docs(event_badge): comment why enable is omitted on import updates

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.ae_brief

@@ -0,0 +1,15 @@
+# Aether Project Brief: aether_api_fastapi
+**Last Updated:** 2026-02-09 19:09:01
+**Current Agent:** mcp_agent
+
+## 🛠️ What I Just Did
+Hardened Data Store search security (account isolation), ensured ID Vision compliance in Data Store search results, refactored and standardized the Data Store E2E test suite, and updated tests/README.md with suite-wide standards.
+
+## 🚧 Current Blockers
+None. V3.1 roadmap is clear.
+
+## ➡️ Exact Next Steps
+Begin [V3.1] ID Vision alignment for Person and Organization modules. Mark legacy V2 routers as [DEPRECATED] to streamline removal for the v3.1 release.
+
+---
+*Generated by ae_brief*

.dockerignore

@@ -0,0 +1,23 @@
+# Docker ignore file for Aether API
+environment/
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+.Python
+env/
+venv/
+.venv/
+pip-log.txt
+pip-delete-this-directory.txt
+.pytest_cache/
+.coverage
+htmlcov/
+.git
+.gitignore
+.dockerignore
+documentation/
+tests/
+logs/
+temp/
+# Don't ignore requirements.txt or Dockerfile!

.gitignore

@@ -83,6 +83,7 @@ celerybeat-schedule
 
 # Environments
 .env
+.env.dev
 .venv
 env/
 venv/
@@ -111,24 +112,36 @@ environment/
 Thumbs.db
 
 # Added by Scott Idem
+# Updated 2024-10-09
 # https://github.com/github/gitignore
+
 *.sock
-*.csv
+*.bak
-*.xlsx
-#*.pdf
 *.cfg
 *.ini
-*.bak
+*.kate-swp
 *.pid
+
+*.csv
+# *.pdf
+*.xlsx
+
+.directory
+.vscode
 flask_config.py
 config.py
-#config.cfg
+!app/config.py
-#users.cfg
+# config.cfg
-.directory
+# users.cfg
-tmp/
+
-temp/
+backups/
-log/
 development/
+log/
+logs/
 myapp/files/
 myapp/file_distribution/
-.vscode
+temp/
+tmp/
+
+# Added 2026-03-23
+gunicorn.ctl

Dockerfile

@@ -0,0 +1,36 @@
+# syntax=docker/dockerfile:1
+# Aether API - FastAPI + Gunicorn
+FROM tiangolo/uvicorn-gunicorn-fastapi:python3.11
+
+LABEL maintainer="Scott Idem <scott.idem@oneskyit.com>"
+
+# 1. Install OS dependencies FIRST.
+# These are the slowest to install and change the least.
+# Doing this before WORKDIR or any COPY ensures maximum caching.
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+    imagemagick ffmpeg curl poppler-utils && \
+    rm -rf /var/lib/apt/lists/*
+
+# 2. Set the working directory
+WORKDIR /srv/aether_api
+
+# 3. Install Python requirements
+# We only copy requirements.txt first to keep the pip install layer cached
+# as long as the dependencies themselves don't change.
+COPY requirements.txt /tmp/requirements.txt
+RUN pip install --no-cache-dir -r /tmp/requirements.txt
+
+# 4. Create a reference of actual installed versions
+RUN pip freeze > /tmp/aether_fastapi_requirements_current.txt
+
+# NOTE: The application source is mounted as a volume in docker-compose.yml
+# for real-time development. We don't COPY the source here to keep the
+# image generic and the build near-instant when code changes.
+
+# Docker health check — verifies DB + Redis connectivity via the /health route.
+HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
+    # CMD curl -f http://localhost/health || exit 1
+    CMD curl -f http://localhost:5005/health || exit 1
+
+CMD ["gunicorn", "--conf", "/conf/gunicorn_fastapi_conf.py"]

GEMINI.md

@@ -0,0 +1,40 @@
+# Aether Backend Agent Context: Gemini CLI Standard
+> **Role:** Aether API Orchestrator (Backend & System Architecture)
+> **Location:** GEMINI.md (Project Root)
+
+## 🚨 MANDATORY PROTOCOL
+You must follow the safety, testing, and coordination standards defined in:
+`documentation/GUIDE__DEVELOPMENT.md`
+
+---
+
+## 🏗️ Technical Domain: Aether Backend
+### Stack & Infrastructure
+- **API Framework:** FastAPI (v0.95+)
+- **Validation:** Pydantic V1 (Strict)
+- **Database:** MariaDB + SQLAlchemy (v1.4+)
+- **ID Standard:** "ID Vision" - Public String IDs (`id_random`) mapping to hidden Internal Integer IDs.
+- **Communication:** RAR Protocol (Request -> Ack -> Result).
+
+### V3 CRUD Architecture
+- Enforce parallel CRUD and Search under `/v3/crud/`.
+- **Registry Pattern:** Object definitions in `app/object_definitions/`, logic in `app/methods/`.
+- **Visibility:** Use `Field(exclude=True)` in Pydantic to hide internal paths from public output.
+
+### Specialized Logic
+- **Universal ID Resolution:** Resolve container IDs (e.g., `event_file`) to physical binaries in actions.
+- **Relational Joins:** Prefer SQL `INNER JOIN` in Views (`v_`) over duplicating columns in tables.
+- **Heal-on-Read:** Standardize fallback resolution for relational IDs during GET operations.
+
+## 🧠 Technical Learnings
+- **Harden `root_validator`:** Ensure pre-validation logic doesn't delete integer IDs during ID Vision resolution.
+- **Pydantic worker boot failures:** Watch for `ValueError`, `NameError`, and `KeyError` during startup.
+- **Inherited Context:** Account context for child objects should be inherited via View joins.
+- **Lookup Hierarchy:** Implemented `ROW_NUMBER() OVER` logic for tiered overrides (Object > Account > Global).
+- **Vision Comparison:** Discovered `load_site_obj` returns Random IDs for accounts; comparison in router must use `account_id_random` strings for reliable 403 authorization.
+
+## 🤝 Coordination & Continuity
+- **Handshake:** Use the `message` tool to notify the Frontend Agent of API changes.
+- **Active Tasks:** Track your progress in `documentation/TODO__Agents.md`.
+- **Lookup Milestone:** Batch 1 (Country, Subdivision, Timezone) complete. V3.1 goal set for Batch 2 and Novi-Mailman bridge.
+- **Learning:** Review `ARCH__V3_CORE_STANDARDS.md` for V4 lifecycle field migration planning.

README.md

@@ -1,2 +1,176 @@
-# Aether API Python FastAPI
+
-The Aether API was created and is being developed by Scott Idem using the Python FastAPI framework.
+# Aether API v3.00.20 (FastAPI)
+
+The **Aether API** is a high-performance, multi-tenant backend for the Aether Platform, built on Python **FastAPI**. It powers both legacy and modern (V3/V4) applications, and is now fully containerized for robust, scalable deployment.
+
+---
+
+
+---
+
+## 🏗️ Architecture Overview
+
+The API is in transition from legacy (V1/V2) to the modern **V3 CRUD Architecture**. All new development follows V3 standards.
+
+
+### V3 CRUD (Modern)
+- **Path:** `/v3/crud/`
+- **Principles:**
+    - **String IDs:** All public APIs use `id_random` (URL-safe string IDs); internal integer IDs are hidden.
+    - **Nested URLs:** Parent-child relationships enforced in URL structure.
+    - **Advanced Search:** POST-based, recursive, with standardized operators.
+    - **Schema Discovery:** Dynamic model/database introspection at `/v3/crud/{obj_type}/schema`.
+    - **Granular Dependencies:** Specialized FastAPI dependencies for account context, pagination, filtering, serialization.
+
+
+### V3 Actions
+- **Path:** `/v3/action/`
+- Handles complex/atomic business logic and binary operations outside standard CRUD.
+- **Features:**
+    - **Atomic Event Uploads:** File storage + event relations in one request.
+    - **Content-Addressable Downloads:** SHA256-based file retrieval for high-performance caching.
+    - **Intelligent ID Resolution:** Download endpoints auto-resolve container IDs.
+
+
+### Legacy API (V1/V2)
+- **Paths:** `/`, `/api/`, `/crud/`, `/v2/crud/`
+- Maintained for backward compatibility, but being systematically deprecated. Accessing legacy routes triggers a warning in logs.
+
+---
+
+
+## 🛠️ Core Technologies
+- **Framework:** FastAPI (v0.95.1+)
+- **Database:** MariaDB (Docker, shared) + SQLAlchemy (v1.4.52)
+- **Caching/ID Resolution:** Redis
+- **Security:** JWT (JSON Web Tokens), API Key Machine Auth
+- **Logging:** Structured, module-level, with rotation
+
+---
+
+
+## 🚀 Quick Start
+
+The Aether API is designed for containerized deployment as part of the unified Aether Docker environment. For full-stack orchestration, see the documentation in the `aether_container_env` project.
+
+### Prerequisites
+- Docker & Docker Compose (for containerized use)
+- Python 3.9+ (for local-only development)
+
+### Local Development (Optional)
+You can run the API locally for debugging:
+```bash
+virtualenv environment
+source environment/bin/activate
+pip install -r requirements.txt
+uvicorn app.main:app --host 0.0.0.0 --port 5005 --reload
+```
+See [GUIDE__LOCAL_DEVELOPMENT.md](documentation/GUIDE__LOCAL_DEVELOPMENT.md) for details.
+
+### Docker Usage
+The API is run and managed via Docker Compose as part of the full Aether stack. Refer to the `aether_container_env` project for orchestration, environment setup, and advanced deployment instructions.
+
+### Service Endpoints (Default Ports)
+- **API Docs:** https://dev-api.oneskyit.com/docs
+- **Frontend:** http://localhost:8888
+- **phpMyAdmin:** http://localhost:8081 (if enabled)
+- **Logs (Dozzle):** http://localhost:8881
+
+
+---
+
+## 🗄️ Database & Backups
+
+All database operations are managed via Docker scripts in `aether_container_env/`:
+- **Backup:** `./backup_db.sh` (saves to `backups/`)
+- **Restore:** `./restore_db.sh [backup_file.gz]`
+- **Export:** `./export_db.sh` (conference-ready backup)
+- **Automated Import:** Drop file in `backups/import/` and run `./check_and_import.sh`
+
+See [GUIDE__DEPLOYMENT_MANUAL.md](documentation/GUIDE__DEPLOYMENT_MANUAL.md) for full deployment and backup/restore instructions.
+
+---
+
+
+---
+
+## 📂 Documentation Index
+
+### **Architecture & Standards**
+- [V3 Core Architecture](documentation/ARCH__V3_CORE.md): Modular structure and boot sequence.
+- [V3 Development Standards](documentation/ARCH__V3_DEVELOPMENT_STANDARDS.md): ID Vision, inheritance, and naming rules.
+- [Unified Agent Arch](documentation/ARCH__UNIFIED_AGENT.md): Vision for cross-stack AI agent awareness.
+
+### **Integration Guides**
+- [V3 Frontend API Guide](documentation/GUIDE__AE_API_V3_for_Frontend.md): How to use the V3 CRUD, Search, and Action endpoints.
+- [V3 Frontend Websockets Guide](documentation/GUIDE__AE_API_V3_for_Frontend_websockets.md): Websocket integration patterns.
+- [Frontend Code Samples](documentation/FRONTEND_API_SAMPLES.md): TypeScript snippets for common API calls.
+
+### **Security**
+- [Project Security Hardening](documentation/PLAN__SECURITY_HARDENING.md): Path towards cryptographic JWT verification.
+
+---
+
+
+## 🧪 Testing Suite
+Tests are under `tests/`:
+- **Unit:** `tests/unit/` (mocked logic)
+- **Integration:** `tests/integration/` (DB/Redis connectivity)
+- **E2E:** `tests/e2e/` (API validation)
+- **Docs:** [tests/README.md](tests/README.md)
+
+---
+
+
+## 🚧 Status & Work in Progress
+
+### Active Workstreams
+- **API Deprecation:** Pruning orphaned routers/methods
+- **ID Vision:** String-ID standardization (Phase 2 complete)
+- **V3 Migration:** Atomic event actions, hash-based file retrieval
+
+### Known Issues
+- **Badge Rendering:** Corrupted numeric `id` fields in `event_badge_template` can cause template load failures
+- **Websockets:** Legacy modules need unification and stability improvements
+- **Intermittent Timeouts:** Some E2E tests occasionally reproduce 403s/timeouts on nested GET calls
+
+---
+
+
+---
+
+## 📜 Release Snapshot
+Current Baseline: **`release/2026-01-28-v3_prod-snapshot`** (Stable v3.0.99)
+
+---
+
+## 🔐 Security & Access
+- **SSH Required:** All git operations now require SSH (Bitbucket app passwords deprecated June 2026). See your Gitea or Bitbucket account for adding SSH keys.
+- **Never commit secrets:** `.env` and credentials are git-ignored.
+- **JWT Key:** Ensure `AE_API_JWT_KEY` is unique and high-entropy in production.
+- **.env precedence:** API uses `.env` credentials for core infra (SMTP/DB) over DB settings.
+
+---
+
+## 🧑‍💻 Management & Operations
+- **Restart API:** `docker compose restart ae_api`
+- **Restart Frontend:** `docker compose restart ae_app`
+- **Rebuild everything:** `docker compose up -d --build`
+- **Logs:** http://localhost:8881 (Dozzle)
+- **phpMyAdmin:** http://localhost:8081 (if enabled)
+
+---
+
+## 🏠 Directory Map (Key Mounts)
+- `conf/` — Nginx/Gunicorn config templates
+- `logs/` — Centralized logs
+- `srv/` — Data/source code mounts
+- `scripts/` — Automation scripts
+- `backups/` — MariaDB snapshots
+
+---
+
+## 📝 Notes
+- For multi-stack setups, ensure unique `AE_NETWORK_NAME` and `CONTAINER_` prefixes in `.env`.
+- All stacks must connect to `aether_shared_net` for shared DB/Redis.
+- See Docker env README and CHEATSHEET for advanced orchestration and troubleshooting.

admin/documentation/gunicorn.service.default

@@ -1,24 +0,0 @@
-[Unit]
-Description=gunicorn daemon
-Requires=gunicorn.socket
-After=network.target
-
-[Service]
-Type=notify
-# the specific user that our service will run as
-User=root
-Group=root
-# another option for an even more restricted service is
-# DynamicUser=yes
-# see http://0pointer.net/blog/dynamic-users-with-systemd.html
-RuntimeDirectory=gunicorn
-WorkingDirectory=/srv/http/dev_fastapi.oneskyit.com
-Environment="PATH=/srv/http/dev_fastapi.oneskyit.com/environment/bin"
-ExecStart=/srv/http/dev_fastapi.oneskyit.com/environment/bin/gunicorn --bind unix:/srv/http/dev_fastapi.oneskyit.com/gunicorn.sock -m 007 app.main:app --workers 4 -k uvicorn.workers.UvicornWorker --access-logfile admin/log/access.log --error-logfile admin/log/error.log, --log-file admin/log/log.log --capture-output --keep-alive 5
-ExecReload=/bin/kill -s HUP $MAINPID
-KillMode=mixed
-TimeoutStopSec=5
-PrivateTmp=true
-
-[Install]
-WantedBy=multi-user.target

admin/documentation/gunicorn.socket.default

@@ -1,14 +0,0 @@
-[Unit]
-Description=gunicorn socket
-
-[Socket]
-ListenStream=/run/gunicorn.sock
-# Our service won't need permissions for the socket, since it
-# inherits the file descriptor by socket activation
-# only the nginx daemon will need access to the socket
-User=http
-# Optionally restrict the socket permissions even more.
-# Mode=600
-
-[Install]
-WantedBy=sockets.target

admin/documentation/nginx_fastapi_server.example.default

@@ -1,85 +0,0 @@
-server {
-  access_log  /var/log/nginx/access_dev_fastapi.oneskyit.com.log;
-
-  listen 443 ssl; # managed by Certbot
-  listen [::]:443 ssl http2; # managed by Certbot
-  #listen 443 http3 reuseport;  # UDP listener for QUIC+HTTP/3
-  server_name dev-fastapi.oneskyit.com;
-
-  ssl_certificate /etc/letsencrypt/live/oneskyit.com-0001/fullchain.pem; # managed by Certbot
-  ssl_certificate_key /etc/letsencrypt/live/oneskyit.com-0001/privkey.pem; # managed by Certbot
-
-  include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
-  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
-
-  #add_header Alt-Svc 'quic=":443"'; # Advertise that QUIC is available
-  #add_header QUIC-Status $quic;     # Sent when QUIC was used
-
-  include brotli.conf;
-  include gzip.conf;
-
-  client_max_body_size 4096M; # or 4G
-
-  location / {
-        proxy_set_header Host $http_host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-
-        proxy_redirect off;
-        proxy_buffering off;
-
-        proxy_pass http://unix:/run/gunicorn.sock;
-  }
-
-  location /ws {
-        proxy_set_header Host $http_host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-
-        proxy_http_version 1.1;
-
-        proxy_redirect off;
-        proxy_buffering off;
-
-        proxy_set_header Upgrade $http_upgrade;
-        proxy_set_header Connection "upgrade";
-
-        #proxy_read_timeout 600;
-        #proxy_headers_hash_max_size 1024;
-
-        proxy_pass http://unix:/run/gunicorn.sock;
-  }
-
-  location /ws_redis {
-        proxy_set_header Host $http_host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-
-        proxy_http_version 1.1;
-
-        proxy_redirect off;
-        proxy_buffering off;
-
-        proxy_set_header Upgrade $http_upgrade;
-        proxy_set_header Connection "upgrade";
-
-        #proxy_read_timeout 600;
-        #proxy_headers_hash_max_size 1024;
-
-        proxy_pass http://unix:/run/gunicorn.sock;
-  }
-}
-
-server {
-  if ($host = dev-fastapi.oneskyit.com) {
-    return 301 https://$host$request_uri;
-  } # managed by Certbot
-
-  listen 80;
-  listen [::]:80;
-  server_name dev-fastapi.oneskyit.com;
-  return 404; # managed by Certbot
-}

admin/documentation/run locally.txt

@@ -1,55 +0,0 @@
-# Go to root of application
-cd ~/path/to/directory/my_application/
-
-# Create new application environment
-virtualenv environment
-
-# Activate application environment
-source environment/bin/activate
-
-# Install application requirements
-pip install -r admin/requirements.txt
-pip install --upgrade --force-reinstall -r admin/requirements.txt
-pip install --ignore-installed -r admin/requirements.txt
-pip list
-
-# Start application
-uvicorn app.main:app --host 0.0.0.0 --port 5005 --reload
-
-# View app
-http://localhost:5005
-
-# Deactivate environment when done
-deactivate
-
-
-# Use git
-# Go to root of application
-cd ~/path/to/directory/my_application/
-
-git init
-
-git remote add origin https://scott_idem@bitbucket.org/oneskyit/one-sky-it-api.git
-git add .
-git commit -m 'Initial commit'
-
-git push -u origin master
-git push -u origin development
-git push -u origin new-branch-name
-
-# List branches
-git branch -a
-
-# Create new branch
-git branch new-branch-name
-
-# Switch branch
-git switch new-branch-name
-
-
-# Clone from Bitbucket:
-git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-api-fastapi.git /srv/http/the_path_to_create
-
-
-
-gunicorn --bind unix:/home/scott/OSIT_dev/aether_api_fastapi/gunicorn.sock --umask 007 app.main:app --workers 2 --worker-class uvicorn.workers.UvicornWorker --access-logfile admin/log/access.log --error-logfile admin/log/error.log, --log-file admin/log/log.log --capture-output --keep-alive 5 --reload

admin/documentation/setup_server.txt

@@ -1,42 +0,0 @@
-sudo git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-api-fastapi.git /srv/http/dev_fastapi.oneskyit.com
-
-sudo mkdir admin/log
-
-sudo ls -lha /srv/http/
-sudo chown http:http -R /srv/http/dev_fastapi.oneskyit.com/
-sudo chmod 775 -R /srv/http/dev_fastapi.oneskyit.com/
-sudo ls -lha /srv/http/
-
-cd /srv/http/dev_fastapi.oneskyit.com/
-rm .gitignore
-
-git branch -a
-git switch development
-
-virtualenv environment
-source environment/bin/activate
-pip install -U -r admin/requirements.txt
-
-sudo vim /etc/systemd/system/gunicorn.socket
-sudo vim /etc/systemd/system/gunicorn.service
-sudo systemctl daemon-reload
-sudo systemctl enable gunicorn.socket
-sudo systemctl start gunicorn.socket
-sudo systemctl status gunicorn.socket
-sudo systemctl status gunicorn.service
-
-# Do not: sudo systemctl enable gunicorn.service
-# Do not? sudo systemctl start gunicorn.service
-
-sudo vim /etc/nginx/sites-available/dev_fastapi.oneskyit.com
-sudo ln -s /etc/nginx/sites-available/dev_fastapi.oneskyit.com /etc/nginx/sites-enabled/dev_fastapi.oneskyit.com
-
-sudo systemctl restart nginx.service
-sudo systemctl status nginx.service
-
-# Troubleshooting:
-systemctl list-units --type=service --state=active
-systemctl list-units --type=service --state=running
-
-sudo systemctl | grep running
-sudo systemctl list-unit-files | grep enabled

aether_api_fastapi.code-workspace

@@ -4,5 +4,9 @@
 			"path": "."
 		}
 	],
-	"settings": {}
+	"settings": {
+		"cSpell.words": [
+			"poolclass"
+		]
+	}
 }

app/ae_obj_types_def.py

@@ -0,0 +1,88 @@
+"""
+This file centralizes the object type definitions for the Aether API.
+It merges definitions from modular files in app/object_definitions/ to support
+both V2 (legacy) and V3 CRUD operations.
+
+🛑 REMINDER: Any field added to 'searchable_fields' in modular definitions
+MUST also exist in the corresponding Pydantic model (mdl) and SQL View (tbl)
+to prevent serialization errors during V3 Search operations.
+"""
+
+# Restore blanket imports for legacy compatibility (V1 and V2 rely on these)
+from app.models.response_models import *
+from app.models.api_crud_models import *
+from app.models.account_models import *
+from app.models.account_cfg_models import *
+from app.models.activity_log_models import *
+from app.models.address_models import *
+from app.models.archive_models import *
+from app.models.archive_content_models import *
+from app.models.contact_models import *
+from app.models.cont_edu_cert_models import *
+from app.models.cont_edu_cert_person_models import *
+from app.models.data_store_models import *
+from app.models.event_models import *
+from app.models.event_abstract_models import *
+from app.models.event_badge_models import *
+from app.models.event_badge_template_models import *
+from app.models.event_device_models import *
+from app.models.event_exhibit_models import *
+from app.models.event_exhibit_tracking_models import *
+from app.models.event_file_models import *
+from app.models.event_location_models import *
+from app.models.event_person_models import *
+from app.models.event_person_tracking_models import *
+from app.models.event_presentation_models import *
+from app.models.event_presenter_models import *
+from app.models.event_registration_models import *
+from app.models.event_session_models import *
+from app.models.event_track_models import *
+from app.models.grant_models import *
+from app.models.hosted_file_models import *
+from app.models.journal_models import *
+from app.models.journal_entry_models import *
+from app.models.log_client_viewing_models import Log_Client_Viewing_Base
+from app.models.membership_cfg_models import *
+from app.models.membership_group_models import *
+from app.models.membership_person_group_models import *
+from app.models.membership_person_models import *
+from app.models.membership_person_profile_models import *
+from app.models.membership_type_models import *
+from app.models.membership_person_type_models import *
+from app.models.order_models import *
+from app.models.order_cart_models import *
+from app.models.organization_models import *
+from app.models.page_models import *
+from app.models.person_models import *
+from app.models.product_models import *
+from app.models.post_models import *
+from app.models.post_comment_models import *
+from app.models.site_models import *
+from app.models.site_domain_models import *
+from app.models.sponsorship_cfg_models import *
+from app.models.sponsorship_models import *
+from app.models.user_models import *
+from app.models.user_role_models import *
+from app.models.e_stripe_models import *
+
+# Modularized definitions
+from app.object_definitions.core import core_obj_li
+from app.object_definitions.events import event_obj_li
+from app.object_definitions.journals import journal_obj_li
+from app.object_definitions.orders import order_obj_li
+from app.object_definitions.cms import cms_obj_li
+from app.object_definitions.lookups import lu_obj_li
+from app.object_definitions.membership import membership_obj_li
+from app.object_definitions.other import other_obj_li
+
+# Merge all modular definitions into the main registry
+obj_type_kv_li = {
+    **core_obj_li,
+    **event_obj_li,
+    **journal_obj_li,
+    **order_obj_li,
+    **cms_obj_li,
+    **lu_obj_li,
+    **membership_obj_li,
+    **other_obj_li,
+}

app/config.py

@@ -0,0 +1,112 @@
+# Configuration for the Aether FastAPI application.
+# All settings are read directly from environment variables (injected by Docker via .env).
+from pydantic import BaseSettings, Field
+from typing import Any, Dict, List
+
+
+class Settings(BaseSettings):
+
+    # --- Application ---
+    APP_NAME: str = "Aether API (FastAPI)"
+
+    # --- Aether Shared Config (DB-driven bootstrap) ---
+    AE_CFG_ID: int = Field(0, env='AE_CFG_ID')
+
+    # --- JWT ---
+    JWT_KEY: str = Field('EHmSXZFKfMEW65E8kxCKmQ', env='AE_API_JWT_KEY')
+
+    # --- Database ---
+    # These flat fields are mutated by the bootstrap process in main.py (lifespan),
+    # which swaps in production credentials after reading from the cfg table.
+    DB_SERVER: str = Field('mariadb', env='AE_DB_SERVER')
+    DB_PORT:   str = Field('3306',    env='AE_DB_PORT')
+    DB_NAME:   str = Field('aether_dev', env='AE_DB_NAME')
+    DB_USER:   str = Field('aether_dev', env='AE_DB_USERNAME')
+    DB_PASS:   str = Field('',           env='AE_DB_PASSWORD')
+
+    # Connection tuning
+    DB_CONNECT_TIMEOUT:  int = Field(20,   env='AE_DB_CONNECTION_TIMEOUT')
+    DB_POOL_RECYCLE:     int = Field(1800, env='AE_DB_POOL_RECYCLE')
+    DB_POOL_SIZE:        int = Field(10,   env='AE_DB_POOL_SIZE')
+    DB_POOL_MAX_OVERFLOW: int = Field(20,  env='AE_DB_POOL_MAX_OVERFLOW')
+
+    # --- Logging ---
+    LOG_PATH_APP: str = Field('/logs/aether_api.log', env='AE_API_LOG_PATH')
+
+    # --- Redis ---
+    REDIS_SERVER: str = Field('redis', env='AE_REDIS_SERVER')
+    REDIS_PORT:   str = Field('6379',  env='AE_REDIS_PORT')
+
+    # --- SMTP ---
+    SMTP_SERVER:   str = Field('linode.oneskyit.com',        env='AE_SMTP_SERVER')
+    SMTP_PORT:     str = Field('465',                        env='AE_SMTP_PORT')
+    SMTP_USERNAME: str = Field('send_mail',                  env='AE_SMTP_USERNAME')
+    SMTP_PASSWORD: str = Field('set-in-ae-sql-db-cnf-tbl',  env='AE_SMTP_PASSWORD')
+
+    # --- File Storage ---
+    FILES_PATH_ROOT: str = Field('/srv/hosted_files', env='AE_FILES_PATH_ROOT')
+    FILES_PATH_TMP:  str = Field('/srv/hosted_tmp',   env='AE_FILES_PATH_TMP')
+
+    # --- CORS ---
+    ORIGINS_REGEX: str = Field(
+        r'(https://.*\.oneskyit\.com)|(https://.*\.oneskyit\.com:4443)',
+        env='AE_API_ORIGINS_REGEX'
+    )
+    ORIGINS: List[str] = ['https://oneskyit.com']
+
+    # -------------------------------------------------------------------------
+    # Computed properties — maintain backwards-compatible dict interface used
+    # throughout the app (e.g. settings.DB['server'], settings.REDIS['port']).
+    # -------------------------------------------------------------------------
+
+    @property
+    def AETHER_CFG(self) -> Dict[str, Any]:
+        return {'id': self.AE_CFG_ID}
+
+    @property
+    def SQLALCHEMY_DB_URI(self) -> str:
+        return f"mysql://{self.DB_USER}:{self.DB_PASS}@{self.DB_SERVER}:{self.DB_PORT}/{self.DB_NAME}"
+
+    @property
+    def DB(self) -> Dict[str, Any]:
+        return {
+            'server':          self.DB_SERVER,
+            'port':            self.DB_PORT,
+            'name':            self.DB_NAME,
+            'username':        self.DB_USER,
+            'password':        self.DB_PASS,
+            'connect_timeout':  self.DB_CONNECT_TIMEOUT,
+            'pool_recycle':     self.DB_POOL_RECYCLE,
+            'pool_size':        self.DB_POOL_SIZE,
+            'max_overflow':     self.DB_POOL_MAX_OVERFLOW,
+        }
+
+    @property
+    def LOG_PATH(self) -> Dict[str, str]:
+        return {'app': self.LOG_PATH_APP}
+
+    @property
+    def REDIS(self) -> Dict[str, str]:
+        return {'server': self.REDIS_SERVER, 'port': self.REDIS_PORT}
+
+    @property
+    def SMTP(self) -> Dict[str, str]:
+        return {
+            'server':   self.SMTP_SERVER,
+            'port':     self.SMTP_PORT,
+            'username': self.SMTP_USERNAME,
+            'password': self.SMTP_PASSWORD,
+        }
+
+    @property
+    def FILES_PATH(self) -> Dict[str, str]:
+        return {
+            'hosted_files_root': self.FILES_PATH_ROOT,
+            'hosted_tmp_root':   self.FILES_PATH_TMP,
+        }
+
+    class Config:
+        case_sensitive = False
+
+
+settings = Settings()

app/config.py.default

@@ -1,85 +1,71 @@
 # Configuration file for this FastAPI app.
 import os
-from pydantic import AnyHttpUrl, BaseSettings, EmailStr, HttpUrl, PostgresDsn, validator
+from pydantic import BaseSettings
 from typing import Any, Dict, List, Optional, Union
 
-
-# ### ### #
-
-
 class Settings(BaseSettings):
-    AETHER_CFG = {}
+    AETHER_CFG: Dict[str, Any] = {
-    AETHER_CFG['id'] = 0
+        "id": os.getenv('AE_CFG_ID', '0')
-    # AETHER_CFG['api_id'] = 0 # NOT CURRENTLY NEED OR USED
+    }
-
+    
-    JWT_KEY = '' # 22 characters; super secret Aether JWT signing key
+    JWT_KEY: str = os.getenv('AE_API_JWT_KEY', 'fake-super-secret-token')
-
-    # APP_NAME: str = "Aether API (FastAPI)"
-    # SUPER_EMAIL: EmailStr = 'Aether.Super@oneskyit.com'
-
 
     # Database Connection
-    DB = {}
+    DB_SERVER: str = os.getenv('AE_DB_SERVER', 'mariadb')
-    DB['server'] = 'db.oneskyit.com'
+    DB_PORT: str = os.getenv('AE_DB_PORT', '3306')
-    DB['port'] = '3306' # default = 3306
+    DB_NAME: str = os.getenv('AE_DB_NAME', 'aether_dev')
-    DB['name'] = 'aether_default'
+    DB_USER: str = os.getenv('AE_DB_USERNAME', 'aether_dev')
-    DB['username'] = ''
+    DB_PASS: str = os.getenv('AE_DB_PASSWORD', '')
-    DB['password'] = ''
-    SQLALCHEMY_DB_URI = 'mysql://'+DB['username']+':'+DB['password']+'@'+DB['server']+'/'+DB['name']
 
-    DB['wait_timeout'] = int(os.getenv('AE_DB_WAIT_TIMEOUT', 1800)) # default = 28800; Time (seconds) that the server waits for a connection to become active before closing it.
+    @property
-    DB['connect_timeout'] = int(os.getenv('AE_DB_CONNECTION_TIMEOUT', 20)) # default = 10; Time (seconds) that the server waits for a connection to become active before closing it.
+    def SQLALCHEMY_DB_URI(self) -> str:
-    DB['pool_recycle'] = int(os.getenv('AE_DB_POOL_RECYCLE', 1800)) # default = ?; Related to SQLAlchemy
+        return f"mysql://{self.DB_USER}:{self.DB_PASS}@{self.DB_SERVER}:{self.DB_PORT}/{self.DB_NAME}"
 
+    @property
+    def DB(self) -> Dict[str, Any]:
+        return {
+            "server": self.DB_SERVER,
+            "port": self.DB_PORT,
+            "name": self.DB_NAME,
+            "username": self.DB_USER,
+            "password": self.DB_PASS,
+            "connect_timeout": int(os.getenv('AE_DB_CONNECTION_TIMEOUT', 20)),
+            "pool_recycle": int(os.getenv('AE_DB_POOL_RECYCLE', 1800))
+        }
 
-    # Aether API log files paths
+    # Logging
-    LOG_PATH = {}
+    LOG_PATH: Dict[str, str] = {
-    LOG_PATH['app'] = '/logs/aether_api.log' # 'admin/log/app.log', '../../logs/aether_api.log'
+        "app": os.getenv('AE_API_LOG_PATH', '/logs/aether_api.log')
-    # LOG_PATH['app_warning'] = '/logs/aether_api_warning.log' # 'admin/log/app_warning.log' '../../logs/aether_api_warning.log'
+    }
-
 
     # Redis
-    REDIS = {}
+    REDIS: Dict[str, str] = {
-    REDIS['server'] = 'localhost' # 'localhost' 'redis'
+        "server": os.getenv('AE_REDIS_SERVER', 'redis'),
-    REDIS['port'] = '6379'
+        "port": os.getenv('AE_REDIS_PORT', '6379')
-
+    }
 
+    # --- CRITICAL CONFIGURATIONS ---
     # Send SMTP Email
-    SMTP = {}
+    SMTP: Dict[str, str] = {
-    # server
+        "server": os.getenv('AE_SMTP_SERVER', ''),
-    # port
+        "port": os.getenv('AE_SMTP_PORT', '465'),
-    # username
+        "username": os.getenv('AE_SMTP_USERNAME', ''),
-    # password
+        "password": os.getenv('AE_SMTP_PASSWORD', '')
-
+    }
 
     # Server Hosted File Paths
-    FILES_PATH = {}
+    FILES_PATH: Dict[str, str] = {
-    # hosted_files_root
+        "hosted_files_root": os.getenv('AE_FILES_PATH_ROOT', '/srv/hosted_files'),
-    # hosted_tmp_root
+        "hosted_tmp_root": os.getenv('AE_FILES_PATH_TMP', '/srv/hosted_tmp')
+    }
+    # --- END CRITICAL CONFIGURATIONS ---
 
-
+    # CORS
-    # CORS Origins
+    ORIGINS_REGEX: str = os.getenv('AE_API_ORIGINS_REGEX', '(https://.*\.oneskyit\.com)|(https://.*\.oneskyit\.com:4443)')
-    ORIGINS_REGEX = '(https://.*\.oneskyit\.com)|(http://.*\.oneskyit\.com:8181)|(https://.*\.oneskyit\.com:4443)|(https://.*\.oneskyit\.com:8443)'
+    ORIGINS: List[str] = [
-    # A reasonable, but fairly open example regular expression for the CORS origins:
-    # '(https://.*\.oneskyit\.com)|(http://.*\.oneskyit\.com)|(http://.*\.oneskyit\.com:8181)|(https://.*\.oneskyit\.com:8443)|(http://.*\.oneskyit\.local)|(http://.*\.oneskyit\.local:5000)|(http://.*.localhost)|(http://.*.localhost:5000)|(http://.*.localhost:8181)'
-
-    ORIGINS = [
         'https://oneskyit.com',
-        # 'http://app-local.oneskyit.com',
-        # 'http://192.168.32.20:3000',
-        # 'http://192.168.32.20:8080',
-
-        # 'http://localhost',
-        # 'http://localhost:3000',
-        # 'http://localhost:5000',
-        # 'http://localhost:8080',
-        # 'http://localhost:7800',
-        # 'http://localhost:8888',
-
         'http://fastapi.localhost',
-
         'http://svelte.oneskyit.local:5555',
-        ]
+    ]
-
 
 settings = Settings()

app/db_connection.py

@@ -0,0 +1,28 @@
+"""
+Independent database connection module to prevent circular imports.
+"""
+import logging
+from sqlalchemy import create_engine
+from app.config import settings
+
+# Use local logger to avoid importing app.log (which might create cycles)
+log = logging.getLogger(__name__)
+
+db_uri = settings.SQLALCHEMY_DB_URI
+engine = create_engine(
+    url = db_uri,
+    echo = False,
+    pool_use_lifo = True,
+    pool_pre_ping = True,
+    pool_recycle = settings.DB['pool_recycle'],
+    isolation_level = 'READ COMMITTED',
+    connect_args = {'connect_timeout': settings.DB['connect_timeout']}
+)
+
+log.info('DB Connection initializing...')
+db = None
+try:
+    db = engine.connect()
+    log.info(f'Connected to database: {db_uri}')
+except Exception:
+    log.exception('Could not connect to database.')

app/lib_api_crud_v3.py

@@ -0,0 +1,266 @@
+from typing import Any, Dict, Optional, Union
+import json
+import logging
+import re
+
+from app.lib_general_v3 import AccountContext, StatusFilterParams
+from app.models.error_models import StandardError
+
+log = logging.getLogger(__name__)
+
+def apply_vision_id_fix(resp_data: dict, obj_type: str, by_alias: bool) -> dict:
+    """
+    V3 contract: {obj_type}_id in responses must be the random string, never the DB integer.
+    Applies to models not yet migrated to the Vision ID pattern (root_validator).
+    Safe to call on already-migrated models — no-op if the value is already a string.
+    """
+    _id_key = f'{obj_type}_id' if by_alias else 'id'
+    _rand_key = f'{obj_type}_id_random' if by_alias else 'id_random'
+    if isinstance(resp_data.get(_id_key), int) and resp_data.get(_rand_key):
+        resp_data[_id_key] = resp_data[_rand_key]
+    return resp_data
+
+
+def format_db_error(raw_error: str) -> StandardError:
+    """
+    Parses raw SQLAlchemy/MariaDB errors into structured StandardError objects.
+    """
+    if not raw_error:
+        return StandardError(
+            category="unknown",
+            message="An unspecified database error occurred."
+        )
+    
+    # 1. Extract Error Code and Message using regex
+    # Standard MariaDB pattern: (code, "message")
+    code = None
+    message = raw_error
+    recoverable = False
+    
+    match = re.search(r'\((\d+),\s*["\'](.*?)["\']\s*\)', raw_error)
+    if match:
+        code = int(match.group(1))
+        message = match.group(2).strip()
+    else:
+        # Fallback: remove all (parenthesized) blocks which often contain codes
+        message = re.sub(r'\(.*?\)', '', raw_error).strip()
+
+    # 2. Categorize based on known MariaDB codes
+    # Ref: https://mariadb.com/kb/en/mariadb-error-codes/
+    if code in [1062]: # Duplicate Entry
+        category = "database_duplicate"
+    elif code in [1451, 1452]: # Foreign Key Constraint
+        category = "database_constraint"
+    elif code in [1045, 2002, 2003, 2006]: # Connection / Auth issues
+        category = "database_connection"
+        recoverable = True
+    elif code in [1054, 1146]: # Unknown column / Table
+        category = "database_schema"
+    elif code == 1364: # Field has no default value — model/schema mismatch
+        category = "database_schema"
+        field_match = re.search(r"Field '([^']+)' doesn't have a default value", message)
+        if field_match:
+            field_name = field_match.group(1)
+            message = f"Schema mismatch: column '{field_name}' is NOT NULL with no default but was not included in the insert. Check the model definition and database schema."
+    else:
+        category = "database"
+
+    return StandardError(
+        category=category,
+        code=code,
+        message=message,
+        recoverable=recoverable,
+        details=raw_error if category == "database" else None # Only include raw details for uncategorized errors
+    )
+
+def check_account_access(sql_result: Any, account: AccountContext, obj_name: str = None) -> bool:
+    """
+    Enforce Multi-Tenant Data Isolation.
+    
+    Verifies that the requested record belongs to the authenticated user's account.
+    Returns True if:
+    - User is a Super User or System (Bypass).
+    - The record's `account_id` matches the user's `account_id`.
+    """
+    if account.super or account.auth_method == 'bypass':
+        return True
+    if not account.account_id:
+        return False
+    
+    res_account_id = None
+    if isinstance(sql_result, dict):
+        if obj_name == 'account':
+            res_account_id = sql_result.get('id')
+        else:
+            res_account_id = sql_result.get('account_id')
+    
+    if res_account_id is not None and res_account_id != account.account_id:
+        return False
+    return True
+
+def apply_forced_account_filter(and_qry_dict: Optional[Dict], account: AccountContext, model: Any, obj_name: str, table_name: str = None) -> Dict:
+    """
+    Secure Search Filtering.
+    
+    Automatically appends an `account_id` filter to database queries to ensure 
+    users only retrieve records associated with their own account.
+    
+    Now schema-aware: checks if the column actually exists in the DB before applying.
+    """
+    forced = and_qry_dict or {}
+    if account.super or account.auth_method == 'bypass':
+        return forced
+    
+    # 1. Determine the target column
+    target_col = 'account_id'
+    if obj_name == 'account':
+        target_col = 'id'
+    
+    # 2. Check if the model even supports it
+    if model and hasattr(model, '__fields__') and target_col not in model.__fields__:
+        return forced
+
+    # 3. If we have a table name, verify the column exists in the physical DB schema
+    # (Important for Views that might exclude account_id for performance/privacy)
+    if table_name:
+        from app import lib_sql_core
+        from sqlalchemy import text
+        try:
+            with lib_sql_core.engine.connect() as conn:
+                conn.execute(text(f"SELECT `{target_col}` FROM `{table_name}` LIMIT 0"))
+            has_col = True
+        except:
+            has_col = False
+        
+        if not has_col:
+            return forced
+        
+    # CRITICAL: Always apply the filter. If account_id is None, it filters for NULL.
+    forced[target_col] = account.account_id
+        
+    return forced
+
+def filter_order_by(order_by_li: Any, model: Any, table_name: str = None) -> Optional[Dict[str, str]]:
+    """
+    Sanitize Sorting Parameters.
+    
+    Prevents SQL injection and logic errors by validating that requested sort columns
+    actually exist in the Pydantic model and/or the database table.
+    """
+    if not order_by_li or not isinstance(order_by_li, dict) or not model:
+        return order_by_li
+    if not hasattr(model, '__fields__'):
+        return order_by_li
+    
+    model_fields = set(model.__fields__.keys())
+    model_fields.update({f.alias for f in model.__fields__.values() if f.alias})
+    filtered = {k: v for k, v in order_by_li.items() if k in model_fields}
+    
+    if table_name and filtered:
+        from app.db_sql import db
+        from sqlalchemy import text
+        final_filtered = {}
+        for column in filtered:
+            try:
+                # Lightweight check to see if column exists in SQL
+                db.execute(text(f"SELECT `{column}` FROM `{table_name}` LIMIT 0"))
+                final_filtered[column] = filtered[column]
+            except Exception:
+                pass
+        filtered = final_filtered
+    return filtered
+
+def get_supported_filters(model: Any, status_filter: StatusFilterParams) -> StatusFilterParams:
+    """
+    Adaptive Status Filtering.
+    
+    Adjusts the default filters (enabled/hidden) based on whether the target object
+    actually supports those concepts (i.e., has those columns).
+    """
+    if not model or not hasattr(model, "__fields__"):
+        return status_filter
+    # We create a new instance to avoid side effects on the dependency object
+    from app.routers.dependencies_v3 import StatusFilterParams as SF
+    adjusted = SF()
+    adjusted.enabled = status_filter.enabled
+    adjusted.hidden = status_filter.hidden
+    
+    if 'enable' not in model.__fields__:
+        adjusted.enabled = 'all'
+    if 'hide' not in model.__fields__:
+        adjusted.hidden = 'all'
+    return adjusted
+
+def safe_json_loads(json_str: Optional[str]) -> Any:
+    if not json_str or json_str == 'undefined': return None
+    try: return json.loads(json_str)
+    except: return None
+
+def sanitize_payload(data: dict, model: Any, ignore_extra: bool = False) -> None:
+    """
+    Sanitizes an input payload before database insertion or update.
+    
+    1. Resolves ID strings to integers:
+       - Handles legacy `*_id_random` fields.
+       - Handles Vision `*_id` fields where the value is a string (e.g., account_id: "random_str").
+    2. Removes virtual lookup fields (ending in `_id_random`) after resolution.
+    3. Removes fields explicitly marked for exclusion in the model's 
+       `fields_to_exclude_from_db` ClassVar (e.g., view-only fields).
+    4. If `ignore_extra` is True, removes all fields NOT present in the model definition.
+       
+    Modifies the `data` dictionary in-place.
+    """
+    if not isinstance(data, dict):
+        return
+
+    from app.db_sql import redis_lookup_id_random
+
+    # Resolve ID strings to integers
+    for k, v in list(data.items()):
+        if not v or not isinstance(v, str):
+            continue
+            
+        target_id_field = None
+        obj_type_lookup = None
+        
+        # Scenario A: Legacy suffix (e.g., account_id_random: "abc")
+        if k.endswith('_id_random') and k != 'id_random':
+            target_id_field = k.replace('_id_random', '_id')
+            obj_type_lookup = k.replace('_id_random', '')
+            
+        # Scenario B: Vision naming (e.g., account_id: "abc")
+        # We only resolve if it's a string of the correct length (random ID format)
+        elif k.endswith('_id') and 11 <= len(v) <= 22:
+            if k == 'external_person_id':
+                continue
+            target_id_field = k
+            obj_type_lookup = k.replace('_id', '')
+            
+        if target_id_field and obj_type_lookup:
+            # Special table mapping if needed
+            if obj_type_lookup == 'address_location': obj_type_lookup = 'address'
+            
+            resolved_id = redis_lookup_id_random(record_id_random=v, table_name=obj_type_lookup)
+            if resolved_id:
+                data[target_id_field] = resolved_id
+                # If we were handling Scenario A, remove the original random key
+                if k.endswith('_id_random'):
+                    del data[k]
+
+    # Filter out model-specific excluded fields (e.g., view-only fields)
+    if hasattr(model, 'fields_to_exclude_from_db'):
+        for k in model.fields_to_exclude_from_db:
+             if k in data:
+                 del data[k]
+
+    # If permissive mode is on, remove any field not in the Pydantic model
+    if ignore_extra and model and hasattr(model, '__fields__'):
+        model_fields = set(model.__fields__.keys())
+        # Also check for aliases
+        for f in model.__fields__.values():
+            if f.alias:
+                model_fields.add(f.alias)
+        
+        extra_keys = [k for k in data.keys() if k not in model_fields]
+        for k in extra_keys:
+            del data[k]

app/lib_config_v3.py

@@ -0,0 +1,128 @@
+import logging
+from typing import Any
+
+log = logging.getLogger('root')
+
+def validate_critical_config(settings: Any):
+    """
+    Validates that essential settings are populated and not using placeholders.
+    Logs warnings or errors for missing critical infrastructure.
+    """
+    log.info("Checking critical system configuration...")
+
+    # 1. Database Check
+    db = getattr(settings, 'DB', {})
+    if not db.get('server') or db.get('server') == 'mariadb':
+        # 'mariadb' is the default in .env, usually fine, but worth noting
+        log.info(f"Database server: {db.get('server')}")
+
+    # 2. SMTP Check
+    smtp = getattr(settings, 'SMTP', {})
+    if not smtp.get('server'):
+        log.warning("CRITICAL: SMTP server not configured. Email features will fail.")
+    if smtp.get('password') == 'set-in-ae-sql-db-cnf-tbl':
+        log.error("CRITICAL: SMTP password is still set to placeholder. Email authentication will fail.")
+
+    # 3. Security Check
+    jwt_key = getattr(settings, 'JWT_KEY', '')
+    if not jwt_key or jwt_key == 'fake-super-secret-token':
+        log.error("SECURITY: JWT_KEY is missing or using a known fake token!")
+
+    log.info("Aether configuration validation complete.")
+
+def bootstrap_db_config(settings: Any) -> bool:
+    """
+    Loads dynamic settings from the 'cfg' table and updates the settings object.
+    Uses deferred import of sql_select to avoid circular dependencies.
+    """
+    # CRITICAL: Deferred import to prevent boot-time circular dependencies
+    from app.db_sql import sql_select
+
+    # log.setLevel(logging.DEBUG)
+
+    cfg_id = settings.AETHER_CFG.get('id', 0)
+    log.info(f"Bootstrapping Aether system configuration from DB (cfg_id={cfg_id})...")
+
+    try:
+        # Fetch the config record
+        aether_cfg_sql = sql_select(
+            table_name='cfg',
+            record_id=int(cfg_id),
+            as_list=False,
+            max_count=1,
+        )
+        log.debug(f"Raw config record from DB: {aether_cfg_sql}")
+
+        # In some cases sql_select might return a single-item list even with as_list=False
+        if isinstance(aether_cfg_sql, list):
+            if len(aether_cfg_sql) > 0:
+                aether_cfg_sql = aether_cfg_sql[0]
+            else:
+                aether_cfg_sql = None
+
+        if not aether_cfg_sql or not isinstance(aether_cfg_sql, dict):
+            log.error(f"FAILED to load system config from DB for ID {cfg_id}. Table 'cfg' might be empty or ID missing.")
+            return False
+
+        # --- Update Database settings ---
+        # ID Vision: Prioritize Environment Variables for core infrastructure.
+        # We only overwrite if the DB value is present AND the environment value is empty OR changed.
+        db_smtp_server = aether_cfg_sql.get('db_server')
+        if db_smtp_server and (not settings.DB_SERVER or settings.DB_SERVER != db_smtp_server):
+            settings.DB_SERVER = db_smtp_server
+
+        db_smtp_port = aether_cfg_sql.get('db_port')
+        if db_smtp_port and (not settings.DB_PORT or settings.DB_PORT != str(db_smtp_port)):
+            settings.DB_PORT = str(db_smtp_port)
+
+        db_smtp_name = aether_cfg_sql.get('db_name')
+        if db_smtp_name and (not settings.DB_NAME or settings.DB_NAME != db_smtp_name):
+            settings.DB_NAME = db_smtp_name
+
+        db_smtp_username = aether_cfg_sql.get('db_username')
+        if db_smtp_username and (not settings.DB_USER or settings.DB_USER != db_smtp_username):
+            settings.DB_USER = db_smtp_username
+
+        db_smtp_password = aether_cfg_sql.get('db_password')
+        if db_smtp_password and (not settings.DB_PASS or settings.DB_PASS != db_smtp_password):
+            settings.DB_PASS = db_smtp_password
+
+        # --- Update SMTP Settings ---
+        # ID Vision: Prioritize Environment Variables for core infrastructure.
+        # We overwrite ONLY if:
+        # 1. The environment value is a known placeholder ('set-in-ae-sql-db-cnf-tbl')
+        # 2. OR the database value has explicitly changed (dynamic refresh)
+        placeholder = 'set-in-ae-sql-db-cnf-tbl'
+
+        db_smtp_server = aether_cfg_sql.get('smtp_server')
+        if db_smtp_server and (settings.SMTP.get('server') in [placeholder, '', None] or settings.SMTP.get('server') != db_smtp_server):
+            log.info(f"Updating SMTP server to {db_smtp_server}")
+            settings.SMTP['server'] = db_smtp_server
+
+        db_smtp_port = aether_cfg_sql.get('smtp_port')
+        if db_smtp_port and (settings.SMTP.get('port') in [placeholder, '', None] or settings.SMTP.get('port') != str(db_smtp_port)):
+            settings.SMTP['port'] = str(db_smtp_port)
+
+        db_smtp_username = aether_cfg_sql.get('smtp_username')
+        if db_smtp_username and (settings.SMTP.get('username') in [placeholder, '', None] or settings.SMTP.get('username') != db_smtp_username):
+            settings.SMTP['username'] = db_smtp_username
+
+        db_smtp_password = aether_cfg_sql.get('smtp_password')
+        if db_smtp_password and (settings.SMTP.get('password') in [placeholder, '', None] or settings.SMTP.get('password') != db_smtp_password):
+            log.info("Updating SMTP password from database (dynamic refresh).")
+            settings.SMTP['password'] = db_smtp_password
+
+        # --- Update File Paths ---
+        # DEPRECATED: Filesystem paths should be controlled by the Environment/Docker, not the DB.
+        # if aether_cfg_sql.get('path_hosted_files_root'): settings.FILES_PATH['hosted_files_root'] = aether_cfg_sql.get('path_hosted_files_root')
+        # if aether_cfg_sql.get('path_hosted_tmp_root'): settings.FILES_PATH['hosted_tmp_root'] = aether_cfg_sql.get('path_hosted_tmp_root')
+
+        # log.setLevel(logging.DEBUG)
+        log.info("Aether API system configuration successfully synchronized with DB.")
+        log.debug(f"Current Database settings after bootstrap: DB_SERVER={settings.DB_SERVER} DB_PORT={settings.DB_PORT} DB_NAME={settings.DB_NAME} DB_USER={settings.DB_USER} DB_PASS={'****' if settings.DB_PASS else ''}")
+        log.debug(f"Current SMTP settings after bootstrap: {settings.SMTP}")
+        return True
+
+    except Exception as e:
+        log.exception(f"Unexpected error during system bootstrap: {e}")
+        return False

app/lib_email.py

@@ -0,0 +1,195 @@
+import html2text
+import smtplib, ssl
+import logging
+from email.message import EmailMessage
+from email.headerregistry import Address
+from typing import Optional
+
+from app.log import logger_reset
+from app.config import settings
+
+log = logging.getLogger(__name__)
+
+# ### BEGIN ### API Lib Email ### send_email() ###
+# Moved from lib_general.py 2026-01-07
+@logger_reset
+def send_email(
+        from_email: str,
+        to_email: str,
+        subject: str,
+        body_html: str,
+
+        from_name: str = '',
+        reply_to_email: str = '',
+        reply_to_name: str = '',
+        to_name: str = '',
+        cc_email: str = '',
+        cc_name: str = '',
+        bcc_email: str = '',
+        bcc_name: str = '',
+        body_text: str = '',
+
+        test: bool = False,
+        log_lvl: int = logging.WARNING, # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+        ):
+    log.setLevel(log_lvl)
+    log.debug(locals())
+
+    if test:
+        log.setLevel(logging.DEBUG)
+        log.debug('[TESTING] Running with send_email() in TEST mode')
+
+    message = EmailMessage()
+    if subject:
+        message['Subject'] = subject
+    else:
+        return False
+
+    if from_email and from_name:
+        try:
+            message['From'] = Address(display_name=from_name, addr_spec=from_email)
+        except Exception as e:
+            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
+            return False
+    elif from_email:
+        try:
+            message['From'] = Address(display_name=from_email, addr_spec=from_email)
+        except Exception as e:
+            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
+            return False
+    else:
+        return False
+
+    if reply_to_email and reply_to_name:
+        try:
+            message['Reply-To'] = Address(display_name=reply_to_name, addr_spec=reply_to_email)
+        except Exception as e:
+            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
+            return False
+    elif reply_to_email:
+        try:
+            message['Reply-To'] = Address(display_name=reply_to_email, addr_spec=reply_to_email)
+        except Exception as e:
+            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
+            return False
+
+    if to_email and to_name:
+        try:
+            message['To'] = Address(display_name=to_name, addr_spec=to_email)
+        except Exception as e:
+            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
+            return False
+    elif to_email:
+        try:
+            message['To'] = Address(display_name=to_email, addr_spec=to_email)
+        except Exception as e:
+            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
+            return False
+    else:
+        return False
+
+    if cc_email and cc_name:
+        try:
+            message['Cc'] = Address(display_name=cc_name, addr_spec=cc_email)
+        except Exception as e:
+            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
+            return False
+    elif cc_email:
+        try:
+            message['Cc'] = Address(display_name=cc_email, addr_spec=cc_email)
+        except Exception as e:
+            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
+            return False
+
+    if bcc_email and bcc_name:
+        try:
+            message['Bcc'] = Address(display_name=bcc_name, addr_spec=bcc_email)
+        except Exception as e:
+            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
+            return False
+    elif bcc_email:
+        try:
+            message['Bcc'] = Address(display_name=bcc_email, addr_spec=bcc_email)
+        except Exception as e:
+            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
+            return False
+
+    html_version = """
+<html>
+<body>
+
+"""+body_html+"""
+
+</body>
+</html>
+    """
+
+    if body_text:
+        text_version = body_text
+    else:
+        text_version = html2text.html2text(html_version)
+
+    message.set_content(text_version)
+    message.add_alternative(html_version, subtype='html')
+
+    log.info('Sending email...')
+    
+    # Safe access to SMTP settings
+    smtp_settings = getattr(settings, 'SMTP', {})
+    if not smtp_settings:
+        log.error('SMTP settings not found in configuration. Returning False.')
+        return False
+
+    log.debug(smtp_settings)
+
+    log.info(f'Subject: {subject}')
+    log.info(f'From: {from_email} Reply To: {reply_to_email} To: {to_email} CC: {cc_email} BCC: {bcc_email}')
+
+    log.debug('Message:')
+    log.debug(message.as_string())
+
+    log.info('Creating SMTP SSL connection...')
+    context = ssl.create_default_context()
+
+    # Validate SMTP settings
+    smtp_server = smtp_settings.get('server')
+    smtp_port = smtp_settings.get('port')
+    smtp_username = smtp_settings.get('username')
+    smtp_password = smtp_settings.get('password')
+
+    if not smtp_server or not smtp_port:
+        log.error(f'Error: SMTP server or port not configured. Server: {smtp_server}, Port: {smtp_port}')
+        return False
+    
+    try:
+        smtp_port = int(smtp_port)
+    except ValueError:
+        log.error(f'Error: Invalid SMTP port: {smtp_port}')
+        return False
+
+    log.info('SMTP configuration, connect, and send')
+    log.info(f'Server: {smtp_server} Port: {smtp_port} Username: {smtp_username}')
+
+    log.info('Trying smtplib.SMTP_SSL in send_email()...')
+    if test:
+        log.info('[TESTING] Email will NOT actually be sent! [TEST MODE]')
+    try:
+        with smtplib.SMTP_SSL(smtp_server, smtp_port, context=context) as server:
+            log.info('SMTP log in...')
+            # Avoid logging password in debug
+            log.debug(f'Server: {smtp_server} Port: {smtp_port} Username: {smtp_username}')
+            
+            if smtp_username and smtp_password:
+                server.login(smtp_username, smtp_password)
+            
+            log.info('SMTP send message...')
+            if not test:
+                log.info('Email sent! Returning True')
+                server.send_message(message)
+            else:
+                log.info('[TESTING] Email (NOT) sent! Returning True [TEST MODE]')
+        return True
+    except Exception as e:
+        log.error(f'Error: Unable to send email. Exception: {e}')
+        return False
+# ### END ### API Lib Email ### send_email() ###

app/lib_export.py

@@ -0,0 +1,116 @@
+import os
+import pandas
+import pathlib
+import logging
+from typing import Optional, Union
+
+from app.log import logger_reset
+from app.config import settings
+
+log = logging.getLogger(__name__)
+
+# ### BEGIN ### API Lib Export ### create_export_file() ###
+# Moved from lib_general.py 2026-01-07
+@logger_reset
+def create_export_file(
+        data_dict_list: list,
+        subdir_path: str,
+        filename: str,
+        column_name_li: list = [],
+        rm_id: bool = True,
+        export_type: str = 'CSV', # CSV, Excel
+        ) -> Union[bool, str]:
+    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.debug(locals())
+
+    hosted_tmp_path = settings.FILES_PATH['hosted_tmp_root']
+    log.info(f'Hosted Temp Path: {hosted_tmp_path}')
+
+    subdirectory_dest = os.path.join(hosted_tmp_path, subdir_path)
+    log.debug(subdirectory_dest)
+    pathlib.Path(subdirectory_dest).mkdir(parents=True, exist_ok=True)
+    file_dest_w_subdir = os.path.join(subdirectory_dest, filename)
+    log.info(f'File Dest With Subdir: {file_dest_w_subdir}')
+
+    if column_name_li:
+        log.info('Using column name list passed')
+    else:
+        log.info('Using an auto generated column name list')
+        column_name_li = list(data_dict_list[0].keys())
+    log.debug(column_name_li)
+
+    if rm_id:
+        for column_name in list(column_name_li):
+            if column_name.endswith('_id'):
+                column_name_li.remove(column_name)
+                log.info(f'Removing column name: {column_name}')
+    log.info(column_name_li)
+
+    data_dataframe = pandas.DataFrame(data_dict_list)
+    log.debug(data_dataframe)
+
+    missing_cols = [col for col in column_name_li if col not in data_dataframe.columns]
+    if missing_cols:
+        column_name_li = [col for col in column_name_li if col not in missing_cols]
+
+    try:
+        if export_type == 'CSV':
+            log.info('Saving dataframe to CSV file')
+            full_dest_path = file_dest_w_subdir+'.csv'
+            filename_w_ext = filename+'.csv'
+            tmp_file_path = os.path.join(subdir_path,filename_w_ext)
+            data_dataframe.to_csv(
+                full_dest_path,
+                na_rep='NULL',
+                columns=column_name_li,
+                index=False,
+                )
+        elif export_type == 'Excel':
+            log.info('Saving dataframe to Excel file')
+            full_dest_path = file_dest_w_subdir+'.xlsx'
+            filename_w_ext = filename+'.xlsx'
+            tmp_file_path = os.path.join(subdir_path,filename_w_ext)
+            data_dataframe.to_excel(
+                full_dest_path,
+                na_rep='NULL',
+                columns=column_name_li,
+                index=False,
+                )
+    except:
+        log.exception('Something went wrong while trying to save the export file.')
+        return False
+
+    log.info(f'Temp File Path: {tmp_file_path}')
+
+    return tmp_file_path 
+# ### END ### API Lib Export ### create_export_file() ###
+
+# ### BEGIN ### API Lib Export ### return_full_tmp_path() ###
+# This is for using with return FileResponse(path=full_tmp_path, filename=filename)
+# Moved from lib_general.py 2026-01-07
+@logger_reset
+def return_full_tmp_path(
+        full_tmp_path: str = None,
+        subdir_path: str = None,
+        filename: str = None,
+        ) -> Union[bool, str]:
+    log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.debug(locals())
+
+    hosted_tmp_path = settings.FILES_PATH['hosted_tmp_root']
+    log.info(f'Hosted Temp Path: {hosted_tmp_path}')
+
+    if full_tmp_path:
+        file_dest = os.path.join(hosted_tmp_path, full_tmp_path)
+        return file_dest
+    elif subdir_path and filename:
+        subdirectory_dest = os.path.join(hosted_tmp_path, subdir_path)
+        log.debug(subdirectory_dest)
+        pathlib.Path(subdirectory_dest).mkdir(parents=True, exist_ok=True)
+        file_dest_w_subdir = os.path.join(subdirectory_dest, filename)
+        log.info(f'File Dest With Subdir: {file_dest_w_subdir}')
+
+        return file_dest_w_subdir
+    else:
+        return False
+# ### END ### API Lib Export ### return_full_tmp_path() ###

app/lib_general.py

@@ -1,23 +1,20 @@
-# from __future__ import annotations
-import datetime, html2text, jwt, os, pandas, pathlib, pytz, redis, time
-from passlib.hash import argon2
-
-# Import smtplib for the actual sending function
-import smtplib, ssl
-
-# Import the email package modules needed
-from email.message import EmailMessage
-from email.headerregistry import Address
-from email.utils import make_msgid
-
-from fastapi import APIRouter, Depends, Header, HTTPException, Response, status
-from pydantic import BaseModel, EmailStr, Field
 from typing import Dict, List, Optional, Set, Union
+import logging
 
-from app.log import log, logging, logger_reset
+from fastapi import Header, HTTPException, Response, status
+
+from app.log import logger_reset
 from app.config import settings
 from app.db_sql import redis_lookup_id_random, sql_select
 
+from app.lib_email import send_email
+from app.lib_export import create_export_file, return_full_tmp_path
+from app.lib_jwt import sign_jwt, decode_jwt
+from app.lib_hash import secure_hash_string, verify_secure_hash_string
+
+log = logging.getLogger(__name__)
+
+
 
 # ### BEGIN ### API Lib General ### async get_token_header() ###
 def get_token_header(x_token: str = Header(...)):
@@ -31,8 +28,9 @@ def get_token_header(x_token: str = Header(...)):
 class Common_Route_Params_No_Account_ID:
     def __init__(
             self,
-            x_account_id: int = None,
+            x_account_id: int|None = None,
-            x_account_id_random: str = None,
+            x_account_id_random: str|None = None,
+            x_no_account_id_token: str|None = None,
             enabled: str = 'enabled',
             limit: int = 10,
             offset: int = 0,
@@ -42,6 +40,7 @@ class Common_Route_Params_No_Account_ID:
             ):
         self.x_account_id = x_account_id
         self.x_account_id_random = x_account_id_random
+        self.x_no_account_id_token = x_no_account_id_token
         self.enabled = enabled
         self.limit = limit
         self.offset = offset
@@ -126,6 +125,7 @@ class Common_Route_Params:
 def common_route_params(
         # x_account_id: str = Header(..., min_length=11, max_length=22), # NOTE WARNING: Commented out 2023-08-17
         x_account_id: str = Header(None, min_length=11, max_length=22), # NOTE WARNING: Changed to this 2023-08-17
+        x_no_account_id: str = Header(None, min_length=11, max_length=22), # NOTE WARNING: Changed to this 2023-08-17
         # x_aether_api_key: Optional[str] = Header(..., min_length=11, max_length=22),
         # x_aether_api_token: Optional[str] = Header(..., min_length=11, max_length=22),
         # x_aether_jwt_token: Optional[str] = Header(..., min_length=11, max_length=50),
@@ -142,7 +142,7 @@ def common_route_params(
         # include: Optional[list] = [], # Leaving this and exclude commented out
         response: Response = Response,
         log_lvl: int = logging.WARNING, # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-        ) -> Common_Route_Params:
+        ) -> Common_Route_Params|Common_Route_Params_No_Account_ID:
     log.setLevel(log_lvl)
     log.debug(locals())
 
@@ -150,10 +150,20 @@ def common_route_params(
 
     x_account_id_random = x_account_id
 
-    if x_account_id := redis_lookup_id_random(table_name='account', record_id_random=x_account_id):
+    if x_account_id:
-        log.info(f'Found the x-account-id header with the value: {x_account_id}')
+        if x_account_id := redis_lookup_id_random(table_name='account', record_id_random=x_account_id):
+            log.info(f'Found the x-account-id header with the value: {x_account_id}')
+        else:
+            log.warning(f'The x-account-id header was found, but the Account ID was not found or is not valid. Account ID: {x_account_id}')
+            raise HTTPException(status_code=403, detail='The x-account-id Account ID was not found.') # Forbidden
+    elif x_no_account_id and len(x_no_account_id) > 10:
+        log.warning(f'Found the x_no_account_id header param with the value: {x_no_account_id}')
+
+        x_account_id = None
+        x_account_id_random = '--- NOT SET ---'
+
     elif x_no_account_id_token and len(x_no_account_id_token) > 10: # NOTE: Not a header value!
-        # NOTE WARNING: This token should be varified and able to be disabled quickly.
+        # NOTE WARNING: This token should be verified and able to be disabled quickly.
         log.warning(f'Found the x_no_account_id_token URL param with the value: {x_no_account_id_token}')
 
         if x_account_id := redis_lookup_id_random(table_name='account', record_id_random=x_no_account_id_token):
@@ -162,11 +172,17 @@ def common_route_params(
         else:
             x_account_id = 0
             x_account_id_random = ''
-    else:
-        log.warning(f'The x-account-id header was found, but the Account ID was not found or is not valid. Account ID: {x_account_id}')
-        raise HTTPException(status_code=403, detail='The x-account-id Account ID was not found.') # Forbidden
 
-    commons = Common_Route_Params( x_account_id=x_account_id, x_account_id_random=x_account_id_random, x_no_account_id_token=x_no_account_id_token, limit=limit, offset=offset, enabled=enabled, by_alias=by_alias, exclude_unset=exclude_unset, response=response )
+        x_account_id = 0
+        x_account_id_random = '--- NOT SET ---'
+    else:
+        log.warning(f'The x-account-id and x-no-account-id-token headers were not found.')
+        raise HTTPException(status_code=403, detail='The x-account-id and x-no-account-id-token headers were not found.') # Forbidden
+
+    if x_account_id:
+        commons = Common_Route_Params( x_account_id=x_account_id, x_account_id_random=x_account_id_random, x_no_account_id_token=x_no_account_id_token, limit=limit, offset=offset, enabled=enabled, by_alias=by_alias, exclude_unset=exclude_unset, response=response )
+    else:
+        commons = Common_Route_Params_No_Account_ID( x_account_id=None, x_account_id_random=None, x_no_account_id_token=x_no_account_id_token, limit=limit, offset=offset, enabled=enabled, by_alias=by_alias, exclude_unset=exclude_unset, response=response )
 
     log.debug(commons)
 
@@ -245,347 +261,4 @@ def common_route_params_min(
 # ### END ### API Lib General ### async common_route_params_min() ###
 
 
-def secure_hash_string(string: str):
-    string_hash = argon2.using(rounds=14, memory_cost=1536, parallelism=2).hash(string)
 
-    return string_hash
-
-
-def verify_secure_hash_string(string: str, string_hash: str):
-    if argon2.verify(string, string_hash):
-        return True
-    else:
-        return False
-
-
-# ### BEGIN ### API Lib General ### sign_jwt() ###
-# Updated 2021-07-14
-@logger_reset
-def sign_jwt(
-        secret_key: str, # Secret/Private/Password
-        ttl: int = 60, # Default to 60 seconds
-        max_renew: int = 0, # Default to 0
-        public_key: str = None, # Will be part of the token. Use to look up secret when verifying.???
-        account_id: str = None,
-        person_id: str = None,
-        user_id: str = None,
-        json_str: str = None,
-        b64_str: str = None,
-        ) -> Dict[str, str]:
-    log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(locals())
-
-    payload = {
-        'iat': time.time(), # Issued at
-        'eat': time.time() + ttl, # Expires at
-        'max_renew': max_renew, # Number of times allowed to request renew without API secret key
-        'public_key': public_key, # Use to lookup the secret/private/password key when verifying
-        'account_id': account_id,
-        'person_id': person_id,
-        'user_id': user_id,
-        'json_str': json_str,
-        'b64_str': b64_str,
-    }
-    secret = secret_key
-    algorithm = 'HS256'
-    token = jwt.encode(payload, secret, algorithm=algorithm)
-
-    log.debug(token)
-
-    return token
-# ### END ### API Lib General ### sign_jwt() ###
-
-
-# ### BEGIN ### API Lib General ### decode_jwt() ###
-# Updated 2021-07-14
-@logger_reset
-def decode_jwt(
-        secret_key: str,
-        token: str,
-        ) -> dict:
-    log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(locals())
-
-    secret = secret_key
-    algorithm = 'HS256'
-
-    try:
-        decoded_token = jwt.decode(token, secret, algorithms=[algorithm])
-        log.debug(decoded_token)
-        if decoded_token['eat'] >= time.time(): return decoded_token
-        else: return False
-    except:
-        return None
-# ### END ### API Lib General ### decode_jwt() ###
-
-
-# ### BEGIN ### API Lib General ### create_export() ###
-# Updated 2021-11-23
-@logger_reset
-def create_export_file(
-        data_dict_list: list,
-        subdir_path: str,
-        filename: str,
-        column_name_li: list = [],
-        rm_id: bool = True,
-        export_type: str = 'CSV', # CSV, Excel
-        ) -> bool|str:
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(locals())
-
-    hosted_tmp_path = settings.FILES_PATH['hosted_tmp_root']
-    # hosted_tmp_path = 'admin/temp'
-    log.info(f'Hosted Temp Path: {hosted_tmp_path}')
-
-    subdirectory_dest = os.path.join(hosted_tmp_path, subdir_path)
-    log.debug(subdirectory_dest)
-    pathlib.Path(subdirectory_dest).mkdir(parents=True, exist_ok=True)
-    file_dest_w_subdir = os.path.join(subdirectory_dest, filename)
-    log.info(f'File Dest With Subdir: {file_dest_w_subdir}')
-
-    if column_name_li:
-        log.info('Using column name list passed')
-    else:
-        log.info('Using an auto generated column name list')
-        column_name_li = list(data_dict_list[0].keys())
-    log.debug(column_name_li)
-
-    if rm_id:
-        for column_name in list(column_name_li):
-            # log.info(f'Checking column name: {column_name}')
-            if column_name.endswith('_id'):
-                column_name_li.remove(column_name)
-                log.info(f'Removing column name: {column_name}')
-        log.debug(column_name_li)
-
-
-    # column_name_li = ['order_line_id_random', 'order_id_random', 'product_id_random', 'product_type', 'product_name', 'quantity', 'amount', 'dollar_amount', 'message', 'person_id_random', 'person_given_name', 'person_family_name', 'person_display_name', 'person_full_name', 'person_contact_email', 'person_contact_cc_email', 'person_contact_address_name', 'person_contact_address_organization_name', 'person_contact_address_line_1', 'person_contact_address_line_2', 'person_contact_address_line_3', 'person_contact_address_city', 'person_contact_address_country_subdivision_code', 'person_contact_address_state_province', 'person_contact_address_postal_code', 'person_contact_address_country_alpha_2_code', 'person_contact_address_country_name', 'person_contact_address_country', 'order_status', 'order_created_on', 'order_updated_on']
-
-    data_dataframe = pandas.DataFrame(data_dict_list)
-    log.debug(data_dataframe)
-
-    try:
-        if export_type == 'CSV':
-            log.info('Saving dataframe to CSV file')
-            full_dest_path = file_dest_w_subdir+'.csv'
-            filename_w_ext = filename+'.csv'
-            tmp_file_path = os.path.join(subdir_path,filename_w_ext)
-            data_dataframe.to_csv(full_dest_path, columns=column_name_li, index=False)
-        elif export_type == 'Excel':
-            log.info('Saving dataframe to Excel file')
-            full_dest_path = file_dest_w_subdir+'.xlsx'
-            filename_w_ext = filename+'.xlsx'
-            tmp_file_path = os.path.join(subdir_path,filename_w_ext)
-            data_dataframe.to_excel(full_dest_path, columns=column_name_li, index=False) # sheet_name='Sheet_name_1'
-    except:
-        log.exception('Something went wrong while trying to save the export file.')
-        return False
-
-    log.info(f'Temp File Path: {tmp_file_path}')
-
-    return tmp_file_path # True
-# ### END ### API Lib General ### create_export() ###
-
-# ### BEGIN ### API Lib General ### return_full_tmp_path() ###
-# This is for using with return FileResponse(path=full_tmp_path, filename=filename)
-# Updated 2022-04-22
-@logger_reset
-def return_full_tmp_path(
-        full_tmp_path: str = None,
-        subdir_path: str = None,
-        filename: str = None,
-        ) -> bool|str:
-    log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(locals())
-
-    hosted_tmp_path = settings.FILES_PATH['hosted_tmp_root']
-    # hosted_tmp_path = 'admin/temp'
-    log.info(f'Hosted Temp Path: {hosted_tmp_path}')
-
-    if full_tmp_path:
-        file_dest = os.path.join(hosted_tmp_path, full_tmp_path)
-        return file_dest
-    elif subdir_path and filename:
-        subdirectory_dest = os.path.join(hosted_tmp_path, subdir_path)
-        log.debug(subdirectory_dest)
-        pathlib.Path(subdirectory_dest).mkdir(parents=True, exist_ok=True)
-        file_dest_w_subdir = os.path.join(subdirectory_dest, filename)
-        log.info(f'File Dest With Subdir: {file_dest_w_subdir}')
-
-        return file_dest_w_subdir
-    else:
-        return False
-# ### END ### API Lib General ### return_full_tmp_path() ###
-
-
-# ### BEGIN ### API Lib General ### send_email() ###
-# Updated 2021-12-02
-@logger_reset
-def send_email(
-        from_email: str,
-        to_email: str,
-        subject: str,
-        body_html: str,
-
-        from_name: str = '',
-        reply_to_email: str = '',
-        reply_to_name: str = '',
-        to_name: str = '',
-        cc_email: str = '',
-        cc_name: str = '',
-        bcc_email: str = '',
-        bcc_name: str = '',
-        body_text: str = '',
-
-        test: bool = False
-        ):
-    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(locals())
-
-    message = EmailMessage()
-    if subject:
-        message['Subject'] = subject
-    else:
-        return False
-
-    if from_email and from_name:
-        #message['From'] = Address(display_name=from_name, username=from_email.split('@')[0], domain=from_email.split('@')[1])
-        try:
-            message['From'] = Address(display_name=from_name, addr_spec=from_email)
-        except Exception as e:
-            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
-            return False
-    elif from_email:
-        try:
-            message['From'] = Address(display_name=from_email, addr_spec=from_email)
-        except Exception as e:
-            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
-            return False
-    else:
-        return False
-
-    if reply_to_email and reply_to_name:
-        try:
-            message['Reply-To'] = Address(display_name=reply_to_name, addr_spec=reply_to_email)
-        except Exception as e:
-            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
-            return False
-    elif reply_to_email:
-        try:
-            message['Reply-To'] = Address(display_name=reply_to_email, addr_spec=reply_to_email)
-        except Exception as e:
-            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
-            return False
-
-    if to_email and to_name:
-        try:
-            message['To'] = Address(display_name=to_name, addr_spec=to_email)
-        except Exception as e:
-            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
-            return False
-    elif to_email:
-        try:
-            message['To'] = Address(display_name=to_email, addr_spec=to_email)
-        except Exception as e:
-            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
-            return False
-    else:
-        return False
-
-    if cc_email and cc_name:
-        try:
-            message['Cc'] = Address(display_name=cc_name, addr_spec=cc_email)
-        except Exception as e:
-            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
-            return False
-    elif cc_email:
-        try:
-            message['Cc'] = Address(display_name=cc_email, addr_spec=cc_email)
-        except Exception as e:
-            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
-            return False
-
-    if bcc_email and bcc_name:
-        try:
-            message['Bcc'] = Address(display_name=bcc_name, addr_spec=bcc_email)
-        except Exception as e:
-            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
-            return False
-    elif bcc_email:
-        try:
-            message['Bcc'] = Address(display_name=bcc_email, addr_spec=bcc_email)
-        except Exception as e:
-            log.exception('**** *** ** * ### BEGIN ### Exception Happened: Returning False * ** *** ****')
-            return False
-
-    html_version = """\
-<html>
-<body>
-
-"""+body_html+"""
-
-</body>
-</html>
-    """
-
-    # html_version = """\
-    #     <html>
-    #     <body>
-    #         <div>
-    #         """+body_html+"""
-    #         </div>
-    #     </body>
-    #     </html>
-    # """
-
-    if body_text:
-        text_version = body_text
-    else:
-        text_version = html2text.html2text(html_version)
-
-    message.set_content(text_version)
-
-    message.add_alternative(html_version, subtype='html')
-
-    log.info('Sending email...')
-    log.debug(settings.SMTP)
-
-    log.info(f'Subject: {subject}')
-    log.info(f'From: {from_email} Reply To: {reply_to_email} To: {to_email} CC: {cc_email} BCC: {bcc_email}')
-
-    log.debug('Message:')
-    log.debug(message.as_string())
-
-    log.info('Creating SMTP SSL connection...')
-    context = ssl.create_default_context()
-
-    # log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.info('SMTP configuration, connect, and send')
-
-    if test:
-        try:
-            with smtplib.SMTP_SSL(settings.SMTP['server'], settings.SMTP['port'], context=context) as server:
-                log.debug('[TEST] SMTP log in...')
-                server.login(settings.SMTP['username'], settings.SMTP['password'])
-                log.debug('[TEST] SMTP send message... [WILL NOT SEND IN TEST MODE]')
-                # server.send_message(message)
-            log.info('[TEST] Email sent!')
-            return True
-        except:
-            #except SMTPException:
-            log.error('[TEST] Error: unable to send email')
-            return False
-
-    try:
-        with smtplib.SMTP_SSL(settings.SMTP['server'], settings.SMTP['port'], context=context) as server:
-            log.debug('SMTP log in...')
-            server.login(settings.SMTP['username'], settings.SMTP['password'])
-            log.debug('SMTP send message...')
-            server.send_message(message)
-        log.info('Email sent!')
-        return True
-    except:
-        #except SMTPException:
-        log.error('Error: unable to send email')
-        return False
-# ### END ### API Lib General ### send_email() ###

app/lib_general_v3.py

@@ -0,0 +1,47 @@
+"""
+This file contains general utility functions and helpers specifically for API v3.
+Refactored 2026-01-07 to move Auth logic to dependencies_v3.py to fix circular dependencies.
+"""
+
+import logging
+from typing import (
+    Any,
+    Dict,
+    List,
+    Optional,
+    Union,
+)
+
+from fastapi import (
+    APIRouter,
+    Depends,
+    Header,
+    HTTPException,
+    Query,
+    Request,
+    Response,
+    status,
+)
+from pydantic import (
+    BaseModel,
+    Field,
+)
+
+# Re-import from the new central auth models
+from app.models.auth_models import AccountContext
+# Import the dependency functions for backward compatibility in existing v3 routes
+from app.routers.dependencies_v3 import (
+    get_account_context,
+    get_account_context_optional,
+    PaginationParams,
+    StatusFilterParams,
+    SerializationParams,
+    DelayParams
+)
+
+from app.config import settings
+from app.log import get_logger
+
+logger = get_logger(__name__)
+
+# Note: Dependency function implementations have moved to app/routers/dependencies_v3.py

app/lib_hash.py

@@ -0,0 +1,16 @@
+from passlib.hash import argon2
+import logging
+
+log = logging.getLogger(__name__)
+
+# Moved from lib_general.py 2026-01-07
+def secure_hash_string(string: str) -> str:
+    string_hash = argon2.using(rounds=14, memory_cost=1536, parallelism=2).hash(string)
+    return string_hash
+
+# Moved from lib_general.py 2026-01-07
+def verify_secure_hash_string(string: str, string_hash: str) -> bool:
+    if argon2.verify(string, string_hash):
+        return True
+    else:
+        return False

app/lib_jwt.py

@@ -0,0 +1,82 @@
+import jwt
+import time
+import logging
+from typing import Dict, Optional
+
+from app.log import logger_reset
+
+log = logging.getLogger(__name__)
+
+# ### BEGIN ### API Lib JWT ### sign_jwt() ###
+# Moved from lib_general.py 2026-01-07
+@logger_reset
+def sign_jwt(
+        secret_key: str, # Secret/Private/Password
+        ttl: int = 60, # Default to 60 seconds
+        max_renew: int = 0, # Default to 0
+        public_key: str = None, # Will be part of the token. Use to look up secret when verifying.???
+        account_id: str = None,
+        person_id: str = None,
+        user_id: str = None,
+        json_str: str = None,
+        b64_str: str = None,
+        **kwargs # Allow arbitrary claims
+        ) -> str:
+    log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.debug(locals())
+
+    # SECURITY CHECK: Ensure we are not signing numeric IDs
+    for label, val in [('account_id', account_id), ('person_id', person_id), ('user_id', user_id)]:
+        if val is not None:
+            if isinstance(val, int) or (isinstance(val, str) and val.isdigit()):
+                log.critical(f"SECURITY BREACH: Attempted to sign a numeric ID for {label}='{val}'. Only random string IDs allowed.")
+                # For now we log and proceed, but in Phase 3 we should raise an Exception
+                # raise ValueError(f"Numeric IDs cannot be signed in JWTs.")
+
+    payload = {
+        'iat': time.time(), # Issued at
+        'eat': time.time() + ttl, # Expires at
+        'max_renew': max_renew, # Number of times allowed to request renew without API secret key
+        'public_key': public_key, # Use to lookup the secret/private/password key when verifying
+        'account_id': account_id,
+        'person_id': person_id,
+        'user_id': user_id,
+        'json_str': json_str,
+        'b64_str': b64_str,
+    }
+    
+    # Merge additional claims
+    if kwargs:
+        payload.update(kwargs)
+
+    secret = secret_key
+    algorithm = 'HS256'
+    token = jwt.encode(payload, secret, algorithm=algorithm)
+
+    log.debug(token)
+
+    return token
+# ### END ### API Lib JWT ### sign_jwt() ###
+
+
+# ### BEGIN ### API Lib JWT ### decode_jwt() ###
+# Moved from lib_general.py 2026-01-07
+@logger_reset
+def decode_jwt(
+        secret_key: str,
+        token: str,
+        ) -> Optional[dict]:
+    log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.debug(locals())
+
+    secret = secret_key
+    algorithm = 'HS256'
+
+    try:
+        decoded_token = jwt.decode(token, secret, algorithms=[algorithm])
+        log.debug(decoded_token)
+        if decoded_token['eat'] >= time.time(): return decoded_token
+        else: return False
+    except:
+        return None
+# ### END ### API Lib JWT ### decode_jwt() ###

app/lib_log_v3.py

@@ -0,0 +1,77 @@
+import functools
+import logging
+import logging.config
+from typing import Any
+
+# Global logger instance used throughout the app
+log = logging.getLogger('root')
+
+def get_logger(name: str):
+    """Returns a logger instance by name."""
+    return logging.getLogger(name)
+
+def setup_logging(settings: Any):
+    """
+    Configures logging based on provided settings.
+    Moving this here prevents immediate execution on module import.
+    """
+    log_file_path = getattr(settings, 'LOG_PATH', {}).get('app', '/logs/aether_api.log')
+    
+    try:
+        logging.config.dictConfig({
+            'version': 1,
+            'disable_existing_loggers': False, # Critical to not kill FastAPI/Uvicorn loggers
+            'formatters': {
+                'default': {'format': '[%(asctime)s] %(levelname)s @ %(module)s.%(funcName)s()#%(lineno)d: %(message)s'},
+                'long': {'format': '[%(asctime)s] %(levelname)s @ %(module)s.%(funcName)s()#%(lineno)d: %(message)s', 'datefmt': '%Y-%m-%d %H:%M:%S'},
+                'short': {'format': '[%(asctime)s] %(levelname)s @ %(module)s.%(funcName)s()#%(lineno)d: %(message)s', 'datefmt': '%H:%M:%S'},
+            },
+            'handlers': {
+                'console': {
+                    'class': 'logging.StreamHandler',
+                    'stream': 'ext://sys.stderr',
+                    'formatter': 'short',
+                },
+                'log_file_all': {
+                    'level': 'NOTSET',
+                    'class': 'logging.handlers.RotatingFileHandler',
+                    'formatter': 'long',
+                    'filename': log_file_path,
+                    'maxBytes': 10485760, # 10 MB
+                    'backupCount': 9
+                },
+            },
+            'loggers': {
+                'uvicorn': {'handlers': ['console'], 'level': 'INFO'},
+            },
+            'root': {
+                'handlers': ['log_file_all'],
+                'level': 'WARNING',
+            }
+        })
+        log.info(f"Logging successfully configured. Path: {log_file_path}")
+    except Exception as e:
+        print(f"Error configuring logging: {e}")
+        logging.basicConfig(level=logging.WARNING)
+
+def logger_reset(func):
+    """
+    Decorator to log function entry/exit and reset log levels.
+    """
+    @functools.wraps(func)
+    def wrapper(*args, **kwargs):
+        # Local safer access to root logger
+        root_log = logging.getLogger('root')
+
+        if func.__name__ not in ['redis_lookup_id_random', 'sql_enable_part', 'sql_hidden_part']:
+            root_log.info(f'*** Function: "{func.__name__}()"')
+        
+        root_log.debug(f'*** Function Positional Args: {args}\nFunction Key Args: {kwargs}')
+        init_log_level = root_log.level
+        
+        returned_result = func(*args, **kwargs)
+        
+        # Reset level in case it was changed during func execution
+        root_log.setLevel(init_log_level)
+        return returned_result
+    return wrapper

app/lib_redis_helpers.py

@@ -0,0 +1,283 @@
+"""
+Redis-based ID resolution and caching helpers for Aether.
+"""
+import datetime
+import random
+import redis
+import logging
+
+from app.config import settings
+
+log = logging.getLogger(__name__)
+
+# --- Global Redis Client ---
+# Using a single client instance with internal connection pooling is more efficient.
+redis_client = redis.Redis(
+    host=settings.REDIS['server'], 
+    port=settings.REDIS['port'], 
+    db=7, 
+    password=None, 
+    decode_responses=True
+)
+
+def redis_lookup_id_random(
+        record_id_random: int|str,
+        table_name: str,
+        check_int_id: bool = False,
+        log_lvl: int = logging.WARNING, # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+        minutes: int = 30, # Expire the Redis key after 30 minutes
+        reset_rate: int = 10, # 1 in 10 chance of resetting the Redis key (DEPRECATED)
+        ) -> str|int|bool|None:
+    """
+    Looks up a record ID in Redis, falling back to SQL if not found.
+    Resolves 'id_random' (URL-safe string) to internal integer 'id'.
+    """
+    from app.db_sql import sql_select, get_id_random
+    log.setLevel(log_lvl)
+
+    if isinstance(record_id_random, str) and len(record_id_random) >= 11 and len(record_id_random) <= 22: pass
+    elif isinstance(record_id_random, int):
+        record_id = record_id_random
+        if check_int_id:
+            log.info(f'Checking the int ID if exists. Table Name: {table_name} ID: {record_id}')
+            if get_id_random_result := get_id_random(
+                    record_id = record_id,
+                    table_name = table_name,
+                    ):
+                log.info(f'The int ID exists. Returning the int ID. ID Random: {get_id_random_result}')
+                return record_id
+            else:
+                log.info(f'The int ID does not exists. Returning False. Table Name: {table_name} ID: {record_id}')
+                return False
+        else:
+            log.debug(f'Not checking if the int ID exists. Returning the int ID. ID: {record_id}')
+            return record_id
+    elif record_id_random is None:
+        log.info(f'No record ID was passed. Returning None')
+        return None
+    else:
+        log.error(f'Unexpected data type or string format: {type(record_id_random)} Expected type is a string 11 or 22 characters long.')
+        return False
+
+    if record_id_random and table_name:
+        if len(record_id_random) < 11:
+            log.error(f'The length of id_random is too short: {record_id_random} ({len(record_id_random)} chars)')
+            return False
+        elif len(record_id_random) > 22:
+            log.error(f'The length of id_random is too long: {record_id_random} ({len(record_id_random)} chars)')
+            return False
+    elif record_id_random:
+        log.error(f'Missing table_name to select from for id_random "{record_id_random}"')
+        return False
+    elif table_name:
+        log.error(f'Missing id_random to select from table "{table_name}"')
+        return False
+    else:
+        log.error('Missing table_name and record_id_random')
+        return False
+
+    key_name = f'{table_name}:{record_id_random}'
+    rev_key_prefix = f'rev:{table_name}:'
+
+    # Use the global redis client instead of creating a new one every time
+    record_id = redis_client.get(key_name)
+
+    if record_id:
+        redis_client.setex(key_name, datetime.timedelta(minutes=minutes), value=record_id)
+        log.info(f'Redis: Entry found for: Key="{key_name}" value="{record_id}" TTL={redis_client.ttl(key_name)} seconds')
+        return int(record_id)
+    elif table_name:
+        data = { 'id_random': record_id_random }
+        sql = f"SELECT id FROM `{table_name}` AS `table` WHERE `table`.id_random = :id_random;"
+
+        if select_results := sql_select(sql=sql, data=data):
+            log.debug(f'SQL: SELECT result: {select_results}')
+            if isinstance(select_results, dict):
+                log.info(f"""SQL: Found ID Random for: {str(record_id_random)} = {str(select_results.get('id'))}""")
+                if record_id := select_results.get('id'):
+                    # Populating BOTH directions in Redis
+                    redis_client.setex(key_name, datetime.timedelta(minutes=minutes), value=record_id)
+                    redis_client.setex(f'{rev_key_prefix}{record_id}', datetime.timedelta(minutes=minutes), value=record_id_random)
+                    return int(record_id)
+                else:
+                    log.error('The SQL result was not what was expected. The ID field was not found.')
+                    return False
+            else:
+                log.error(f'SQL: More than one record found in "{table_name}". Duplicate id_random!')
+                return redis_lookup_id_random(record_id_random=record_id_random, table_name=table_name)
+        else:
+            log.warning(f'SQL: ID Random "{record_id_random}" not found in "{table_name}". Returning None.')
+            return None
+
+    log.error('Unexpected state in redis_lookup_id_random.')
+    return False
+
+
+def get_id_random(
+        record_id: int,
+        table_name: str,
+        log_lvl: int = logging.WARNING, # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+        minutes: int = 30, # Expire the Redis key after 30 minutes
+        ) -> str|bool|None:
+    """
+    Looks up the 'id_random' for a given internal integer ID.
+    Uses Redis caching for performance.
+    """
+    from app.db_sql import sql_select, get_last_sql_error
+    log.setLevel(log_lvl)
+
+    # Hardened check: Skip lookups for tables known to not have random IDs (e.g. lu_ tables)
+    if not table_name or table_name.startswith('lu_') or table_name.startswith('v_lu_'):
+        return None
+
+    # Check Redis cache first (using 'rev:' prefix for integer -> string mappings)
+    key_name = f'rev:{table_name}:{record_id}'
+    if cached_val := redis_client.get(key_name):
+        # Extend TTL on hit
+        redis_client.setex(key_name, datetime.timedelta(minutes=minutes), value=cached_val)
+        return str(cached_val)
+
+    data = { 'id': record_id }
+    sql = f"SELECT id_random FROM `{table_name}` AS `table` WHERE `table`.id = :id;"
+
+    select_results = sql_select(sql=sql, data=data)
+
+    # Check for "Unknown column 'id_random'" error if sql_select failed
+    if select_results is False:
+        err = str(get_last_sql_error())
+        if "1054" in err and "id_random" in err:
+            log.info(f"Table '{table_name}' does not have an 'id_random' column. Skipping.")
+            return None
+        return False
+
+    if select_results:
+        if isinstance(select_results, dict):
+            if record_id_random := select_results.get('id_random'):
+                # Populating BOTH directions in Redis
+                redis_client.setex(key_name, datetime.timedelta(minutes=minutes), value=record_id_random)
+                redis_client.setex(f'{table_name}:{record_id_random}', datetime.timedelta(minutes=minutes), value=record_id)
+                return str(record_id_random)
+            else:
+                log.error('The SQL result was not what was expected.')
+                return False
+        elif isinstance(select_results, list):
+            log.exception('More than one record may have been found. Duplicate ID!')
+            return False
+        else:
+            log.exception(f'Got an unexpected result while trying to look up the ID.')
+            return False
+    elif select_results is None:
+        return None
+    else:
+        return False
+
+
+def reset_redis():
+    """Flushes the Redis database used for ID caching."""
+    r = redis.Redis(host=settings.REDIS['server'], port=settings.REDIS['port'], db=7, password=None, decode_responses=True)
+    r.flushdb()
+    return True
+
+
+def lookup_id_random_pop(
+        obj_data: dict,
+        log_lvl: int = logging.WARNING, # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+        ):
+    """
+    Look up and resolve id_random values to their id
+    Remove the unneeded *_id_random key from the dict
+    """
+    log.setLevel(log_lvl)
+    
+    # Common prefixes for ID resolution
+    id_prefixes = [
+        'account', 'activity_log', 'address', 'address_location', 'archive', 
+        'contact', 'contact_1', 'contact_2', 'cont_edu_cert', 'cont_edu_cert_person',
+        'entry', 'event', 'event_id_random_only', 'event_abstract', 'event_badge', 
+        'event_badge_template', 'event_exhibit', 'event_file', 'event_location', 
+        'event_person', 'event_person_profile', 'event_presentation', 
+        'event_presenter', 'event_registration', 'event_session', 'event_track', 
+        'grant', 'hosted_file', 'journal', 'journal_entry', 'membership_group', 
+        'membership_person_group', 'membership_person', 'membership_type', 
+        'membership_person_type', 'order', 'order_line', 'order_cart', 
+        'order_cart_line', 'organization', 'page', 'person', 'poc_event_person', 
+        'poc_person', 'post', 'product', 'sponsorship', 'sponsorship_cfg', 
+        'site', 'user'
+    ]
+
+    for prefix in id_prefixes:
+        key_random = f'{prefix}_id_random'
+        key_id = f'{prefix}_id'
+        
+        # Table name mapping
+        table = prefix
+        if prefix == 'address_location': table = 'address'
+        elif prefix in ['contact_1', 'contact_2']: table = 'contact'
+        elif prefix == 'entry': table = 'journal_entry'
+        elif prefix == 'event_id_random_only': table = 'event'
+        elif prefix == 'poc_event_person': table = 'event_person'
+        elif prefix == 'poc_person': table = 'person'
+
+        resolved_id = None
+
+        # Scenario A: Legacy suffix (e.g., account_id_random: "abc")
+        if key_random in obj_data:
+            resolved_id = redis_lookup_id_random(record_id_random=obj_data[key_random], table_name=table)
+            obj_data.pop(key_random)
+            
+        # Scenario B: Vision naming (e.g., account_id: "abc")
+        # Only resolve if it's a string of the correct length (random ID format)
+        elif key_id in obj_data and isinstance(obj_data[key_id], str) and 11 <= len(obj_data[key_id]) <= 22:
+            resolved_id = redis_lookup_id_random(record_id_random=obj_data[key_id], table_name=table)
+
+        if resolved_id is not None:
+            # Set the target ID field
+            target_id_key = key_id
+            if prefix == 'event_id_random_only': target_id_key = 'event_id_only'
+            obj_data[target_id_key] = resolved_id
+            
+            # Removed the short prefix version (e.g., obj_data['account'] = 1) 
+            # as it causes 'Unknown column' errors in direct table inserts.
+
+    # Polymorphic links
+    polymorphic = [
+        ('for_type', 'for_id_random', 'for_id'),
+        ('link_to_type', 'link_to_id_random', 'link_to_id'),
+        ('object_type', 'object_id_random', 'object_id'),
+        ('to_object_type', 'to_object_id_random', 'to_object_id'),
+        ('from_object_type', 'from_object_id_random', 'from_object_id')
+    ]
+
+    for type_key, rand_key, id_key in polymorphic:
+        # Handle random key if present
+        if type_key in obj_data and rand_key in obj_data:
+            obj_data[id_key] = redis_lookup_id_random(
+                record_id_random=obj_data.get(rand_key), 
+                table_name=obj_data.get(type_key)
+            )
+            obj_data.pop(rand_key)
+        # Handle Vision naming (id_key contains the string)
+        elif type_key in obj_data and id_key in obj_data and isinstance(obj_data[id_key], str) and 11 <= len(obj_data[id_key]) <= 22:
+            obj_data[id_key] = redis_lookup_id_random(
+                record_id_random=obj_data.get(id_key), 
+                table_name=obj_data.get(type_key)
+            )
+
+    return obj_data
+
+
+def get_account_id_w_for_type_id(
+        for_type: str, # This is the table name
+        for_id: int|str,
+        ) -> bool|int|None:
+    """Helper to find an account_id associated with an object."""
+    from app.db_sql import sql_select
+    log.setLevel(logging.WARNING)
+    
+    if fid := redis_lookup_id_random(record_id_random=for_id, table_name=for_type):
+        data = {'for_id': fid}
+        sql = f"SELECT account_id FROM `{for_type}` WHERE id = :for_id LIMIT 1;"
+        if result := sql_select(data=data, sql=sql):
+            return result.get('account_id')
+        return False
+    return None

app/lib_schema_v3.py

@@ -0,0 +1,67 @@
+from typing import Any, Dict
+from sqlalchemy import text
+from app.db_sql import db
+from app.ae_obj_types_def import obj_type_kv_li
+
+def get_object_schema_info(obj_type: str, view: str = 'default', variant: str = 'base') -> Dict[str, Any]:
+    """
+    Introspects an object type to return its database and model structure.
+    
+    Args:
+        obj_type: The name of the object (e.g., 'person').
+        view: The SQL view to describe (default, detail, etc.).
+        variant: The model variant to describe (base, in, out).
+        
+    Returns:
+        A dictionary containing database column info and Pydantic field info.
+    """
+    if obj_type not in obj_type_kv_li:
+        return {"error": f"Object type '{obj_type}' not found."}
+
+    obj_cfg = obj_type_kv_li[obj_type]
+    table_name = obj_cfg.get(f'tbl_{view}', obj_cfg.get('tbl_default', obj_cfg.get('tbl')))
+    model_key = f'mdl_{variant}' if variant != 'base' else 'mdl'
+    model = obj_cfg.get(model_key, obj_cfg.get('mdl_default', obj_cfg.get('mdl')))
+
+    if not table_name:
+        return {"error": f"Table configuration for '{obj_type}' is missing."}
+
+    schema_info = {
+        "object_type": obj_type,
+        "view": view,
+        "variant": variant,
+        "database": {"table_name": table_name, "columns": []},
+        "model": {"name": model.__name__ if hasattr(model, '__name__') else str(model), "fields": {}}
+    }
+
+    # 1. Database Introspection
+    try:
+        db_result = db.execute(text(f"DESCRIBE `{table_name}`"))
+        for row in db_result.fetchall():
+            # row format: (Field, Type, Null, Key, Default, Extra)
+            schema_info["database"]["columns"].append({
+                "field": row[0], 
+                "db_type": row[1], 
+                "nullable": row[2] == 'YES',
+                "required": row[2] == 'NO', # Explicitly capture NOT NULL
+                "key": row[3], 
+                "db_default": row[4], 
+                "extra": row[5]
+            })
+    except Exception as e:
+        schema_info["database"]["error"] = str(e)
+
+    # 2. Pydantic Model Introspection
+    if model and hasattr(model, "__fields__"):
+        for field_name, field in model.__fields__.items():
+            field_info = {
+                "alias": field.alias, 
+                "type": str(field.outer_type_), 
+                "required": field.required, 
+                "default": field.default
+            }
+            if field.field_info.description: 
+                field_info["description"] = field.field_info.description
+            schema_info["model"]["fields"][field_name] = field_info
+
+    return schema_info

app/lib_sql_core.py

@@ -0,0 +1,88 @@
+"""
+Foundational SQL connection management for the Aether API.
+Isolates the SQLAlchemy engine and global connection state to prevent circular imports.
+"""
+import logging
+import threading
+from typing import Any, Optional
+from sqlalchemy import create_engine
+from app.config import settings
+
+log = logging.getLogger('root')
+log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+
+# 1. Thread-local storage for capturing last SQL error message
+_sql_error_state = threading.local()
+
+def get_last_sql_error() -> Optional[str]:
+    """Retrieves and clears the last captured SQL error message."""
+    error = getattr(_sql_error_state, 'last_error', None)
+    _sql_error_state.last_error = None
+    return error
+
+def set_last_sql_error(error: Any):
+    """Sets the last captured SQL error message."""
+    _sql_error_state.last_error = str(error)
+
+
+# 2. Initial Engine Setup
+db_uri = settings.SQLALCHEMY_DB_URI
+
+def create_ae_engine(uri: str):
+    return create_engine(
+        url = uri,
+        echo = False,
+        pool_size = settings.DB.get('pool_size', 10),
+        max_overflow = settings.DB.get('max_overflow', 20),
+        pool_use_lifo = True,
+        pool_pre_ping = True,
+        pool_recycle = settings.DB['pool_recycle'],
+        isolation_level = 'READ COMMITTED',
+        connect_args = {'connect_timeout': settings.DB['connect_timeout']}
+    )
+
+engine = create_ae_engine(db_uri)
+
+# DEPRECATED: Global shared 'db' connection. Still used by lib_schema_v3.py and lib_api_crud_v3.py.
+# TODO (P3 full fix): migrate those two call sites to engine.connect() context managers, then remove this.
+# Bare connect guarded so a Docker startup race (MariaDB not yet ready) doesn't crash the worker.
+# If this fails, db=None — callers that hit it before reconnect_db() runs will raise AttributeError.
+try:
+    db = engine.connect()
+except Exception:
+    log.warning("DB SQL Core: Initial db connection failed at startup (MariaDB not ready?). Will retry via reconnect_db().")
+    db = None
+
+log.info('DB SQL Core: Initializing engine...')
+
+
+# 3. Connection Management Logic
+def reconnect_db() -> bool:
+    """
+    Re-initializes the global database engine using current settings.
+    Useful after bootstrapping new credentials from the 'cfg' table.
+    """
+    global engine, db, db_uri
+
+    log.info("DB SQL Core: Refreshing database connection engine...")
+    try:
+        if engine:
+            engine.dispose()
+            log.info("DB SQL Core: Disposed of previous database engine.")
+
+        db_uri = settings.SQLALCHEMY_DB_URI
+        engine = create_ae_engine(db_uri)
+        db = engine.connect()
+
+        safe_uri = db_uri.split('@')[-1] if '@' in db_uri else db_uri
+        log.info(f"DB SQL Core: Database engine re-established successfully: {safe_uri}")
+        return True
+    except Exception:
+        log.exception("DB SQL Core: FAILED to refresh database engine!")
+        return False
+
+def sql_connect(current_db=None, log_lvl: int = logging.INFO) -> bool:
+    """Refreshes the global database connection."""
+    log.setLevel(log_lvl)
+    log.info('DB SQL Core: Refreshing database connection via sql_connect...')
+    return reconnect_db()

app/lib_sql_crud.py

@@ -0,0 +1,436 @@
+"""
+Standardized SQL CRUD operations for the Aether API.
+Provides high-level helpers for INSERT, UPDATE, SELECT, and DELETE.
+"""
+import logging
+import json
+from typing import Any, List, Optional
+from sqlalchemy import text, Time
+from sqlalchemy.exc import IntegrityError, OperationalError, ProgrammingError
+
+from app.log import log, logger_reset
+# CRITICAL: Import the core module to access current global state
+from app import lib_sql_core
+from app.lib_sql_core import set_last_sql_error
+
+# log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+
+# Helper for resolving random IDs
+from app.lib_redis_helpers import lookup_id_random_pop
+
+# ### BEGIN ### API DB SQL ### sql_insert() ###
+@logger_reset
+def sql_insert(
+        sql: str|None = None,
+        data: dict|None = None,
+        table_name: str|None = None,
+        rm_id_random: bool = False,
+        id_random_length: int = 8,
+        log_lvl: int = logging.WARNING,
+        ) -> None|bool|int:
+    log.setLevel(log_lvl)
+
+    if sql:
+        sql_insert_stmt = text(sql)
+    elif table_name and data:
+        if rm_id_random:
+            data = lookup_id_random_pop(obj_data=data)
+        if not data.get('id_random', None) and id_random_length:
+            import secrets
+            data['id_random'] = secrets.token_urlsafe(id_random_length)
+
+        fields = []
+        values = []
+        for key, value in data.items():
+            if key != 'id':
+                fields.append('`'+str(key)+'`')
+                values.append(':'+str(key))
+                if isinstance(value, (dict, list)):
+                    data[key] = json.dumps(value)
+
+        sql_insert_stmt = text(f"INSERT INTO `{table_name}` ({', '.join(fields)}) VALUES ({', '.join(values)});")
+    else:
+        log.error('SQL INSERT statement could not be created. Missing params.')
+        return False
+
+    trans = None
+    try:
+        with lib_sql_core.engine.connect() as conn:
+            trans = conn.begin()
+            result_insert = conn.execute(sql_insert_stmt, data)
+            trans.commit()
+            if result_insert.rowcount == 1 and result_insert.lastrowid > 0:
+                return result_insert.lastrowid
+            return False
+    except IntegrityError as e:
+        # Data constraint violation (duplicate key, FK mismatch, NOT NULL) — do NOT retry;
+        # the same data would fail again. Return None so callers can distinguish from errors.
+        if trans: trans.rollback()
+        log.error('Integrity error (likely duplicate). Returning None')
+        log.debug(e)
+        set_last_sql_error(e)
+        return None
+    except OperationalError:
+        # Transient connection failure. The broken connection rolls back on MariaDB's side,
+        # so retrying with a fresh connection is safe.
+        if trans: trans.rollback()
+        log.warning('Operational error in sql_insert. Retrying once with fresh connection...')
+        try:
+            with lib_sql_core.engine.connect() as conn:
+                trans = conn.begin()
+                result_insert = conn.execute(sql_insert_stmt, data)
+                trans.commit()
+                if result_insert.rowcount == 1 and result_insert.lastrowid > 0:
+                    return result_insert.lastrowid
+                return False
+        except Exception as e:
+            set_last_sql_error(e)
+            return False
+    except Exception as e:
+        if trans: trans.rollback()
+        log.error('Unknown exception in sql_insert. Returning False')
+        log.exception(e)
+        set_last_sql_error(e)
+        return False
+# ### END ### API DB SQL ### sql_insert() ###
+
+
+# ### BEGIN ### API DB SQL ### sql_update() ###
+@logger_reset
+def sql_update(
+        sql: str|None = None,
+        data: dict|None = None,
+        table_name: str|None = None,
+        record_id: int|None = None,
+        record_id_random: str|None = None,
+        rm_id_random: bool = False,
+        id_random_length: None|int = None,
+        log_lvl: int = logging.WARNING,
+        ):
+    log.setLevel(log_lvl)
+
+    if sql:
+        sql_update_stmt = text(sql)
+    elif table_name and data:
+        if rm_id_random:
+            data = lookup_id_random_pop(obj_data=data)
+        if not data.get('id_random', None) and id_random_length:
+            import secrets
+            data['id_random'] = secrets.token_urlsafe(id_random_length)
+
+        field_list = []
+        for key, value in data.items():
+            if key != 'id':
+                field_list.append('`'+str(key) + '` = :' + str(key))
+                if isinstance(value, (dict, list)):
+                    data[key] = json.dumps(value)
+
+        sql_set = ', '.join(field_list)
+        if len(sql_set) < 4:
+            return None
+
+        if record_id is not None:
+            data['id'] = record_id
+            sql_update_stmt = text(f'UPDATE `{table_name}` SET {sql_set} WHERE id = :id')
+        elif record_id_random:
+            data['id_random'] = record_id_random
+            sql_update_stmt = text(f'UPDATE `{table_name}` SET {sql_set} WHERE id_random = :id_random')
+        elif 'id' in data:
+            sql_update_stmt = text(f'UPDATE `{table_name}` SET {sql_set} WHERE id = :id')
+        elif 'id_random' in data:
+            sql_update_stmt = text(f'UPDATE `{table_name}` SET {sql_set} WHERE id_random = :id_random')
+        else:
+            return False
+    else:
+        return False
+
+    trans = None
+    try:
+        with lib_sql_core.engine.connect() as conn:
+            trans = conn.begin()
+            result_update = conn.execute(sql_update_stmt, data)
+            trans.commit()
+            if result_update.rowcount >= 1:
+                return True
+            return None
+    except OperationalError:
+        if trans: trans.rollback()
+        log.error('Operational error (gone away?). Retrying once...')
+        try:
+            with lib_sql_core.engine.connect() as conn:
+                trans = conn.begin()
+                result_update = conn.execute(sql_update_stmt, data)
+                trans.commit()
+                if result_update.rowcount >= 1:
+                    return True
+                return None
+        except Exception as e:
+            set_last_sql_error(e)
+            return False
+    except Exception as e:
+        if trans: trans.rollback()
+        log.exception(e)
+        set_last_sql_error(e)
+        return False
+# ### END ### API DB SQL ### sql_update() ###
+
+
+# ### BEGIN ### Core Help CRUD ### sql_insert_or_update() ###
+@logger_reset
+def sql_insert_or_update(
+        sql: str|None = None,
+        data: dict|None = None,
+        table_name: str|None = None,
+        rm_id_random: bool = False,
+        id_random_length: int|None = None,
+        log_lvl: int = logging.DEBUG,
+        ):
+    log.setLevel(log_lvl)
+
+    if sql:
+        stmt = text(sql)
+    elif table_name and data:
+        if rm_id_random:
+            data = lookup_id_random_pop(obj_data=data)
+        if not data.get('id_random', None) and id_random_length:
+            import secrets
+            data['id_random'] = secrets.token_urlsafe(id_random_length)
+
+        fields = [f'`{k}`' for k in data.keys() if k != 'id']
+        placeholders = [f':{k}' for k in data.keys() if k != 'id']
+        updates = [f'`{k}` = :{k}' for k in data.keys() if k != 'id']
+
+        for k, v in data.items():
+            if isinstance(v, (dict, list)):
+                data[k] = json.dumps(v)
+
+        stmt = text(f"INSERT INTO `{table_name}` ({', '.join(fields)}) VALUES ({', '.join(placeholders)}) "
+                    f"ON DUPLICATE KEY UPDATE {', '.join(updates)};")
+    else:
+        return False
+
+    trans = None
+    try:
+        with lib_sql_core.engine.connect() as conn:
+            trans = conn.begin()
+            res = conn.execute(stmt, data)
+            trans.commit()
+            return res.lastrowid if res.lastrowid > 0 else True
+    except OperationalError:
+        # ON DUPLICATE KEY UPDATE is idempotent — safe to retry.
+        if trans: trans.rollback()
+        log.warning('Operational error in sql_insert_or_update. Retrying once...')
+        try:
+            with lib_sql_core.engine.connect() as conn:
+                trans = conn.begin()
+                res = conn.execute(stmt, data)
+                trans.commit()
+                return res.lastrowid if res.lastrowid > 0 else True
+        except Exception as e:
+            set_last_sql_error(e)
+            return False
+    except Exception as e:
+        if trans: trans.rollback()
+        log.exception(e)
+        return False
+# ### END ### Core Help CRUD ### sql_insert_or_update() ###
+
+
+# ### BEGIN ### Core Help CRUD ### sql_select() ###
+@logger_reset
+def sql_select(
+        table_name: str|None = None,
+        record_id: int|None = None,
+        record_id_random: str|None = None,
+        field_name: str|None = None,
+        field_value = None,
+        enabled: str|None = None,
+        hidden: str|None = None,
+        qry_dict_li: dict|None = None,
+        fulltext_qry_dict: dict|None = None,
+        and_qry_dict: dict|None = None,
+        and_like_dict: dict|None = None,
+        or_like_dict: dict|None = None,
+        and_in_dict_li: dict|None = None,
+        search_query: Any|None = None,
+        searchable_fields: List[str]|None = None,
+        order_by_li: dict|None = None,
+        limit: int = 9999999,
+        offset: int = 0,
+        sql: str|None = None,
+        data: dict|None = None,
+        rm_id_random: bool = False,
+        as_dict: bool|None = True,
+        as_list: bool|None = False,
+        max_count: int = 100000,
+        log_lvl: int = logging.WARNING,
+        ) -> None|bool|dict|list:
+    from app.lib_sql_search import (
+        sql_enable_part, sql_hidden_part, sql_search_qry_part,
+        sql_where_qry_part, sql_fulltext_qry_part, sql_and_qry_part,
+        sql_and_like_part, sql_or_like_part, sql_and_in_dict_li_part
+    )
+
+    log.setLevel(log_lvl)
+
+    sql_limit_offset = f'LIMIT {limit} OFFSET {offset}' if limit >= 0 and offset >= 0 else ''
+
+    sql_order_by = ''
+    if order_by_li and isinstance(order_by_li, dict):
+        order_by_str_li = [f'`{table_name}`.`{k}` {v}' for k, v in order_by_li.items()]
+        sql_order_by = f"ORDER BY {', '.join(order_by_str_li)}"
+
+    if table_name and record_id is None and not (record_id_random or field_name or field_value or sql or data):
+        data = {}
+        s_en, d_en = sql_enable_part(table_name, enabled) if enabled else ('', None)
+        s_hi, d_hi = sql_hidden_part(table_name, hidden) if hidden else ('', None)
+        if d_en is not None: data['enabled'] = d_en
+        if d_hi is not None: data['hidden'] = d_hi
+
+        s_search, d_search = ('', {})
+        if search_query:
+            s_search, d_search = sql_search_qry_part(search_query, searchable_fields, table_name=table_name)
+            data.update(d_search)
+
+        stmt = text(f"SELECT * FROM `{table_name}` WHERE 1=1 {s_search} {s_en} {s_hi} {sql_order_by} {sql_limit_offset};")
+
+    elif table_name and (record_id is not None or record_id_random) and not (field_name or field_value or sql or data):
+        data = {'rid': record_id} if record_id is not None else {'ridr': record_id_random}
+        where = f"`{table_name}`.id = :rid" if record_id is not None else f"`{table_name}`.id_random = :ridr"
+        stmt = text(f"SELECT * FROM `{table_name}` WHERE {where} {sql_order_by} {sql_limit_offset};")
+
+    elif table_name and field_name and field_value and not (record_id is not None or record_id_random or sql or data):
+        data = {field_name: field_value}
+        s_where, d_where = sql_where_qry_part(qry_dict_li) if qry_dict_li else ('', {})
+        s_ft, d_ft = sql_fulltext_qry_part(fulltext_qry_dict) if fulltext_qry_dict else ('', {})
+        s_and, d_and = sql_and_qry_part(and_qry_dict) if and_qry_dict else ('', {})
+        s_alike, d_alike = sql_and_like_part(and_like_dict) if and_like_dict else ('', {})
+        s_olike, d_olike = sql_or_like_part(or_like_dict) if or_like_dict else ('', {})
+        s_in, d_in = sql_and_in_dict_li_part(and_in_dict_li) if and_in_dict_li else ('', {})
+        s_search, d_search = sql_search_qry_part(search_query, searchable_fields, table_name=table_name) if search_query else ('', {})
+        s_en, d_en = sql_enable_part(table_name, enabled) if enabled else ('', None)
+        s_hi, d_hi = sql_hidden_part(table_name, hidden) if hidden else ('', None)
+
+        data.update(d_where); data.update(d_ft); data.update(d_and); data.update(d_alike)
+        data.update(d_olike); data.update(d_in); data.update(d_search)
+        if d_en is not None: data['enabled'] = d_en
+        if d_hi is not None: data['hidden'] = d_hi
+
+        stmt = text(f"SELECT * FROM `{table_name}` WHERE `{table_name}`.{field_name} = :{field_name} "
+                     f"{s_where} {s_ft} {s_and} {s_alike} {s_olike} {s_in} {s_search} {s_en} {s_hi} {sql_order_by} {sql_limit_offset};")
+    elif table_name and data and not (record_id or record_id_random or field_name or field_value or sql):
+        if rm_id_random: data = lookup_id_random_pop(obj_data=data)
+        where_clauses = [f"`{table_name}`.{k} = :{k}" for k in data.keys()]
+        stmt = text(f"SELECT * FROM `{table_name}` WHERE {' AND '.join(where_clauses)} {sql_order_by} {sql_limit_offset};")
+    elif sql:
+        stmt = text(sql)
+    else:
+        return False
+
+    try:
+        with lib_sql_core.engine.connect() as conn:
+            result = conn.execute(stmt, data)
+            if not result:
+                return [] if as_list else None
+
+            # Fetch all rows first to determine actual count reliably
+            if hasattr(result, 'returns_rows') and not result.returns_rows:
+                log.warning("SQL Result does not return rows (ResourceClosedError prevented).")
+                return [] if as_list else None
+
+            rows = result.all()
+    except OperationalError:
+        # Transient connection failure — reads are always safe to retry.
+        log.error('Operational error in sql_select. Retrying once with fresh connection...')
+        try:
+            with lib_sql_core.engine.connect() as conn:
+                result = conn.execute(stmt, data)
+                if not result:
+                    return [] if as_list else None
+                if hasattr(result, 'returns_rows') and not result.returns_rows:
+                    return [] if as_list else None
+                rows = result.all()
+        except Exception as e:
+            log.error(f"SQL Fetch Error on retry: {e}")
+            set_last_sql_error(e)
+            return False
+    except Exception as e:
+        log.error(f"SQL Fetch Error: {e}")
+        set_last_sql_error(e)
+        return False
+
+    count = len(rows)
+
+    if count == 0:
+        return [] if as_list else None
+
+    if count == 1:
+        record = dict(rows[0]) if as_dict else rows[0]
+        return [record] if as_list else record
+
+    # count > 1
+    records = [dict(r) for r in rows] if as_dict else rows
+    return records
+# ### END ### Core Help CRUD ### sql_select() ###
+
+
+# ### BEGIN ### API DB SQL ### run_sql_select() ###
+@logger_reset
+def run_sql_select(
+        sql: text,
+        data: dict|None = None,
+        log_lvl: int = logging.WARNING,
+        ) -> Any:
+    log.setLevel(log_lvl)
+
+    try:
+        with lib_sql_core.engine.connect() as conn:
+            return conn.execute(sql, data)
+    except (OperationalError, ProgrammingError) as e:
+        log.error(f'DB Error: {e}. Retrying once...')
+        try:
+            with lib_sql_core.engine.connect() as conn:
+                return conn.execute(sql, data)
+        except Exception as e2:
+            set_last_sql_error(e2)
+            raise e2 # RAISING instead of returning False
+    except Exception as e:
+        log.exception(e)
+        set_last_sql_error(e)
+        raise e # RAISING instead of returning False
+# ### END ### API DB SQL ### run_sql_select() ###
+
+# ### BEGIN ### Core Help CRUD ### sql_delete() ###
+@logger_reset
+def sql_delete(
+        table_name: str|None = None,
+        record_id: int|None = None,
+        record_id_random: str|None = None,
+        field_name: str|None = None,
+        field_value = None,
+        sql: str|None = None,
+        data: dict|None = None,
+        log_lvl: int = logging.INFO,
+        ) -> None|bool:
+    log.setLevel(log_lvl)
+
+    if table_name and (record_id is not None or record_id_random) and not (field_name or field_value or sql or data):
+        data = {'rid': record_id} if record_id is not None else {'ridr': record_id_random}
+        where = f"`{table_name}`.id = :rid" if record_id is not None else f"`{table_name}`.id_random = :ridr"
+        stmt = text(f"DELETE FROM `{table_name}` WHERE {where}")
+    elif table_name and field_name and field_value and not (record_id is not None or record_id_random or sql or data):
+        data = {field_name: field_value}
+        stmt = text(f"DELETE FROM `{table_name}` WHERE `{table_name}`.{field_name} = :{field_name}")
+    elif sql:
+        stmt = text(sql)
+    else:
+        return False
+
+    try:
+        with lib_sql_core.engine.connect() as conn:
+            result = conn.execute(stmt, data) if data else conn.execute(stmt)
+            return True if result.rowcount >= 1 else None
+    except Exception as e:
+        log.exception(e)
+        return False
+# ### END ### Core Help CRUD ### sql_delete() ###

app/lib_sql_search.py

@@ -0,0 +1,299 @@
+"""
+Modular search builder and query generators for Aether.
+"""
+import logging
+from typing import Any, List, Optional
+from fastapi import HTTPException
+from sqlalchemy import text
+
+log = logging.getLogger(__name__)
+
+def sql_limit_offset_part(limit: int, offset: int = 0) -> str:
+    """Creates a partial SQL string for LIMIT and OFFSET."""
+    if limit >= 0 and offset >= 0:
+        log.info(f'Creating partial SQL string for LIMIT and OFFSET. Limit: {limit}; Offset: {offset}')
+        return f'LIMIT {limit} OFFSET {offset}'
+    else:
+        return ''
+
+def sql_and_like_part(and_like_dict_obj: dict) -> tuple[str, dict]:
+    """Creates a partial SQL string for AND LIKE queries."""
+    data = {}
+    if and_like_dict_obj and isinstance(and_like_dict_obj, dict):
+        log.info('Creating partial SQL string for additional AND LIKE queries.')
+        clauses = []
+        for key, value in and_like_dict_obj.items():
+            clauses.append(f"{key} LIKE :and_like_{key}")
+            data[f'and_like_{key}'] = value
+        return f"AND ({' AND '.join(clauses)})", data
+    return '', {}
+
+def sql_or_like_part(or_like_dict_obj: dict) -> tuple[str, dict]:
+    """Creates a partial SQL string for OR LIKE queries."""
+    data = {}
+    if or_like_dict_obj and isinstance(or_like_dict_obj, dict):
+        log.info('Creating partial SQL string for additional OR LIKE queries.')
+        clauses = []
+        for key, value in or_like_dict_obj.items():
+            clauses.append(f"{key} LIKE :or_like_{key}")
+            data[f'or_like_{key}'] = value
+        return f"AND ({' OR '.join(clauses)})", data
+    return '', {}
+
+def sql_and_in_dict_li_part(and_in_dict_li_dict_obj: dict) -> tuple[str, dict]:
+    """Creates a partial SQL string for AND IN queries."""
+    data = {}
+    if and_in_dict_li_dict_obj and isinstance(and_in_dict_li_dict_obj, dict):
+        log.info('Creating partial SQL string for additional AND IN queries.')
+        clauses = []
+        for key, value in and_in_dict_li_dict_obj.items():
+            clauses.append(f"{key} IN :and_in_{key}")
+            data[f'and_in_{key}'] = value
+        return f"AND ({' AND '.join(clauses)})", data
+    return '', {}
+
+def sql_and_qry_part(and_qry_dict_obj: dict) -> tuple[str, dict]:
+    """Creates a partial SQL string for additional AND queries (equals)."""
+    data = {}
+    if and_qry_dict_obj and isinstance(and_qry_dict_obj, dict):
+        log.info('Creating partial SQL string for additional AND queries.')
+        clauses = []
+        for key, value in and_qry_dict_obj.items():
+            clauses.append(f"{key} = :and_{key}")
+            data[f'and_{key}'] = value
+        return f"AND ({' AND '.join(clauses)})", data
+    return '', {}
+
+def sql_fulltext_qry_part(fulltext_qry_dict: dict) -> tuple[str, dict]:
+    """Creates a partial SQL string for fulltext search."""
+    data = {}
+    if fulltext_qry_dict and isinstance(fulltext_qry_dict, dict):
+        log.info('Creating partial SQL string for fulltext search.')
+        clauses = []
+        for key, value in fulltext_qry_dict.items():
+            clauses.append(f"MATCH( {key} ) AGAINST( :ft_{key} IN BOOLEAN MODE )")
+            data[f'ft_{key}'] = value
+        return f"AND ({' OR '.join(clauses)})", data
+    return '', {}
+
+def sql_enable_part(table_name: str, enabled: str) -> tuple[str, bool|None]:
+    """Handles enabled/disabled status filtering with schema check."""
+    from app import lib_sql_core
+    if not table_name: return '', None
+    if enabled in ['enabled', 'disabled', 'not_enabled', 'all']:
+        if enabled == 'all': return '', None
+        try:
+            with lib_sql_core.engine.connect() as conn:
+                conn.execute(text(f"SELECT enable FROM `{table_name}` LIMIT 0"))
+        except:
+            log.warning(f"Table '{table_name}' missing 'enable' column. Skipping filter.")
+            return '', None
+        val = (enabled == 'enabled')
+        return f"AND `{table_name}`.enable = {str(val).lower()}", val
+    return '', None
+
+def sql_hidden_part(table_name: str, hidden: str) -> tuple[str, bool|None]:
+    """Handles hidden status filtering with schema check."""
+    from app import lib_sql_core
+    if not table_name: return '', None
+    if hidden in ['hidden', 'not_hidden', 'all']:
+        if hidden == 'all': return '', None
+        try:
+            with lib_sql_core.engine.connect() as conn:
+                conn.execute(text(f"SELECT hide FROM `{table_name}` LIMIT 0"))
+        except:
+            log.warning(f"Table '{table_name}' missing 'hide' column. Skipping filter.")
+            return '', None
+        if hidden == 'hidden':
+            return f"AND `{table_name}`.hide = true", True
+        return f"AND (`{table_name}`.hide = false OR `{table_name}`.hide IS NULL)", False
+    return '', None
+
+def sql_where_qry_part(qry_dict_li: list) -> tuple[str, dict]:
+    """Standard v2 style WHERE clause builder."""
+    data = {}
+    if qry_dict_li and isinstance(qry_dict_li, list):
+        log.info('Creating partial SQL string for WHERE queries.')
+        clauses = []
+        for qry in qry_dict_li:
+            field = qry.get('field')
+            op = qry.get('operator')
+            val = qry.get('value')
+            type_ = qry.get('type', 'AND') or 'AND'
+            if op == 'MATCH':
+                clauses.append(f'{type_} MATCH( {field} ) AGAINST( :{field} IN BOOLEAN MODE )')
+            else:
+                clauses.append(f'{type_} {field} {op} :{field}')
+            data[field] = val
+        return ' '.join(clauses), data
+    return '', {}
+
+def sql_search_qry_part(
+        search_query: Any,
+        searchable_fields: List[str]|None = None,
+        max_depth: int = 5,
+        table_name: str|None = None,
+        ) -> tuple[str, dict]:
+    """Recursively builds a SQL WHERE clause from a SearchQuery model."""
+    from app import lib_sql_core
+    data = {}
+    param_counter = [0]
+
+    def get_param_name():
+        param_counter[0] += 1
+        return f"sp_{param_counter[0]}"
+
+    operator_map = {
+        "eq": "=", "ne": "!=", "gt": ">", "gte": ">=", "lt": "<", "lte": "<=",
+        "like": "LIKE", "in": "IN", "is_null": "IS NULL", "is_not_null": "IS NOT NULL",
+        "contains": "LIKE", "icontains": "LIKE", "startswith": "LIKE", "istartswith": "LIKE",
+        "endswith": "LIKE", "iendswith": "LIKE"
+    }
+
+    def process_node(query_node, current_depth: int) -> str:
+        if current_depth > max_depth:
+            raise HTTPException(status_code=400, detail=f"Search query too complex.")
+        clauses = []
+        if hasattr(query_node, 'query_string') and query_node.query_string:
+            if query_node.query_string == '%': pass
+            else:
+                use_match = True
+                if table_name:
+                    try:
+                        with lib_sql_core.engine.connect() as conn:
+                            conn.execute(text(f"SELECT default_qry_str FROM `{table_name}` LIMIT 0"))
+                    except:
+                        use_match = False
+                else:
+                    use_match = False
+
+                if use_match:
+                    p_name = get_param_name()
+                    clauses.append(f"MATCH( default_qry_str ) AGAINST( :{p_name} IN BOOLEAN MODE )")
+                    data[p_name] = query_node.query_string
+                elif searchable_fields:
+                    like_clauses = []
+                    # Fields to exclude from a generic text 'q' search (numeric, technical, or date fields)
+                    exclude_patterns = [
+                        'enable', 'hide', 'priority', 'sort', 'group',
+                        'created_on', 'updated_on'
+                    ]
+                    for field in searchable_fields:
+                        # Exclude internal integer IDs specifically
+                        if field.endswith('_id') or field == 'id':
+                            continue
+
+                        # Exclude other technical/meta fields
+                        if any(x == field for x in exclude_patterns):
+                            continue
+
+                        f_p_name = get_param_name()
+                        like_clauses.append(f"`{field}` LIKE :{f_p_name}")
+                        data[f_p_name] = f"%{query_node.query_string}%"
+
+                    if like_clauses: clauses.append(f"({' OR '.join(like_clauses)})")
+        for filter_attr in ['and_filters', 'or_filters']:
+            if hasattr(query_node, filter_attr) and getattr(query_node, filter_attr):
+                node_clauses = []
+                for item in getattr(query_node, filter_attr):
+                    if hasattr(item, 'field'):
+                        clause, item_data = process_filter(item)
+                        node_clauses.append(clause); data.update(item_data)
+                    else:
+                        # Recurse into nested SearchQuery; only append if non-empty
+                        sub_clause = process_node(item, current_depth + 1)
+                        if sub_clause:
+                            node_clauses.append(f"({sub_clause})")
+                if node_clauses:
+                    joiner = ' AND ' if 'and' in filter_attr else ' OR '
+                    clauses.append(f"({joiner.join(node_clauses)})")
+        return ' AND '.join(clauses)
+
+    def process_filter(f) -> tuple[str, dict]:
+        # --- ID VISION MAPPING ---
+        # If the frontend uses clean names (id, account_id),
+        # map them to the database columns (id_random, account_id_random)
+        # ONLY if those columns actually exist in this table/view.
+        target_field = f.field
+        vision_fields = [
+            'id', 'account_id', 'site_id', 'person_id', 'user_id',
+            'archive_id', 'archive_content_id',
+            'event_id',
+            'event_session_id', 'event_presentation_id', 'event_presenter_id',
+            'event_device_id', 'event_location_id', 'event_track_id',
+            'event_exhibit_id',
+            'event_person_id', 'event_registration_id',
+            'order_id', 'product_id', 'order_cart_id', 'membership_id', 'sponsorship_id',
+            'journal_id', 'journal_entry_id', 'page_id',
+            'post_id', 'post_comment_id',
+            'organization_id', 'address_id', 'contact_id',
+            'hosted_file_id'
+        ]
+
+        if target_field in vision_fields:
+            # ONLY map to _random if the value is a string (looks like a random ID)
+            # If it's an integer, we want to query the original integer column.
+            is_int_val = isinstance(f.value, int) or (isinstance(f.value, str) and f.value.isdigit())
+
+            if not is_int_val:
+                candidate_field = 'id_random' if target_field == 'id' else f"{target_field}_random"
+
+                # Schema Check: Verify if the random version exists in the current table/view
+                use_random = False
+                if table_name:
+                    try:
+                        with lib_sql_core.engine.connect() as conn:
+                            conn.execute(text(f"SELECT `{candidate_field}` FROM `{table_name}` LIMIT 0"))
+                        use_random = True
+                    except Exception:
+                        pass
+
+                if use_random:
+                    target_field = candidate_field
+                    # print(f"Search Trace: Mapping filter field '{f.field}' -> '{target_field}'", flush=True)
+                else:
+                    # Fallback: Resolve ID if random column is missing from view
+                    try:
+                        from app.lib_redis_helpers import redis_lookup_id_random
+                        # Infer table name (e.g., 'event_id' -> 'event')
+                        if target_field.endswith('_id') and target_field != 'id':
+                            lookup_tbl = target_field[:-3] # remove '_id'
+                            resolved_id = redis_lookup_id_random(record_id_random=f.value, table_name=lookup_tbl)
+                            if resolved_id:
+                                # Update the filter value to use the resolved integer
+                                f.value = resolved_id
+                    except Exception as e:
+                        log.warning(f"Failed to resolve random ID for field {target_field}: {e}")
+
+        # site_domain: 'access_key' is a virtual field.
+        # site_access_key (site-level) takes priority; fall back to site_domain_access_key
+        # when site_access_key is not set (NULL or empty).
+        if target_field == 'access_key' and table_name and 'site_domain' in table_name:
+            sql_op = operator_map.get(f.op.lower())
+            if not sql_op: raise HTTPException(status_code=400, detail=f"Unsupported operator: {f.op}")
+            p1, p2 = get_param_name(), get_param_name()
+            return (
+                f"(site_access_key {sql_op} :{p1} OR "
+                f"((site_access_key IS NULL OR site_access_key = '') AND site_domain_access_key {sql_op} :{p2}))"
+            ), {p1: f.value, p2: f.value}
+
+        if searchable_fields is not None and target_field not in searchable_fields:
+            # Fallback check for original field just in case
+            if f.field not in searchable_fields:
+                raise HTTPException(status_code=400, detail=f"Unauthorized search field '{f.field}' (mapped to '{target_field}')")
+
+        sql_op = operator_map.get(f.op.lower())
+        if not sql_op: raise HTTPException(status_code=400, detail=f"Unsupported operator: {f.op}")
+        filter_data = {}
+        if f.op.lower() in ['is_null', 'is_not_null']: clause = f"`{target_field}` {sql_op}"
+        else:
+            p_name = get_param_name()
+            if f.op.lower() == 'in': clause = f"`{target_field}` IN (:{p_name})"; filter_data[p_name] = f.value
+            elif f.op.lower() in ['contains', 'icontains']: clause = f"`{target_field}` LIKE :{p_name}"; filter_data[p_name] = f"%{f.value}%"
+            elif f.op.lower() in ['startswith', 'istartswith']: clause = f"`{target_field}` LIKE :{p_name}"; filter_data[p_name] = f"{f.value}%"
+            elif f.op.lower() in ['endswith', 'iendswith']: clause = f"`{target_field}` LIKE :{p_name}"; filter_data[p_name] = f"%{f.value}"
+            else: clause = f"`{target_field}` {sql_op} :{p_name}"; filter_data[p_name] = f.value
+        return clause, filter_data
+
+    sql_where = process_node(search_query, 1)
+    return (f"AND ({sql_where})", data) if sql_where else ("", {})

app/lib_websockets_v3.py

@@ -0,0 +1,119 @@
+import datetime
+import json
+import logging
+from typing import Any, Dict, List, Optional, Union
+from pydantic import BaseModel, Field
+
+import redis.asyncio as redis
+from app.config import settings
+
+log = logging.getLogger(__name__)
+
+# --- Models ---
+
+class WS_Message_V3(BaseModel):
+    """
+    Standardized message schema for WebSockets V3.
+    """
+    version: str = "3"
+    msg_type: str = Field(..., description="'msg', 'cmd', 'heartbeat', 'presence'")
+    target: str = Field(..., description="'direct', 'group', 'broadcast', 'echo'")
+    
+    from_id: str = Field(..., description="client_id_random of the sender")
+    to_id: Optional[str] = Field(None, description="target client_id_random (for direct messages)")
+    group_id: Optional[str] = Field(None, description="target group_id_random (for group messages)")
+    
+    cmd: Optional[str] = Field(None, description="Specific command string (e.g., 'RELOAD', 'OPEN_FILE')")
+    msg: Optional[str] = Field(None, description="Human-readable message content")
+    
+    payload: Dict[str, Any] = Field(default_factory=dict, description="Flexible JSON data payload")
+    sent_at: datetime.datetime = Field(default_factory=lambda: datetime.datetime.now(datetime.timezone.utc))
+
+    class Config:
+        json_encoders = {
+            datetime.datetime: lambda v: v.isoformat()
+        }
+
+# --- Manager ---
+
+class WS_Manager_V3:
+    """
+    Manages Redis Granular Pub/Sub and Presence for WebSockets V3.
+    """
+    def __init__(self, redis_db: int = 6):
+        self.redis_db = redis_db
+        self.redis_url = f"redis://{settings.REDIS['server']}:{settings.REDIS['port']}"
+        self._redis_conn: Optional[redis.Redis] = None
+
+    async def get_redis(self) -> redis.Redis:
+        """Lazy-loaded async Redis connection."""
+        if self._redis_conn is None:
+            log.info(f"WS V3: Connecting to Redis DB {self.redis_db}")
+            self._redis_conn = redis.Redis.from_url(
+                self.redis_url, 
+                db=self.redis_db, 
+                encoding='utf-8', 
+                decode_responses=True
+            )
+        return self._redis_conn
+
+    def get_channel_names(self, client_id: str, group_id: Optional[str] = None) -> List[str]:
+        """
+        Generates the list of Redis channels a client should subscribe to.
+        """
+        channels = [
+            f"ws:client:{client_id}",  # Direct messages
+            "ws:broadcast"             # System-wide messages
+        ]
+        if group_id:
+            channels.append(f"ws:group:{group_id}") # Group messages
+        return channels
+
+    async def update_presence(self, client_id: str, group_id: str, online: bool = True):
+        """
+        Tracks which clients are online in which groups using Redis Sets.
+        """
+        r = await self.get_redis()
+        key = f"ws:presence:{group_id}"
+        if online:
+            await r.sadd(key, client_id)
+            await r.expire(key, 3600) # Auto-expire in 1 hour if not refreshed
+        else:
+            await r.srem(key, client_id)
+
+    async def get_online_clients(self, group_id: str) -> List[str]:
+        """Returns list of online client IDs in a group."""
+        r = await self.get_redis()
+        return await r.smembers(f"ws:presence:{group_id}")
+
+    async def publish_message(self, message: WS_Message_V3):
+        """
+        Publishes a structured message to the correct granular Redis channel.
+        """
+        r = await self.get_redis()
+        channel = ""
+        
+        if message.target == "direct":
+            if not message.to_id:
+                log.warning("WS V3: Attempted direct publish without to_id")
+                return
+            channel = f"ws:client:{message.to_id}"
+            
+        elif message.target == "group":
+            if not message.group_id:
+                log.warning("WS V3: Attempted group publish without group_id")
+                return
+            channel = f"ws:group:{message.group_id}"
+            
+        elif message.target == "broadcast":
+            channel = "ws:broadcast"
+            
+        elif message.target == "echo":
+            channel = f"ws:client:{message.from_id}"
+
+        if channel:
+            log.debug(f"WS V3: Publishing to {channel}")
+            await r.publish(channel, message.json())
+
+# Global instance
+ws_manager_v3 = WS_Manager_V3()

app/log.py

@@ -1,97 +1,5 @@
-import functools, logging
+import logging
+from app.lib_log_v3 import log, logger_reset, get_logger, setup_logging
 
-from app.config import settings
+# Re-exporting for backward compatibility with ~200 existing imports
-
+__all__ = ['log', 'logging', 'logger_reset', 'get_logger', 'setup_logging']
-# stream options: 'ext://sys.stderr' or 'ext://sys.stdout'
-
-# NOTE: This log config is confusing and may need work... 2022-10-07
-# 'uvicorn' under 'loggers' creates an output to the 'console' handler
-# Do not also add 'console' handler to the 'root' 'handlers' list
-# For now just using that to add or remove file logging options.
-logging.config.dictConfig({
-    'version': 1,
-    'formatters': {
-        'default': {'format': '[%(asctime)s] %(levelname)s @ %(module)s.%(funcName)s()#%(lineno)d: %(message)s'},
-        'long': {'format': '[%(asctime)s] %(levelname)s @ %(module)s.%(funcName)s()#%(lineno)d: %(message)s', 'datefmt': '%Y-%m-%d %H:%M:%S'},
-        'short': {'format': '[%(asctime)s] %(levelname)s @ %(module)s.%(funcName)s()#%(lineno)d: %(message)s', 'datefmt': '%H:%M:%S', 'use_colors': True},
-    },
-    #'filename': 'example.log',
-    # 'level': logging.ERROR,
-    'handlers': {
-        'console': {
-            'class': 'logging.StreamHandler',
-            'stream': 'ext://sys.stderr',
-            'formatter': 'short',
-        },
-        'log_file_all': {
-            'level': 'NOTSET',
-            'class': 'logging.handlers.RotatingFileHandler',
-            'formatter': 'long',
-            'filename': settings.LOG_PATH['app'],
-            'maxBytes': 10485760, # 5,242,880 = 5 MB; 10,485,760 = 10 MB
-            'backupCount': 9
-        },
-        # 'log_file_warning': {
-        #     'level': 'WARNING',
-        #     'class': 'logging.handlers.RotatingFileHandler',
-        #     'formatter': 'long',
-        #     'filename': settings.LOG_PATH['app_warning'],
-        #     'maxBytes': 512000, # 524,288 =  512KB
-        #     'backupCount': 9
-        # },
-        # 'test_handler': {
-        #     'class': 'logging.StreamHandler',
-        #     'level': 'INFO',
-        #     'formatter': 'short',
-        # },
-        # 'test_handler_all_rotate': {
-        #     'class': 'logging.handlers.RotatingFileHandler',
-        #     'level': 'NOTSET',
-        #     'formatter': 'short',
-        #     'filename': '/logs/test_rotate.log',
-        #     'maxBytes': 100000, # 5120000 = 5 MB
-        #     'backupCount': 2,
-        # }
-    },
-    'loggers': {
-        # 'uvicorn': {'handlers': ['default'], 'level': 'INFO'},
-        'uvicorn': {'handlers': ['console'], 'level': 'INFO'},
-        # 'uvicorn.error': {'level': 'INFO', 'handlers': ['default'], 'propagate': True},
-        # 'uvicorn.error': {'level': 'INFO', 'handlers': ['console'], 'propagate': True},
-        # 'uvicorn.access': {'handlers': ['access'], 'level': 'INFO', 'propagate': False},
-        # 'gunicorn': {'handlers': ['console'], 'level': 'INFO'},
-    },
-    'root': {
-        'handlers': ['log_file_all'], #, 'log_file_all', 'log_file_warning'],
-        # 'handlers': ['console', 'log_file_all'], #, 'log_file_all', 'log_file_warning'],
-        'level': 'WARNING', # WARNING
-    }
-})
-
-
-log = logging.getLogger('root')
-# log.setLevel(logging.INFO) # DEBUG > INFO > WARNING > ERROR > CRITICAL
-# logging.basicConfig(
-#     format='[%(asctime)s] %(levelname)s @ %(module)s.%(funcName)s()#%(lineno)d: %(message)s'
-#     )
-
-
-
-# ### BEGIN ### Log ### logger_reset() ###
-# https://realpython.com/primer-on-python-decorators/
-# Updated 2022-02-15
-def logger_reset(func):
-    # log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    # log.info(locals())
-    @functools.wraps(func)
-    def wrapper(*args, **kwargs):
-        if func.__name__ not in ['redis_lookup_id_random']:
-            log.info(f'*** Function: "{func.__name__}()"')
-        log.debug(f'*** Function Positional Args: {args}\nFunction Key Args: {kwargs}')
-        init_log_level = log.level
-        returned_result = func(*args, **kwargs)
-        log.debug(f'*** Function finished: "{func.__name__}()". Resetting logger level to level: {log.level} ***')
-        log.setLevel(init_log_level)
-        return returned_result
-    return wrapper
-# ### END ### Log ### logger_reset() ###

app/main.py

@@ -1,4 +1,4 @@
-import datetime, json, os, pytz, random, secrets # , uvicorn
+import datetime, json, os, pytz, random, secrets, contextlib # , uvicorn
 
 from enum import Enum
 #from datetime import datetime, time, timedelta
@@ -10,78 +10,98 @@ from functools import lru_cache
 from pydantic import BaseModel, EmailStr, Field
 from typing import Dict, List, Optional, Set, Union
 
-# from sqlalchemy import create_engine, text
-# from sqlalchemy.exc import IntegrityError, OperationalError
-
 from . import config
 # from app.lib_general import common_route_params, Common_Route_Params
-from app.log import log, logging
+import logging
+import app.log
+from app.log import setup_logging
 
-# Import the routers here first:
+# Import middleware with alias to avoid shadowing 'app' FastAPI instance
-from app.routers import aether_cfg, api_crud, api, importing, sql, account, activity_log, address, archive, archive_content, contact, cont_edu_cert, cont_edu_cert_person, data_store, event, event_abstract, event_badge, event_badge_importing, event_badge_template, event_device, event_exhibit, event_exhibit_tracking, event_file, event_importing, event_location, event_person, event_person_detail, event_person_tracking, event_presentation, event_presenter, event_registration, event_session, flask_cfg, fundraising, grant, hosted_file, journal, journal_entry, log_client_viewing, lookup, membership_cfg, membership_group, membership_person_group, membership_person, membership_person_profile, membership_type, membership_person_type, order, order_v3, order_line, order_cart, organization, page, person, person_user, post, post_comment, product, qr, site, site_domain, user, util_email, websockets_redis, e_confex, e_cvent, c_idaa, e_impexium, e_stripe
+from app.middleware import add_process_time_header as process_time_middleware
 
-# from app.routers import aether_cfg, sql
+# Centralized router registry
+from app.routers.registry import setup_routers
 
-from app.db_sql import sql_select # , sql_connect
+from app.db_sql import sql_select, reset_redis, reconnect_db
+from app.lib_config_v3 import bootstrap_db_config, validate_critical_config
 
 
-print('### **** *** ** * The Aether API v4 using FastAPI is loading... * ** *** **** ###')
+print('### **** *** ** * The Aether API v3.0 using FastAPI is loading... * ** *** **** ###')
 
 
-#log = logging.getLogger('root')
+log = logging.getLogger(__name__)
 # log.setLevel(logging.DEBUG) # DEBUG > INFO > WARNING > ERROR > CRITICAL
 #logging.basicConfig(
     #format='[%(asctime)s] %(levelname)s @ %(module)s.%(funcName)s()#%(lineno)d: %(message)s'
     #)
 
+@contextlib.asynccontextmanager
+async def lifespan(app: FastAPI):
+    """
+    Handles application startup and shutdown lifecycle.
+    """
+    # 1. Initialize Logging early but safely
+    setup_logging(config.settings)
+    log.info('### **** *** ** * Aether API v3.0 using FastAPI - Startup Lifespan Initiated * ** *** **** ###')
 
+    # 2. Bootstrapping Configuration from DB with robust error handling
+    log.info("Bootstrapping Configuration...")
+
+    # Save original settings for fallback
+    orig_db_server = config.settings.DB_SERVER
+    orig_db_user = config.settings.DB_USER
+    orig_db_pass = config.settings.DB_PASS
+    orig_db_name = config.settings.DB_NAME
+    orig_db_port = config.settings.DB_PORT
+
+    try:
+        if bootstrap_db_config(config.settings):
+            log.info("Successfully bootstrapped configuration from database.")
+            # Re-initialize the database engine with new credentials/URI
+            if reconnect_db():
+                log.info("Database connection re-established with production configuration.")
+            else:
+                log.warning("FAILED to re-establish database connection after bootstrap. Reverting to .env settings.")
+                config.settings.DB_SERVER = orig_db_server
+                config.settings.DB_USER = orig_db_user
+                config.settings.DB_PASS = orig_db_pass
+                config.settings.DB_NAME = orig_db_name
+                config.settings.DB_PORT = orig_db_port
+                reconnect_db()
+        else:
+            log.warning("System bootstrap from DB returned no results. Using environment defaults.")
+    except Exception as e:
+        log.error(f"Unexpected error during configuration bootstrap: {e}. Falling back to .env settings.")
+        config.settings.DB_SERVER = orig_db_server
+        config.settings.DB_USER = orig_db_user
+        config.settings.DB_PASS = orig_db_pass
+        config.settings.DB_NAME = orig_db_name
+        config.settings.DB_PORT = orig_db_port
+        reconnect_db()
+
+    # 3. Final validation of critical infrastructure
+    validate_critical_config(config.settings)
+
+    log.info('### **** *** ** * Aether API v3.0 using FastAPI - Startup Sequence Complete * ** *** **** ###')
+
+    yield
+
+    # Shutdown logic
+    log.info('### **** *** ** * Aether API v3.0 using FastAPI - Shutdown Lifespan Initiated * ** *** **** ###')
+    log.info('The Aether FastAPI API is shutting down...')
+
+
+print('### **** *** ** * Aether API v3.0 using FastAPI - About to try FastAPI() while loading... * ** *** **** ###')
 app = FastAPI(
     # debug = True,
     title = 'Aether API',
-    description = 'One Sky IT\'s Aether API v4 using FastAPI.',
+    description = 'One Sky IT\'s Aether API v3.0 using FastAPI.',
-    version = '4.9.0',
+    version = '3.00.10',
     operationsSorter = 'method',
+    lifespan = lifespan,
     )
 
 
-log.setLevel(logging.INFO)
-# log.debug(config.settings)
-
-if aether_cfg_sql_result := sql_select(
-        table_name = 'cfg',
-        record_id = config.settings.AETHER_CFG['id'],
-        as_list = False,
-        max_count = 1,
-        ):
-    aether_cfg_sql = aether_cfg_sql_result
-
-    config.settings.DB['server'] = aether_cfg_sql.get('db_server')
-    config.settings.DB['port'] = aether_cfg_sql.get('db_port')
-    config.settings.DB['name'] = aether_cfg_sql.get('db_name')
-    config.settings.DB['username'] = aether_cfg_sql.get('db_username')
-    config.settings.DB['password'] = aether_cfg_sql.get('db_password')
-
-    DB = config.settings.DB
-    config.settings.SQLALCHEMY_DB_URI = 'mysql://'+DB['username']+':'+DB['password']+'@'+DB['server']+'/'+DB['name']
-    # db_result = sql_connect(config.settings.SQLALCHEMY_DB_URI)
-    log.debug(config.settings.DB)
-
-    config.settings.SMTP['server'] = aether_cfg_sql.get('smtp_server')
-    config.settings.SMTP['port'] = aether_cfg_sql.get('smtp_port')
-    config.settings.SMTP['username'] = aether_cfg_sql.get('smtp_username')
-    config.settings.SMTP['password'] = aether_cfg_sql.get('smtp_password')
-
-    # config.settings.FILES_PATH['hosted_files_root'] = aether_cfg_sql.get('PATH_HOSTED_FILES_ROOT')
-    # config.settings.FILES_PATH['hosted_tmp_root'] = aether_cfg_sql.get('PATH_HOSTED_TMP_ROOT')
-
-    config.settings.FILES_PATH['hosted_files_root'] = aether_cfg_sql.get('path_hosted_files_root')
-    config.settings.FILES_PATH['hosted_tmp_root'] = aether_cfg_sql.get('path_hosted_tmp_root')
-else:
-    # aether_cfg_sql_result
-    pass
-log.debug(aether_cfg_sql_result)
-log.debug(config.settings)
-
 # @lru_cache()
 # def get_settings():
 #     return config.Settings()
@@ -90,350 +110,11 @@ log.debug(config.settings)
 app.mount('/static', StaticFiles(directory='static'), name='static')
 
 
-# Set up each route once the router has been imported
+# Register all application routes
-app.include_router(
+setup_routers(app)
-    aether_cfg.router,
-    tags=['Aether Config'],
-)
-app.include_router(
-    api_crud.router,
-    prefix='/crud',
-    tags=['CRUD'],
-    #dependencies=[Depends(get_token_header)],
-    #dependencies=[Depends(get_account_header)],
-    #responses={404: {'description': 'Not found'}},
-)
-app.include_router(
-    api.router,
-    prefix='/api',
-    tags=['API'],
-)
-app.include_router(
-    flask_cfg.router,
-    prefix='/flask_cfg',
-    tags=['Flask CFG'],
-)
-app.include_router(
-    importing.router,
-    prefix='/importing',
-    tags=['Importing'],
-)
-app.include_router(
-    sql.router,
-    # prefix='/sql',
-    tags=['SQL'],
-)
-# # app.include_router(
-# #     flask_cfg.router,
-# #     prefix='/redis',
-# #     tags=['Redis'],
-# # )
-
-app.include_router(
-    account.router,
-    # prefix='/account',
-    tags=['Account'],
-)
-app.include_router(
-    activity_log.router,
-    prefix='/activity_log',
-    tags=['Activity Log'],
-)
-app.include_router(
-    address.router,
-    prefix='/address',
-    tags=['Address'],
-)
-app.include_router(
-    archive.router,
-    # prefix='/archive',
-    tags=['Archive'],
-)
-app.include_router(
-    archive_content.router,
-    prefix='/archive/content',
-    tags=['Archive Content'],
-)
-app.include_router(
-    contact.router,
-    prefix='/contact',
-    tags=['Contact'],
-)
-app.include_router(
-    cont_edu_cert.router,
-    tags=['Cont Edu Cert'],
-)
-app.include_router(
-    cont_edu_cert_person.router,
-    tags=['Cont Edu Cert Person'],
-)
-app.include_router(
-    data_store.router,
-    # prefix='/data_store',
-    tags=['Data Store'],
-)
-app.include_router(
-    event.router,
-    # prefix='/event',
-    tags=['Event'],
-)
-app.include_router(
-    event_abstract.router,
-    tags=['Event Abstract'],
-)
-app.include_router(
-    event_badge.router,
-    tags=['Event Badge'],
-)
-app.include_router(
-    event_badge_importing.router,
-    tags=['Event Badge Importing'],
-)
-app.include_router(
-    event_badge_template.router,
-    # prefix='/event/badge/template',
-    tags=['Event Badge Template'],
-)
-app.include_router(
-    event_device.router,
-    # prefix='/event/device',
-    tags=['Event Device'],
-)
-app.include_router(
-    event_exhibit.router,
-    # prefix='/event/exhibit',
-    tags=['Event Exhibit'],
-)
-app.include_router(
-    event_exhibit_tracking.router,
-    # prefix='/event/exhibit/tracking',
-    tags=['Event Exhibit Tracking'],
-)
-app.include_router(
-    event_file.router,
-    # prefix='/event/file',
-    tags=['Event File'],
-)
-app.include_router(
-    event_importing.router,
-    # prefix='/event/importing',
-    tags=['Event Importing'],
-)
-app.include_router(
-    event_location.router,
-    # prefix='/event/location',
-    tags=['Event Location'],
-)
-app.include_router(
-    event_person.router,
-    # prefix='/event/person',
-    tags=['Event Person'],
-)
-app.include_router(
-    event_person.router,
-    prefix='/event/person/detail',
-    tags=['Event Person Detail'],
-)
-app.include_router(
-    event_person_tracking.router,
-    tags=['Event Person Tracking'],
-)
-app.include_router(
-    event_presentation.router,
-    # prefix='/event/presentation',
-    tags=['Event Presentation'],
-)
-app.include_router(
-    event_presenter.router,
-    prefix='/event/presenter',
-    tags=['Event Presenter'],
-)
-app.include_router(
-    event_registration.router,
-    prefix='/event/registration',
-    tags=['Event Registration'],
-)
-app.include_router(
-    event_session.router,
-    # prefix='/event/session',
-    tags=['Event Session'],
-)
-app.include_router(
-    fundraising.router,
-    tags=['Fundraising'],
-)
-app.include_router(
-    grant.router,
-    tags=['Grant'],
-)
-app.include_router(
-    hosted_file.router,
-    prefix='/hosted_file',
-    tags=['Hosted File'],
-)
-app.include_router(
-    journal.router,
-    prefix='/journal',
-    tags=['Journal'],
-)
-app.include_router(
-    journal_entry.router,
-    # prefix='/journal/entry',
-    tags=['Journal Entry'],
-)
-app.include_router(
-    log_client_viewing.router,
-    # prefix='/log/client_viewing',
-    tags=['Log Client Viewing'],
-)
-app.include_router(
-    lookup.router,
-    prefix='/lu',
-    tags=['Lookup'],
-)
-app.include_router(
-    membership_cfg.router,
-    tags=['Membership Config'],
-)
-app.include_router(
-    membership_group.router,
-    tags=['Membership Group'],
-)
-app.include_router(
-    membership_person_group.router,
-    tags=['Membership Group Person'],
-)
-app.include_router(
-    membership_person_profile.router,
-    tags=['Membership Person Profile'],
-)
-app.include_router(
-    membership_person.router,
-    tags=['Membership Person'],
-)
-app.include_router(
-    membership_type.router,
-    tags=['Membership Type'],
-)
-app.include_router(
-    membership_person_type.router,
-    tags=['Membership Type Person'],
-)
-app.include_router(
-    order.router,
-    # prefix='/order',
-    tags=['Order'],
-)
-app.include_router(
-    order_v3.router,
-    # prefix='/order',
-    tags=['Order v3'],
-)
-app.include_router(
-    order_line.router,
-    # prefix='/order',
-    tags=['Order Line'],
-)
-app.include_router(
-    order_cart.router,
-    prefix='/order/cart',
-    tags=['Order Cart'],
-)
-app.include_router(
-    organization.router,
-    prefix='/organization',
-    tags=['Organization'],
-)
-app.include_router(
-    page.router,
-    prefix='/page',
-    tags=['Page'],
-)
-app.include_router(
-    person.router,
-    tags=['Person'],
-)
-app.include_router(
-    person_user.router,
-    prefix='/person_user',
-    tags=['Person User'],
-)
-app.include_router(
-    post.router,
-    # prefix='/post',
-    tags=['Post'],
-)
-app.include_router(
-    post_comment.router,
-    prefix='/post/comment',
-    tags=['Post Comment'],
-)
-app.include_router(
-    product.router,
-    # prefix='/product',
-    tags=['Product'],
-)
-app.include_router(
-    qr.router,
-    tags=['QR'],
-)
-app.include_router(
-    site.router,
-    # prefix='/site',
-    tags=['Site'],
-)
-app.include_router(
-    site_domain.router,
-    # prefix='/site/domain',
-    tags=['Site Domain'],
-)
-app.include_router(
-    user.router,
-    tags=['User'],
-)
-app.include_router(
-    util_email.router,
-    tags=['Utility: Email'],
-)
-# app.include_router(
-#     websockets.router,
-#     # prefix='/websocket',
-#     tags=['Websockets'],
-#     # dependencies=[Depends(get_token_header)],
-#     # responses={404: {'description': 'Not found'}},
-# )
-app.include_router(
-    websockets_redis.router,
-    tags=['Websockets (Redis)'],
-)
-app.include_router(
-    e_confex.router,
-    prefix='/e/confex',
-    tags=['External Service: Confex'],
-)
-app.include_router(
-    e_cvent.router,
-    prefix='/e/cvent',
-    tags=['External Service: Cvent'],
-)
-app.include_router(
-    e_impexium.router,
-    prefix='/e/impexium',
-    tags=['External Service: Impexium'],
-)
-app.include_router(
-    e_stripe.router,
-    prefix='/e/stripe',
-    tags=['External Service: Stripe'],
-)
-
-app.include_router(
-    c_idaa.router,
-    prefix='/c/idaa',
-    tags=['Client: IDAA'],
-)
 
 
+# Updated 2026-02-23
 # BEGIN: CORS
 # NOTE: Eventually this should query the DB for the specific list based on the cfg table and or site_domain table. That way it is dynamic and only allowing those defined in the DB. No wildcards or regex.
 # NOTE: Need to include .localhost for less browser restrictions! Mainly for audio and video.
@@ -452,35 +133,48 @@ app.add_middleware(
 # END: CORS
 
 
-@app.on_event('startup')
+# Updated 2026-02-23
-async def startup():
+# Add middleware to ensure Access-Control-Allow-Private-Network is present
-    log.setLevel(logging.INFO) # DEBUG, INFO, WARN, WARNING, ERROR, EXCEPTION, CRITICAL
+# when the response already includes CORS allow-origin (i.e. origin was allowed).
-    log.debug(locals())
+@app.middleware("http")
-
+async def cors_pna_middleware(request: Request, call_next):
-    log.info('The Aether FastAPI API is starting up...')
+    """Add `Access-Control-Allow-Private-Network: true` to responses
-    #await database.connect()
+    only when CORS has already allowed the request's origin. This avoids
-
+    echoing PNA for disallowed origins and leverages the existing
-
+    CORSMiddleware origin validation.
-@app.on_event('shutdown')
+    """
-async def shutdown():
-    log.setLevel(logging.INFO) # DEBUG, INFO, WARN, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(locals())
-
-    log.info('The Aether FastAPI API is shutting down...')
-    #await database.disconnect()
-
-
-#Add the processing time to the response header.
-@app.middleware('http')
-async def add_process_time_header(request: Request, call_next):
-    import time
-    start_time = time.time()
     response = await call_next(request)
-    process_time = time.time() - start_time
+    # Rely on existing CORS logic (CORSMiddleware) to validate origin.
-    response.headers['X-Process-Time'] = str(process_time)
+    # Only add the PNA header if an Allow-Origin header is present.
+    if response.headers.get('access-control-allow-origin') or response.headers.get('Access-Control-Allow-Origin'):
+        response.headers['Access-Control-Allow-Private-Network'] = 'true'
     return response
 
 
+# Updated 2026-02-23
+# Temporary debug middleware: logs Origin and PNA-related request/response headers
+# Activate only when an Origin header is present to limit log noise.
+# @app.middleware("http")
+# async def debug_pna_logging_middleware(request: Request, call_next):
+#     origin = request.headers.get('origin')
+#     if origin:
+#         acrpn = request.headers.get('access-control-request-private-network')
+#         acrm = request.headers.get('access-control-request-method')
+#         log.debug(f"PNA_DEBUG REQ: method={request.method} path={request.url.path} remote={getattr(request.client, 'host', None)} origin={origin} acr_method={acrm} acr_private_network={acrpn}")
+
+#     response = await call_next(request)
+
+#     if origin:
+#         # collect CORS/PNA-related response headers for visibility
+#         interesting = {k: v for k, v in response.headers.items() if k.lower().startswith('access-control-') or k.lower() == 'vary'}
+#         log.debug(f"PNA_DEBUG RESP: status={response.status_code} headers={interesting}")
+
+#     return response
+
+# Register utility middleware from external module
+app.middleware('http')(process_time_middleware)
+
+
 # ### BEGIN ### API Main ### fastapi_root() ###
 @app.get('/', tags=['Root'], response_class=PlainTextResponse)
 async def fastapi_root(response: Response = Response):
@@ -499,6 +193,10 @@ async def fastapi_root(response: Response = Response):
     log.critical('This is critical') # 50 CRITICAL
     log.info('^^^')
 
+    log.warning('Resetting Redis...')
+    reset_redis()
+    log.info('Reset Redis')
+
     response_data = {}
     response_data['message'] = 'This is One Sky IT\'s Aether API root (FastAPI).'
 
@@ -566,45 +264,3 @@ async def generate_id_random(response: Response = Response):
 
     return HTMLResponse(content=html_list, status_code=200)
 # ### END ### API Main ### generate_id_random() ###
-
-
-# ### BEGIN ### API Main ### sql_test() ###
-# ### TEST TEST TEST ### #
-@app.get('/sql_test', tags=['Testing'], response_class=PlainTextResponse)
-async def sql_test(response: Response = Response):
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARN, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(locals())
-
-    return mk_resp(data=False, status_code=501, response=response)
-
-    log.info('Getting all accounts from DB...')
-
-    sql = text(
-        """
-        SELECT id, id_random, name, enable
-        FROM `account`
-        """
-        )
-    try:
-        result = db.execute(sql)
-    except Exception as e:
-        log.error('*** An exception happened. ***')
-        log.error(repr(e))
-        log.error('***')
-        log.error(str(e))
-        log.error('^^^ exception ^^^')
-    else:
-        if result.rowcount:
-            record_li = [dict(record) for record in result.fetchall()]
-            log.debug(record_li)
-        else:
-            log.error('No records found. Something went wrong.')
-
-    log.info('Got the account list')
-
-    response_data = {}
-    response_data['message'] = 'This is the Aether API using FastAPI.'
-    response_data['data'] = record_li
-
-    return json.dumps(response_data, indent=4) # , sort_keys=True
-# ### END ### API Main ### sql_test() ###

app/methods/address_methods.py

@@ -110,7 +110,7 @@ def create_update_address_obj_v4(
         fail_any: bool = False, # Fail if any thing goes wrong for sub objects
         return_outline: bool = False,
         ) -> int|bool:
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
     log.info('Checking requirements...')
@@ -172,7 +172,7 @@ def create_update_address_obj_v4(
             address_dict['for_id'] = for_id
         try:
             address_obj = Address_Base(**address_dict)
-            log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+            log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
             log.debug(address_obj)
         except ValidationError as e:
             log.error(e.json())
@@ -231,7 +231,7 @@ def create_address_obj(
         for_id: int|str = None,
         fail_any: bool = False, # Fail if any thing goes wrong for sub objects
         ) -> int|bool:
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
     # ### SECTION ### Secondary data validation
@@ -312,7 +312,7 @@ def update_address_obj(
         create_sub_obj: bool = False,
         fail_any: bool = False, # Fail if any thing goes wrong for sub objects
         ) -> bool:
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
     # ### SECTION ### Secondary data validation
@@ -381,7 +381,7 @@ def create_update_address_obj(
         process_address: bool = False,
         process_organization: bool = False,
         ) -> bool:
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
     if address_id:

app/methods/api_crud_methods.py

@@ -0,0 +1,110 @@
+from fastapi import Query, Response
+from typing import Optional, Union, List
+import logging
+
+from app.db_sql import sql_insert, sql_update, sql_select, sql_delete, redis_lookup_id_random, lookup_id_random_pop
+from app.models.response_models import mk_resp
+from app.object_definitions.legacy_v1 import obj_type_li
+
+log = logging.getLogger(__name__)
+
+def post_obj_template(
+        obj_type: str,
+        data: dict,
+        id_random_length: int = 8,
+        return_obj: bool = True,
+        by_alias: bool = True,
+        exclude_unset: Optional[bool] = True,
+        response: Response = Response,
+        **kwargs
+        ):
+    obj_data = lookup_id_random_pop(data)
+    table_name_select = obj_type_li[obj_type]['table_name']
+    base_name = obj_type_li[obj_type]['base_name']
+
+    if sql_insert_result := sql_insert(table_name=obj_type, data=obj_data, id_random_length=id_random_length):
+        obj_id = sql_insert_result
+    else:
+        return mk_resp(data=False, status_code=400, response=response)
+
+    if sql_select_result := sql_select(table_name=table_name_select, record_id=obj_id):
+        resp_data = base_name(**sql_select_result).dict(by_alias=by_alias, exclude_unset=exclude_unset)
+        return mk_resp(data=resp_data, response=response)
+    return mk_resp(data=False, status_code=404, response=response)
+
+def patch_obj_template(
+        obj_type: str,
+        data: dict,
+        obj_id: str,
+        by_alias: bool=True,
+        exclude_unset: Optional[bool] = True,
+        response: Response = Response,
+        **kwargs
+        ):
+    data['id_random'] = obj_id
+    obj_data = lookup_id_random_pop(data)
+    table_name_select = obj_type_li[obj_type]['table_name']
+    base_name = obj_type_li[obj_type]['base_name']
+
+    if sql_update(table_name=obj_type, data=obj_data):
+        obj_id_int = data['id']
+    else:
+        return mk_resp(data=False, status_code=400, response=response)
+
+    if sql_select_result := sql_select(table_name=table_name_select, record_id=obj_id_int):
+        resp_data = base_name(**sql_select_result).dict(by_alias=by_alias, exclude_unset=exclude_unset)
+        return mk_resp(data=resp_data, response=response)
+    return mk_resp(data=False, status_code=404, response=response)
+
+def get_obj_li_template(
+        obj_type: str,
+        for_obj_type: Optional[str] = None,
+        for_obj_id: Optional[Union[int,str]] = None,
+        by_alias: Optional[bool] = True,
+        exclude_unset: Optional[bool] = True,
+        response: Response = Response,
+        **kwargs
+        ):
+    if isinstance(for_obj_id, str):
+        for_obj_id = redis_lookup_id_random(record_id_random=for_obj_id, table_name=for_obj_type)
+
+    table_name_select = obj_type_li[obj_type]['table_name']
+    base_name = obj_type_li[obj_type]['base_name']
+
+    if for_obj_type and for_obj_id:
+        sql_result = sql_select(table_name=table_name_select, field_name=f'{for_obj_type}_id', field_value=for_obj_id)
+    else:
+        sql_result = sql_select(table_name=table_name_select)
+
+    resp_data_li = [base_name(**record).dict(by_alias=by_alias, exclude_unset=exclude_unset) for record in (sql_result or [])]
+    return mk_resp(data=resp_data_li, response=response)
+
+def get_obj_template(
+        obj_id: Union[int,str],
+        obj_type: str,
+        by_alias: Optional[bool] = True,
+        exclude_unset: Optional[bool] = True,
+        response: Response = Response,
+        **kwargs
+        ):
+    if isinstance(obj_id, str):
+        obj_id = redis_lookup_id_random(record_id_random=obj_id, table_name=obj_type)
+
+    table_name_select = obj_type_li[obj_type]['table_name']
+    if not obj_id: return mk_resp(data=False, status_code=404, response=response)
+
+    if sql_result := sql_select(table_name=table_name_select, record_id=obj_id):
+        base_name = obj_type_li[obj_type]['base_name']
+        resp_data = base_name(**sql_result).dict(by_alias=by_alias, exclude_unset=exclude_unset)
+        return mk_resp(data=resp_data, response=response)
+    return mk_resp(data=False, status_code=404, response=response)
+
+def delete_obj_template(
+        obj_type: str,
+        obj_id: str,
+        response: Response = Response,
+        **kwargs
+        ):
+    if sql_delete(table_name=obj_type, record_id_random=obj_id):
+        return mk_resp(data=True, response=response)
+    return mk_resp(data=False, status_code=404, response=response)

app/methods/contact_methods.py

@@ -224,7 +224,7 @@ def create_update_contact_obj_v4(
             contact_dict['for_id'] = for_id
         try:
             contact_obj = Contact_Base(**contact_dict)
-            log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+            log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
             log.debug(contact_obj)
         except ValidationError as e:
             log.error(e.json())
@@ -540,7 +540,7 @@ def create_update_contact_obj(
         process_address: bool = False,
         process_organization: bool = False,
         ) -> bool:
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
     if contact_id:

app/methods/data_store_methods.py

@@ -55,7 +55,7 @@ def load_data_store_obj(
 def load_data_store_obj_w_code(
         account_id: int,
         code: str,
-        for_type: int = None,
+        for_type: str = None,
         for_id: int = None,
         enabled: str = 'enabled', # enabled, disabled, all
         limit: int = 1,
@@ -64,9 +64,11 @@ def load_data_store_obj_w_code(
         exclude_unset: bool = True, # NOTE: For now this is ignored
         model_as_dict: bool = False, # NOTE: For now this is ignored
         ) -> Data_Store_Base|dict|bool:
-    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
+    log.info(f'Getting Data Store record with code: {code} for Account ID: {account_id} and For Type: {for_type} and For ID: {for_id}')
+
     data = {}
     data['account_id'] = account_id
     data['code'] = code
@@ -78,17 +80,9 @@ def load_data_store_obj_w_code(
     # if for_id := redis_lookup_id_random(record_id_random=for_id, table_name=for_type): pass
     #     else: return False
 
-    if for_type and for_id:
-        sql_for_type_id = 'AND `data_store`.for_type = :for_type AND `data_store`.for_id = :for_id'
-    else:
-        sql_for_type_id = 'AND `data_store`.for_type IS NULL AND `data_store`.for_id IS NULL'
-
     sql_enabled, data['enable'] = sql_enable_part(table_name='data_store', enabled=enabled) # Reasonably safe return str and bool
     sql_limit = sql_limit_offset_part(limit=limit, offset=offset) # Reasonably safe return str
 
-    log.debug(data)
-    # log.warning(f'Where are we now??????????? {code}')
-
     sql = f"""
         SELECT *
         FROM `v_data_store` AS `data_store`
@@ -96,20 +90,23 @@ def load_data_store_obj_w_code(
             (
                 `data_store`.account_id = :account_id
                 OR `data_store`.account_id IS NULL
+                OR (`data_store`.for_type = :for_type AND `data_store`.for_id = :for_id)
             )
             AND `data_store`.code = :code
-            {sql_for_type_id}
             {sql_enabled}
-        ORDER BY `data_store`.account_id DESC, `data_store`.created_on DESC, `data_store`.updated_on DESC
+        ORDER BY `data_store`.for_id DESC, `data_store`.account_id DESC, `data_store`.created_on DESC, `data_store`.updated_on DESC
         {sql_limit};
         """
     # log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(sql)
 
+
     if data_store_rec_li_result := sql_select(data=data, sql=sql, as_list=True):
         data_store_rec_li = data_store_rec_li_result
     else: # [] or False
         data_store_rec_li = data_store_rec_li_result
+        log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+        log.info(f'No Data Store records found with code: {code} for Account ID: {account_id} and For Type: {for_type} and For ID: {for_id}')
 
     log.debug(data_store_rec_li_result)
 
@@ -119,21 +116,16 @@ def load_data_store_obj_w_code(
             try:
                 data_store_obj = Data_Store_Base(**data_store_rec)
                 data_store_obj_li.append(data_store_obj)
+                log.debug(data_store_obj)
             except ValidationError as e:
                 log.error(e.json())
                 data_store_obj_li.append(None)
                 # return False
-            log.debug(data_store_obj)
     else: pass
 
-    # log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.info(f'Found {len(data_store_obj_li)} Data Store records with code: {code} for Account ID: {account_id} and For Type: {for_type} and For ID: {for_id}')
     log.debug(data_store_obj_li)
     return data_store_obj_li
-
-    # if model_as_dict:
-    #     return data_store_obj.dict(by_alias=by_alias, exclude_unset=exclude_unset) # pylint: disable=no-member
-    # else:
-    #     return data_store_obj
 # ### END ### API Data Store Methods ### load_data_store_obj_w_code() ###
 
 

app/methods/e_confex_methods.py

@@ -15,8 +15,8 @@ from app.models.event_file_models import Event_File_Base
 
 
 api = {}
-# api['base_url'] = 'https://aapor.confex.com/aapor/2023/meetingapi.cgi/[object]/[id]'
+# api['base_url'] = 'https://aapor.confex.com/aapor/20xx/meetingapi.cgi/[object]/[id]'
-api['base_url'] = 'https://aapor.confex.com/aapor/2023/meetingapi.cgi'
+api['base_url'] = 'https://aapor.confex.com/aapor/2024/meetingapi.cgi'
 api['headers'] = { 'Content-Type': 'application/json;charset=UTF-8' }
 api['username'] = None
 api['password'] = None
@@ -83,7 +83,7 @@ def get_event_session_list(
 
                 log.warning('Something may have gone wrong during the request.')
 
-                # log.warning('Something may have gone wrong. Setting the API app_user_token_datetime value to None to re-authenticate with Impexium on the next request.')
+                # log.warning('Something may have gone wrong. Setting the API app_user_token_datetime value to None to re-authenticate with Confex on the next request.')
                 # api['app_user_token_datetime'] = None # Resetting this just in case the App and or User token expired.
 
     return confex_session_list
@@ -152,7 +152,7 @@ def get_event_session_detail(
 
                 log.warning('Something may have gone wrong during the request.')
 
-                # log.warning('Something may have gone wrong. Setting the API app_user_token_datetime value to None to re-authenticate with Impexium on the next request.')
+                # log.warning('Something may have gone wrong. Setting the API app_user_token_datetime value to None to re-authenticate with Confex on the next request.')
                 # api['app_user_token_datetime'] = None # Resetting this just in case the App and or User token expired.
 
     return confex_session_detail
@@ -223,7 +223,7 @@ def get_event_presentation_detail(
 
                 log.warning('Something may have gone wrong during the request.')
 
-                # log.warning('Something may have gone wrong. Setting the API app_user_token_datetime value to None to re-authenticate with Impexium on the next request.')
+                # log.warning('Something may have gone wrong. Setting the API app_user_token_datetime value to None to re-authenticate with Confex on the next request.')
                 # api['app_user_token_datetime'] = None # Resetting this just in case the App and or User token expired.
 
     return confex_presentation_detail
@@ -294,7 +294,7 @@ def get_event_presenter_detail(
 
                 log.warning('Something may have gone wrong during the request.')
 
-                # log.warning('Something may have gone wrong. Setting the API app_user_token_datetime value to None to re-authenticate with Impexium on the next request.')
+                # log.warning('Something may have gone wrong. Setting the API app_user_token_datetime value to None to re-authenticate with Confex on the next request.')
                 # api['app_user_token_datetime'] = None # Resetting this just in case the App and or User token expired.
 
     return confex_presenter_detail

app/methods/e_novi_mailman_methods.py

@@ -0,0 +1,448 @@
+import json, requests
+from typing import Dict, List, Optional
+from app.db_sql import sql_select
+from app.lib_general import log, logging, logger_reset
+
+# ---------------------------------------------------------------------------
+# Novi-Mailman Bridge — IDAA
+#
+# Credentials live in site.cfg_json for the IDAA site (id_random='58_gJESdlUh').
+# Novi keys already present:
+#   novi_api_root_url  — e.g. "https://www.idaa.org/api"
+#   novi_idaa_api_key  — Base64 API key (Basic auth)
+#
+# Keys that must be added to cfg_json before Mailman or webhooks can work:
+#   mailman_base_url     — e.g. "http://lists.idaa.org:8001"
+#   mailman_username     — Mailman REST admin user (usually "restadmin")
+#   mailman_password     — Mailman REST admin password
+#   mailman_list_id      — Target list, e.g. "members@idaa.org"
+#   novi_webhook_secret  — Shared secret for HMAC-SHA256 webhook validation
+# ---------------------------------------------------------------------------
+
+IDAA_SITE_ID_RANDOM = '58_gJESdlUh'
+
+
+# ── Config Helper ─────────────────────────────────────────────────────────
+
+@logger_reset
+def _load_idaa_cfg() -> Optional[Dict]:
+    """
+    Load IDAA site cfg_json.  Returns the parsed dict, or None on failure.
+    """
+    from app.methods.site_methods import load_site_obj
+    site = load_site_obj(site_id=IDAA_SITE_ID_RANDOM, model_as_dict=True)
+    if not site:
+        log.error("Could not load IDAA site record (id_random='%s').", IDAA_SITE_ID_RANDOM)
+        return None
+    cfg = site.get('cfg_json')
+    if isinstance(cfg, str):
+        try:
+            cfg = json.loads(cfg)
+        except Exception as e:
+            log.error("Failed to parse IDAA cfg_json: %s", e)
+            return None
+    if not isinstance(cfg, dict):
+        log.error("IDAA cfg_json is not a dict after parsing.")
+        return None
+    return cfg
+
+
+# ── Novi AMS Methods ──────────────────────────────────────────────────────
+
+@logger_reset
+def test_novi_connection() -> Dict:
+    """
+    Verify Novi AMS API credentials from IDAA site cfg_json.
+    Uses the first group GUID in novi_idaa_group_guid_li as a lightweight auth probe.
+    Returns {'ok': True, 'member_count': N} on success.
+    """
+    cfg = _load_idaa_cfg()
+    if not cfg:
+        return {"ok": False, "error": "Could not load IDAA site config."}
+
+    base_url = cfg.get('novi_api_root_url', '').rstrip('/')
+    api_key  = cfg.get('novi_idaa_api_key', '')
+
+    if not base_url or not api_key:
+        return {"ok": False, "error": "novi_api_root_url or novi_idaa_api_key missing from cfg_json."}
+
+    group_guid_li = cfg.get('novi_idaa_group_guid_li') or []
+    if not group_guid_li:
+        return {"ok": False, "error": "novi_idaa_group_guid_li missing from cfg_json."}
+
+    headers = {"Authorization": f"Basic {api_key}", "Accept": "application/json"}
+    try:
+        # Use first group as a lightweight auth probe (pageSize=1)
+        guid = group_guid_li[0]
+        resp = requests.get(f"{base_url}/groups/{guid}/members",
+                            headers=headers, params={"pageSize": 1}, timeout=10)
+        if resp.status_code == 200:
+            return {"ok": True, "probe_group": guid}
+        return {"ok": False, "error": f"HTTP {resp.status_code}: {resp.text[:200]}"}
+    except Exception as e:
+        log.exception("Novi connection test failed: %s", e)
+        return {"ok": False, "error": str(e)}
+
+
+@logger_reset
+def get_novi_members(status_filter: Optional[str] = None, page_size: int = 500, offset: int = 0) -> Optional[List[Dict]]:
+    """
+    Fetch member records from Novi AMS.
+
+    Novi has no flat member-list endpoint.  Members are fetched per group from
+    novi_idaa_group_guid_li, deduped by UniqueID, then each member's full record
+    (including Email) is fetched via GET /customers/{uuid}.
+
+    status_filter and pagination (page_size/offset) are not supported at the
+    Novi API level for this approach — all group members are returned.
+    """
+    cfg = _load_idaa_cfg()
+    if not cfg:
+        return None
+
+    base_url      = cfg.get('novi_api_root_url', '').rstrip('/')
+    api_key       = cfg.get('novi_idaa_api_key', '')
+    group_guid_li = cfg.get('novi_idaa_group_guid_li') or []
+    headers       = {"Authorization": f"Basic {api_key}", "Accept": "application/json"}
+
+    if not group_guid_li:
+        log.error("novi_idaa_group_guid_li missing from cfg_json.")
+        return None
+
+    # Step 1: collect unique member UUIDs across all configured groups
+    seen_uuids: set = set()
+    uuid_list: List[str] = []
+    for guid in group_guid_li:
+        try:
+            resp = requests.get(f"{base_url}/groups/{guid}/members",
+                                headers=headers, params={"pageSize": page_size}, timeout=30)
+            if resp.status_code != 200:
+                log.error("Novi group %s fetch error: %s", guid, resp.status_code)
+                continue
+            for entry in resp.json():
+                uid = entry.get('UniqueID')
+                if uid and uid not in seen_uuids:
+                    seen_uuids.add(uid)
+                    uuid_list.append(uid)
+        except Exception as e:
+            log.exception("Failed to fetch Novi group %s: %s", guid, e)
+
+    log.info("Novi: %d unique members across %d group(s).", len(uuid_list), len(group_guid_li))
+
+    # Step 2: fetch full customer record (including Email) for each UUID
+    members: List[Dict] = []
+    for uid in uuid_list:
+        try:
+            resp = requests.get(f"{base_url}/customers/{uid}", headers=headers, timeout=10)
+            if resp.status_code == 200:
+                members.append(resp.json())
+            else:
+                log.warning("Novi customer %s fetch error: %s", uid, resp.status_code)
+        except Exception as e:
+            log.exception("Failed to fetch Novi customer %s: %s", uid, e)
+
+    return members
+
+
+# ── Mailman 3 Methods ─────────────────────────────────────────────────────
+
+@logger_reset
+def test_mailman_connection() -> Dict:
+    """
+    Verify Mailman 3 REST API credentials from IDAA site cfg_json.
+    Returns {'ok': True, 'version': '...'} on success.
+    """
+    cfg = _load_idaa_cfg()
+    if not cfg:
+        return {"ok": False, "error": "Could not load IDAA site config."}
+
+    base_url = cfg.get('mailman_base_url', '').rstrip('/')
+    username = cfg.get('mailman_username', 'restadmin')
+    password = cfg.get('mailman_password', '')
+
+    if not base_url or not password:
+        return {"ok": False, "error": "mailman_base_url or mailman_password missing from cfg_json."}
+
+    try:
+        resp = requests.get(f"{base_url}/3.1/system/versions", auth=(username, password), timeout=10)
+        if resp.status_code == 200:
+            return {"ok": True, "version": resp.json().get('mailman_version')}
+        return {"ok": False, "error": f"HTTP {resp.status_code}: {resp.text[:200]}"}
+    except Exception as e:
+        log.exception("Mailman connection test failed: %s", e)
+        return {"ok": False, "error": str(e)}
+
+
+@logger_reset
+def get_mailman_list_members(list_id: str, count: int = 100, page: int = 1) -> Optional[List[Dict]]:
+    """
+    Return members of a specific Mailman 3 list.
+
+    Args:
+        list_id: fqdn_listname e.g. 'mm3@idaa.org' or dot-notation 'mm3.idaa.org'
+        count:   page size (Mailman default 20, max typically 100)
+        page:    1-based page number
+    """
+    cfg = _load_idaa_cfg()
+    if not cfg:
+        return None
+
+    base_url    = cfg.get('mailman_base_url', '').rstrip('/')
+    auth        = (cfg.get('mailman_username', 'restadmin'), cfg.get('mailman_password', ''))
+    list_id_dot = list_id.replace('@', '.')
+
+    try:
+        resp = requests.get(
+            f"{base_url}/3.1/lists/{list_id_dot}/roster/member",
+            auth=auth,
+            params={"count": count, "page": page},
+            timeout=10,
+        )
+        if resp.status_code != 200:
+            log.error("Mailman member list fetch failed for %s: %s", list_id, resp.status_code)
+            return None
+        data = resp.json()
+        return data.get('entries', [])
+    except Exception as e:
+        log.exception("Failed to fetch members for list %s: %s", list_id, e)
+        return None
+
+
+@logger_reset
+def get_mailman_lists() -> Optional[List[Dict]]:
+    """Return all mailing lists from this Mailman 3 instance."""
+    cfg = _load_idaa_cfg()
+    if not cfg:
+        return None
+
+    base_url = cfg.get('mailman_base_url', '').rstrip('/')
+    auth     = (cfg.get('mailman_username', 'restadmin'), cfg.get('mailman_password', ''))
+
+    try:
+        resp = requests.get(f"{base_url}/3.1/lists", auth=auth, timeout=10)
+        if resp.status_code != 200:
+            log.error("Mailman list fetch failed: %s", resp.status_code)
+            return None
+        return resp.json().get('entries', [])
+    except Exception as e:
+        log.exception("Failed to fetch Mailman lists: %s", e)
+        return None
+
+
+@logger_reset
+def subscribe_member_to_list(list_id: str, email: str, display_name: str = '') -> bool:
+    """
+    Subscribe an email address to a Mailman 3 list (pre-confirmed, no welcome email).
+    Returns True on success or already-subscribed, False on error.
+    """
+    cfg = _load_idaa_cfg()
+    if not cfg:
+        return False
+
+    base_url = cfg.get('mailman_base_url', '').rstrip('/')
+    auth     = (cfg.get('mailman_username', 'restadmin'), cfg.get('mailman_password', ''))
+
+    payload = {
+        "list_id":              list_id.replace('@', '.'),
+        "subscriber":           email,
+        "display_name":         display_name,
+        "pre_verified":         True,
+        "pre_confirmed":        True,
+        "pre_approved":         True,
+        "send_welcome_message": False,
+    }
+
+    try:
+        resp = requests.post(f"{base_url}/3.1/members", auth=auth, json=payload, timeout=10)
+        if resp.status_code in (200, 201):
+            log.info("Subscribed %s to %s", email, list_id)
+            return True
+        if resp.status_code == 409:
+            log.debug("%s already subscribed to %s — skipping.", email, list_id)
+            return True
+        log.error("Subscribe failed for %s: %s - %s", email, resp.status_code, resp.text[:200])
+        return False
+    except Exception as e:
+        log.exception("Error subscribing %s to %s: %s", email, list_id, e)
+        return False
+
+
+@logger_reset
+def unsubscribe_member_from_list(list_id: str, email: str) -> bool:
+    """
+    Unsubscribe an email address from a Mailman 3 list.
+    Returns True on success or not-found, False on error.
+    """
+    cfg = _load_idaa_cfg()
+    if not cfg:
+        return False
+
+    base_url    = cfg.get('mailman_base_url', '').rstrip('/')
+    auth        = (cfg.get('mailman_username', 'restadmin'), cfg.get('mailman_password', ''))
+    list_id_dot = list_id.replace('@', '.')
+
+    try:
+        resp = requests.delete(
+            f"{base_url}/3.1/lists/{list_id_dot}/member/{email}",
+            auth=auth,
+            timeout=10,
+        )
+        if resp.status_code in (200, 204):
+            log.info("Unsubscribed %s from %s", email, list_id)
+            return True
+        if resp.status_code == 404:
+            log.debug("%s not found in %s — skipping.", email, list_id)
+            return True
+        log.error("Unsubscribe failed for %s: %s - %s", email, resp.status_code, resp.text[:200])
+        return False
+    except Exception as e:
+        log.exception("Error unsubscribing %s from %s: %s", email, list_id, e)
+        return False
+
+
+# ── Mirror Sync Engine ────────────────────────────────────────────────────
+
+@logger_reset
+def mirror_novi_group_to_mailman_list(novi_group_guid: str, mailman_list_id: str) -> Optional[Dict]:
+    """
+    Mirror a single Novi group to a Mailman 3 list.
+
+    - Fetches all members of `novi_group_guid` from Novi.
+    - For each member, fetches their customer record to get Email, name, and
+      membership flags.  Members with `Active=False` or `UnsubscribeFromEmails=True`
+      are excluded from the target set.
+    - Fetches current members of `mailman_list_id`.
+    - Subscribes addresses in Novi but not in Mailman.
+    - Unsubscribes addresses in Mailman but not in Novi (mirror / full reconcile).
+    - Returns a result dict with counts.
+    """
+    cfg = _load_idaa_cfg()
+    if not cfg:
+        return None
+
+    base_url = cfg.get('novi_api_root_url', '').rstrip('/')
+    api_key  = cfg.get('novi_idaa_api_key', '')
+    headers  = {"Authorization": f"Basic {api_key}", "Accept": "application/json"}
+
+    log.info("Mirror sync: Novi group %s → Mailman list %s", novi_group_guid, mailman_list_id)
+
+    # ── Step 1: Novi group → UUIDs ────────────────────────────────────────
+    try:
+        resp = requests.get(f"{base_url}/groups/{novi_group_guid}/members",
+                            headers=headers, params={"pageSize": 500}, timeout=30)
+        if resp.status_code != 200:
+            log.error("Novi group fetch failed (%s): %s", novi_group_guid, resp.status_code)
+            return None
+        uuid_list = [m['UniqueID'] for m in resp.json() if m.get('UniqueID')]
+    except Exception as e:
+        log.exception("Failed to fetch Novi group %s: %s", novi_group_guid, e)
+        return None
+
+    log.info("Novi group %s has %d member(s).", novi_group_guid, len(uuid_list))
+
+    # ── Step 2: UUID → customer record → email ────────────────────────────
+    # email.lower() → display_name
+    novi_members: Dict[str, str] = {}
+    skipped_inactive = 0
+    skipped_unsub    = 0
+    skipped_no_email = 0
+
+    for uid in uuid_list:
+        try:
+            r = requests.get(f"{base_url}/customers/{uid}", headers=headers, timeout=10)
+            if r.status_code != 200:
+                log.warning("Novi customer %s fetch failed: %s", uid, r.status_code)
+                continue
+            c = r.json()
+            if not c.get('Active', False):
+                skipped_inactive += 1
+                continue
+            if c.get('UnsubscribeFromEmails', False):
+                skipped_unsub += 1
+                continue
+            email = (c.get('Email') or '').strip()
+            if not email:
+                skipped_no_email += 1
+                continue
+            display = f"{c.get('FirstName', '')} {c.get('LastName', '')}".strip()
+            novi_members[email.lower()] = display
+        except Exception as e:
+            log.exception("Failed to fetch Novi customer %s: %s", uid, e)
+
+    log.info("Novi active/subscribed members with email: %d (skipped: inactive=%d unsub=%d no_email=%d)",
+             len(novi_members), skipped_inactive, skipped_unsub, skipped_no_email)
+
+    # ── Step 3: Current Mailman members ───────────────────────────────────
+    mailman_entries = get_mailman_list_members(mailman_list_id)
+    if mailman_entries is None:
+        log.error("Could not fetch current Mailman members for %s — aborting.", mailman_list_id)
+        return None
+    mailman_emails = {m['email'].lower() for m in mailman_entries}
+
+    # ── Step 4: Diff ──────────────────────────────────────────────────────
+    novi_email_set  = set(novi_members.keys())
+    to_subscribe    = novi_email_set - mailman_emails
+    to_unsubscribe  = mailman_emails - novi_email_set
+
+    log.info("Diff — to subscribe: %d, to unsubscribe: %d", len(to_subscribe), len(to_unsubscribe))
+
+    results = {
+        "novi_group_guid":    novi_group_guid,
+        "mailman_list_id":    mailman_list_id,
+        "novi_count":         len(novi_email_set),
+        "mailman_count_before": len(mailman_emails),
+        "subscribed":         0,
+        "unsubscribed":       0,
+        "errors":             0,
+        "skipped_inactive":   skipped_inactive,
+        "skipped_unsub":      skipped_unsub,
+        "skipped_no_email":   skipped_no_email,
+    }
+
+    # ── Step 5: Apply ─────────────────────────────────────────────────────
+    for email in to_subscribe:
+        ok = subscribe_member_to_list(mailman_list_id, email, novi_members[email])
+        if ok: results['subscribed'] += 1
+        else:  results['errors'] += 1
+
+    for email in to_unsubscribe:
+        ok = unsubscribe_member_from_list(mailman_list_id, email)
+        if ok: results['unsubscribed'] += 1
+        else:  results['errors'] += 1
+
+    log.info("Mirror sync complete: %s", results)
+    return results
+
+
+@logger_reset
+def mirror_all_configured_mappings() -> Optional[List[Dict]]:
+    """
+    Run mirror_novi_group_to_mailman_list for every entry in
+    cfg_json['novi_mailman_sync'].
+
+    Expected cfg_json shape:
+        "novi_mailman_sync": [
+            {"novi_group_guid": "...", "mailman_list_id": "members@idaa.org"},
+            ...
+        ]
+    """
+    cfg = _load_idaa_cfg()
+    if not cfg:
+        return None
+
+    sync_map = cfg.get('novi_mailman_sync') or []
+    if not sync_map:
+        log.warning("novi_mailman_sync not configured in IDAA cfg_json.")
+        return []
+
+    results = []
+    for mapping in sync_map:
+        guid    = mapping.get('novi_group_guid', '').strip()
+        list_id = mapping.get('mailman_list_id', '').strip()
+        if not guid or not list_id:
+            log.warning("Skipping incomplete novi_mailman_sync entry: %s", mapping)
+            continue
+        result = mirror_novi_group_to_mailman_list(guid, list_id)
+        results.append(result or {"novi_group_guid": guid, "mailman_list_id": list_id, "error": "sync failed"})
+
+    return results

app/methods/e_zoom_methods.py

@@ -0,0 +1,226 @@
+import datetime, json, requests, time
+from typing import Dict, List, Optional, Set, Union
+from app.db_sql import sql_select, sql_update, sql_insert
+from app.lib_general import log, logging, logger_reset
+
+# Zoom API Configuration (Defaults)
+ZOOM_OAUTH_URL = "https://zoom.us/oauth/token"
+ZOOM_API_BASE = "https://api.zoom.us/v2"
+
+# In-memory token cache (Standard Aether pattern)
+zoom_api_cache = {
+    "access_token": None,
+    "expire_on": None,
+    "headers": {}
+}
+
+@logger_reset
+def get_zoom_access_token(force_refresh: bool = False):
+    """
+    Retrieves a Zoom Access Token using Server-to-Server OAuth.
+    Credentials should be stored in the 'data_store' table with code 'zoom_api_config'.
+    """
+    log.setLevel(logging.INFO)
+    
+    # 1. Check Cache
+    if not force_refresh and zoom_api_cache["access_token"] and zoom_api_cache["expire_on"]:
+        if datetime.datetime.now() < zoom_api_cache["expire_on"]:
+            log.debug("Using cached Zoom access token.")
+            return zoom_api_cache
+
+    # 2. Load Credentials from Data Store
+    # Logic: Look for 'zoom_api_config' in data_store
+    config_rec = sql_select(
+        table_name='data_store',
+        field_name='code',
+        field_value='zoom_api_config'
+    )
+    
+    if not config_rec:
+        log.error("Zoom API credentials not found in data_store (code='zoom_api_config').")
+        return False
+
+    try:
+        config_data = json.loads(config_rec['text'])
+        client_id = config_data['client_id']
+        client_secret = config_data['client_secret']
+        account_id = config_data['account_id']
+    except Exception as e:
+        log.error(f"Failed to parse Zoom credentials from data_store: {e}")
+        return False
+
+    # 3. Request Token
+    log.info("Requesting new Zoom Access Token...")
+    params = {
+        "grant_type": "account_credentials",
+        "account_id": account_id
+    }
+    
+    try:
+        resp = requests.post(
+            ZOOM_OAUTH_URL,
+            params=params,
+            auth=(client_id, client_secret),
+            timeout=10
+        )
+        
+        if resp.status_code != 200:
+            log.error(f"Zoom OAuth failure: {resp.status_code} - {resp.text}")
+            return False
+            
+        data = resp.json()
+        zoom_api_cache["access_token"] = data["access_token"]
+        # Set expiry with a 60s safety buffer
+        zoom_api_cache["expire_on"] = datetime.datetime.now() + datetime.timedelta(seconds=data["expires_in"] - 60)
+        zoom_api_cache["headers"] = {
+            "Authorization": f"Bearer {data['access_token']}",
+            "Accept": "application/json"
+        }
+        
+        log.info("Successfully obtained Zoom access token.")
+        return zoom_api_cache
+
+    except Exception as e:
+        log.exception(f"Unexpected error during Zoom OAuth: {e}")
+        return False
+
+@logger_reset
+def get_zoom_tickets(event_id: str, page_size: int = 300, next_page_token: str = None):
+    """
+    Retrieves 'Tickets' (Attendees) for a specific Zoom Event.
+    Endpoint: GET /zoom_events/events/{eventId}/tickets
+    """
+    auth = get_zoom_access_token()
+    if not auth: return False
+
+    url = f"{ZOOM_API_BASE}/zoom_events/events/{event_id}/tickets"
+    params = {"page_size": page_size}
+    if next_page_token: params["next_page_token"] = next_page_token
+
+    try:
+        resp = requests.get(url, headers=auth["headers"], params=params, timeout=15)
+        if resp.status_code != 200:
+            log.error(f"Zoom API Error: {resp.status_code} - {resp.text}")
+            return False
+            
+        return resp.json()
+    except Exception as e:
+        log.exception(f"Failed to fetch tickets from Zoom: {e}")
+        return False
+
+@logger_reset
+def sync_zoom_attendees_to_event(event_id_random: str, zoom_event_id: str):
+    """
+    Atomic sync action: Pulls Zoom tickets and upserts Aether event_person records.
+    Uses Zoom ticket_id as the primary external identifier.
+    """
+    from app.methods.event_person_methods import create_update_event_person_obj_v4
+    
+    log.info(f"Starting Zoom sync for event {event_id_random} (Zoom ID: {zoom_event_id})")
+    
+    # 1. Fetch tickets from Zoom
+    zoom_data = get_zoom_tickets(zoom_event_id)
+    if not zoom_data:
+        log.error("Failed to retrieve tickets from Zoom API.")
+        return False
+    
+    tickets = zoom_data.get("tickets", [])
+    log.info(f"Found {len(tickets)} tickets in Zoom.")
+
+    # 2. Resolve Local Context
+    if event_id_int := redis_lookup_id_random(record_id_random=event_id_random, table_name='event'):
+        pass
+    else:
+        log.error(f"Aether Event ID {event_id_random} not found.")
+        return False
+
+    # Get account_id for this event
+    res = sql_select(sql="SELECT account_id FROM event WHERE id = :id", data={'id': event_id_int})
+    account_id_int = res.get('account_id') if res else None
+    if not account_id_int:
+        log.error("Could not resolve account_id for event.")
+        return False
+
+    sync_results = {
+        "total": len(tickets),
+        "created": 0,
+        "updated": 0,
+        "failed": 0
+    }
+
+    # 3. Iterate and Upsert
+    for ticket in tickets:
+        try:
+            ticket_id = ticket.get('ticket_id')
+            email = ticket.get('email')
+            first_name = ticket.get('first_name', '').strip()
+            last_name = ticket.get('last_name', '').strip()
+            
+            # Standard External ID Pattern for Zoom: {zoom_event_id}:{ticket_id}
+            external_id = f"{zoom_event_id}:{ticket_id}"
+            
+            # Prepare Aether Data structure (event_person_methods v4 expects this nested shape)
+            event_person_data = {
+                "enable": True,
+                "external_id": external_id,
+                "external_person_id": ticket.get('registrant_id'), # Zoom User ID
+                "external_registration_id": ticket_id,
+                "allow_tracking": True, # Lead retrieval demo default
+                
+                # Nested Person Profile (Person Table)
+                "event_person_profile": {
+                    "given_name": first_name,
+                    "family_name": last_name,
+                    "full_name": f"{first_name} {last_name}".strip(),
+                    "email": email,
+                    "enable": True
+                },
+                
+                # Nested Badge Data (Event Badge Table)
+                "event_badge": {
+                    "given_name": first_name,
+                    "family_name": last_name,
+                    "full_name": f"{first_name} {last_name}".strip(),
+                    "email": email,
+                    "badge_type": ticket.get('ticket_type_name', 'Attendee'),
+                    "badge_type_code": ticket.get('ticket_type_id'),
+                    "external_id": external_id,
+                    "enable": True
+                }
+            }
+
+            # 4. Check for existing Event Person to determine create vs update
+            # (Matches Impexium pattern of looking up by external_id)
+            existing = sql_select(
+                sql="SELECT id, event_badge_id, event_person_profile_id FROM event_person WHERE event_id = :eid AND external_id = :ext",
+                data={'eid': event_id_int, 'ext': external_id}
+            )
+
+            if existing:
+                log.info(f"Updating existing record for {email} ({external_id})")
+                res = create_update_event_person_obj_v4(
+                    event_person_dict_obj = event_person_data,
+                    event_person_id = existing['id'],
+                    account_id = account_id_int,
+                    event_id = event_id_int,
+                    event_badge_id = existing['event_badge_id'],
+                    event_person_profile_id = existing['event_person_profile_id']
+                )
+                if res: sync_results["updated"] += 1
+                else: sync_results["failed"] += 1
+            else:
+                log.info(f"Creating new record for {email} ({external_id})")
+                res = create_update_event_person_obj_v4(
+                    event_person_dict_obj = event_person_data,
+                    account_id = account_id_int,
+                    event_id = event_id_int
+                )
+                if res: sync_results["created"] += 1
+                else: sync_results["failed"] += 1
+
+        except Exception as e:
+            log.exception(f"Failed to process ticket {ticket.get('ticket_id')}: {e}")
+            sync_results["failed"] += 1
+
+    log.info(f"Zoom sync complete: {sync_results}")
+    return sync_results

app/methods/event_abstract_methods.py

@@ -70,7 +70,7 @@ def load_event_abstract_obj(
 
     # Updated 2023-03-20
     if inc_event_file_list:
-        # log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+        # log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
         log.info('Need to include event file list...')
 
         from app.methods.event_file_methods import get_event_file_rec_list, load_event_file_obj
@@ -123,7 +123,7 @@ def load_event_abstract_obj(
             log.debug(event_person_obj)
             event_abstract_obj.event_person = event_person_obj
         else:
-            log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+            log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
             log.debug(event_person_obj)
             event_abstract_obj.event_person = None
         log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
@@ -227,7 +227,7 @@ def get_event_abstract_rec_list(
         ORDER BY event_abstract.priority DESC, event_abstract.sort DESC, event_abstract.name ASC, `event_abstract`.created_on DESC, `event_abstract`.updated_on DESC
         {sql_limit};
         """
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(sql)
 
     if event_abstract_rec_li_result := sql_select(data=data, sql=sql, as_list=True):
@@ -329,7 +329,7 @@ def create_update_event_abstract_obj_old(
         fail_any: bool = False, # Fail if any thing goes wrong for sub objects
         return_outline: bool = False,
         ) -> int|bool:
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
     log.info('Checking requirements...')
@@ -379,7 +379,7 @@ def create_update_event_abstract_obj_old(
             event_abstract_dict['event_person_id'] = event_person_id
         try:
             event_abstract_obj = Event_Abstract_In(**event_abstract_dict)
-            log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+            log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
             log.debug(event_abstract_obj)
         except ValidationError as e:
             log.error(e.json())

app/methods/event_badge_methods.py

@@ -322,10 +322,9 @@ def create_update_event_badge_obj_v4(
     elif event_person_id := event_badge_obj.event_person_id: pass
 
     if event_badge_id:
-        if event_badge_dict_up_result := sql_update(data=event_badge_dict, table_name='event_badge', rm_id_random=True): pass
+        event_badge_dict_up_result = sql_update(data=event_badge_dict, table_name='event_badge', record_id=event_badge_id, rm_id_random=True)
-        else:
+        if event_badge_dict_up_result is False:
-            log.warning(f'Event Badge not updated. Event Badge ID: {event_badge_id}')
+            log.warning(f'Event Badge update failed (DB error). Event Badge ID: {event_badge_id}')
-            log.debug(event_badge_dict_up_result)
             return False
         log.debug(event_badge_dict_up_result)
     else:

app/methods/event_file_methods.py

@@ -83,7 +83,15 @@ def load_event_file_obj(
                 enabled = enabled,
                 ):
             event_file_obj.hosted_file = hosted_file_obj
-            # event_file_obj.hosted_file = hosted_file_obj.dict(by_alias=by_alias, exclude_unset=exclude_unset)
+            # Explicitly populate convenience fields from hosted_file_obj
+            if hosted_file_obj.hash_sha256:
+                event_file_obj.hosted_file_hash_sha256 = hosted_file_obj.hash_sha256
+            if hosted_file_obj.subdirectory_path:
+                event_file_obj.hosted_file_subdirectory_path = hosted_file_obj.subdirectory_path
+            if hosted_file_obj.content_type:
+                event_file_obj.hosted_file_content_type = hosted_file_obj.content_type
+            if hosted_file_obj.size:
+                event_file_obj.hosted_file_size = str(hosted_file_obj.size) # Ensure it's a string as per model definition
         else:
             event_file_obj.hosted_file = {}
     else:

app/methods/event_list_methods.py

@@ -1,122 +0,0 @@
-from __future__ import annotations
-import datetime
-
-from typing import Dict, List, Optional, Set, Union
-from pydantic import BaseModel, EmailStr, Field, PrivateAttr, ValidationError, validator
-
-from app.db_sql import redis_lookup_id_random, sql_insert, sql_select, sql_update
-from app.lib_general import log, logging
-
-from app.methods.event_methods import load_event_obj
-
-
-# ### BEGIN ### API Event Methods ### load_event_obj_list() ###
-def load_event_obj_list(
-        account_id: int|str,
-        limit: int = 1000,
-        model_as_dict: bool = False,
-        enabled: str = 'enabled', # enabled, disabled, all
-        inc_contact_1: bool = False,
-        inc_contact_2: bool = False,
-        inc_contact_3: bool = False,
-        inc_event_abstract_list: bool = False,
-        inc_event_badge_list: bool = False,
-        inc_event_cfg: bool = False,
-        inc_event_device_list: bool = False,
-        inc_event_exhibit_list: bool = False,
-        inc_event_file_list: bool = False,
-        inc_event_location: bool = False, # For event_session child object
-        inc_event_location_list: bool = False,
-        inc_event_person_list: bool = False,
-        inc_event_presentation_list: bool = False,
-        inc_event_presenter_cat: bool = False, # For event_session child object
-        inc_event_presenter_list: bool = False,
-        inc_event_registration_cfg: bool = False,
-        inc_event_registration_list: bool = False,
-        inc_event_session_list: bool = False,
-        inc_event_track: bool = False, # For event_session child object
-        inc_event_track_list: bool = False,
-        inc_location_address: bool = False,
-        inc_poc_event_person: bool = False,
-        inc_person: bool = False,
-        inc_user: bool = False,
-        ) -> list|bool:
-    log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(locals())
-
-    if account_id := redis_lookup_id_random(record_id_random=account_id, table_name='account'): pass
-    else: return False
-
-    data = {}
-    data['account_id'] = account_id
-
-    if enabled in ['enabled', 'disabled', 'all']:
-        if enabled == 'enabled':
-            data['enable'] = True
-            sql_enabled = f'AND `tbl`.enable = :enable'
-        elif enabled == 'disabled':
-            data['enable'] = False
-            sql_enabled = f'AND `tbl`.enable = :enable'
-        elif enabled == 'all':
-            sql_enabled = ''
-    # else: tbl_obj['account'] = None
-
-    if limit:
-        data['limit'] = limit
-        sql_limit = f'LIMIT :limit'
-    else:
-        sql_limit = ''
-
-    sql = f"""
-        SELECT `tbl`.id AS 'event_id', `tbl`.id_random AS 'event_id_random'
-        FROM `event` AS `tbl`
-        WHERE `tbl`.account_id = :account_id
-        {sql_enabled}
-        ORDER BY `tbl`.created_on DESC, `tbl`.updated_on DESC
-        {sql_limit};
-        """
-
-    if event_rec_li_result := sql_select(data=data, sql=sql, as_list=True):
-        log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-        log.debug(event_rec_li_result)
-        event_result_li = []
-        for event_rec in event_rec_li_result:
-            event_id = event_rec.get('event_id', None)
-            if event_result := load_event_obj(
-                event_id = event_id,
-                limit = limit,
-                model_as_dict = model_as_dict,
-                enabled = enabled,
-                # inc_location_address = inc_address,
-                # inc_contact_1 = inc_contact,
-                # inc_contact_2 = inc_contact,
-                # inc_contact_3 = inc_contact,
-                # inc_event_abstract_list = inc_event_abstract_list,
-                # inc_event_badge_list = inc_event_badge_list,
-                # inc_event_device_list = inc_event_device_list,
-                inc_event_exhibit_list = inc_event_exhibit_list,
-                inc_event_file_list = inc_event_file_list,
-                inc_event_location_list = inc_event_location_list,
-                inc_event_person_list = inc_event_person_list,
-                inc_event_presentation_list = inc_event_presentation_list,
-                inc_event_presenter_list = inc_event_presenter_list,
-                inc_event_registration_list = inc_event_registration_list,
-                inc_event_session_list = inc_event_session_list,
-                inc_event_track_list = inc_event_track_list,
-                # inc_person = inc_person,
-                # inc_user = inc_user,
-                ):
-                log.debug(event_result)
-                event_result_li.append(event_result)
-            else:
-                log.debug(event_result)
-                event_result_li.append(None)
-        log.debug(event_result_li)
-    else:
-        log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-        log.debug(event_rec_li_result)
-        event_result_li = []
-    log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-
-    return event_result_li
-# ### END ### API Event Methods ### load_event_obj_list() ###

app/methods/event_person_methods.py

@@ -3,7 +3,7 @@ import datetime
 from typing import Dict, List, Optional, Set, Union
 from pydantic import BaseModel, EmailStr, Field, PrivateAttr, ValidationError, validator
 
-from app.db_sql import get_account_id_w_for_type_id, redis_lookup_id_random, sql_insert, sql_select, sql_update
+from app.db_sql import get_account_id_w_for_type_id, redis_lookup_id_random, sql_insert, sql_select, sql_update, get_id_random
 from app.lib_general import log, logging, logger_reset
 
 # from app.methods.event_abstract_methods import load_event_abstract_obj
@@ -355,7 +355,7 @@ def create_update_event_person_obj_v4(
         fail_any: bool = False, # Fail if any thing goes wrong for sub objects
         return_outline: bool = False,
         ) -> int|bool:
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
     # ### SECTION ### Secondary data validation
@@ -420,7 +420,19 @@ def create_update_event_person_obj_v4(
         if account_id:
             event_person_dict['account_id'] = account_id
         if event_id:
-            event_person_dict['event_id'] = event_id
+            # The model expects random-string IDs (eg. id_random). If we have an
+            # integer internal ID, convert it to the random string form so the
+            # Pydantic root_validator preserves it. This ensures `event_id` is
+            # present when inserting a new `event_person` record.
+            if isinstance(event_id, int):
+                if idr := get_id_random(record_id=event_id, table_name='event'):
+                    event_person_dict['event_id_random'] = idr
+                else:
+                    # Fallback: set the integer (will likely be removed by the model),
+                    # but allow downstream logic to attempt insertion.
+                    event_person_dict['event_id'] = event_id
+            else:
+                event_person_dict['event_id'] = event_id
         try:
             event_person_obj = Event_Person_Base(**event_person_dict)
         except ValidationError as e:
@@ -434,7 +446,16 @@ def create_update_event_person_obj_v4(
         if account_id:
             event_person_obj.account_id = account_id
         if event_id:
-            event_person_obj.event_id = event_id
+            # If an integer internal ID was provided, convert to the random ID
+            # string form for the Pydantic object so it is preserved when
+            # serializing to the DB insert/update payload.
+            if isinstance(event_id, int):
+                if idr := get_id_random(record_id=event_id, table_name='event'):
+                    event_person_obj.event_id = idr
+                else:
+                    event_person_obj.event_id = event_id
+            else:
+                event_person_obj.event_id = event_id
     log.debug(event_person_obj)
 
     event_person_dict = event_person_obj.dict(by_alias=False, exclude_defaults=False, exclude_unset=True, exclude={'event_badge', 'event_person_profile', 'event_registration', 'created_on', 'updated_on', 'external_id_old'})
@@ -453,11 +474,11 @@ def create_update_event_person_obj_v4(
             event_person_profile_id = event_person_obj.event_person_profile_id
 
     if event_person_id:
-        if event_person_dict_up_result := sql_update(data=event_person_dict, table_name='event_person', rm_id_random=True): pass
+        event_person_dict_up_result = sql_update(data=event_person_dict, table_name='event_person', record_id=event_person_id, rm_id_random=True)
-        else:
+        if event_person_dict_up_result is False:
-            log.warning(f'Event Person not updated. Event Person ID: {event_person_id}')
+            log.warning(f'Event Person update failed (DB error). Event Person ID: {event_person_id}')
-            log.debug(event_person_dict_up_result)
             return False
+        # None means 0 rows affected (record unchanged) — not an error, continue to sub-objects
         log.debug(event_person_dict_up_result)
     else:
         if event_person_dict_in_result := sql_insert(data=event_person_dict, table_name='event_person', rm_id_random=True, id_random_length=None): pass

app/methods/event_person_profile_methods.py

@@ -154,11 +154,12 @@ def create_update_event_person_profile_obj_v4(
         contact_id = event_person_profile_obj.contact_id
 
     if event_person_profile_id:
-        if event_person_profile_dict_up_result := sql_update(data=event_person_profile_dict, table_name='event_person_profile', rm_id_random=True): pass
+        event_person_profile_dict_up_result = sql_update(data=event_person_profile_dict, table_name='event_person_profile', record_id=event_person_profile_id, rm_id_random=True)
-        else:
+        if event_person_profile_dict_up_result is False:
-            log.warning(f'Event Person Profile not updated. Event Person Profile ID: {event_person_profile_id}')
+            log.warning(f'Event Person Profile update failed (DB error). Event Person Profile ID: {event_person_profile_id}')
             log.debug(event_person_profile_dict_up_result)
             return False
+        # None means 0 rows affected (record unchanged) — not an error
         log.debug(event_person_profile_dict_up_result)
     else:
         if event_person_profile_dict_in_result := sql_insert(data=event_person_profile_dict, table_name='event_person_profile', rm_id_random=True, id_random_length=8): pass

app/methods/event_presentation_methods.py

@@ -165,7 +165,7 @@ def load_event_presentation_obj(
             WHERE `event_presenter`.event_presentation_id = :event_presentation_id
                 {sql_hidden}
                 {sql_enabled}
-            ORDER BY `event_presenter`.priority DESC, -`event_presenter`.sort DESC, `event_presenter`.family_name ASC, `event_presenter`.given_name ASC, `event_presenter`.display_name ASC, `event_presenter`.full_name ASC, `event_presenter`.created_on DESC, `event_presenter`.updated_on DESC;
+            ORDER BY `event_presenter`.priority DESC, -`event_presenter`.sort DESC, `event_presenter`.family_name ASC, `event_presenter`.given_name ASC, `event_presenter`.full_name ASC, `event_presenter`.created_on DESC, `event_presenter`.updated_on DESC;
             """
         # log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
         log.debug(sql)
@@ -429,9 +429,9 @@ def create_update_event_presentation_obj_v4(
         event_presentation_dict = event_presentation_obj.dict(by_alias=False, exclude_defaults=False, exclude_unset=True, exclude={'event_presenter', 'event_presenter_list', 'created_on', 'updated_on'})
 
     if event_presentation_id:
-        if event_presentation_dict_up_result := sql_update(data=event_presentation_dict, table_name='event_presentation', rm_id_random=True): pass
+        event_presentation_dict_up_result = sql_update(data=event_presentation_dict, table_name='event_presentation', record_id=event_presentation_id, rm_id_random=True)
-        else:
+        if event_presentation_dict_up_result is False:
-            log.warning(f'Event Presentation not updated. Event Presentation ID: {event_presentation_id}')
+            log.warning(f'Event Presentation update failed (DB error). Event Presentation ID: {event_presentation_id}')
             log.debug(event_presentation_dict_up_result)
             return False
         log.debug(event_presentation_dict_up_result)

app/methods/event_presenter_methods.py

@@ -262,7 +262,7 @@ def get_event_presenter_rec_list(
             {sql_where_event_presentation_id}
             {sql_hidden}
             {sql_enabled}
-        ORDER BY `event_presenter`.priority DESC, -`event_presenter`.sort DESC, `event_presenter`.family_name ASC, `event_presenter`.given_name ASC, `event_presenter`.display_name ASC, `event_presenter`.full_name ASC, `event_presenter`.created_on DESC, `event_presenter`.updated_on DESC
+        ORDER BY `event_presenter`.priority DESC, -`event_presenter`.sort DESC, `event_presenter`.family_name ASC, `event_presenter`.given_name ASC, `event_presenter`.full_name ASC, `event_presenter`.created_on DESC, `event_presenter`.updated_on DESC
         {sql_limit};
         """
 
@@ -404,9 +404,9 @@ def create_update_event_presenter_obj_v4(
         event_presenter_dict = event_presenter_obj.dict(by_alias=False, exclude_defaults=False, exclude_unset=True, exclude={'created_on', 'updated_on'})
 
     if event_presenter_id:
-        if event_presenter_dict_up_result := sql_update(data=event_presenter_dict, table_name='event_presenter', rm_id_random=True): pass
+        event_presenter_dict_up_result = sql_update(data=event_presenter_dict, table_name='event_presenter', record_id=event_presenter_id, rm_id_random=True)
-        else:
+        if event_presenter_dict_up_result is False:
-            log.warning(f'Event Presenter not updated. Event Presenter ID: {event_presenter_id}')
+            log.warning(f'Event Presenter update failed (DB error). Event Presenter ID: {event_presenter_id}')
             log.debug(event_presenter_dict_up_result)
             return False
         log.debug(event_presenter_dict_up_result)

app/methods/hosted_file_methods.py

@@ -1,24 +1,71 @@
-import datetime, hashlib, os, pathlib, shutil, time
+import datetime, hashlib, mimetypes, os, pathlib, shutil, time
 
 from fastapi import File, UploadFile
 from typing import Dict, List, Optional, Set, Union
 from pydantic import BaseModel, EmailStr, Field, PrivateAttr, ValidationError, validator
 
 from app.config import settings
-from app.db_sql import redis_lookup_id_random, sql_delete, sql_enable_part, sql_insert, sql_limit_offset_part, sql_select, sql_update
+from app.db_sql import redis_lookup_id_random, sql_delete, sql_enable_part, sql_insert, sql_limit_offset_part, sql_select, sql_update, get_id_random
 from app.lib_general import log, logging, logger_reset
 
 from app.models.hosted_file_models import Hosted_File_Base
 
 
+# ### BEGIN ### API Hosted File Methods ### directory_check_method() ###
+# Extracted 2026-02-03
+def directory_check_method(rm_orphan: bool = False):
+    """
+    Logic for scanning the hosted_files root and migrating legacy files to 2-char subdirectories.
+    Returns a list of processed files.
+    """
+    hosted_files_path = settings.FILES_PATH['hosted_files_root']
+    if not os.path.isdir(hosted_files_path):
+        return False
+
+    directory_list = os.listdir(hosted_files_path)
+    result_list = []
+    count = 0
+
+    for item in directory_list:
+        if count >= 100: break # Rate limited per call
+        
+        file_path = os.path.join(hosted_files_path, item)
+        if os.path.isfile(file_path):
+            if '.file' not in item: continue
+            
+            log.info(f'Migrating legacy file to subdirectory: {item}')
+            result_list.append(file_path)
+
+            # Create a subdirectory with the first 2 characters of the hash
+            full_subdirectory_path = os.path.join(hosted_files_path, item[:2])
+            os.makedirs(full_subdirectory_path, exist_ok=True)
+
+            # Move the file
+            shutil.move(file_path, os.path.join(full_subdirectory_path, item))
+            count += 1
+            
+    return result_list
+# ### END ### API Hosted File Methods ### directory_check_method() ###
+
+
 # ### BEGIN ### API Hosted File Methods ### create_hosted_file_obj() ###
 @logger_reset
 def create_hosted_file_obj(hosted_file_obj_new:Hosted_File_Base):
     log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    # hosted_file_obj_data = hosted_file_obj_new.dict(by_alias=False, exclude_defaults=False, exclude_unset=True, exclude={'created_on', 'updated_on'})
+    # We need to explicitly include subdirectory_path because it has Field(exclude=True) in the model
-    hosted_file_obj_data = hosted_file_obj_new.dict(by_alias=False, exclude_defaults=False, exclude_unset=True, exclude={'saved', 'already_exists', 'copy_timer', 'created_on', 'updated_on'})
+    # which prevents it from showing in the public API, but also strips it from .dict() by default.
+    hosted_file_obj_data = hosted_file_obj_new.dict(
+        by_alias=False, 
+        exclude_defaults=False, 
+        exclude_unset=True, 
+        exclude={'saved', 'already_exists', 'copy_timer', 'created_on', 'updated_on'}
+    )
+    
+    # Force inclusion of subdirectory_path if present in the object
+    if hasattr(hosted_file_obj_new, 'subdirectory_path') and hosted_file_obj_new.subdirectory_path:
+        hosted_file_obj_data['subdirectory_path'] = hosted_file_obj_new.subdirectory_path
 
     if hosted_file_obj_in_result := sql_insert(data=hosted_file_obj_data, table_name='hosted_file', rm_id_random=True, id_random_length=8): pass
     else:
@@ -195,59 +242,38 @@ async def save_file(
     log.debug(locals())
 
     hosted_files_path = settings.FILES_PATH['hosted_files_root']
-    # hosted_files_path = '/home/scott/tmp/hosted_files_dev/'
     log.info(f'Hosted Files Path: {hosted_files_path}')
     log.debug(shutil.disk_usage(hosted_files_path))
 
-    log.debug(dir(file))
-    log.debug(f'{file.filename}')
-
     if file.filename.endswith('.docwin'):
-        log.warning('Fixing win extension')
         file.filename = file.filename.replace('.docwin', '.doc')
     if file.filename.endswith('.docxwin'):
-        log.warning('Fixing win extension')
         file.filename = file.filename.replace('.docxwin', '.docx')
-
     if file.filename.endswith('.odpmac'):
-        log.warning('Fixing mac extension')
         file.filename = file.filename.replace('.odpmac', '.odp')
-
     if file.filename.endswith('.odpwin'):
-        log.warning('Fixing win extension')
         file.filename = file.filename.replace('.odpwin', '.odp')
-
     if file.filename.endswith('.pdfmac'):
-        log.warning('Fixing mac extension')
         file.filename = file.filename.replace('.pdfmac', '.pdf')
-
     if file.filename.endswith('.pdfwin'):
-        log.warning('Fixing win extension')
         file.filename = file.filename.replace('.pdfwin', '.pdf')
-
     if file.filename.endswith('.pptmac'):
-        log.warning('Fixing mac extension')
         file.filename = file.filename.replace('.pptmac', '.ppt')
     if file.filename.endswith('.pptxmac'):
-        log.warning('Fixing mac extension')
         file.filename = file.filename.replace('.pptxmac', '.pptx')
-
     if file.filename.endswith('.pptwin'):
-        log.warning('Fixing win extension')
         file.filename = file.filename.replace('.pptwin', '.ppt')
     if file.filename.endswith('.pptxwin'):
-        log.warning('Fixing win extension')
         file.filename = file.filename.replace('.pptxwin', '.pptx')
-
     if file.filename.endswith('.xlswin'):
-        log.warning('Fixing win extension')
         file.filename = file.filename.replace('.xlswin', '.xls')
     if file.filename.endswith('.xlsxwin'):
-        log.warning('Fixing win extension')
         file.filename = file.filename.replace('.xlsxwin', '.xlsx')
 
     file_info: dict = {}
     file_info['saved'] = None
+    file_info['account_id'] = account_id
+    file_info['account_id_random'] = account_id_random
     file_info['link_to_type'] = link_to_type
     file_info['link_to_id'] = link_to_id
     file_info['link_to_id_random'] = link_to_id_random
@@ -264,136 +290,48 @@ async def save_file(
     else:
         file_info['extension_allowed'] = None
 
-    # There is a difference between Content-Type and MIME type.
+    file_info['content_type'] = file.content_type
-    # https://stackoverflow.com/questions/3452381/whats-the-difference-of-contenttype-and-mimetype
-    file_info['content_type'] = file.content_type # might also include charset or other parameters
-    # file_info['mimetype'] = file.mimetype # This may need to be filled in a different way?
 
     file.file.seek(0, os.SEEK_END)
     file_size = file.file.tell()
-    file.file.seek(0) # The file will not properly save if seek is not reset to 0.
+    file.file.seek(0)
-    log.debug(file_size)
     file_info['size'] = file_size
 
     file_hash = await get_file_object_hash(file.file)
-    log.debug(file_hash)
     file_info['hash_sha256'] = file_hash
 
-    # 16384 bytes is the default
-    # 4096 8192 16384 32768 65536 131072 262144 524288 1048576 bytes
     buffer_size = 524288
-
+    f_src = file.file
-    #f_src = open(file_src, 'rb')
-    f_src = file.file # Don't need to do open(file_src, 'rb') since it is already "open"
 
     file_hash_subdirectory = file_hash[0:2]
     subdirectory_dest = os.path.join(hosted_files_path, file_hash_subdirectory)
-    log.debug(subdirectory_dest)
+    log.info(f"Subdirectory Dest: {subdirectory_dest}")
     pathlib.Path(subdirectory_dest).mkdir(parents=True, exist_ok=True)
     file_info['subdirectory_path'] = file_hash_subdirectory
 
-    #file_dest = f'{hosted_files_path}{file.filename}'
-    # file_dest = f'{hosted_files_path}{file_hash}.file'
-
-    file_dest = os.path.join(hosted_files_path, f'{file_hash}.file')
     file_dest_w_subdir = os.path.join(subdirectory_dest, f'{file_hash}.file')
-
-    existing_file_check = pathlib.Path(file_dest)
     existing_file_check_subdir = pathlib.Path(file_dest_w_subdir)
 
-
+    if existing_file_check_subdir.exists():
-    if existing_file_check.exists():
-        log.warning('This file already exists at the destination without the subdirectory. Not re-saving. Going to move the current file and update the database later.')
-        file_info['already_exists'] = True
-        file_info['already_exists_subdir'] = False
-        try:
-            log.info('Moving file to sub directory destination...')
-            timer_start = time.process_time()
-            shutil.move(existing_file_check, existing_file_check_subdir)
-            timer_end = time.process_time()
-            elapsed_time = timer_end - timer_start
-            log.debug(f'Elapsed time: {elapsed_time}')
-            file_info['copy_timer'] = elapsed_time
-            file_info['saved'] = True
-
-            log.info(f'File moved to: {hosted_files_path}')
-        except Exception as e:
-            log.exception('*** An exception happened. ***')
-            log.exception(repr(e))
-            log.exception('***')
-            log.exception(str(e))
-            log.exception('^^^ exception ^^^')
-
-            file_info['copy_timer'] = 0
-            file_info['saved'] = False
-    elif existing_file_check_subdir.exists():
-        log.warning('This file already exists at the destination with the subdirectory. Not re-saving.')
         file_info['already_exists'] = True
         file_info['already_exists_subdir'] = True
         file_info['copy_timer'] = 0
         file_info['saved'] = True
     else:
-        # log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-        log.warning('This file does not already exist at the destination with or without the subdirectory.')
         file_info['already_exists'] = False
         file_info['already_exists_subdir'] = False
         try:
-            log.info('Saving file to destination...')
             f_dest = open(file_dest_w_subdir, 'wb')
             timer_start = time.process_time()
             shutil.copyfileobj(f_src, f_dest, buffer_size)
             timer_end = time.process_time()
-            elapsed_time = timer_end - timer_start
+            file_info['copy_timer'] = timer_end - timer_start
-            log.debug(f'Elapsed time: {elapsed_time}')
-            file_info['copy_timer'] = elapsed_time
             file_info['saved'] = True
-
-            log.info(f'File saved to: {hosted_files_path}')
         except Exception as e:
-            log.exception('*** An exception happened. ***')
+            log.exception(f'Error saving file: {e}')
-            log.exception(repr(e))
-            log.exception('***')
-            log.exception(str(e))
-            log.exception('^^^ exception ^^^')
-
             file_info['copy_timer'] = 0
             file_info['saved'] = False
             return False
-    log.info(f'Disk usage: {shutil.disk_usage(hosted_files_path)}')
-    log.info(f"Filename: {file_info['filename']}")
-    log.info(f"Subdirectory Path: {file_info['subdirectory_path']}")
-    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(file_info)
-
-
-    # if existing_file_check.exists():
-    #     file_info['already_exists'] = True
-    #     file_info['copy_timer'] = 0
-    #     file_info['saved'] = True
-    # else:
-    #     file_info['already_exists'] = False
-    #     try:
-    #         f_dest = open(file_dest, 'wb')
-    #         timer_start = time.process_time()
-    #         shutil.copyfileobj(f_src, f_dest, buffer_size)
-    #         timer_end = time.process_time()
-    #         elapsed_time = timer_end - timer_start
-    #         log.debug(f'Elapsed time: {elapsed_time}')
-    #         file_info['copy_timer'] = elapsed_time
-    #         file_info['saved'] = True
-    #     except Exception as e:
-    #         log.exception('*** An exception happened. ***')
-    #         log.exception(repr(e))
-    #         log.exception('***')
-    #         log.exception(str(e))
-    #         log.exception('^^^ exception ^^^')
-
-    #         file_info['copy_timer'] = 0
-    #         file_info['saved'] = False
-
-
-    log.debug(shutil.disk_usage(hosted_files_path))
-
     return file_info
 # ### END ### API Hosted File Methods ### save_file() ###
 
@@ -408,119 +346,65 @@ async def save_file_to_hosted_file(
         account_id: int,
         link_to_type: str,
         link_to_id: int,
+        account_id_random: str = None,
+        link_to_id_random: str = None,
         ):
     log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
     hosted_files_path = settings.FILES_PATH['hosted_files_root']
-    log.info(f'Hosted Files Path: {hosted_files_path}')
-    log.debug(shutil.disk_usage(hosted_files_path))
-
-    log.debug(file_path)
-    log.debug(f'Filename: {filename} Extension: {extension}')
-
     file_obj = open(file_path, 'rb')
 
-
     file_info: dict = {}
     file_info['saved'] = None
+    file_info['account_id'] = account_id
+    file_info['account_id_random'] = account_id_random
     file_info['link_to_type'] = link_to_type
     file_info['link_to_id'] = link_to_id
+    file_info['link_to_id_random'] = link_to_id_random
     file_info['filename'] = filename
-    file_info['extension'] = extension # guess_file_extension(filename=filename)
+    file_info['extension'] = extension
-
+    file_info['content_type'] = mimetypes.guess_type(filename)[0]
-    # if check_allowed_extension:
-    #     if allowed_file_extension(extension=file_info['extension'], extension_list=['jpg','png','webp']):
-    #         file_info['extension_allowed'] = True
-    #     else:
-    #         file_info['extension_allowed'] = False
-    #         file_info['saved'] = False
-    #         return file_info
-    # else:
-    #     file_info['extension_allowed'] = None
-
-    # There is a difference between Content-Type and MIME type.
-    # https://stackoverflow.com/questions/3452381/whats-the-difference-of-contenttype-and-mimetype
-    # file_info['content_type'] = file.content_type # might also include charset or other parameters
-    # file_info['mimetype'] = file.mimetype # This may need to be filled in a different way?
 
     file_obj.seek(0, os.SEEK_END)
     file_size = file_obj.tell()
-    file_obj.seek(0) # The file will not properly save if seek is not reset to 0.
+    file_obj.seek(0)
-    log.debug(file_size)
     file_info['size'] = file_size
 
     file_hash = await get_file_object_hash(file_obj)
-    log.debug(file_hash)
     file_info['hash_sha256'] = file_hash
 
-    # 16384 bytes is the default
-    # 4096 8192 16384 32768 65536 131072 262144 524288 1048576 bytes
     buffer_size = 524288
-
+    f_src = file_obj
-    #f_src = open(file_src, 'rb')
-    f_src = file_obj # Don't need to do open(file_src, 'rb') since it is already "open"
 
     file_hash_subdirectory = file_hash[0:2]
     subdirectory_dest = os.path.join(hosted_files_path, file_hash_subdirectory)
-    log.debug(subdirectory_dest)
     pathlib.Path(subdirectory_dest).mkdir(parents=True, exist_ok=True)
     file_info['subdirectory_path'] = file_hash_subdirectory
 
-    #file_dest = f'{hosted_files_path}{file.filename}'
-    # file_dest = f'{hosted_files_path}{file_hash}.file'
-
-    file_dest = os.path.join(hosted_files_path, f'{file_hash}.file')
     file_dest_w_subdir = os.path.join(subdirectory_dest, f'{file_hash}.file')
-
-    existing_file_check = pathlib.Path(file_dest)
     existing_file_check_subdir = pathlib.Path(file_dest_w_subdir)
 
-    log.debug(existing_file_check_subdir)
-    # return file_info
-
-
     if existing_file_check_subdir.exists():
-        log.warning('This file already exists at the destination with the subdirectory. Not re-saving.')
         file_info['already_exists'] = True
         file_info['already_exists_subdir'] = True
         file_info['copy_timer'] = 0
         file_info['saved'] = True
     else:
-        # log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-        log.warning('This file does not already exist at the destination subdirectory.')
         file_info['already_exists'] = False
         file_info['already_exists_subdir'] = False
         try:
-            log.info('Saving file to destination...')
             f_dest = open(file_dest_w_subdir, 'wb')
             timer_start = time.process_time()
             shutil.copyfileobj(f_src, f_dest, buffer_size)
             timer_end = time.process_time()
-            elapsed_time = timer_end - timer_start
+            file_info['copy_timer'] = timer_end - timer_start
-            log.debug(f'Elapsed time: {elapsed_time}')
-            file_info['copy_timer'] = elapsed_time
             file_info['saved'] = True
-
-            log.info(f'File saved to: {hosted_files_path}')
         except Exception as e:
-            log.exception('*** An exception happened. ***')
+            log.exception(f'Error saving to hosted storage: {e}')
-            log.exception(repr(e))
-            log.exception('***')
-            log.exception(str(e))
-            log.exception('^^^ exception ^^^')
-
             file_info['copy_timer'] = 0
             file_info['saved'] = False
             return False
-    log.info(f'Disk usage: {shutil.disk_usage(hosted_files_path)}')
-    log.info(f"Filename: {file_info['filename']}")
-    log.info(f"Subdirectory Path: {file_info['subdirectory_path']}")
-    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(file_info)
-
-    log.debug(shutil.disk_usage(hosted_files_path))
-
     return file_info
 # ### END ### API Hosted File Methods ### save_file_to_hosted_file() ###
 
@@ -547,235 +431,116 @@ def create_hosted_file_link(
     hosted_file_link_data: dict = {}
     hosted_file_link_data['account_id'] = account_id
     hosted_file_link_data['hosted_file_id'] = hosted_file_id
-    hosted_file_link_data['link_to_type'] = link_to_type # Should this be renamed to "link_to_type" for clarity?
+    hosted_file_link_data['link_to_type'] = link_to_type
-    hosted_file_link_data['link_to_id'] = link_to_id # Should this be renamed to "link_to_id" for clarity?
+    hosted_file_link_data['link_to_id'] = link_to_id
 
-    # hosted_file_link_data['test'] = 'test'
-
-    # NOTE: Currently sql_insert does not handle all successful inserts correctly. If there is not an autonum ID then it will return 0 as the ID.
     if hosted_file_link_data_in_result := sql_insert(data=hosted_file_link_data, table_name='hosted_file_link', id_random_length=0):
         log.info('The hosted_file_link was created.')
-        pass # This should be improved
     elif hosted_file_link_data_in_result is None:
         log.info('The hosted_file_link probably already exists.')
         return None
     else:
-        # This should be improved
-        log.warning('Because the hosted_file_link table does not have a primary autonum this check is incorrect even when successful.')
-        log.warning('Something may have gone wrong while trying to create the hosted_file_link record.')
-        log.warning('The hosted_file_link was probably created fine though.')
         return False
-
-    log.debug(hosted_file_link_data_in_result)
     return True
 # ### END ### API Hosted File Methods ### create_hosted_file_link() ###
 
 
 # ### BEGIN ### API Hosted File Methods ### handle_delete_hosted_file() ###
-# Updated 2022-08-09
+# Updated 2026-02-03
 @logger_reset
 def handle_delete_hosted_file(
         account_id: int|str,
         hosted_file_id: int|str,
-
         link_to_type: str = None,
         link_to_id: int|str = None,
-
         rm_all_links: bool = False,
         rm_orphan: bool = False,
         ):
     log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    if account_id := redis_lookup_id_random(record_id_random=account_id, table_name='account'): pass
+    # Resolve account_id if it's a string (Vision ID or 'bypass')
-    else: return False
+    if isinstance(account_id, str):
+        if res_acc := redis_lookup_id_random(record_id_random=account_id, table_name='account'):
+            account_id_int = res_acc
+        else:
+            # If bypass or not found, we still proceed but log it. 
+            # In many maintenance cases, we don't want to block the deletion.
+            log.warning(f"Could not resolve account_id '{account_id}'. Proceeding without account restriction.")
+            account_id_int = None
+    else:
+        account_id_int = account_id
 
-    if hosted_file_id := redis_lookup_id_random(record_id_random=hosted_file_id, table_name='hosted_file'): pass
+    if hosted_file_id_int := redis_lookup_id_random(record_id_random=hosted_file_id, table_name='hosted_file'): pass
     else: return False
 
-    # ### SECTION ### Handle links NOTE NOTE NOTE NOTE NOTE NOTE
-    # NOTE: If link_to_type and link_to_id passed then try and remove that link record first.
-
     if link_to_type and link_to_id:
         if hosted_file_link_result := delete_hosted_file_link(
-                account_id = account_id,
+                account_id = account_id_int,
-                hosted_file_id = hosted_file_id,
+                hosted_file_id = hosted_file_id_int,
-
                 link_to_type = link_to_type,
                 link_to_id = link_to_id,
-
-                # rm_orphan = rm_orphan,
                 ):
             log.info('The hosted file link record was deleted.')
         elif hosted_file_link_result is None:
-            log.warning('The hosted file link record was not found and may have already been deleted. Odd, but this can happen. event_file has a trigger to delete hosted_file_link when being deleted.')
+            log.warning('The hosted file link record was not found.')
-            # return None
         else:
-            log.error('Something went wrong while trying to delete the hosted file link record.')
             return False
 
-    # ### SECTION ### Handle orphan check and deletion of hosted_file record and file on server NOTE NOTE NOTE NOTE NOTE NOTE
+    if not rm_orphan: return True
-    # NOTE: If not rm_orphan then do nothing else.
-    # NOTE: If rm_orphan then get list of links for file.
-    # NOTE:     If 0 links result then delete the hosted_file record and file on the server.
-    # NOTE:     If >0 links result then do nothing else.
 
-    # NOTE: Don't check or remove orphan
+    if hosted_file_obj := load_hosted_file_obj(hosted_file_id = hosted_file_id_int, inc_hosted_file_link_list = True): pass
-    if not rm_orphan:
+    else: return False
-        log.info('Removed hosted file link. No orphan check.')
+
+    if hosted_file_link_rec_list_result := get_hosted_file_link_rec_list(hosted_file_id=hosted_file_id_int):
+        log.info('Still not an orphan file.')
         return True
-
+    
-    if hosted_file_obj := load_hosted_file_obj(
+    # Orphan: Delete physical file
-            hosted_file_id = hosted_file_id,
-            # inc_hosted_file = True,
-            inc_hosted_file_link_list = True, # if rm_orphan (True) then need to include hosted_file_link_list (True)
-            ):
-        log.info('Hosted File object loaded.')
-        pass
-    elif hosted_file_obj is None:
-        log.warning('Hosted File object not found. Can not attempt to delete file from the server if there is one.')
-        # pass
-        return None
-    else:
-        log.error('Something went wrong while trying to load the Hosted File object.')
-        return False
-    log.debug(hosted_file_obj)
-
-    # NOTE: Check and remove orphan
-    if hosted_file_link_rec_list_result := get_hosted_file_link_rec_list(hosted_file_id=hosted_file_id):
-        log.info('This hosted file has linked records to it.')
-        hosted_file_link_result_list = []
-        for hosted_file_link_rec in hosted_file_link_rec_list_result:
-            hosted_file_link_result_list.append(hosted_file_link_rec)
-        # log.debug( )
-        hosted_file_list = hosted_file_link_result_list
-        # NOT safe to delete the hosted_file record and file from server!!!
-        # STOP!
-        log.info('Removed hosted file link (above). Still not an orphan file.')
-        return True
-    elif isinstance(hosted_file_link_rec_list_result, list) or hosted_file_link_rec_list_result is None:
-        log.info('This hosted file has no link records to it.')
-        hosted_file_list = []
-        # Safe to delete the hosted_file record and file from server???
-        # CONTINUE
-    else:
-        hosted_file_list = False
-        # Safe to delete the hosted_file record and file from server???
-        # CONTINUE???
-        log.error('Something went wrong while trying to get a list of the hosted file link records.')
-        return False
-
-    # ### Orphan file: ### Delete file from server
-    hosted_files_path = settings.FILES_PATH['hosted_files_root']
-    # hosted_files_path = '/home/scott/tmp/hosted_files_dev/'
-    log.info(f'Hosted Files Path: {hosted_files_path}')
-
-    dir_path = hosted_file_obj.directory_path
     subdir_path = hosted_file_obj.subdirectory_path
     hash_sha256 = hosted_file_obj.hash_sha256
-    hash_filename = hash_sha256+'.file'
+    file_path = os.path.join(settings.FILES_PATH['hosted_files_root'], subdir_path or '', f'{hash_sha256}.file')
 
-    if subdir_path:
+    if os.path.exists(file_path):
-        full_subdirectory_path = os.path.join(hosted_files_path, subdir_path)
-    else:
-        full_subdirectory_path = hosted_files_path
-    log.debug(full_subdirectory_path)
-    file_path_w_subdir = os.path.join(full_subdirectory_path, hash_filename)
-    log.info(f'Full file path with subdirectory: {file_path_w_subdir}')
-
-    if os.path.exists(file_path_w_subdir):
-        log.info('File exists!')
-        log.info('Going remove the file if it is an orphan...')
         try:
-            pathlib.Path(file_path_w_subdir).unlink()
+            pathlib.Path(file_path).unlink()
+            log.info(f"Unlinked physical file: {file_path}")
         except OSError as e:
-            log.error("Error: %s : %s" % (file_path, e.strerror))
+            log.error(f"Error unlinking: {e}")
             return False
-        pass
-        # return True
-    else:
-        log.warning(f'The hosted file was not found on the server. Hash: {hash_sha256}')
-        pass
-        # return None
 
-    # ### Orphan file: ### Delete hosted_file record
+    # Delete record
-    sql = f"""
+    sql = "DELETE FROM hosted_file WHERE id = :hosted_file_id"
-        DELETE FROM hosted_file
+    if sql_delete(sql=sql, data={'hosted_file_id': hosted_file_id_int}):
-        WHERE hosted_file.id = :hosted_file_id
+        log.info(f"Deleted record for hosted_file {hosted_file_id_int}")
-        """
-    log.debug(sql)
-
-    hosted_file_data = {}
-    hosted_file_data['hosted_file_id'] = hosted_file_id
-    log.debug(hosted_file_data)
-
-    if hosted_file_delete_result := sql_delete(sql=sql, data=hosted_file_data):
-        log.info(f'Deleted Hosted File record. Hosted File ID: {hosted_file_id}')
         return True
-    elif hosted_file_delete_result is None:
+    return False
-        log.warning(f'Hosted File record was not found and may have already been removed. Hosted File ID: {hosted_file_id}')
-        return None
-        # pass
-    else:
-        log.error('Something went wrong while trying to delete the hosted file record.')
-        return False
 # ### END ### API Hosted File Methods ### handle_delete_hosted_file() ###
 
 
 
 # ### BEGIN ### API Hosted File Methods ### delete_hosted_file_link() ###
-# Updated 2022-08-09
 @logger_reset
 def delete_hosted_file_link(
         account_id: int|str,
         hosted_file_id: int|str,
-
         link_to_type: str,
         link_to_id: int|str,
-
-        # rm_orphan: bool = False,
         ):
-    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(locals())
-
-    # if account_id := redis_lookup_id_random(record_id_random=account_id, table_name='account'): pass
-    # else: return False
-
     if hosted_file_id := redis_lookup_id_random(record_id_random=hosted_file_id, table_name='hosted_file'): pass
     else: return False
 
     if link_to_id := redis_lookup_id_random(record_id_random=link_to_id, table_name=link_to_type): pass
     else: return False
 
-    sql = f"""
+    sql = "DELETE FROM hosted_file_link WHERE hosted_file_id = :hosted_file_id AND link_to_type = :link_to_type AND link_to_id = :link_to_id"
-        DELETE FROM hosted_file_link
+    if sql_delete(sql=sql, data={'hosted_file_id': hosted_file_id, 'link_to_type': link_to_type, 'link_to_id': link_to_id}):
-        WHERE hosted_file_id = :hosted_file_id
+        return True
-            AND link_to_type = :link_to_type
+    return False
-            AND link_to_id = :link_to_id
-        """
-    log.debug(sql)
-
-    hosted_file_link_data = {}
-    hosted_file_link_data['hosted_file_id'] = hosted_file_id
-    hosted_file_link_data['link_to_type'] = link_to_type
-    hosted_file_link_data['link_to_id'] = link_to_id
-    log.debug(hosted_file_link_data)
-
-    if hosted_file_delete_result := sql_delete(sql=sql, data=hosted_file_link_data):
-        log.info(f'Deleted Hosted File Link. Hosted File ID: {hosted_file_id}, Link To Type: {link_to_type}, Link To ID: {link_to_id}')
-    elif hosted_file_delete_result is None:
-        return None
-    else:
-        return False
-
-    return True
 # ### END ### API Hosted File Methods ### delete_hosted_file_link() ###
 
 
 # ### BEGIN ### API Hosted File Methods ### get_hosted_file_rec_list() ###
-# This needs to be improved. Currently it does not really do anything.
-# Need to allow for list by account? Probably have the same actual hosted file have two hosted_file entries if it was uploaded for two separate accounts.
-# Updated 2022-09-22
 @logger_reset
 def get_hosted_file_rec_list(
         for_obj_type: str,
@@ -783,92 +548,34 @@ def get_hosted_file_rec_list(
         limit: int = 1000,
         enabled: str = 'enabled', # enabled, disabled, all
         ) -> list|bool:
-    log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(locals())
-
     if for_obj_id := redis_lookup_id_random(record_id_random=for_obj_id, table_name=for_obj_type): pass
     else: return False
-    data = {}
+    
-    data[f'{for_obj_type}_id'] = for_obj_id
+    data = {f'{for_obj_type}_id': for_obj_id, 'limit': limit}
-    # data['for_obj_type'] = for_obj_type
+    sql_enabled = "AND enable = :enable" if enabled == 'enabled' else ("AND enable = :enable" if enabled == 'disabled' else "")
-    sql_obj_type_id = f'`tbl`.{for_obj_type}_id = :{for_obj_type}_id'
+    if enabled != 'all': data['enable'] = (enabled == 'enabled')
-
-    if enabled in ['enabled', 'disabled', 'all']:
-        if enabled == 'enabled':
-            data['enable'] = True
-            sql_enabled = f'AND `tbl`.enable = :enable'
-        elif enabled == 'disabled':
-            data['enable'] = False
-            sql_enabled = f'AND `tbl`.enable = :enable'
-        elif enabled == 'all':
-            sql_enabled = ''
-
-    if limit:
-        data['limit'] = limit
-        sql_limit = f'LIMIT :limit'
-    else:
-        sql_limit = ''
 
     sql = f"""
-        SELECT `hosted_file`.id AS 'hosted_file_id', `hosted_file`.id_random AS 'hosted_file_id_random'
+        SELECT id AS 'hosted_file_id', id_random AS 'hosted_file_id_random'
-        FROM `hosted_file` AS `hosted_file`
+        FROM hosted_file
-        WHERE
+        WHERE {for_obj_type}_id = :{for_obj_type}_id {sql_enabled}
-            {sql_obj_type_id}
+        ORDER BY created_on DESC, updated_on DESC, filename ASC
-            {sql_enabled}
+        LIMIT :limit;
-        ORDER BY `hosted_file`.created_on DESC, `hosted_file`.updated_on DESC, `hosted_file`.filename ASC, `hosted_file`.extension ASC
-        {sql_limit};
         """
-
+    if res := sql_select(data=data, sql=sql, as_list=True): return res
-    # NOTE: Use the ORDER BY below if priority and sort fields are added to the hosted_file table.
+    return []
-    # /* ORDER BY `hosted_file`.priority DESC, -`hosted_file`.sort DESC, `hosted_file`.created_on DESC, `hosted_file`.updated_on DESC, `hosted_file`.filename ASC, `hosted_file`.extension ASC */
-
-    if hosted_file_rec_li_result := sql_select(data=data, sql=sql, as_list=True):
-        hosted_file_rec_li = hosted_file_rec_li_result
-    else:
-        hosted_file_rec_li = []
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(hosted_file_rec_li_result)
-
-    return hosted_file_rec_li
 # ### END ### API Hosted File Methods ### get_hosted_file_rec_list() ###
 
 
 # ### BEGIN ### API Hosted File Methods ### get_hosted_file_link_rec_list() ###
-# Updated 2022-08-09
 @logger_reset
 def get_hosted_file_link_rec_list(
         hosted_file_id: int|str,
-
-        link_to_type: str = None,
-        link_to_id: int|str = None,
-
         limit: int = 10,
         offset: int = 0,
-        enabled: str = 'enabled', # enabled, disabled, all
         ) -> list|bool:
-    log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    data = {'hosted_file_id': hosted_file_id, 'limit': limit, 'offset': offset}
-    log.debug(locals())
+    sql = "SELECT * FROM hosted_file_link WHERE hosted_file_id = :hosted_file_id ORDER BY created_on DESC LIMIT :limit OFFSET :offset"
-
+    if res := sql_select(data=data, sql=sql, as_list=True): return res
-    data = {'hosted_file_id': hosted_file_id}
+    return []
-
+# ### END ### API Hosted File Methods ### get_hosted_file_link_rec_list() ###
-    # sql_enabled, data['enable'] = sql_enable_part(table_name='hosted_file', enabled=enabled) # Reasonably safe return str and bool
-    sql_limit = sql_limit_offset_part(limit=limit, offset=offset) # Reasonably safe return str
-
-    sql = f"""
-        SELECT *
-        FROM `hosted_file_link` AS `hosted_file_link`
-        WHERE
-            `hosted_file_link`.hosted_file_id = :hosted_file_id
-        ORDER BY `hosted_file_link`.created_on DESC, `hosted_file_link`.updated_on DESC
-            {sql_limit};
-        """
-
-    if hosted_file_link_rec_li_result := sql_select(data=data, sql=sql, as_list=True):
-        hosted_file_link_rec_li = hosted_file_link_rec_li_result
-    else:
-        hosted_file_link_rec_li = []
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
-    log.debug(hosted_file_link_rec_li_result)
-
-    return hosted_file_link_rec_li
-# ### END ### API Hosted File Methods ### get_hosted_file_link_rec_list() ###

app/methods/idaa_novi_verify_methods.py

@@ -0,0 +1,154 @@
+import datetime
+import json
+import requests
+from typing import Dict, Optional
+
+from app.lib_general import log, logger_reset
+
+IDAA_SITE_ID_RANDOM = '58_gJESdlUh'
+_CACHE_TTL = datetime.timedelta(hours=4)
+
+
+# ── Config ────────────────────────────────────────────────────────────────
+
+@logger_reset
+def _load_idaa_cfg() -> Optional[Dict]:
+    """Load IDAA site cfg_json. Returns parsed dict or None on failure."""
+    from app.methods.site_methods import load_site_obj
+    site = load_site_obj(site_id=IDAA_SITE_ID_RANDOM, model_as_dict=True)
+    if not site:
+        log.error("Could not load IDAA site record (id_random='%s').", IDAA_SITE_ID_RANDOM)
+        return None
+    cfg = site.get('cfg_json')
+    if isinstance(cfg, str):
+        try:
+            cfg = json.loads(cfg)
+        except Exception as e:
+            log.error("Failed to parse IDAA cfg_json: %s", e)
+            return None
+    if not isinstance(cfg, dict):
+        log.error("IDAA cfg_json is not a dict after parsing.")
+        return None
+    return cfg
+
+
+def _cache_key(uuid: str) -> str:
+    return f'idaa:novi_member:{uuid}'
+
+
+# ── Public API ────────────────────────────────────────────────────────────
+
+@logger_reset
+def verify_novi_member(uuid: str) -> Dict:
+    """
+    Proxy GET /customers/{uuid} to Novi AMS and return normalized member data.
+
+    Returns a dict with one of:
+      {'status': 200, 'verified': True,  'full_name': '...', 'email': '...'}
+      {'status': 404, 'reason': '...'}
+      {'status': 429, 'reason': '...'}
+      {'status': 503, 'reason': '...'}
+
+    Redis cache key: idaa:novi_member:{uuid}, TTL 4 hours.
+    Only 200 (verified) results are cached — 404 is never cached.
+    """
+    from app.lib_redis_helpers import redis_client
+
+    cache_key = _cache_key(uuid)
+
+    # ── Cache hit ─────────────────────────────────────────────────────────
+    cached_raw = redis_client.get(cache_key)
+    if cached_raw:
+        try:
+            cached = json.loads(cached_raw)
+            log.info("Novi verify cache hit: %s", uuid)
+            return cached
+        except Exception:
+            pass  # corrupt cache entry — fall through to Novi
+
+    # ── Load credentials ──────────────────────────────────────────────────
+    cfg = _load_idaa_cfg()
+    if not cfg:
+        return {'status': 503, 'reason': 'IDAA site configuration unavailable.'}
+
+    base_url = cfg.get('novi_api_root_url', '').rstrip('/')
+    api_key  = cfg.get('novi_idaa_api_key', '')
+
+    if not base_url or not api_key:
+        log.error("novi_api_root_url or novi_idaa_api_key missing from IDAA cfg_json.")
+        return {'status': 503, 'reason': 'Novi credentials not configured.'}
+
+    headers = {'Authorization': f'Basic {api_key}', 'Accept': 'application/json'}
+
+    # ── Call Novi ─────────────────────────────────────────────────────────
+    try:
+        resp = requests.get(f'{base_url}/customers/{uuid}', headers=headers, timeout=10)
+    except requests.exceptions.ConnectionError as e:
+        log.error("Novi unreachable: %s", e)
+        return {'status': 503, 'reason': 'Novi API unreachable.'}
+    except requests.exceptions.Timeout:
+        log.error("Novi request timed out for UUID %s", uuid)
+        return {'status': 503, 'reason': 'Novi API timed out.'}
+    except Exception as e:
+        log.exception("Unexpected error calling Novi for UUID %s: %s", uuid, e)
+        return {'status': 503, 'reason': 'Unexpected error contacting Novi.'}
+
+    if resp.status_code == 429:
+        log.warning("Novi rate limit hit for UUID %s", uuid)
+        return {'status': 429, 'reason': 'Novi rate limit exceeded. Try again shortly.'}
+
+    if resp.status_code >= 500:
+        log.error("Novi server error %s for UUID %s", resp.status_code, uuid)
+        return {'status': 503, 'reason': f'Novi server error ({resp.status_code}).'}
+
+    if resp.status_code == 404:
+        log.info("Novi returned 404 for UUID %s", uuid)
+        return {'status': 404, 'reason': 'Member not found in Novi.'}
+
+    if resp.status_code != 200:
+        log.error("Unexpected Novi status %s for UUID %s: %s", resp.status_code, uuid, resp.text[:200])
+        return {'status': 503, 'reason': f'Unexpected Novi response ({resp.status_code}).'}
+
+    # ── Parse response ────────────────────────────────────────────────────
+    try:
+        data = resp.json()
+    except Exception:
+        log.error("Novi returned non-JSON for UUID %s", uuid)
+        return {'status': 503, 'reason': 'Novi returned an unparseable response.'}
+
+    if not isinstance(data, dict):
+        log.warning("Novi returned non-dict body for UUID %s", uuid)
+        return {'status': 404, 'reason': 'Member not found in Novi (empty response).'}
+
+    # Empty-member anti-pattern: Novi 200 with no identity data
+    email_raw = (data.get('Email') or '').strip()
+    if not email_raw:
+        log.info("Novi 200 with no Email for UUID %s — empty-member anti-pattern", uuid)
+        return {'status': 404, 'reason': 'Member not found in Novi (no identity data).'}
+
+    email = email_raw.replace(' ', '+')
+
+    # Build display name: "FirstName LastName[0]." — fall back to Name field
+    first = (data.get('FirstName') or '').strip()
+    last  = (data.get('LastName')  or '').strip()
+    if first and last:
+        full_name = f'{first} {last[0]}.'
+    elif first:
+        full_name = first
+    else:
+        full_name = (data.get('Name') or '').strip() or 'Member'
+
+    result = {
+        'status': 200,
+        'verified': True,
+        'full_name': full_name,
+        'email': email,
+    }
+
+    # ── Cache verified result ─────────────────────────────────────────────
+    try:
+        redis_client.setex(cache_key, _CACHE_TTL, json.dumps(result))
+    except Exception as e:
+        log.warning("Failed to cache Novi verify result for %s: %s", uuid, e)
+
+    return result

app/methods/lib_media.py

@@ -0,0 +1,169 @@
+import os
+import pathlib
+import shutil
+import time
+import tempfile
+import subprocess
+import shlex
+import logging
+import mimetypes
+
+from app.config import settings
+from app.lib_general import log, logging
+from app.db_sql import sql_select, sql_update, sql_insert, get_id_random
+from app.methods.hosted_file_methods import (
+    load_hosted_file_obj, create_hosted_file_obj, save_file_to_hosted_file
+)
+from app.models.hosted_file_models import Hosted_File_Base
+
+# ### BEGIN ### API Hosted File Methods ### clip_video_method() ###
+async def clip_video_method(
+        hosted_file_id: str,
+        start_time: str,
+        end_time: str,
+        account_id: int,
+        link_to_type: str,
+        link_to_id: int,
+        account_id_random: str = None,
+        filename_no_ext: str = 'automated_hosted_file_clip_video',
+        to_type: str = 'mp4',
+        reencode: bool = False,
+        scale_down: bool = False,
+        ):
+    """
+    Business logic for clipping a video using ffmpeg and saving as a new hosted_file.
+    Returns the new hosted_file dict or False.
+    """
+    # NOTE: This function is invoked by the hosted_file router at
+    # `/hosted_file/{hosted_file_id}/clip_video` and returns the created
+    # hosted_file metadata (or False) so the router can build the standard
+    # response body consumed by frontends.
+    hosted_file_obj = load_hosted_file_obj(hosted_file_id=hosted_file_id)
+    if not hosted_file_obj: return False
+
+    file_hash = hosted_file_obj.hash_sha256
+    hosted_files_path = settings.FILES_PATH['hosted_files_root']
+    full_file_path = os.path.join(hosted_files_path, file_hash[0:2], f'{file_hash}.file')
+
+    if not os.path.exists(full_file_path): return False
+
+    with tempfile.NamedTemporaryFile(delete=False, suffix='.mp4') as tmp_video_file_clip:
+        tmp_video_file_clip_path = tmp_video_file_clip.name
+
+    try:
+        if scale_down:
+            new_filename = f'{filename_no_ext}_[clip_scaled].{to_type}'
+            cmd = f'ffmpeg -hide_banner -loglevel error -nostats -y -i {full_file_path} -ss {start_time} -to {end_time} -vf "scale=w=1920:h=1080:force_original_aspect_ratio=decrease" -c:v libx264 -crf 23 -maxrate 2M -bufsize 2M -c:a copy -movflags +faststart {tmp_video_file_clip_path}'
+        elif reencode:
+            new_filename = f'{filename_no_ext}_[clip_reencode].{to_type}'
+            cmd = f"ffmpeg -hide_banner -loglevel error -nostats -y -i {full_file_path} -ss {start_time} -to {end_time} -c:v libx264 -crf 23 -maxrate 2M -bufsize 2M -c:a copy -movflags +faststart {tmp_video_file_clip_path}"
+        else:
+            new_filename = f'{filename_no_ext}_[clip].{to_type}'
+            cmd = f"ffmpeg -hide_banner -loglevel error -nostats -y -i {full_file_path} -ss {start_time} -to {end_time} -c:v copy -c:a copy -movflags +faststart {tmp_video_file_clip_path}"
+
+        args = shlex.split(cmd)
+        try:
+            subprocess.run(args, check=True, capture_output=True, text=True)
+        except subprocess.CalledProcessError as e:
+            log.exception(f'ffmpeg failed: returncode={e.returncode}; stdout={e.stdout}; stderr={e.stderr}')
+            return False
+
+        file_info = await save_file_to_hosted_file(
+            file_path = tmp_video_file_clip_path,
+            filename = new_filename,
+            extension = to_type,
+            account_id = account_id,
+            account_id_random = account_id_random,
+            link_to_type = link_to_type,
+            link_to_id = link_to_id,
+        )
+
+        if file_info.get('saved'):
+            if sel := sql_select(table_name='hosted_file', field_name='hash_sha256', field_value=file_info['hash_sha256']):
+                return load_hosted_file_obj(hosted_file_id=sel.get('id'), model_as_dict=True)
+            else:
+                new_obj = Hosted_File_Base(**file_info)
+                if res_id := create_hosted_file_obj(hosted_file_obj_new=new_obj):
+                    return load_hosted_file_obj(hosted_file_id=res_id, model_as_dict=True)
+    finally:
+        try:
+            if os.path.exists(tmp_video_file_clip_path):
+                os.unlink(tmp_video_file_clip_path)
+        except Exception:
+            log.exception('Failed to remove temporary video clip file')
+    return False
+# ### END ### API Hosted File Methods ### clip_video_method() ###
+
+
+# ### BEGIN ### API Hosted File Methods ### convert_file_method() ###
+async def convert_file_method(
+        hosted_file_id: str,
+        link_to_type: str,
+        link_to_id: int,
+        account_id: int,
+        account_id_random: str = None,
+        filename_no_ext: str = 'automated_hosted_file_conversion',
+        to_type: str = 'webp',
+        ):
+    from pdf2image import convert_from_path
+
+    # NOTE: Invoked by the hosted_file router at
+    # `/hosted_file/{hosted_file_id}/convert_file`. This helper currently
+    # converts the first page of a PDF to an image (webp/png) and saves a
+    # new hosted_file record; it returns that record or False on failure.
+
+    hosted_file_obj = load_hosted_file_obj(hosted_file_id=hosted_file_id)
+    if not hosted_file_obj: return False
+
+    # Ensure input is a PDF (pdf2image is designed for PDFs)
+    if (getattr(hosted_file_obj, 'extension', None) or '').lower() != 'pdf' and (getattr(hosted_file_obj, 'content_type', None) or '') != 'application/pdf':
+        log.warning('convert_file_method called on non-PDF file')
+        return False
+
+    full_file_path = os.path.join(settings.FILES_PATH['hosted_files_root'], hosted_file_obj.hash_sha256[0:2], f'{hosted_file_obj.hash_sha256}.file')
+    if not os.path.exists(full_file_path): return False
+    if not os.path.exists(full_file_path): return False
+
+    save_path = os.path.join(settings.FILES_PATH['hosted_tmp_root'], 'convert_file', f'conv_{int(time.time())}.{to_type}')
+    os.makedirs(os.path.dirname(save_path), exist_ok=True)
+
+    try:
+        images = convert_from_path(full_file_path, size=(3840, None))
+        image = images[0]
+
+        if to_type == 'webp':
+            image.save(save_path, lossless=False, quality=90)
+        elif to_type == 'png':
+            image.save(save_path, compress_level=9)
+        else:
+            log.warning(f'Unsupported target type for convert_file_method: {to_type}')
+            return False
+    except Exception:
+        log.exception('Error converting file to image')
+        return False
+
+    file_info = await save_file_to_hosted_file(
+        file_path = save_path,
+        filename = f'{filename_no_ext}.{to_type}',
+        extension = to_type,
+        account_id = account_id,
+        account_id_random = account_id_random,
+        link_to_type = link_to_type,
+        link_to_id = link_to_id,
+    )
+
+    if file_info.get('saved'):
+        if sel := sql_select(table_name='hosted_file', field_name='hash_sha256', field_value=file_info['hash_sha256']):
+            return load_hosted_file_obj(hosted_file_id=sel.get('id'), model_as_dict=True)
+        else:
+            new_obj = Hosted_File_Base(**file_info)
+            if res_id := create_hosted_file_obj(hosted_file_obj_new=new_obj):
+                # cleanup tmp file
+                try:
+                    if os.path.exists(save_path):
+                        os.unlink(save_path)
+                except Exception:
+                    log.exception('Failed to remove temporary converted file')
+                return load_hosted_file_obj(hosted_file_id=res_id, model_as_dict=True)
+    return False
+# ### END ### API Hosted File Methods ### convert_file_method() ###

app/methods/lookup_methods.py

@@ -0,0 +1,96 @@
+import logging
+from typing import List, Optional
+from sqlalchemy import text
+from app.lib_sql_core import engine
+from app.lib_general_v3 import AccountContext
+
+log = logging.getLogger(__name__)
+
+def get_lookup_list_v3(
+    lu_type: str, 
+    account_ctx: AccountContext,
+    for_type: Optional[str] = None,
+    for_id: Optional[int] = None,
+    include_disabled: bool = False,
+    whitelist: Optional[List[str]] = None,
+    only_priority: bool = False
+) -> List[dict]:
+    """
+    Retrieves a ranked, deduplicated list of lookup records.
+    Priority: Object Override > Account Override > Global Default.
+    Supports an optional whitelist and priority filtering.
+    """
+    table_name = f"v_lu_v3_{lu_type}"
+    
+    # We use ROW_NUMBER() to handle the hierarchy
+    
+    sql = f"""
+        SELECT * FROM (
+            SELECT *, 
+                   ROW_NUMBER() OVER (
+                       PARTITION BY `group`
+                       ORDER BY
+                          (for_type = :for_type AND for_id = :for_id) DESC,
+                          (account_id = :account_id) DESC, 
+                          created_on DESC
+                   ) as rank_priority
+            FROM `{table_name}`
+            WHERE ((for_type = :for_type AND for_id = :for_id)
+               OR account_id = :account_id 
+               OR account_id IS NULL)
+    """
+    
+    if whitelist:
+        sql += " AND `group` IN :whitelist"
+
+    sql += f"""
+        ) AS ranked
+        WHERE rank_priority = 1
+    """
+    
+    if not include_disabled:
+        sql += " AND enable = 1"
+        
+    if only_priority:
+        sql += " AND priority = 1"
+        
+    sql += " ORDER BY COALESCE(priority, 0) DESC, COALESCE(sort, 0) DESC, name ASC"
+    
+    params = {
+        "account_id": account_ctx.account_id,
+        "for_type": for_type,
+        "for_id": for_id,
+        "whitelist": tuple(whitelist) if whitelist else None
+    }
+    
+    try:
+        with engine.connect() as conn:
+            result = conn.execute(text(sql), params)
+            return [dict(row._mapping) for row in result]
+    except Exception as e:
+        log.error(f"Error in get_lookup_list_v3: {e}")
+        return []
+
+def resolve_lookup_v3(
+    lu_type: str,
+    query: str,
+    account_ctx: AccountContext,
+    identity_fields: List[str]
+) -> Optional[dict]:
+    """
+    Resolves a query string to a single lookup record by scanning multiple identity fields.
+    Returns the highest-priority match.
+    """
+    # Simple implementation: get the full ranked list and find first match in identity fields
+    # For performance with large tables (like timezones), we might want a specific SQL query
+    full_list = get_lookup_list_v3(lu_type, account_ctx)
+    
+    query_clean = query.strip().lower()
+    
+    for item in full_list:
+        for field in identity_fields:
+            val = item.get(field)
+            if val and str(val).lower() == query_clean:
+                return item
+                
+    return None

app/methods/organization_methods.py

@@ -34,7 +34,7 @@ def load_organization_obj(
     if organization_rec := sql_select(table_name='v_organization', record_id=organization_id): pass
     else: return False
 
-    #log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    #log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(organization_rec)
 
     try:
@@ -114,7 +114,7 @@ def get_organization_rec_list(
         organization_rec_li = organization_rec_li_result
     else:
         organization_rec_li = []
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(organization_rec_li_result)
 
     return organization_rec_li
@@ -154,11 +154,11 @@ def update_organization_obj(
                 account_id = contact_obj_in.account_id,
                 contact_dict_obj=contact_obj_in,
                 ):
-            # log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+            # log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
             log.debug(contact_obj_in_result)
             organization_obj_up.contact_id = contact_obj_in_result
         else:
-            # log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+            # log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
             log.debug(contact_obj_in_result)
             return False
 
@@ -180,7 +180,7 @@ def create_update_organization_obj(
         organization_obj: Organization_Base,
         process_contact: bool = False,
         ) -> bool:
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
     if organization_id:

app/methods/person_methods.py

@@ -1837,8 +1837,8 @@ def handle_email_person_auth_key_url(
     from_email = account_cfg.default_no_reply_email
     from_name = account_cfg.default_no_reply_name
 
-    to_name = person_obj.display_name
+    to_name = person_obj.full_name
-    to_email = person_obj.email
+    to_email = person_obj.email or person_obj.user_email or person_obj.primary_email
 
     bcc_email = account_cfg.confirm_email
     bcc_name = account_cfg.confirm_name
@@ -1856,21 +1856,39 @@ def handle_email_person_auth_key_url(
     # subject = f'{account_short_name}: One Time Use Create Account Link ({new_auth_key})'
 
     body_html = f"""
-        <p>{to_name},</p>
+    <div style="font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif; max-width: 600px; margin: 20px auto; padding: 30px; border: 1px solid #ddd; border-radius: 10px; background-color: #ffffff; color: #333;">
-
+        <div style="text-align: center; margin-bottom: 20px;">
-        <p>If you did not request this account creation link, please delete this email. It is suggested that you delete this email after the account creation link has been used or if a new link has been requested.</p>
+            <h2 style="color: #444; margin: 0;">{account_short_name}</h2>
-
+            <div style="height: 2px; background: linear-gradient(to right, #eee, #ccc, #eee); margin: 10px auto; width: 80%;"></div>
-        <p>The link below can only be used once. If you would like try again using this method, you must <a href="NOT READY YET">request a new account creation link</a>. If you request multiple links, only the newest link will work.</p>
+        </div>
-
+        
-        <p><strong><a href="{person_auth_key_url}" style="appearance: button; display: inline-block; text-align: center; text-decoration: none; padding: .2rem .4rem; border: solid thin gray; border-radius: .2rem; background-color: lightyellow; color: black; font-size: larger;">Click to Finish Account Creation With One Time Use Link</a></strong></p>
+        <p style="font-size: 16px;">Hello <strong>{to_name}</strong>,</p>
-
+        
-        <p>Or copy and paste the link:<br>
+        <p style="line-height: 1.6;">You have requested a one-time use link to complete your account registration. This link will allow you to set up your account securely.</p>
-        <strong style="background-color: lightyellow; color: black; font-size: larger;"><a href="{person_auth_key_url}">{person_auth_key_url}</a></strong></p>
+        
-
+        <div style="text-align: center; margin: 35px 0;">
-        <p>If you have questions about this email or trouble with this one time use link, you can email <a href="mailto:{help_tech_email}">{help_tech_name} ({help_tech_email})</a>.</p>
+            <a href="{person_auth_key_url}" style="display: inline-block; padding: 14px 28px; background-color: #fdfd96; color: #000; text-decoration: none; border: 1px solid #ccc; border-radius: 6px; font-weight: bold; font-size: 18px; box-shadow: 0 2px 4px rgba(0,0,0,0.1);">
-
+                Finish Account Creation
-        <p>Thank you!</p>
+            </a>
-        """
+        </div>
+        
+        <p style="font-size: 14px; color: #666; line-height: 1.6;">
+            <strong>Security Note:</strong> If you did not request this link, please delete this email. The link above can only be used once. If you request multiple links, only the newest one will be active.
+        </p>
+        
+        <div style="margin-top: 40px; padding-top: 20px; border-top: 1px solid #eee; font-size: 12px; color: #888;">
+            <p style="margin-bottom: 5px;">If the button above doesn't work, copy and paste the following URL into your browser:</p>
+            <p style="word-break: break-all; color: #0056b3; background-color: #f9f9f9; padding: 10px; border-radius: 4px; border: 1px dashed #ccc;">
+                {person_auth_key_url}
+            </p>
+            
+            <p style="margin-top: 20px;">
+                Questions or trouble? Contact <a href="mailto:{help_tech_email}" style="color: #0056b3; text-decoration: underline;">{help_tech_name}</a>.
+            </p>
+            <p style="text-align: center; margin-top: 30px; font-weight: bold; color: #aaa;">Thank you!</p>
+        </div>
+    </div>
+    """
 
     if send_email(from_email=from_email, from_name=from_name, to_email=to_email, to_name=to_name, bcc_email=bcc_email, bcc_name=bcc_name, subject=subject, body_text=None, body_html=body_html):
         log.info(f'An email with a one time use sign in link was sent to {to_email}.')

app/methods/site_domain_methods.py

@@ -147,6 +147,9 @@ def get_site_domain_rec_list(
 # ### BEGIN ### API Site Domain Methods ### lookup_site_domain_fqdn() ###
 def lookup_site_domain_fqdn(
         fqdn: str,
+    # Accept access_key as an argument for validation (str|None)
+    access_key: Optional[str] = None,
+    referrer: Optional[str] = None,
         enabled: str = 'enabled', # enabled, disabled, all
         limit: int = 100,
         offset: int = 0,
@@ -156,15 +159,37 @@ def lookup_site_domain_fqdn(
 
     data = {}
     data['fqdn'] = fqdn
+    # If access_key is provided, add it to the data dict for SQL parameterization
+    data['domain_access_key'] = access_key
+    if referrer:
+        data['required_referrer'] = referrer
 
     sql_enabled, data['enable'] = sql_enable_part(table_name='site_domain', enabled=enabled) # Reasonably safe return str and bool
     sql_limit = sql_limit_offset_part(limit=limit, offset=offset) # Reasonably safe return str
 
+    # Build access key / referrer SQL similar to router.lookup_fqdn behavior
+    if access_key and referrer:
+        sql_access_key_referrer = """
+            AND site_domain.domain_access_key = :domain_access_key
+            AND site_domain.required_referrer = :required_referrer
+            """
+    elif access_key:
+        sql_access_key_referrer = """
+            AND site_domain.domain_access_key = :domain_access_key
+            AND (site_domain.required_referrer IS NULL OR site_domain.required_referrer = '')
+            """
+    else:
+        sql_access_key_referrer = """
+            AND (site_domain.domain_access_key IS NULL OR site_domain.domain_access_key  = '')
+            AND (site_domain.required_referrer IS NULL OR site_domain.required_referrer  = '')
+            """
+
     sql = f"""
         SELECT `site_domain`.id AS 'site_domain_id', `site_domain`.id_random AS 'site_domain_id_random'
         FROM `v_site_domain` AS site_domain
         WHERE
             site_domain.fqdn = :fqdn
+            {sql_access_key_referrer}
             {sql_enabled}
         ORDER BY `site_domain`.fqdn ASC, `site_domain`.access_key ASC, `site_domain`.required_referrer ASC, `site_domain`.created_on DESC, `site_domain`.updated_on DESC
         {sql_limit};
@@ -176,4 +201,11 @@ def lookup_site_domain_fqdn(
         site_domain_rec_li = []
 
     return site_domain_rec_li
+
+# ---
+# To restore access_key validation:
+# 1. Accept access_key as a parameter to this function (and any API endpoint calling it).
+# 2. Add access_key to the SQL WHERE clause (see above) so only matching records are returned.
+# 3. If access_key is required, return empty or error if not matched.
+# 4. Update API docs and tests to reflect the new/required parameter.
 # ### END ### API Site Domain Methods ### get_site_domain_rec_list() ###

app/methods/site_methods.py

@@ -100,7 +100,7 @@ def update_site_obj(
         site_obj_up: Site_Base,
         create_sub_obj: bool = False,
         ) -> bool:
-    log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
     if site_id := redis_lookup_id_random(record_id_random=site_id, table_name='site'): pass

app/methods/user_methods.py

@@ -605,12 +605,13 @@ def get_user_rec_list(
 
 
 # ### BEGIN ### User Methods ### email_user_auth_key_url() ###
-# This emails the actual one time use sign in URL for a user.
+# This generates a new auth_key token and emails the actual one time use sign in URL to the user's email.
-# Updated 2021-12-02
+# Updated 2025-04-08
 def email_user_auth_key_url(
     account_id: int|str,
     user_id: int|str,
     root_url: str,
+    key_param_name: str = 'auth_key',
     ):
     log.setLevel(logging.DEBUG) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
@@ -653,7 +654,7 @@ def email_user_auth_key_url(
     else: return False
     log.debug(account_cfg)
 
-    user_id_random = user_obj.id_random # NOTE: Not user_id_random because of alias
+    user_id_random = user_obj.id or user_obj.user_id # Vision ID: User_Out_Base uses 'id'/'user_id', not 'id_random'
 
     from_email = account_cfg.default_no_reply_email
     from_name = account_cfg.default_no_reply_name
@@ -684,8 +685,13 @@ def email_user_auth_key_url(
     else: enable_to_str = '-- Not Set --'
     auth_key = user_obj.auth_key
 
-    user_login_url = f'{root_url}user/login?username={urllib.parse.quote(username)}&email={urllib.parse.quote(to_email)}'
+    user_login_url = f'{root_url}?username={urllib.parse.quote(username)}&user_email={urllib.parse.quote(to_email)}'
-    user_login_auth_key_url = f'{root_url}?user_id={urllib.parse.quote(user_id_random)}&auth_key={urllib.parse.quote(new_auth_key)}&valid_email={True}'
+    # user_login_url = f'{root_url}user/login?username={urllib.parse.quote(username)}&email={urllib.parse.quote(to_email)}'
+
+    if key_param_name == 'auth_key':
+        user_login_auth_key_url = f'{root_url}?user_id={urllib.parse.quote(user_id_random)}&auth_key={urllib.parse.quote(new_auth_key)}&valid_email={True}'
+    elif key_param_name:
+        user_login_auth_key_url = f'{root_url}?user_id={urllib.parse.quote(user_id_random)}&{key_param_name}={urllib.parse.quote(new_auth_key)}&valid_email={True}'
 
     subject = f'{account_short_name}: One Time Use Sign In Link ({new_auth_key})'
 

app/middleware.py

@@ -0,0 +1,12 @@
+import time
+from fastapi import Request
+
+async def add_process_time_header(request: Request, call_next):
+    """
+    Middleware to add the processing time to the response header.
+    """
+    start_time = time.time()
+    response = await call_next(request)
+    process_time = time.time() - start_time
+    response.headers['X-Process-Time'] = str(process_time)
+    return response

app/models/account_cfg_models.py

@@ -23,6 +23,7 @@ class Account_Cfg_Base(BaseModel):
     id: Optional[int] = Field(
         alias = 'account_cfg_id'
     )
+
     account_id_random: Optional[str]
     account_id: Optional[int]
 

app/models/account_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -22,22 +22,23 @@ class Account_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        **base_fields['account_id_random'],
+    id: Optional[str] = Field(None, **base_fields['account_id_random'])
-        alias = 'account_id_random',
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-        # default_factory = lambda:secrets.token_urlsafe(default_num_bytes),
+
-    )
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    id: Optional[int] = Field(
+    id_random: Optional[str] = Field(None, alias='account_id_random', exclude=True)
-        alias = 'account_id'
-    )
-    # account_id: Optional[int] = Field(
-    # )
 
     code: Optional[str]
     name: Optional[str]
     short_name: Optional[str]
     description: Optional[str]
 
+    hide: Optional[bool]
+    priority: Optional[bool]
+    sort: Optional[int]
+    group: Optional[str]
+
     enable: Optional[bool]
     enable_from: Optional[datetime.datetime] = None
     enable_to: Optional[datetime.datetime] = None
@@ -72,27 +73,29 @@ class Account_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
+    @root_validator(pre=True)
-    def account_id_lookup(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        log.setLevel(logging.WARNING)
+        """
-        log.debug(locals())
+        Vision Transformer:
-
+        Map DB keys to clean API keys and strip internal integers.
-        if values['id_random']:
+        """
-            log.debug(values['id_random'])
+        # 1. Map Random Strings to Clean Names
-            return redis_lookup_id_random(record_id_random=values['id_random'], table_name='account')
+        if rid := values.get('id_random') or values.get('account_id_random'):
-        return None
+            values['id'] = rid
-
+            values['account_id'] = rid
-    # @validator('account_id', always=True)
+            
-    # def account_id_duplicate(cls, v, values, **kwargs):
+        # 2. Final Vision Enforcement: Strip internal integers from public fields
-    #     log.setLevel(logging.DEBUG)
+        for k in ['id', 'account_id']:
-    #     log.debug(locals())
+            val = values.get(k)
-
+            if val is not None:
-    #     if values['id']:
+                # If it's not a valid random string ID
-    #         log.debug(values['id'])
+                if not isinstance(val, str) or len(val) < 11:
-    #         return values['id']
+                    values[k] = None
-    #     return None
+                
+        return values
 
     class Config:
         underscore_attrs_are_private = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Account Models ### Account_Base() ###

app/models/activity_log_models.py

@@ -67,7 +67,15 @@ class Activity_Log_Base(BaseModel):
     other_json: Optional[str] # When getting the dict version for SQL this should be a string.
     meta_json: Optional[str] # When getting the dict version for SQL this should be a string.
 
+    enable: Optional[bool]
+
+    hide: Optional[bool]
+    priority: Optional[bool]
+    sort: Optional[int]
+    group: Optional[str]
+    
     notes: Optional[str]
+
     created_on: Optional[datetime.datetime] = None
     updated_on: Optional[datetime.datetime] = None
 
@@ -94,29 +102,23 @@ class Activity_Log_Base(BaseModel):
 
     @validator('account_id', always=True)
     def account_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
+        if isinstance(v, int) and v > 0: return v
-        log.debug(locals())
+        elif id_random := values.get('account_id_random'):
-
+            return redis_lookup_id_random(record_id_random=id_random, table_name='account')
-        if values['account_id_random']:
-            return redis_lookup_id_random(record_id_random=values['account_id_random'], table_name='account')
         return None
 
     @validator('person_id', always=True)
     def person_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
+        if isinstance(v, int) and v > 0: return v
-        log.debug(locals())
+        elif id_random := values.get('person_id_random'):
-
+            return redis_lookup_id_random(record_id_random=id_random, table_name='person')
-        if values['person_id_random']:
-            return redis_lookup_id_random(record_id_random=values['person_id_random'], table_name='person')
         return None
 
     @validator('user_id', always=True)
     def user_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
+        if isinstance(v, int) and v > 0: return v
-        log.debug(locals())
+        elif id_random := values.get('user_id_random'):
-
+            return redis_lookup_id_random(record_id_random=id_random, table_name='user')
-        if values['user_id_random']:
-            return redis_lookup_id_random(record_id_random=values['user_id_random'], table_name='user')
         return None
 
     class Config:

app/models/address_models.py

@@ -1,9 +1,9 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
-from app.db_sql import redis_lookup_id_random
+from app.db_sql import get_id_random, redis_lookup_id_random
 from app.lib_general import log, logging
 
 from app.models.common_field_schema import base_fields, default_num_bytes
@@ -14,23 +14,21 @@ class Address_Base(BaseModel):
     log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # Standardized Vision IDs (Strings) ---
-        **base_fields['address_id_random'],
+    id: Optional[str] = Field(None, **base_fields['address_id_random'])
-        alias = 'address_id_random',
+    address_id: Optional[str] = Field(None, **base_fields['address_id_random'])
-        # default_factory = lambda:secrets.token_urlsafe(default_num_bytes),
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-    )
+    contact_id: Optional[str] = Field(None, **base_fields['contact_id_random'])
-    id: Optional[int] = Field(
-        alias = 'address_id'
-    )
-    account_id_random: Optional[str]
-    account_id: Optional[int]
 
+    # Standardized Polymorphic Target
     for_type: Optional[str]
-    for_id_random: Optional[str]
+    for_id: Optional[str] = Field(**base_fields['obj_id_random'])
-    for_id: Optional[int]
 
-    contact_id_random: Optional[str]
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    contact_id: Optional[int]
+    id_random: Optional[str] = Field(None, alias='address_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    contact_id_random: Optional[str] = Field(None, exclude=True)
+    for_id_random: Optional[str] = Field(None, exclude=True)
 
     #organization: Optional[Organization_Base] = Organization_Base()
 
@@ -50,7 +48,7 @@ class Address_Base(BaseModel):
     country_name: Optional[str] # From country lookup table
     country: Optional[str] # Avoid using
 
-    lu_time_zone_id: Optional[str]
+    lu_time_zone_id: Optional[str] = Field(None, exclude=True)
     timezone: Optional[str]
 
     latitude: Optional[str]
@@ -60,65 +58,75 @@ class Address_Base(BaseModel):
 
     congressional_district: Optional[str]
 
-    #priority: Optional[int]
+    enable: Optional[bool]
-    #sort: Optional[int]
+    hide: Optional[bool]
-    #group: Optional[str]
+    priority: Optional[bool]
+    sort: Optional[int]
+    group: Optional[str]
 
+    notes: Optional[str]
     created_on: Optional[datetime.datetime] = None
     updated_on: Optional[datetime.datetime] = None
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('address_id_random', always=True)
+    @root_validator(pre=True)
-    def address_id_random_copy(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        log.setLevel(logging.WARNING)
+        """
-        log.debug(locals())
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers.
+        """
+        # 1. Map Primary Object ID
+        if rid := values.get('id_random') or values.get('address_id_random'):
+            values['id'] = rid
+            values['address_id'] = rid
+            
+        # 2. Map & Resolve Relational IDs
+        id_map = [
+            ('account_id', 'account'),
+            ('contact_id', 'contact'),
+        ]
 
-        if values['id_random']:
+        for field, table in id_map:
-            return values['id_random']
+            r_val = values.get(f'{field}_random')
-        return None
+            if r_val and isinstance(r_val, str):
+                values[field] = r_val
+            elif values.get(field) and isinstance(values[field], (int, str)):
+                is_random = isinstance(values[field], str) and len(values[field]) >= 11
+                if not is_random:
+                    resolved_rid = get_id_random(values[field], table)
+                    if resolved_rid:
+                        values[field] = resolved_rid
+                        values[f'{field}_random'] = resolved_rid
 
-    @validator('id', always=True)
+        # 3. Handle Polymorphic for_id
-    def address_id_lookup(cls, v, values, **kwargs):
+        if f_rid := values.get('for_id_random'):
-        log.setLevel(logging.WARNING)
+            values['for_id'] = f_rid
-        log.debug(locals())
+        elif values.get('for_id') and values.get('for_type'):
+            # Resolve based on the for_type
+            is_random = isinstance(values['for_id'], str) and len(values['for_id']) >= 11
+            if not is_random:
+                resolved_for_rid = get_id_random(values['for_id'], values['for_type'])
+                if resolved_for_rid:
+                    values['for_id'] = resolved_for_rid
+                    values['for_id_random'] = resolved_for_rid
 
-        if isinstance(v, int) and v > 0: return v
+        # 4. Final Vision Enforcement
-        elif id_random := values.get('id_random'):
+        for k in ['id', 'address_id', 'account_id', 'contact_id', 'for_id']:
-            return redis_lookup_id_random(record_id_random=id_random, table_name='address')
+            val = values.get(k)
-        return None
+            if val is not None:
+                if not isinstance(val, str) or len(val) < 11:
+                    values[k] = None
+                
+        return values
 
-    @validator('account_id', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def account_id_lookup(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        log.setLevel(logging.WARNING)
+        'country_subdivision_name', 'country_name'
-        log.debug(locals())
+    ]
-
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('account_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='account')
-        return None
-
-    @validator('contact_id', always=True)
-    def contact_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('contact_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='contact')
-        return None
-
-    @validator('for_id', always=True)
-    def for_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['for_id_random'] and values['for_type']:
-            return redis_lookup_id_random(record_id_random=values['for_id_random'], table_name=values['for_type'])
-        return None
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
-# ### END ### API Address Models ### Address_Base() ###
+# ### END ### API Address Models ### Address_Base() ###

app/models/api_crud_models.py

@@ -1,14 +1,37 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Any, Dict, List, Optional, Set, Union
 from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
 
-from app.db_sql import redis_lookup_id_random
+import logging
-from app.lib_general import log, logging
+log = logging.getLogger(__name__)
 
 from app.models.common_field_schema import base_fields, default_num_bytes
 
 
+# ### BEGIN ### API Search Models ###
+class SearchFilter(BaseModel):
+    """
+    Represents a single filter condition.
+    Example: {"field": "price", "op": "gt", "value": 100}
+    """
+    field: str
+    op: str  # eq, ne, gt, gte, lt, lte, like, in, is_null, is_not_null, contains, startswith, endswith
+    value: Optional[Any] = None
+
+class SearchQuery(BaseModel):
+    """
+    Represents a complex search query with optional logical grouping.
+    """
+    query_string: Optional[str] = Field(None, alias="q")
+    and_filters: Optional[List[Union[SearchFilter, 'SearchQuery']]] = Field(None, alias="and")
+    or_filters: Optional[List[Union[SearchFilter, 'SearchQuery']]] = Field(None, alias="or")
+
+# Support recursive models in Pydantic v1
+SearchQuery.update_forward_refs()
+# ### END ### API Search Models ###
+
+
 # ### BEGIN ### API CRUD Models ### Fundraising_Cfg_Base() ###
 class Api_Crud_Base(BaseModel):
     log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
@@ -16,6 +39,8 @@ class Api_Crud_Base(BaseModel):
 
     super_key: Optional[str] = None # Query(None, min_length=8,  max_length=50),
 
+    jwt: Optional[str] = None
+
     create_key: Optional[str] = None # Query(None, min_length=6,  max_length=50),
     read_key: Optional[str] = None # Query(None, min_length=5,  max_length=50),
     update_key: Optional[str] = None # Query(None, min_length=6,  max_length=50),

app/models/archive_content_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -11,22 +11,21 @@ from app.models.common_field_schema import base_fields, default_num_bytes
 
 # ### BEGIN ### API Archive Content Models ### Archive_Content_Base() ###
 class Archive_Content_Base(BaseModel):
-    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
     # def testing(test_var=None):
     #     log.debug(test_var)
     #     return test_var
 
-    id_random: Optional[str] = Field(
+    # --- Vision IDs (primary public identifiers — always random strings) ---
-        # **base_fields['archive_content_id_random'],
+    id: Optional[str] = Field(None, **base_fields['archive_content_id_random'])
-        alias = 'archive_content_id_random',
+    archive_content_id: Optional[str] = Field(None, **base_fields['archive_content_id_random'])
-    )
+    # Legacy alias kept for backward compatibility; populated by root_validator
-    id: Optional[int] = Field(
+    id_random: Optional[str] = Field(None, alias='archive_content_id_random')
-        alias = 'archive_content_id'
+
-    )
+    account_id_random: Optional[str]
-    # account_id_random: Optional[str] # Is this field really needed?
+    account_id: Optional[int]
-    # account_id: Optional[int] # Is this field really needed?
 
     archive_id_random: Optional[str]
     archive_id: Optional[int]
@@ -37,6 +36,9 @@ class Archive_Content_Base(BaseModel):
     lu_media_type_id: Optional[int]
     lu_media_type: Optional[str]
 
+    external_id: Optional[str]
+    code: Optional[str]
+
     name: Optional[str]
     description: Optional[str]
 
@@ -85,14 +87,38 @@ class Archive_Content_Base(BaseModel):
     created_on: Optional[datetime.datetime]
     updated_on: Optional[datetime.datetime]
 
+    # Including convenience data
+    # This is only for convenience. Probably going to keep unless it causes a problem.
+    hosted_file_hash_sha256: Optional[str]
+    hosted_file_subdirectory_path: Optional[str] = Field(None, exclude=True)
+    hosted_file_content_type: Optional[str]
+    hosted_file_size: Optional[str]
+
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
+    fields_to_exclude_from_db: ClassVar[list] = [
+        'id', 'archive_content_id', 'id_random',
+        'account_id', 'account_id_random', 'archive_id_random', 'hosted_file_id_random',
+        'hosted_file_path', 'api_hosted_file_path_download', 'api_hosted_file_path_stream',
+        'hosted_file_hash_sha256', 'hosted_file_content_type', 'hosted_file_size'
+    ]
+
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
+    @root_validator(pre=True)
-    def archive_content_id_lookup(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if isinstance(v, int) and v > 0: return v
+        """
-        elif id_random := values.get('id_random'):
+        Vision Transformer: Map DB random-string keys to clean Vision ID fields.
-            return redis_lookup_id_random(record_id_random=id_random, table_name='archive_content')
+        Collision prevention strips any integer that snuck into the string ID fields.
-        return None
+        """
+        rid = values.get('id_random') or values.get('archive_content_id_random')
+        if rid and isinstance(rid, str):
+            values['id'] = rid
+            values['archive_content_id'] = rid
+        # Collision prevention: reject integer values in Vision string fields
+        for k in ['id', 'archive_content_id']:
+            if k in values and not isinstance(values[k], str):
+                del values[k]
+        return values
 
     @validator('archive_id', always=True)
     def archive_id_lookup(cls, v, values, **kwargs):

app/models/archive_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
 from typing import Dict, List, Optional, Set, Union
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -14,15 +14,39 @@ class Archive_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings for API, Integers for DB) ---
-        **base_fields['archive_id_random'],
+    id: Optional[Union[int, str]] = Field(None, **base_fields['archive_id_random'])
-        alias = 'archive_id_random',
+    archive_id: Optional[Union[int, str]] = Field(None, **base_fields['archive_id_random'])
-    )
+    account_id: Optional[Union[int, str]] = Field(None, **base_fields['account_id_random'])
-    id: Optional[int] = Field(
+
-        alias = 'archive_id'
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    )
+    id_random: Optional[str] = Field(None, alias='archive_id_random', exclude=True)
-    account_id_random: Optional[str]
+    account_id_random: Optional[str] = Field(None, exclude=True)
-    account_id: Optional[int]
+
+    @root_validator(pre=True)
+    def map_v3_ids(cls, values):
+        """
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers during READ operations.
+        During CREATE (POST) operations, we ensure resolved integers are preserved.
+        """
+        # 1. Map Random Strings to Clean Names
+        rid = values.get('id_random') or values.get('archive_id_random')
+        if rid and isinstance(rid, str):
+            values['id'] = rid
+            values['archive_id'] = rid
+            
+        if a_rid := values.get('account_id_random'): values['account_id'] = a_rid
+            
+        # 2. Prevent leakage of integers during API responses (Vision Standard)
+        for k in ['id', 'archive_id', 'account_id']:
+            val = values.get(k)
+            if val is not None and not isinstance(val, str):
+                if values.get(f'{k}_random') or (k=='id' and values.get('id_random')):
+                    del values[k]
+                
+        return values
+
     archive_type_id: Optional[int]
     archive_type: Optional[str]
 
@@ -70,22 +94,8 @@ class Archive_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
-    def archive_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='archive')
-        return None
-
-    @validator('account_id', always=True)
-    def account_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('account_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='account')
-        return None
-
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Archive Models ### Archive_Base() ###

app/models/auth_models.py

@@ -0,0 +1,15 @@
+from typing import Optional
+from pydantic import BaseModel
+
+# Zero-dependency auth models for V3
+# Created 2026-01-07 to resolve circular dependencies in FastAPI startup
+
+class AccountContext(BaseModel):
+    account_id: Optional[int]
+    account_id_random: Optional[str]
+    administrator: bool = False
+    manager: bool = False
+    super: bool = False
+    auth_method: str = 'legacy_header'
+    token_payload: Optional[dict] = None
+    auth_error: Optional[str] = None

app/models/common_field_schema.py

@@ -49,11 +49,13 @@ base_fields['event_person_tracking_id_random'] = xxx_id_random_field_schema
 base_fields['event_presentation_id_random'] = xxx_id_random_field_schema
 base_fields['event_presenter_id_random'] = xxx_id_random_field_schema
 base_fields['event_registration_id_random'] = xxx_id_random_field_schema
+base_fields['event_registration_cfg_id_random'] = xxx_id_random_field_schema
 base_fields['event_session_id_random'] = xxx_id_random_field_schema
 base_fields['event_track_id_random'] = xxx_id_random_field_schema
 base_fields['flask_cfg_id_random'] = xxx_id_random_field_schema
 base_fields['fundraising_id_random'] = xxx_id_random_field_schema
 base_fields['fundraising_cfg_id_random'] = xxx_id_random_field_schema
+base_fields['grant_id_random'] = xxx_id_random_field_schema
 base_fields['hosted_file_id_random'] = xxx_id_random_field_schema
 base_fields['journal_id_random'] = xxx_id_random_field_schema
 base_fields['journal_entry_id_random'] = xxx_id_random_field_schema
@@ -79,6 +81,9 @@ base_fields['post_comment_id_random'] = xxx_id_random_field_schema
 base_fields['product_id_random'] = xxx_id_random_field_schema
 base_fields['site_id_random'] = xxx_id_random_field_schema
 base_fields['site_domain_id_random'] = xxx_id_random_field_schema
+base_fields['status_id_random'] = xxx_id_random_field_schema
+base_fields['sponsorship_cfg_id_random'] = xxx_id_random_field_schema
+base_fields['sponsorship_id_random'] = xxx_id_random_field_schema
 base_fields['user_id_random'] = xxx_id_random_field_schema
 base_fields['user_role_id_random'] = xxx_id_random_field_schema
 

app/models/contact_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import get_id_random, redis_lookup_id_random
 from app.lib_general import log, logging
@@ -16,26 +16,26 @@ class Contact_Base(BaseModel):
     log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        **base_fields['contact_id_random'],
+    id: Optional[str] = Field(None, **base_fields['contact_id_random'])
-        alias = 'contact_id_random',
+    contact_id: Optional[str] = Field(None, **base_fields['contact_id_random'])
-        # default_factory = lambda:secrets.token_urlsafe(default_num_bytes),
+    
-    )
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-    id: Optional[int] = Field(
+    address_id: Optional[str] = Field(None, **base_fields['address_id_random'])
-        alias = 'contact_id'
+    
-    )
+    # NOTE: Linked Address ID is actually the old contact.address_id (Legacy?)
-    account_id_random: Optional[str]
+    linked_address_id: Optional[str] = Field(None, **base_fields['address_id_random'])
-    account_id: Optional[int]
-
-    address_id_random: Optional[str]
-    address_id: Optional[int]
-
-    linked_address_id_random: Optional[str]
-    linked_address_id: Optional[int]
 
+    # Standardized Polymorphic Target
     for_type: Optional[str]
-    for_id: Optional[int]
+    for_id: Optional[str] = Field(**base_fields['obj_id_random'])
-    for_id_random: Optional[Union[str,None]] = None # lambda:get_id_random(values.get('for_id'), table_name=values.get('for_type')),
+
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
+    id_random: Optional[str] = Field(None, alias='contact_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    address_id_random: Optional[str] = Field(None, exclude=True)
+    linked_address_id_random: Optional[str] = Field(None, exclude=True)
+    for_id_random: Optional[str] = Field(None, exclude=True)
 
     name: Optional[str]
     title: Optional[str]
@@ -72,10 +72,13 @@ class Contact_Base(BaseModel):
     other_text: Optional[str]
     other_json: Optional[Json]
 
+    enable: Optional[bool]
+    hide: Optional[bool]
     priority: Optional[bool]
     sort: Optional[int]
     group: Optional[str]
 
+    notes: Optional[str]
     created_on: Optional[datetime.datetime] = None
     updated_on: Optional[datetime.datetime] = None
 
@@ -85,99 +88,64 @@ class Contact_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('contact_id_random', always=True)
+    @root_validator(pre=True)
-    def contact_id_random_copy(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        log.setLevel(logging.WARNING)
+        """
-        log.debug(locals())
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers.
+        """
+        # 1. Map Primary Object ID
+        if rid := values.get('id_random') or values.get('contact_id_random'):
+            values['id'] = rid
+            values['contact_id'] = rid
+            
+        # 2. Map & Resolve Relational IDs
+        id_map = [
+            ('account_id', 'account'),
+            ('address_id', 'address'),
+            ('linked_address_id', 'address'),
+        ]
 
-        if values['id_random']:
+        for field, table in id_map:
-            return values['id_random']
+            r_val = values.get(f'{field}_random')
-        return None
+            if r_val and isinstance(r_val, str):
+                values[field] = r_val
+            elif values.get(field) and isinstance(values[field], (int, str)):
+                is_random = isinstance(values[field], str) and len(values[field]) >= 11
+                if not is_random:
+                    resolved_rid = get_id_random(values[field], table)
+                    if resolved_rid:
+                        values[field] = resolved_rid
+                        values[f'{field}_random'] = resolved_rid
 
-    @validator('id', always=True)
+        # 3. Handle Polymorphic for_id
-    def contact_id_lookup(cls, v, values, **kwargs):
+        if f_rid := values.get('for_id_random'):
-        log.setLevel(logging.WARNING)
+            values['for_id'] = f_rid
-        log.debug(locals())
+        elif values.get('for_id') and values.get('for_type'):
+            # Resolve based on the for_type
+            is_random = isinstance(values['for_id'], str) and len(values['for_id']) >= 11
+            if not is_random:
+                resolved_for_rid = get_id_random(values['for_id'], values['for_type'])
+                if resolved_for_rid:
+                    values['for_id'] = resolved_for_rid
+                    values['for_id_random'] = resolved_for_rid
 
-        if isinstance(v, int) and v > 0: return v
+        # 4. Final Vision Enforcement
-        elif id_random := values.get('id_random'):
+        for k in ['id', 'contact_id', 'account_id', 'address_id', 'linked_address_id', 'for_id']:
-            return redis_lookup_id_random(record_id_random=id_random, table_name='contact')
+            val = values.get(k)
-        return None
+            if val is not None:
+                if not isinstance(val, str) or len(val) < 11:
+                    values[k] = None
+                
+        return values
 
-    @validator('account_id', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def account_id_lookup(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        log.setLevel(logging.WARNING)
+        'linked_address_id', 'timezone_name', 'address'
-        log.debug(locals())
+    ]
-
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('account_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='account')
-        return None
-
-    @validator('address_id', always=True)
-    def address_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('address_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='address')
-        return None
-
-    # NOTE: Linked Address ID is actually the old contact.address_id
-    # This should no longer be used...
-    @validator('linked_address_id', always=True)
-    def linked_address_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('linked_address_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='address')
-        return None
-
-    @validator('for_id', pre=True, always=True)
-    def for_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.DEBUG)
-        log.debug(locals())
-
-        for_type = values.get('for_type')
-        for_id = v # values.get('for_id')
-        for_id_random = values.get('for_id_random')
-
-        if for_id and for_type:
-            log.info(f'Got For ID: {for_id}; For Type: {for_type}')
-            for_id_random = get_id_random(for_id, table_name=for_type)
-            values['for_id_random'] = for_id_random
-            return for_id
-        elif values.get('for_id_random') and values.get('for_type'):
-            log.info(f'Got For ID Random: {for_id_random}; For Type: {for_type}')
-            return redis_lookup_id_random(record_id_random=values['for_id_random'], table_name=values['for_type'])
-        log.info(f'Got nothing? For ID: {for_id}; For ID Random: {for_id_random}; For Type: {for_type}')
-        return None
-
-    @validator('for_id_random', always=True)
-    def for_id_random_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.DEBUG)
-        log.debug(locals())
-
-        for_type = values.get('for_type')
-        for_id = values.get('for_id')
-        for_id_random = v
-
-        if for_id_random:
-            log.info(f'Got For ID Random: {for_id_random}')
-            return for_id_random
-        elif for_id and for_type:
-            log.info(f'Got For ID: {for_id}; For Type: {for_type}')
-            for_id_random = get_id_random(for_id, table_name=for_type)
-            log.info(f'Got ID Random: {for_id_random}')
-            return for_id_random
-        log.info(f'Got nothing? For ID: {for_id}; For ID Random: {for_id_random}; For Type: {for_type}')
-        return None
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Contact Models ### Contact_Base() ###

app/models/core_object_models.py

@@ -68,6 +68,10 @@ class Core_Std_Obj_Base(BaseModel):
             # return redis_lookup_id_random(record_id_random=id_random, table_name=otype)
         return None
 
+    class Config:
+        underscore_attrs_are_private = True
+        allow_population_by_field_name = True
+
 
 class Core_Object_Base(BaseModel):
     log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
@@ -148,6 +152,10 @@ class Core_Object_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
+    class Config:
+        underscore_attrs_are_private = True
+        allow_population_by_field_name = True
+
 
 class Example_Object_Base(Core_Object_Base): # Based on Core_Object_Base
     title: Optional[str] = None

app/models/data_store_models.py

@@ -1,53 +1,60 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
-from app.db_sql import redis_lookup_id_random
+from app.db_sql import get_id_random, redis_lookup_id_random
 from app.lib_general import log, logging
 
-from app.models.common_field_schema import base_fields, default_num_bytes
+from app.models.common_field_schema import base_fields
 
 
 # ### BEGIN ### API Data Store Models ### Data_Store_Base() ###
 class Data_Store_Base(BaseModel):
-    # log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        **base_fields['data_store_id_random'],
+    id: Optional[str] = Field(None, **base_fields['data_store_id_random'])
-        alias = 'data_store_id_random',
+    data_store_id: Optional[str] = Field(None, **base_fields['data_store_id_random'])
-    )
+    
-    id: Optional[int] = Field(
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-        alias = 'data_store_id'
+    person_id: Optional[str] = Field(None, **base_fields['person_id_random'])
-    )
+    user_id: Optional[str] = Field(None, **base_fields['user_id_random'])
-
-    account_id_random: Optional[str]
-    account_id: Optional[int]
 
+    # Standardized Polymorphic Target
     for_type: Optional[str]
-    for_id_random: Optional[str]
+    for_id: Optional[str] = Field(**base_fields['obj_id_random'])
-    for_id: Optional[int]
 
-    person_id_random: Optional[str]
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    person_id: Optional[int]
+    id_random: Optional[str] = Field(None, alias='data_store_id_random', exclude=True)
-
+    account_id_random: Optional[str] = Field(None, exclude=True)
-    user_id_random: Optional[str]
+    person_id_random: Optional[str] = Field(None, exclude=True)
-    user_id: Optional[int]
+    user_id_random: Optional[str] = Field(None, exclude=True)
+    for_id_random: Optional[str] = Field(None, exclude=True)
 
     code: Optional[str]
-
     name: Optional[str]
     description: Optional[str]
 
-    # json: Optional[str] # "json" is reserved; need to change field name? json_str?
+    type: Optional[str] # html, json, md, text
+
+    # The JSON fields are case sensitive
     json_str: Optional[Union[Json, None]] = Field(
         alias = 'json',
     )
+
+    # The text fields are case insensitive
     text: Optional[str]
+
     meta_json: Optional[str]
     meta_text: Optional[str]
 
+    access: Optional[str]
+    access_read: Optional[str]
+    access_write: Optional[str]
+    access_delete: Optional[str]
+
     enable: Optional[bool]
 
     hide: Optional[bool]
@@ -60,58 +67,68 @@ class Data_Store_Base(BaseModel):
     created_on: Optional[datetime.datetime] = None
     updated_on: Optional[datetime.datetime] = None
 
-    # Including convenience data
-    # This is only for convenience. Probably going to keep unless it causes a problem.
-
-    # Including JSON data
-    # other_json: Optional[Json]
-    # meta_json: Optional[Json]
-
-    # Including other related objects
-
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
+    @root_validator(pre=True)
-    def data_store_id_lookup(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if isinstance(v, int) and v > 0: return v
+        """
-        elif id_random := values.get('id_random'):
+        Vision Transformer:
-            return redis_lookup_id_random(record_id_random=id_random, table_name='data_store')
+        Map DB keys to clean API keys and strip internal integers.
-        return None
+        """
+        # 0. Scrub stringified NULLs from database
+        for k, v in list(values.items()):
+            if isinstance(v, str) and v.upper() == 'NULL':
+                values[k] = None
 
-    @validator('account_id', always=True)
+        # 1. Map Primary Object ID
-    def account_id_lookup(cls, v, values, **kwargs):
+        if rid := values.get('id_random') or values.get('data_store_id_random'):
-        if isinstance(v, int) and v > 0: return v
+            values['id'] = rid
-        elif id_random := values.get('account_id_random'):
+            values['data_store_id'] = rid
-            return redis_lookup_id_random(record_id_random=id_random, table_name='account')
+            
-        return None
+        # 2. Map & Resolve Relational IDs
+        id_map = [
+            ('account_id', 'account'),
+            ('person_id', 'person'),
+            ('user_id', 'user'),
+        ]
 
-    @validator('for_id', always=True)
+        for field, table in id_map:
-    def for_id_lookup(cls, v, values, **kwargs):
+            r_val = values.get(f'{field}_random')
-        log.setLevel(logging.WARNING)
+            if r_val and isinstance(r_val, str):
-        log.debug(locals())
+                values[field] = r_val
-        if isinstance(v, int) and v > 0: return v
+            elif values.get(field) and isinstance(values[field], (int, str)):
-        elif values.get('for_id_random') and values.get('for_type'):
+                # If it's a string but doesn't look like a random ID (e.g. integer string), resolve it
-            for_id_random = values.get('for_id_random')
+                is_random = isinstance(values[field], str) and len(values[field]) >= 11
-            for_type = values.get('for_type')
+                if not is_random:
-            return redis_lookup_id_random(record_id_random=for_id_random, table_name=for_type)
+                    resolved_rid = get_id_random(values[field], table)
-        return None
+                    if resolved_rid:
+                        values[field] = resolved_rid
+                        values[f'{field}_random'] = resolved_rid
 
-    @validator('person_id', always=True)
+        # 3. Handle Polymorphic for_id
-    def person_id_lookup(cls, v, values, **kwargs):
+        if f_rid := values.get('for_id_random'):
-        if isinstance(v, int) and v > 0: return v
+            values['for_id'] = f_rid
-        elif id_random := values.get('person_id_random'):
+        elif values.get('for_id') and values.get('for_type'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='person')
+            # Resolve based on the for_type
-        return None
+            is_random = isinstance(values['for_id'], str) and len(values['for_id']) >= 11
+            if not is_random:
+                resolved_for_rid = get_id_random(values['for_id'], values['for_type'])
+                if resolved_for_rid:
+                    values['for_id'] = resolved_for_rid
+                    values['for_id_random'] = resolved_for_rid
 
-    @validator('user_id', always=True)
+        # 4. Final Vision Enforcement: Strip internal integers from public fields
-    def user_id_lookup(cls, v, values, **kwargs):
+        for k in ['id', 'data_store_id', 'account_id', 'person_id', 'user_id', 'for_id']:
-        if isinstance(v, int) and v > 0: return v
+            val = values.get(k)
-        elif id_random := values.get('user_id_random'):
+            # If value is present but not a valid random string ID
-            return redis_lookup_id_random(record_id_random=id_random, table_name='user')
+            if val is not None:
-        return None
+                if not isinstance(val, str) or len(val) < 11:
+                    values[k] = None
+                
+        return values
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Data Store Models ### Data_Store_Base() ###

app/models/error_models.py

@@ -0,0 +1,23 @@
+from typing import Optional, Any, Dict
+from pydantic import BaseModel, Field
+
+class StandardError(BaseModel):
+    """
+    Standardized machine-readable error structure for Aether.
+    Helps the frontend decide how to handle failures.
+    """
+    category: str = Field(..., description="Error category (e.g., 'database', 'validation', 'security')")
+    code: Optional[int] = Field(None, description="Specific error code (e.g., MariaDB error code)")
+    message: str = Field(..., description="Developer-friendly error message")
+    recoverable: bool = Field(False, description="If True, the frontend might want to retry or ask for user input")
+    details: Optional[Any] = Field(None, description="Raw technical details or traceback (if permitted)")
+
+    class Config:
+        schema_extra = {
+            "example": {
+                "category": "database",
+                "code": 1062,
+                "message": "Duplicate entry for key 'id_random'",
+                "recoverable": False
+            }
+        }

app/models/event_abstract_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -23,31 +23,19 @@ class Event_Abstract_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        # **base_fields['event_abstract_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_abstract_id_random'])
-        alias = 'event_abstract_id_random',
+    event_abstract_id: Optional[str] = Field(None, **base_fields['event_abstract_id_random'])
-    )
+    
-    id: Optional[int] = Field(
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-        alias = 'event_abstract_id'
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
-    )
+    event_person_id: Optional[str] = Field(None, **base_fields['event_person_id_random'])
+    event_presentation_id: Optional[str] = Field(None, **base_fields['event_presentation_id_random'])
+    event_presenter_id: Optional[str] = Field(None, **base_fields['event_presenter_id_random'])
+    event_session_id: Optional[str] = Field(None, **base_fields['event_session_id_random'])
+    grant_id: Optional[str] = Field(None, **base_fields['grant_id_random'])
 
-    event_id_random: Optional[str]
+    # event_track_id: Optional[str] = Field(None, **base_fields['event_track_id_random'])
-    event_id: Optional[int]
-
-    event_person_id_random: Optional[str] # This is the primary person/submitter
-    event_person_id: Optional[int]
-
-    event_presentation_id_random: Optional[str]
-    event_presentation_id: Optional[int]
-
-    event_presenter_id_random: Optional[str]
-    event_presenter_id: Optional[int]
-
-    event_session_id_random: Optional[str]
-    event_session_id: Optional[int]
-
-    # event_track_id_random: Optional[str]
-    # event_track_id: Optional[int]
 
     # poc_event_person_id_random: Optional[str] # Maybe change this to primary_event_person?
     # poc_event_person_id: Optional[int] # Maybe change this to primary_event_person?
@@ -55,9 +43,6 @@ class Event_Abstract_Base(BaseModel):
     external_id: Optional[str]
     code: Optional[str]
 
-    grant_id_random: Optional[str]
-    grant_id: Optional[int]
-
     grant_code: Optional[str]
     # grant_type_code: Optional[str]
     # grant_json: Optional[Union[Json, None]]
@@ -101,67 +86,48 @@ class Event_Abstract_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
+    @root_validator(pre=True)
-    def event_abstract_id_lookup(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if isinstance(v, int) and v > 0: return v
+        """
-        elif id_random := values.get('id_random'):
+        Vision Transformer:
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_abstract')
+        Map DB keys to clean API keys and strip internal integers.
-        return None
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_abstract_id_random'):
+            values['id'] = rid
+            values['event_abstract_id'] = rid
+            
+        if a_rid := values.get('account_id_random'):
+            values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'):
+            values['event_id'] = e_rid
+        if ep_rid := values.get('event_person_id_random'):
+            values['event_person_id'] = ep_rid
+        if epr_rid := values.get('event_presentation_id_random'):
+            values['event_presentation_id'] = epr_rid
+        if eps_rid := values.get('event_presenter_id_random'):
+            values['event_presenter_id'] = eps_rid
+        if es_rid := values.get('event_session_id_random'):
+            values['event_session_id'] = es_rid
+        if g_rid := values.get('grant_id_random'):
+            values['grant_id'] = g_rid
+            
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'event_abstract_id', 'account_id', 'event_id', 'event_person_id', 'event_presentation_id', 'event_presenter_id', 'event_session_id', 'grant_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+                
+        return values
 
-    @validator('event_id', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def event_id_lookup(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        if isinstance(v, int) and v > 0: return v
+        'account_id', 'event_session_code', 'event_session_name',
-        elif id_random := values.get('event_id_random'):
+        'event_file_list', 'event_person', 'event_presenter_list'
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
+    ]
-        return None
-
-    @validator('event_person_id', always=True)
-    def event_person_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_person_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_person')
-        return None
-
-    @validator('event_presentation_id', always=True)
-    def event_presentation_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_presentation_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_presentation')
-        return None
-
-    @validator('event_presenter_id', always=True)
-    def event_presenter_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_presenter_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_presenter')
-        return None
-
-    @validator('event_session_id', always=True)
-    def event_session_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_session_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_session')
-        return None
-
-    @validator('grant_id', always=True)
-    def grant_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('grant_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='grant')
-        return None
-
-    # @validator('poc_event_person_id', always=True)
-    # def poc_event_person_id_lookup(cls, v, values, **kwargs):
-    #     log.setLevel(logging.WARNING)
-    #     log.debug(locals())
-
-    #     if values['poc_event_person_id_random']:
-    #         return redis_lookup_id_random(record_id_random=values['poc_event_person_id_random'], table_name='poc_event_person')
-    #     return None
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Event Abstract Models ### Event_Abstract_Base() ###
 
@@ -175,24 +141,17 @@ class Event_Abstract_Base_New(Core_Std_Obj_Base):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        alias = 'event_abstract_id_random',
+    id: Optional[str] = Field(None, **base_fields['event_abstract_id_random'])
-    )
+    event_abstract_id: Optional[str] = Field(None, **base_fields['event_abstract_id_random'])
-
+    
-    event_id_random: Optional[str]
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-    event_id: Optional[int]
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
-
+    event_person_id: Optional[str] = Field(None, **base_fields['event_person_id_random'])
-    event_person_id: Optional[int]
+    event_presentation_id: Optional[str] = Field(None, **base_fields['event_presentation_id_random'])
-
+    event_presenter_id: Optional[str] = Field(None, **base_fields['event_presenter_id_random'])
-    event_session_id: Optional[int]
+    event_session_id: Optional[str] = Field(None, **base_fields['event_session_id_random'])
-
+    grant_id: Optional[str] = Field(None, **base_fields['grant_id_random'])
-    event_person_id_random: Optional[str]
-
-    event_presentation_id_random: Optional[str]
-
-    event_presenter_id_random: Optional[str]
-
-    event_session_id_random: Optional[str]
 
     # event_track_id_random: Optional[str]
 
@@ -209,9 +168,6 @@ class Event_Abstract_Base_New(Core_Std_Obj_Base):
 
     passcode: Optional[str]
 
-    grant_id_random: Optional[str]
-    grant_id: Optional[int]
-
     grant_code: Optional[str]
     grant_type_code: Optional[str]
     grant_json: Optional[Union[Json, None]]
@@ -224,23 +180,47 @@ class Event_Abstract_Base_New(Core_Std_Obj_Base):
     submitter_json: Optional[Union[Json, None]]
     coauthors_json: Optional[Union[Json, None]]
 
-    @validator('event_person_id', always=True)
+    @root_validator(pre=True)
-    def event_person_id_lookup(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if isinstance(v, int) and v > 0: return v
+        """
-        elif id_random := values.get('event_person_id_random'):
+        Vision Transformer:
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_person')
+        Map DB keys to clean API keys and strip internal integers.
-        return None
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_abstract_id_random'):
+            values['id'] = rid
+            values['event_abstract_id'] = rid
+            
+        if a_rid := values.get('account_id_random'):
+            values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'):
+            values['event_id'] = e_rid
+        if ep_rid := values.get('event_person_id_random'):
+            values['event_person_id'] = ep_rid
+        if epr_rid := values.get('event_presentation_id_random'):
+            values['event_presentation_id'] = epr_rid
+        if eps_rid := values.get('event_presenter_id_random'):
+            values['event_presenter_id'] = eps_rid
+        if es_rid := values.get('event_session_id_random'):
+            values['event_session_id'] = es_rid
+        if g_rid := values.get('grant_id_random'):
+            values['grant_id'] = g_rid
+            
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'event_abstract_id', 'account_id', 'event_id', 'event_person_id', 'event_presentation_id', 'event_presenter_id', 'event_session_id', 'grant_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+                
+        return values
 
-    # @validator('event_session_id', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    # def event_session_id_lookup(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-    #     if isinstance(v, int) and v > 0: return v
+        'account_id'
-    #     elif id_random := values.get('event_session_id_random'):
+    ]
-    #         return redis_lookup_id_random(record_id_random=id_random, table_name='event_session')
-    #     return None
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Event Abstract Models ### Event_Abstract_Base() ###
 
@@ -250,68 +230,11 @@ class Event_Abstract_Base_New(Core_Std_Obj_Base):
 class Event_Abstract_In(Event_Abstract_Base_New):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
+    
+    # Inherits everything from Event_Abstract_Base_New including the Vision ID pattern.
+    # We do NOT redefine 'id' as int here.
 
-    id: Optional[int] = Field(
+    pass
-        alias = 'event_abstract_id'
-    )
-
-    event_id: Optional[int]
-
-    event_person_id: Optional[int]
-
-    # event_session_id: Optional[int]
-
-    # grant_json: Optional[Union[str, None]]
-
-
-    @validator('id', always=True)
-    def event_abstract_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_abstract')
-        return None
-
-    @validator('event_id', always=True)
-    def event_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
-        return None
-
-    @validator('event_person_id', always=True)
-    def event_person_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_person_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_person')
-        return None
-
-    # @validator('event_presentation_id', always=True)
-    # def event_presentation_id_lookup(cls, v, values, **kwargs):
-    #     if isinstance(v, int) and v > 0: return v
-    #     elif id_random := values.get('event_presentation_id_random'):
-    #         return redis_lookup_id_random(record_id_random=id_random, table_name='event_presentation')
-    #     return None
-
-    # @validator('event_presenter_id', always=True)
-    # def event_presenter_id_lookup(cls, v, values, **kwargs):
-    #     if isinstance(v, int) and v > 0: return v
-    #     elif id_random := values.get('event_presenter_id_random'):
-    #         return redis_lookup_id_random(record_id_random=id_random, table_name='event_presenter')
-    #     return None
-
-    # @validator('event_session_id', always=True)
-    # def event_session_id_lookup(cls, v, values, **kwargs):
-    #     if isinstance(v, int) and v > 0: return v
-    #     elif id_random := values.get('event_session_id_random'):
-    #         return redis_lookup_id_random(record_id_random=id_random, table_name='event_session')
-    #     return None
-
-    @validator('grant_id', always=True)
-    def grant_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('grant_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='grant')
-        return None
 # ### END ### API Event Abstract Models ### Event_Abstract_In() ###
 
 
@@ -340,4 +263,9 @@ class Event_Abstract_Ext(Event_Abstract_Base_New):
     # event_track: Optional[Event_Track_Base]
     # poc_event_person: Optional[Event_Person_Base] # Maybe change this to primary_event_person?
     # poc_person: Optional[Person_Base] # Maybe change this to primary_person?
+
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
+    fields_to_exclude_from_db: ClassVar[list] = Event_Abstract_Base_New.fields_to_exclude_from_db + [
+        'event_session_code', 'event_session_name', 'event_cfg', 'event_file_list', 'event_person'
+    ]
 # ### END ### API Event Abstract Models ### Event_Abstract_Ext() ###

app/models/event_badge_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -11,33 +11,60 @@ from app.models.event_badge_template_models import Event_Badge_Template_Base
 from app.models.order_models import Order_Base
 
 
+# ### BEGIN ### API Event Badge Models ### Event_Badge_Base() ###
 class Event_Badge_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings for API, Integers for DB) ---
-        # **base_fields['event_badge_id_random'],
+    id: Optional[Union[int, str]] = Field(**base_fields['event_badge_id_random'])
-        alias = 'event_badge_id_random',
+    event_badge_id: Optional[Union[int, str]] = Field(**base_fields['event_badge_id_random'])
-    )
+    account_id: Optional[Union[int, str]] = Field(None, **base_fields['account_id_random'])
-    id: Optional[int] = Field(
+    event_id: Optional[Union[int, str]] = Field(**base_fields['event_id_random'])
-        alias = 'event_badge_id'
+    
-    )
-
-    event_id_random: Optional[str]
-    event_id: Optional[int]
-
     # NOTE: This should only be used when the event_person record can not be created. And records before 2022.
-    event_id_random_only: Optional[str]
+    event_id_only: Optional[Union[int, str]] = Field(**base_fields['event_id_random'])
-    event_id_only: Optional[int]
 
-    event_badge_template_id_random: Optional[str]
+    event_badge_template_id: Optional[Union[int, str]] = Field(**base_fields['event_badge_template_id_random'])
-    event_badge_template_id: Optional[int]
+    event_person_id: Optional[Union[int, str]] = Field(**base_fields['event_person_id_random'])
+    person_id: Optional[Union[int, str]] = Field(**base_fields['person_id_random'])
 
-    event_person_id_random: Optional[str]
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    event_person_id: Optional[int]
+    id_random: Optional[str] = Field(None, alias='event_badge_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random_only: Optional[str] = Field(None, exclude=True)
+    event_badge_template_id_random: Optional[str] = Field(None, exclude=True)
+    event_person_id_random: Optional[str] = Field(None, exclude=True)
+    person_id_random: Optional[str] = Field(None, exclude=True)
 
-    person_id_random: Optional[str]
+    @root_validator(pre=True)
-    person_id: Optional[int]
+    def map_v3_ids(cls, values):
+        """
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers during READ operations.
+        During CREATE (POST) operations, we ensure resolved integers are preserved.
+        """
+        # 1. Map Random Strings to Clean Names
+        rid = values.get('id_random') or values.get('event_badge_id_random')
+        if rid and isinstance(rid, str):
+            values['id'] = rid
+            values['event_badge_id'] = rid
+            
+        if a_rid := values.get('account_id_random'): values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'): values['event_id'] = e_rid
+        if eo_rid := values.get('event_id_random_only'): values['event_id_only'] = eo_rid
+        if et_rid := values.get('event_badge_template_id_random'): values['event_badge_template_id'] = et_rid
+        if ep_rid := values.get('event_person_id_random'): values['event_person_id'] = ep_rid
+        if p_rid := values.get('person_id_random'): values['person_id'] = p_rid
+            
+        # 2. Prevent leakage of integers during API responses (Vision Standard)
+        for k in ['id', 'event_badge_id', 'account_id', 'event_id', 'event_id_only', 'event_badge_template_id', 'event_person_id', 'person_id']:
+            val = values.get(k)
+            if val is not None and not isinstance(val, str):
+                values[k] = None
+                
+        return values
 
     external_id: Optional[str] # Generated internally or externally. Needs to be stable. It should not change.
     external_event_id: Optional[str] # Event ID generated by external system. Needs to be stable. It should not change.
@@ -141,7 +168,11 @@ class Event_Badge_Base(BaseModel):
     # affiliations_font_size: Optional[str] # Not currently used 2023-01-25
     # location_font_size: Optional[str] # Not currently used 2023-01-25
     # css: Optional[str] # Not currently used 2023-01-25
-    # other_json: Optional[str] # Not currently used 2023-01-25
+
+    cfg_json: Optional[Union[Json, None]] # Store per badge config options like font size; Not currently used 2024-06-11
+    data_json: Optional[Union[Json, None]] # For key value data. Careful with overwriting existing fields! Not currently used 2024-06-11
+
+    default_qry_str: Optional[str] # Default query string used for searching and filtering badges. Updated using SQL triggers and a SQL function
 
     hide: Optional[bool]
     priority: Optional[bool]
@@ -160,47 +191,10 @@ class Event_Badge_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def event_badge_id_lookup(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        if isinstance(v, int) and v > 0: return v
+        'account_id', 'order', 'ticket_list', 'event_badge_template'
-        elif id_random := values.get('id_random'):
+    ]
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_badge')
-        return None
-
-    @validator('event_id', always=True)
-    def event_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
-        return None
-
-    @validator('event_id_only', always=True)
-    def event_id_only_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_id_random_only'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
-        return None
-
-    @validator('event_badge_template_id', always=True)
-    def event_badge_template_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_badge_template_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_badge_template')
-        return None
-
-    @validator('event_person_id', always=True)
-    def event_person_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_person_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_person')
-        return None
-
-    @validator('person_id', always=True)
-    def person_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('person_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='person')
-        return None
 
     class Config:
         underscore_attrs_are_private = True
@@ -212,15 +206,44 @@ class Event_Badge_Basic_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings for API, Integers for DB) ---
-        **base_fields['event_badge_id_random'],
+    id: Optional[Union[int, str]] = Field(None, **base_fields['event_badge_id_random'])
-        alias = 'event_badge_id_random',
+    event_badge_id: Optional[Union[int, str]] = Field(None, **base_fields['event_badge_id_random'])
-    )
+    account_id: Optional[Union[int, str]] = Field(None, **base_fields['account_id_random'])
-    event_badge_template_id_random: Optional[str]
+    event_badge_template_id: Optional[Union[int, str]] = Field(None, **base_fields['event_badge_template_id_random'])
-    # event_badge_template_id: Optional[int]
+    event_person_id: Optional[Union[int, str]] = Field(None, **base_fields['event_person_id_random'])
 
-    event_person_id_random: Optional[str]
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    # event_person_id: Optional[int]
+    id_random: Optional[str] = Field(None, alias='event_badge_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    event_badge_template_id_random: Optional[str] = Field(None, exclude=True)
+    event_person_id_random: Optional[str] = Field(None, exclude=True)
+
+    @root_validator(pre=True)
+    def map_v3_ids(cls, values):
+        """
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers during READ operations.
+        During CREATE (POST) operations, we ensure resolved integers are preserved.
+        """
+        # 1. Map Random Strings to Clean Names
+        rid = values.get('id_random') or values.get('event_badge_id_random')
+        if rid and isinstance(rid, str):
+            values['id'] = rid
+            values['event_badge_id'] = rid
+            
+        if a_rid := values.get('account_id_random'): values['account_id'] = a_rid
+        if et_rid := values.get('event_badge_template_id_random'): values['event_badge_template_id'] = et_rid
+        if ep_rid := values.get('event_person_id_random'): values['event_person_id'] = ep_rid
+            
+        # 2. Prevent "Collision Population" or leakage of integers during API responses
+        for k in ['id', 'event_badge_id', 'account_id', 'event_badge_template_id', 'event_person_id']:
+            val = values.get(k)
+            if val is not None and not isinstance(val, str):
+                if values.get(f'{k}_random') or (k=='id' and values.get('id_random')):
+                    del values[k]
+                
+        return values
 
     external_id: Optional[str] # Generated internally or externally. Needs to be stable. It should not change.
     # external_sys_id: Optional[str] # Person ID generated by external system (should be stable and not change)
@@ -290,19 +313,19 @@ class Event_Badge_Basic_Base(BaseModel):
     allow_tracking: Optional[bool] # Allow tracking for lead retrieval and other marketing
     # agree_to_tc: Optional[bool] # Agree to terms and conditions
 
-    # print_first_datetime: Optional[datetime.datetime] = None
+    print_first_datetime: Optional[datetime.datetime] = None
-    # print_last_datetime: Optional[datetime.datetime] = None
+    print_last_datetime: Optional[datetime.datetime] = None
-    # print_count: Optional[int]
+    print_count: Optional[int]
 
-    # hide: Optional[bool]
+    hide: Optional[bool]
-    # priority: Optional[bool]
+    priority: Optional[bool]
-    # sort: Optional[int]
+    sort: Optional[int]
-    # group: Optional[str]
+    group: Optional[str]
-    # enable: Optional[bool]
+    enable: Optional[bool]
 
-    # notes: Optional[str]
+    notes: Optional[str]
-    # created_on: Optional[datetime.datetime] = None
+    created_on: Optional[datetime.datetime] = None
-    # updated_on: Optional[datetime.datetime] = None
+    updated_on: Optional[datetime.datetime] = None
 
     # Including other related objects
     # order: Optional[Union[Order_Base, None]]
@@ -311,7 +334,13 @@ class Event_Badge_Basic_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
+    fields_to_exclude_from_db: ClassVar[list] = [
+        'account_id', 'event_badge_template'
+    ]
+
     class Config:
         underscore_attrs_are_private = True
         allow_population_by_field_name = True
         fields = base_fields
+

app/models/event_badge_template_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -15,19 +15,16 @@ class Event_Badge_Template_Base(BaseModel):
 
     # log.info('Using base template')
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        **base_fields['event_badge_template_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_badge_template_id_random'])
-        alias = 'event_badge_template_id_random',
+    event_badge_template_id: Optional[str] = Field(None, **base_fields['event_badge_template_id_random'])
-    )
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
-    id: Optional[int] = Field(
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-        alias = 'event_badge_template_id'
-    )
 
-    # account_id_random: Optional[str]
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    # account_id: Optional[int]
+    id_random: Optional[str] = Field(None, alias='event_badge_template_id_random', exclude=True)
-
+    event_id_random: Optional[str] = Field(None, exclude=True)
-    event_id_random: Optional[str]
+    account_id_random: Optional[str] = Field(None, exclude=True)
-    event_id: Optional[int]
 
     name: Optional[str]
     description: Optional[str]
@@ -40,6 +37,7 @@ class Event_Badge_Template_Base(BaseModel):
     header_background: Optional[str]
 
     secondary_header_path: Optional[str] # Path to image file for back of badge and other sections
+    background_image_path: Optional[str]
 
     footer_path: Optional[str] # Path to image file
     footer_title: Optional[str]
@@ -49,6 +47,8 @@ class Event_Badge_Template_Base(BaseModel):
 
     badge_type_list: Optional[Json]
 
+    duplex: Optional[bool]
+
     ticket_list: Optional[Json]
     ticket_1_text: Optional[str]
     ticket_2_text: Optional[str]
@@ -74,29 +74,47 @@ class Event_Badge_Template_Base(BaseModel):
     script_src: Optional[str]
     passcode: Optional[str]
 
-    other_json: Optional[str]
+    other_json: Optional[Json]
+    cfg_json: Optional[Json]
 
+    enable: Optional[bool]
+    hide: Optional[bool]
+    priority: Optional[int]
+    sort: Optional[int]
+    group: Optional[str]
     notes: Optional[str]
+    created_on: Optional[datetime.datetime] = None
+    updated_on: Optional[datetime.datetime] = None
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
+    @root_validator(pre=True)
-    def event_badge_template_id_lookup(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if isinstance(v, int) and v > 0: return v
+        """
-        elif id_random := values.get('id_random'):
+        Vision Transformer:
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_badge_template')
+        Map DB keys to clean API keys and strip internal integers.
-        return None
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_badge_template_id_random'):
+            values['id'] = rid
+            values['event_badge_template_id'] = rid
 
-    @validator('event_id', always=True)
+        if e_rid := values.get('event_id_random'):
-    def event_id_lookup(cls, v, values, **kwargs):
+            values['event_id'] = e_rid
-        if isinstance(v, int) and v > 0: return v
+
-        elif id_random := values.get('event_id_random'):
+        if a_rid := values.get('account_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
+            values['account_id'] = a_rid
-        return None
+
+        # 2. Prevent "Collision Population" (ensure no integers leak into the clean string fields)
+        for k in ['id', 'event_badge_template_id', 'event_id', 'account_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+
+        return values
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 
 
@@ -116,6 +134,5 @@ class Event_Badge_Template_Base_Out(Event_Badge_Template_Base):
     # log.info('Using Out template')
 
     # badge_type_list: Optional[Json]
+    event_name: Optional[str]
 
-    created_on: Optional[datetime.datetime] = None
-    updated_on: Optional[datetime.datetime] = None

app/models/event_cfg_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -15,16 +15,40 @@ class Event_Cfg_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        # **base_fields['event_cfg_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_cfg_id_random'])
-        alias = 'event_cfg_id_random',
+    event_cfg_id: Optional[str] = Field(None, **base_fields['event_cfg_id_random'])
-    )
-    id: Optional[int] = Field(
-        alias = 'event_cfg_id'
-    )
 
-    event_id_random: Optional[str]
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-    event_id: Optional[int]
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
+
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
+    id_random: Optional[str] = Field(None, alias='event_cfg_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random: Optional[str] = Field(None, exclude=True)
+
+    @root_validator(pre=True)
+    def map_v3_ids(cls, values):
+        """
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers.
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_cfg_id_random'):
+            values['id'] = rid
+            values['event_cfg_id'] = rid
+            
+        if a_rid := values.get('account_id_random'):
+            values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'):
+            values['event_id'] = e_rid
+            
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'event_cfg_id', 'account_id', 'event_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+                
+        return values
 
     enable: Optional[bool]
     enable_from: Optional[datetime.datetime]
@@ -105,6 +129,11 @@ class Event_Cfg_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
+    fields_to_exclude_from_db: ClassVar[list] = [
+        'account_id', 'event_id', 'event_registration_cfg'
+    ]
+
     class Config:
         underscore_attrs_are_private = True
         allow_population_by_field_name = True

app/models/event_device_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -16,22 +16,13 @@ class Event_Device_Base(BaseModel):
     log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        **base_fields['event_device_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_device_id_random'])
-        alias = 'event_device_id_random',
+    event_device_id: Optional[str] = Field(None, **base_fields['event_device_id_random'])
-    )
-    id: Optional[int] = Field(
-        alias = 'event_device_id'
-    )
 
-    account_id_random: Optional[str]
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-    account_id: Optional[int]
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
-
+    event_location_id: Optional[str] = Field(None, **base_fields['event_location_id_random'])
-    event_id_random: Optional[str]
-    event_id: Optional[int]
-
-    event_location_id_random: Optional[str]
-    event_location_id: Optional[int]
 
     code: Optional[str]
 
@@ -45,11 +36,11 @@ class Event_Device_Base(BaseModel):
     api_secret_key: Optional[str]
 
     api_base_url: Optional[str]
-    app_server_base_url: Optional[str]
+    app_base_url: Optional[str]
     file_server_base_url: Optional[str]
 
     api_base_url_bak: Optional[str] # Backup URL
-    app_server_base_url_bak: Optional[str] # Backup URL
+    app_base_url_bak: Optional[str] # Backup URL
     file_server_base_url_bak: Optional[str] # Backup URL
 
     trigger_open_filename: Optional[str] # The file hash filename
@@ -83,20 +74,34 @@ class Event_Device_Base(BaseModel):
     check_event_location_loop_period: Optional[int]
     check_event_session_loop_period: Optional[int]
 
+    passcode: Optional[str]
+
     alert: Optional[bool]
     alert_msg: Optional[str]
+    alert_on: Optional[datetime.datetime]
+    status: Optional[str]
+    status_msg: Optional[str]
+    status_msg_on: Optional[datetime.datetime]
+    record_status: Optional[str]
+    record_status_msg: Optional[str]
+    record_status_on: Optional[datetime.datetime]
+
+    heartbeat: Optional[datetime.datetime]
 
     info_hostname: Optional[str]
     info_ip: Optional[str]
     info_ip_list: Optional[str] # string list of IPs separated by ;
     info_os: Optional[str]
 
+    cfg_json: Optional[Union[Json, None]] # Store per device config options like theme, language, etc
+    data_json: Optional[Union[Json, None]] # For key value data. Careful with overwriting existing fields!
+
     enable: Optional[bool]
 
-    # hide: Optional[bool]
+    hide: Optional[bool]
-    # priority: Optional[bool]
+    priority: Optional[bool]
-    # sort: Optional[int]
+    sort: Optional[int]
-    # group: Optional[str]
+    group: Optional[str]
 
     event_notes: Optional[str]
     notes: Optional[str]
@@ -117,45 +122,36 @@ class Event_Device_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('event_device_id_random', always=True)
+    @root_validator(pre=True)
-    # def event_device_id_random_copy(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-    #     log.setLevel(logging.WARNING)
+        """
-    #     log.debug(locals())
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers.
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_device_id_random'):
+            values['id'] = rid
+            values['event_device_id'] = rid
+            
+        if a_rid := values.get('account_id_random'):
+            values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'):
+            values['event_id'] = e_rid
+        if el_rid := values.get('event_location_id_random'):
+            values['event_location_id'] = el_rid
+            
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'event_device_id', 'account_id', 'event_id', 'event_location_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+                
+        return values
 
-    #     if values['id_random']:
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    #         return values['id_random']
+    fields_to_exclude_from_db: ClassVar[list] = ['account_id', 'event_cfg', 'event_location']
-    #     return None
-
-    @validator('id', always=True)
-    def event_device_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_device')
-        return None
-
-    @validator('account_id', always=True)
-    def account_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('account_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='account')
-        return None
-
-    @validator('event_id', always=True)
-    def event_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
-        return None
-
-    @validator('event_location_id', always=True)
-    def event_location_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_location_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_location')
-        return None
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Event Device Models ### Event_Device_Base() ###

app/models/event_exhibit_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -15,32 +15,78 @@ class Event_Exhibit_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings for API, Integers for DB) ---
-        **base_fields['event_exhibit_id_random'],
+    id: Optional[Union[int, str]] = Field(**base_fields['event_exhibit_id_random'])
-        alias = 'event_exhibit_id_random',
+    event_exhibit_id: Optional[Union[int, str]] = Field(**base_fields['event_exhibit_id_random'])
-    )
+    account_id: Optional[Union[int, str]] = Field(**base_fields['account_id_random'])
-    id: Optional[int] = Field(
+    event_id: Optional[Union[int, str]] = Field(**base_fields['event_id_random'])
-        alias = 'event_exhibit_id'
+    organization_id: Optional[Union[int, str]] = Field(**base_fields['organization_id_random'])
-    )
+    contact_id: Optional[Union[int, str]] = Field(**base_fields['contact_id_random'])
+    person_id: Optional[Union[int, str]] = Field(**base_fields['person_id_random'])
+    status_id: Optional[Union[int, str]] = Field(**base_fields['status_id_random'])
+
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
+    id_random: Optional[str] = Field(None, alias='event_exhibit_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random: Optional[str] = Field(None, exclude=True)
+    organization_id_random: Optional[str] = Field(None, exclude=True)
+    contact_id_random: Optional[str] = Field(None, exclude=True)
+    person_id_random: Optional[str] = Field(None, exclude=True)
+    status_id_random: Optional[str] = Field(None, exclude=True)
+
+    @root_validator(pre=True)
+    def map_v3_ids(cls, values):
+        """
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers during READ operations.
+        Falls back to Redis/DB lookups if random string IDs are missing from the view.
+        """
+        from app.db_sql import get_id_random
+
+        # 1. Map Primary Object ID
+        rid = values.get('id_random') or values.get('event_exhibit_id_random')
+        if rid and isinstance(rid, str):
+            values['id'] = rid
+            values['event_exhibit_id'] = rid
+        elif values.get('id') and isinstance(values.get('id'), int):
+            # Fallback for primary ID
+            resolved_rid = get_id_random(values['id'], 'event_exhibit')
+            if resolved_rid:
+                values['id'] = resolved_rid
+                values['event_exhibit_id'] = resolved_rid
+                values['id_random'] = resolved_rid
+
+        # 2. Map & Resolve Relational IDs
+        id_map = [
+            ('account_id', 'account'),
+            ('event_id', 'event'),
+            ('organization_id', 'organization'),
+            ('contact_id', 'contact'),
+            ('person_id', 'person'),
+            ('status_id', 'status'),
+        ]
+
+        for field, table in id_map:
+            r_val = values.get(f'{field}_random')
+            if r_val and isinstance(r_val, str):
+                values[field] = r_val
+            elif values.get(field) and isinstance(values[field], int):
+                # Fallback: Resolve from Redis/DB if missing from view result
+                resolved_rid = get_id_random(values[field], table)
+                if resolved_rid:
+                    values[field] = resolved_rid
+                    values[f'{field}_random'] = resolved_rid
+
+        # 3. Final Vision Enforcement: Strip internal integers
+        for k in ['id', 'event_exhibit_id', 'account_id', 'event_id', 'organization_id', 'contact_id', 'person_id', 'status_id']:
+            val = values.get(k)
+            if val is not None and not isinstance(val, str):
+                values[k] = None
+                
+        return values
 
     code: Optional[str] # The assigned booth number or ID
 
-    account_id_random: Optional[str]
-    account_id: Optional[int]
-    event_id_random: Optional[str]
-    event_id: Optional[int]
-    organization_id_random: Optional[str]
-    organization_id: Optional[int]
-    contact_id_random: Optional[str]
-    contact_id: Optional[int]
-
-    # Point of Contact person ID
-    person_id_random: Optional[str]
-    person_id: Optional[int]
-
-    status_id_random: Optional[str]
-    status_id: Optional[int]
-
     staff_passcode: Optional[str]
 
     name: Optional[str]
@@ -65,6 +111,11 @@ class Event_Exhibit_Base(BaseModel):
     leads_device_sm_qty: Optional[int] # NOTE: Cell phone sized devices rented by exhibitor. Should this be a separate linked table (event_device)?
     leads_device_lg_qty: Optional[int] # NOTE: Tablet (8 or 9 inch) sized devices rented by exhibitor. Should this be a separate linked table (event_device)?
 
+    data_json: Optional[Union[Json, None]]
+    license_max: Optional[int]
+    license_li_json: Optional[Union[Json, None]]
+    cfg_json: Optional[Union[Json, None]]
+
     enable_organization_name_change: Optional[bool]
     enable_name_change: Optional[bool]
     enable_banner_image: Optional[bool]
@@ -74,10 +125,11 @@ class Event_Exhibit_Base(BaseModel):
 
     # access_key: Optional[str] # Maybe use in the future?
 
-    # enable: Optional[bool] # Maybe use in the future?
+    enable: Optional[bool]
     # enable_from: Optional[datetime.datetime] = None # Maybe use in the future?
     # enable_to: Optional[datetime.datetime] = None # Maybe use in the future?
 
+    hide: Optional[bool]
     priority: Optional[bool]
     sort: Optional[int]
     group: Optional[str]
@@ -91,78 +143,10 @@ class Event_Exhibit_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('event_exhibit_id_random', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def event_exhibit_id_random_copy(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        log.setLevel(logging.WARNING)
+        'event_exhibit_tracking_list'
-        log.debug(locals())
+    ]
-
-        if values['id_random']:
-            return values['id_random']
-        return None
-
-    @validator('id', always=True)
-    def event_exhibit_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['id_random']:
-            log.debug(values['id_random'])
-            return redis_lookup_id_random(record_id_random=values['id_random'], table_name='event_exhibit')
-        return None
-
-    @validator('account_id', always=True)
-    def account_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['account_id_random']:
-            return redis_lookup_id_random(record_id_random=values['account_id_random'], table_name='account')
-        return None
-
-    @validator('event_id', always=True)
-    def event_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['event_id_random']:
-            return redis_lookup_id_random(record_id_random=values['event_id_random'], table_name='event')
-        return None
-
-    @validator('organization_id', always=True)
-    def organization_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['organization_id_random']:
-            return redis_lookup_id_random(record_id_random=values['organization_id_random'], table_name='organization')
-        return None
-
-    @validator('contact_id', always=True)
-    def contact_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['contact_id_random']:
-            return redis_lookup_id_random(record_id_random=values['contact_id_random'], table_name='contact')
-        return None
-
-    @validator('person_id', always=True)
-    def person_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['person_id_random']:
-            return redis_lookup_id_random(record_id_random=values['person_id_random'], table_name='person')
-        return None
-
-    # @validator('leads_custom_questions_json', always=True)
-    # def leads_custom_questions_json_fix(cls, v, values, **kwargs):
-    #     if isinstance(v, str):
-    #         return json.loads(v)
-    #         # return v.dict()
-    #     else:
-    #         return v
-    #     return None
 
     class Config:
         underscore_attrs_are_private = True

app/models/event_exhibit_tracking_models.py

@@ -1,14 +1,14 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
 
 from app.models.common_field_schema import base_fields, default_num_bytes
 
-from app.models.event_badge_models import Event_Badge_Base
+# from app.models.event_badge_models import Event_Badge_Base
 from app.models.event_person_models import Event_Person_Base
 
 
@@ -17,34 +17,60 @@ class Event_Exhibit_Tracking_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings for API, Integers/Strings for DB) ---
-        **base_fields['event_exhibit_tracking_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_exhibit_tracking_id_random'])
-        alias = 'event_exhibit_tracking_id_random',
+    event_exhibit_tracking_id: Optional[str] = Field(None, **base_fields['event_exhibit_tracking_id_random'])
-    )
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-    id: Optional[int] = Field(
-        alias = 'event_exhibit_tracking_id'
-    )
 
-    event_id_random: Optional[str]
+    event_id: Optional[Union[int, str]] = Field(None, **base_fields['event_id_random'])
-    event_id: Optional[int]
+    event_exhibit_id: Optional[Union[int, str]] = Field(None, **base_fields['event_exhibit_id_random'])
+    event_person_id: Optional[Union[int, str]] = Field(None, **base_fields['event_person_id_random'])
+    event_badge_id: Optional[Union[int, str]] = Field(None, **base_fields['event_badge_id_random'])
 
-    event_exhibit_id_random: Optional[str]
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    event_exhibit_id: Optional[int]
+    id_random: Optional[str] = Field(None, alias='event_exhibit_tracking_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random: Optional[str] = Field(None, exclude=True)
+    event_exhibit_id_random: Optional[str] = Field(None, exclude=True)
+    event_person_id_random: Optional[str] = Field(None, exclude=True)
+    event_badge_id_random: Optional[str] = Field(None, exclude=True)
 
-    event_person_id_random: Optional[str]
+    @root_validator(pre=True)
-    event_person_id: Optional[int]
+    def map_v3_ids(cls, values):
+        """
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers.
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_exhibit_tracking_id_random'):
+            values['id'] = rid
+            values['event_exhibit_tracking_id'] = rid
+            
+        if a_rid := values.get('account_id_random'): values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'): values['event_id'] = e_rid
+        if ee_rid := values.get('event_exhibit_id_random'): values['event_exhibit_id'] = ee_rid
+        if ep_rid := values.get('event_person_id_random'): values['event_person_id'] = ep_rid
+        if eb_rid := values.get('event_badge_id_random'): values['event_badge_id'] = eb_rid
+            
+        # 2. Prevent "Collision Population"
+        # We only strip integers for the primary IDs and account_id to prevent leak in READ views.
+        # Relational IDs (event_id, exhibit_id, etc.) are allowed to remain as integers during 
+        # POST/PUT operations so they reach the database correctly.
+        for k in ['id', 'event_exhibit_tracking_id', 'account_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+                
+        return values
 
-    event_badge_id_random: Optional[str]
+    external_person_id: Optional[str] # This is probably an email address
-    event_badge_id: Optional[int]
 
     exhibitor_notes: Optional[str]
     responses_json: Optional[Json] # NOTE: Responses to custom questions
-    # responses_json: Json = [{'test': ''}] # NOTE: Responses to custom questions
+    # Example:
-    # responses_json: Optional[Json] = Field(
+    # {"5_years": {"response": "I see myself in 5 years doing something."}, "colors": {"response": "green"}}
-    #     default_factory = lambda:[{'test': ''}]
+    # {"example_text": {"response": "This is an example of an text answer."}, "example_option_list": {"response": "no"}, "the_code": {"response": "yes"}, "question_everything": {"response": "tomorrow"}, "pre_assesment": {"response": "yes"}}
-    # )
+
-    data_json: Optional[Json]
+    data_json: Optional[Json] # NOTE: Additional data
-    # data_json: Optional[str]
 
     enable: Optional[bool]
     hide: Optional[bool]
@@ -167,40 +193,27 @@ class Event_Exhibit_Tracking_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def event_exhibit_tracking_id_lookup(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        if isinstance(v, int) and v > 0: return v
+        'account_id',
-        elif id_random := values.get('id_random'):
+        'event_badge_pronouns', 'event_badge_pronouns_override', 'event_badge_informal_name',
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_exhibit_tracking')
+        'event_badge_title_names', 'event_badge_given_name', 'event_badge_middle_name',
-        return None
+        'event_badge_family_name', 'event_badge_designations',
-
+        'event_badge_professional_title', 'event_badge_professional_title_override',
-    @validator('event_id', always=True)
+        'event_badge_full_name', 'event_badge_full_name_override',
-    def event_id_lookup(cls, v, values, **kwargs):
+        'event_badge_affiliations', 'event_badge_affiliations_override',
-        if isinstance(v, int) and v > 0: return v
+        'event_badge_email', 'event_badge_email_override',
-        elif id_random := values.get('event_id_random'):
+        'event_badge_phone', 'event_badge_phone_override',
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
+        'event_badge_address_line_1', 'event_badge_address_line_2', 'event_badge_address_line_3',
-        return None
+        'event_badge_city', 'event_badge_county', 'event_badge_country_subdivision_code',
-
+        'event_badge_state_province', 'event_badge_state_province_abb', 'event_badge_postal_code',
-    @validator('event_exhibit_id', always=True)
+        'event_badge_country_alpha_2_code', 'event_badge_country',
-    def event_exhibit_id_lookup(cls, v, values, **kwargs):
+        'event_badge_location', 'event_badge_location_override',
-        if isinstance(v, int) and v > 0: return v
+        'event_person_informal_name', 'event_person_given_name', 'event_person_family_name',
-        elif id_random := values.get('event_exhibit_id_random'):
+        'event_person_full_name', 'event_person_full_name_override',
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_exhibit')
+        'event_person_affiliations', 'event_person_email', 'event_exhibit_name',
-        return None
+        'event_person'
-
+    ]
-    @validator('event_person_id', always=True)
-    def event_person_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_person_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_person')
-        return None
-
-    @validator('event_badge_id', always=True)
-    def event_badge_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_badge_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_badge')
-        return None
 
     class Config:
         underscore_attrs_are_private = True

app/models/event_file_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import get_id_random, redis_lookup_id_random
 # from app.lib_general import log, logging
@@ -16,38 +16,110 @@ class Event_File_Base(BaseModel):
     log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings for API, Integers for DB) ---
-        # **base_fields['event_file_id_random'],
+    id: Optional[Union[int, str]] = Field(**base_fields['event_file_id_random'])
-        alias = 'event_file_id_random',
+    event_file_id: Optional[Union[int, str]] = Field(**base_fields['event_file_id_random'])
-    )
+    account_id: Optional[Union[int, str]] = Field(None, **base_fields['account_id_random'])
-    id: Optional[int] = Field(
+    hosted_file_id: Optional[Union[int, str]] = Field(**base_fields['hosted_file_id_random'])
-        alias = 'event_file_id'
+    
-    )
+    # Generic Relational target
+    for_id: Optional[Union[int, str]] = Field(**base_fields['obj_id_random'])
+    
+    event_id: Optional[Union[int, str]] = Field(**base_fields['event_id_random'])
+    event_exhibit_id: Optional[Union[int, str]] = Field(**base_fields['event_exhibit_id_random'])
+    event_location_id: Optional[Union[int, str]] = Field(**base_fields['event_location_id_random'])
+    event_presentation_id: Optional[Union[int, str]] = Field(**base_fields['event_presentation_id_random'])
+    event_presenter_id: Optional[Union[int, str]] = Field(**base_fields['event_presenter_id_random'])
+    event_session_id: Optional[Union[int, str]] = Field(**base_fields['event_session_id_random'])
+    event_track_id: Optional[Union[int, str]] = Field(**base_fields['event_track_id_random'])
+
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
+    id_random: Optional[str] = Field(None, alias='event_file_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    hosted_file_id_random: Optional[str] = Field(None, exclude=True)
+    for_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random: Optional[str] = Field(None, exclude=True)
+    event_exhibit_id_random: Optional[str] = Field(None, exclude=True)
+    event_location_id_random: Optional[str] = Field(None, exclude=True)
+    event_presentation_id_random: Optional[str] = Field(None, exclude=True)
+    event_presenter_id_random: Optional[str] = Field(None, exclude=True)
+    event_session_id_random: Optional[str] = Field(None, exclude=True)
+    event_track_id_random: Optional[str] = Field(None, exclude=True)
+
+    # Internal flag to signal the model to load nested hosted_file
+    inc_hosted_file: Optional[bool] = Field(False, exclude=True)
+
+    @root_validator(pre=True)
+    def map_v3_ids(cls, values):
+        """
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers during READ operations.
+        Falls back to Redis/DB lookups if random string IDs are missing from the view.
+        """
+        # 1. Map Primary Object ID
+        rid = values.get('id_random') or values.get('event_file_id_random')
+        if rid and isinstance(rid, str):
+            values['id'] = rid
+            values['event_file_id'] = rid
+            
+        # 2. Map & Resolve Relational IDs
+        # (Field Name, Table Name)
+        id_map = [
+            ('account_id', 'account'),
+            ('hosted_file_id', 'hosted_file'),
+            ('event_id', 'event'),
+            ('event_exhibit_id', 'event_exhibit'),
+            ('event_location_id', 'event_location'),
+            ('event_presentation_id', 'event_presentation'),
+            ('event_presenter_id', 'event_presenter'),
+            ('event_session_id', 'event_session'),
+            ('event_track_id', 'event_track'),
+        ]
+        
+        # 2a. Handle specific relational fields
+        for field, table in id_map:
+            # Check for existing random string version
+            r_val = values.get(f'{field}_random')
+            if r_val and isinstance(r_val, str):
+                values[field] = r_val
+            elif values.get(field) and isinstance(values[field], int):
+                # Fallback: Resolve from Redis/DB if missing from view result
+                resolved_rid = get_id_random(values[field], table)
+                if resolved_rid:
+                    values[field] = resolved_rid
+                    values[f'{field}_random'] = resolved_rid
+                    
+        # 2b. Handle Polymorphic for_id
+        if f_rid := values.get('for_id_random'):
+            values['for_id'] = f_rid
+        elif values.get('for_id') and isinstance(values.get('for_id'), int) and values.get('for_type'):
+            # Resolve based on the for_type
+            resolved_for_rid = get_id_random(values['for_id'], values['for_type'])
+            if resolved_for_rid:
+                values['for_id'] = resolved_for_rid
+                values['for_id_random'] = resolved_for_rid
+
+        # 3. Final Vision Enforcement: Strip internal integers
+        id_fields = [f for f, t in id_map] + ['id', 'event_file_id', 'for_id']
+        for k in id_fields:
+            val = values.get(k)
+            if val is not None and not isinstance(val, str):
+                values[k] = None
+
+        # 4. Conditionally load nested 'hosted_file' object
+        if values.get('inc_hosted_file') and values.get('hosted_file_id'):
+            from app.methods.hosted_file_methods import load_hosted_file_obj
+            if hosted_file_obj := load_hosted_file_obj(hosted_file_id=values['hosted_file_id']):
+                values['hosted_file'] = hosted_file_obj
+        
+        # Clean up internal inc_hosted_file flag after processing
+        if 'inc_hosted_file' in values:
+            del values['inc_hosted_file']
+                
+        return values
 
-    hosted_file_id_random: Optional[str]
-    hosted_file_id: Optional[int]
 
-    # NOTE: Handling this outside of the Pydantic model and model validation. See below as well. -STI 2021-09-10
     for_type: Optional[str]
-    for_id: Optional[int] # NOTE: This is reversed with for_id_random
-    for_id_random: Optional[str] # NOTE: This is reversed with for_id
-    # for_id_random: Optional[str] = None # Need to override value from common_field_schema.py
-    # for_id: Optional[int]
-
-    event_id_random: Optional[str]
-    event_id: Optional[int]
-    event_exhibit_id_random: Optional[str]
-    event_exhibit_id: Optional[int]
-    event_location_id_random: Optional[str]
-    event_location_id: Optional[int]
-    event_presentation_id_random: Optional[str]
-    event_presentation_id: Optional[int]
-    event_presenter_id_random: Optional[str]
-    event_presenter_id: Optional[int]
-    event_session_id_random: Optional[str]
-    event_session_id: Optional[int]
-    event_track_id_random: Optional[str]
-    event_track_id: Optional[int]
 
     filename: Optional[str]
     filename_no_ext: Optional[str] # Currently created with a view
@@ -56,7 +128,7 @@ class Event_File_Base(BaseModel):
     title: Optional[str]
     description: Optional[str]
 
-    lu_file_purpose_id: Optional[int]
+    lu_file_purpose_id: Optional[int] = Field(None, exclude=True)
     file_purpose: Optional[str]
 
     # New internal use fields to help with logistics and planning 2022-09-15
@@ -81,126 +153,77 @@ class Event_File_Base(BaseModel):
     sort: Optional[int]
     group: Optional[str] # Same or similar as file_purpose?
 
-    # notes: Optional[str]
+    notes: Optional[str]
     created_on: Optional[datetime.datetime] = None
     updated_on: Optional[datetime.datetime] = None
 
-    # Including convenience data
+    # Including convenience data for Hosted Files (top-level properties)
-    # This is only for convenience. Probably going to keep unless it causes a problem.
+    # These fields provide direct access to frequently needed properties from the associated
-    hosted_file_hash_sha256: Optional[str] = Field(
+    # hosted file, effectively flattening some aspects of the nested 'hosted_file' object.
-        alias = 'hash_sha256'
+    #
-    )
+    # IMPORTANT: These fields are designed to be populated directly from the SQL View
-    hosted_file_subdirectory_path: Optional[str] = Field(  # NOTE: This will frequently only contain numbers, but it still needs to be a string
+    # (e.g., `v_event_file_simple`) via JOINs. They should **NOT** have Pydantic `alias`
-        alias = 'subdirectory_path'
+    # definitions here if the view provides them with matching names (e.g., `hosted_file_hash_sha256`).
-    )
+    # Pydantic's default mapping will handle them directly from the incoming data dictionary
-    hosted_file_content_type: Optional[str] = Field(
+    # (the `sql_result` in `api_crud_v3.py`).
-        alias = 'content_type'
+    # The `root_validator` does **NOT** populate these top-level fields; its role is
-    )
+    # solely to conditionally load the *nested* `hosted_file` object.
-    hosted_file_size: Optional[str] = Field(
+    hosted_file_hash_sha256: Optional[str]
-        alias = 'file_size'
+    hosted_file_subdirectory_path: Optional[str]
-    )
+    hosted_file_content_type: Optional[str]
+    hosted_file_size: Optional[str]
 
     lu_event_file_purpose_name: Optional[str] = Field(
         alias = 'file_purpose_name'
     )
 
+    event_name: Optional[str]
+    event_code: Optional[str]
+    event_start_datetime: Optional[datetime.datetime]
+    event_end_datetime: Optional[datetime.datetime]
+    event_location_code: Optional[str]
+    event_location_name: Optional[str]
+    event_presentation_code: Optional[str]
+    event_presentation_type_code: Optional[str]
+    event_presentation_name: Optional[str]
+    event_presentation_start_datetime: Optional[datetime.datetime]
+    event_presentation_end_datetime: Optional[datetime.datetime]
+    event_presenter_code: Optional[str]
+    event_presenter_given_name: Optional[str]
+    event_presenter_family_name: Optional[str]
+    event_presenter_full_name: Optional[str]
+    event_presenter_email: Optional[str]
+    event_session_code: Optional[str]
+    event_session_type_code: Optional[str]
+    event_session_name: Optional[str]
+    event_session_start_datetime: Optional[datetime.datetime]
+    event_session_end_datetime: Optional[datetime.datetime]
+    event_track_code: Optional[str]
+    event_track_name: Optional[str]
+
     # Including other related objects
     hosted_file: Optional[Union[Hosted_File_Base, None]]
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('event_file_id_random', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def event_file_id_random_copy(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        if values['id_random']:
+        'account_id', 'filename_no_ext', 'filename_w_ext',
-            return values['id_random']
+        'hosted_file_hash_sha256', 'hosted_file_subdirectory_path', 
-        return None
+        'hosted_file_content_type', 'hosted_file_size',
-
+        'event_name', 'event_code', 'event_start_datetime', 'event_end_datetime',
-    @validator('id', always=True)
+        'event_location_code', 'event_location_name', 'event_presentation_code',
-    def event_file_id_lookup(cls, v, values, **kwargs):
+        'event_presentation_type_code', 'event_presentation_name',
-        if isinstance(v, int) and v > 0: return v
+        'event_presentation_start_datetime', 'event_presentation_end_datetime',
-        elif id_random := values.get('id_random'):
+        'event_presenter_code', 'event_presenter_given_name', 'event_presenter_family_name',
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_file')
+        'event_presenter_full_name', 'event_presenter_email', 'event_session_code',
-        return None
+        'event_session_type_code', 'event_session_name', 'event_session_start_datetime',
-
+        'event_session_end_datetime', 'event_track_code', 'event_track_name',
-    @validator('hosted_file_id', always=True)
+        'hosted_file'
-    def hosted_file_id_lookup(cls, v, values, **kwargs):
+    ]
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('hosted_file_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='hosted_file')
-        return None
-
-    @validator('event_id', always=True)
-    def event_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
-        return None
-
-    @validator('event_exhibit_id', always=True)
-    def event_exhibit_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_exhibit_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_exhibit')
-        return None
-
-    @validator('event_location_id', always=True)
-    def event_location_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_location_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_location')
-        return None
-
-    @validator('event_presentation_id', always=True)
-    def event_presentation_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_presentation_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_presentation')
-        return None
-
-    @validator('event_presenter_id', always=True)
-    def event_presenter_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_presenter_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_presenter')
-        return None
-
-    @validator('event_session_id', always=True)
-    def event_session_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_session_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_session')
-        return None
-
-    @validator('event_track_id', always=True)
-    def event_track_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_track_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_track')
-        return None
-
-    # NOTE: I kind of give up on this. Handeling this outside of Pydantic and before the data is even attempted to be loaded into the Event_File_Base model. -STI 2021-09-10
-    # NOTE: This validator will try to find and "set" the for_id_random value. However, The value is not really "set" in Pydantic. To get this value, exclude_unset=True when returning a dict from the model.
-    # @validator('for_id_random', always=True)
-    # def for_id_random_lookup(cls, v, values, **kwargs):
-    #     log.setLevel(logging.WARNING)
-    #     log.debug(locals())
-
-    #     if values.get('for_id') and values['for_type']:
-    #         return get_id_random(record_id=values['for_id'], table_name=values['for_type'])
-    #     return None
-
-    # @validator('for_id', always=True)
-    # def for_id_lookup(cls, v, values, **kwargs):
-    #     log.setLevel(logging.DEBUG)
-    #     log.debug(locals())
-
-    #     if values.get('for_id_random', None) and values['for_type']:
-    #         return redis_lookup_id_random(record_id_random=values['for_id_random'], table_name=values['for_type'])
-    #     # return None
-    #     else: return v
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Event File Models ### Event_File_Base() ###

app/models/event_location_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -15,28 +15,28 @@ class Event_Location_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        # **base_fields['event_location_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_location_id_random'])
-        alias = 'event_location_id_random',
+    event_location_id: Optional[str] = Field(None, **base_fields['event_location_id_random'])
-    )
+    
-    id: Optional[int] = Field(
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-        alias = 'event_location_id'
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
-    )
+    event_track_id: Optional[str] = Field(None, **base_fields['event_track_id_random'])
+
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
+    id_random: Optional[str] = Field(None, alias='event_location_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random: Optional[str] = Field(None, exclude=True)
+    event_track_id_random: Optional[str] = Field(None, exclude=True)
 
     code: Optional[str] = Field(
         # alias = 'event_location_code'
     )
 
     external_id: Optional[str] = Field(
-        alias = 'event_location_external_id'
+        # alias = 'event_location_external_id'
     )
 
-    event_id_random: Optional[str]
-    event_id: Optional[int]
-
-    event_track_id_random: Optional[str] # Can a track be assigned to one location?
-    event_track_id: Optional[int] # Can a track be assigned to one location?
-
     lu_location_type_id: Optional[int]
     location_type_code: Optional[str]
     location_type: Optional[str]
@@ -55,8 +55,15 @@ class Event_Location_Base(BaseModel):
     internal_notes_it: Optional[str] # IT and networking
     internal_notes_staff: Optional[str] # staffing and labor
 
+    passcode: Optional[str]
+
+    cfg_json: Optional[Union[Json, None]] # Store per location config options
+    data_json: Optional[Union[Json, None]] # For key value data. Careful with overwriting existing fields!
+
     file_count: Optional[int]
     internal_use_count: Optional[int] # Should be renamed to "internal_use_file_count"???
+    event_file_id_li_json: Optional[Union[Json, None]] # List of file IDs (actually id_random)
+    file_count_all: Optional[int] # Of all files under a location
 
     alert: Optional[bool]
     alert_msg: Optional[str]
@@ -97,29 +104,43 @@ class Event_Location_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
+    @root_validator(pre=True)
-    def event_location_id_lookup(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if isinstance(v, int) and v > 0: return v
+        """
-        elif id_random := values.get('id_random'):
+        Vision Transformer:
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_location')
+        Map DB keys to clean API keys and strip internal integers.
-        return None
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_location_id_random'):
+            values['id'] = rid
+            values['event_location_id'] = rid
+            
+        if a_rid := values.get('account_id_random'):
+            values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'):
+            values['event_id'] = e_rid
+        if et_rid := values.get('event_track_id_random'):
+            values['event_track_id'] = et_rid
+            
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'event_location_id', 'account_id', 'event_id', 'event_track_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+                
+        return values
 
-    @validator('event_id', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def event_id_lookup(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        if isinstance(v, int) and v > 0: return v
+        'account_id',
-        elif id_random := values.get('event_id_random'):
+        'location_type', 'file_count', 'internal_use_count', 'event_file_id_li_json', 'file_count_all',
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
+        'event_name', 'event_start_datetime', 'event_end_datetime',
-        return None
+        'event_abstract_list', 'event_device_list', 'event_file_list',
-
+        'event_file_internal_use_list', 'event_presentation_list', 
-    @validator('event_track_id', always=True)
+        'event_presenter_list', 'event_session_list', 'event_track_list'
-    def event_track_id_lookup(cls, v, values, **kwargs):
+    ]
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_track_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_track')
-        return None
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Event Location Models ### Event_Location_Base() ###

app/models/event_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -22,33 +22,71 @@ class Event_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings for API, Integers for DB) ---
-        **base_fields['event_id_random'],
+    # We use Union[int, str] to allow both public string IDs and resolved DB integers to pass validation.
-        alias = 'event_id_random',
+    id: Optional[Union[int, str]] = Field(None, **base_fields['event_id_random'])
-    )
+    event_id: Optional[Union[int, str]] = Field(None, **base_fields['event_id_random'])
-    id: Optional[int] = Field(
+    account_id: Optional[Union[int, str]] = Field(None, **base_fields['account_id_random'])
-        alias = 'event_id'
+
-    )
+    poc_event_person_id: Optional[Union[int, str]] = Field(None, **base_fields['event_person_id_random'])
+    poc_person_id: Optional[Union[int, str]] = Field(None, **base_fields['person_id_random'])
+    user_id: Optional[Union[int, str]] = Field(None, **base_fields['user_id_random'])
+    address_location_id: Optional[Union[int, str]] = Field(None, **base_fields['address_id_random'])
+
+    contact_1_id: Optional[Union[int, str]] = Field(None, **base_fields['contact_id_random'])
+    contact_2_id: Optional[Union[int, str]] = Field(None, **base_fields['contact_id_random'])
+    contact_3_id: Optional[Union[int, str]] = Field(None, **base_fields['contact_id_random'])
+
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
+    id_random: Optional[str] = Field(None, alias='event_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    poc_event_person_id_random: Optional[str] = Field(None, exclude=True)
+    poc_person_id_random: Optional[str] = Field(None, exclude=True)
+    user_id_random: Optional[str] = Field(None, exclude=True)
+    address_location_id_random: Optional[str] = Field(None, exclude=True)
+    contact_1_id_random: Optional[str] = Field(None, exclude=True)
+    contact_2_id_random: Optional[str] = Field(None, exclude=True)
+    contact_3_id_random: Optional[str] = Field(None, exclude=True)
+
+    @root_validator(pre=True)
+    def map_v3_ids(cls, values):
+        """
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers during READ operations.
+        """
+        # 1. Map Random Strings to Clean Names
+        rid = values.get('id_random') or values.get('event_id_random')
+        if rid and isinstance(rid, str):
+            values['id'] = rid
+            values['event_id'] = rid
+
+        if a_rid := values.get('account_id_random'): values['account_id'] = a_rid
+        if pep_rid := values.get('poc_event_person_id_random'): values['poc_event_person_id'] = pep_rid
+        if pp_rid := values.get('poc_person_id_random'): values['poc_person_id'] = pp_rid
+        if u_rid := values.get('user_id_random'): values['user_id'] = u_rid
+        if al_rid := values.get('address_location_id_random'): values['address_location_id'] = al_rid
+        if c1_rid := values.get('contact_1_id_random'): values['contact_1_id'] = c1_rid
+        if c2_rid := values.get('contact_2_id_random'): values['contact_2_id'] = c2_rid
+        if c3_rid := values.get('contact_3_id_random'): values['contact_3_id'] = c3_rid
+
+        # 2. Prevent "Collision Population" or leakage of integers during API responses
+        # WE MUST NOT DELETE these if they are already integers during a POST operation
+        # as they have been resolved by sanitize_payload.
+        for k in ['id', 'event_id', 'account_id', 'poc_event_person_id', 'poc_person_id', 'user_id', 'address_location_id', 'contact_1_id', 'contact_2_id', 'contact_3_id']:
+            val = values.get(k)
+            if val is not None and not isinstance(val, str):
+                if values.get(f'{k}_random') or (k=='id' and values.get('id_random')):
+                    del values[k]
+
+        return values
 
     code: Optional[str] = Field(
         alias = 'event_code'
     )
 
-    account_id_random: Optional[str]
-    account_id: Optional[int]
-
-    poc_event_person_id_random: Optional[str]
-    poc_event_person_id: Optional[int]
-
-    poc_person_id_random: Optional[str]
-    poc_person_id: Optional[int]
-
     external_person_id: Optional[str] # Person ID generated by external system (should be stable and not change)
 
-    user_id_random: Optional[str]
+    lu_event_type_id: Optional[int] = Field(None, exclude=True)
-    user_id: Optional[int]
-
-    lu_event_type_id: Optional[int]
     #lu_event_type: Optional[str] # Needs to be reviewed
 
     conference: Optional[bool] # Also in Event_Cfg_Base model
@@ -70,8 +108,8 @@ class Event_Base(BaseModel):
 
     recurring: Optional[bool]
     recurring_pattern: Optional[str]
-    recurring_start_time: Optional[datetime.time]
+    recurring_start_time: Optional[str]
-    recurring_end_time: Optional[datetime.time]
+    recurring_end_time: Optional[str]
     recurring_text: Optional[str]
 
     weekday_sunday: Optional[bool]
@@ -82,8 +120,6 @@ class Event_Base(BaseModel):
     weekday_friday: Optional[bool]
     weekday_saturday: Optional[bool]
 
-    address_location_id_random: Optional[str]
-    address_location_id: Optional[int]
     location_address_json: Optional[Union[Json, None]]
     location_text: Optional[str]
 
@@ -101,35 +137,46 @@ class Event_Base(BaseModel):
     physical: Optional[bool] # physical in person event
     virtual: Optional[bool] # virtual remote access event
 
-    contact_1_id_random: Optional[str]
-    contact_1_id: Optional[int]
-    contact_2_id_random: Optional[str]
-    contact_2_id: Optional[int]
-    contact_3_id_random: Optional[str]
-    contact_3_id: Optional[int]
     contact_li_json: Optional[Union[Json, None]] # list of dicts (custom for client); this is SQL FULLTEXT() indexed
 
     attend_url: Optional[str]
+    attend_url_code: Optional[str] # ID, code, nickname
     attend_url_passcode: Optional[str]
     attend_phone: Optional[str]
     attend_phone_passcode: Optional[str]
     attend_text: Optional[str]
-
+    attend_json: Optional[Union[Json, None]]
-    # NOT FINISHED YET
 
     # access_key: Optional[str] # Maybe use in the future?
+    passcode: Optional[str]
 
     file_count: Optional[int]
+    internal_use_count: Optional[int] # Should be renamed to "internal_use_file_count"???
+    event_file_id_li_json: Optional[Union[Json, None]] # List of file IDs (actually id_random)
+    file_count_all: Optional[int] # Of all files under a session
+
+    status: Optional[str]
+    review: Optional[bool]
+    approve: Optional[bool]
+    ready: Optional[bool]
+    ready_on: Optional[datetime.datetime]
+    archive: Optional[bool] # Also in Event_Cfg_Base model
+    archive_on: Optional[datetime.datetime] # Also in Event_Cfg_Base model
+
+    mod_abstracts_json: Optional[Union[Json, None]]
+    mod_badges_json: Optional[Union[Json, None]]
+    mod_exhibits_json: Optional[Union[Json, None]]
+    mod_meetings_json: Optional[Union[Json, None]]
+    mod_pres_mgmt_json: Optional[Union[Json, None]]
+
+    cfg_json: Optional[Union[Json, None]] # Store per event config options; Not currently used 2024-06-11
+    data_json: Optional[Union[Json, None]] # For key value data. Careful with overwriting existing fields! Not currently used 2024-06-11
+    default_qry_str: Optional[str] # Default query string used for searching and filtering events. Updated using SQL triggers and a SQL function
 
     enable: Optional[bool] # Also in Event_Cfg_Base model
     enable_from: Optional[datetime.datetime] = None
     enable_to: Optional[datetime.datetime] = None
 
-    archive: Optional[bool] # Also in Event_Cfg_Base model
-    archive_on: Optional[datetime.datetime] # Also in Event_Cfg_Base model
-
-    cfg_json: Optional[Union[Json, None]]
-
     hide: Optional[bool] # Also in Event_Cfg_Base model
     priority: Optional[bool]
     sort: Optional[int]
@@ -164,95 +211,6 @@ class Event_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('event_id_random', always=True)
-    def event_id_random_copy(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['id_random']:
-            return values['id_random']
-        return None
-
-    @validator('id', always=True)
-    def event_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['id_random']:
-            log.debug(values['id_random'])
-            return redis_lookup_id_random(record_id_random=values['id_random'], table_name='event')
-        return None
-
-    @validator('account_id', always=True)
-    def account_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('account_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='account')
-        return None
-
-    @validator('poc_event_person_id', always=True)
-    def poc_event_person_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['poc_event_person_id_random']:
-            return redis_lookup_id_random(record_id_random=values['poc_event_person_id_random'], table_name='event_person')
-        return None
-
-    @validator('poc_person_id', always=True)
-    def poc_person_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['poc_person_id_random']:
-            return redis_lookup_id_random(record_id_random=values['poc_person_id_random'], table_name='person')
-        return None
-
-    @validator('user_id', always=True)
-    def user_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['user_id_random']:
-            return redis_lookup_id_random(record_id_random=values['user_id_random'], table_name='user')
-        return None
-
-    @validator('address_location_id', always=True)
-    def address_location_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['address_location_id_random']:
-            return redis_lookup_id_random(record_id_random=values['address_location_id_random'], table_name='address')
-        return None
-
-    @validator('contact_1_id', always=True)
-    def contact_1_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['contact_1_id_random']:
-            return redis_lookup_id_random(record_id_random=values['contact_1_id_random'], table_name='contact')
-        return None
-
-    @validator('contact_2_id', always=True)
-    def contact_2_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['contact_2_id_random']:
-            return redis_lookup_id_random(record_id_random=values['contact_2_id_random'], table_name='contact')
-        return None
-
-    @validator('contact_3_id', always=True)
-    def contact_3_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['contact_3_id_random']:
-            return redis_lookup_id_random(record_id_random=values['contact_3_id_random'], table_name='contact')
-        return None
-
     @validator('created_on', always=True)
     def created_on_utc(cls, v, values, **kwargs):
         if isinstance(v, datetime.datetime):
@@ -265,6 +223,23 @@ class Event_Base(BaseModel):
             return v.astimezone(pytz.UTC).isoformat()
         else: return v
 
+    @validator('recurring_start_time', 'recurring_end_time', pre=True, always=True)
+    def time_to_str(cls, v):
+        if isinstance(v, (datetime.time, datetime.timedelta)):
+            return str(v)
+        return v
+
+    # Fields that are part of the model (for reading) but should not be saved to the DB table.
+    # These convenience fields and related objects are joined in the view.
+    fields_to_exclude_from_db: ClassVar[list] = [
+        'file_count', 'internal_use_count', 'event_file_id_li_json', 'file_count_all',
+        'address_location', 'contact_1', 'contact_2', 'contact_3',
+        'event_abstract_list', 'event_cfg', 'event_device_list', 'event_exhibit_list',
+        'event_file_list', 'event_location_list', 'event_person_list',
+        'event_presentation_list', 'event_presenter_list', 'event_session_list',
+        'event_track_list', 'poc_person'
+    ]
+
     class Config:
         underscore_attrs_are_private = True
         allow_population_by_field_name = True
@@ -278,31 +253,68 @@ class Event_Meeting_Flat_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings for API, Integers for DB) ---
-        **base_fields['event_id_random'],
+    id: Optional[Union[int, str]] = Field(None, **base_fields['event_id_random'])
-        alias = 'event_id_random',
+    event_id: Optional[Union[int, str]] = Field(None, **base_fields['event_id_random'])
-    )
+    account_id: Optional[Union[int, str]] = Field(None, **base_fields['account_id_random'])
-    id: Optional[int] = Field(
+
-        alias = 'event_id'
+    poc_event_person_id: Optional[Union[int, str]] = Field(None, **base_fields['event_person_id_random'])
-    )
+    poc_person_id: Optional[Union[int, str]] = Field(None, **base_fields['person_id_random'])
+    user_id: Optional[Union[int, str]] = Field(None, **base_fields['user_id_random'])
+    address_location_id: Optional[Union[int, str]] = Field(None, **base_fields['address_id_random'])
+
+    contact_1_id: Optional[Union[int, str]] = Field(None, **base_fields['contact_id_random'])
+    contact_2_id: Optional[Union[int, str]] = Field(None, **base_fields['contact_id_random'])
+    contact_3_id: Optional[Union[int, str]] = Field(None, **base_fields['contact_id_random'])
+
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
+    id_random: Optional[str] = Field(None, alias='event_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    poc_event_person_id_random: Optional[str] = Field(None, exclude=True)
+    poc_person_id_random: Optional[str] = Field(None, exclude=True)
+    user_id_random: Optional[str] = Field(None, exclude=True)
+    address_location_id_random: Optional[str] = Field(None, exclude=True)
+    contact_1_id_random: Optional[str] = Field(None, exclude=True)
+    contact_2_id_random: Optional[str] = Field(None, exclude=True)
+    contact_3_id_random: Optional[str] = Field(None, exclude=True)
+
+    @root_validator(pre=True)
+    def map_v3_ids(cls, values):
+        """
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers during READ operations.
+        """
+        # 1. Map Random Strings to Clean Names
+        rid = values.get('id_random') or values.get('event_id_random')
+        if rid and isinstance(rid, str):
+            values['id'] = rid
+            values['event_id'] = rid
+
+        if a_rid := values.get('account_id_random'): values['account_id'] = a_rid
+        if pep_rid := values.get('poc_event_person_id_random'): values['poc_event_person_id'] = pep_rid
+        if pp_rid := values.get('poc_person_id_random'): values['poc_person_id'] = pp_rid
+        if u_rid := values.get('user_id_random'): values['user_id'] = u_rid
+        if al_rid := values.get('address_location_id_random'): values['address_location_id'] = al_rid
+        if c1_rid := values.get('contact_1_id_random'): values['contact_1_id'] = c1_rid
+        if c2_rid := values.get('contact_2_id_random'): values['contact_2_id'] = c2_rid
+        if c3_rid := values.get('contact_3_id_random'): values['contact_3_id'] = c3_rid
+
+        # 2. Prevent "Collision Population" or leakage of integers during API responses
+        for k in ['id', 'event_id', 'account_id', 'poc_event_person_id', 'poc_person_id', 'user_id', 'address_location_id', 'contact_1_id', 'contact_2_id', 'contact_3_id']:
+            val = values.get(k)
+            if val is not None and not isinstance(val, str):
+                if values.get(f'{k}_random') or (k=='id' and values.get('id_random')):
+                    del values[k]
+
+        return values
 
     code: Optional[str] = Field(
         alias = 'event_code'
     )
 
-    account_id_random: Optional[str]
+    external_person_id: Optional[str] # Person ID generated by external system (should be stable and not change)
-    account_id: Optional[int]
 
-    # poc_event_person_id_random: Optional[str]
+    lu_event_type_id: Optional[int] = Field(None, exclude=True)
-    # poc_event_person_id: Optional[int]
-
-    # poc_person_id_random: Optional[str]
-    # poc_person_id: Optional[int]
-
-    # user_id_random: Optional[str]
-    # user_id: Optional[int]
-
-    lu_event_type_id: Optional[int]
     #lu_event_type: Optional[str] # Needs to be reviewed
 
     conference: Optional[bool] # Also in Event_Cfg_Base model
@@ -324,8 +336,8 @@ class Event_Meeting_Flat_Base(BaseModel):
 
     recurring: Optional[bool]
     recurring_pattern: Optional[str]
-    recurring_start_time: Optional[datetime.time]
+    recurring_start_time: Optional[str]
-    recurring_end_time: Optional[datetime.time]
+    recurring_end_time: Optional[str]
     recurring_text: Optional[str]
 
     weekday_sunday: Optional[bool]
@@ -336,8 +348,6 @@ class Event_Meeting_Flat_Base(BaseModel):
     weekday_friday: Optional[bool]
     weekday_saturday: Optional[bool]
 
-    address_location_id_random: Optional[str]
-    address_location_id: Optional[int]
     location_address_json: Optional[Union[Json, None]]
     location_text: Optional[str]
 
@@ -355,14 +365,6 @@ class Event_Meeting_Flat_Base(BaseModel):
     physical: Optional[bool] # physical in person event
     virtual: Optional[bool] # virtual remote access event
 
-    external_person_id: Optional[str] # Person ID generated by external system (should be stable and not change)
-
-    contact_1_id_random: Optional[str]
-    contact_1_id: Optional[int]
-    contact_2_id_random: Optional[str]
-    contact_2_id: Optional[int]
-    contact_3_id_random: Optional[str]
-    contact_3_id: Optional[int]
     contact_li_json: Optional[Union[Json, None]] # list of dicts (custom for client); this is SQL FULLTEXT() indexed
 
     attend_url: Optional[str]
@@ -370,20 +372,37 @@ class Event_Meeting_Flat_Base(BaseModel):
     attend_phone: Optional[str]
     attend_phone_passcode: Optional[str]
     attend_text: Optional[str]
-
+    attend_json: Optional[Union[Json, None]]
-    # NOT FINISHED YET
 
     # access_key: Optional[str] # Maybe use in the future?
+    passcode: Optional[str]
 
     file_count: Optional[int]
+    internal_use_count: Optional[int] # Should be renamed to "internal_use_file_count"???
+    event_file_id_li_json: Optional[Union[Json, None]] # List of file IDs (actually id_random)
+    file_count_all: Optional[int] # Of all files under a session
+
+    status: Optional[str]
+    review: Optional[bool]
+    approve: Optional[bool]
+    ready: Optional[bool]
+    ready_on: Optional[datetime.datetime]
+    archive: Optional[bool] # Also in Event_Cfg_Base model
+    archive_on: Optional[datetime.datetime] # Also in Event_Cfg_Base model
+
+    mod_abstracts_json: Optional[Union[Json, None]]
+    mod_badges_json: Optional[Union[Json, None]]
+    mod_exhibits_json: Optional[Union[Json, None]]
+    mod_pres_mgmt_json: Optional[Union[Json, None]]
+
+    cfg_json: Optional[Union[Json, None]] # Store per event config options; Not currently used 2024-06-11
+    data_json: Optional[Union[Json, None]] # For key value data. Careful with overwriting existing fields! Not currently used 2024-06-11
+    default_qry_str: Optional[str] # Default query string used for searching and filtering events. Updated using SQL triggers and a SQL function
 
     enable: Optional[bool] # Also in Event_Cfg_Base model
     enable_from: Optional[datetime.datetime] = None
     enable_to: Optional[datetime.datetime] = None
 
-    archive: Optional[bool] # Also in Event_Cfg_Base model
-    archive_on: Optional[datetime.datetime] # Also in Event_Cfg_Base model
-
     hide: Optional[bool] # Also in Event_Cfg_Base model
     priority: Optional[bool]
     sort: Optional[int]
@@ -393,9 +412,11 @@ class Event_Meeting_Flat_Base(BaseModel):
     created_on: Optional[datetime.datetime] = None
     updated_on: Optional[datetime.datetime] = None
 
-    # Including convenience data
+    # --- IDAA Recovery Meetings: Convenience Data (Flat) ---
-    address_id_random: Optional[str]
+    # These fields are primarily for the flat "Meeting" view used by the IDAA mobile/web apps.
-    address_id: Optional[int]
+    # Note: We prioritize string IDs (id_random) for all external API consumers.
+
+    address_id_random: Optional[str] = Field(None, **base_fields['address_id_random'])
     address_name: Optional[str]
     address_line_1: Optional[str]
     address_line_2: Optional[str]
@@ -407,8 +428,6 @@ class Event_Meeting_Flat_Base(BaseModel):
     address_country_alpha_2_code: Optional[str]
     address_country_name: Optional[str]
 
-    contact_1_id_random: Optional[str]
-    contact_1_id: Optional[int]
     contact_1_name: Optional[str] # Avoid using or use as something different?
     contact_1_full_name: Optional[str] # Yes... it is the same as "name"
     contact_1_email: Optional[str]
@@ -420,8 +439,6 @@ class Event_Meeting_Flat_Base(BaseModel):
     contact_1_phone_other: Optional[str]
     contact_1_other_text: Optional[str]
 
-    contact_2_id_random: Optional[str]
-    contact_2_id: Optional[int]
     contact_2_name: Optional[str] # Avoid using or use as something different?
     contact_2_full_name: Optional[str] # Yes... it is the same as "name"
     contact_2_email: Optional[str]
@@ -435,70 +452,6 @@ class Event_Meeting_Flat_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('event_id_random', always=True)
-    def event_id_random_copy(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['id_random']:
-            return values['id_random']
-        return None
-
-    @validator('id', always=True)
-    def event_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['id_random']:
-            log.debug(values['id_random'])
-            return redis_lookup_id_random(record_id_random=values['id_random'], table_name='event')
-        return None
-
-    # @validator('account_id', always=True)
-    # def account_id_lookup(cls, v, values, **kwargs):
-    #     log.setLevel(logging.WARNING)
-    #     log.debug(locals())
-
-    #     if values['account_id_random']:
-    #         return redis_lookup_id_random(record_id_random=values['account_id_random'], table_name='account')
-    #     return None
-
-    # @validator('address_id', always=True)
-    # def address_id_lookup(cls, v, values, **kwargs):
-    #     log.setLevel(logging.WARNING)
-    #     log.debug(locals())
-
-    #     if values['address_id_random']:
-    #         return redis_lookup_id_random(record_id_random=values['address_id_random'], table_name='address')
-    #     return None
-
-    # @validator('contact_1_id', always=True)
-    # def contact_1_id_lookup(cls, v, values, **kwargs):
-    #     log.setLevel(logging.WARNING)
-    #     log.debug(locals())
-
-    #     if values['contact_1_id_random']:
-    #         return redis_lookup_id_random(record_id_random=values['contact_1_id_random'], table_name='contact')
-    #     return None
-
-    # @validator('contact_2_id', always=True)
-    # def contact_2_id_lookup(cls, v, values, **kwargs):
-    #     log.setLevel(logging.WARNING)
-    #     log.debug(locals())
-
-    #     if values['contact_2_id_random']:
-    #         return redis_lookup_id_random(record_id_random=values['contact_2_id_random'], table_name='contact')
-    #     return None
-
-    # @validator('contact_3_id', always=True)
-    # def contact_3_id_lookup(cls, v, values, **kwargs):
-    #     log.setLevel(logging.WARNING)
-    #     log.debug(locals())
-
-    #     if values['contact_3_id_random']:
-    #         return redis_lookup_id_random(record_id_random=values['contact_3_id_random'], table_name='contact')
-    #     return None
-
     @validator('created_on', always=True)
     def created_on_utc(cls, v, values, **kwargs):
         if isinstance(v, datetime.datetime):
@@ -511,8 +464,22 @@ class Event_Meeting_Flat_Base(BaseModel):
             return v.astimezone(pytz.UTC).isoformat()
         else: return v
 
+    # Fields that are part of the model (for reading) but should not be saved to the DB table.
+    # These convenience fields are joined in the view.
+    fields_to_exclude_from_db: ClassVar[list] = [
+        'address_name', 'address_line_1', 'address_line_2', 'address_line_3', 'address_city',
+        'address_country_subdivision_code', 'address_country_subdivision_name', 'address_postal_code',
+        'address_country_alpha_2_code', 'address_country_name',
+        'contact_1_name', 'contact_1_full_name', 'contact_1_email', 'contact_1_phone_mobile',
+        'contact_1_phone_home', 'contact_1_phone_office', 'contact_1_phone_land', 'contact_1_phone_fax',
+        'contact_1_phone_other', 'contact_1_other_text',
+        'contact_2_name', 'contact_2_full_name', 'contact_2_email', 'contact_2_phone_mobile',
+        'contact_2_phone_home', 'contact_2_phone_office', 'contact_2_phone_land', 'contact_2_phone_fax',
+        'contact_2_phone_other', 'contact_2_other_text'
+    ]
+
     class Config:
         underscore_attrs_are_private = True
         allow_population_by_field_name = True
         fields = base_fields
-# ### END ### API Event Models ### Event_Meeting_Flat_Base() ###
+# ### END ### API Event Models ### Event_Meeting_Flat_Base() ###

app/models/event_person_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -22,40 +22,33 @@ class Event_Person_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        # **base_fields['event_person_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_person_id_random'])
-        alias = 'event_person_id_random',
+    event_person_id: Optional[str] = Field(None, **base_fields['event_person_id_random'])
-    )
-    id: Optional[int] = Field(
-        alias = 'event_person_id'
-    )
 
-    account_id_random: Optional[str]
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-    account_id: Optional[int]
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
 
-    event_id_random: Optional[str]
+    event_badge_id: Optional[str] = Field(None, **base_fields['event_badge_id_random'])
-    event_id: Optional[int]
+    event_badge_vendor_id: Optional[str] = Field(None, **base_fields['event_badge_id_random'])
+    event_badge_vip_id: Optional[str] = Field(None, **base_fields['event_badge_id_random'])
 
-    event_badge_id_random: Optional[str] # Default attendee badge
+    event_person_profile_id: Optional[str] = Field(None, **base_fields['event_person_profile_id_random'])
-    event_badge_id: Optional[int]
+    event_registration_id: Optional[str] = Field(None, **base_fields['event_registration_id_random'])
+    person_id: Optional[str] = Field(None, **base_fields['person_id_random'])
+    user_id: Optional[str] = Field(None, **base_fields['user_id_random'])
 
-    event_badge_vendor_id_random: Optional[str] # Additional vendor badge
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    event_badge_vendor_id: Optional[int]
+    id_random: Optional[str] = Field(None, alias='event_person_id_random', exclude=True)
-
+    account_id_random: Optional[str] = Field(None, exclude=True)
-    event_badge_vip_id_random: Optional[str] # Additional VIP badge
+    event_id_random: Optional[str] = Field(None, exclude=True)
-    event_badge_vip_id: Optional[int]
+    event_badge_id_random: Optional[str] = Field(None, exclude=True)
-
+    event_badge_vendor_id_random: Optional[str] = Field(None, exclude=True)
-    event_person_profile_id_random: Optional[str]
+    event_badge_vip_id_random: Optional[str] = Field(None, exclude=True)
-    event_person_profile_id: Optional[int]
+    event_person_profile_id_random: Optional[str] = Field(None, exclude=True)
-
+    event_registration_id_random: Optional[str] = Field(None, exclude=True)
-    event_registration_id_random: Optional[str]
+    person_id_random: Optional[str] = Field(None, exclude=True)
-    event_registration_id: Optional[int]
+    user_id_random: Optional[str] = Field(None, exclude=True)
-
-    person_id_random: Optional[str]
-    person_id: Optional[int]
-
-    user_id_random: Optional[str]
-    user_id: Optional[int]
 
     external_id: Optional[str] # Generated internally or externally. Needs to be stable. It should not change.
     external_event_id: Optional[str] # Event ID generated by external system. Needs to be stable. It should not change.
@@ -67,12 +60,18 @@ class Event_Person_Base(BaseModel):
     agree_to_tc: Optional[bool] # Agree to terms and conditions
     allow_tracking: Optional[bool] # Allow tracking for lead retrieval and other marketing
 
+    passcode: Optional[str] # Passcode for accessing the event
+
+    cfg_json: Optional[Union[Json, None]] # Store per person config options like theme, language, etc
+    data_json: Optional[Union[Json, None]] # For key value data. Careful with overwriting existing fields!
+
     file_count: Optional[int]
 
     priority: Optional[bool]
     sort: Optional[int]
     group: Optional[str]
     enable: Optional[bool]
+    hide: Optional[bool]
 
     notes: Optional[str]
     created_on: Optional[datetime.datetime] = None
@@ -121,8 +120,9 @@ class Event_Person_Base(BaseModel):
     person_given_name: Optional[str]
     person_middle_name: Optional[str]
     person_family_name: Optional[str]
-    person_display_name: Optional[str]
     person_full_name: Optional[str]
+    person_full_name_override: Optional[str]
+    # person_display_name: Optional[str]
     person_affiliations: Optional[str]
     person_email: Optional[str]
 
@@ -152,81 +152,54 @@ class Event_Person_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('event_person_id_random', always=True)
+    @root_validator(pre=True)
-    def event_person_id_random_copy(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if values['id_random']:
+        """
-            return values['id_random']
+        Vision Transformer:
-        return None
+        Map DB keys to clean API keys and strip internal integers.
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_person_id_random'):
+            values['id'] = rid
+            values['event_person_id'] = rid
+            
+        if a_rid := values.get('account_id_random'): values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'): values['event_id'] = e_rid
+        if b_rid := values.get('event_badge_id_random'): values['event_badge_id'] = b_rid
+        if bv_rid := values.get('event_badge_vendor_id_random'): values['event_badge_vendor_id'] = bv_rid
+        if bvip_rid := values.get('event_badge_vip_id_random'): values['event_badge_vip_id'] = bvip_rid
+        if ep_rid := values.get('event_person_profile_id_random'): values['event_person_profile_id'] = ep_rid
+        if er_rid := values.get('event_registration_id_random'): values['event_registration_id'] = er_rid
+        if p_rid := values.get('person_id_random'): values['person_id'] = p_rid
+        if u_rid := values.get('user_id_random'): values['user_id'] = u_rid
+            
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'event_person_id', 'account_id', 'event_id', 'event_badge_id', 'event_badge_vendor_id', 'event_badge_vip_id', 'event_person_profile_id', 'event_registration_id', 'person_id', 'user_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+                
+        return values
 
-    @validator('id', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def event_person_id_lookup(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        if isinstance(v, int) and v > 0: return v
+        'file_count',
-        elif id_random := values.get('id_random'):
+        'informal_name', 'given_name', 'middle_name', 'family_name', 'full_name_override', 'full_name',
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_person')
+        'affiliations', 'email', 'website_url',
-        return None
+        'event_badge_informal_name', 'event_badge_given_name', 'event_badge_middle_name',
-
+        'event_badge_family_name', 'event_badge_full_name', 'event_badge_full_name_override',
-    @validator('account_id', always=True)
+        'event_badge_affiliations', 'event_badge_email', 'event_badge_city',
-    def account_id_lookup(cls, v, values, **kwargs):
+        'event_badge_state_province', 'event_badge_country_alpha_2_code', 'event_badge_country',
-        if isinstance(v, int) and v > 0: return v
+        'event_person_informal_name', 'event_person_given_name', 'event_person_middle_name',
-        elif id_random := values.get('account_id_random'):
+        'event_person_family_name', 'event_person_name_override', 'event_person_full_name',
-            return redis_lookup_id_random(record_id_random=id_random, table_name='account')
+        'event_person_affiliations', 'event_person_email', 'event_person_extended_json',
-        return None
+        'person_informal_name', 'person_given_name', 'person_middle_name', 'person_family_name',
-
+        'person_full_name', 'person_full_name_override', 'person_affiliations', 'person_email',
-    @validator('event_id', always=True)
+        'user_email', 'user_name', 'user_username',
-    def event_id_lookup(cls, v, values, **kwargs):
+        'event_badge', 'event_badge_vendor', 'event_badge_vip', 'event_exhibit_list',
-        if isinstance(v, int) and v > 0: return v
+        'event_file_list', 'event_location_list', 'event_person_profile',
-        elif id_random := values.get('event_id_random'):
+        'event_presentation_list', 'event_presenter_list', 'event_registration',
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
+        'event_session', 'event_track', 'person', 'user'
-        return None
+    ]
-
-    @validator('event_badge_id', always=True)
-    def event_badge_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_badge_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_badge')
-        return None
-
-    @validator('event_badge_vendor_id', always=True)
-    def event_badge_vendor_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_badge_vendor_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_badge')
-        return None
-
-    @validator('event_badge_vip_id', always=True)
-    def event_badge_vip_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_badge_vip_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_badge')
-        return None
-
-    @validator('event_person_profile_id', always=True)
-    def event_person_profile_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_person_profile_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_person_profile')
-        return None
-
-    @validator('event_registration_id', always=True)
-    def event_registration_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_registration_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_registration')
-        return None
-
-    @validator('person_id', always=True)
-    def person_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('person_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='person')
-        return None
-
-    @validator('user_id', always=True)
-    def user_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('user_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='user')
-        return None
 
     class Config:
         underscore_attrs_are_private = True
@@ -240,17 +213,17 @@ class Event_Person_New_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        **base_fields['event_person_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_person_id_random'])
-        alias = 'event_person_id_random',
+    event_person_id: Optional[str] = Field(None, **base_fields['event_person_id_random'])
-    )
+
-    id: Optional[int] = Field(
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-        alias = 'event_person_id'
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
-    )
+
-    account_id_random: Optional[str]
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    account_id: Optional[int]
+    id_random: Optional[str] = Field(None, alias='event_person_id_random', exclude=True)
-    event_id_random: Optional[str]
+    account_id_random: Optional[str] = Field(None, exclude=True)
-    event_id: Optional[int]
+    event_id_random: Optional[str] = Field(None, exclude=True)
 
     extended_json: Optional[Union[Json, None]]
 
@@ -295,7 +268,8 @@ class Event_Person_New_Base(BaseModel):
     person_middle_name: Optional[str]
     person_family_name: Optional[str]
     person_full_name: Optional[str]
-    person_display_name: Optional[str]
+    person_full_name_override: Optional[str]
+    # person_display_name: Optional[str]
 
     # affiliations: Optional[str] # One or more affiliations with organizations, companies, and other groups
 
@@ -311,32 +285,43 @@ class Event_Person_New_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('event_person_id_random', always=True)
+    @root_validator(pre=True)
-    def event_person_id_random_copy(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if values['id_random']:
+        """
-            return values['id_random']
+        Vision Transformer:
-        return None
+        Map DB keys to clean API keys and strip internal integers.
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_person_id_random'):
+            values['id'] = rid
+            values['event_person_id'] = rid
+            
+        if a_rid := values.get('account_id_random'):
+            values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'):
+            values['event_id'] = e_rid
+            
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'event_person_id', 'account_id', 'event_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+                
+        return values
 
-    @validator('id', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def event_person_id_lookup(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        if isinstance(v, int) and v > 0: return v
+        'informal_name', 'given_name', 'middle_name', 'family_name', 'full_name',
-        elif id_random := values.get('id_random'):
+        'full_name_override', 'affiliations', 'email', 'website_url', 'state_province_name',
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_person')
+        'event_badge_informal_name', 'event_badge_given_name', 'event_badge_middle_name',
-        return None
+        'event_badge_family_name', 'event_badge_full_name', 'event_badge_full_name_override',
-
+        'event_badge_affiliations', 'event_badge_email', 'event_badge_city',
-    @validator('account_id', always=True)
+        'event_badge_state_province', 'event_badge_country_alpha_2_code', 'event_badge_country',
-    def account_id_lookup(cls, v, values, **kwargs):
+        'event_person_informal_name', 'event_person_given_name', 'event_person_middle_name',
-        if isinstance(v, int) and v > 0: return v
+        'event_person_family_name', 'event_person_name_override', 'event_person_full_name',
-        elif id_random := values.get('account_id_random'):
+        'event_person_affiliations', 'event_person_email',
-            return redis_lookup_id_random(record_id_random=id_random, table_name='account')
+        'person_given_name', 'person_middle_name', 'person_family_name',
-        return None
+        'person_full_name', 'person_full_name_override', 'new_password'
-
+    ]
-    @validator('event_id', always=True)
-    def event_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
-        return None
 
     class Config:
         underscore_attrs_are_private = True

app/models/event_person_profile_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -17,28 +17,23 @@ class Event_Person_Profile_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        **base_fields['event_person_profile_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_person_profile_id_random'])
-        alias = 'event_person_profile_id_random',
+    event_person_profile_id: Optional[str] = Field(None, **base_fields['event_person_profile_id_random'])
-    )
-    id: Optional[int] = Field(
-        alias = 'event_person_profile_id'
-    )
 
-    account_id_random: Optional[str]
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-    account_id: Optional[int]
+    contact_id: Optional[str] = Field(None, **base_fields['contact_id_random'])
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
+    event_person_id: Optional[str] = Field(None, **base_fields['event_person_id_random'])
+    organization_id: Optional[str] = Field(None, **base_fields['organization_id_random'])
 
-    contact_id_random: Optional[str]
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    contact_id: Optional[int]
+    id_random: Optional[str] = Field(None, alias='event_person_profile_id_random', exclude=True)
-
+    account_id_random: Optional[str] = Field(None, exclude=True)
-    event_id_random: Optional[str] # Only in view
+    contact_id_random: Optional[str] = Field(None, exclude=True)
-    event_id: Optional[int] # Only in view
+    event_id_random: Optional[str] = Field(None, exclude=True)
-
+    event_person_id_random: Optional[str] = Field(None, exclude=True)
-    event_person_id_random: Optional[str] # Only in view
+    organization_id_random: Optional[str] = Field(None, exclude=True)
-    event_person_id: Optional[int] # Only in view
-
-    organization_id_random: Optional[str]
-    organization_id: Optional[int]
 
     pronouns: Optional[str] # Preferred pronouns
     informal_name: Optional[str]
@@ -65,21 +60,18 @@ class Event_Person_Profile_Base(BaseModel):
     email: Optional[str]
     website_url: Optional[str]
 
-    thumbnail_hosted_file_id: Optional[int]
+    thumbnail_hosted_file_id: Optional[str] = Field(None, **base_fields['hosted_file_id_random'])
-    thumbnail_hosted_file_id_random: Optional[str]
     thumbnail_path: Optional[str]
     thumbnail_bg_color: Optional[str]
 
     # photo_path: Optional[str]
     # photo_bg_color: Optional[str]
 
-    picture_hosted_file_id: Optional[int]
+    picture_hosted_file_id: Optional[str] = Field(None, **base_fields['hosted_file_id_random'])
-    picture_hosted_file_id_random: Optional[str]
     picture_path: Optional[str]
     picture_bg_color: Optional[str]
 
-    about_hosted_file_id: Optional[int]
+    about_hosted_file_id: Optional[str] = Field(None, **base_fields['hosted_file_id_random'])
-    about_hosted_file_id_random: Optional[str]
     about_path: Optional[str]
 
     email_allowed: Optional[bool]
@@ -110,66 +102,37 @@ class Event_Person_Profile_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('event_person_profile_id_random', always=True)
+    @root_validator(pre=True)
-    def event_person_profile_id_random_copy(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if values['id_random']:
+        """
-            return values['id_random']
+        Vision Transformer:
-        return None
+        Map DB keys to clean API keys and strip internal integers.
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_person_profile_id_random'):
+            values['id'] = rid
+            values['event_person_profile_id'] = rid
+            
+        if a_rid := values.get('account_id_random'): values['account_id'] = a_rid
+        if c_rid := values.get('contact_id_random'): values['contact_id'] = c_rid
+        if e_rid := values.get('event_id_random'): values['event_id'] = e_rid
+        if ep_rid := values.get('event_person_id_random'): values['event_person_id'] = ep_rid
+        if o_rid := values.get('organization_id_random'): values['organization_id'] = o_rid
+            
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'event_person_profile_id', 'account_id', 'contact_id', 'event_id', 'event_person_id', 'organization_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+                
+        return values
 
-    @validator('id', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def event_person_profile_id_lookup(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        if isinstance(v, int) and v > 0: return v
+        'account_id', 'event_id', 'event_person_id',
-        elif id_random := values.get('id_random'):
+        'full_name', 'full_name_override', 'display_name',
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_person_profile')
+        'thumbnail_path', 'picture_path', 'about_path',
-        return None
+        'contact', 'event_cfg', 'organization'
-
+    ]
-    @validator('account_id', always=True)
-    def account_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('account_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='account')
-        return None
-
-    @validator('contact_id', always=True)
-    def contact_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('contact_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='contact')
-        return None
-
-    @validator('organization_id', always=True)
-    def organization_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('organization_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='organization')
-        return None
-
-    @validator('thumbnail_hosted_file_id', always=True)
-    def thumbnail_hosted_file_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values.get('thumbnail_hosted_file_id_random', None):
-            return redis_lookup_id_random(record_id_random=values['thumbnail_hosted_file_id_random'], table_name='thumbnail_hosted_file')
-        return None
-
-    @validator('picture_hosted_file_id', always=True)
-    def picture_hosted_file_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values.get('picture_hosted_file_id_random', None):
-            return redis_lookup_id_random(record_id_random=values['picture_hosted_file_id_random'], table_name='picture_hosted_file')
-        return None
-
-    @validator('about_hosted_file_id', always=True)
-    def about_hosted_file_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values.get('about_hosted_file_id_random', None):
-            return redis_lookup_id_random(record_id_random=values['about_hosted_file_id_random'], table_name='about_hosted_file')
-        return None
 
     class Config:
         underscore_attrs_are_private = True

app/models/event_person_tracking_models.py

@@ -2,7 +2,7 @@ from __future__ import annotations
 import datetime, hashlib, logging, os, pytz, redis, secrets
 
 from typing import Dict, List, Optional, Set, Union
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -14,26 +14,69 @@ class Event_Person_Tracking_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings for API, Integers for DB) ---
-        **base_fields['event_person_tracking_id_random'],
+    id: Optional[Union[int, str]] = Field(**base_fields['event_person_tracking_id_random'])
-        alias = 'event_person_tracking_id_random',
+    event_person_tracking_id: Optional[Union[int, str]] = Field(**base_fields['event_person_tracking_id_random'])
-        default_factory = lambda:secrets.token_urlsafe(default_num_bytes),
+    account_id: Optional[Union[int, str]] = Field(**base_fields['account_id_random'])
-    )
+    event_id: Optional[Union[int, str]] = Field(**base_fields['event_id_random'])
-    id: Optional[int] = Field(
+    event_session_id: Optional[Union[int, str]] = Field(**base_fields['event_session_id_random'])
-        alias = 'event_person_tracking_id'
+    event_person_id: Optional[Union[int, str]] = Field(**base_fields['event_person_id_random'])
-    )
 
-    # account_id_random: Optional[str]
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    # account_id: Optional[int]
+    id_random: Optional[str] = Field(None, alias='event_person_tracking_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random: Optional[str] = Field(None, exclude=True)
+    event_session_id_random: Optional[str] = Field(None, exclude=True)
+    event_person_id_random: Optional[str] = Field(None, exclude=True)
 
-    event_id_random: Optional[str]
+    @root_validator(pre=True)
-    event_id: Optional[int]
+    def map_v3_ids(cls, values):
+        """
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers during READ operations.
+        Falls back to Redis/DB lookups if random string IDs are missing from the view.
+        """
+        from app.db_sql import get_id_random
 
-    event_session_id_random: Optional[str]
+        # 1. Map Primary Object ID
-    event_session_id: Optional[int]
+        rid = values.get('id_random') or values.get('event_person_tracking_id_random')
+        if rid and isinstance(rid, str):
+            values['id'] = rid
+            values['event_person_tracking_id'] = rid
+        elif values.get('id') and isinstance(values.get('id'), int):
+            # Fallback for primary ID
+            resolved_rid = get_id_random(values['id'], 'event_person_tracking')
+            if resolved_rid:
+                values['id'] = resolved_rid
+                values['event_person_tracking_id'] = resolved_rid
+                values['id_random'] = resolved_rid
 
-    event_person_id_random: Optional[str]
+        # 2. Map & Resolve Relational IDs
-    event_person_id: Optional[int]
+        id_map = [
+            ('account_id', 'account'),
+            ('event_id', 'event'),
+            ('event_session_id', 'event_session'),
+            ('event_person_id', 'event_person'),
+        ]
+
+        for field, table in id_map:
+            r_val = values.get(f'{field}_random')
+            if r_val and isinstance(r_val, str):
+                values[field] = r_val
+            elif values.get(field) and isinstance(values[field], int):
+                # Fallback: Resolve from Redis/DB if missing from view result
+                resolved_rid = get_id_random(values[field], table)
+                if resolved_rid:
+                    values[field] = resolved_rid
+                    values[f'{field}_random'] = resolved_rid
+
+        # 3. Final Vision Enforcement: Strip internal integers
+        for k in ['id', 'event_person_tracking_id', 'account_id', 'event_id', 'event_session_id', 'event_person_id']:
+            val = values.get(k)
+            if val is not None and not isinstance(val, str):
+                values[k] = None
+                
+        return values
 
     check_in_out: Optional[bool]
     break_in_out: Optional[bool]
@@ -43,15 +86,11 @@ class Event_Person_Tracking_Base(BaseModel):
     in_datetime: Optional[datetime.datetime] # This should generally default to the created datetime and be overridden as needed
     out_datetime: Optional[datetime.datetime] # This should generally default to the updated datetime and be overridden as needed
 
-    # Maybe add minutes or hours?
+    check_in: Optional[bool]
-    # Maybe add timezone?
+    break_out: Optional[bool]
-
+    break_in: Optional[bool]
-
+    check_out: Optional[bool]
-    check_in: Optional[bool] # Does this make sense to use instead?
+    datetime: Optional[datetime.datetime]
-    break_out: Optional[bool] # Does this make sense to use instead?
-    break_in: Optional[bool] # Does this make sense to use instead?
-    check_out: Optional[bool] # Does this make sense to use instead?
-    datetime: Optional[datetime.datetime] # This should generally default to the created datetime and be overridden as needed
 
     enable: Optional[bool]
 
@@ -60,14 +99,6 @@ class Event_Person_Tracking_Base(BaseModel):
     updated_on: Optional[datetime.datetime] = None
 
     # Including convenience data
-    # This is only for convenience. Probably going to keep unless it causes a problem.
-    # full_name: Optional[str] = Field(
-    #     alias = 'event_person_full_name'
-    # )
-    # display_name: Optional[str] = Field(
-    #     alias = 'event_person_display_name'
-    # )
-
     event_person_informal_name: Optional[str]
     event_person_given_name: Optional[str]
     event_person_family_name: Optional[str]
@@ -83,57 +114,10 @@ class Event_Person_Tracking_Base(BaseModel):
     track_name: Optional[str] = Field(
         alias = 'event_track_name'
     )
-    # Maybe add timezone in the future?
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('event_person_tracking_id_random', always=True)
-    def event_person_tracking_id_random_copy(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['id_random']:
-            return values['id_random']
-        return None
-
-    @validator('id', always=True)
-    def event_person_tracking_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values.get('id_random', None):
-            log.debug(values['id_random'])
-            return redis_lookup_id_random(record_id_random=values['id_random'], table_name='event_person_tracking')
-        return None
-
-    @validator('event_id', always=True)
-    def event_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values.get('event_id_random', None):
-            return redis_lookup_id_random(record_id_random=values['event_id_random'], table_name='event')
-        return None
-
-    @validator('event_session_id', always=True)
-    def event_session_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values.get('event_session_id_random', None):
-            return redis_lookup_id_random(record_id_random=values['event_session_id_random'], table_name='event_session')
-        return None
-
-    @validator('event_person_id', always=True)
-    def event_person_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values.get('event_person_id_random', None):
-            return redis_lookup_id_random(record_id_random=values['event_person_id_random'], table_name='event_person')
-        return None
-
     class Config:
         underscore_attrs_are_private = True
         allow_population_by_field_name = True
-        fields = base_fields
+        fields = base_fields

app/models/event_presentation_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -18,36 +18,31 @@ class Event_Presentation_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        **base_fields['event_presentation_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_presentation_id_random'])
-        alias = 'event_presentation_id_random',
+    event_presentation_id: Optional[str] = Field(None, **base_fields['event_presentation_id_random'])
-    )
+    
-    id: Optional[int] = Field(
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-        alias = 'event_presentation_id'
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
-    )
+    event_abstract_id: Optional[str] = Field(None, **base_fields['event_abstract_id_random'])
+    event_location_id: Optional[str] = Field(None, **base_fields['event_location_id_random'])
+    event_session_id: Optional[str] = Field(None, **base_fields['event_session_id_random'])
+    event_track_id: Optional[str] = Field(None, **base_fields['event_track_id_random'])
+
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
+    id_random: Optional[str] = Field(None, alias='event_presentation_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random: Optional[str] = Field(None, exclude=True)
+    event_abstract_id_random: Optional[str] = Field(None, exclude=True)
+    event_location_id_random: Optional[str] = Field(None, exclude=True)
+    event_session_id_random: Optional[str] = Field(None, exclude=True)
+    event_track_id_random: Optional[str] = Field(None, exclude=True)
 
     external_id: Optional[str] = Field(
-        alias = 'event_presentation_external_id'
+        # alias = 'event_presentation_external_id'
     )
 
-    code: Optional[str] = Field(
+    code: Optional[str]
-        # alias = 'event_presentation_code'
-    )
-
-    event_id_random: Optional[str]
-    event_id: Optional[int]
-
-    event_abstract_id_random: Optional[str]
-    event_abstract_id: Optional[int]
-
-    event_location_id_random: Optional[str]
-    event_location_id: Optional[int]
-
-    event_session_id_random: Optional[str]
-    event_session_id: Optional[int]
-
-    event_track_id_random: Optional[str]
-    event_track_id: Optional[int]
 
     poc_event_person: Optional[Event_Person_Base]
     poc_person: Optional[Person_Base]
@@ -55,6 +50,8 @@ class Event_Presentation_Base(BaseModel):
     for_type: Optional[str]
     for_id: Optional[int]
 
+    abstract_code: Optional[str]
+
     # FUTURE: This event_presentation.type_code should override, the type_code of the event_session.type_code.
     type_code: Optional[str] # None, poster (image, video), assume presentation (PPT, Key, PDF, etc)
 
@@ -64,6 +61,8 @@ class Event_Presentation_Base(BaseModel):
     start_datetime: Optional[datetime.datetime]
     end_datetime: Optional[datetime.datetime]
 
+    passcode: Optional[str]
+
     file_count: Optional[int]
 
     enable: Optional[bool]
@@ -109,36 +108,49 @@ class Event_Presentation_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
+    @root_validator(pre=True)
-    def event_presentation_id_lookup(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if isinstance(v, int) and v > 0: return v
+        """
-        elif id_random := values.get('id_random'):
+        Vision Transformer:
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_presentation')
+        Map DB keys to clean API keys and strip internal integers.
-        return None
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_presentation_id_random'):
+            values['id'] = rid
+            values['event_presentation_id'] = rid
+            
+        if a_rid := values.get('account_id_random'):
+            values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'):
+            values['event_id'] = e_rid
+        if ea_rid := values.get('event_abstract_id_random'):
+            values['event_abstract_id'] = ea_rid
+        if el_rid := values.get('event_location_id_random'):
+            values['event_location_id'] = el_rid
+        if es_rid := values.get('event_session_id_random'):
+            values['event_session_id'] = es_rid
+        if et_rid := values.get('event_track_id_random'):
+            values['event_track_id'] = et_rid
+            
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'event_presentation_id', 'account_id', 'event_id', 'event_abstract_id', 'event_location_id', 'event_session_id', 'event_track_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+                
+        return values
 
-    @validator('event_id', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def event_id_lookup(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        if isinstance(v, int) and v > 0: return v
+        'account_id', 'file_count',
-        elif id_random := values.get('event_id_random'):
+        'event_name', 'event_start_datetime', 'event_end_datetime', 'event_location_name',
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
+        'event_session_type_code', 'event_session_name', 'event_session_start_datetime',
-        return None
+        'event_session_end_datetime', 'event_track_name',
-
+        'poc_event_person', 'poc_person', 'event_abstract_list', 'event_file_list',
-    @validator('event_abstract_id', always=True)
+        'event_presenter_list', 'event_session'
-    def event_abstract_id_lookup(cls, v, values, **kwargs):
+    ]
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_abstract_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_abstract')
-        return None
-
-    @validator('event_session_id', always=True)
-    def event_session_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_session_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_session')
-        return None
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Event Presentation Models ### Event_Presentation_Base() ###

app/models/event_presenter_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -19,42 +19,31 @@ class Event_Presenter_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        **base_fields['event_presenter_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_presenter_id_random'])
-        alias = 'event_presenter_id_random',
+    event_presenter_id: Optional[str] = Field(None, **base_fields['event_presenter_id_random'])
-    )
-    id: Optional[int] = Field(
-        alias = 'event_presenter_id'
-    )
 
-    external_id: Optional[str] = Field(
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-        alias = 'event_presenter_external_id'
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
-    )
+    event_person_id: Optional[str] = Field(None, **base_fields['event_person_id_random'])
+    event_presentation_id: Optional[str] = Field(None, **base_fields['event_presentation_id_random'])
+    event_session_id: Optional[str] = Field(None, **base_fields['event_session_id_random'])
+    event_track_id: Optional[str] = Field(None, **base_fields['event_track_id_random'])
+    person_id: Optional[str] = Field(None, **base_fields['person_id_random'])
 
-    code: Optional[str] = Field(
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-        # alias = 'event_presenter_code'
+    id_random: Optional[str] = Field(None, alias='event_presenter_id_random', exclude=True)
-    )
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random: Optional[str] = Field(None, exclude=True)
+    event_person_id_random: Optional[str] = Field(None, exclude=True)
+    event_presentation_id_random: Optional[str] = Field(None, exclude=True)
+    event_session_id_random: Optional[str] = Field(None, exclude=True)
+    event_track_id_random: Optional[str] = Field(None, exclude=True)
+    person_id_random: Optional[str] = Field(None, exclude=True)
 
-    event_id_random: Optional[str]
+    external_id: Optional[str]
-    event_id: Optional[int]
 
-    # event_abstract_id_random: Optional[str]
+    code: Optional[str]
-    # event_abstract_id: Optional[int]
-
-    event_location_id_random: Optional[str]
-    event_location_id: Optional[int]
-
-    event_person_id_random: Optional[str]
-    event_person_id: Optional[int]
-
-    event_presentation_id_random: Optional[str]
-    event_presentation_id: Optional[int]
-
-    event_session_id_random: Optional[str]
-    event_session_id: Optional[int]
-
-    event_track_id_random: Optional[str]
-    event_track_id: Optional[int]
 
     for_type: Optional[str]
     for_id: Optional[int]
@@ -89,15 +78,36 @@ class Event_Presenter_Base(BaseModel):
     email: Optional[str]
     website_url: Optional[str]
 
+    phone_li_json: Optional[Union[Json, None]]
+
+    # For social media in a JSON object format. The Aether standard field names should be used. Examples: url, url_text, icon, etc.
+    social_li_json: Optional[Union[Json, None]]
+
     tagline: Optional[str]
     biography: Optional[str]
 
-    picture_path: Optional[str]
+    picture_path: Optional[str] # Start using image_li_json instead
     picture_bg_color: Optional[str]
 
+    # For image files only in a JSON object format. The Aether standard field names should be used. Examples: url, url_text, alt_text, width, height, size (in bytes), etc.
+    image_li_json: Optional[Union[Json, None]] # "headshot" is probably the most common
+    # media_li_json: Optional[Union[Json, None]]
+
     role: Optional[str]
 
-    file_count: Optional[int]
+    passcode: Optional[str]
+
+    cfg_json: Optional[Union[Json, None]] # Store per presenter config options like theme, language, etc
+    data_json: Optional[Union[Json, None]] # For key value data. Careful with overwriting existing fields!
+
+    file_count: Optional[int] # File count for the presenter
+    event_file_id_li_json: Optional[Union[Json, None]] # List of file IDs (actually id_random)
+
+    # General catchall for agreement or consent
+    agree: Optional[bool]
+
+    # Comments from the presenter. This is for internal use only.
+    comments: Optional[str]
 
     enable: Optional[bool]
     enable_from: Optional[datetime.datetime] = None
@@ -115,6 +125,7 @@ class Event_Presenter_Base(BaseModel):
     notes: Optional[str]
     created_on: Optional[datetime.datetime] = None
     updated_on: Optional[datetime.datetime] = None
+    default_qry_str: Optional[str] # Default query string used for searching and filtering presenters. Updated using SQL triggers and a SQL function
 
     # Including convenience data
     # This is only for convenience. Probably going to keep unless it causes a problem.
@@ -136,6 +147,16 @@ class Event_Presenter_Base(BaseModel):
     event_track_code: Optional[str]
     event_track_name: Optional[str]
 
+    person_external_id: Optional[str]
+    person_external_sys_id: Optional[str]
+    person_given_name: Optional[str]
+    person_family_name: Optional[str]
+    person_professional_title: Optional[str]
+    person_full_name: Optional[str]
+    person_affiliations: Optional[str]
+    person_primary_email: Optional[str]
+    person_passcode: Optional[str]
+
     # Including other related objects
     # event: Optional[Event_Base]
     # event_abstract: Optional[Event_Abstract_Base]
@@ -160,43 +181,181 @@ class Event_Presenter_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
+    @root_validator(pre=True)
-    def event_presenter_id_lookup(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if isinstance(v, int) and v > 0: return v
+        """
-        elif id_random := values.get('id_random'):
+        Vision Transformer:
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_presenter')
+        Map DB keys to clean API keys and strip internal integers.
-        return None
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_presenter_id_random'):
+            values['id'] = rid
+            values['event_presenter_id'] = rid
 
-    @validator('event_id', always=True)
+        if a_rid := values.get('account_id_random'): values['account_id'] = a_rid
-    def event_id_lookup(cls, v, values, **kwargs):
+        if e_rid := values.get('event_id_random'): values['event_id'] = e_rid
-        if isinstance(v, int) and v > 0: return v
+        if ep_rid := values.get('event_person_id_random'): values['event_person_id'] = ep_rid
-        elif id_random := values.get('event_id_random'):
+        if epr_rid := values.get('event_presentation_id_random'): values['event_presentation_id'] = epr_rid
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
+        if es_rid := values.get('event_session_id_random'): values['event_session_id'] = es_rid
-        return None
+        if et_rid := values.get('event_track_id_random'): values['event_track_id'] = et_rid
+        if p_rid := values.get('person_id_random'): values['person_id'] = p_rid
 
-    @validator('event_person_id', always=True)
+        # 2. Prevent "Collision Population"
-    def event_person_id_lookup(cls, v, values, **kwargs):
+        for k in ['id', 'event_presenter_id', 'account_id', 'event_id', 'event_person_id', 'event_presentation_id', 'event_session_id', 'event_track_id', 'person_id']:
-        if isinstance(v, int) and v > 0: return v
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
-        elif id_random := values.get('event_person_id_random'):
+                del values[k]
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_person')
-        return None
 
-    @validator('event_presentation_id', always=True)
+        return values
-    def event_presentation_id_lookup(cls, v, values, **kwargs):
-        if isinstance(v, int) and v > 0: return v
-        elif id_random := values.get('event_presentation_id_random'):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_presentation')
-        return None
 
-    @validator('event_session_id', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def event_session_id_lookup(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        if isinstance(v, int) and v > 0: return v
+        'account_id', 'file_count', 'event_file_id_li_json',
-        elif id_random := values.get('event_session_id_random'):
+        'event_name', 'event_start_datetime', 'event_end_datetime',
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_session')
+        'event_location_code', 'event_location_name',
-        return None
+        'event_presentation_code', 'event_presentation_type_code', 'event_presentation_name',
+        'event_presentation_start_datetime', 'event_presentation_end_datetime',
+        'event_session_code', 'event_session_type_code', 'event_session_name',
+        'event_session_start_datetime', 'event_session_end_datetime',
+        'event_track_code', 'event_track_name',
+        'person_external_id', 'person_external_sys_id', 'person_given_name',
+        'person_family_name', 'person_professional_title', 'person_full_name',
+        'person_affiliations', 'person_primary_email', 'person_passcode',
+        'event_abstract', 'event_abstract_list', 'event_cfg', 'event_file_list',
+        'event_person', 'event_presentation', 'event_session'
+    ]
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
+        fields = base_fields
+# ### END ### API Event Presenter Models ### Event_Presenter_Base() ###
+
+
+
+# ### BEGIN ### API Event Presenter Models ### Event_Presenter_Base() ###
+class Event_Presenter_Out_Base(BaseModel):
+    log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
+    log.debug(locals())
+
+    # --- Standardized Vision IDs (Strings) ---
+    id: Optional[str] = Field(None, **base_fields['event_presenter_id_random'])
+    event_presenter_id: Optional[str] = Field(None, **base_fields['event_presenter_id_random'])
+
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
+    person_id: Optional[str] = Field(None, **base_fields['person_id_random'])
+    event_presentation_id: Optional[str] = Field(None, **base_fields['event_presentation_id_random'])
+    event_session_id: Optional[str] = Field(None, **base_fields['event_session_id_random'])
+
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
+    id_random: Optional[str] = Field(None, alias='event_presenter_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random: Optional[str] = Field(None, exclude=True)
+    person_id_random: Optional[str] = Field(None, exclude=True)
+    event_presentation_id_random: Optional[str] = Field(None, exclude=True)
+    event_session_id_random: Optional[str] = Field(None, exclude=True)
+
+    external_id: Optional[str]
+
+    code: Optional[str]
+
+    pronouns: Optional[str] # Preferred pronouns
+    informal_name: Optional[str] # Informal or nick name they commonly go by
+
+    title_names: Optional[str] # Title for generation, official position, or professional or academic qualification, other honorific, or other name prefix
+    # prefix: Optional[str] # NOTE: Phasing out! Use *title_names* instead.
+    given_name: Optional[str]
+    middle_name: Optional[str]
+    family_name: Optional[str]
+    designations: Optional[str] # Temporary or long-term designations related to family, relationships, person differentiation (Junior/Senior), location, social status, professional qualifications, legal status, or other name suffix
+    # suffix: Optional[str] # NOTE: Phasing out! Use *designations* instead.
+
+    professional_title: Optional[str] # Professional title
+    # title: Optional[str] # NOTE: Phasing out! Use *professional_title* instead.
+
+    # BEGIN # Auto created name variations
+    full_name: Optional[str] # title_names given_name middle_name family_name designations
+    full_name_override: Optional[str] # Override full_name; Actual name shown for presenter
+
+    affiliations: Optional[str] # One or more affiliations with organizations, companies, and other groups
+    # affiliation: Optional[str] # NOTE: Phasing out! Use *affiliations* instead.
+
+    email: Optional[str]
+    website_url: Optional[str]
+
+    # For social media in a JSON object format. The Aether standard field names should be used. Examples: url, url_text, icon, etc.
+    social_li_json: Optional[Union[Json, None]]
+
+    tagline: Optional[str]
+    biography: Optional[str]
+
+    # For image files only in a JSON object format. The Aether standard field names should be used. Examples: url, url_text, alt_text, width, height, size (in bytes), etc.
+    image_li_json: Optional[Union[Json, None]] # "headshot" is probably the most common
+
+    data_json: Optional[Union[Json, None]] # For key value data. Careful with overwriting existing fields!
+    cfg_json: Optional[Union[Json, None]] # Store per presenter config options like theme, language, etc
+
+    file_count: Optional[int]
+
+    # General catchall for agreement or consent
+    agree: Optional[bool]
+
+    # Comments from the presenter. This is for internal use only.
+    comments: Optional[str]
+
+    enable: Optional[bool]
+
+    hide: Optional[bool]
+
+    priority: Optional[bool]
+    sort: Optional[int] # The presenter number if given
+    group: Optional[str]
+
+    notes: Optional[str]
+    created_on: Optional[datetime.datetime] = None
+    updated_on: Optional[datetime.datetime] = None
+    default_qry_str: Optional[str] # Default query string used for searching and filtering presenters. Updated using SQL triggers and a SQL function
+
+    person_external_id: Optional[str]
+    person_external_sys_id: Optional[str]
+    person_given_name: Optional[str]
+    person_family_name: Optional[str]
+    person_professional_title: Optional[str]
+    person_full_name: Optional[str]
+    person_affiliations: Optional[str]
+    person_primary_email: Optional[str]
+    person_passcode: Optional[str]
+
+    # Including other related objects
+
+    _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
+
+    @root_validator(pre=True)
+    def map_v3_ids(cls, values):
+        """
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers.
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_presenter_id_random'):
+            values['id'] = rid
+            values['event_presenter_id'] = rid
+
+        if a_rid := values.get('account_id_random'): values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'): values['event_id'] = e_rid
+        if epr_rid := values.get('event_presentation_id_random'): values['event_presentation_id'] = epr_rid
+        if es_rid := values.get('event_session_id_random'): values['event_session_id'] = es_rid
+        if p_rid := values.get('person_id_random'): values['person_id'] = p_rid
+
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'event_presenter_id', 'account_id', 'event_id', 'event_presentation_id', 'event_session_id', 'person_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+
+        return values
+
+    class Config:
+        underscore_attrs_are_private = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Event Presenter Models ### Event_Presenter_Base() ###

app/models/event_registration_cfg_models.py

@@ -1,8 +1,8 @@
 from __future__ import annotations
 import datetime, hashlib, logging, os, pytz, redis, secrets
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -14,6 +14,41 @@ class Event_Registration_Cfg_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
+    # --- Standardized Vision IDs (Strings) ---
+    id: Optional[str] = Field(None, **base_fields['event_registration_cfg_id_random'])
+    event_registration_cfg_id: Optional[str] = Field(None, **base_fields['event_registration_cfg_id_random'])
+
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
+
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
+    id_random: Optional[str] = Field(None, alias='event_registration_cfg_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random: Optional[str] = Field(None, exclude=True)
+
+    @root_validator(pre=True)
+    def map_v3_ids(cls, values):
+        """
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers.
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_registration_cfg_id_random'):
+            values['id'] = rid
+            values['event_registration_cfg_id'] = rid
+            
+        if a_rid := values.get('account_id_random'):
+            values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'):
+            values['event_id'] = e_rid
+            
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'event_registration_cfg_id', 'account_id', 'event_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+                
+        return values
+
     start_on: Optional[datetime.datetime]
     end_on: Optional[datetime.datetime]
 
@@ -50,6 +85,9 @@ class Event_Registration_Cfg_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
+    fields_to_exclude_from_db: ClassVar[list] = []
+
     class Config:
         underscore_attrs_are_private = True
         allow_population_by_field_name = True

app/models/event_registration_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import *
@@ -15,24 +15,23 @@ class Event_Registration_Base(BaseModel):
     log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        **base_fields['event_registration_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_registration_id_random'])
-        alias = 'event_registration_id_random',
+    event_registration_id: Optional[str] = Field(None, **base_fields['event_registration_id_random'])
-    )
-    id: Optional[int] = Field(
-        alias = 'event_registration_id'
-    )
 
-    account_id_random: Optional[str]
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-    account_id: Optional[int]
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
-    event_id_random: Optional[str]
+    organization_id: Optional[str] = Field(None, **base_fields['organization_id_random'])
-    event_id: Optional[int]
+    contact_id: Optional[str] = Field(None, **base_fields['contact_id_random'])
-    organization_id_random: Optional[str]
+    person_id: Optional[str] = Field(None, **base_fields['person_id_random'])
-    organization_id: Optional[int]
+
-    contact_id_random: Optional[str]
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    contact_id: Optional[int]
+    id_random: Optional[str] = Field(None, alias='event_registration_id_random', exclude=True)
-    person_id_random: Optional[str]
+    account_id_random: Optional[str] = Field(None, exclude=True)
-    person_id: Optional[int]
+    event_id_random: Optional[str] = Field(None, exclude=True)
+    organization_id_random: Optional[str] = Field(None, exclude=True)
+    contact_id_random: Optional[str] = Field(None, exclude=True)
+    person_id_random: Optional[str] = Field(None, exclude=True)
 
     priority: Optional[bool]
     sort: Optional[int]
@@ -48,69 +47,32 @@ class Event_Registration_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('event_registration_id_random', always=True)
+    @root_validator(pre=True)
-    def event_registration_id_random_copy(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        log.setLevel(logging.WARNING)
+        """
-        log.debug(locals())
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers.
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_registration_id_random'):
+            values['id'] = rid
+            values['event_registration_id'] = rid
+            
+        if a_rid := values.get('account_id_random'): values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'): values['event_id'] = e_rid
+        if o_rid := values.get('organization_id_random'): values['organization_id'] = o_rid
+        if c_rid := values.get('contact_id_random'): values['contact_id'] = c_rid
+        if p_rid := values.get('person_id_random'): values['person_id'] = p_rid
+            
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'event_registration_id', 'account_id', 'event_id', 'organization_id', 'contact_id', 'person_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+                
+        return values
 
-        if values['id_random']:
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-            return values['id_random']
+    fields_to_exclude_from_db: ClassVar[list] = ['cfg', 'event_person_list']
-        return None
-
-    @validator('id', always=True)
-    def event_registration_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['id_random']:
-            log.debug(values['id_random'])
-            return redis_lookup_id_random(record_id_random=values['id_random'], table_name='event_registration')
-        return None
-
-    @validator('account_id', always=True)
-    def account_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['account_id_random']:
-            return redis_lookup_id_random(record_id_random=values['account_id_random'], table_name='account')
-        return None
-
-    @validator('event_id', always=True)
-    def event_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['event_id_random']:
-            return redis_lookup_id_random(record_id_random=values['event_id_random'], table_name='event')
-        return None
-
-    @validator('organization_id', always=True)
-    def organization_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['organization_id_random']:
-            return redis_lookup_id_random(record_id_random=values['organization_id_random'], table_name='organization')
-        return None
-
-    @validator('contact_id', always=True)
-    def contact_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['contact_id_random']:
-            return redis_lookup_id_random(record_id_random=values['contact_id_random'], table_name='contact')
-        return None
-
-    @validator('person_id', always=True)
-    def person_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['person_id_random']:
-            return redis_lookup_id_random(record_id_random=values['person_id_random'], table_name='person')
-        return None
 
     class Config:
         underscore_attrs_are_private = True

app/models/event_session_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -19,61 +19,69 @@ class Event_Session_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        # **base_fields['event_session_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_session_id_random'])
-        alias = 'event_session_id_random',
+    event_session_id: Optional[str] = Field(None, **base_fields['event_session_id_random'])
-    )
+
-    id: Optional[int] = Field(
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-        alias = 'event_session_id'
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
-    )
+    event_location_id: Optional[str] = Field(None, **base_fields['event_location_id_random'])
+    event_track_id: Optional[str] = Field(None, **base_fields['event_track_id_random'])
+    poc_event_person_id: Optional[str] = Field(None, **base_fields['event_person_id_random'])
+    poc_person_id: Optional[str] = Field(None, **base_fields['person_id_random'])
+
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
+    id_random: Optional[str] = Field(None, alias='event_session_id_random', exclude=True)
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random: Optional[str] = Field(None, exclude=True)
+    event_location_id_random: Optional[str] = Field(None, exclude=True)
+    event_track_id_random: Optional[str] = Field(None, exclude=True)
+    poc_event_person_id_random: Optional[str] = Field(None, exclude=True)
+    poc_person_id_random: Optional[str] = Field(None, exclude=True)
 
     external_id: Optional[str] = Field(
-        alias = 'event_session_external_id'
+        # alias = 'event_session_external_id'
     )
 
     code: Optional[str] = Field(
         # alias = 'event_session_code'
     )
 
-    event_id_random: Optional[str]
+    # General catchall for agreement or consent
-    event_id: Optional[int]
+    poc_agree: Optional[bool]
 
-    event_location_id_random: Optional[str]
+    poc_kv_json: Optional[Union[Json, None]]
-    event_location_id: Optional[int]
-
-    event_track_id_random: Optional[str]
-    event_track_id: Optional[int]
-
-    poc_event_person_id_random: Optional[str]
-    poc_event_person_id: Optional[int]
-
-    # poc_person_id_random: Optional[str] # Not used or needed?
-    # poc_person_id: Optional[int] # Not used or needed?
 
     # type_id_random: Optional[str] # Not used or needed?
     # type_id: Optional[int] # Not used or needed?
-    type_code: Optional[str] # None, poster (image, video), assume presentation (PPT, Key, PDF, etc)
+    type_code: Optional[str] # From client; max 25 characters for now; This is a bug with MariaDB?
 
     name: Optional[str]
     description: Optional[str]
 
+    proposal_json: Optional[Union[Json, None]] # Is this still used or needed? 2024-09-12
+
     start_datetime: Optional[datetime.datetime]
     end_datetime: Optional[datetime.datetime]
 
-    attend_url: Optional[str]
+    # Need to redo this using a JSON field
-    attend_url_text: Optional[str]
+    # attend_url: Optional[str]
-    attend_url_passcode: Optional[str]
+    # attend_url_text: Optional[str]
-    attend_phone: Optional[str]
+    # attend_url_passcode: Optional[str]
-    attend_phone_passcode: Optional[str]
+    # attend_phone: Optional[str]
-    attend_text: Optional[str]
+    # attend_phone_passcode: Optional[str]
+    # attend_text: Optional[str]
+    attend_json: Optional[Union[Json, None]]
 
-    rehearsal_start_datetime: Optional[datetime.datetime]
+    # Need to redo this using a JSON field
-    rehearsal_end_datetime: Optional[datetime.datetime]
+    # rehearsal_start_datetime: Optional[datetime.datetime]
-    rehearsal_url: Optional[str]
+    # rehearsal_end_datetime: Optional[datetime.datetime]
-    rehearsal_url_passcode: Optional[str]
+    # rehearsal_url: Optional[str]
-    rehearsal_phone: Optional[str]
+    # rehearsal_url_passcode: Optional[str]
-    rehearsal_phone_passcode: Optional[str]
+    # rehearsal_phone: Optional[str]
-    rehearsal_text: Optional[str]
+    # rehearsal_phone_passcode: Optional[str]
+    # rehearsal_text: Optional[str]
+    rehearsal_json: Optional[Union[Json, None]]
 
     image_path: Optional[str] # Not currently in use. For a banner or logo
     # presentation_file_path: Optional[str] # No longer used 2022-09-15
@@ -94,8 +102,12 @@ class Event_Session_Base(BaseModel):
     internal_notes_it: Optional[str] # IT and networking
     internal_notes_staff: Optional[str] # staffing and labor
 
-    file_count: Optional[int]
+    passcode: Optional[str]
+
+    file_count: Optional[int] # Only files directly under the session
     internal_use_count: Optional[int] # Should be renamed to "internal_use_file_count"???
+    event_file_id_li_json: Optional[Union[Json, None]] # List of file IDs (actually id_random)
+    file_count_all: Optional[int] # Of all files under a session
 
     status: Optional[int]
     review: Optional[bool]
@@ -105,6 +117,11 @@ class Event_Session_Base(BaseModel):
     alert: Optional[bool]
     alert_msg: Optional[str]
 
+    # Options: 'colloquium', 'lecture', 'panel', 'poster', 'symposium', 'workshop'
+    # This is mainly reflected in the Launcher.
+    ux_mode: Optional[str]
+    # Other options??? None, poster (image, video), assume presentation (PPT, Key, PDF, etc)
+
     enable: Optional[bool]
     enable_from: Optional[datetime.datetime] = None
     enable_to: Optional[datetime.datetime] = None
@@ -121,6 +138,9 @@ class Event_Session_Base(BaseModel):
     notes: Optional[str]
     created_on: Optional[datetime.datetime] = None
     updated_on: Optional[datetime.datetime] = None
+    default_qry_str: Optional[str] # Default query string used for searching and filtering sessions. Updated using SQL triggers and a SQL function
+    event_presentation_li_qry_str: Optional[str] # Concatenated query string of presentation data for this session (from v_event_session_w_file_count)
+    event_presenter_li_qry_str: Optional[str] # Concatenated query string of presenter data for this session (from v_event_session_w_file_count)
 
     # Including convenience data
     # This is only for convenience. Probably going to keep unless it causes a problem.
@@ -153,42 +173,67 @@ class Event_Session_Base(BaseModel):
     event_presentation_list: Optional[list[Event_Presentation_Base]] # Optional[Event_Presentation_Base]
     event_presenter_list: Optional[list] # Optional[Event_Presenter_Base]
     event_track: Optional[Event_Track_Base]
+
     poc_event_person: Optional[Event_Person_Base] # NOTE: Using thi will probably create an import loop
 
-    proposal_json: Optional[Union[Json, None]]
+    poc_person: Optional[Person_Base]
+    poc_person_external_id: Optional[str]
+    poc_person_given_name: Optional[str]
+    poc_person_family_name: Optional[str]
+    poc_person_full_name: Optional[str]
+    poc_person_primary_email: Optional[str]
+    poc_person_passcode: Optional[str]
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
+    @root_validator(pre=True)
-    def event_session_id_lookup(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if isinstance(v, int) and v > 0: return v
+        """
-        elif id_random := values.get('id_random'):
+        Vision Transformer:
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_session')
+        Map DB keys to clean API keys and strip internal integers.
-        return None
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_session_id_random'):
+            values['id'] = rid
+            values['event_session_id'] = rid
 
-    @validator('event_id', always=True)
+        if a_rid := values.get('account_id_random'):
-    def event_id_lookup(cls, v, values, **kwargs):
+            values['account_id'] = a_rid
-        if isinstance(v, int) and v > 0: return v
+        if e_rid := values.get('event_id_random'):
-        elif id_random := values.get('event_id_random'):
+            values['event_id'] = e_rid
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event')
+        if el_rid := values.get('event_location_id_random'):
-        return None
+            values['event_location_id'] = el_rid
+        if et_rid := values.get('event_track_id_random'):
+            values['event_track_id'] = et_rid
+        if pep_rid := values.get('poc_event_person_id_random'):
+            values['poc_event_person_id'] = pep_rid
+        if pp_rid := values.get('poc_person_id_random'):
+            values['poc_person_id'] = pp_rid
 
-    @validator('event_location_id', always=True)
+        # 2. Prevent "Collision Population"
-    def event_location_id_lookup(cls, v, values, **kwargs):
+        for k in ['id', 'event_session_id', 'account_id', 'event_id', 'event_location_id', 'event_track_id', 'poc_event_person_id', 'poc_person_id']:
-        if isinstance(v, int) and v > 0: return v
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
-        elif id_random := values.get('event_location_id_random'):
+                del values[k]
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_location')
-        return None
 
-    @validator('event_track_id', always=True)
+        return values
-    def event_track_id_lookup(cls, v, values, **kwargs):
+
-        if isinstance(v, int) and v > 0: return v
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-        elif id_random := values.get('event_track_id_random'):
+    fields_to_exclude_from_db: ClassVar[list] = [
-            return redis_lookup_id_random(record_id_random=id_random, table_name='event_track')
+        'account_id',
-        return None
+        'file_count', 'internal_use_count', 'event_file_id_li_json', 'file_count_all',
+        'event_name', 'event_start_datetime', 'event_end_datetime',
+        'event_location_name', 'event_track_name',
+        'event_abstract_list', 'event_badge_list', 'event_device_list',
+        'event_file_list', 'event_file_internal_use_list', 'event_location',
+        'event_location_list', 'event_person_list', 'event_presenter_cat',
+        'event_presentation_list', 'event_presenter_list', 'event_track',
+        'poc_event_person', 'poc_person',
+        'poc_person_external_id', 'poc_person_given_name', 'poc_person_family_name',
+        'poc_person_full_name', 'poc_person_primary_email', 'poc_person_passcode'
+    ]
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Event Session Models ### Event_Session_Base() ###

app/models/event_track_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -15,17 +15,19 @@ class Event_Track_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        **base_fields['event_track_id_random'],
+    id: Optional[str] = Field(None, **base_fields['event_track_id_random'])
-        alias = 'event_track_id_random',
+    event_track_id: Optional[str] = Field(None, **base_fields['event_track_id_random'])
-    )
+
-    id: Optional[int] = Field(
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-        alias = 'event_track_id'
+    event_id: Optional[str] = Field(None, **base_fields['event_id_random'])
-    )
+    event_location_id: Optional[str] = Field(None, **base_fields['event_location_id_random'])
-    event_id_random: Optional[str]
+
-    event_id: Optional[int]
+    # --- Standardized Legacy / Internal IDs (Excluded) ---
-    event_location_id_random: Optional[str] # Can a location be assigned to one track?
+    id_random: Optional[str] = Field(None, alias='event_track_id_random', exclude=True)
-    event_location_id: Optional[int] # Can a location be assigned to one track?
+    account_id_random: Optional[str] = Field(None, exclude=True)
+    event_id_random: Optional[str] = Field(None, exclude=True)
+    event_location_id_random: Optional[str] = Field(None, exclude=True)
 
     lu_track_type_id: Optional[int]
     track_type_code: Optional[str]
@@ -54,6 +56,11 @@ class Event_Track_Base(BaseModel):
     created_on: Optional[datetime.datetime] = None
     updated_on: Optional[datetime.datetime] = None
 
+    # Including convenience data
+    event_name: Optional[str]
+    event_start_datetime: Optional[datetime.datetime]
+    event_end_datetime: Optional[datetime.datetime]
+
     # Including other related objects
     #event: Optional[Event_Base]
     event_abstract_list: Optional[list] # Optional[Event_Abstract_Base]
@@ -66,45 +73,41 @@ class Event_Track_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('event_track_id_random', always=True)
+    @root_validator(pre=True)
-    def event_track_id_random_copy(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        log.setLevel(logging.WARNING)
+        """
-        log.debug(locals())
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers.
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('event_track_id_random'):
+            values['id'] = rid
+            values['event_track_id'] = rid
+            
+        if a_rid := values.get('account_id_random'):
+            values['account_id'] = a_rid
+        if e_rid := values.get('event_id_random'):
+            values['event_id'] = e_rid
+        if el_rid := values.get('event_location_id_random'):
+            values['event_location_id'] = el_rid
+            
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'event_track_id', 'account_id', 'event_id', 'event_location_id']:
+            if k in values and not isinstance(values[k], str) and values[k] is not None:
+                del values[k]
+                
+        return values
 
-        if values['id_random']:
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-            return values['id_random']
+    fields_to_exclude_from_db: ClassVar[list] = [
-        return None
+        'account_id', 'track_type',
-
+        'event_name', 'event_start_datetime', 'event_end_datetime',
-    @validator('id', always=True)
+        'event_abstract_list', 'event_device_list', 'event_file_list',
-    def event_track_id_lookup(cls, v, values, **kwargs):
+        'event_presentation_list', 'event_presenter_list', 'event_session_list', 'event_track_list'
-        log.setLevel(logging.WARNING)
+    ]
-        log.debug(locals())
-
-        if values['id_random']:
-            log.debug(values['id_random'])
-            return redis_lookup_id_random(record_id_random=values['id_random'], table_name='event_track')
-        return None
-
-    @validator('event_id', always=True)
-    def event_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['event_id_random']:
-            return redis_lookup_id_random(record_id_random=values['event_id_random'], table_name='event')
-        return None
-
-    @validator('event_location_id', always=True)
-    def event_location_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['event_location_id_random']:
-            return redis_lookup_id_random(record_id_random=values['event_location_id_random'], table_name='event_location')
-        return None
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Event Track Models ### Event_Track_Base() ###

app/models/hosted_file_link_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -14,22 +14,13 @@ class Hosted_File_Link_Base(BaseModel):
     log.setLevel(logging.WARNING) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    # id_random: Optional[str] = Field(
+    id: Optional[Union[int, str]] = Field(None)
-    #     **base_fields['hosted_file_link_id_random'],
+    account_id: Optional[Union[int, str]] = Field(None, **base_fields['account_id_random'])
-    #     alias = 'hosted_file_link_id_random',
-    # )
-    id: Optional[int] = Field(
-        #alias = 'hosted_file_link_id'
-    )
-    account_id_random: Optional[str]
-    account_id: Optional[int]
 
-    hosted_file_id_random: Optional[str]
+    hosted_file_id: Optional[Union[int, str]] = Field(None, **base_fields['hosted_file_id_random'])
-    hosted_file_id: Optional[int]
 
     link_to_type: Optional[str] # Should this be renamed to "link_to_obj_type" for clarity?
-    link_to_id_random: Optional[str] # Should this be renamed to "link_to_obj_id_random" for clarity?
+    link_to_id: Optional[Union[int, str]] = Field(None) # Random string or integer
-    link_to_id: Optional[int] # Should this be renamed to "link_to_obj_id" for clarity?
 
     # notes: Optional[str]
     created_on: Optional[datetime.datetime] = None
@@ -40,21 +31,33 @@ class Hosted_File_Link_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('account_id', always=True)
+    @root_validator(pre=True)
-    def account_id_lookup(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if isinstance(v, int) and v > 0: return v
+        """
-        elif id_random := values.get('account_id_random'):
+        Vision Transformer:
-            return redis_lookup_id_random(record_id_random=id_random, table_name='account')
+        Map DB keys to clean API keys and strip internal integers.
-        return None
+        """
+        # 1. Map account_id
+        if a_rid := values.get('account_id_random'):
+            if not isinstance(values.get('account_id'), int):
+                values['account_id'] = a_rid
+                
+        # 2. Map hosted_file_id
+        if f_rid := values.get('hosted_file_id_random'):
+            if not isinstance(values.get('hosted_file_id'), int):
+                values['hosted_file_id'] = f_rid
+                
+        # 3. Map link_to_id
+        if l_rid := values.get('link_to_id_random'):
+            if not isinstance(values.get('link_to_id'), int):
+                values['link_to_id'] = l_rid
+                
+        return values
 
-    @validator('link_to_id', always=True)
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-    def link_to_id_lookup(cls, v, values, **kwargs):
+    fields_to_exclude_from_db: ClassVar[list] = [
-        log.setLevel(logging.WARNING)
+        'link_to'
-        log.debug(locals())
+    ]
-
-        if values['link_to_id_random'] and values['link_to_type']:
-            return redis_lookup_id_random(record_id_random=values['link_to_id_random'], table_name=values['link_to_type'])
-        return None
 
     class Config:
         underscore_attrs_are_private = True

app/models/hosted_file_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
 from typing import Dict, List, Optional, Set, Union
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -14,17 +14,10 @@ class Hosted_File_Base(BaseModel):
     log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        **base_fields['hosted_file_id_random'],
+    id: Optional[Union[int, str]] = Field(None, **base_fields['hosted_file_id_random'])
-        alias = 'hosted_file_id_random',
+    hosted_file_id: Optional[Union[int, str]] = Field(None, **base_fields['hosted_file_id_random'])
-        # default_factory = lambda:secrets.token_urlsafe(default_num_bytes),
+    account_id: Optional[Union[int, str]] = Field(None, **base_fields['account_id_random'])
-    )
-    id: Optional[int] = Field(
-        alias = 'hosted_file_id'
-    )
-
-    account_id_random: Optional[str]
-    account_id: Optional[int]
 
     hash_sha256: Optional[str]
     title: Optional[str]
@@ -32,30 +25,25 @@ class Hosted_File_Base(BaseModel):
 
     version: Optional[int]
 
-    directory_path: Optional[str]
+    subdirectory_path: Optional[str] = Field(None, exclude=True) # NOTE: This will frequently only contain numbers, but it still needs to be a string
-    subdirectory_path: Optional[str] # NOTE: This will frequently only contain numbers, but it still needs to be a string
     filename: Optional[str]
+    filename_no_ext: Optional[str]
+    filename_w_ext: Optional[str]
     extension: Optional[str]
     content_type: Optional[str]
     mimetype: Optional[str]
     size: Optional[int] # In bytes
 
-    cloud_storage: Optional[str]
-    owner_user_id: Optional[int]
-    group_user_id: Optional[str]
-
-    package_name: Optional[str]
-
     already_exists: Optional[str] # This will probably only be populated on upload results
     copy_timer: Optional[str] # This will probably only be populated on upload results
     saved: Optional[str] # This will probably only be populated on upload results
 
-    # metadata: Optional[str]
+    enable: Optional[bool]
 
-    # hide: Optional[bool]
+    hide: Optional[bool]
-    # priority: Optional[bool]
+    priority: Optional[bool]
-    # sort: Optional[int]
+    sort: Optional[int]
-    # group: Optional[str]
+    group: Optional[str]
 
     notes: Optional[str]
     created_on: Optional[datetime.datetime] = None
@@ -75,28 +63,32 @@ class Hosted_File_Base(BaseModel):
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    #@validator('hosted_file_id_random', always=True)
+    @root_validator(pre=True)
-    def hosted_file_id_random_copy(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        if values['id_random']:
+        """
-            return values['id_random']
+        Vision Transformer:
-        return None
+        Map DB keys to clean API keys and strip internal integers during READ operations.
-
+        During CREATE (POST) operations, we ensure resolved integers are preserved.
-    @validator('id', always=True)
+        """
-    def hosted_file_id_lookup(cls, v, values, **kwargs):
+        # 1. Map Random Strings to Clean Names
-        if isinstance(v, int) and v > 0: return v
+        rid = values.get('id_random') or values.get('hosted_file_id_random')
-        elif id_random := values.get('id_random'):
+        if rid and isinstance(rid, str):
-            return redis_lookup_id_random(record_id_random=id_random, table_name='hosted_file')
+            values['id'] = rid
-        return None
+            values['hosted_file_id'] = rid
-
+            
-    @validator('account_id', always=True)
+        if a_rid := values.get('account_id_random'): values['account_id'] = a_rid
-    def account_id_lookup(cls, v, values, **kwargs):
+            
-        if isinstance(v, int) and v > 0: return v
+        # 2. Prevent "Collision Population" or leakage of integers during API responses
-        elif id_random := values.get('account_id_random'):
+        for k in ['id', 'hosted_file_id', 'account_id']:
-            return redis_lookup_id_random(record_id_random=id_random, table_name='account')
+            val = values.get(k)
-        return None
+            if val is not None and not isinstance(val, str):
+                if values.get(f'{k}_random') or (k=='id' and values.get('id_random')):
+                    del values[k]
+                
+        return values
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Hosted File Models ### Hosted_File_Base() ###

app/models/journal_entry_models.py

@@ -1,7 +1,7 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
@@ -14,34 +14,84 @@ class Journal_Entry_Base(BaseModel):
     log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings) ---
-        **base_fields['journal_entry_id_random'],
+    id: Optional[str] = Field(None, **base_fields['journal_entry_id_random'])
-        alias = 'journal_entry_id_random',
+    journal_entry_id: Optional[str] = Field(None, **base_fields['journal_entry_id_random'])
-    )
+    journal_id: Optional[str] = Field(None, **base_fields['journal_id_random'])
-    id: Optional[int] = Field(
+    account_id: Optional[str] = Field(None, **base_fields['account_id_random'])
-        alias = 'journal_entry_id'
-    )
-
-    journal_id_random: Optional[str]
-    journal_id: Optional[int]
 
+    external_id: Optional[str] # ID generated by or for external systems (should be stable and not change)
+    import_id: Optional[str] # Used for import purposes to track the source of the data
     code: Optional[str]
 
+    for_type: Optional[str] # 'person', 'user', 'account', etc
+    for_id: Optional[int] = Field(None, exclude=True)
+    for_id_random: Optional[str]
+
     name: Optional[str]
+    short_name: Optional[str]
     summary: Optional[str]
+    outline: Optional[str]
+
     content: Optional[str]
+    content_html: Optional[str]
+    content_json: Optional[Union[Json, None]]
+    content_encrypted: Optional[str]
+
+    history: Optional[str] # Used to store the history of the journal entry content
+    history_encrypted: Optional[str]
+
+    passcode_hash: Optional[str] # Used to store the hash of the passcode for looking up the passcode
+
+    template: Optional[bool] = False # If this is a template entry, it can be used to create new entries based on this template
+
+    type_code: Optional[str] # 'log', 'tracking', 'personal', 'professional', etc
+    topic_code: Optional[str]
+    category_code: Optional[str]
+    # keywords: Optional[str]
+    tags: Optional[str]
+
+    start_datetime: Optional[datetime.datetime]
+    end_datetime: Optional[datetime.datetime]
+    timezone: Optional[str] # = 'UTC' # Default to UTC
+    seconds: Optional[int]
+
+    location: Optional[str]
+    latitude: Optional[float]
+    longitude: Optional[float]
+
+    billable: Optional[bool]
+    bill_to: Optional[str]
+
+    alert: Optional[bool] = False
+    alert_msg: Optional[str] = None
 
     private: Optional[bool] = True
     public: Optional[bool] = False
     personal: Optional[bool] = True
     professional: Optional[bool] = False
 
-    keywords: Optional[str]
+    parent_id: Optional[str] = Field(None, **base_fields['journal_entry_id_random'])
+    # parent_id_random: Optional[str]
 
+    related_entry_id_random: Optional[List[str]]
+    related_entry_id_li: Optional[List[int]] = Field(None, exclude=True)
+
+    due_datetime: Optional[datetime.datetime]
+    due_alert: Optional[bool]
     archive_on: Optional[datetime.datetime]
     archive: Optional[bool]
 
-    data_json: Optional[Json]
+    passcode: Optional[str] # Used to read and write to the journal entry
+    passcode_timeout: Optional[int] # Number of seconds before asking for the passcode again
+    passcode_read: Optional[str] # Used to read the journal entry
+    passcode_read_expire: Optional[int] # Number of seconds to expire the read passcode
+    # passcode_write: Optional[str] # Used to write to the journal entry
+    # passcode_write_expire: Optional[int] # Number of seconds to expire the write passcode
+
+    url_kv_json: Optional[Union[Json, None]]
+    data_json: Optional[Union[Json, None]] # Used to store additional data for the journal
+    meta_json: Optional[Union[Json, None]] # Used to store additional data about the journal entry
 
     enable: Optional[bool]
     hide: Optional[bool]
@@ -52,30 +102,46 @@ class Journal_Entry_Base(BaseModel):
     notes: Optional[str]
     created_on: Optional[datetime.datetime] = None
     updated_on: Optional[datetime.datetime] = None
+    default_qry_str: Optional[str] = None  # Default query string used for searching and filtering journal entries
+
+    # Including other related objects
+    # This is only for convenience. Probably going to keep unless it causes a problem.
+    file_count: Optional[int]
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
+    @root_validator(pre=True)
-    def journal_entry_id_lookup(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        log.setLevel(logging.WARNING)
+        """
-        log.debug(locals())
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers.
+        """
+        # 1. Map Random Strings to Clean Names
+        if rid := values.get('id_random') or values.get('journal_entry_id_random'):
+            values['id'] = rid
+            values['journal_entry_id'] = rid
 
-        if values['id_random']:
+        if j_rid := values.get('journal_id_random'):
-            log.debug(values['id_random'])
+            values['journal_id'] = j_rid
-            return redis_lookup_id_random(record_id_random=values['id_random'], table_name='journal_entry')
-        return None
 
-    @validator('journal_id', always=True)
+        if a_rid := values.get('account_id_random'):
-    def journal_id_lookup(cls, v, values, **kwargs):
+            values['account_id'] = a_rid
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
 
-        if values['journal_id_random']:
+        if p_rid := values.get('parent_id_random'):
-            return redis_lookup_id_random(record_id_random=values['journal_id_random'], table_name='journal')
+            values['parent_id'] = p_rid
-        return None
+
+        # 2. Prevent "Collision Population"
+        for k in ['id', 'journal_entry_id', 'journal_id', 'account_id', 'parent_id']:
+            if k in values and not isinstance(values[k], str):
+                del values[k]
+
+        return values
+
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
+    fields_to_exclude_from_db: ClassVar[list] = ['file_count']
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Journal Entry Models ### Journal_Entry_Base() ###

app/models/journal_models.py

@@ -1,13 +1,14 @@
 import datetime, pytz
 
-from typing import Dict, List, Optional, Set, Union
+from typing import Dict, List, Optional, Set, Union, ClassVar
-from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator
+from pydantic import BaseModel, EmailStr, Field, Json, PrivateAttr, ValidationError, validator, root_validator
 
 from app.db_sql import redis_lookup_id_random
 from app.lib_general import log, logging
 
 from app.models.common_field_schema import base_fields, default_num_bytes
 from app.models.journal_entry_models import Journal_Entry_Base
+# from app.models.person_models import Person_Base
 
 
 # ### BEGIN ### API Journal Models ### Journal_Base() ###
@@ -15,34 +16,86 @@ class Journal_Base(BaseModel):
     log.setLevel(logging.INFO) # DEBUG, INFO, WARNING, ERROR, EXCEPTION, CRITICAL
     log.debug(locals())
 
-    id_random: Optional[str] = Field(
+    # --- Standardized Vision IDs (Strings for API, Integers for DB) ---
-        **base_fields['journal_id_random'],
+    id: Optional[Union[int, str]] = Field(None, **base_fields['journal_id_random'])
-        alias = 'journal_id_random',
+    journal_id: Optional[Union[int, str]] = Field(None, **base_fields['journal_id_random'])
-    )
+    account_id: Optional[Union[int, str]] = Field(None, **base_fields['account_id_random'])
-    id: Optional[int] = Field(
+    person_id: Optional[Union[int, str]] = Field(None, **base_fields['person_id_random'])
-        alias = 'journal_id'
+    user_id: Optional[Union[int, str]] = Field(None, **base_fields['user_id_random'])
-    )
-    account_id_random: Optional[str]
-    account_id: Optional[int]
-
-    person_id_random: Optional[str]
-    person_id: Optional[int]
-
-    user_id_random: Optional[str]
-    user_id: Optional[int]
 
+    external_id: Optional[str] # ID generated by or for external systems (should be stable and not change)
+    import_id: Optional[str] # Used for import purposes to track the source of the data
     code: Optional[str]
 
+    for_type: Optional[str] # 'person', 'user', 'account', etc
+    for_id: Optional[int] = Field(None, exclude=True)
+    for_id_random: Optional[str]
+
+    name: Optional[str]
+    short_name: Optional[str]
+    summary: Optional[str]
+    outline: Optional[str]
+
+    description: Optional[str]
+    description_html: Optional[str]
+    description_json: Optional[Union[Json, None]]
+
+    type_code: Optional[str] # 'log', 'tracking', 'personal', 'professional', etc
+    tags: Optional[str]
+
+    start_datetime: Optional[datetime.datetime]
+    end_datetime: Optional[datetime.datetime]
+    timezone: Optional[str] # = 'UTC' # Default to UTC
+    seconds: Optional[int]
+
+    location: Optional[str]
+    latitude: Optional[float]
+    longitude: Optional[float]
+
+    billable: Optional[bool]
+    bill_to: Optional[str]
+
+    alert: Optional[bool] = False
+    alert_msg: Optional[str] = None
+
+    private: Optional[bool] = True
+    public: Optional[bool] = False
+    personal: Optional[bool] = True
+    professional: Optional[bool] = False
+
+    # Default the Journal Entries to private, public, personal, and professional
     default_private: Optional[bool]
     default_public: Optional[bool]
     default_personal: Optional[bool]
     default_professional: Optional[bool]
 
-    private_passcode: Optional[str]
+    due_datetime: Optional[datetime.datetime]
-    public_passcode: Optional[str]
+    due_alert: Optional[bool]
+    archive_on: Optional[datetime.datetime]
+    archive: Optional[bool]
 
-    name: Optional[str]
+    allow_auth: Optional[bool]
-    summary: Optional[str]
+    auth_key: Optional[str]
+
+    passcode: Optional[str] # Used to read and write to the journal entry
+    passcode_timeout: Optional[int] # Number of seconds before asking for the passcode again
+    # private_passcode: Optional[str]
+    # public_passcode: Optional[str]
+
+    passcode_read: Optional[str] # Used to read the journal
+    passcode_read_expire: Optional[int] # Number of seconds to expire the read passcode
+    # passcode_write: Optional[str] # Used to write to the journal
+    # passcode_write_expire: Optional[int] # Number of seconds to expire the write passcode
+
+    private_passcode: Optional[str] # Used with the passcode to encrypt and decrypt the Journal Entries
+    public_passcode: Optional[str] # Used to allow external people to view a Journal Entry
+
+    sort_by: Optional[str]
+    sort_by_desc: Optional[bool]
+
+    cfg_json: Optional[Union[Json, None]]
+    data_json: Optional[Union[Json, None]] # Used to store additional data for the journal
+    meta_json: Optional[Union[Json, None]] # Used to store additional data for about the journal
 
     enable: Optional[bool]
     hide: Optional[bool]
@@ -55,40 +108,61 @@ class Journal_Base(BaseModel):
     updated_on: Optional[datetime.datetime] = None
 
     # Including other related objects
+    journal_entry_count: Optional[int] # Number of journal entries in the journal
     journal_entry_list: Optional[list[Journal_Entry_Base]] # Journal_Entry_Base()
+    # This is only for convenience. Probably going to keep unless it causes a problem.
+    file_count: Optional[int] # Only files directly under the journal
+    file_count_all: Optional[int] # All files under a journal and entries
+
+    # This person is essentially the owner of the journal
+    # person: Optional[Person_Base]
+    person_external_id: Optional[str]
+    person_given_name: Optional[str] = None
+    person_family_name: Optional[str] = None
+    person_full_name: Optional[str] = None
+    person_primary_email: Optional[str] = None
+    person_passcode: Optional[str] = None
+
+    # person: Optional[Person_Base]
+    # user: Optional[User_Base]
 
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
-    @validator('id', always=True)
+    @root_validator(pre=True)
-    def journal_id_lookup(cls, v, values, **kwargs):
+    def map_v3_ids(cls, values):
-        log.setLevel(logging.WARNING)
+        """
-        log.debug(locals())
+        Vision Transformer:
+        Map DB keys to clean API keys and strip internal integers during READ operations.
+        During CREATE (POST) operations, we ensure resolved integers are preserved.
+        """
+        # 1. Map Random Strings to Clean Names
+        rid = values.get('id_random') or values.get('journal_id_random')
+        if rid and isinstance(rid, str):
+            values['id'] = rid
+            values['journal_id'] = rid
+            
+        if a_rid := values.get('account_id_random'): values['account_id'] = a_rid
+        if p_rid := values.get('person_id_random'): values['person_id'] = p_rid
+        if u_rid := values.get('user_id_random'): values['user_id'] = u_rid
+            
+        # 2. Prevent "Collision Population" or leakage of integers during API responses
+        for k in ['id', 'journal_id', 'account_id', 'person_id', 'user_id']:
+            val = values.get(k)
+            if val is not None and not isinstance(val, str):
+                if values.get(f'{k}_random') or (k=='id' and values.get('id_random')):
+                    del values[k]
+                
+        return values
 
-        if values['id_random']:
+    # Fields that are part of the model (for reading) but should not be saved to the DB table
-            log.debug(values['id_random'])
+    fields_to_exclude_from_db: ClassVar[list] = [
-            return redis_lookup_id_random(record_id_random=values['id_random'], table_name='journal')
+        'person_external_id', 'person_given_name', 'person_family_name', 
-        return None
+        'person_full_name', 'person_primary_email', 'person_passcode',
-
+        'journal_entry_count', 'file_count', 'file_count_all'
-    @validator('person_id', always=True)
+    ]
-    def person_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['person_id_random']:
-            return redis_lookup_id_random(record_id_random=values['person_id_random'], table_name='person')
-        return None
-
-    @validator('user_id', always=True)
-    def user_id_lookup(cls, v, values, **kwargs):
-        log.setLevel(logging.WARNING)
-        log.debug(locals())
-
-        if values['user_id_random']:
-            return redis_lookup_id_random(record_id_random=values['user_id_random'], table_name='user')
-        return None
 
     class Config:
         underscore_attrs_are_private = True
-        allow_population_by_field_name = True
+        allow_population_by_field_name = False
         fields = base_fields
 # ### END ### API Journal Models ### Journal_Base() ###

app/models/lookup_models.py

@@ -0,0 +1,38 @@
+from typing import Optional
+from pydantic import Field
+from .core_object_models import Core_Std_Obj_Base
+
+class Lookup_Base(Core_Std_Obj_Base):
+    """
+    Standardized Baseline for Aether V3 Lookups.
+    Follows the Hierarchical, Identity-Agnostic System.
+    """
+    id_random: Optional[str] = Field(None, description="Public String ID (ID Vision)")
+    account_id: Optional[int] = Field(None, description="Internal Account ID (NULL = Global)")
+    account_id_random: Optional[str] = Field(None, description="Public Account ID")
+    
+    for_type: Optional[str] = Field(None, description="Polymorphic Context Type")
+    for_id: Optional[int] = Field(None, description="Polymorphic Context Internal ID")
+    for_id_random: Optional[str] = Field(None, description="Polymorphic Context Public ID")
+    
+    group: Optional[str] = Field(None, description="Primary Business Key / Cluster Key")
+    name: Optional[str] = Field(None, description="Primary Display Label")
+    description: Optional[str] = Field(None, description="Detailed Explanation")
+    
+    enable: Optional[bool] = Field(True, description="Active status (Shadowing/Negative Overrides)")
+    hide: Optional[bool] = Field(False, description="UI Visibility flag")
+    sort: Optional[int] = Field(0, description="Ordering priority")
+
+class Lu_Country_V3_Base(Lookup_Base):
+    alpha_2_code: Optional[str] = None
+    alpha_3_code: Optional[str] = None
+    numeric_code: Optional[str] = None
+    english_short_name: Optional[str] = None
+
+class Lu_Country_Subdivision_V3_Base(Lookup_Base):
+    country_alpha_2_code: Optional[str] = None
+    code: Optional[str] = None
+
+class Lu_Time_Zone_V3_Base(Lookup_Base):
+    timezone: Optional[str] = None
+    offset: Optional[str] = None

app/models/membership_group_models.py

@@ -42,6 +42,15 @@ class Membership_Group_Base(BaseModel):
 
     expire_in_days: Optional[int]
 
+    enable: Optional[bool]
+    enable_from: Optional[datetime.datetime] = None
+    enable_to: Optional[datetime.datetime] = None
+
+    hide: Optional[bool]
+    priority: Optional[bool]
+    sort: Optional[int]
+    group: Optional[str]
+
     notes: Optional[str]
     created_on: Optional[datetime.datetime] = None
     updated_on: Optional[datetime.datetime] = None
@@ -104,5 +113,6 @@ class Membership_Group_Base(BaseModel):
     class Config:
         underscore_attrs_are_private = True
         fields = base_fields
+        allow_population_by_field_name = True
 
 # Membership_Group_Base.update_forward_refs()

app/models/membership_person_group_models.py

@@ -55,6 +55,12 @@ class Membership_Person_Group_Base(BaseModel):
     flag: Optional[bool]
     flag_message: Optional[str]
 
+    enable: Optional[bool]
+    hide: Optional[bool]
+    priority: Optional[bool]
+    sort: Optional[int]
+    group: Optional[str]
+
     notes: Optional[str]
 
     created_on: Optional[datetime.datetime] = None
@@ -117,5 +123,6 @@ class Membership_Person_Group_Base(BaseModel):
     class Config:
         underscore_attrs_are_private = True
         fields = base_fields
+        allow_population_by_field_name = True
 
 # Membership_Base.update_forward_refs()

app/models/membership_type_models.py

@@ -58,6 +58,7 @@ class Membership_Type_Base(BaseModel):
     enable_from: Optional[datetime.datetime] = None
     enable_to: Optional[datetime.datetime] = None
 
+    hide: Optional[bool]
     priority: Optional[bool]
     sort: Optional[int] = Field(0, ge=0, lt=100) # Essentially the membership level. Should there be a range limit?
     group: Optional[str]
@@ -106,4 +107,9 @@ class Membership_Type_Base(BaseModel):
     #         return redis_lookup_id_random(record_id_random=values['account_id_random'], table_name='account')
     #     return None
 
+    class Config:
+        underscore_attrs_are_private = True
+        fields = base_fields
+        allow_population_by_field_name = True
+
 # Membership_Type_Base.update_forward_refs()

app/models/order_cfg_models.py

@@ -34,6 +34,16 @@ class Order_Cfg_Base(BaseModel):
     stripe_publishable_key: Optional[str] # Publish/Sharable
     stripe_account_id: Optional[str] # Connected Stripe Account ID
 
+    enable: Optional[bool]
+    hide: Optional[bool]
+    priority: Optional[bool]
+    sort: Optional[int]
+    group: Optional[str]
+
+    notes: Optional[str]
+    created_on: Optional[datetime.datetime] = None
+    updated_on: Optional[datetime.datetime] = None
+
     _processed_at: datetime.datetime = PrivateAttr(default_factory=datetime.datetime.now)
 
     @validator('account_id', always=True)
@@ -44,3 +54,8 @@ class Order_Cfg_Base(BaseModel):
         if values['account_id_random']:
             return redis_lookup_id_random(record_id_random=values['account_id_random'], table_name='account')
         return None
+
+    class Config:
+        underscore_attrs_are_private = True
+        fields = base_fields
+        allow_population_by_field_name = True