141 Commits

Author SHA1 Message Date
Scott Idem
6c6de37419 fix: restrict Dozzle to localhost-only binding
Bind Dozzle to 127.0.0.1 to prevent exposure on external/LAN interfaces.
Previously bound to 0.0.0.0, allowing unauthenticated access to container
logs from any network-reachable host.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-29 14:02:48 -04:00
Scott Idem
47fe502dc1 Minor clean up 2026-04-19 15:24:28 -04:00
Scott Idem
a56213569a docs: expand .env.default comments for API and DB tuning settings
Updated AE_API_GUNICORN_WORKERS default from 2 → 4 based on stress
testing (nearly 2x throughput improvement confirmed). Added detailed
comments to Gunicorn, DB pool, and connection tuning settings explaining
what each parameter does, how they interact, and capacity planning math.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 18:55:28 -04:00
Scott Idem
8d1c27471f feat: expose DB pool_size and max_overflow as env vars
Documents AE_DB_POOL_SIZE and AE_DB_POOL_MAX_OVERFLOW in .env.default
with per-replica connection math comment for capacity planning.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 18:06:19 -04:00
Scott Idem
d1ed06a4c4 fix: resolve logrotate permission issues in maintenance container 2026-04-03 17:10:22 -04:00
Scott Idem
3c6b67b149 chore: unify timezone and implement containerized log rotation 2026-04-03 17:06:34 -04:00
Scott Idem
75fc650ba8 docs(cheatsheet): update multi-stack isolation section with full container name var list
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 19:20:52 -04:00
Scott Idem
c136c2e50c chore(env): clean up .env.default and parameterize container names
- Remove 16 dead variables (OSIT_ENV, AE_API_ENV, AE_APP_ENV, AE_FLASK_APP_SRC,
  AE_DB_ROOT_PASSWORD, OSIT_WEB_HTTPS_PORT, 5x DOCKER_AE_*_EXTRA_HOST,
  CONTAINER_AE_API/APP/MARIADB/PMA)
- Add missing vars: AE_NETWORK_NAME, CONTAINER_DOZZLE, AE_DOZZLE_PORT
- Parameterize hardcoded container names in compose: CONTAINER_MARIADB,
  CONTAINER_PMA, CONTAINER_AE_OPS (all with :-default fallbacks)
- Fix AE_DB_EXTERNAL_PORT default: 3306 → 32768 (avoids host MariaDB conflict)
- Reorganize: AE_APP_GATEWAY_PORT moved next to AE_API_GATEWAY_PORT

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 19:02:46 -04:00
Scott Idem
4f15386d93 docs: update CHEATSHEET and README for new build/deploy commands
Replace AE_APP_BUILD_MODE=staging references and old docker compose
build-ui instructions with current build-docker-* and deploy-remote-*
Makefile targets.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 18:05:43 -04:00
Scott Idem
352cca8a27 fix(compose): update BUILD_MODE fallback from staging to dev
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 17:28:10 -04:00
Scott Idem
dbfa9754d9 chore(deploy): add deploy.sh remote script, update Makefile
- deploy.sh: SSH-triggered deploy for prod and test environments on
  srv-nyx (linode.oneskyit.com). Pulls repos, builds ae_app container
  with correct BUILD_MODE, restarts ae_api.
- Makefile: rename build-ui → build-docker-dev/test/prod to match new
  naming convention; add deploy-remote-test and deploy-remote-prod targets
- .env.default: AE_APP_BUILD_MODE staging → dev (from prior session)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 17:18:52 -04:00
Scott Idem
bb437ce5cb chore(env): add .env.default template and track it in .gitignore
The Docker env project had no committed .env template — new contributors
had to reverse-engineer the required variables from the compose files.

Added .env.default with all required variables, secrets replaced with XXXX,
and comments explaining each section. Notable: AE_API_GUNICORN_TIMEOUT is
documented as 900 (needed for long ffmpeg video jobs like clip_video).

Updated .gitignore to whitelist .env.default via !.env.default.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 14:08:11 -04:00
Scott Idem
bd035f8c17 fix(nginx,gunicorn): raise send_timeout and proxy_send_timeout for long-running endpoints
Nginx was closing the client connection after exactly 60 seconds on requests
like clip_video (ffmpeg, 5-40 min) because send_timeout and proxy_send_timeout
both default to 60s. proxy_read_timeout was already 2100s but the other two
timeouts were still at defaults.

With proxy_buffering off, Nginx holds the write path to the client open as soon
as the upstream connection is established. If the upstream sends no data for 60s
(e.g. ffmpeg processing), Nginx treats the idle write path as stalled and closes
the client connection, logging 499 (Client Closed Request).

Fixed: raise proxy_send_timeout and send_timeout to 2100s to match
proxy_read_timeout in the main location block.

Also raised the Gunicorn default timeout from 30s to 120s in gunicorn_conf.py
as a belt-and-suspenders measure (AE_API_GUNICORN_TIMEOUT env var takes precedence).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 14:05:22 -04:00
Scott Idem
6fd6899879 fix(gunicorn): set control_socket to /dev/shm path
Newer gunicorn (post-23.0.0) added _get_control_socket_path() which
calls os.path.isabs() on the value — crashing when it is None.
Point the socket to /dev/shm (already used for worker_tmp_dir) so it
is writable inside the container and satisfies the new gunicorn code.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-24 20:24:12 -04:00
Scott Idem
cd208ef25c Clean up of old stuff 2026-03-24 17:07:19 -04:00
Scott Idem
0d81958bfc Cleaning up old legacy files. Sorry, no more Flask. 2026-03-24 15:51:44 -04:00
Scott Idem
8c9d263afb Disable Gunicorn control_socket in FastAPI configs
Gunicorn 25.1.0+ enables a 'control_socket' by default, which creates
a root-owned 'gunicorn.ctl' file in the chdir directory. When this
directory is a volume mount (as in our dev/test setups), it causes
permission errors during Docker build context gathering.

This change explicitly sets 'control_socket = None' to prevent the
creation of this file.
2026-03-24 15:41:54 -04:00
Scott Idem
facf453991 Added .dockerignore to this file to help with a build issue. 2026-03-24 15:10:29 -04:00
Scott Idem
90a42a68b3 More domains here 2026-03-12 03:15:58 -04:00
Scott Idem
22efb9c832 More sub domains 2026-03-12 03:00:26 -04:00
Scott Idem
1d7200639c fix: moved API healthcheck to docker-compose to override base image defaults and force port 5005. 2026-03-12 02:53:13 -04:00
Scott Idem
f636c021bc docs: updated README and CHEATSHEET with multi-stack and shared network architecture. 2026-03-12 02:28:43 -04:00
Scott Idem
0072a16c25 fix: introduced aether_shared_net to allow isolated stacks to reach shared DB/Redis. 2026-03-12 02:22:35 -04:00
Scott Idem
d80e2aa1ff Adding more server names... 2026-03-12 02:04:42 -04:00
Scott Idem
26e943b066 fix: added AE_NETWORK_NAME variable to isolate networks between stacks. 2026-03-12 01:40:19 -04:00
Scott Idem
ef54720e78 chore: cleaned up Nginx server_name conflicts and fixed API healthcheck port. 2026-03-12 01:22:44 -04:00
Scott Idem
122ae1efc6 fix: added AE_DOZZLE_PORT variable to resolve port conflicts between environments. 2026-03-12 00:40:30 -04:00
Scott Idem
dca75ab990 fix: added CONTAINER_DOZZLE variable to support multiple environments and updated env template. 2026-03-12 00:32:08 -04:00
Scott Idem
7afbc6ffa3 feat: implemented scalable regex for Nginx server_names across App and API. 2026-03-11 23:54:27 -04:00
Scott Idem
4c68cd2ce3 feat: added test-demo and test-api domains to Nginx server_name lists. 2026-03-11 23:50:25 -04:00
Scott Idem
55350ddd7c Add more server names.... 2026-03-11 23:13:45 -04:00
Scott Idem
a7b6112f4d chore: disabled internal SSL/HTTPS in Nginx configs to support host-level SSL termination. 2026-03-11 23:05:51 -04:00
Scott Idem
14173cfc22 feat: Added Makefile for fast stack management and optimized API volume mounts. 2026-03-11 16:15:08 -04:00
Scott Idem
50f4ddf39d chore: Remove mounted API config file — now lives in aether_api_fastapi repo
API config is no longer injected via volume mount. app/config.py in the
aether_api_fastapi repo reads all settings directly from env vars (.env).
Updated README to reflect the new config location.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 18:45:18 -04:00
Scott Idem
1aa4186f4a docs: update README and CHEATSHEET to reflect final correct architecture
- Traffic diagram corrected: both app and api route through ae_web_dev
- Removed outdated 'two options' scaling section for ae_app
- Added port reference table to CHEATSHEET
- Scaling section simplified: change replicas, done, home nginx never changes
- localhost link corrected from 3001 to 8888 (LAN HTTP via ae_web_dev)
2026-03-10 18:24:31 -04:00
Scott Idem
57ec65144d refactor: route ae_app through ae_web_dev like ae_api
Both API and App now scale transparently via Docker DNS round-robin.
Home server nginx points at a single port for each service:
  - workstation:5060 -> ae_web_dev:80 -> ae_api replicas
  - workstation:3001 -> ae_web_dev:80 -> ae_app replicas

ae_app no longer needs host port bindings. AE_APP_REPLICAS scales freely.
HTTPS (443) commented out -- SSL terminates at home server, not internally.
2026-03-10 18:16:06 -04:00
Scott Idem
0ea5373390 docs: clarify traffic architecture and ae_app vs ae_api scaling options 2026-03-10 17:50:55 -04:00
Scott Idem
16f98bc93d fix: use port range for ae_app to support scaling with host port binding
Replace single AE_APP_NODE_PORT with AE_APP_NODE_PORT_RANGE (e.g. 3001-3006).
Docker assigns one port from the range to each replica, enabling external
nginx upstreams to reference individual ports per instance.

ae_api needs no host ports -- it scales via Docker DNS through ae_web_dev.
2026-03-10 17:40:26 -04:00
Scott Idem
055afda9c4 fix: restore ae_app host port mapping, scale=1 required for static port bind 2026-03-10 17:30:32 -04:00
Scott Idem
6d7cd97bd5 fix: remove host port mapping from ae_app service
With scale > 1, multiple replicas cannot bind to the same host port.
ae_app does not need a host port -- nginx routes to it internally via
the Docker service name 'ae_app:3000' with round-robin load balancing.
Removed ports: '${AE_APP_NODE_PORT}:3000' to fix the port conflict.
2026-03-10 17:11:38 -04:00
Scott Idem
83770ffc00 chore: add SvelteKit nginx config, clean up Flask upstream, wire AE_APP_REPLICAS
- Add conf/nginx/site-enabled_aether_app_svelte_node.conf (SvelteKit node upstream)
- Clean up site-enabled_aether_flask_gunicorn.conf: remove dead comments, update
  upstream to app-node:3000
- docker-compose.yml: ae_app scale now uses ${AE_APP_REPLICAS:-1} instead of hardcoded 1
2026-03-10 16:11:46 -04:00
Scott Idem
f51f12755c docs: Simplified setup instructions by removing redundant symlink steps. 2026-03-10 15:29:54 -04:00
Scott Idem
5fd57dc11a chore: Updated ae_api orchestration to use project-internal build context and Dockerfile. 2026-03-10 15:23:13 -04:00
Scott Idem
decba7f7c8 chore: Pruned redundant FastAPI dependencies (watchgod, six, python-dotenv, itsdangerous, baize, async-timeout). 2026-03-10 15:01:38 -04:00
Scott Idem
8abc3b4c7e docs: Update for unified SvelteKit + FastAPI architecture. Added autonomous SvelteKit build process and updated cheatsheet commands. 2026-03-10 13:33:23 -04:00
Scott Idem
b4866c2f23 docs: modernize README and finalize environment synchronization
Updated README.md to reflect V3 architecture, documented the physical database management suite, and finalized synchronization between env.default and active .env configuration.
2026-02-06 13:35:20 -05:00
Scott Idem
49539d52c1 chore(env): synchronize env.default and stabilize API config
Updated env.default with self-documenting comments and all active environment variables. Hardened conf/aether_api_config.py to ensure SMTP and FILES_PATH dictionaries are preserved during refactors. Integrated v3 websocket routes into Nginx template.
2026-02-06 13:15:48 -05:00
Scott Idem
a303b23d54 More comments 2026-01-13 13:59:12 -05:00
Scott Idem
6297df094d Bug fix for the restore script. It now deletes the temp backup directories automatically. 2026-01-13 12:47:08 -05:00
Scott Idem
ea45d99f13 Enhance DB automation: Added conference export, absolute path restore logic, and automated multi-user credential resetting. Updated README and CHEATSHEET. 2026-01-12 20:28:02 -05:00
Scott Idem
f886250ae3 Automation: Final robust DB restore fixes and updated gitignore rules. 2026-01-12 18:33:05 -05:00
Scott Idem
7bd22d1086 Fix: Robust API configuration and password handling. Resolved 502 Bad Gateway. 2026-01-12 18:20:01 -05:00
Scott Idem
5044a4fc5b Ops: Added backup script, dashboard UI, and project cheatsheet. 2026-01-12 17:35:44 -05:00
Scott Idem
6bad495dce Environment: Cleaned up and standardized env.default template. Removed legacy colored variables. 2026-01-12 17:19:33 -05:00
Scott Idem
7bce390e5f Architecture: Finalized dynamic scaling and Nginx gateway port configuration. 2026-01-12 17:10:16 -05:00
Scott Idem
6ee6f24c00 Environment: Renamed .env.default to env.default for better visibility and updated it with new MariaDB and path parameters. 2026-01-12 16:50:48 -05:00
Scott Idem
00092d2058 MariaDB: Parameterized performance settings via .env and docker-compose command flags. 2026-01-12 16:47:12 -05:00
Scott Idem
0b4c13c84b MariaDB: Applied optimized production server.cnf and enabled audit logging. 2026-01-12 16:40:45 -05:00
Scott Idem
5a2316537c Modernization: Standardized Docker env, added local MariaDB/phpMyAdmin, added automated restore script. 2026-01-12 16:28:43 -05:00
Scott Idem
129cb84254 Snapshot: Pre-cleanup state. Tracking legacy symlinks. 2026-01-12 15:28:42 -05:00
Scott Idem
d5153cda76 Cleanup: Remove commented-out services (Nextcloud, PHP7, PMA) from docker-compose.yml 2026-01-06 19:14:25 -05:00
Scott Idem
9e291ba528 Updates to get FastAPI reasonably up to date. 2024-11-20 15:26:36 -05:00
Scott Idem
97ddb96829 Minor text changes 2024-10-09 14:42:42 -04:00
Scott Idem
2911007b1f Comment out extra servers 2024-10-09 14:00:32 -04:00
Scott Idem
f2798551b2 Updates to make things more efficient. 2024-10-09 13:47:52 -04:00
Scott Idem
6604584556 Ignore files update. 2024-10-09 12:24:59 -04:00
Scott Idem
5f9b7fab6d Ignore update 2024-10-09 11:27:48 -04:00
Scott Idem
485e51488d Merge remote-tracking branch 'refs/remotes/origin/dev_cluster' into dev_cluster
Trying to merge changes...
2024-09-06 18:46:51 -04:00
Scott Idem
991ae88a86 Update related to ASCM dev 2024-09-06 18:45:51 -04:00
Scott Idem
c795bb00f4 More documentation 2024-05-30 23:28:40 -04:00
Scott Idem
a8f62b8022 Minor change 2024-05-30 22:42:31 -04:00
Scott Idem
4c458f378a Finally got multiple server names working for Docker in nginx!!! 2024-05-30 22:40:32 -04:00
Scott Idem
294716fefc Enabled poppler-utils for Python pdf2image to work 2024-05-22 19:22:20 -04:00
Scott Idem
341f5ccbee Everything seems to be working with FastAPI 110 now!! Need to work on SQLAlchemy upgrade to 2.0.29 soon. 2024-04-26 17:58:17 -04:00
Scott Idem
f43f13b9e1 Minor changes 2024-04-25 16:51:26 -04:00
Scott Idem
68150f857b Ignore more 2024-04-24 20:42:48 -04:00
Scott Idem
2ccb3486e9 Don't store the DB 2024-04-24 20:38:11 -04:00
Scott Idem
43c4dffa9a Tweaking settings... 2024-04-24 16:50:09 -04:00
Scott Idem
f9b5186afc Trial and error fixes... 2024-04-24 16:04:36 -04:00
Scott Idem
1d279a2644 Minor update before trying live 2024-04-24 12:08:57 -04:00
Scott Idem
c983ab610d Merge remote-tracking branch 'refs/remotes/origin/dev_cluster' into dev_cluster
Merging?
2024-04-24 12:06:17 -04:00
Scott Idem
60428abb33 Merging changes 2024-04-24 11:58:57 -04:00
Scott Idem
fc04c04044 Tweaks to get API to stop hanging? 2024-04-24 11:56:48 -04:00
9854ba2479 Updated readme 2024-04-07 10:37:27 -04:00
Scott Idem
648224add7 Enabling "http2 on" for all the sites. 2024-03-30 18:54:41 -04:00
Scott Idem
653f55b64d Checking if this works on Linode 2024-03-30 17:31:16 -04:00
Scott Idem
acf953439f Tweaking things... 2024-03-30 14:15:44 -04:00
Scott Idem
91eac68ee1 hosts... 2024-03-29 20:22:31 -04:00
Scott Idem
3ddc6cc5f6 Stuff... 2024-03-29 19:21:27 -04:00
Scott Idem
96fbca2b2c I don't know 2024-03-29 19:08:16 -04:00
Scott Idem
74dc0e1a7e No more white API container 2024-03-29 18:31:52 -04:00
Scott Idem
adb490948a No more white API 2024-03-29 18:30:30 -04:00
Scott Idem
cb584b0734 Now with HTTP2 enabled correctly everywhere. 2024-03-29 17:08:38 -04:00
Scott Idem
90ecc8206a General clean up 2024-03-29 16:46:41 -04:00
Scott Idem
8a4434d0b8 Now with more colors! 2024-03-29 14:51:06 -04:00
Scott Idem
ee4e68ddcb Changing load balancing 2024-03-08 09:10:50 -05:00
Scott Idem
61c49b7f6c Trying to get things to work better. Why are the SQL results getting mixed...??? 2024-03-08 00:39:19 -05:00
Scott Idem
c9bf237db9 Changes 2024-03-07 22:23:36 -05:00
Scott Idem
3e72c4299c Hopefully this works.... 2024-03-07 22:08:15 -05:00
Scott Idem
f1a36b2bc4 Working on using a cluster of API servers.... 2024-03-07 21:33:10 -05:00
Scott Idem
d265064cf6 Clean up before wrapping up for the night. Seems to be working well. 2024-02-14 20:37:18 -05:00
Scott Idem
56da44234a Trying again 2024-02-14 20:32:52 -05:00
Scott Idem
4d50418238 Making things easier to configure 2024-02-14 20:27:17 -05:00
Scott Idem
b87fd7f200 Ooops 2024-02-14 18:54:15 -05:00
Scott Idem
f171d314f2 Still trying to fix a Docker compose bug? 2024-02-14 18:53:15 -05:00
Scott Idem
2ac5a39706 Trying to fix a Docker compose bug? 2024-02-14 18:30:01 -05:00
Scott Idem
9fe60c418d Updated the cert filenames and paths 2024-02-13 17:42:20 -05:00
Scott Idem
3a14925540 General updates. Enabled CHOW. 2024-02-13 16:32:38 -05:00
Scott Idem
3c7fb0afdb Adding in ImageMagick and ffmpeg related tools 2023-11-10 17:29:03 -05:00
Scott Idem
c897f4b439 Getting rid of possible host names in Nginx conf 2023-10-20 14:05:03 -04:00
Scott Idem
040fdfe2ae General config clean up. 2023-10-19 19:15:14 -04:00
Scott Idem
a4f578b400 General config clean up. 2023-10-19 19:04:16 -04:00
Scott Idem
b17420e584 General config clean up. 2023-10-19 18:58:11 -04:00
Scott Idem
14e046b77c Work to get Flask updated. General config clean up. 2023-10-19 12:05:21 -04:00
Scott Idem
92baaccb48 Clean up and upgrades related to Python requirements, Flask, and FastAPI. 2023-09-12 15:41:09 -04:00
Scott Idem
42b6e8ed08 Updating Python requirements file 2023-09-08 17:08:45 -04:00
Scott Idem
8bf5b49552 Now with more .env vars! 2023-09-08 15:24:48 -04:00
Scott Idem
e8e7fc2383 More work on configs and env 2023-09-08 14:45:02 -04:00
Scott Idem
9262e9987e More work on configs and env 2023-09-08 14:23:08 -04:00
Scott Idem
be1bb21b7e Better configs with environment vars 2023-09-08 14:01:00 -04:00
Scott Idem
cccf9fd24f Updates to make Docker Compose better 2023-09-08 13:13:01 -04:00
Scott Idem
7679a62cf1 General improvements. 2023-07-12 15:39:45 -04:00
Scott Idem
630bd1a61e Now with variable key and max body size 2023-06-13 17:50:31 -04:00
Scott Idem
3a5024a2d0 Now with variable names for containers 2023-06-13 17:26:00 -04:00
Scott Idem
37c346efd7 Less configs to worry about 2023-06-13 17:11:35 -04:00
Scott Idem
37af3019c6 One less file to change hopefully 2023-06-13 16:25:08 -04:00
Scott Idem
5b65bd5a1f More clean up of Docker config related files 2023-06-13 16:09:42 -04:00
Scott Idem
8969dfb33a Clean up to go live 2023-06-13 15:20:01 -04:00
Scott Idem
2d545a2a66 Work towards new API version 5 2023-06-07 19:29:54 -04:00
cc54f95cd8 Minor update to documentation 2023-05-17 18:35:43 -04:00
34f1ab89c2 Updates from Asus and BGH event. 2023-04-28 15:50:14 -04:00
3e6c8db324 General clean up 2023-04-16 22:54:27 -04:00
Scott Idem
7bdfe41aa1 Unknown 2023-04-16 19:45:11 -04:00
Scott Idem
70db400290 General updates and clean up. 2023-04-04 20:12:06 -04:00
Scott Idem
26c8b5121b The Docker Compose seems to be working correctly now... 2023-02-24 19:04:43 -05:00
Scott Idem
6cc7f33f6f Still working on Docker Compose 2023-02-24 18:09:00 -05:00
Scott Idem
f85a97001c Still working on Docker Compose 2023-02-24 17:02:44 -05:00
Scott Idem
733db642eb Still working on Docker Compose 2023-02-24 16:51:05 -05:00
Scott Idem
e7a2f2313c Still working on Docker Compose 2023-02-24 16:47:04 -05:00
Scott Idem
f6a00b0145 Still working on Docker Compose 2023-02-24 16:46:58 -05:00
Scott Idem
56fb54a38e Work on nginx and server names 2023-02-24 14:21:54 -05:00
63 changed files with 1930 additions and 1996 deletions

23
.dockerignore Normal file
View File

@@ -0,0 +1,23 @@
# Ignore Git
.git
.gitignore
# Ignore Data Directories (CRITICAL for build speed and permissions)
srv/
logs/
temp/
tmp/
backups/
# Ignore Environment Files
.env
.env.*
*.env
# Ignore IDE settings
.vscode/
*.code-workspace
# Ignore miscellaneous
README.md
gunicorn.ctl

View File

@@ -1,63 +1,201 @@
# One Sky IT's Aether Framework and System
# ------------------------------------------------------------------------------
# AETHER FRAMEWORK - DOCKER ENVIRONMENT CONFIGURATION TEMPLATE
# ------------------------------------------------------------------------------
# Copy this file to .env and fill in real values.
# .env is gitignored — never commit the live file.
# ------------------------------------------------------------------------------
OSIT_ENV=development
# OSIT_ENV=production
# OSIT_ENV=testing
# ------------------------------------------------------------------------------
# SYSTEM SETTINGS
# ------------------------------------------------------------------------------
# System timezone for all containers
TZ=US/Eastern
OSIT_WEB_HTTP_PORT=8080
OSIT_WEB_HTTPS_PORT=4443
# Logging level for the API and background workers (debug, info, warning, error)
AE_LOG_LVL=debug
# For now this extra host variable is important for the AE Flask app to connect to the AE FastAPI API.
DOCKER_AE_APP_EXTRA_HOST=dev-api.oneskyit.com:192.168.32.20
# Docker Compose Profiles
# 'database' includes: mariadb, phpmyadmin, ae_ops
# Comment out or leave empty for "app-only" nodes that connect to a remote DB
# COMPOSE_PROFILES=database
# Aether general shared config options
# For general shared config options like API access and use, database access and use, Redis, and SMTP
# home development, live testing, live production, onsite development, onsite testing, onsite production???
AE_CFG_ID=0
# ------------------------------------------------------------------------------
# STACK ISOLATION
# ------------------------------------------------------------------------------
# Unique Docker network name per stack (prevents collisions when running test/prod on same host)
AE_NETWORK_NAME=ae_dev_net
## Aether API access and use
AE_API_PROTOCOL=https
AE_API_SERVER=dev-api.oneskyit.com
AE_API_PORT=443
AE_API_PATH=
AE_API_SECRET_KEY=the-secret-api-key
# Internal Docker container names (must be unique per stack)
# Note: ae_api and ae_app are scaled services — Docker does not allow container_name on those.
CONTAINER_WEB=ae_web_dev
CONTAINER_REDIS=ae_redis_dev
CONTAINER_DOZZLE=ae_dozzle_dev
CONTAINER_MARIADB=ae_mariadb_dev
CONTAINER_PMA=ae_pma_dev
CONTAINER_AE_OPS=ae_ops_dev
## Aether DB access and use
AE_DB_SERVER=linode.oneskyit.com
# ------------------------------------------------------------------------------
# NETWORK & PROXY SETTINGS
# ------------------------------------------------------------------------------
# Local Nginx listener ports on the host system
OSIT_WEB_HTTP_PORT=8888
# Maximum allowed file upload size (Global for Nginx)
OSIT_WEB_MAX_BODY_SIZE=5120M
# Gateway ports for external reverse proxy (Home Server → this node → Docker Nginx)
AE_APP_GATEWAY_PORT=3001
AE_API_GATEWAY_PORT=5060
# Dozzle log viewer port
AE_DOZZLE_PORT=8881
# Nginx Server Names (used in vhost config templates)
DOCKER_AE_API_SERVER_NAME=dev-api.oneskyit.com
DOCKER_AE_APP_SERVER_NAME=dev-example.oneskyit.com
DOCKER_PHPMYADMIN_SERVER_NAME=dev-phpmyadmin.oneskyit.com
DOCKER_OSIT_SERVER_NAME=dev.oneskyit.com
# ------------------------------------------------------------------------------
# DATABASE SETTINGS (MariaDB)
# ------------------------------------------------------------------------------
# To use an EXTERNAL database:
# 1. Set COMPOSE_PROFILES= (empty) above to disable local DB containers.
# 2. Set AE_DB_SERVER to the external IP or hostname.
# 3. Ensure the external DB allows connections from this host's IP.
# DB Hostname (use 'mariadb' for the local container, or a remote IP/FQDN)
AE_DB_SERVER=vpn-db.oneskyit.com
# AE_DB_SERVER=mariadb
AE_DB_PORT=3306
AE_DB_NAME=aether_dev
# AE_DB_USERNAME=osit_aether
AE_DB_PASSWORD="the password with $$ escape"
## Aether Redis access and use
# Port to expose on the host system if running a local MariaDB container
AE_DB_EXTERNAL_PORT=32768
# Database credentials
AE_DB_NAME=aether_dev
AE_DB_USERNAME=aether_dev
AE_DB_PASSWORD=XXXX
# Connection Tuning
# Seconds to wait when establishing a new connection before giving up.
# Lower values fail fast on DB outage rather than hanging requests.
AE_DB_CONNECTION_TIMEOUT=7
# Seconds before a pooled connection is recycled (closed and reopened).
# Prevents "MySQL server has gone away" errors from MariaDB's wait_timeout.
# Must be less than MariaDB's wait_timeout (default 28800s / 8 hours).
# 900s (15 min) is a safe conservative value for active workloads.
AE_DB_POOL_RECYCLE=900
# Connections held open per API replica at idle (the "warm" pool).
# Each replica maintains this many persistent connections to MariaDB.
AE_DB_POOL_SIZE=10
# Additional connections a replica can open beyond AE_DB_POOL_SIZE under burst load.
# These are created on demand and closed when the burst subsides.
# Max connections per replica = AE_DB_POOL_SIZE + AE_DB_POOL_MAX_OVERFLOW.
# Total max DB connections across all replicas = AE_API_REPLICAS × (AE_DB_POOL_SIZE + AE_DB_POOL_MAX_OVERFLOW).
# Example: 3 replicas × (10 + 20) = 90 max connections. MARIADB_MAX_CONNECTIONS must exceed this.
AE_DB_POOL_MAX_OVERFLOW=20
# ------------------------------------------------------------------------------
# REDIS SETTINGS
# ------------------------------------------------------------------------------
# Redis is used for caching, ID resolution, and messaging
AE_REDIS_SERVER=redis
AE_REDIS_PORT=6379
## Aether SMTP access and use
# ------------------------------------------------------------------------------
# API SETTINGS (FastAPI)
# ------------------------------------------------------------------------------
# Number of API container instances (Docker Compose replica scaling).
# Each replica is an independent container with its own Gunicorn process and
# connection pool. Total DB connections = AE_API_REPLICAS × (AE_DB_POOL_SIZE + AE_DB_POOL_MAX_OVERFLOW).
# Increase for horizontal scaling across CPU cores. On a single-node Linode,
# 2-4 replicas is typical; more replicas won't help if the DB is the bottleneck.
AE_API_REPLICAS=3
# --- Gunicorn / Uvicorn Tuning ---
# Internal port Gunicorn listens on inside the container. Nginx proxies to this.
# Each replica uses this same port within its own network namespace.
AE_API_GUNICORN_PORT=5065
# Worker timeout in seconds. A request that takes longer than this causes Gunicorn
# to kill and restart the worker. Default in gunicorn_conf.py is 120s.
# Raise for endpoints that run long ffmpeg operations (clip_video, convert_file, etc.).
# Dev typically uses 900s to accommodate 5-15 min video jobs.
AE_API_GUNICORN_TIMEOUT=900
# Uvicorn worker processes per replica. Each worker handles requests independently
# using async I/O, but SQLAlchemy DB calls are synchronous and block the worker.
# More workers = more parallel DB queries. Recommended: 2-4 per replica.
# Total parallel DB query capacity ≈ AE_API_REPLICAS × AE_API_GUNICORN_WORKERS.
# Stress testing at 4 workers/replica yielded ~2x throughput vs 2 workers (14 req/s vs 7.5 req/s).
# Rule of thumb: (2 × CPU cores) + 1 per replica, but DB throughput caps before CPU becomes the limit.
AE_API_GUNICORN_WORKERS=4
# Threads per Gunicorn worker. Uvicorn workers use async I/O, so threading provides
# minimal benefit here. Leave at 1 unless explicitly benchmarked otherwise.
AE_API_GUNICORN_THREADS=1
# Security & CORS
# JWT_KEY should be a 22+ character secret string. Rotate if compromised.
AE_API_JWT_KEY=XXXX
# Regex for allowed CORS origins. Requests from non-matching origins are blocked.
# Extend the pattern if adding new domains or local dev ports.
AE_API_ORIGINS_REGEX="(https://.*\.oneskyit\.com)|(http://.*\.oneskyit\.com)|(http://.*.localhost)|(http://.*.localhost:5173)"
# ------------------------------------------------------------------------------
# SMTP SETTINGS (Email)
# ------------------------------------------------------------------------------
# Core SMTP configuration for system notifications and user emails
AE_SMTP_SERVER=linode.oneskyit.com
AE_SMTP_PORT=465
AE_SMTP_USERNAME=send_mail
# AE_SMTP_PASSWORD="not currently used"
AE_SMTP_PASSWORD=XXXX
# ------------------------------------------------------------------------------
# APP SETTINGS (SvelteKit)
# ------------------------------------------------------------------------------
# Build mode baked into the Docker image at build time (dev, test, prod)
AE_APP_BUILD_MODE=dev
AE_APP_REPLICAS=2
# Aether API specific config options (FastAPI)
# AE_API_CFG_ID=0 # NOT CURRENTLY NEED OR USED
AE_API_ENV=development
AE_API_DIR=/srv/aether_api
AE_API_LOG_PATH="/logs/aether_api.log"
AE_API_WORKERS=1
AE_API_THREADS=1
AE_API_RELOAD=False
# ------------------------------------------------------------------------------
# SOURCE PATHS (Absolute paths on Host Machine)
# ------------------------------------------------------------------------------
# IMPORTANT: These paths must exist on the machine running Docker.
# They are mounted into containers as volumes for real-time development.
AE_API_SRC=/home/scott/OSIT_dev/aether_api_fastapi
AE_APP_SRC=/home/scott/OSIT_dev/aether_app_sveltekit
# Physical File Storage (Images, Documents, etc.)
# NOTE: Shared between environments to ensure binary availability
HOSTED_FILES_SRC=/home/scott/OSIT/hosted_files
HOSTED_TMP_SRC=/home/scott/OSIT/hosted_tmp
# Aether app specific config (Flask with Svelte)
AE_APP_CFG_ID=0
AE_APP_ENV=development
AE_APP_UX_MODE=default
# AE_APP_UX_MODE=onsite
# AE_APP_UX_MODE=native
AE_APP_DIR=/srv/aether_app
AE_APP_LOG_PATH="/logs/aether_app.log"
AE_APP_WORKERS=1
AE_APP_THREADS=1
AE_APP_RELOAD=True
# ------------------------------------------------------------------------------
# SERVICE TUNING & PERFORMANCE
# ------------------------------------------------------------------------------
# phpMyAdmin Host Port
AE_PMA_PORT=8081
# MariaDB Performance (Injected via Docker Compose command flags)
MARIADB_MAX_CONNECTIONS=500
MARIADB_INNODB_BUFFER_POOL_SIZE=512M
MARIADB_QUERY_CACHE_SIZE=32M
MARIADB_TMP_TABLE_SIZE=384M
MARIADB_TABLE_OPEN_CACHE=4000
# ------------------------------------------------------------------------------
# AETHER SHARED CONFIG (DB Driven)
# ------------------------------------------------------------------------------
# Specifies which record from the 'cfg' table to use for shared settings
# (SMTP, API routes, and external service keys)
# common options: 1=Default, 5=Home Dev, 7=Live Test
AE_CFG_ID=5

49
.gitignore vendored
View File

@@ -57,6 +57,7 @@ Thumbs.db
.env
.venv
*.env
!.env.default
env/
venv/
ENV/
@@ -65,21 +66,28 @@ venv.bak/
environment/
*.bak
*.kate-swp
.directory
archives/
archives_and_old/
backups/
bak/
downloads/
hold/
log/
# logs/
logs/ae_api_blue/
logs/ae_api_green/
logs/ae_api_red/
# logs/ae_api/
# logs/ae_app/
logs/apache2/
logs/mailman2/
# logs/php7/
# logs/web/
tmp/
old/
temp/
tmp/
# OSIT and Aether specific:
@@ -107,18 +115,29 @@ srv/mailman2/
# srv/hosted_tmp_link/
# srv/hosted_tmp_link
srv/aether_api_ln/
srv/aether_api_ln
srv/aether_app_ln/
srv/aether_app_ln
srv/hosted_files_dev_ln/
srv/hosted_files_dev_ln
srv/hosted_files_ln/
srv/hosted_files_ln
srv/hosted_tmp_dev_ln/
srv/hosted_tmp_dev_ln
srv/hosted_tmp_ln/
srv/hosted_tmp_ln
# srv/aether_api_ln/
# srv/aether_api_ln
# srv/aether_app_ln/
# srv/aether_app_ln
# srv/hosted_files_dev_ln/
# srv/hosted_files_dev_ln
# srv/hosted_files_ln/
# srv/hosted_files_ln
# srv/hosted_tmp_dev_ln/
# srv/hosted_tmp_dev_ln
# srv/hosted_tmp_ln/
# srv/hosted_tmp_ln
srv/mariadb_ln/
srv/mariadb_ln
# srv/mariadb_ln/
# srv/mariadb_ln
srv/mariadb/
srv/mariadb
# Aether DB Snapshots and Backups
srv/mariadb_bak_*
backups/imported/
backups/auto_backup_*
srv/aether_api_v5_ln/
srv/aether_api_v5_ln

44
CHEATSHEET.md Normal file
View File

@@ -0,0 +1,44 @@
# Aether Docker Cheat Sheet 🚀
## 🚀 Deployment & Updates
- **Full Rebuild:** `docker compose up -d --build`
- **Rebuild SvelteKit only:** `docker compose up -d --build ae_app`
- **Restart API (pick up Python changes):** `docker compose restart ae_api`
- **Rebuild SvelteKit (local):** `make build-docker-dev` / `build-docker-test` / `build-docker-prod`
- **Deploy to remote:** `make deploy-remote-test` / `deploy-remote-prod` (SSH → linode.oneskyit.com)
- **Shut everything down:** `npm run compose:down` (from `aether_app_sveltekit/`)
## 🛠️ Management Links
- **SvelteKit Frontend:** [http://localhost:8888](http://localhost:8888) (LAN HTTP via ae_web_dev)
- **FastAPI Docs:** [https://dev-api.oneskyit.com/docs](https://dev-api.oneskyit.com/docs)
- **Database (phpMyAdmin):** [http://localhost:8081](http://localhost:8081) (requires `--profile database`)
- **Logs (Dozzle):** [http://localhost:8881](http://localhost:8881)
## 🔌 Multi-Stack Port Reference
To run multiple stacks (`test`, `bak`, `prod`) on one host, you **must** assign unique host ports in each `.env`.
| Port Type | Variable | Example (Test) | Example (Bak) | Example (Prod) |
|-----------------|-----------------------|----------------|---------------|----------------|
| App Gateway | `AE_APP_GATEWAY_PORT` | 3009 | 3002 | 3001 |
| API Gateway | `AE_API_GATEWAY_PORT` | 5063 | 5062 | 5061 |
| LAN Direct HTTP | `OSIT_WEB_HTTP_PORT` | 32887 | 32891 | 32890 |
| Dozzle Logs | `AE_DOZZLE_PORT` | 8889 | 8882 | 8881 |
| MariaDB Direct | `AE_DB_EXTERNAL_PORT` | 32769 | 32770 | 32768 |
## 🏗️ Multi-Stack Isolation
1. **Network Name:** Set `AE_NETWORK_NAME=ae_test_net` (etc) to prevent Docker network name collisions.
2. **Container Names:** All service container names are now `.env` variables with `:-default` fallbacks. Set unique values per stack:
- `CONTAINER_WEB`, `CONTAINER_REDIS`, `CONTAINER_DOZZLE`
- `CONTAINER_MARIADB`, `CONTAINER_PMA`, `CONTAINER_AE_OPS` (database profile only)
- Note: `ae_api` and `ae_app` use `scale` — Docker does not allow `container_name` on scaled services.
3. **Internal Shared Net:** All stacks must connect to `aether_shared_net` to reach a shared MariaDB/Redis.
## 💾 Database Operations
- **Manual Backup:** `./backup_db.sh` (hot backup, live container)
- **Manual Restore:** `./restore_db.sh [path_to_file.gz]`
- **Conference Export:** `./export_db.sh` (saves to `backups/conference_export/`)
- **Automated Import:** Drop file in `backups/import/` → run `./check_and_import.sh`
## 🧹 Maintenance
- **Internal Logs:** Docker handles rotation automatically (10MB limit).
- **Dozzle:** Live log viewer at port 8881 — no auth currently (LAN only).

50
Makefile Normal file
View File

@@ -0,0 +1,50 @@
# Aether Platform - Operations Makefile
# Use these shortcuts for faster development and deployment.
.PHONY: up down restart-api build-api build-docker-dev build-docker-test build-docker-prod logs ps deploy-remote-test deploy-remote-prod
# Start the entire stack
up:
docker compose up -d
# Stop the entire stack
down:
docker compose down
# FAST UPDATE: Pick up Python code changes without rebuilding
# Since source is mounted as a volume, we just need to restart the container.
# This takes ~2 seconds instead of ~60 seconds.
restart-api:
docker compose restart ae_api
# REBUILD API: Use this only when requirements.txt or Dockerfile changes.
build-api:
docker compose up -d --build ae_api
# BUILD DOCKER UI: Build the SvelteKit container for the given mode.
# Use 'npm run dev' for active development (Vite HMR, no Docker).
# Use these only when testing the production-like Docker build locally.
build-docker-dev:
docker compose build ae_app && docker compose up -d ae_app
build-docker-test:
docker compose build --build-arg BUILD_MODE=test ae_app && docker compose up -d ae_app
build-docker-prod:
docker compose build --build-arg BUILD_MODE=prod ae_app && docker compose up -d --remove-orphans ae_app
# View combined logs
logs:
docker compose logs -f --tail=100
# Check service status
ps:
docker compose ps
# Remote deploy (SSH to linode.oneskyit.com, run deploy.sh)
# Requires key-based SSH and deploy.sh committed + pulled on the server.
deploy-remote-test:
ssh linode.oneskyit.com 'bash /srv/env/test_aether/deploy.sh test'
deploy-remote-prod:
ssh linode.oneskyit.com 'bash /srv/env/prod_aether/deploy.sh prod'

228
README.md
View File

@@ -1,133 +1,147 @@
This can be used to create a Docker server cluster for the Aether App using Flask, Aether API using FastAPI.
# Aether Framework - Docker Environment (Unified V3)
## Initialize
### Part 1
* Create directory and clone the Aether environment.
```bash
sudo mkdir /srv/env
sudo chown -R scott:scott /srv/env/
git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-container-environment.git /srv/env/test_aether
This repository provides the unified Docker orchestration and configuration for the Aether Platform. It manages the lifecycle of the Aether API (FastAPI), Aether App (SvelteKit), and supporting infrastructure (MariaDB, Redis, Nginx).
## 🌐 Traffic Architecture
The V3 environment uses a **Dual-Network** strategy to support multiple isolated stacks (`test`, `bak`, `prod`) on a single host while sharing core services like MariaDB or Redis.
```
External Internet
Home Server Nginx (SSL termination, domain routing)
↓ ↓
workstation:3001 workstation:5060
(AE_APP_GATEWAY_PORT) (AE_API_GATEWAY_PORT)
↓ ↓
ae_web_dev (Docker nginx, port 80)
↓ ↓
svelte_backend fastapi_backend
(Docker DNS round-robin) (Docker DNS round-robin)
↓ ↓
ae_app replicas ae_api replicas
```
### Part 2
* Create links to needed for the srv/ directories. See the README.md file under srv/ for details.
* Copy Let's Encrypt certificates to the conf/certs/ directory. See the README.md file under conf/certs/ for details.
* Copy MariaDB database files to the srv/mariadb/ directory using rsync. The original files may need to be copied from the normal Arch Linux location (/var/lib/mysql) first and then possibly reset the root password.
**Key Improvements:**
- **Timezone Support:** All containers use the `TZ` variable from `.env` for consistent logging and database timestamps.
- **Scalable Routing:** Nginx uses Regex (`~^(dev|test|bak|sr|prod)?-?...`) to automatically handle any environment prefix without configuration changes.
- **Isolated Stacks:** Each deployment uses a unique `AE_NETWORK_NAME` and `CONTAINER_` prefix to prevent collisions.
- **Shared Services:** Core infrastructure (DB/Redis) resides on the `aether_shared_net` which must be created manually once.
---
## 🚀 Quick Start
### 1. Initialize Host Network
Before starting your first stack, create the shared internal network:
```bash
sudo rsync -vhr -progress /var/lib/mysql/ /srv/env/test_aether/srv/mariadb/
sudo rsync -vhrz scott@linode.oneskyit.com:/srv/env/test_aether/srv/mariadb/ /srv/env/test_aether/srv/mariadb/
sudo chown -R scott:scott /srv/env/test_aether/srv/mariadb/
docker network create aether_shared_net
```
### Part 3
* Create the environment settings file and place it under the root of the Docker Compose directory. Copy the .env.default file as a template.
### 2. Initialize Directory Structure
Create the base directory and clone this environment:
```bash
cp /srv/env/test_aether/.env /srv/env/test_aether/.env.bak
cp /srv/env/test_aether/.env.default /srv/env/test_aether/.env
```
**/.env**
```sh
# One Sky IT's Aether Framework and System
OSIT_ENV=development
# Aether general shared config options
## Aether API access and use
## Aether DB access and use
## Aether Redis access and use
## Aether SMTP access and use
# Aether API specific config options (FastAPI)
# Aether app specific config (Flask with Svelte)
KEY="The Value"
sudo mkdir -p /srv/env/aether
sudo chown -R $USER:$USER /srv/env/aether
git clone https://bitbucket.org/oneskyit/one-sky-it-container-environment.git /srv/env/aether/container_env
```
## Manage Docker Compose Environment
### 3. Configure Environment Settings
Copy the template and update it with your unique stack identifiers:
```bash
docker restart ae_api_dev
docker restart ae_app_dev
docker restart ae_mariadb_dev
cd /srv/env/aether/container_env
cp env.default .env
# CRITICAL: Set unique CONTAINER_ prefixes and AE_NETWORK_NAME for each stack
vim .env
```
### 3. Configure Data & Source Paths
The containers locate data and source code using absolute paths defined in your `.env` file. Ensure these variables point to the correct locations on your host system:
- **`AE_API_SRC`**: Path to your `aether_api_fastapi` repository.
- **`AE_APP_SRC`**: Path to your `aether_app_sveltekit` repository.
- **`HOSTED_FILES_SRC`**: Path to the physical storage for images/documents.
- **`HOSTED_TMP_SRC`**: Path for temporary file processing.
### 4. Certificates & Database
* **SSL:** Place your wild-card certificates in `conf/certs/` (matching the filenames in `docker-compose.yml`).
* **Database:** Use the restoration scripts (see below) to import a MariaDB snapshot.
---
## 🛠️ Management Commands
### Orchestration (Unified Stack)
```bash
docker compose up -d --build # Build and start all services (Autonomous SvelteKit build)
docker compose down # Stop all services
docker compose restart ae_app # Restart the SvelteKit UI
docker compose restart ae_api # Restart the FastAPI Backend
```
### Deployment Workflow
The SvelteKit application is built **inside** the container using `vite build --mode <env>`, which reads the corresponding `.env.<env>` file for `PUBLIC_` variables.
From `aether_app_sveltekit/`:
```bash
# Build Docker image locally
npm run build:docker:dev # uses .env.dev
npm run build:docker:test # uses .env.test
npm run build:docker:prod # uses .env.prod
# Deploy to remote server (linode.oneskyit.com)
npm run deploy:remote:test
npm run deploy:remote:prod
```
Or via Makefile targets in this directory:
```bash
make build-docker-dev
make deploy-remote-prod
```
---
## More Notes
## 🗄️ Database Management (Physical Backups)
... (rest of the file remains the same) ...
The system uses physical hot backups via `mariabackup` for maximum speed and data integrity.
```bash
ln -s /srv/http/dev_app.oneskyit.com /srv/env/test_aether/srv/aether_app_ln
ln -s /srv/http/dev_fastapi.oneskyit.com /srv/env/test_aether/srv/aether_api_ln
ln -s /mnt/data/speaker_ready/hosted_tmp /srv/env/test_aether/srv/hosted_tmp_ln
ln -s /mnt/data/speaker_ready/hosted_files /srv/env/test_aether/srv/hosted_files_ln
### User-Facing Scripts
These scripts are located in the root directory:
ls -lha /srv/env/test_aether/srv/aether_app_ln/
ls -lha /srv/env/test_aether/srv/aether_api_ln/
ls -lha /srv/env/test_aether/srv/hosted_tmp_ln/
ls -lha /srv/env/test_aether/srv/hosted_files_ln/
````
* **`./backup_db.sh`**: Triggers an immediate hot backup. Results are stored in `backups/`.
* **`./export_db.sh`**: Creates a "Conference Ready" backup in `backups/conference_export/` with host-user ownership.
* **`./restore_db.sh [backup_file.gz]`**: Performs a full "Clean Slate" restoration.
* *Warning:* This stops MariaDB, archives current data, and resets `root` passwords to match your `.env`.
* **`./check_and_import.sh`**: A watchdog script that monitors `backups/import/` for new snapshots and triggers automated restoration.
### Part 2
```bash
sudo ls -lha /var/lib/mysql/
```
---
## 📂 Directory Map
* **`conf/`**: Configuration templates for Nginx and Gunicorn. API config now lives in the `aether_api_fastapi` repo as `app/config.py` and reads settings directly from env vars.
* **`logs/`**: Centralized logging for all containers. Automatic rotation is managed by the `ae_ops` service (7-day retention).
* **`srv/`**: Mount points for data and source code (managed via symlinks).
* **`scripts/`**: Internal automation logic.
* **`backups/`**: Storage for MariaDB snapshots.
## Common:
* conf/ = All config files
* logs/ = All log files
* srv/ = All files and data directories that are being served up in some way.
* srv/hosted_files = All hashed hosted files (/mnt/data_drive/srv/data/osit_app/hosted_files_dev/)
* srv/static_files = All static files
* ---
* srv/aether_api = Aether API
* srv/aether_app = Aether App
* ---
* srv/html_php = The default HTML and PHP directory
* srv/oneskyit_site = One Sky IT, LLC main site
* ---
* srv/mailman2 = All Mailman2 files
* srv/mariadb = All MariaDB database files
* srv/nextcloud = All Nextcloud app files
---
## Servics:
nginx
php
gunicorn
mariadb
phpmyadmin
## 📝 Configuration Guidelines
redis
postfix?
mailman?
Nextcloud???
Syncthing???
restic???
memcached???
### Environment Profiles (`COMPOSE_PROFILES`)
* **`database`**: Includes MariaDB, phpMyAdmin, and the `ae_ops` maintenance service.
* **App-Only**: Leave empty if connecting to a remote/shared database server.
### Aether Config ID (`AE_CFG_ID`)
Specifies the record from the `cfg` table to load during the API bootstrap process:
* `1`: Default / Template
* `5`: Home Development
* `7`: Live Testing
## Setup
## Check for in use services and ports
Note that the Aether FastAPI will hang if redis is not found.
sudo systemctl status mariadb.services
sudo systemctl status nginx.service
sudo systemctl status php-fpm.service
sudo systemctl status postfix.service
sudo systemctl status redis.service
### Create links to Aether API and app directories
Make sure the docker-compose.yml file is updated with the correct paths.
ln -s /home/scott/OSIT_dev/aether_api_fastapi /home/scott/OSIT_dev/aether_container_env/srv/aether_api_link
ln -s /home/scott/OSIT_dev/aether_app /home/scott/OSIT_dev/aether_container_env/srv/aether_app_link
ln -s /mnt/data_drive/srv/data/osit_app/hosted_files /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_link
ln -s /mnt/data_drive/srv/data/osit_app/hosted_files_dev /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_dev_link
ln -s /mnt/data_drive/srv/data/osit_app/hosted_tmp /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_link
ln -s /mnt/data_drive/srv/data/osit_app/hosted_tmp_dev /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_dev_link
---
## ⚠️ Security Notes
* Never commit the `.env` file (it is ignored by git).
* Ensure `AE_API_JWT_KEY` is a unique, high-entropy string in production.
* The API prioritizes `.env` credentials over DB settings for core infrastructure (SMTP/DB) to prevent accidental lockouts.

View File

@@ -0,0 +1,8 @@
{
"folders": [
{
"path": "."
}
],
"settings": {}
}

View File

@@ -1,11 +0,0 @@
FROM tiangolo/uvicorn-gunicorn-fastapi:latest
# FROM tiangolo/uvicorn-gunicorn-fastapi:python3.10
LABEL maintainer="Scott Idem <scott.idem@oneskyit.com>"
WORKDIR /srv/aether_api
COPY conf/aether_fastapi_requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt
CMD ["gunicorn", "--conf", "/conf/gunicorn_fastapi_conf.py"]

View File

@@ -1,4 +1,5 @@
FROM python:3
# FROM python:latest
FROM python:3.11
LABEL maintainer="Scott Idem <scott.idem@oneskyit.com>"
@@ -7,4 +8,7 @@ WORKDIR /srv/aether_app
COPY conf/aether_flask_requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt
RUN pip freeze > /aether_flask_requirements_current.txt
RUN pip freeze > /tmp/aether_flask_requirements_current.txt
CMD ["gunicorn", "--conf", "/conf/gunicorn_flask_conf.py"]

19
backup_db.sh Executable file
View File

@@ -0,0 +1,19 @@
#!/bin/bash
# Aether MariaDB Backup Script (Physical Backup)
set -e
PROJECT_ROOT="/home/scott/OSIT_dev/aether_container_env"
BACKUP_DIR="${PROJECT_ROOT}/backups"
TIMESTAMP=$(date +%Y%m%d_%H%M)
BACKUP_FILE="${BACKUP_DIR}/local_backup_${TIMESTAMP}.gz"
echo "--- Starting Aether Local Database Backup ---"
mkdir -p "${BACKUP_DIR}"
# Increased open-files-limit to prevent OS error 24
echo ">>> Backing up to ${BACKUP_FILE}..."
docker exec ae_mariadb_dev mariabackup --user=root --password='$1sky.AE_dev.2023' \
--backup --stream=xbstream --open-files-limit=65535 | gzip > "${BACKUP_FILE}"
echo "--- Backup Complete! ---"
ls -lh "${BACKUP_FILE}"

29
check_and_import.sh Executable file
View File

@@ -0,0 +1,29 @@
#!/bin/bash
# Aether Automated Import Watchdog
# Checks 'backups/import/' for new database files and restores them.
set -e
PROJECT_ROOT="/home/scott/OSIT_dev/aether_container_env"
IMPORT_DIR="${PROJECT_ROOT}/backups/import"
ARCHIVE_DIR="${PROJECT_ROOT}/backups/imported"
mkdir -p "$IMPORT_DIR" "$ARCHIVE_DIR"
# Find the newest .gz file in the import directory
NEW_BACKUP=$(ls -t "$IMPORT_DIR"/*.gz 2>/dev/null | head -n 1)
if [ -n "$NEW_BACKUP" ]; then
echo "--- New Backup Detected: $(basename "$NEW_BACKUP") ---"
# Run the restore
"${PROJECT_ROOT}/restore_db.sh" "$NEW_BACKUP"
# Move to archive
echo ">>> Archiving imported file..."
mv "$NEW_BACKUP" "$ARCHIVE_DIR/"
echo "--- Automated Import Finished ---"
else
echo "No files found in $IMPORT_DIR. Nothing to do."
fi

View File

@@ -1,104 +0,0 @@
# Configuration file for this FastAPI app.
import os
from pydantic import AnyHttpUrl, BaseSettings, EmailStr, HttpUrl, PostgresDsn, validator
from typing import Any, Dict, List, Optional, Union
# ### ### #
class Settings(BaseSettings):
AETHER_CFG = {}
AETHER_CFG['id'] = os.getenv('AE_CFG_ID', None)
# AETHER_CFG['api_id'] = os.getenv('AE_API_CFG_ID', None) # NOT CURRENTLY NEED OR USED
# APP_NAME: str = "Aether API (FastAPI)"
# SUPER_EMAIL: EmailStr = 'Aether.Super@oneskyit.com'
# Database Connection
DB = {}
DB['server'] = os.getenv('AE_DB_SERVER', 'mariadb') # 'linode.oneskyit.com' # linode.oneskyit.com, vpn-linode linode.oneskyit.local
DB['port'] = os.getenv('AE_DB_PORT', '3306') # default = 3306
DB['name'] = os.getenv('AE_DB_NAME', None) # 'aether_dev' #onesky_ams_dev
DB['username'] = os.getenv('AE_DB_USERNAME', None) # 'osit_aether' # 'onesky_aether'
DB['password'] = os.getenv('AE_DB_PASSWORD', None) #
SQLALCHEMY_DB_URI = 'mysql://'+DB['username']+':'+DB['password']+'@'+DB['server']+'/'+DB['name']
# Aether API log files paths
LOG_PATH = {}
LOG_PATH['app'] = os.getenv('AE_API_LOG_PATH', 'admin/log/app.log') # 'admin/log/app.log', '../../logs/aether_api.log'
# LOG_PATH['app_warning'] = '/logs/aether_api_warning.log' # 'admin/log/app_warning.log' '../../logs/aether_api_warning.log'
# Redis
REDIS = {}
REDIS['server'] = os.getenv('AE_REDIS_SERVER', 'redis') # 'localhost' 'redis'
REDIS['port'] = os.getenv('AE_REDIS_PORT', '6379') # '6379'
# Send SMTP Email
SMTP = {}
# server
# port
# username
# password
# Server Hosted File Paths
FILES_PATH = {}
# hosted_files_root
# hosted_tmp_root
# CORS Origins
ORIGINS_REGEX = '(https://.*\.oneskyit\.com)|(http://.*\.oneskyit\.com)|(https://.*\.oneskyit\.com:4443)|(http://.*\.oneskyit\.com:8080)|(http://.*\.oneskyit\.com:8181)|(https://.*\.oneskyit\.com:8443)|(http://.*\.oneskyit\.local)|(http://.*\.oneskyit\.local:5000)|(http://.*.localhost)|(http://.*.localhost:5000)|(http://.*.localhost:8181)'
# A reasonable, but fairly open example regular expression for the CORS origins:
# '(https://.*\.oneskyit\.com)|(http://.*\.oneskyit\.com)|(http://.*\.oneskyit\.com:8181)|(https://.*\.oneskyit\.com:8443)|(http://.*\.oneskyit\.local)|(http://.*\.oneskyit\.local:5000)|(http://.*.localhost)|(http://.*.localhost:5000)|(http://.*.localhost:8181)'
ORIGINS = [
'https://oneskyit.com',
# 'http://app-local.oneskyit.com',
'http://192.168.32.20:3000',
'http://192.168.32.20:8080',
'http://localhost',
'http://localhost:3000',
# 'http://localhost:5000',
'http://localhost:7800',
# 'http://localhost:8080',
# 'http://localhost:8888',
# 'http://fastapi.localhost',
'http://svelte.oneskyit.local:5555',
# 'http://connect.localhost:5000', # Using localhost
# 'http://dev-svelte.oneskyit.local:5555',
# 'http://lci.internal:5000', # Using internal; just in case guess before LCI
# 'http://lci.oneskyit.internal:5000', # Using internal; just in case guess before LCI
# 'http://lci.oneskyit.internal', # Using internal; just in case guess before LCI
]
# HTTP Status Dict List
HTTP_STATUS_LI = {}
# HTTP_STATUS_LI[200] = { 'name': 'OK', 'message': 'The request has succeeded.' }
# HTTP_STATUS_LI[400] = { 'name': 'Bad Request', 'message': 'The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.' }
# HTTP_STATUS_LI[401] = { 'name': 'Unauthorized', 'message': 'The server could not verify that you are authorized to access the URL requested. You either supplied the wrong credentials (e.g. a bad password), or your browser does not understand how to supply the credentials required.' }
# HTTP_STATUS_LI[402] = { 'name': '?Request Failed?', 'message': '??The parameters were valid but the request failed.??' }
# HTTP_STATUS_LI[403] = { 'name': 'Forbidden', 'message': 'The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. If the request method was not HEAD and the server wishes to make public why the request has not been fulfilled, it SHOULD describe the reason for the refusal in the entity. If the server does not wish to make this information available to the client, the status code 404 (Not Found) can be used instead.' }
# HTTP_STATUS_LI[404] = { 'name': 'Not Found', 'message': 'The requested resource does not exist.' }
# HTTP_STATUS_LI[409] = { 'name': 'Conflict', 'message': 'The request conflicts with another request (perhaps due to using the same idempotent key).' }
# HTTP_STATUS_LI[429] = { 'name': 'Too Many Requests', 'message': 'Too many requests hit the API too quickly. We recommend an exponential backoff of your requests.' }
# HTTP_STATUS_LI[500] = { 'name': 'Internal Server Error', 'message': 'The server encountered an unexpected condition which prevented it from fulfilling the request.' }
# HTTP_STATUS_LI[501] = { 'name': 'Not Implemented', 'message': 'The server does not support the functionality required to fulfill the request. This is the appropriate response when the server does not recognize the request method and is not capable of supporting it for any resource.' }
# HTTP_STATUS_LI[502] = { 'name': 'Bad Gateway', 'message': 'The server, while acting as a gateway or proxy, received an invalid response from the upstream server it accessed in attempting to fulfill the request.' }
# HTTP_STATUS_LI[503] = { 'name': 'Service Unavailable', 'message': 'The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay. If known, the length of the delay MAY be indicated in a Retry-After header. If no Retry-After is given, the client SHOULD handle the response as it would for a 500 response.' }
# HTTP_STATUS_LI[504] = { 'name': 'Gateway Timeout', 'message': 'The server, while acting as a gateway or proxy, did not receive a timely response from the upstream server specified by the URI (e.g. HTTP, FTP, LDAP) or some other auxiliary server (e.g. DNS) it needed to access in attempting to complete the request.' }
settings = Settings()

View File

@@ -14,7 +14,7 @@ ENV = 'development' # None, 'backup', 'default', 'development', 'production', 't
DEBUG = True
TESTING = True
SECRET_KEY = '\x08}\xe1q?\xb2\x16o9\xf1\x1d\xc7\xa8\xfb!\xff' # Generate a new key with: # python -c 'import os; print(os.urandom(16))'
SECRET_KEY = os.getenv('AE_APP_CACHE_SECRET_KEY', None) # Generate a new key with: # python -c 'import os; print(os.urandom(16))'
# SESSION_COOKIE_NAME = 'session'
# SESSION_COOKIE_DOMAIN = '.onesky.com'
@@ -31,19 +31,19 @@ SECRET_KEY = '\x08}\xe1q?\xb2\x16o9\xf1\x1d\xc7\xa8\xfb!\xff' # Generate a new k
# 7862400 is 91 days or 13 weeks (.25 year)
# 31536000 is 365 days
# 33868800 is 9408 hours or 392 days
PERMANENT_SESSION_LIFETIME = 3600
PERMANENT_SESSION_LIFETIME = int(os.getenv('AE_APP_SESSION_LIFETIME', 86400))
SESSION_REFRESH_EACH_REQUEST = True
# Files and caching
# SERVER_NAME = ''
# UPLOAD_FOLDER = ''
MAX_CONTENT_LENGTH = 5120 * 1024 * 1024 # 5 GB
# MAX_CONTENT_LENGTH = 5120 * 1024 * 1024 # 5 GB
SEND_FILE_MAX_AGE_DEFAULT = 3 # default 43200 (in seconds), 1800 = 30 minutes
CACHE_TYPE = 'SimpleCache'
CACHE_DEFAULT_TIMEOUT = 1
CACHE_DEFAULT_TIMEOUT = int(os.getenv('AE_APP_CACHE_TIMEOUT', 5))
JSONIFY_PRETTYPRINT_REGULAR = True
@@ -67,7 +67,7 @@ ISOLATION_LEVEL = 'READ COMMITTED'
AETHER_CFG = {}
## General Aether configuration options
AETHER_CFG['id'] = 5 # Aether config ID
AETHER_CFG['id'] = os.getenv('AE_CFG_ID', None) # Aether config ID
## Aether Flask app (not a specific browser client) configuration (pull from "cfg_flask" table)
AETHER_CFG['app'] = {}
@@ -81,18 +81,18 @@ AETHER_CFG['app']['ux_mode'] = None # In this case it is the same as mode. None,
AETHER_CFG['app']['theme'] = None # For future use or at least for 'light', 'dark', 'contrast' themes?
AETHER_CFG['app']['path_hosted_files_root']: str = None
AETHER_CFG['app']['path_hosted_tmp_root']: str = None
AETHER_CFG['app']['path_hosted_qr_images']: str = None # Will contain only QR code image files
# AETHER_CFG['app']['path_hosted_qr_images']: str = None # Will contain only QR code image files; Use path_hosted_tmp_root instead
## Aether API for Flask app configuration (pull from "cfg_flask" table)
AETHER_CFG['api'] = {}
AETHER_CFG['api']['protocol'] = os.getenv('AE_API_PROTOCOL', 'https') # 'https' # https
AETHER_CFG['api']['server'] = os.getenv('AE_API_SERVER', None) # 'dev-api.oneskyit.com' # linode.oneskyit.com vpn-linode linode.oneskyit.local
AETHER_CFG['api']['port'] = os.getenv('AE_API_PORT', '443') # '443' # default = 3306
AETHER_CFG['api']['port'] = os.getenv('AE_API_PORT', '443')
AETHER_CFG['api']['path'] = os.getenv('AE_API_PATH', '') # ''
AETHER_CFG['api']['secret_key'] = os.getenv('AE_API_SECRET_KEY', None)
AETHER_CFG['api']['protocol_backup'] = 'http' # https
AETHER_CFG['api']['server_backup'] = 'dev-fastapi.oneskyit.local' # linode.oneskyit.com vpn-linode linode.oneskyit.local
AETHER_CFG['api']['port_backup'] = '5005' # default = 3306
AETHER_CFG['api']['protocol_backup'] = 'https' # https
AETHER_CFG['api']['server_backup'] = 'bak-api.oneskyit.com' # linode.oneskyit.com vpn-linode linode.oneskyit.local
AETHER_CFG['api']['port_backup'] = '443'
AETHER_CFG['api']['path_backup'] = ''
AETHER_CFG['api']['secret_key_backup'] = 'the secret backup key'
AETHER_CFG['api']['temporary_token'] = {}

View File

@@ -1,43 +1,30 @@
import os
# Gunicorn config variables
loglevel = "warning"
loglevel = os.getenv('AE_LOG_LVL', 'warning')
accesslog = "/logs/gunicorn_access.log" # "-" # stdout
errorlog = "/logs/gunicorn_error.log" # "-" # stderr
# "logfile" does not seem to actually do anything
# logfile = "/logs/gunicorn.log" # "-" # stderr
accesslog = "-" # stdout
errorlog = "-" # stderr
# Use /dev/shm for the control socket to avoid permission issues on volume mounts
control_socket = "/dev/shm/gunicorn.ctl"
# ... (existing bind/chdir) ...
bind = "0.0.0.0:5005"
# bind = "unix:/tmp/gunicorn.sock"
worker_tmp_dir = "/dev/shm"
chdir = "/srv/aether_api"
# home = /path/to/environment
wsgi_app = "app.main:app"
# module = "run_server"
# callable = "app"
# plugins = "python"
# default_proc_name = "app.main:app"
# Setting a long timeout since some FastAPI API requests may take a while
timeout = 1200 # default 30; worker process silent then kill and restart
graceful_timeout = 20 # default 30; timeout after restart signal
keepalive = 300 # default 2; setting higher because behind load balancer (nginx)
# Numeric variables must be integers
timeout = int(os.getenv('AE_API_GUNICORN_TIMEOUT', 120))
graceful_timeout = int(os.getenv('AE_API_GUNICORN_GRACEFUL_TIMEOUT', 30))
keepalive = int(os.getenv('AE_API_GUNICORN_KEEPALIVE', 4))
# Reload does not work correctly with UvicornWorker
# https://github.com/benoitc/gunicorn/issues/2339
# Disable reload if using more than one thread
reload = True
reload = False
worker_class = "uvicorn.workers.UvicornWorker"
# reload_engine = "poll"
workers = int(os.getenv('AE_API_GUNICORN_WORKERS', 2))
threads = int(os.getenv('AE_API_GUNICORN_THREADS', 2))
worker_class = "uvicorn.workers.UvicornWorker" # default "sync"
# Works are processes, not threads
# workers = 9 # default 1; use 10ish for production; 2 to 4 times the number of cores
# threads = 1 # default 1; only affects Gthread worker type
workers = os.getenv('AE_API_WORKERS', None)
threads = os.getenv('AE_API_THREADS', None)
# umask = '007'

View File

@@ -0,0 +1,63 @@
# aioredis # BAD! Not maintained!
anyio
argon2-cffi
argon2-cffi-bindings
asgiref
async-timeout
certifi
cffi
charset-normalizer
click
Deprecated
dnspython
email-validator
et-xmlfile
fastapi
greenlet
gunicorn
h11
html2text
httpcore
httptools
httpx
idna
itsdangerous
Jinja2
MarkupSafe
mysqlclient
numpy
openpyxl
orjson
packaging
pandas
passlib
pdf2image
Pillow
pycparser
pydantic
PyJWT
pyparsing
python-dateutil
python-dotenv
python-multipart
pytz
PyYAML
qrcode
redis[hiredis]
requests
rfc3986
six
sniffio
SQLAlchemy==1.4.47 # 1.4.47 is the newest I am working with
starlette
stripe
typing_extensions
ujson
urllib3
uvicorn
uvloop
watchfiles
watchgod
websockets
wrapt
xlrd

View File

@@ -1,61 +0,0 @@
anyio==3.6.2
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asgiref==3.5.2
async-timeout==4.0.2
certifi==2022.9.24
cffi==1.15.1
charset-normalizer==2.1.1
click==8.1.3
Deprecated==1.2.13
dnspython==2.2.1
email-validator==1.3.0
et-xmlfile==1.1.0
fastapi==0.88.0
greenlet==2.0.0
gunicorn==20.1.0
h11==0.14.0
html2text==2020.1.16
httpcore==0.16.2
httptools==0.5.0
httpx==0.23.1
idna==3.4
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.1
mysqlclient==2.1.1
numpy==1.23.4
openpyxl==3.0.10
orjson==3.8.1
packaging==21.3
pandas==1.5.2
passlib==1.7.4
Pillow==9.3.0
pycparser==2.21
pydantic==1.10.2
PyJWT==2.6.0
pyparsing==3.0.9
python-dateutil==2.8.2
python-dotenv==0.21.0
python-multipart==0.0.5
pytz==2022.6
PyYAML==6.0
qrcode==7.3.1
redis==4.3.5
requests==2.28.1
rfc3986==1.5.0
six==1.16.0
sniffio==1.3.0
SQLAlchemy==1.4.44
starlette==0.22.0
stripe==5.0.0
typing_extensions==4.4.0
ujson==5.5.0
urllib3==1.26.12
uvicorn==0.18.3
uvloop==0.17.0
watchfiles==0.18.0
watchgod==0.8.2
websockets==10.4
wrapt==1.14.1
xlrd==2.0.1

View File

@@ -0,0 +1,70 @@
# Updated manually 2024-04-26 with a lot of trial and error.
# A few are commented out even though they are actually used and required. Other packages already pull them in.
# SQLAlchemy needs to be upgraded to 2.x. There are issues with async IO or something related to that.
# https://docs.sqlalchemy.org/en/14/changelog/migration_20.html
# aioredis # BAD! Not maintained!
aiofiles
anyio
argon2-cffi
argon2-cffi-bindings
# asgiref
async-timeout
baize # added 2023-08-17
# certifi
# cffi
charset-normalizer
click
Deprecated
dnspython
email-validator
et-xmlfile
fastapi==0.94.1 # working 0.94.1, 0.88.0; not working >= 0.95.0
greenlet>=2.0.2
gunicorn>=22.0.0
h11
html2text>=2020.1.16
httpcore
httptools
httpx
idna
itsdangerous
# Jinja2>=3.1.2
MarkupSafe
mysqlclient
numpy>=1.25.2
openpyxl
orjson
# packaging
pandas>=2.1.0
passlib
pdf2image>=1.16.3
Pillow>=10.0.0
pycparser
pydantic>=1.10.12
PyJWT>=2.8.0
pyparsing
python-dateutil
python-dotenv
python-multipart
pytz
PyYAML>=6.0.1
qrcode>=7.4.2
redis[hiredis] # redis==5.0.0 hiredis==2.2.3
requests
rfc3986
six
sniffio
SQLAlchemy==1.4.49 # 1.4.49 is the newest (2.0.20) I am working with
starlette>=0.22.0
stripe>=6.4.0
typing_extensions
ujson
urllib3
uvicorn
uvloop
watchfiles
watchgod
websockets>=11.0.3
wrapt
xlrd

View File

@@ -1,59 +0,0 @@
import os
# Gunicorn config variables
loglevel = "debug"
# accesslog = "/logs/gunicorn_access.log" # "-" # stdout
errorlog = "/logs/gunicorn_error.log" # "-" # stderr
# "logfile" does not seem to actually do anything
# logfile = "/logs/gunicorn.log" # "-" # stderr
bind = "0.0.0.0:5005"
# bind = "unix:/tmp/gunicorn.sock"
worker_tmp_dir = "/dev/shm"
chdir = "/srv/aether_app"
# home = /path/to/environment
wsgi_app = "run_server:app"
# module = "run_server"
# callable = "app"
# plugins = "python"
# default_proc_name = "run_server:app"
# Setting a longer timeout since some Flask app requests may take a while
timeout = 1200 # default 30; worker process silent then kill and restart
graceful_timeout = 20
keepalive = 300 # default 2; setting higher because behind load balancer (nginx)
# Disable reload if using more than one thread
reload = True
# worker_class = "sync" # default "sync"
# Works are processes, not threads
# workers = 9 # default 1; use 10ish for production; 2 to 4 times the number of cores
# threads = 1 # default 1; only affects Gthread worker type
workers = os.getenv('AE_APP_WORKERS', None)
threads = os.getenv('AE_APP_THREADS', None)
# umask = '007'
# capture_output = True
# default_proc_name = "run_server:app"
# worker_class = "worker_class"
## From FastAPI for reference
# --bind unix:/home/scott/OSIT_dev/aether_api_fastapi/gunicorn.sock
# --umask 007 app.main:app
# --workers 2
# --worker-class uvicorn.workers.UvicornWorker
# --log-level debug
# --access-logfile admin/log/access.log
# --error-logfile admin/log/error.log
# --log-file admin/log/log.log
# --capture-output
# --keep-alive 5
# --reload

View File

@@ -1,77 +0,0 @@
argon2-cffi==21.3.0
# argon2-cffi-bindings==21.2.0
async-timeout==4.0.2
bidict==0.21.4
Brotli==1.0.9
cachelib==0.9.0
certifi==2021.10.8
# cffi==1.15.0
charset-normalizer==2.0.9
click==7.1.2
cssselect2==0.4.1
# Cython==0.29.32
Deprecated==1.2.13
# dnspython==2.1.0
eventlet==0.33.2
Flask==1.1.4
Flask-Caching==2.0.1
Flask-Cors==3.0.10
Flask-MySQLdb==1.0.1
Flask-SocketIO==5.3.2
Flask-SQLAlchemy==2.5.1
Flask-WeasyPrint==0.6
fonttools==4.28.4
gevent==22.10.2
greenlet==2.0.1
gunicorn==20.1.0
html2text==2020.1.16
html5lib==1.1
idna==3.3
itsdangerous==1.1.0
Jinja2==2.11.3
MarkupSafe==2.0.1
mypy==0.930
mypy-extensions==0.4.3
mysqlclient==2.1.0
numpy
packaging==21.3
pandas==1.5.2
passlib==1.7.4
Pillow==9.3.0
pycparser==2.21
# pycrypto==2.6.1
# pycryptodome==3.16.0
pydantic==1.10.2
pydyf==0.1.2
pyparsing==3.0.9
pyphen==0.11.0
python-dateutil==2.8.2
python-engineio==4.3.0
python-socketio==5.5.0
pytz==2021.3
qrcode==7.3.1
redis==4.3.5
requests==2.28.1
# simple-crypt==4.1.7
six==1.16.0
SQLAlchemy==1.4.28
stripe==5.0.0
suds-py3==1.4.5.0
tinycss2==1.1.1
tomli==2.0.0
types-pytz==2022.1.2
types-requests==2.28.10
types-urllib3==1.26.24
typing_extensions==4.4.0
urllib3==1.26.7
uvicorn==0.18.3
# uWSGI==2.0.21
weasyprint==53.4
webencodings==0.5.1
Werkzeug==0.16.1
wrapt==1.13.3
xlrd==2.0.1
xmltodict==0.13.0
# zope.event==4.5.0
# zope.interface==5.4.0
# zopfli==0.1.9

View File

@@ -2,3 +2,4 @@
*
# Except for this file
!.gitignore
!README.md

20
conf/certs/README.md Normal file
View File

@@ -0,0 +1,20 @@
Copy or create a links to the pem files
* ./conf/certs/oneskyit.com_fullchain.pem
* ./conf/certs/privkey.pem
* ./conf/certs/ssl-dhparams.pem
## Copy certs
### Copy Let's Encrypt
ssl-dhparams.pem should only need to be copied one time. The others need to be copied over after they expire.
Does the ownership or other permissions need to change?
```bash
sudo rsync -vhr -LK --archive -progress /etc/letsencrypt/live/ /home/scott/backups/letsencrypt_live_certs
sudo chown scott:scott -R ~/backups/letsencrypt_live_certs
sudo cp /etc/letsencrypt/live/oneskyit.com-0001/fullchain.pem /srv/env/test_aether/conf/certs/fullchain.pem
sudo cp /etc/letsencrypt/live/oneskyit.com-0001/privkey.pem /srv/env/test_aether/conf/certs/privkey.pem
sudo cp /etc/letsencrypt/ssl-dhparams.pem /srv/env/test_aether/conf/certs/ssl-dhparams.pem
# sudo chown -R scott:scott ~/srv/env/test_aether/certs/
```

2
conf/crontab Normal file
View File

@@ -0,0 +1,2 @@
55 * * * * bash /scripts/backup_internal.sh >> /logs/backup_cron.log 2>&1
0 0 * * * /usr/sbin/logrotate /etc/logrotate.internal.conf

12
conf/logrotate.conf Normal file
View File

@@ -0,0 +1,12 @@
# Logrotate configuration for Aether Docker Logs (Internal container version)
/logs/*/*.log /logs/web/*/*.log {
su aether aether
daily
rotate 7
missingok
notifempty
compress
delaycompress
copytruncate
}

42
conf/mariadb/server.cnf Normal file
View File

@@ -0,0 +1,42 @@
[server]
[mysqld]
# Global defaults for Aether Dev (Overrides are in docker-compose.yml via .env)
slow_query_log = ON
long_query_time = 1
expire_logs_days = 90
max_binlog_size = 512M
wait_timeout = 1800
net_read_timeout = 900
net_write_timeout = 900
interactive_timeout = 1800
connect_timeout = 90
max_allowed_packet = 128M
key_buffer_size = 64M
# Buffer Settings
join_buffer_size = 16M
read_buffer_size = 4M
read_rnd_buffer_size = 8M
sort_buffer_size = 16M
innodb_sort_buffer_size = 32M
innodb_open_files = 4096
query_cache_type = ON
query_cache_limit = 4M
[galera]
[embedded]
[mariadb]
plugin_load_add = server_audit
server_audit=FORCE_PLUS_PERMANENT
server_audit_file_path=/var/log/mysql/mariadb-audit.log
server_audit_logging=ON
server_audit_events = 'CONNECT,TABLE'
# skip-name-resolve helps significantly with Docker network overhead
skip-name-resolve
[mariadb-10.11]

View File

@@ -1,151 +0,0 @@
# DO NOT EDIT: created by update.sh from Dockerfile-debian.template
FROM php:8.1-fpm-bullseye
# entrypoint.sh and cron.sh dependencies
RUN set -ex; \
\
apt-get update; \
apt-get install -y --no-install-recommends \
rsync \
bzip2 \
busybox-static \
libldap-common \
libmagickcore-6.q16-6-extra \
; \
rm -rf /var/lib/apt/lists/*; \
\
mkdir -p /var/spool/cron/crontabs; \
echo '*/5 * * * * php -f /var/www/html/cron.php' > /var/spool/cron/crontabs/www-data
# install the PHP extensions we need
# see https://docs.nextcloud.com/server/stable/admin_manual/installation/source_installation.html
ENV PHP_MEMORY_LIMIT 512M
ENV PHP_UPLOAD_LIMIT 512M
RUN set -ex; \
\
savedAptMark="$(apt-mark showmanual)"; \
\
apt-get update; \
apt-get install -y --no-install-recommends \
libcurl4-openssl-dev \
libevent-dev \
libfreetype6-dev \
libicu-dev \
libjpeg-dev \
libldap2-dev \
libmcrypt-dev \
libmemcached-dev \
libpng-dev \
libpq-dev \
libxml2-dev \
libmagickwand-dev \
libzip-dev \
libwebp-dev \
libgmp-dev \
; \
\
debMultiarch="$(dpkg-architecture --query DEB_BUILD_MULTIARCH)"; \
docker-php-ext-configure gd --with-freetype --with-jpeg --with-webp; \
docker-php-ext-configure ldap --with-libdir="lib/$debMultiarch"; \
docker-php-ext-install -j "$(nproc)" \
bcmath \
exif \
gd \
intl \
ldap \
opcache \
pcntl \
pdo_mysql \
pdo_pgsql \
zip \
gmp \
; \
\
# pecl will claim success even if one install fails, so we need to perform each install separately
pecl install APCu-5.1.22; \
pecl install memcached-3.2.0; \
pecl install redis-5.3.7; \
pecl install imagick-3.7.0; \
\
docker-php-ext-enable \
apcu \
memcached \
redis \
imagick \
; \
rm -r /tmp/pear; \
\
# reset apt-mark's "manual" list so that "purge --auto-remove" will remove all build dependencies
apt-mark auto '.*' > /dev/null; \
apt-mark manual $savedAptMark; \
ldd "$(php -r 'echo ini_get("extension_dir");')"/*.so \
| awk '/=>/ { print $3 }' \
| sort -u \
| xargs -r dpkg-query -S \
| cut -d: -f1 \
| sort -u \
| xargs -rt apt-mark manual; \
\
apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \
rm -rf /var/lib/apt/lists/*
# set recommended PHP.ini settings
# see https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html#enable-php-opcache
RUN { \
echo 'opcache.enable=1'; \
echo 'opcache.interned_strings_buffer=16'; \
echo 'opcache.max_accelerated_files=10000'; \
echo 'opcache.memory_consumption=128'; \
echo 'opcache.save_comments=1'; \
echo 'opcache.revalidate_freq=60'; \
} > "${PHP_INI_DIR}/conf.d/opcache-recommended.ini"; \
\
echo 'apc.enable_cli=1' >> "${PHP_INI_DIR}/conf.d/docker-php-ext-apcu.ini"; \
\
{ \
echo 'memory_limit=${PHP_MEMORY_LIMIT}'; \
echo 'upload_max_filesize=${PHP_UPLOAD_LIMIT}'; \
echo 'post_max_size=${PHP_UPLOAD_LIMIT}'; \
} > "${PHP_INI_DIR}/conf.d/nextcloud.ini"; \
\
mkdir /var/www/data; \
chown -R www-data:root /var/www; \
chmod -R g=u /var/www
VOLUME /var/www/html
ENV NEXTCLOUD_VERSION 25.0.2
RUN set -ex; \
fetchDeps=" \
gnupg \
dirmngr \
"; \
apt-get update; \
apt-get install -y --no-install-recommends $fetchDeps; \
\
curl -fsSL -o nextcloud.tar.bz2 \
"https://download.nextcloud.com/server/releases/nextcloud-${NEXTCLOUD_VERSION}.tar.bz2"; \
curl -fsSL -o nextcloud.tar.bz2.asc \
"https://download.nextcloud.com/server/releases/nextcloud-${NEXTCLOUD_VERSION}.tar.bz2.asc"; \
export GNUPGHOME="$(mktemp -d)"; \
# gpg key from https://nextcloud.com/nextcloud.asc
gpg --batch --keyserver keyserver.ubuntu.com --recv-keys 28806A878AE423A28372792ED75899B9A724937A; \
gpg --batch --verify nextcloud.tar.bz2.asc nextcloud.tar.bz2; \
tar -xjf nextcloud.tar.bz2 -C /usr/src/; \
gpgconf --kill all; \
rm nextcloud.tar.bz2.asc nextcloud.tar.bz2; \
rm -rf "$GNUPGHOME" /usr/src/nextcloud/updater; \
mkdir -p /usr/src/nextcloud/data; \
mkdir -p /usr/src/nextcloud/custom_apps; \
chmod +x /usr/src/nextcloud/occ; \
\
apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false $fetchDeps; \
rm -rf /var/lib/apt/lists/*
COPY *.sh upgrade.exclude /
COPY config/* /usr/src/nextcloud/config/
ENTRYPOINT ["/entrypoint.sh"]
CMD ["php-fpm"]

View File

@@ -1,4 +0,0 @@
<?php
$CONFIG = array (
'memcache.local' => '\OC\Memcache\APCu',
);

View File

@@ -1,15 +0,0 @@
<?php
$CONFIG = array (
'apps_paths' => array (
0 => array (
'path' => OC::$SERVERROOT.'/apps',
'url' => '/apps',
'writable' => false,
),
1 => array (
'path' => OC::$SERVERROOT.'/custom_apps',
'url' => '/custom_apps',
'writable' => true,
),
),
);

View File

@@ -1,41 +0,0 @@
<?php
$autoconfig_enabled = false;
if (getenv('SQLITE_DATABASE')) {
$AUTOCONFIG['dbtype'] = 'sqlite';
$AUTOCONFIG['dbname'] = getenv('SQLITE_DATABASE');
$autoconfig_enabled = true;
} elseif (getenv('MYSQL_DATABASE_FILE') && getenv('MYSQL_USER_FILE') && getenv('MYSQL_PASSWORD_FILE') && getenv('MYSQL_HOST')) {
$AUTOCONFIG['dbtype'] = 'mysql';
$AUTOCONFIG['dbname'] = trim(file_get_contents(getenv('MYSQL_DATABASE_FILE')));
$AUTOCONFIG['dbuser'] = trim(file_get_contents(getenv('MYSQL_USER_FILE')));
$AUTOCONFIG['dbpass'] = trim(file_get_contents(getenv('MYSQL_PASSWORD_FILE')));
$AUTOCONFIG['dbhost'] = getenv('MYSQL_HOST');
$autoconfig_enabled = true;
} elseif (getenv('MYSQL_DATABASE') && getenv('MYSQL_USER') && getenv('MYSQL_PASSWORD') && getenv('MYSQL_HOST')) {
$AUTOCONFIG['dbtype'] = 'mysql';
$AUTOCONFIG['dbname'] = getenv('MYSQL_DATABASE');
$AUTOCONFIG['dbuser'] = getenv('MYSQL_USER');
$AUTOCONFIG['dbpass'] = getenv('MYSQL_PASSWORD');
$AUTOCONFIG['dbhost'] = getenv('MYSQL_HOST');
$autoconfig_enabled = true;
} elseif (getenv('POSTGRES_DB_FILE') && getenv('POSTGRES_USER_FILE') && getenv('POSTGRES_PASSWORD_FILE') && getenv('POSTGRES_HOST')) {
$AUTOCONFIG['dbtype'] = 'pgsql';
$AUTOCONFIG['dbname'] = trim(file_get_contents(getenv('POSTGRES_DB_FILE')));
$AUTOCONFIG['dbuser'] = trim(file_get_contents(getenv('POSTGRES_USER_FILE')));
$AUTOCONFIG['dbpass'] = trim(file_get_contents(getenv('POSTGRES_PASSWORD_FILE')));
$AUTOCONFIG['dbhost'] = getenv('POSTGRES_HOST');
$autoconfig_enabled = true;
} elseif (getenv('POSTGRES_DB') && getenv('POSTGRES_USER') && getenv('POSTGRES_PASSWORD') && getenv('POSTGRES_HOST')) {
$AUTOCONFIG['dbtype'] = 'pgsql';
$AUTOCONFIG['dbname'] = getenv('POSTGRES_DB');
$AUTOCONFIG['dbuser'] = getenv('POSTGRES_USER');
$AUTOCONFIG['dbpass'] = getenv('POSTGRES_PASSWORD');
$AUTOCONFIG['dbhost'] = getenv('POSTGRES_HOST');
$autoconfig_enabled = true;
}
if ($autoconfig_enabled) {
$AUTOCONFIG['directory'] = getenv('NEXTCLOUD_DATA_DIR') ?: '/var/www/html/data';
}

View File

@@ -1,17 +0,0 @@
<?php
if (getenv('REDIS_HOST')) {
$CONFIG = array(
'memcache.distributed' => '\OC\Memcache\Redis',
'memcache.locking' => '\OC\Memcache\Redis',
'redis' => array(
'host' => getenv('REDIS_HOST'),
'password' => (string) getenv('REDIS_HOST_PASSWORD'),
),
);
if (getenv('REDIS_HOST_PORT') !== false) {
$CONFIG['redis']['port'] = (int) getenv('REDIS_HOST_PORT');
} elseif (getenv('REDIS_HOST')[0] != '/') {
$CONFIG['redis']['port'] = 6379;
}
}

View File

@@ -1,30 +0,0 @@
<?php
$overwriteHost = getenv('OVERWRITEHOST');
if ($overwriteHost) {
$CONFIG['overwritehost'] = $overwriteHost;
}
$overwriteProtocol = getenv('OVERWRITEPROTOCOL');
if ($overwriteProtocol) {
$CONFIG['overwriteprotocol'] = $overwriteProtocol;
}
$overwriteCliUrl = getenv('OVERWRITECLIURL');
if ($overwriteCliUrl) {
$CONFIG['overwrite.cli.url'] = $overwriteCliUrl;
}
$overwriteWebRoot = getenv('OVERWRITEWEBROOT');
if ($overwriteWebRoot) {
$CONFIG['overwritewebroot'] = $overwriteWebRoot;
}
$overwriteCondAddr = getenv('OVERWRITECONDADDR');
if ($overwriteCondAddr) {
$CONFIG['overwritecondaddr'] = $overwriteCondAddr;
}
$trustedProxies = getenv('TRUSTED_PROXIES');
if ($trustedProxies) {
$CONFIG['trusted_proxies'] = array_filter(array_map('trim', explode(' ', $trustedProxies)));
}

View File

@@ -1,27 +0,0 @@
<?php
if (getenv('OBJECTSTORE_S3_BUCKET')) {
$use_ssl = getenv('OBJECTSTORE_S3_SSL');
$use_path = getenv('OBJECTSTORE_S3_USEPATH_STYLE');
$use_legacyauth = getenv('OBJECTSTORE_S3_LEGACYAUTH');
$autocreate = getenv('OBJECTSTORE_S3_AUTOCREATE');
$CONFIG = array(
'objectstore' => array(
'class' => '\OC\Files\ObjectStore\S3',
'arguments' => array(
'bucket' => getenv('OBJECTSTORE_S3_BUCKET'),
'key' => getenv('OBJECTSTORE_S3_KEY') ?: '',
'secret' => getenv('OBJECTSTORE_S3_SECRET') ?: '',
'region' => getenv('OBJECTSTORE_S3_REGION') ?: '',
'hostname' => getenv('OBJECTSTORE_S3_HOST') ?: '',
'port' => getenv('OBJECTSTORE_S3_PORT') ?: '',
'objectPrefix' => getenv("OBJECTSTORE_S3_OBJECT_PREFIX") ? getenv("OBJECTSTORE_S3_OBJECT_PREFIX") : "urn:oid:",
'autocreate' => (strtolower($autocreate) === 'false' || $autocreate == false) ? false : true,
'use_ssl' => (strtolower($use_ssl) === 'false' || $use_ssl == false) ? false : true,
// required for some non Amazon S3 implementations
'use_path_style' => $use_path == true && strtolower($use_path) !== 'false',
// required for older protocol versions
'legacy_auth' => $use_legacyauth == true && strtolower($use_legacyauth) !== 'false'
)
)
);
}

View File

@@ -1,22 +0,0 @@
<?php
if (getenv('SMTP_HOST') && getenv('MAIL_FROM_ADDRESS') && getenv('MAIL_DOMAIN')) {
$CONFIG = array (
'mail_smtpmode' => 'smtp',
'mail_smtphost' => getenv('SMTP_HOST'),
'mail_smtpport' => getenv('SMTP_PORT') ?: (getenv('SMTP_SECURE') ? 465 : 25),
'mail_smtpsecure' => getenv('SMTP_SECURE') ?: '',
'mail_smtpauth' => getenv('SMTP_NAME') && (getenv('SMTP_PASSWORD') || (getenv('SMTP_PASSWORD_FILE') && file_exists(getenv('SMTP_PASSWORD_FILE')))),
'mail_smtpauthtype' => getenv('SMTP_AUTHTYPE') ?: 'LOGIN',
'mail_smtpname' => getenv('SMTP_NAME') ?: '',
'mail_from_address' => getenv('MAIL_FROM_ADDRESS'),
'mail_domain' => getenv('MAIL_DOMAIN'),
);
if (getenv('SMTP_PASSWORD_FILE') && file_exists(getenv('SMTP_PASSWORD_FILE'))) {
$CONFIG['mail_smtppassword'] = trim(file_get_contents(getenv('SMTP_PASSWORD_FILE')));
} elseif (getenv('SMTP_PASSWORD')) {
$CONFIG['mail_smtppassword'] = getenv('SMTP_PASSWORD');
} else {
$CONFIG['mail_smtppassword'] = '';
}
}

View File

@@ -1,31 +0,0 @@
<?php
if (getenv('OBJECTSTORE_SWIFT_URL')) {
$autocreate = getenv('OBJECTSTORE_SWIFT_AUTOCREATE');
$CONFIG = array(
'objectstore' => [
'class' => 'OC\\Files\\ObjectStore\\Swift',
'arguments' => [
'autocreate' => $autocreate == true && strtolower($autocreate) !== 'false',
'user' => [
'name' => getenv('OBJECTSTORE_SWIFT_USER_NAME'),
'password' => getenv('OBJECTSTORE_SWIFT_USER_PASSWORD'),
'domain' => [
'name' => (getenv('OBJECTSTORE_SWIFT_USER_DOMAIN')) ?: 'Default',
],
],
'scope' => [
'project' => [
'name' => getenv('OBJECTSTORE_SWIFT_PROJECT_NAME'),
'domain' => [
'name' => (getenv('OBJECTSTORE_SWIFT_PROJECT_DOMAIN')) ?: 'Default',
],
],
],
'serviceName' => (getenv('OBJECTSTORE_SWIFT_SERVICE_NAME')) ?: 'swift',
'region' => getenv('OBJECTSTORE_SWIFT_REGION'),
'url' => getenv('OBJECTSTORE_SWIFT_URL'),
'bucket' => getenv('OBJECTSTORE_SWIFT_CONTAINER_NAME'),
]
]
);
}

View File

@@ -1,4 +0,0 @@
#!/bin/sh
set -eu
exec busybox crond -f -l 0 -L /dev/stdout

View File

@@ -1,250 +0,0 @@
#!/bin/sh
set -eu
# version_greater A B returns whether A > B
version_greater() {
[ "$(printf '%s\n' "$@" | sort -t '.' -n -k1,1 -k2,2 -k3,3 -k4,4 | head -n 1)" != "$1" ]
}
# return true if specified directory is empty
directory_empty() {
[ -z "$(ls -A "$1/")" ]
}
run_as() {
if [ "$(id -u)" = 0 ]; then
su -p "$user" -s /bin/sh -c "$1"
else
sh -c "$1"
fi
}
# usage: file_env VAR [DEFAULT]
# ie: file_env 'XYZ_DB_PASSWORD' 'example'
# (will allow for "$XYZ_DB_PASSWORD_FILE" to fill in the value of
# "$XYZ_DB_PASSWORD" from a file, especially for Docker's secrets feature)
file_env() {
local var="$1"
local fileVar="${var}_FILE"
local def="${2:-}"
local varValue=$(env | grep -E "^${var}=" | sed -E -e "s/^${var}=//")
local fileVarValue=$(env | grep -E "^${fileVar}=" | sed -E -e "s/^${fileVar}=//")
if [ -n "${varValue}" ] && [ -n "${fileVarValue}" ]; then
echo >&2 "error: both $var and $fileVar are set (but are exclusive)"
exit 1
fi
if [ -n "${varValue}" ]; then
export "$var"="${varValue}"
elif [ -n "${fileVarValue}" ]; then
export "$var"="$(cat "${fileVarValue}")"
elif [ -n "${def}" ]; then
export "$var"="$def"
fi
unset "$fileVar"
}
if expr "$1" : "apache" 1>/dev/null; then
if [ -n "${APACHE_DISABLE_REWRITE_IP+x}" ]; then
a2disconf remoteip
fi
fi
if expr "$1" : "apache" 1>/dev/null || [ "$1" = "php-fpm" ] || [ "${NEXTCLOUD_UPDATE:-0}" -eq 1 ]; then
uid="$(id -u)"
gid="$(id -g)"
if [ "$uid" = '0' ]; then
case "$1" in
apache2*)
user="${APACHE_RUN_USER:-http}"
group="${APACHE_RUN_GROUP:-http}"
# strip off any '#' symbol ('#1000' is valid syntax for Apache)
user="${user#'#'}"
group="${group#'#'}"
;;
*) # php-fpm
user='http'
group='http'
;;
esac
else
user="$uid"
group="$gid"
fi
if [ -n "${REDIS_HOST+x}" ]; then
echo "Configuring Redis as session handler"
{
file_env REDIS_HOST_PASSWORD
echo 'session.save_handler = redis'
# check if redis host is an unix socket path
if [ "$(echo "$REDIS_HOST" | cut -c1-1)" = "/" ]; then
if [ -n "${REDIS_HOST_PASSWORD+x}" ]; then
echo "session.save_path = \"unix://${REDIS_HOST}?auth=${REDIS_HOST_PASSWORD}\""
else
echo "session.save_path = \"unix://${REDIS_HOST}\""
fi
# check if redis password has been set
elif [ -n "${REDIS_HOST_PASSWORD+x}" ]; then
echo "session.save_path = \"tcp://${REDIS_HOST}:${REDIS_HOST_PORT:=6379}?auth=${REDIS_HOST_PASSWORD}\""
else
echo "session.save_path = \"tcp://${REDIS_HOST}:${REDIS_HOST_PORT:=6379}\""
fi
echo "redis.session.locking_enabled = 1"
echo "redis.session.lock_retries = -1"
# redis.session.lock_wait_time is specified in microseconds.
# Wait 10ms before retrying the lock rather than the default 2ms.
echo "redis.session.lock_wait_time = 10000"
} > /usr/local/etc/php/conf.d/redis-session.ini
fi
installed_version="0.0.0.0"
if [ -f /var/www/html/version.php ]; then
# shellcheck disable=SC2016
installed_version="$(php -r 'require "/var/www/html/version.php"; echo implode(".", $OC_Version);')"
fi
# shellcheck disable=SC2016
image_version="$(php -r 'require "/usr/src/nextcloud/version.php"; echo implode(".", $OC_Version);')"
if version_greater "$installed_version" "$image_version"; then
echo "Can't start Nextcloud because the version of the data ($installed_version) is higher than the docker image version ($image_version) and downgrading is not supported. Are you sure you have pulled the newest image version?"
exit 1
fi
if version_greater "$image_version" "$installed_version"; then
echo "Initializing nextcloud $image_version ..."
if [ "$installed_version" != "0.0.0.0" ]; then
echo "Upgrading nextcloud from $installed_version ..."
run_as 'php /var/www/html/occ app:list' | sed -n "/Enabled:/,/Disabled:/p" > /tmp/list_before
fi
if [ "$(id -u)" = 0 ]; then
rsync_options="-rlDog --chown $user:$group"
else
rsync_options="-rlD"
fi
# If another process is syncing the html folder, wait for
# it to be done, then escape initalization.
# You need to define the NEXTCLOUD_INIT_LOCK environment variable
lock=/var/www/html/nextcloud-init-sync.lock
count=0
limit=10
if [ -f "$lock" ] && [ -n "${NEXTCLOUD_INIT_LOCK+x}" ]; then
until [ ! -f "$lock" ] || [ "$count" -gt "$limit" ]
do
count=$((count+1))
wait=$((count*10))
echo "Another process is initializing Nextcloud. Waiting $wait seconds..."
sleep $wait
done
if [ "$count" -gt "$limit" ]; then
echo "Timeout while waiting for an ongoing initialization"
exit 1
fi
echo "The other process is done, assuming complete initialization"
else
# Prevent multiple images syncing simultaneously
touch $lock
rsync $rsync_options --delete --exclude-from=/upgrade.exclude /usr/src/nextcloud/ /var/www/html/
for dir in config data custom_apps themes; do
if [ ! -d "/var/www/html/$dir" ] || directory_empty "/var/www/html/$dir"; then
rsync $rsync_options --include "/$dir/" --exclude '/*' /usr/src/nextcloud/ /var/www/html/
fi
done
rsync $rsync_options --include '/version.php' --exclude '/*' /usr/src/nextcloud/ /var/www/html/
# Install
if [ "$installed_version" = "0.0.0.0" ]; then
echo "New nextcloud instance"
file_env NEXTCLOUD_ADMIN_PASSWORD
file_env NEXTCLOUD_ADMIN_USER
if [ -n "${NEXTCLOUD_ADMIN_USER+x}" ] && [ -n "${NEXTCLOUD_ADMIN_PASSWORD+x}" ]; then
# shellcheck disable=SC2016
install_options='-n --admin-user "$NEXTCLOUD_ADMIN_USER" --admin-pass "$NEXTCLOUD_ADMIN_PASSWORD"'
if [ -n "${NEXTCLOUD_DATA_DIR+x}" ]; then
# shellcheck disable=SC2016
install_options=$install_options' --data-dir "$NEXTCLOUD_DATA_DIR"'
fi
file_env MYSQL_DATABASE
file_env MYSQL_PASSWORD
file_env MYSQL_USER
file_env POSTGRES_DB
file_env POSTGRES_PASSWORD
file_env POSTGRES_USER
install=false
if [ -n "${SQLITE_DATABASE+x}" ]; then
echo "Installing with SQLite database"
# shellcheck disable=SC2016
install_options=$install_options' --database-name "$SQLITE_DATABASE"'
install=true
elif [ -n "${MYSQL_DATABASE+x}" ] && [ -n "${MYSQL_USER+x}" ] && [ -n "${MYSQL_PASSWORD+x}" ] && [ -n "${MYSQL_HOST+x}" ]; then
echo "Installing with MySQL database"
# shellcheck disable=SC2016
install_options=$install_options' --database mysql --database-name "$MYSQL_DATABASE" --database-user "$MYSQL_USER" --database-pass "$MYSQL_PASSWORD" --database-host "$MYSQL_HOST"'
install=true
elif [ -n "${POSTGRES_DB+x}" ] && [ -n "${POSTGRES_USER+x}" ] && [ -n "${POSTGRES_PASSWORD+x}" ] && [ -n "${POSTGRES_HOST+x}" ]; then
echo "Installing with PostgreSQL database"
# shellcheck disable=SC2016
install_options=$install_options' --database pgsql --database-name "$POSTGRES_DB" --database-user "$POSTGRES_USER" --database-pass "$POSTGRES_PASSWORD" --database-host "$POSTGRES_HOST"'
install=true
fi
if [ "$install" = true ]; then
echo "Starting nextcloud installation"
max_retries=10
try=0
until run_as "php /var/www/html/occ maintenance:install $install_options" || [ "$try" -gt "$max_retries" ]
do
echo "Retrying install..."
try=$((try+1))
sleep 10s
done
if [ "$try" -gt "$max_retries" ]; then
echo "Installing of nextcloud failed!"
exit 1
fi
if [ -n "${NEXTCLOUD_TRUSTED_DOMAINS+x}" ]; then
echo "Setting trusted domains…"
NC_TRUSTED_DOMAIN_IDX=1
for DOMAIN in $NEXTCLOUD_TRUSTED_DOMAINS ; do
DOMAIN=$(echo "$DOMAIN" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')
run_as "php /var/www/html/occ config:system:set trusted_domains $NC_TRUSTED_DOMAIN_IDX --value=$DOMAIN"
NC_TRUSTED_DOMAIN_IDX=$((NC_TRUSTED_DOMAIN_IDX+1))
done
fi
else
echo "Please run the web-based installer on first connect!"
fi
fi
# Upgrade
else
run_as 'php /var/www/html/occ upgrade'
run_as 'php /var/www/html/occ app:list' | sed -n "/Enabled:/,/Disabled:/p" > /tmp/list_after
echo "The following apps have been disabled:"
diff /tmp/list_before /tmp/list_after | grep '<' | cut -d- -f2 | cut -d: -f1
rm -f /tmp/list_before /tmp/list_after
fi
# Initialization done, reset lock
rm $lock
echo "Initializing finished"
fi
fi
# Update htaccess after init if requested
if [ -n "${NEXTCLOUD_INIT_HTACCESS+x}" ] && [ "$installed_version" != "0.0.0.0" ]; then
run_as 'php /var/www/html/occ maintenance:update:htaccess'
fi
fi
exec "$@"

View File

@@ -1,6 +0,0 @@
/config/
/data/
/custom_apps/
/themes/
/version.php
/nextcloud-init-sync.lock

View File

@@ -1,20 +0,0 @@
server {
listen 80;
listen [::]:80;
server_name mailman2-oneskyit.localhost mailman2.oneskyit.com;
access_log /logs/nginx/access_oneskyit_mailman2.log;
index index.php;
location / {
proxy_pass http://mailman2:80;
proxy_pass_header Content-Type;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade; # allow websockets
proxy_pass_header Connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
}
}

View File

@@ -1,194 +0,0 @@
upstream php-handler {
server nextcloud25:9000;
# server unix:/var/run/php/php7.4-fpm.sock;
}
# Set the `immutable` cache control options only for assets with a cache busting `v` argument
map $arg_v $asset_immutable {
"" "";
default "immutable";
}
server {
listen 80;
listen [::]:80;
server_name nextcloud.oneskyit.com;
# Prevent nginx HTTP Server Detection
server_tokens off;
# Enforce HTTPS
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name nextcloud.oneskyit.com;
access_log /logs/nginx/access_oneskyit_nextcloud.log;
# Path to the root of your installation
# root /srv/nextcloud;
root /var/www/html;
# Use Mozilla's guidelines for SSL/TLS settings
# https://mozilla.github.io/server-side-tls/ssl-config-generator/
# ssl_certificate /etc/ssl/nginx/nextcloud.oneskyit.com.crt;
# ssl_certificate_key /etc/ssl/nginx/nextcloud.oneskyit.com.key;
include /etc/nginx/options-ssl-nginx.conf;
ssl_certificate /etc/certs/fullchain.pem;
ssl_certificate_key /etc/certs/privkey.pem;
ssl_dhparam /etc/certs/ssl-dhparams.pem;
# Prevent nginx HTTP Server Detection
server_tokens off;
# HSTS settings
# WARNING: Only add the preload option once you read about
# the consequences in https://hstspreload.org/. This option
# will add the domain to a hardcoded list that is shipped
# in all major browsers and getting removed from this list
# could take several months.
#add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload" always;
# set max upload size and increase upload timeout:
client_max_body_size 512M;
client_body_timeout 300s;
fastcgi_buffers 64 4K;
# Enable gzip but do not remove ETag headers
gzip on;
gzip_vary on;
gzip_comp_level 4;
gzip_min_length 256;
gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/wasm application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;
# Pagespeed is not supported by Nextcloud, so if your server is built
# with the `ngx_pagespeed` module, uncomment this line to disable it.
#pagespeed off;
# The settings allows you to optimize the HTTP2 bandwitdth.
# See https://blog.cloudflare.com/delivering-http-2-upload-speed-improvements/
# for tunning hints
client_body_buffer_size 512k;
# HTTP response headers borrowed from Nextcloud `.htaccess`
add_header Referrer-Policy "no-referrer" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-Download-Options "noopen" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Permitted-Cross-Domain-Policies "none" always;
add_header X-Robots-Tag "none" always;
add_header X-XSS-Protection "1; mode=block" always;
# Remove X-Powered-By, which is an information leak
fastcgi_hide_header X-Powered-By;
# Specify how to handle directories -- specifying `/index.php$request_uri`
# here as the fallback means that Nginx always exhibits the desired behaviour
# when a client requests a path that corresponds to a directory that exists
# on the server. In particular, if that directory contains an index.php file,
# that file is correctly served; if it doesn't, then the request is passed to
# the front-end controller. This consistent behaviour means that we don't need
# to specify custom rules for certain paths (e.g. images and other assets,
# `/updater`, `/ocm-provider`, `/ocs-provider`), and thus
# `try_files $uri $uri/ /index.php$request_uri`
# always provides the desired behaviour.
index index.php index.html /index.php$request_uri;
# Rule borrowed from `.htaccess` to handle Microsoft DAV clients
location = / {
if ( $http_user_agent ~ ^DavClnt ) {
return 302 /remote.php/webdav/$is_args$args;
}
}
location = /robots.txt {
allow all;
log_not_found off;
access_log off;
}
# Make a regex exception for `/.well-known` so that clients can still
# access it despite the existence of the regex rule
# `location ~ /(\.|autotest|...)` which would otherwise handle requests
# for `/.well-known`.
location ^~ /.well-known {
# The rules in this block are an adaptation of the rules
# in `.htaccess` that concern `/.well-known`.
location = /.well-known/carddav { return 301 /remote.php/dav/; }
location = /.well-known/caldav { return 301 /remote.php/dav/; }
location /.well-known/acme-challenge { try_files $uri $uri/ =404; }
location /.well-known/pki-validation { try_files $uri $uri/ =404; }
# Let Nextcloud's API for `/.well-known` URIs handle all other
# requests by passing them to the front-end controller.
return 301 /index.php$request_uri;
}
# Rules borrowed from `.htaccess` to hide certain paths from clients
location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/) { return 404; }
location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) { return 404; }
# Ensure this block, which passes PHP files to the PHP process, is above the blocks
# which handle static assets (as seen below). If this block is not declared first,
# then Nginx will encounter an infinite rewriting loop when it prepends `/index.php`
# to the URI, resulting in a HTTP 500 error response.
location ~ \.php(?:$|/) {
# Required for legacy support
rewrite ^/(?!index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+|.+\/richdocumentscode\/proxy) /index.php$request_uri;
fastcgi_split_path_info ^(.+?\.php)(/.*)$;
set $path_info $fastcgi_path_info;
try_files $fastcgi_script_name =404;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
# fastcgi_param SCRIPT_FILENAME /var/www/html/$fastcgi_script_name;
fastcgi_param PATH_INFO $path_info;
fastcgi_param HTTPS on;
fastcgi_param modHeadersAvailable true; # Avoid sending the security headers twice
fastcgi_param front_controller_active true; # Enable pretty urls
fastcgi_pass php-handler;
fastcgi_intercept_errors on;
fastcgi_request_buffering off;
fastcgi_max_temp_file_size 0;
}
location ~ \.(?:css|js|svg|gif|png|jpg|ico|wasm|tflite|map)$ {
try_files $uri /index.php$request_uri;
add_header Cache-Control "public, max-age=15778463, $asset_immutable";
access_log off; # Optional: Don't log access to assets
location ~ \.wasm$ {
default_type application/wasm;
}
}
location ~ \.woff2?$ {
try_files $uri /index.php$request_uri;
expires 7d; # Cache-Control policy borrowed from `.htaccess`
access_log off; # Optional: Don't log access to assets
}
# Rule borrowed from `.htaccess`
location /remote {
return 301 /remote.php$request_uri;
}
location / {
try_files $uri $uri/ /index.php$request_uri;
}
}

View File

@@ -2,7 +2,8 @@ server {
listen 80;
listen [::]:80;
server_name phpmyadmin-oneskyit.localhost phpmyadmin.oneskyit.com dev-phpmyadmin.oneskyit.com test-phpmyadmin.oneskyit.com;
server_name ${DOCKER_PHPMYADMIN_SERVER_NAME};
# server_name phpmyadmin-oneskyit.localhost phpmyadmin.oneskyit.com dev-phpmyadmin.oneskyit.com test-phpmyadmin.oneskyit.com;
access_log /logs/nginx/access_oneskyit_phpmyadmin.log;
@@ -23,12 +24,20 @@ server {
server {
listen 443 ssl;
listen [::]:443 ssl http2;
listen [::]:443 ssl;
http2 on;
# server_name ${DOCKER_PHPMYADMIN_SERVER_NAME};
server_name phpmyadmin-oneskyit.localhost phpmyadmin.oneskyit.com dev-phpmyadmin.oneskyit.com test-phpmyadmin.oneskyit.com;
access_log /logs/nginx/access_oneskyit_phpmyadmin.log;
include /etc/nginx/options-ssl-nginx.conf;
ssl_certificate /etc/certs/fullchain.pem;
ssl_certificate_key /etc/certs/privkey.pem;
ssl_dhparam /etc/certs/ssl-dhparams.pem;
index index.php;
location / {

View File

@@ -0,0 +1,82 @@
server {
listen 80;
listen [::]:80;
server_name
${DOCKER_AE_APP_SERVER_NAME}
~^(dev|test|bak|sk|sr|prod)?-?(app|demo|connect|aacc|aapor|ascm|axonius|bgh|businessgroup|chow|cmsc|idaa|ishlt|lci|ncsd|npa|rli|scott|dgr)\.oneskyit\.com$
app.localhost
demo.localhost
connect.localhost
svelte.localhost
dev.localhost
;
access_log /logs/nginx/access_svelte_node.log;
error_log /logs/nginx/error_svelte_node.log;
client_max_body_size ${OSIT_WEB_MAX_BODY_SIZE};
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
proxy_buffering off;
proxy_read_timeout 1500s;
proxy_pass http://svelte_backend;
}
}
# server {
# listen 443 ssl;
# listen [::]:443 ssl;
# http2 on;
#
# server_name
# ${DOCKER_AE_APP_SERVER_NAME}
# ~^(dev|test|bak|sr|prod)?-?(app|demo|connect|aacc|aapor|ascm|businessgroup|chow|cmsc|idaa|ishlt|lci|ncsd|npa|rli)\.oneskyit\.com$
# app.localhost
# demo.localhost
# connect.localhost
# svelte.localhost
# dev.localhost
# localhost
# ;
#
# access_log /logs/nginx/access_svelte_node.log;
# error_log /logs/nginx/error_svelte_node.log;
#
# include /etc/nginx/options-ssl-nginx.conf;
#
# ssl_certificate /etc/certs/fullchain_wild.pem;
# ssl_certificate_key /etc/certs/privkey_wild.pem;
# ssl_dhparam /etc/certs/ssl-dhparams.pem;
#
# client_max_body_size ${OSIT_WEB_MAX_BODY_SIZE};
#
# location / {
# proxy_set_header Host $http_host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header X-Forwarded-Proto $scheme;
#
# proxy_redirect off;
# proxy_buffering off;
#
# proxy_read_timeout 1500s;
#
# proxy_pass http://svelte_backend;
# }
# }
upstream svelte_backend {
ip_hash;
server ae_app:3000 weight=20 max_fails=3 fail_timeout=30s;
}

View File

@@ -1,103 +1,220 @@
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 80;
listen [::]:80;
server_name
fastapi_gunicorn.localhost
dev-api.localhost
dev-api.oneskyit.com
test-api.oneskyit.com
${DOCKER_AE_API_SERVER_NAME}
~^(dev|test|bak|sr|prod)?-?(api|fastapi)\.oneskyit\.com$
api.localhost
fastapi.localhost
;
# server_name
# fastapi_gunicorn.localhost
# dev-api.localhost
# dev-api.oneskyit.com
# test-api.oneskyit.com
# ;
access_log /logs/nginx/access_fastapi_gunicorn.log;
error_log /logs/nginx/error_fastapi_gunicorn.log;
client_max_body_size 5120M; #4096M or 4G; 5120M or 5G;
client_max_body_size ${OSIT_WEB_MAX_BODY_SIZE}; # 5120M; #4096M or 4G; 5120M or 5G;
location / {
# Based on recommendations here: https://www.uvicorn.org/deployment/#running-behind-nginx
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_redirect off;
proxy_buffering off;
# I think "X-Real-IP" might be needed for some things?
proxy_set_header X-Real-IP $remote_addr;
# This is needed for long running Python code. Default is 60 seconds
# Increased from 1200 to 1500 on 2022-04-17
# Increased from 1500 to 2000 on 2023-03-15
# Increased proxy read timeout to 2100 and decreased fastcgi options to 35s on 2023-03-16
# fastcgi_connect_timeout 4s;
# fastcgi_send_timeout 5s;
# fastcgi_read_timeout 5s;
# proxy read timeout being too low will cause 504 Gateway Time-out on the client browser
proxy_read_timeout 2100s;
# proxy_send_timeout and send_timeout default to 60s. For long-running endpoints
# (clip_video, ffmpeg operations that take 5-40 min), raise to match proxy_read_timeout.
proxy_send_timeout 2100s;
send_timeout 2100s;
proxy_pass http://fastapi_backend;
}
location /ws {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_redirect off;
proxy_buffering off;
# This is needed for long running Python code. Default is 60 seconds
# Increased from 1200 to 1500 on 2022-04-17
fastcgi_connect_timeout 1500s;
fastcgi_send_timeout 1500s;
fastcgi_read_timeout 1500s;
proxy_read_timeout 1500s;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# proxy_read_timeout 600;
# proxy_headers_hash_max_size 1024;
proxy_pass http://fastapi_backend;
access_log /logs/nginx/access_fastapi_gunicorn_ws.log;
error_log /logs/nginx/error_fastapi_gunicorn_ws.log;
}
}
server {
listen 443 ssl;
listen [::]:443 ssl http2;
server_name
fastapi_gunicorn.localhost
dev-api.localhost
dev-api.oneskyit.com
test-api.oneskyit.com
;
access_log /logs/nginx/access_fastapi_gunicorn.log;
error_log /logs/nginx/error_fastapi_gunicorn.log;
include /etc/nginx/options-ssl-nginx.conf;
ssl_certificate /etc/certs/fullchain.pem;
ssl_certificate_key /etc/certs/privkey.pem;
ssl_dhparam /etc/certs/ssl-dhparams.pem;
# include brotli.conf;
# include gzip.conf;
client_max_body_size 5120M; #4096M or 4G; 5120M or 5G;
location / {
location /v3/ws {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_redirect off;
proxy_buffering off;
# This is needed for long running Python code. Default is 60 seconds
# Increased from 1200 to 1500 on 2022-04-17
fastcgi_connect_timeout 1500s;
fastcgi_send_timeout 1500s;
fastcgi_read_timeout 1500s;
proxy_read_timeout 1500s;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 2100s;
proxy_pass http://fastapi_backend;
access_log /logs/nginx/access_fastapi_gunicorn_v3_ws.log;
error_log /logs/nginx/error_fastapi_gunicorn_v3_ws.log;
}
}
# server {
# listen 443 ssl;
# listen [::]:443 ssl;
# http2 on;
#
# server_name
# ${DOCKER_AE_API_SERVER_NAME}
# fastapi.localhost
# api.localhost
# localhost
# ;
#
# # server_name
# # fastapi_gunicorn.localhost
# # dev-api.localhost
# # dev-api.oneskyit.com
# # test-api.oneskyit.com
# # ;
#
# access_log /logs/nginx/access_fastapi_gunicorn.log;
# error_log /logs/nginx/error_fastapi_gunicorn.log;
#
# include /etc/nginx/options-ssl-nginx.conf;
#
# ssl_certificate /etc/certs/fullchain_wild.pem;
# ssl_certificate_key /etc/certs/privkey_wild.pem;
# ssl_dhparam /etc/certs/ssl-dhparams.pem;
#
# # include brotli.conf;
# # include gzip.conf;
#
# client_max_body_size ${OSIT_WEB_MAX_BODY_SIZE}; # 5120M; #4096M or 4G; 5120M or 5G;
#
# location / {
# # Based on recommendations here: https://www.uvicorn.org/deployment/#running-behind-nginx
# proxy_set_header Host $http_host;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header X-Forwarded-Proto $scheme;
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection $connection_upgrade;
#
# proxy_redirect off;
# proxy_buffering off;
#
# # I think "X-Real-IP" might be needed for some things?
# proxy_set_header X-Real-IP $remote_addr;
#
# # # This is needed for long running Python code. Default is 60 seconds
# # # Increased from 1200 to 1500 on 2022-04-17
# # # Increased from 1500 to 2000 on 2023-03-15
# # # Increased proxy read timeout to 2100 and decreased fastcgi options to 35s on 2023-03-16
# # fastcgi_connect_timeout 35s;
# # fastcgi_send_timeout 35s;
# # fastcgi_read_timeout 35s;
#
# # proxy read timeout being too low will cause 504 Gateway Time-out on the client browser
# proxy_read_timeout 2100s;
#
# proxy_pass http://fastapi_backend;
# }
#
# location /ws {
# proxy_set_header Host $http_host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header X-Forwarded-Proto $scheme;
#
# proxy_http_version 1.1;
#
# proxy_redirect off;
# proxy_buffering off;
#
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection "upgrade";
#
# # proxy_read_timeout 600;
# # proxy_headers_hash_max_size 1024;
#
# proxy_pass http://fastapi_backend;
#
# access_log /logs/nginx/access_fastapi_gunicorn_ws.log;
# error_log /logs/nginx/error_fastapi_gunicorn_ws.log;
# }
#
# location /v3/ws {
# proxy_set_header Host $http_host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header X-Forwarded-Proto $scheme;
#
# proxy_http_version 1.1;
#
# proxy_redirect off;
# proxy_buffering off;
#
# proxy_set_header Upgrade $http_upgrade;
# proxy_set_header Connection "upgrade";
#
# proxy_read_timeout 2100s;
#
# proxy_pass http://fastapi_backend;
#
# access_log /logs/nginx/access_fastapi_gunicorn_v3_ws.log;
# error_log /logs/nginx/error_fastapi_gunicorn_v3_ws.log;
# }
# }
upstream fastapi_backend {
# sticky sessions
ip_hash;
# enable least connections balancing method
# least_conn;
# zone backend 64k; # Use NGINX Plus' shared memory
# server webserver1 weight=1;
# server webserver2 weight=4;
# larger number will recieve more requests
# Example of 20 vs 10: 20 will recieve twice as many requests as 10
server aether_api_gunicorn:5005 weight=20 max_fails=3 fail_timeout=30s;
# server aether_api_gunicorn_bak:5005 weight=10 max_fails=1 fail_timeout=30s;
# maintain up to 20 idle connections to the group of upstream servers
# keepalive 20;
least_conn;
server ae_api:5005 weight=20 max_fails=1 fail_timeout=5s;
keepalive 10;
}

View File

@@ -1,131 +0,0 @@
server {
listen 80;
listen [::]:80;
server_name
flask_gunicorn.localhost demo.localhost dev.localhost
dev.oneskyit.com
dev-app.oneskyit.com
dev-connect.oneskyit.com *.dev-connect.oneskyit.com
dev-demo.oneskyit.com *.dev-demo.oneskyit.com
dev-aapor.oneskyit.com *.dev-aapor.oneskyit.com
dev-businessgroup.oneskyt.com *.dev-businessgroup.oneskyt.com
dev-cmsc.oneskyit.com *.dev-cmsc.oneskyit.com
dev-idaa.oneskyit.com *.dev-idaa.oneskyit.com
dev-ishlt.oneskyit.com *.dev-ishlt.oneskyit.com
dev-ncsd.oneskyit.com *.dev-ncsd.oneskyit.com
dev-npa.oneskyit.com *.dev-npa.oneskyit.com
dev-rli.oneskyit.com *.dev-rli.oneskyit.com
test-app.oneskyit.com
;
access_log /logs/nginx/access_flask_gunicorn.log;
error_log /logs/nginx/error_flask_gunicorn.log;
client_max_body_size 5120M; #4096M or 4G; 5120M or 5G;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
proxy_buffering off;
# This is needed for long running Python code. Default is 60 seconds
# Increased from 1200 to 1500 on 2022-04-17
fastcgi_connect_timeout 1500s;
fastcgi_send_timeout 1500s;
fastcgi_read_timeout 1500s;
proxy_read_timeout 1500s;
proxy_pass http://flask_backend;
}
}
server {
listen 443 ssl;
listen [::]:443 ssl http2;
server_name
flask_gunicorn.localhost demo.localhost dev.localhost
dev.oneskyit.com
dev-app.oneskyit.com
dev-connect.oneskyit.com *.dev-connect.oneskyit.com
dev-demo.oneskyit.com *.dev-demo.oneskyit.com
dev-aapor.oneskyit.com *.dev-aapor.oneskyit.com
dev-businessgroup.oneskyt.com *.dev-businessgroup.oneskyt.com
dev-cmsc.oneskyit.com *.dev-cmsc.oneskyit.com
dev-idaa.oneskyit.com *.dev-idaa.oneskyit.com
dev-ishlt.oneskyit.com *.dev-ishlt.oneskyit.com
dev-ncsd.oneskyit.com *.dev-ncsd.oneskyit.com
dev-npa.oneskyit.com *.dev-npa.oneskyit.com
dev-rli.oneskyit.com *.dev-rli.oneskyit.com
test-app.oneskyit.com
;
access_log /logs/nginx/access_flask_gunicorn.log;
error_log /logs/nginx/error_flask_gunicorn.log;
include /etc/nginx/options-ssl-nginx.conf;
ssl_certificate /etc/certs/fullchain.pem;
ssl_certificate_key /etc/certs/privkey.pem;
ssl_dhparam /etc/certs/ssl-dhparams.pem;
# include brotli.conf;
# include gzip.conf;
client_max_body_size 5120M; #4096M or 4G; 5120M or 5G;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
proxy_buffering off;
# This is needed for long running Python code. Default is 60 seconds
# Increased from 1200 to 1500 on 2022-04-17
fastcgi_connect_timeout 1500s;
fastcgi_send_timeout 1500s;
fastcgi_read_timeout 1500s;
proxy_read_timeout 1500s;
proxy_pass http://flask_backend;
}
}
upstream flask_backend {
# sticky sessions
ip_hash;
# enable least connections balancing method
# least_conn;
# zone backend 64k; # Use NGINX Plus' shared memory
# server webserver1 weight=1;
# server webserver2 weight=4;
# larger number will recieve more requests
# Example of 20 vs 10: 20 will recieve twice as many requests as 10
server aether_app_gunicorn:5005 weight=20 max_fails=3 fail_timeout=30s;
# server aether_app_gunicorn_bak:5005 weight=10 max_fails=1 fail_timeout=30s;
# maintain up to 20 idle connections to the group of upstream servers
# keepalive 20;
}

View File

@@ -1,150 +1,101 @@
server {
listen 80;
listen [::]:80;
server_name oneskyit.localhost;
listen 80;
listen [::]:80;
access_log /logs/nginx/access_oneskyit.log;
server_name docker.oneskyit.com ${DOCKER_OSIT_SERVER_NAME};
# server_name oneskyit.localhost;
# Do not overflow the SSL send buffer (causes extra round trips)
#ssl_buffer_size 8k;
access_log /logs/nginx/access_oneskyit.log;
root /srv/oneskyit_site;
# Do not overflow the SSL send buffer (causes extra round trips)
#ssl_buffer_size 8k;
index index.php index.html;
# index index.html index.htm index.php;
# include php.conf;
# include brotli.conf;
# include gzip.conf;
# include expires.conf;
# These two locations remove .html and .php from filenames.
location / {
try_files $uri $uri/ $uri.html $uri.php$is_args$query_string;
}
location ~ \.php$ {
root /srv/oneskyit_site;
index index.php index.html;
# index index.html index.htm index.php;
try_files $uri =404;
# try_files $uri $document_root$fastcgi_script_name =404;
# include php.conf;
# include brotli.conf;
# include gzip.conf;
# include expires.conf;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass php7:9000;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
}
# location ~ \.php$ {
# try_files $uri =404;
# }
}
# # Redirect http to https
# server {
# listen 80;
# listen [::]:80;
# server_name oneskyit.com;
# return 301 https://oneskyit.com$request_uri;
# }
#
# #upstream oneskyit {
# #least_conn;
# # ip_hash;
# #server localhost:8889;
# #server localhost:8889;
# #}
#
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name docker.oneskyit.com;
access_log /logs/nginx/access_oneskyit.log;
error_log /logs/nginx/error_oneskyit.log;
# Do not overflow the SSL send buffer (causes extra round trips)
#ssl_buffer_size 8k;
include /etc/nginx/options-ssl-nginx.conf;
ssl_certificate /etc/certs/fullchain.pem;
ssl_certificate_key /etc/certs/privkey.pem;
ssl_dhparam /etc/certs/ssl-dhparams.pem;
root /srv/oneskyit_site;
index index.php index.html;
#
# root /srv/http/oneskyit.com/;
# index index.php index.html;
#
# include php.conf;
# include brotli.conf;
# include gzip.conf;
# include expires.conf;
#
# These two locations remove .html and .php from filenames.
location / {
try_files $uri $uri/ $uri.html $uri.php$is_args$query_string;
try_files $uri $uri/ $uri.html $uri.php$is_args$query_string;
}
location ~ \.php$ {
root /srv/oneskyit_site;
location ~ \.php$ {
root /srv/oneskyit_site;
# index index.html index.htm index.php;
# index index.html index.htm index.php;
try_files $uri =404;
# try_files $uri $document_root$fastcgi_script_name =404;
try_files $uri =404;
# try_files $uri $document_root$fastcgi_script_name =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass php7:9000;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
}
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass php7:9000;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
}
}
# location ~ \.php$ {
# try_files $uri =404;
# }
#
# #location / {
# # include uwsgi_params;
# # #uwsgi_pass oneskyit;
# # proxy_pass http://oneskyit;
# # #uwsgi_pass uwsgi://oneskyit.com:8889;
# #
# # #proxy_pass http://apptest;
# # #uwsgi_pass uwsgi://oneskyit.com:8890;
# # #uwsgi_pass uwsgi://oneskyit.com:8889;
# #}
#
# ssl_certificate /etc/letsencrypt/live/oneskyit.com/fullchain.pem; # managed by Certbot
# #ssl_certificate /etc/letsencrypt/live/oneskyit.com-0001/fullchain.pem; # managed by Certbot
# ssl_certificate_key /etc/letsencrypt/live/oneskyit.com/privkey.pem; # managed by Certbot
# #ssl_certificate_key /etc/letsencrypt/live/oneskyit.com-0001/privkey.pem; # managed by Certbot
# include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
# ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
#
# #ssl_session_cache shared:SSL:5m; # was 1m (1 MB)
# #ssl_session_timeout 1h; # was 5m (5 minutes)
#
# ssl_buffer_size 8k;
#
#
# if ($scheme != "https"){
# return 301 https://$host$request_uri;
# } # managed by Certbot
#
# }
server {
listen 443 ssl;
listen [::]:443 ssl;
http2 on;
# server_name docker.oneskyit.com ${DOCKER_OSIT_SERVER_NAME};
server_name docker.oneskyit.com ${DOCKER_OSIT_SERVER_NAME};
# server_name docker.oneskyit.com dev.oneskyit.com test.oneskyit.com prod.oneskyit.com;
# server_name docker.oneskyit.com;
access_log /logs/nginx/access_oneskyit.log;
error_log /logs/nginx/error_oneskyit.log;
# Do not overflow the SSL send buffer (causes extra round trips)
#ssl_buffer_size 8k;
include /etc/nginx/options-ssl-nginx.conf;
ssl_certificate /etc/certs/fullchain.pem;
ssl_certificate_key /etc/certs/privkey.pem;
ssl_dhparam /etc/certs/ssl-dhparams.pem;
root /srv/oneskyit_site;
index index.php index.html;
#
# root /srv/http/oneskyit.com/;
# index index.php index.html;
#
# include php.conf;
# include brotli.conf;
# include gzip.conf;
# include expires.conf;
#
# These two locations remove .html and .php from filenames.
location / {
try_files $uri $uri/ $uri.html $uri.php$is_args$query_string;
}
location ~ \.php$ {
root /srv/oneskyit_site;
# index index.html index.htm index.php;
try_files $uri =404;
# try_files $uri $document_root$fastcgi_script_name =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass php7:9000;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
}
}

View File

@@ -1,60 +1,38 @@
# Aether Platform - Default Nginx Site Config
# This file handles the default (non-matching) requests.
server {
listen 80;
server_name docker.localhost docker.oneskyit.com;
listen 80 default_server;
server_name _;
error_log /logs/nginx/error_docker.log;
access_log /logs/nginx/access_docker.log;
access_log /logs/nginx/access_docker_default.log;
error_log /logs/nginx/error_docker_default.log;
root /srv/html_php;
index index.html index.htm index.php;
# location / {
# # root /usr/share/nginx/html;
# index index.html index.htm;
# }
location ~ \.php$ {
index index.html index.htm index.php;
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass php7:9000;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
# Just return a 404 for any non-matching domains
location / {
return 404;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
# error_page 500 502 503 504 /50x.html;
# location = /50x.html {
# root /usr/share/nginx/html;
# }
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
# SSL is disabled by default for internal containers.
# If you need SSL termination INSIDE the container, uncomment this block
# and ensure valid certs are in /etc/certs/
#
# server {
# listen 443 ssl;
# listen [::]:443 ssl;
# server_name _;
#
# access_log /logs/nginx/access_docker_ssl.log;
# error_log /logs/nginx/error_docker_ssl.log;
#
# include /etc/nginx/options-ssl-nginx.conf;
#
# ssl_certificate /etc/certs/fullchain.pem;
# ssl_certificate_key /etc/certs/privkey.pem;
# ssl_dhparam /etc/certs/ssl-dhparams.pem;
#
# location / {
# return 404;
# }
# }

88
deploy.sh Executable file
View File

@@ -0,0 +1,88 @@
#!/bin/bash
# deploy.sh — Remote deploy for Aether Platform
# Run on srv-nyx directly, or triggered via SSH from the workstation.
#
# Usage: ./deploy.sh <prod|test> [app_branch] [api_branch]
# Example: ./deploy.sh prod
# ./deploy.sh test ae_app_3x_llm development
#
# From workstation (npm run deploy:remote:prod / deploy:remote:test):
# ssh linode.oneskyit.com 'bash /srv/env/prod_aether/deploy.sh prod'
# ssh linode.oneskyit.com 'bash /srv/env/test_aether/deploy.sh test'
#
# NOTE: bak_aether shares the same app/api dirs as prod.
# After a prod deploy, restart bak containers manually if running:
# cd /srv/env/bak_aether && docker compose restart ae_app ae_api
set -euo pipefail
ENV=${1:-}
if [ -z "$ENV" ]; then
echo "Usage: $0 <prod|test> [app_branch] [api_branch]"
exit 1
fi
# --- Environment config ---
# TODO: Update default branches once prod/test branch strategy is finalized.
# Currently both envs pull from the same working branches.
if [ "$ENV" = "prod" ]; then
APP_DIR=/srv/apps/prod_aether_app_sveltekit
API_DIR=/srv/apps/prod_aether_api_fastapi
COMPOSE_DIR=/srv/env/prod_aether
BUILD_MODE=prod
APP_BRANCH=${2:-ae_app_3x_llm}
API_BRANCH=${3:-development}
elif [ "$ENV" = "test" ]; then
APP_DIR=/srv/apps/test_aether_app_sveltekit
API_DIR=/srv/apps/test_aether_api_fastapi
COMPOSE_DIR=/srv/env/test_aether
BUILD_MODE=test
APP_BRANCH=${2:-ae_app_3x_llm}
API_BRANCH=${3:-development}
else
echo "Unknown environment: '$ENV' (expected: prod or test)"
exit 1
fi
echo ""
echo "========================================"
echo " Aether Deploy: $ENV"
echo " App: $APP_DIR [$APP_BRANCH]"
echo " API: $API_DIR [$API_BRANCH]"
echo " Mode: $BUILD_MODE"
echo "========================================"
echo ""
# --- Pull repos ---
echo "[1/4] Pulling container env..."
git -C "$COMPOSE_DIR" pull --ff-only
echo ""
echo "[2/4] Pulling app ($APP_BRANCH)..."
git -C "$APP_DIR" pull --ff-only origin "$APP_BRANCH"
echo ""
echo "[3/4] Pulling API ($API_BRANCH)..."
git -C "$API_DIR" pull --ff-only origin "$API_BRANCH"
# --- Build and deploy ---
echo ""
echo "[4/4] Building and deploying..."
cd "$COMPOSE_DIR"
docker compose build --build-arg BUILD_MODE="$BUILD_MODE" ae_app
docker compose up -d ae_app
docker compose restart ae_api
echo ""
echo "========================================"
echo " Done: $ENV deployed successfully"
echo "========================================"
if [ "$ENV" = "prod" ]; then
echo ""
echo " bak_aether uses the same code dirs — if its containers"
echo " are running, restart them:"
echo " cd /srv/env/bak_aether && docker compose restart ae_app ae_api"
fi
echo ""

View File

@@ -1,302 +1,243 @@
version: "3.9"
services:
web:
restart: unless-stopped
container_name: ae_web_dev
container_name: ${CONTAINER_WEB}
networks:
- default
- shared
build:
# context: ./builds
context: ./
dockerfile: aether_nginx.Dockerfile
env_file:
- ./.env
environment:
# This does not seem to work (yet???)
- AE_DOMAIN_LIST:'dev-aapor.oneskyit.com dev-businessgroup.oneskyt.com dev-cmsc.oneskyit.com dev-idaa.oneskyit.com dev-ishlt.oneskyit.com dev-ncsd.oneskyit.com dev-npa.oneskyit.com dev-rli.oneskyit.com'
- PUID=1000
- PGID=1000
- TZ=${TZ}
- NGINX_SERVER_NAMES="demo.localhost dev.localhost dev.oneskyit.com dev-app.oneskyit.com dev-connect.oneskyit.com dev-demo.oneskyit.com dev-aacc.oneskyit.com dev-aapor.oneskyit.com dev-ascm.oneskyit.com dev-businessgroup.oneskyt.com dev-chow.oneskyit.com dev-cmsc.oneskyit.com dev-idaa.oneskyit.com dev-ishlt.oneskyit.com dev-lci.oneskyit.com dev-ncsd.oneskyit.com dev-npa.oneskyit.com dev-rli.oneskyit.com test-app.oneskyit.com test-api.oneskyit.com test-demo.oneskyit.com test-lci.oneskyit.com test-idaa.oneskyit.com scott.oneskyit.com dgr.oneskyit.com"
ports:
- "${OSIT_WEB_HTTP_PORT}:80"
- "${OSIT_WEB_HTTPS_PORT}:443"
# - "80:80"
# - "443:443"
# - "8181:80"
# - "8443:443"
# networks:
# - local-net
- "${OSIT_WEB_HTTP_PORT}:80" # LAN HTTP (local access without SSL)
# - "${OSIT_WEB_HTTPS_PORT}:443" # HTTPS — not needed internally; terminate SSL at home server
- "${AE_API_GATEWAY_PORT}:80" # API gateway: home nginx → workstation:5060 → ae_api replicas
- "${AE_APP_GATEWAY_PORT}:80" # App gateway: home nginx → workstation:3001 → ae_app replicas
volumes:
- ./srv/html_php:/srv/html_php
- ./srv/oneskyit_site:/srv/oneskyit_site
- ./srv/hosted_files_ln:/srv/hosted_files
- ./srv/hosted_tmp_ln:/srv/hosted_tmp
# NOTE: Nextcloud Docker container requires (sort of) the path to be /var/www/html
# - ./srv/nextcloud:/srv/nextcloud
# - ./srv/nextcloud/app:/var/www/html
# - ./srv/nextcloud/apps:/var/www/html/apps
# - ./srv/nextcloud/config:/var/www/html/config
# - ./srv/nextcloud/data:/var/www/html/data
- ${HOSTED_FILES_SRC}:/srv/hosted_files
- ${HOSTED_TMP_SRC}:/srv/hosted_tmp
- ./conf/nginx/options-ssl-nginx.conf:/etc/nginx/options-ssl-nginx.conf
- ./conf/nginx/site.conf:/etc/nginx/conf.d/site.conf
# - ./conf/nginx/site-enabled_aether-phpmyadmin.conf:/etc/nginx/conf.d/site-enabled_aether-phpmyadmin.conf
# - ./conf/nginx/site-enabled_aether-mailman2.conf:/etc/nginx/conf.d/site-enabled_aether-mailman2.conf
# - ./conf/nginx/site-enabled_aether-nextcloud.conf:/etc/nginx/conf.d/site-enabled_aether-nextcloud.conf
- ./conf/nginx/site-enabled_oneskyit.conf:/etc/nginx/conf.d/site-enabled_oneskyit.conf
- ./conf/nginx/site-enabled_aether_fastapi_gunicorn.conf:/etc/nginx/conf.d/site-enabled_aether_fastapi_gunicorn.conf
# - ./conf/nginx/site-enabled_aether_fastapi_2_gunicorn.conf:/etc/nginx/conf.d/site-enabled_aether_fastapi_2_gunicorn.conf
- ./conf/nginx/site-enabled_aether_flask_gunicorn.conf:/etc/nginx/conf.d/site-enabled_aether_flask_gunicorn.conf
- ./conf/certs/fullchain.pem:/etc/certs/fullchain.pem
- ./conf/certs/privkey.pem:/etc/certs/privkey.pem
- ./conf/nginx/site.conf:/etc/nginx/conf.d/0_site.conf
- ./conf/nginx/site-enabled_aether_fastapi_gunicorn.conf:/etc/nginx/templates/site-enabled_aether_fastapi_gunicorn.conf.template
- ./conf/nginx/site-enabled_aether_app_svelte_node.conf:/etc/nginx/templates/site-enabled_aether_app_svelte_node.conf.template
- ./conf/certs/oneskyit_wild_fullchain.pem:/etc/certs/fullchain_wild.pem
- ./conf/certs/oneskyit_wild_privkey.pem:/etc/certs/privkey_wild.pem
- ./conf/certs/oneskyit.com_fullchain.pem:/etc/certs/fullchain.pem
- ./conf/certs/oneskyit.com_privkey.pem:/etc/certs/privkey.pem
- ./conf/certs/ssl-dhparams.pem:/etc/certs/ssl-dhparams.pem
- ./logs/web:/logs
# volumes_from:
# - nextcloud25
depends_on:
- php7
- aether_api_gunicorn
- aether_app_gunicorn
# links:
# # - nextcloud25
# - php7
# - aether_api_gunicorn
# - aether_app_gunicorn
# # - aether_api_gunicorn_bak
# stdin_open: true # docker run -i
# tty: true # docker run -t
# mailman2:
# image: d3fk/mailman2
# container_name: ae_mailman2
# # hostname: mails.oneskyit.com
# hostname: mail.localhost
# restart: unless-stopped
# ports:
# - "8889:80"
# # - "8443:443"
# - "2525:25"
# - "25465:465"
# - "25587:587"
# # env_file:
# # - ./conf/mailman2.env
# environment:
# - EMAIL_HOST=mail.oneskyit.com
# # - URL_HOST=lists.localhost
# - URL_HOST=mailman2-oneskyit.localhost
# - LIST_ADMIN=admin@oneskyit.com
# - MASTER_PASSWORD=strong_pass_321
# - URL_PATTERN=http
# # - SSL_FROM_CONTAINER="true"
# # - SSL_SELFSIGNED="true"
# # - ENABLE_SPF_CHECK="true"
# - URL_ROOT=lists/
# extra_hosts:
# - "mail.oneskyit.com:127.0.0.1"
# # - "oneskyit.com:the linode ip"
# volumes:
# - ./srv/mailman2/archives:/var/lib/mailman/archives
# - ./srv/mailman2/lists:/var/lib/mailman/lists
# - ./srv/mailman2/keys:/etc/exim4/tls.d
# - ./logs/mailman2/apache2:/var/log/apache2
# - ./logs/mailman2/exim4:/var/log/exim4
# - ./logs/mailman2/mailman:/var/log/mailman
# # - ./logs/mailman2/mailman_error.log:/var/lib/mailman/logs/error
#
# # - ./customcert.pem:/etc/ssl/certs/ssl-cert-snakeoil.pem
# # - ./customcertkey.key:/etc/ssl/private/ssl-cert-snakeoil.key
# php5:
# restart: always
# container_name: ae_php5_dev
# # image: php:5-fpm
# build:
# context: ./
# dockerfile: php5.Dockerfile
# volumes:
# - ./srv/html_php:/srv/html_php
#
# - ./conf/php/custom_php5.ini:/usr/local/etc/php/conf.d/custom_php5.ini
#
# - ./logs:/logs
# ports:
# - "9005:9000"
# # networks:
# # - local-net
php7:
restart: always
container_name: ae_php7_dev
# image: php:fpm
build:
context: ./
dockerfile: php7.Dockerfile
volumes:
- ./srv/html_php:/srv/html_php
- ./srv/oneskyit_site:/srv/oneskyit_site
# - ./srv/nextcloud:/srv/nextcloud
- ./conf/php/custom_php7.ini:/usr/local/etc/php/conf.d/custom_php7.ini
- ./logs/php7:/logs
# ports:
# - "9007:9000"
# networks:
# - local-net
phpmyadmin:
image: phpmyadmin
container_name: ae_phpmyadmin
restart: unless-stopped
depends_on:
- mariadb
ports:
- 8888:80
env_file:
- ./.env
environment:
- PMA_ARBITRARY=1
- UPLOAD_LIMIT=1G
mariadb:
container_name: ae_mariadb_dev
# image: mariadb/server:latest
image: mariadb:10.9
# image: mariadb:10.6
restart: always
# env_file:
# - ./.env
# - filename.env
ports:
- "3307:3306"
volumes:
- ./srv/mariadb:/var/lib/mysql
# - ./conf/mariadb/password_reset.sql:/docker-entrypoint-initdb.d/init.sql:ro
# - ./conf/mariadb/password_reset.sql:/password_reset.sql:z
# - ./srv/mariadb_ln:/var/lib/mysql
# - ./conf/mariadb/my.cnf:/etc/my.cnf
# environment:
# - MARIADB_ROOT_PASSWORD=$$1sky.Adapting.7e2
# - MARIADB_ROOT_PASSWORD=CentauriStar123
# - MARIADB_DATABASE: 'my_env_db'
# - MYSQL_ROOT_PASSWORD=$$1sky.Adapting.7e2
# - MYSQL_ROOT_PASSWORD=CentauriStar123
# - MYSQL_PASSWORD=MyPassword
# - MYSQL_DATABASE=nextcloud
# - MYSQL_USER=nextcloud
# nextcloud25:
# container_name: ae_nextcloud25_dev
# build:
# # context: ./builds
# context: ./
# dockerfile: nextcloud25_fpm.Dockerfile
# # image: nextcloud:fpm
# restart: unless-stopped
# links:
# - mariadb
# depends_on:
# - mariadb
# volumes:
# - ./srv/nextcloud/app:/var/www/html
# - ./srv/nextcloud/apps:/var/www/html/apps
# - ./srv/nextcloud/custom_apps:/var/www/html/custom_apps
# - ./srv/nextcloud/config:/var/www/html/config
# - ./srv/nextcloud/data:/var/www/html/data
# - ./srv/nextcloud/themes:/var/www/html/themes
# environment:
# - MYSQL_PASSWORD=MyPassword.1248
# - MYSQL_DATABASE=nextcloud
# - MYSQL_USER=nextcloud
# - MYSQL_HOST=mariadb
# - NEXTCLOUD_TRUSTED_DOMAINS=oneskyit.com
- ae_api
- ae_app
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
redis:
container_name: ae_redis_dev
image: redis
ports:
# host to image
# default port is 6379
- "6389:6379"
aether_api_gunicorn:
restart: always
container_name: ae_api_dev
container_name: ${CONTAINER_REDIS}
image: redis
networks:
- default
- shared
environment:
- TZ=${TZ}
command: redis-server --save "" --loglevel warning
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
mariadb:
restart: always
image: mariadb:10.11
container_name: ${CONTAINER_MARIADB:-ae_mariadb_dev}
profiles: ["database"]
networks:
- shared
command: [
"mysqld",
"--max-connections=${MARIADB_MAX_CONNECTIONS}",
"--innodb-buffer-pool-size=${MARIADB_INNODB_BUFFER_POOL_SIZE}",
"--query-cache-size=${MARIADB_QUERY_CACHE_SIZE}",
"--tmp-table-size=${MARIADB_TMP_TABLE_SIZE}",
"--max-heap-table-size=${MARIADB_TMP_TABLE_SIZE}",
"--table-open-cache=${MARIADB_TABLE_OPEN_CACHE}"
]
environment:
MYSQL_ROOT_PASSWORD: ${AE_DB_PASSWORD}
MYSQL_DATABASE: ${AE_DB_NAME}
MYSQL_USER: ${AE_DB_USERNAME}
MYSQL_PASSWORD: ${AE_DB_PASSWORD}
TZ: ${TZ}
ports:
- "${AE_DB_EXTERNAL_PORT}:3306"
volumes:
- ./srv/mariadb:/var/lib/mysql
- ./conf/mariadb/server.cnf:/etc/mysql/conf.d/server.cnf
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
phpmyadmin:
restart: always
image: phpmyadmin/phpmyadmin
container_name: ${CONTAINER_PMA:-ae_pma_dev}
profiles: ["database"]
networks:
- shared
environment:
PMA_HOST: mariadb
UPLOAD_LIMIT: 64M
TZ: ${TZ}
ports:
- "${AE_PMA_PORT}:80"
depends_on:
- mariadb
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
ae_api:
restart: always
build:
# context: ./builds
context: ./
dockerfile: aether_fastapi_gunicorn.Dockerfile
context: ${AE_API_SRC}
dockerfile: Dockerfile
scale: ${AE_API_REPLICAS}
networks:
- default
- shared
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:5005/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 60s
env_file:
- ./.env
ports:
- "5065:5005"
# expose:
# - 5005
# networks:
# - local-net
extra_hosts:
dev.oneskyit.com: "192.168.32.7"
dev-app.oneskyit.com: "192.168.32.7"
dev-api.oneskyit.com: "192.168.32.7"
test-api.oneskyit.com: "104.237.143.4"
vpn-db.oneskyit.com: "192.168.64.5"
linode.oneskyit.com: "104.237.143.4"
volumes:
- ./conf/aether_fastapi_gunicorn_conf.py:/conf/gunicorn_fastapi_conf.py
- ./conf/aether_api_config.py:/srv/aether_api/app/config.py
- ./logs/ae_api:/logs
# - ./logs/ae_api/aether_fastapi_gunicorn.log:/logs/gunicorn.log
# - ./logs/aether_fastapi_gunicorn_access.log:/logs/gunicorn_access.log
# - ./logs/aether_fastapi_gunicorn_error.log:/logs/gunicorn_error.log
# - ./logs/aether_api.log:/logs/aether_api.log
# - ./logs/ae_api/aether_api.log.1:/logs/aether_api.log.1
# - ./logs/ae_api/aether_api.log.2:/logs/aether_api.log.2
# - ./logs/ae_api/aether_api.log.3:/logs/aether_api.log.3
# - ./logs/ae_api/aether_api.log.4:/logs/aether_api.log.4
# - ./logs/ae_api/aether_api.log.5:/logs/aether_api.log.5
# - ./logs/ae_api/aether_api_warning.log:/logs/aether_api_warning.log
- ./srv/aether_api_ln:/srv/aether_api
- ./srv/hosted_files_ln:/srv/hosted_files
- ./srv/hosted_tmp_ln:/srv/hosted_tmp
# links:
# - redis
- ${AE_API_SRC}:/srv/aether_api
- ${HOSTED_FILES_SRC}:/srv/hosted_files
- ${HOSTED_TMP_SRC}:/srv/hosted_tmp
- ./temp/ae_api:/temp
depends_on:
- redis
stdin_open: true # docker run -i
tty: true # docker run -t
stdin_open: true
tty: true
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
aether_app_gunicorn:
ae_app:
restart: always
container_name: ae_app_dev
build:
# context: ./builds
context: ./
dockerfile: aether_flask_gunicorn.Dockerfile
# image: tiangolo/uvicorn-gunicorn:latest
context: ${AE_APP_SRC}
dockerfile: Dockerfile
target: deploy-node
args:
BUILD_MODE: ${AE_APP_BUILD_MODE:-dev}
scale: ${AE_APP_REPLICAS:-1}
networks:
- default
- shared
env_file:
- ./.env
ports:
- "5055:5005"
# expose:
# - 5005
# networks:
# - local-net
# No host ports — ae_web_dev proxies to ae_app:3000 via Docker DNS,
# round-robining across all replicas. Scale freely with AE_APP_REPLICAS.
extra_hosts:
# - dev-api.oneskyit.com:192.168.32.20
- "${DOCKER_AE_APP_EXTRA_HOST}"
srv-nyx.oneskyit.com: "104.237.143.4"
dev-app.oneskyit.com: "104.237.143.4"
api.oneskyit.com: "104.237.143.4"
bak-api.oneskyit.com: "104.237.143.4"
test-api.oneskyit.com: "104.237.143.4"
dev-api.oneskyit.com: "192.168.32.7"
home.oneskyit.com: "71.126.159.102"
static.oneskyit.com: "104.237.143.4"
dev.oneskyit.com: "192.168.32.7"
volumes:
- ./conf/aether_flask_gunicorn_conf.py:/conf/gunicorn_flask_conf.py
- ./conf/aether_app_config.py:/srv/aether_app/flask_config_v2.py
- ./logs/ae_app:/logs
# - ./logs/aether_flask_gunicorn_access.log:/logs/gunicorn_access.log
# - ./logs/aether_flask_gunicorn_error.log:/logs/gunicorn_error.log
# - ./logs/aether_app.log:/logs/aether_app.log
# - ./logs/aether_app_warning.log:/logs/aether_app_warning.log
- ./srv/aether_app_ln:/srv/aether_app
- ./srv/hosted_files_ln:/srv/hosted_files
- ./srv/hosted_tmp_ln:/srv/hosted_tmp
depends_on:
- aether_api_gunicorn
stdin_open: true # docker run -i
tty: true # docker run -t
- ae_api
- redis
# networks:
# local-net:
# driver: bridge
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
dozzle:
container_name: ${CONTAINER_DOZZLE:-ae_dozzle_dev}
image: amir20/dozzle:latest
environment:
- TZ=${TZ}
volumes:
- /var/run/docker.sock:/var/run/docker.sock
ports:
- "127.0.0.1:${AE_DOZZLE_PORT:-8881}:8080"
restart: unless-stopped
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
ae_ops:
container_name: ${CONTAINER_AE_OPS:-ae_ops_dev}
image: alpine:latest
restart: always
profiles: ["database"]
networks:
- shared
env_file:
- ./.env
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./backups:/backups
- ./scripts:/scripts
- ./logs:/logs
- ./conf/crontab:/etc/crontabs/root
- ./conf/logrotate.conf:/etc/logrotate.conf
command: sh -c "apk add --no-cache docker-cli bash logrotate && adduser -u 1000 -D aether && cp /etc/logrotate.conf /etc/logrotate.internal.conf && chown root:root /etc/logrotate.internal.conf && crond -f -l 2"
depends_on:
- mariadb
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
networks:
default:
name: ${AE_NETWORK_NAME:-ae_dev_net}
shared:
name: aether_shared_net
external: true

View File

@@ -0,0 +1,52 @@
# Aether Platform - Strategic TODO (Agents & Operations)
This document tracks high-impact architectural improvements to the Aether Docker Environment and its connected services. These tasks focus on stability, security, and developer experience (DX).
---
## 🛠️ Infrastructure & Orchestration
### **1. Container Healthchecks (Self-Healing)**
- [x] **FastAPI Healthcheck:** Added a `/health` endpoint to `aether_api_fastapi` that verifies DB and Redis connectivity.
- [ ] **Docker Integration:** Update `docker-compose.yml` to use `healthcheck` for `ae_api` and `ae_app`. (Manual testing complete, next step is automation).
- [ ] **Dependency Ordering:** Use `condition: service_healthy` in `depends_on` blocks to ensure services start in the correct order.
### **2. Environment Abstraction & Safety**
- [ ] **IP Abstraction:** Move the hardcoded workstation IP (`192.168.32.7`) to an `.env` variable (e.g., `AE_HOST_IP`) and reference it in `extra_hosts`.
- [ ] **Env Validation:** Create a `scripts/validate_env.sh` to compare `.env` against `env.default` and catch missing keys or malformed values.
- [ ] **Secret Scanning:** Implement a pre-commit hook or script to ensure no sensitive credentials (from `.env` or backups) are accidentally staged.
### **3. Operational Tooling (The "Easy Button")**
- [ ] **Master Makefile:** Create a `Makefile` in the orchestration root for common commands:
- `make up` / `make down`
- `make build-ui` / `make build-api`
- `make db-backup` / `make db-restore`
- `make logs`
- [ ] **Unified Logs:** Enhance `ae_ops` to provide a consolidated view of critical system errors across all containers.
---
## 🐍 Backend (FastAPI) Modernization
### **4. Configuration via Pydantic Settings**
- [x] **Refactor `app/config.py`:** Switched from the mounted file pattern to `pydantic-settings`.
- [x] **Environment Injection:** API now inherits all settings directly from Docker environment variables.
- [ ] **V2 Migration:** (Long Term) Prepare for the upgrade to Pydantic V2 and SQLAlchemy 2.0.
### **5. Dependency Management**
- [x] **Lockfiles:** Created `requirements.lock` to ensure bit-identical builds across environments.
- [x] **Pruning:** Conducted a final audit of the FastAPI base image and removed 6 redundant Python dependencies.
---
## 🌐 Frontend (SvelteKit) Enhancements
### **6. Build & Runtime Optimization**
- [ ] **Image Size:** Optimize the multi-stage Dockerfile to further reduce the final runtime image size.
- [ ] **Cache Warming:** Implement a mechanism to warm the SvelteKit / Dexie cache on first load for better UX.
---
## 📝 Governance
- This list is managed by **Scott Idem** and **Aether Agents**.
- Tasks should be moved to the [Kanban Board] (via `ae_task_add`) when active work begins.

188
env.default Normal file
View File

@@ -0,0 +1,188 @@
# ------------------------------------------------------------------------------
# AETHER FRAMEWORK - DOCKER ENVIRONMENT CONFIGURATION (TEMPLATE)
# ------------------------------------------------------------------------------
# Instructions: Copy this to .env and update the paths and credentials.
# This file serves as the master reference for all available environment variables.
# ------------------------------------------------------------------------------
# ------------------------------------------------------------------------------
# SYSTEM SETTINGS
# ------------------------------------------------------------------------------
# System timezone for all containers
TZ=US/Eastern
# Environment mode (development, testing, production)
OSIT_ENV=development
# Logging level for the API and background workers (debug, info, warning, error)
AE_LOG_LVL=warning
# Docker Compose Profiles
# 'database' includes: mariadb, phpmyadmin, ae_ops
# Comment out or leave empty for "app-only" nodes that connect to a remote DB
COMPOSE_PROFILES=database
# ------------------------------------------------------------------------------
# CONTAINER NAMES
# ------------------------------------------------------------------------------
# Internal Docker container names (should be unique per environment)
CONTAINER_WEB=ae_web_default
CONTAINER_AE_API=ae_api_default
CONTAINER_AE_APP=ae_app_default
CONTAINER_REDIS=ae_redis_default
CONTAINER_MARIADB=ae_mariadb_default
CONTAINER_PMA=ae_pma_default
CONTAINER_DOZZLE=ae_dozzle_default
# ------------------------------------------------------------------------------
# NETWORK & PROXY SETTINGS
# ------------------------------------------------------------------------------
# Internal Docker network name (should be unique per environment)
AE_NETWORK_NAME=ae_dev_net
# Local Nginx listener ports on the host system
OSIT_WEB_HTTP_PORT=8080
OSIT_WEB_HTTPS_PORT=4443
# Gateway ports (External entry points into the container mesh)
AE_API_GATEWAY_PORT=5060
AE_APP_GATEWAY_PORT=3001
AE_DOZZLE_PORT=8881
# Maximum allowed file upload size (Global for Nginx)
OSIT_WEB_MAX_BODY_SIZE=5120M
# Gateway Port for External Reverse Proxy
# Used when a master proxy (e.g. Home Server) forwards traffic to this node
AE_API_GATEWAY_PORT=5060
# DNS Overrides (Injected into containers' /etc/hosts)
# Format: DOMAIN:IP_ADDRESS
# Useful for container-to-container routing when using real domain names
DOCKER_AE_SERVER_EXTRA_HOST=example.oneskyit.com:127.0.0.1
DOCKER_AE_APP_SERVER_EXTRA_HOST=example-app.oneskyit.com:127.0.0.1
DOCKER_AE_API_SERVER_EXTRA_HOST=example-api.oneskyit.com:127.0.0.1
DOCKER_AE_API_BAK_SERVER_EXTRA_HOST=example-bak-api.oneskyit.com:127.0.0.1
DOCKER_AE_DB_SERVER_EXTRA_HOST=db.oneskyit.com:127.0.0.1
# Nginx Server Names (Used in vhost configuration templates)
DOCKER_AE_API_SERVER_NAME=example-api.oneskyit.com
DOCKER_AE_APP_SERVER_NAME=example-app.oneskyit.com
DOCKER_PHPMYADMIN_SERVER_NAME=example-phpmyadmin.oneskyit.com
DOCKER_OSIT_SERVER_NAME=example-docker.oneskyit.com
# ------------------------------------------------------------------------------
# DATABASE SETTINGS (MariaDB)
# ------------------------------------------------------------------------------
# To use an EXTERNAL database:
# 1. Set COMPOSE_PROFILES= (empty) above to disable local DB containers.
# 2. Set AE_DB_SERVER to the external IP or Hostname.
# 3. Ensure the external DB allows connections from this host's IP.
# DB Hostname (use 'mariadb' for the local container, or a remote IP/FQDN)
AE_DB_SERVER=mariadb
AE_DB_PORT=3306
# Port to expose on the host system if running a local MariaDB container
AE_DB_EXTERNAL_PORT=3306
# Database credentials
AE_DB_NAME=aether_dev
AE_DB_USERNAME=osit_aether
AE_DB_PASSWORD="your-secure-password-here"
AE_DB_ROOT_PASSWORD="your-mariadb-root-password-here"
# Connection Tuning
AE_DB_CONNECTION_TIMEOUT=15
AE_DB_POOL_RECYCLE=1800
# ------------------------------------------------------------------------------
# REDIS SETTINGS
# ------------------------------------------------------------------------------
# Redis is used for caching, ID resolution, and messaging
AE_REDIS_SERVER=redis
AE_REDIS_PORT=6379
# ------------------------------------------------------------------------------
# API SETTINGS (FastAPI)
# ------------------------------------------------------------------------------
AE_API_ENV=development
# Number of API container instances to run (Docker Compose Scaling)
AE_API_REPLICAS=2
# Gunicorn / Uvicorn Tuning
AE_API_GUNICORN_PORT=5065
AE_API_GUNICORN_TIMEOUT=2100
AE_API_GUNICORN_WORKERS=2
AE_API_GUNICORN_THREADS=2
# Security & CORS
# JWT_KEY should be a 22+ character secret string
AE_API_JWT_KEY="your-22-char-secret-key"
# Regex for allowed CORS origins
AE_API_ORIGINS_REGEX="(https://.*\.oneskyit\.com)|(https://.*\.oneskyit\.com:4443)"
# ------------------------------------------------------------------------------
# APP SETTINGS (SvelteKit)
# ------------------------------------------------------------------------------
AE_APP_ENV=development
AE_APP_BUILD_MODE=staging
AE_APP_REPLICAS=1
AE_APP_NODE_PORT=3001
# ------------------------------------------------------------------------------
# SMTP SETTINGS (Email)
# ------------------------------------------------------------------------------
# Core SMTP configuration for system notifications and user emails
AE_SMTP_SERVER=smtp.example.com
AE_SMTP_PORT=465
AE_SMTP_USERNAME=send_mail
AE_SMTP_PASSWORD="your-smtp-password-here"
# ------------------------------------------------------------------------------
# LEGACY APP SETTINGS (Flask)
# ------------------------------------------------------------------------------
AE_FLASK_APP_ENV=development
AE_FLASK_APP_GUNICORN_PORT=5055
AE_FLASK_APP_CACHE_SECRET_KEY="your-secret-key"
AE_FLASK_APP_SESSION_LIFETIME=86400
AE_FLASK_APP_CACHE_TIMEOUT=5
# ------------------------------------------------------------------------------
# SOURCE PATHS (Absolute paths on Host Machine)
# ------------------------------------------------------------------------------
# IMPORTANT: These paths must exist on the machine running Docker
# They are mounted into containers as volumes for real-time development
# Project Source Code
AE_API_SRC=/path/to/aether_api_fastapi
AE_APP_SRC=/path/to/aether_app_sveltekit
AE_FLASK_APP_SRC=/path/to/aether_app_flask
# Physical File Storage (Images, Documents, etc.)
# NOTE: Shared between environments to ensure binary availability
HOSTED_FILES_SRC=/path/to/hosted_files
HOSTED_TMP_SRC=/path/to/hosted_tmp
# ------------------------------------------------------------------------------
# SERVICE TUNING & PERFORMANCE
# ------------------------------------------------------------------------------
# phpMyAdmin Host Port
AE_PMA_PORT=8081
# MariaDB Performance (Injected via Docker Compose command flags)
MARIADB_MAX_CONNECTIONS=500
MARIADB_INNODB_BUFFER_POOL_SIZE=512M
MARIADB_QUERY_CACHE_SIZE=32M
MARIADB_TMP_TABLE_SIZE=384M
MARIADB_TABLE_OPEN_CACHE=4000
# ------------------------------------------------------------------------------
# AETHER SHARED CONFIG (DB Driven)
# ------------------------------------------------------------------------------
# Specifies which record from the 'cfg' table to use for shared settings
# (SMTP, API routes, and external service keys)
# common options: 1=Default, 5=Home Dev, 7=Live Test
AE_CFG_ID=1

35
export_db.sh Executable file
View File

@@ -0,0 +1,35 @@
#!/bin/bash
# Aether Conference Export Script
# Manually triggers a hot backup for off-site use.
set -e
PROJECT_ROOT="/home/scott/OSIT_dev/aether_container_env"
EXPORT_DIR="${PROJECT_ROOT}/backups/conference_export"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
EXPORT_FILE="conference_backup_${TIMESTAMP}.gz"
mkdir -p "$EXPORT_DIR"
echo "--- Starting Conference Database Export ---"
# Trigger the internal backup script inside the ops container
# This will create an 'auto_backup_...' file in the backups folder
docker exec ae_ops_dev bash /scripts/backup_internal.sh
# Find the most recent backup file created in the backups folder
LATEST_BACKUP=$(ls -t "${PROJECT_ROOT}/backups"/auto_backup_*.gz | head -n 1)
if [ -n "$LATEST_BACKUP" ]; then
echo ">>> Moving latest backup to export directory: ${EXPORT_FILE}"
mv "$LATEST_BACKUP" "${EXPORT_DIR}/${EXPORT_FILE}"
# Ensure final ownership is correct
chown 1000:1000 "${EXPORT_DIR}/${EXPORT_FILE}"
echo "--- Export Complete! ---"
echo "File location: ${EXPORT_DIR}/${EXPORT_FILE}"
else
echo "ERROR: Failed to find the generated backup file."
exit 1
fi

44
html_php/index.html Normal file
View File

@@ -0,0 +1,44 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Aether Workstation Dashboard</title>
<style>
body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; background: #1a1a1a; color: #e0e0e0; display: flex; justify-content: center; align-items: center; height: 100vh; margin: 0; }
.container { background: #2d2d2d; padding: 2rem; border-radius: 12px; box-shadow: 0 10px 30px rgba(0,0,0,0.5); width: 400px; }
h1 { color: #4CAF50; margin-top: 0; font-size: 1.5rem; border-bottom: 1px solid #444; padding-bottom: 1rem; }
.links { display: grid; gap: 1rem; margin-top: 1.5rem; }
a { background: #3d3d3d; color: #fff; text-decoration: none; padding: 1rem; border-radius: 8px; transition: transform 0.1s, background 0.2s; display: flex; align-items: center; justify-content: space-between; }
a:hover { background: #4d4d4d; transform: translateY(-2px); }
.port { font-family: monospace; color: #888; font-size: 0.9rem; }
.status { font-size: 0.8rem; color: #4CAF50; margin-top: 2rem; text-align: center; border-top: 1px solid #444; padding-top: 1rem; }
</style>
</head>
<body>
<div class="container">
<h1>Aether Dev Environment</h1>
<div class="links">
<a href="http://localhost:8881" target="_blank">
<span>Dozzle (Live Logs)</span>
<span class="port">:8881</span>
</a>
<a href="http://localhost:8081" target="_blank">
<span>phpMyAdmin (Database)</span>
<span class="port">:8081</span>
</a>
<a href="https://dev-api.oneskyit.com/docs" target="_blank">
<span>API Docs (Swagger)</span>
<span class="port">/docs</span>
</a>
<a href="https://dev-app.oneskyit.com" target="_blank">
<span>Aether App (Flask)</span>
<span class="port">:443</span>
</a>
</div>
<div class="status">
Workstation Mode • Arch Linux
</div>
</div>
</body>
</html>

18
html_php/index.php Normal file
View File

@@ -0,0 +1,18 @@
<html>
<head>
<title>Aether Docker Compose</title>
<meta content="">
<style></style>
</head>
<body>
<h1>Running with Docker Compose</h1>
<hr>
<?php
echo phpinfo();
?>
</body>
</html>

0
logs/ae_api/.gitignore vendored Normal file → Executable file
View File

0
logs/ae_app/.gitignore vendored Normal file → Executable file
View File

0
logs/php7/.gitignore vendored Normal file → Executable file
View File

0
logs/web/nginx/.gitignore vendored Normal file → Executable file
View File

View File

@@ -1,3 +0,0 @@
# FROM php:5.6.20
FROM php:5-fpm
RUN docker-php-ext-install mysqli

View File

@@ -1,2 +0,0 @@
FROM php:7-fpm
RUN docker-php-ext-install mysqli

86
restore_db.sh Executable file
View File

@@ -0,0 +1,86 @@
#!/bin/bash
# Aether MariaDB Restore Script (Physical Backup)
set -e
PROJECT_ROOT="/home/scott/OSIT_dev/aether_container_env"
DEFAULT_BACKUP="${PROJECT_ROOT}/backups/mariadbbackup_1555.gz"
BACKUP_FILE="${1:-$DEFAULT_BACKUP}"
MARIADB_DATA="${PROJECT_ROOT}/srv/mariadb"
RESTORE_TEMP="${PROJECT_ROOT}/srv/restore_temp"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
# Load env for password
source "${PROJECT_ROOT}/.env"
if [ ! -f "$BACKUP_FILE" ]; then
echo "ERROR: Backup file not found: $BACKUP_FILE"
exit 1
fi
# Convert to absolute path for Docker volume mounting
BACKUP_FILE_ABS=$(readlink -f "$BACKUP_FILE")
echo "--- Starting Aether Database Restore ---"
# 1. Stop MariaDB
echo ">>> Stopping MariaDB..."
cd "${PROJECT_ROOT}" && docker compose stop mariadb
# 2. Archive current data
if [ -d "$MARIADB_DATA" ] && [ "$(ls -A $MARIADB_DATA)" ]; then
echo ">>> Archiving current data..."
BACKUP_DIR="${PROJECT_ROOT}/srv/mariadb_bak_${TIMESTAMP}"
mv "${MARIADB_DATA}" "${BACKUP_DIR}"
# Fix ownership of archived data so host user can manage it
docker run --rm -v "${BACKUP_DIR}":/bak alpine chown -R 1000:1000 /bak
fi
mkdir -p "${MARIADB_DATA}" "${RESTORE_TEMP}"
# 3. Extract and Prepare
echo ">>> Running extraction and preparation..."
docker run --rm --user 0 \
-v "${BACKUP_FILE_ABS}":/backups/import.gz \
-v "${RESTORE_TEMP}":/restore \
-v "${PROJECT_ROOT}/scripts/restore_internal.sh":/restore.sh \
mariadb:10.11 bash -c "export BACKUP_FILE=/backups/import.gz && bash /restore.sh"
# 4. Move prepared data (Using container to avoid permission issues)
echo ">>> Moving prepared data..."
docker run --rm --user 0 \
-v "${RESTORE_TEMP}":/src \
-v "${MARIADB_DATA}":/dst \
alpine sh -c "mv /src/* /dst/ 2>/dev/null || true; mv /src/.* /dst/ 2>/dev/null || true"
rmdir "${RESTORE_TEMP}"
# 5. Fix Permissions
echo ">>> Fixing ownership (999:999)..."
docker run --rm -v "${MARIADB_DATA}":/var/lib/mysql alpine chown -R 999:999 /var/lib/mysql
# 6. Start MariaDB in Maintenance Mode to reset password
echo ">>> Resetting passwords to match local .env..."
docker run -d --name ae_mariadb_maint -v "${MARIADB_DATA}":/var/lib/mysql mariadb:10.11 --skip-grant-tables
sleep 5
# Maintenance SQL: Sets root password AND ensures app user exists with correct password/grants
MAINT_SQL="FLUSH PRIVILEGES;
ALTER USER 'root'@'localhost' IDENTIFIED BY '${AE_DB_ROOT_PASSWORD}';
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '${AE_DB_ROOT_PASSWORD}' WITH GRANT OPTION;
CREATE USER IF NOT EXISTS '${AE_DB_USERNAME}'@'%' IDENTIFIED BY '${AE_DB_PASSWORD}';
ALTER USER '${AE_DB_USERNAME}'@'%' IDENTIFIED BY '${AE_DB_PASSWORD}';
GRANT ALL PRIVILEGES ON \`${AE_DB_NAME}\`.* TO '${AE_DB_USERNAME}'@'%';
FLUSH PRIVILEGES;"
docker exec ae_mariadb_maint mariadb -e "$MAINT_SQL"
docker stop ae_mariadb_maint && docker rm ae_mariadb_maint
# 7. Start MariaDB Normally
echo ">>> Starting MariaDB container normally..."
docker compose start mariadb
echo "--- Restore and Password Reset Complete! ---"
# 8. Cleanup Safety Snapshot (Only on success)
if [ -n "$BACKUP_DIR" ] && [ -d "$BACKUP_DIR" ]; then
echo ">>> Removing safety snapshot (Restore successful)..."
rm -rf "$BACKUP_DIR"
fi

View File

@@ -0,0 +1,23 @@
#!/bin/bash
# Aether Internal Backup Script (Runs inside the Cron Container)
set -e
# These are paths INSIDE the cron container
BACKUP_DIR="/backups"
TIMESTAMP=$(date +%Y%m%d_%H%M)
BACKUP_FILE="${BACKUP_DIR}/auto_backup_${TIMESTAMP}.gz"
echo "[$(date)] Starting Scheduled Backup..."
# We use the Docker CLI inside this container to talk to the MariaDB container
# The password is taken from the environment variable passed to this service
docker exec ${CONTAINER_MARIADB} mariabackup --user=root --password="${AE_DB_ROOT_PASSWORD}" \
--backup --stream=xbstream --open-files-limit=65535 | gzip > "${BACKUP_FILE}"
echo "[$(date)] Backup Complete: ${BACKUP_FILE}"
# Ensure host user can manage the backup files
chown 1000:1000 "${BACKUP_FILE}"
# Optional: Clean up backups older than 7 days
find "${BACKUP_DIR}" -name "auto_backup_*.gz" -mtime +7 -delete

View File

@@ -0,0 +1,31 @@
#!/bin/bash
set -e
# Configuration
BACKUP_FILE="${BACKUP_FILE:-/backups/import.gz}"
RESTORE_DIR="/restore"
echo ">>> Phase 0: Wiping restore directory..."
rm -rf "${RESTORE_DIR:?}"/*
echo ">>> Phase 1: Extracting ${BACKUP_FILE} to ${RESTORE_DIR}..."
gunzip -c "${BACKUP_FILE}" | mbstream -x -C "${RESTORE_DIR}"
echo ">>> Phase 2: Metadata Check..."
cd "${RESTORE_DIR}"
if [ -f "mariadb_backup_checkpoints" ] && [ ! -f "xtrabackup_checkpoints" ]; then
echo ">>> Linking mariadb_backup_checkpoints to xtrabackup_checkpoints..."
ln -sf mariadb_backup_checkpoints xtrabackup_checkpoints
fi
if [ -f "mariadb_backup_info" ] && [ ! -f "xtrabackup_info" ]; then
echo ">>> Linking mariadb_backup_info to xtrabackup_info..."
ln -sf mariadb_backup_info xtrabackup_info
fi
echo ">>> Phase 3: Decompressing data..."
mariabackup --decompress --target-dir="${RESTORE_DIR}" --open-files-limit=65535
echo ">>> Phase 4: Preparing backup (Applying logs)..."
mariabackup --prepare --target-dir="${RESTORE_DIR}" --open-files-limit=65535
echo ">>> Restore preparation complete!"

View File

@@ -7,6 +7,14 @@ Create links to the actual directories as needed
* ./srv/oneskyit_site
* ./srv/static_files
git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-api-fastapi.git /srv/http/aether_api_fastapi/
git pull origin development
git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-app.git /srv/http/aether_app/
git pull origin development
git status
## Create links examples
### Flask App
```bash
@@ -25,6 +33,9 @@ ln -s ~/OSIT_dev/aether_api_fastapi ~/OSIT_dev/aether_container_env/srv/aether_a
ln -s /mnt/data/speaker_ready/hosted_tmp /srv/env/test_aether/srv/hosted_tmp_ln
ln -s /mnt/data_drive/srv/data/osit_app/hosted_tmp /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_ln
ln -s /mnt/data_drive/srv/data/osit_app/hosted_tmp_dev /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_ln
# scott-laptop-main:
ln -s /data/OSIT/hosted_tmp /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_ln
```
### Hosted (hashed) files
@@ -32,6 +43,9 @@ ln -s /mnt/data_drive/srv/data/osit_app/hosted_tmp_dev /home/scott/OSIT_dev/aeth
ln -s /mnt/data/speaker_ready/hosted_files /srv/env/test_aether/srv/hosted_files_ln
ln -s /mnt/data_drive/srv/data/osit_app/hosted_files /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_ln
ln -s /mnt/data_drive/srv/data/osit_app/hosted_files_dev /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_ln
# scott-laptop-main:
ln -s /data/OSIT/hosted_files /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_ln
```
### MariaDB

View File

@@ -1,4 +0,0 @@
# Ignore everything in this directory
*
# Except for this file
!.gitignore

View File

@@ -1,4 +0,0 @@
# Ignore everything in this directory
*
# Except for this file
!.gitignore