Scott.Idem/OSIT-AE-Docker-Env
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: modernize README and finalize environment synchronization

Browse Source 
Updated README.md to reflect V3 architecture, documented the physical database management suite, and finalized synchronization between env.default and active .env configuration.
This commit is contained in:
Scott Idem
2026-02-06 13:35:20 -05:00
parent 49539d52c1
commit b4866c2f23
1 changed files with 77 additions and 227 deletions
Download Patch File Download Diff File Expand all files Collapse all files

304
README.md
View File

@@ -1,252 +1,102 @@
This can be used to create a Docker server cluster for the Aether App using Flask, Aether API using FastAPI. # Aether Framework - Docker Environment
## Initialize This repository provides the Docker orchestration and configuration for the Aether Platform. It manages the lifecycle of the Aether API (FastAPI), Aether App (SvelteKit/Flask), and supporting infrastructure (MariaDB, Redis, Nginx).
### Part 1
* Create directory and clone the Aether environment. ## 🚀 Quick Start
### 1. Initialize Directory Structure
Create the base directory and clone this environment:
```bash ```bash
sudo mkdir /srv/env sudo mkdir -p /srv/env/aether
sudo chown -R scott:scott /srv/env/ sudo chown -R $USER:$USER /srv/env/aether
git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-container-environment.git /srv/env/test_aether git clone https://bitbucket.org/oneskyit/one-sky-it-container-environment.git /srv/env/aether/container_env
``` ```
* Using rsync is also an option. The symlinked directories will need to be copied separately.
### 2. Configure Environment Settings
Copy the template and update it with your local paths and credentials:
```bash ```bash
sudo rsync -vhr -progress /srv/env/prod_aether/ /srv/env/bak_aether cd /srv/env/aether/container_env
rsync -av /srv/http/aether_app/ /srv/apps/prod_aether_app_flask cp env.default .env
rsync -av /srv/http/aether_api/ /srv/apps/prod_aether_api_fastapi # Edit .env to match your host system
vim .env
``` ```
### Part 2 ### 3. Setup Persistent Data & Symlinks
* Create links to needed for the srv/ directories. See the README.md file under srv/ for details. The containers expect data and source code to be available in the `srv/` directory via symlinks:
* Copy Let's Encrypt certificates to the conf/certs/ directory. See the README.md file under conf/certs/ for details.
* Copy MariaDB database files to the srv/mariadb/ directory using rsync. The original files may need to be copied from the normal Arch Linux location (/var/lib/mysql) first and then possibly reset the root password.
```bash ```bash
sudo rsync -vhr -progress /var/lib/mysql/ /srv/env/test_aether/srv/mariadb/ # Link your local source code
sudo rsync -vhrz scott@linode.oneskyit.com:/srv/env/test_aether/srv/mariadb/ /srv/env/test_aether/srv/mariadb/ ln -s ~/OSIT_dev/aether_api_fastapi srv/aether_api
sudo chown -R scott:scott /srv/env/test_aether/srv/mariadb/ ln -s ~/OSIT_dev/aether_app_sveltekit srv/aether_app
# Link physical file storage
ln -s /mnt/data/aether/hosted_files srv/hosted_files
ln -s /mnt/data/aether/hosted_tmp srv/hosted_tmp
``` ```
### Part 3 ### 4. Certificates & Database
* Create the environment settings file and place it under the root of the Docker Compose directory. Copy the .env.default file as a template. * **SSL:** Place your Let's Encrypt certificates in `conf/certs/`.
```bash * **Database:** Use the restoration scripts (see below) to import a MariaDB snapshot.
cp /srv/env/test_aether/.env /srv/env/test_aether/.env.bak
cp /srv/env/test_aether/.env.default /srv/env/test_aether/.env
```
**/.env [very basic example]**
The AE_CFG_ID and AE_APP_CFG_ID can be something generic (dev, test, prod a, prod b) when using Docker Compose.
```cnf
# One Sky IT's Aether Framework and System
OSIT_ENV=development
# Aether general shared config options
AE_CFG_ID=0
## Aether API access and use
## Aether DB access and use
AE_DB_PASSWORD="the password with $$ escape"
## Aether Redis access and use
## Aether SMTP access and use
# Aether API specific config options (FastAPI)
# Aether app specific config (Flask with Svelte)
AE_APP_CFG_ID=0
KEY="The Value"
```
#### Part 4 - Still figuring things out...
Update /etc/hosts with something like this.
```cnf
127.0.0.1 dev-api.oneskyit.com dev-app.oneskyit.com
127.0.0.1 dev-demo.oneskyit.com dev-businessgroup.oneskyit.com dev-ishlt.oneskyit.com
```
* Nothing in aether_flask_gunicorn_conf.py needs to be changed?
* Nothing in aether_fastapi_gunicorn_conf.py needs to be changed?
* Nothing in aether_api_config.py needs to be changed?
* Nothing in aether_app_config.py needs to be changed?
## Manage Docker Compose Environment
```bash
docker restart ae_api_dev
docker restart ae_app_dev
docker restart ae_mariadb_dev
```
## Switch AE Docker branch
```bash
git status
git pull --all
git switch branch_to_change_to
git status
git pull origin branch_to_git
less docker-compose.yml
```
## Clone Svelte App(s)
### One Sky IT NPM library
git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-npm-library.git /srv/env/aether_npm_library
### Actual Svelte apps (note the branches)
Clone:
* git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-app-svelte.git /srv/env/aether_xxx_yyy_svelte
* git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-app-svelte.git ~/OSIT_dev/aether_xxx_yyy_svelte
Module groups:
* aether_app_core_svelte
* aether_app_mods_svelte
* aether_app_mod_events_svelte
Note:
Will need to change branches.
```bash
git branch --all
git switch ae_core_dev
git pull origin ae_core_dev
git switch ae_mods_dev
git pull origin ae_mods_dev
git switch ae_mod_events_dev
git pull origin ae_mod_events_dev
git branch --all
```
Also... don't forget to npm update before you npm run build or dev.
### Native Electron app
git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-app-native.git /srv/env/aether_app_native
sdfasdfasd git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-container-environment.git /srv/env/test_aether
--- ---
## 🛠️ Management Commands
### Orchestration
## More Notes
```bash ```bash
sudo ln -s /srv/apps/prod_aether_app_flask /srv/env/bak_aether/srv/aether_app_ln docker compose up -d # Start all services
sudo ln -s /srv/apps/prod_aether_api_fastapi /srv/env/bak_aether/srv/aether_api_ln docker compose down # Stop all services
sudo ln -s /mnt/data/speaker_ready/hosted_tmp /srv/env/bak_aether/srv/hosted_tmp_ln docker compose restart ae_api # Restart specific service
sudo ln -s /mnt/data/speaker_ready/hosted_files /srv/env/bak_aether/srv/hosted_files_ln
# Asus laptop
sudo ln -s /mnt/data_drive/OSIT/hosted_files /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_ln
sudo ln -s /mnt/data_drive/OSIT/hosted_tmp /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_ln
ln -s /srv/http/dev_app.oneskyit.com /srv/env/test_aether/srv/aether_app_ln
ln -s /srv/http/dev_fastapi.oneskyit.com /srv/env/test_aether/srv/aether_api_ln
ln -s /mnt/data/speaker_ready/hosted_tmp /srv/env/test_aether/srv/hosted_tmp_ln
ln -s /mnt/data/speaker_ready/hosted_files /srv/env/test_aether/srv/hosted_files_ln
ls -lha /srv/env/test_aether/srv/aether_app_ln/
ls -lha /srv/env/test_aether/srv/aether_api_ln/
ls -lha /srv/env/test_aether/srv/hosted_tmp_ln/
ls -lha /srv/env/test_aether/srv/hosted_files_ln/
````
### Part 2
```bash
sudo ls -lha /var/lib/mysql/
``` ```
### Branch Management
```bash
git pull --all
git switch development
docker compose up -d --build
```
---
## Common: ## 🗄️ Database Management (Physical Backups)
* conf/ = All config files
* logs/ = All log files
* srv/ = All files and data directories that are being served up in some way.
* srv/hosted_files = All hashed hosted files (/mnt/data_drive/srv/data/osit_app/hosted_files_dev/)
* srv/static_files = All static files
* ---
* srv/aether_api = Aether API
* srv/aether_app = Aether App
* ---
* srv/html_php = The default HTML and PHP directory
* srv/oneskyit_site = One Sky IT, LLC main site
* ---
* srv/mailman2 = All Mailman2 files
* srv/mariadb = All MariaDB database files
* srv/nextcloud = All Nextcloud app files
## Servics: The system uses physical hot backups via `mariabackup` for maximum speed and data integrity.
nginx
php
gunicorn
mariadb
phpmyadmin
redis
postfix?
mailman?
Nextcloud???
Syncthing???
restic???
memcached???
## Setup
## Check for in use services and ports
Note that the Aether FastAPI will hang if redis is not found.
sudo systemctl status mariadb.services
sudo systemctl status nginx.service
sudo systemctl status php-fpm.service
sudo systemctl status postfix.service
sudo systemctl status redis.service
### Create links to Aether API and app directories
Make sure the docker-compose.yml file is updated with the correct paths.
ln -s /home/scott/OSIT_dev/aether_api_fastapi /home/scott/OSIT_dev/aether_container_env/srv/aether_api_link
ln -s /home/scott/OSIT_dev/aether_app /home/scott/OSIT_dev/aether_container_env/srv/aether_app_link
ln -s /mnt/data_drive/srv/data/osit_app/hosted_files /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_link
ln -s /mnt/data_drive/srv/data/osit_app/hosted_files_dev /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_dev_link
ln -s /mnt/data_drive/srv/data/osit_app/hosted_tmp /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_link
ln -s /mnt/data_drive/srv/data/osit_app/hosted_tmp_dev /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_dev_link
## Database Management (Physical Backups & Restores)
The system uses physical hot backups via `mariabackup` for speed and data integrity.
### User-Facing Scripts ### User-Facing Scripts
These scripts are located in the `aether_container_env/` root directory. These scripts are located in the root directory:
* **`./backup_db.sh`**: * **`./backup_db.sh`**: Triggers an immediate hot backup. Results are stored in `backups/`.
* Triggers an immediate hot backup of the local MariaDB instance. * **`./export_db.sh`**: Creates a "Conference Ready" backup in `backups/conference_export/` with host-user ownership.
* Stores results in `backups/local_backup_YYYYMMDD_HHMM.gz`. * **`./restore_db.sh [backup_file.gz]`**: Performs a full "Clean Slate" restoration.
* **`./export_db.sh`**: * *Warning:* This stops MariaDB, archives current data, and resets `root` passwords to match your `.env`.
* Creates a "Conference Ready" backup. * **`./check_and_import.sh`**: A watchdog script that monitors `backups/import/` for new snapshots and triggers automated restoration.
* Triggers a backup inside the `ae_ops_dev` container and moves it to `backups/conference_export/`.
* Ensures the resulting file is owned by the host user (UID 1000).
* **`./restore_db.sh [backup_file.gz]`**:
* Performs a full "Clean Slate" restoration from a `.gz` stream file.
* **Logic**: Stops MariaDB -> Archives current data (`srv/mariadb_bak_...`) -> Wipes `srv/mariadb` -> Extracts/Prepares data -> Resets `root` and `aether_dev` passwords to match your current `.env`.
* **`./check_and_import.sh`**:
* Watchdog script that monitors `backups/import/` for new `.gz` files.
* If a file is found, it triggers `./restore_db.sh` and then moves the processed file to `backups/imported/`.
### Internal Logic Scripts ---
These are located in `aether_container_env/scripts/` and are primarily called by the user-facing scripts or the `ae_ops` cron service.
* **`scripts/backup_internal.sh`**: ## 📂 Directory Map
* Runs inside the `ae_ops` container.
* Connects to MariaDB, streams the backup, and fixes file ownership for the host.
* **`scripts/restore_internal.sh`**:
* Runs inside a temporary restoration container.
* Handles decompression, metadata linking (handling `mariadb_backup_...` vs legacy `xtrabackup_...` filenames), and log application (`--prepare`).
### Backup Directory Structure * **`conf/`**: Configuration templates for Nginx, Gunicorn, and the API.
* `backups/`: General local backups. * **`logs/`**: Centralized logging for all containers.
* `backups/import/`: Drop files here for `check_and_import.sh`. * **`srv/`**: Mount points for data and source code (managed via symlinks).
* `backups/imported/`: Archive of processed import files. * **`scripts/`**: Internal automation logic.
* `backups/conference_export/`: Specialized manual exports. * **`backups/`**: Storage for MariaDB snapshots.
---
## 📝 Configuration Guidelines
### Environment Profiles (`COMPOSE_PROFILES`)
* **`database`**: Includes MariaDB, phpMyAdmin, and the `ae_ops` maintenance service.
* **App-Only**: Leave empty if connecting to a remote/shared database server.
### Aether Config ID (`AE_CFG_ID`)
Specifies the record from the `cfg` table to load during the API bootstrap process:
* `1`: Default / Template
* `5`: Home Development
* `7`: Live Testing
---
## ⚠️ Security Notes
* Never commit the `.env` file (it is ignored by git).
* Ensure `AE_API_JWT_KEY` is a unique, high-entropy string in production.
* The API prioritizes `.env` credentials over DB settings for core infrastructure (SMTP/DB) to prevent accidental lockouts.