Scott.Idem/OSIT-AE-Docker-Env
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: modernize README and finalize environment synchronization

Browse Source 
Updated README.md to reflect V3 architecture, documented the physical database management suite, and finalized synchronization between env.default and active .env configuration.
This commit is contained in:
Scott Idem
2026-02-06 13:35:20 -05:00
parent 49539d52c1
commit b4866c2f23
1 changed files with 77 additions and 227 deletions
Download Patch File Download Diff File Expand all files Collapse all files

304
README.md
View File

@@ -1,252 +1,102 @@
This can be used to create a Docker server cluster for the Aether App using Flask, Aether API using FastAPI.
# Aether Framework - Docker Environment
## Initialize
### Part 1
* Create directory and clone the Aether environment.
This repository provides the Docker orchestration and configuration for the Aether Platform. It manages the lifecycle of the Aether API (FastAPI), Aether App (SvelteKit/Flask), and supporting infrastructure (MariaDB, Redis, Nginx).
## 🚀 Quick Start
### 1. Initialize Directory Structure
Create the base directory and clone this environment:
```bash
sudo mkdir /srv/env
sudo chown -R scott:scott /srv/env/
git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-container-environment.git /srv/env/test_aether
sudo mkdir -p /srv/env/aether
sudo chown -R $USER:$USER /srv/env/aether
git clone https://bitbucket.org/oneskyit/one-sky-it-container-environment.git /srv/env/aether/container_env
```
* Using rsync is also an option. The symlinked directories will need to be copied separately.
### 2. Configure Environment Settings
Copy the template and update it with your local paths and credentials:
```bash
sudo rsync -vhr -progress /srv/env/prod_aether/ /srv/env/bak_aether
rsync -av /srv/http/aether_app/ /srv/apps/prod_aether_app_flask
rsync -av /srv/http/aether_api/ /srv/apps/prod_aether_api_fastapi
cd /srv/env/aether/container_env
cp env.default .env
# Edit .env to match your host system
vim .env
```
### Part 2
* Create links to needed for the srv/ directories. See the README.md file under srv/ for details.
* Copy Let's Encrypt certificates to the conf/certs/ directory. See the README.md file under conf/certs/ for details.
* Copy MariaDB database files to the srv/mariadb/ directory using rsync. The original files may need to be copied from the normal Arch Linux location (/var/lib/mysql) first and then possibly reset the root password.
### 3. Setup Persistent Data & Symlinks
The containers expect data and source code to be available in the `srv/` directory via symlinks:
```bash
sudo rsync -vhr -progress /var/lib/mysql/ /srv/env/test_aether/srv/mariadb/
sudo rsync -vhrz scott@linode.oneskyit.com:/srv/env/test_aether/srv/mariadb/ /srv/env/test_aether/srv/mariadb/
sudo chown -R scott:scott /srv/env/test_aether/srv/mariadb/
# Link your local source code
ln -s ~/OSIT_dev/aether_api_fastapi srv/aether_api
ln -s ~/OSIT_dev/aether_app_sveltekit srv/aether_app
# Link physical file storage
ln -s /mnt/data/aether/hosted_files srv/hosted_files
ln -s /mnt/data/aether/hosted_tmp srv/hosted_tmp
```
### Part 3
* Create the environment settings file and place it under the root of the Docker Compose directory. Copy the .env.default file as a template.
```bash
cp /srv/env/test_aether/.env /srv/env/test_aether/.env.bak
cp /srv/env/test_aether/.env.default /srv/env/test_aether/.env
```
**/.env [very basic example]**
The AE_CFG_ID and AE_APP_CFG_ID can be something generic (dev, test, prod a, prod b) when using Docker Compose.
```cnf
# One Sky IT's Aether Framework and System
OSIT_ENV=development
# Aether general shared config options
AE_CFG_ID=0
## Aether API access and use
## Aether DB access and use
AE_DB_PASSWORD="the password with $$ escape"
## Aether Redis access and use
## Aether SMTP access and use
# Aether API specific config options (FastAPI)
# Aether app specific config (Flask with Svelte)
AE_APP_CFG_ID=0
KEY="The Value"
```
#### Part 4 - Still figuring things out...
Update /etc/hosts with something like this.
```cnf
127.0.0.1 dev-api.oneskyit.com dev-app.oneskyit.com
127.0.0.1 dev-demo.oneskyit.com dev-businessgroup.oneskyit.com dev-ishlt.oneskyit.com
```
* Nothing in aether_flask_gunicorn_conf.py needs to be changed?
* Nothing in aether_fastapi_gunicorn_conf.py needs to be changed?
* Nothing in aether_api_config.py needs to be changed?
* Nothing in aether_app_config.py needs to be changed?
## Manage Docker Compose Environment
```bash
docker restart ae_api_dev
docker restart ae_app_dev
docker restart ae_mariadb_dev
```
## Switch AE Docker branch
```bash
git status
git pull --all
git switch branch_to_change_to
git status
git pull origin branch_to_git
less docker-compose.yml
```
## Clone Svelte App(s)
### One Sky IT NPM library
git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-npm-library.git /srv/env/aether_npm_library
### Actual Svelte apps (note the branches)
Clone:
* git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-app-svelte.git /srv/env/aether_xxx_yyy_svelte
* git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-app-svelte.git ~/OSIT_dev/aether_xxx_yyy_svelte
Module groups:
* aether_app_core_svelte
* aether_app_mods_svelte
* aether_app_mod_events_svelte
Note:
Will need to change branches.
```bash
git branch --all
git switch ae_core_dev
git pull origin ae_core_dev
git switch ae_mods_dev
git pull origin ae_mods_dev
git switch ae_mod_events_dev
git pull origin ae_mod_events_dev
git branch --all
```
Also... don't forget to npm update before you npm run build or dev.
### Native Electron app
git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-app-native.git /srv/env/aether_app_native
sdfasdfasd git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-container-environment.git /srv/env/test_aether
### 4. Certificates & Database
* **SSL:** Place your Let's Encrypt certificates in `conf/certs/`.
* **Database:** Use the restoration scripts (see below) to import a MariaDB snapshot.
---
## 🛠️ Management Commands
## More Notes
### Orchestration
```bash
sudo ln -s /srv/apps/prod_aether_app_flask /srv/env/bak_aether/srv/aether_app_ln
sudo ln -s /srv/apps/prod_aether_api_fastapi /srv/env/bak_aether/srv/aether_api_ln
sudo ln -s /mnt/data/speaker_ready/hosted_tmp /srv/env/bak_aether/srv/hosted_tmp_ln
sudo ln -s /mnt/data/speaker_ready/hosted_files /srv/env/bak_aether/srv/hosted_files_ln
# Asus laptop
sudo ln -s /mnt/data_drive/OSIT/hosted_files /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_ln
sudo ln -s /mnt/data_drive/OSIT/hosted_tmp /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_ln
ln -s /srv/http/dev_app.oneskyit.com /srv/env/test_aether/srv/aether_app_ln
ln -s /srv/http/dev_fastapi.oneskyit.com /srv/env/test_aether/srv/aether_api_ln
ln -s /mnt/data/speaker_ready/hosted_tmp /srv/env/test_aether/srv/hosted_tmp_ln
ln -s /mnt/data/speaker_ready/hosted_files /srv/env/test_aether/srv/hosted_files_ln
ls -lha /srv/env/test_aether/srv/aether_app_ln/
ls -lha /srv/env/test_aether/srv/aether_api_ln/
ls -lha /srv/env/test_aether/srv/hosted_tmp_ln/
ls -lha /srv/env/test_aether/srv/hosted_files_ln/
````
### Part 2
```bash
sudo ls -lha /var/lib/mysql/
docker compose up -d # Start all services
docker compose down # Stop all services
docker compose restart ae_api # Restart specific service
```
### Branch Management
```bash
git pull --all
git switch development
docker compose up -d --build
```
---
## Common:
* conf/ = All config files
* logs/ = All log files
* srv/ = All files and data directories that are being served up in some way.
* srv/hosted_files = All hashed hosted files (/mnt/data_drive/srv/data/osit_app/hosted_files_dev/)
* srv/static_files = All static files
* ---
* srv/aether_api = Aether API
* srv/aether_app = Aether App
* ---
* srv/html_php = The default HTML and PHP directory
* srv/oneskyit_site = One Sky IT, LLC main site
* ---
* srv/mailman2 = All Mailman2 files
* srv/mariadb = All MariaDB database files
* srv/nextcloud = All Nextcloud app files
## 🗄️ Database Management (Physical Backups)
## Servics:
nginx
php
gunicorn
mariadb
phpmyadmin
redis
postfix?
mailman?
Nextcloud???
Syncthing???
restic???
memcached???
## Setup
## Check for in use services and ports
Note that the Aether FastAPI will hang if redis is not found.
sudo systemctl status mariadb.services
sudo systemctl status nginx.service
sudo systemctl status php-fpm.service
sudo systemctl status postfix.service
sudo systemctl status redis.service
### Create links to Aether API and app directories
Make sure the docker-compose.yml file is updated with the correct paths.
ln -s /home/scott/OSIT_dev/aether_api_fastapi /home/scott/OSIT_dev/aether_container_env/srv/aether_api_link
ln -s /home/scott/OSIT_dev/aether_app /home/scott/OSIT_dev/aether_container_env/srv/aether_app_link
ln -s /mnt/data_drive/srv/data/osit_app/hosted_files /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_link
ln -s /mnt/data_drive/srv/data/osit_app/hosted_files_dev /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_dev_link
ln -s /mnt/data_drive/srv/data/osit_app/hosted_tmp /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_link
ln -s /mnt/data_drive/srv/data/osit_app/hosted_tmp_dev /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_dev_link
## Database Management (Physical Backups & Restores)
The system uses physical hot backups via `mariabackup` for speed and data integrity.
The system uses physical hot backups via `mariabackup` for maximum speed and data integrity.
### User-Facing Scripts
These scripts are located in the `aether_container_env/` root directory.
These scripts are located in the root directory:
* **`./backup_db.sh`**:
* Triggers an immediate hot backup of the local MariaDB instance.
* Stores results in `backups/local_backup_YYYYMMDD_HHMM.gz`.
* **`./export_db.sh`**:
* Creates a "Conference Ready" backup.
* Triggers a backup inside the `ae_ops_dev` container and moves it to `backups/conference_export/`.
* Ensures the resulting file is owned by the host user (UID 1000).
* **`./restore_db.sh [backup_file.gz]`**:
* Performs a full "Clean Slate" restoration from a `.gz` stream file.
* **Logic**: Stops MariaDB -> Archives current data (`srv/mariadb_bak_...`) -> Wipes `srv/mariadb` -> Extracts/Prepares data -> Resets `root` and `aether_dev` passwords to match your current `.env`.
* **`./check_and_import.sh`**:
* Watchdog script that monitors `backups/import/` for new `.gz` files.
* If a file is found, it triggers `./restore_db.sh` and then moves the processed file to `backups/imported/`.
* **`./backup_db.sh`**: Triggers an immediate hot backup. Results are stored in `backups/`.
* **`./export_db.sh`**: Creates a "Conference Ready" backup in `backups/conference_export/` with host-user ownership.
* **`./restore_db.sh [backup_file.gz]`**: Performs a full "Clean Slate" restoration.
* *Warning:* This stops MariaDB, archives current data, and resets `root` passwords to match your `.env`.
* **`./check_and_import.sh`**: A watchdog script that monitors `backups/import/` for new snapshots and triggers automated restoration.
### Internal Logic Scripts
These are located in `aether_container_env/scripts/` and are primarily called by the user-facing scripts or the `ae_ops` cron service.
---
* **`scripts/backup_internal.sh`**:
* Runs inside the `ae_ops` container.
* Connects to MariaDB, streams the backup, and fixes file ownership for the host.
* **`scripts/restore_internal.sh`**:
* Runs inside a temporary restoration container.
* Handles decompression, metadata linking (handling `mariadb_backup_...` vs legacy `xtrabackup_...` filenames), and log application (`--prepare`).
## 📂 Directory Map
### Backup Directory Structure
* `backups/`: General local backups.
* `backups/import/`: Drop files here for `check_and_import.sh`.
* `backups/imported/`: Archive of processed import files.
* `backups/conference_export/`: Specialized manual exports.
* **`conf/`**: Configuration templates for Nginx, Gunicorn, and the API.
* **`logs/`**: Centralized logging for all containers.
* **`srv/`**: Mount points for data and source code (managed via symlinks).
* **`scripts/`**: Internal automation logic.
* **`backups/`**: Storage for MariaDB snapshots.
---
## 📝 Configuration Guidelines
### Environment Profiles (`COMPOSE_PROFILES`)
* **`database`**: Includes MariaDB, phpMyAdmin, and the `ae_ops` maintenance service.
* **App-Only**: Leave empty if connecting to a remote/shared database server.
### Aether Config ID (`AE_CFG_ID`)
Specifies the record from the `cfg` table to load during the API bootstrap process:
* `1`: Default / Template
* `5`: Home Development
* `7`: Live Testing
---
## ⚠️ Security Notes
* Never commit the `.env` file (it is ignored by git).
* Ensure `AE_API_JWT_KEY` is a unique, high-entropy string in production.
* The API prioritizes `.env` credentials over DB settings for core infrastructure (SMTP/DB) to prevent accidental lockouts.