OSIT-AE-Docker-Env/README.md

This can be used to create a Docker server cluster for the Aether App using Flask, Aether API using FastAPI.

Initialize

Part 1

• Create directory and clone the Aether environment.

sudo mkdir /srv/env
sudo chown -R scott:scott /srv/env/
git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-container-environment.git /srv/env/test_aether

• Using rsync is also an option. The symlinked directories will need to be copied separately.

sudo rsync -vhr -progress /srv/env/prod_aether/ /srv/env/bak_aether
rsync -av /srv/http/aether_app/ /srv/apps/prod_aether_app_flask
rsync -av /srv/http/aether_api/ /srv/apps/prod_aether_api_fastapi

Part 2

• Create links to needed for the srv/ directories. See the README.md file under srv/ for details.
• Copy Let's Encrypt certificates to the conf/certs/ directory. See the README.md file under conf/certs/ for details.
• Copy MariaDB database files to the srv/mariadb/ directory using rsync. The original files may need to be copied from the normal Arch Linux location (/var/lib/mysql) first and then possibly reset the root password.

sudo rsync -vhr -progress /var/lib/mysql/ /srv/env/test_aether/srv/mariadb/
sudo rsync -vhrz scott@linode.oneskyit.com:/srv/env/test_aether/srv/mariadb/ /srv/env/test_aether/srv/mariadb/
sudo chown -R scott:scott /srv/env/test_aether/srv/mariadb/

Part 3

• Create the environment settings file and place it under the root of the Docker Compose directory. Copy the .env.default file as a template.

cp /srv/env/test_aether/.env /srv/env/test_aether/.env.bak
cp /srv/env/test_aether/.env.default /srv/env/test_aether/.env

/.env [very basic example] The AE_CFG_ID and AE_APP_CFG_ID can be something generic (dev, test, prod a, prod b) when using Docker Compose.

# One Sky IT's Aether Framework and System

OSIT_ENV=development

# Aether general shared config options
AE_CFG_ID=0

## Aether API access and use
## Aether DB access and use
AE_DB_PASSWORD="the password with $$ escape"
## Aether Redis access and use
## Aether SMTP access and use

# Aether API specific config options (FastAPI)

# Aether app specific config (Flask with Svelte)
AE_APP_CFG_ID=0

KEY="The Value"

Part 4 - Still figuring things out...

Update /etc/hosts with something like this.

127.0.0.1       dev-api.oneskyit.com dev-app.oneskyit.com
127.0.0.1       dev-demo.oneskyit.com dev-businessgroup.oneskyit.com dev-ishlt.oneskyit.com

• Nothing in aether_flask_gunicorn_conf.py needs to be changed?
• Nothing in aether_fastapi_gunicorn_conf.py needs to be changed?
• Nothing in aether_api_config.py needs to be changed?
• Nothing in aether_app_config.py needs to be changed?

Manage Docker Compose Environment

docker restart ae_api_dev
docker restart ae_app_dev
docker restart ae_mariadb_dev

Switch AE Docker branch

git status
git pull --all
git switch branch_to_change_to
git status
git pull origin branch_to_git
less docker-compose.yml

Clone Svelte App(s)

One Sky IT NPM library

git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-npm-library.git /srv/env/aether_npm_library

Actual Svelte apps (note the branches)

Clone:

• git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-app-svelte.git /srv/env/aether_xxx_yyy_svelte
• git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-app-svelte.git ~/OSIT_dev/aether_xxx_yyy_svelte Module groups:
• aether_app_core_svelte
• aether_app_mods_svelte
• aether_app_mod_events_svelte Note: Will need to change branches.

git branch --all
git switch ae_core_dev
git pull origin ae_core_dev

git switch ae_mods_dev
git pull origin ae_mods_dev

git switch ae_mod_events_dev
git pull origin ae_mod_events_dev

git branch --all

Also... don't forget to npm update before you npm run build or dev.

Native Electron app

git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-app-native.git /srv/env/aether_app_native

sdfasdfasd git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-container-environment.git /srv/env/test_aether

---

More Notes

sudo ln -s /srv/apps/prod_aether_app_flask /srv/env/bak_aether/srv/aether_app_ln
sudo ln -s /srv/apps/prod_aether_api_fastapi /srv/env/bak_aether/srv/aether_api_ln
sudo ln -s /mnt/data/speaker_ready/hosted_tmp /srv/env/bak_aether/srv/hosted_tmp_ln
sudo ln -s /mnt/data/speaker_ready/hosted_files /srv/env/bak_aether/srv/hosted_files_ln

# Asus laptop
sudo ln -s /mnt/data_drive/OSIT/hosted_files /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_ln
sudo ln -s /mnt/data_drive/OSIT/hosted_tmp /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_ln


ln -s /srv/http/dev_app.oneskyit.com /srv/env/test_aether/srv/aether_app_ln
ln -s /srv/http/dev_fastapi.oneskyit.com /srv/env/test_aether/srv/aether_api_ln
ln -s /mnt/data/speaker_ready/hosted_tmp /srv/env/test_aether/srv/hosted_tmp_ln
ln -s /mnt/data/speaker_ready/hosted_files /srv/env/test_aether/srv/hosted_files_ln

ls -lha /srv/env/test_aether/srv/aether_app_ln/
ls -lha /srv/env/test_aether/srv/aether_api_ln/
ls -lha /srv/env/test_aether/srv/hosted_tmp_ln/
ls -lha /srv/env/test_aether/srv/hosted_files_ln/

Part 2

sudo ls -lha /var/lib/mysql/

Common:

• conf/ = All config files
• logs/ = All log files
• srv/ = All files and data directories that are being served up in some way.
• srv/hosted_files = All hashed hosted files (/mnt/data_drive/srv/data/osit_app/hosted_files_dev/)
• srv/static_files = All static files

• ---

• srv/aether_api = Aether API
• srv/aether_app = Aether App

• ---

• srv/html_php = The default HTML and PHP directory
• srv/oneskyit_site = One Sky IT, LLC main site

• ---

• srv/mailman2 = All Mailman2 files
• srv/mariadb = All MariaDB database files
• srv/nextcloud = All Nextcloud app files

Servics:

nginx php gunicorn mariadb phpmyadmin

redis postfix? mailman? Nextcloud??? Syncthing??? restic??? memcached???

Setup

Check for in use services and ports

Note that the Aether FastAPI will hang if redis is not found. sudo systemctl status mariadb.services sudo systemctl status nginx.service sudo systemctl status php-fpm.service sudo systemctl status postfix.service sudo systemctl status redis.service

Create links to Aether API and app directories

Make sure the docker-compose.yml file is updated with the correct paths.

ln -s /home/scott/OSIT_dev/aether_api_fastapi /home/scott/OSIT_dev/aether_container_env/srv/aether_api_link ln -s /home/scott/OSIT_dev/aether_app /home/scott/OSIT_dev/aether_container_env/srv/aether_app_link

ln -s /mnt/data_drive/srv/data/osit_app/hosted_files /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_link ln -s /mnt/data_drive/srv/data/osit_app/hosted_files_dev /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_dev_link

ln -s /mnt/data_drive/srv/data/osit_app/hosted_tmp /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_link ln -s /mnt/data_drive/srv/data/osit_app/hosted_tmp_dev /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_dev_link

Database Management (Physical Backups & Restores)

The system uses physical hot backups via mariabackup for speed and data integrity.

User-Facing Scripts

These scripts are located in the aether_container_env/ root directory.

• ./backup_db.sh:
  • Triggers an immediate hot backup of the local MariaDB instance.
  • Stores results in backups/local_backup_YYYYMMDD_HHMM.gz.
• ./export_db.sh:
  • Creates a "Conference Ready" backup.
  • Triggers a backup inside the ae_ops_dev container and moves it to backups/conference_export/.
  • Ensures the resulting file is owned by the host user (UID 1000).
• ./restore_db.sh [backup_file.gz]:
  • Performs a full "Clean Slate" restoration from a .gz stream file.
  • Logic: Stops MariaDB -> Archives current data (srv/mariadb_bak_...) -> Wipes srv/mariadb -> Extracts/Prepares data -> Resets root and aether_dev passwords to match your current .env.
• ./check_and_import.sh:
  • Watchdog script that monitors backups/import/ for new .gz files.
  • If a file is found, it triggers ./restore_db.sh and then moves the processed file to backups/imported/.

Internal Logic Scripts

These are located in aether_container_env/scripts/ and are primarily called by the user-facing scripts or the ae_ops cron service.

• scripts/backup_internal.sh:
  • Runs inside the ae_ops container.
  • Connects to MariaDB, streams the backup, and fixes file ownership for the host.
• scripts/restore_internal.sh:
  • Runs inside a temporary restoration container.
  • Handles decompression, metadata linking (handling mariadb_backup_... vs legacy xtrabackup_... filenames), and log application (--prepare).

Backup Directory Structure

• backups/: General local backups.
• backups/import/: Drop files here for check_and_import.sh.
• backups/imported/: Archive of processed import files.
• backups/conference_export/: Specialized manual exports.