diff --git a/README.md b/README.md index 958630c..8addaed 100644 --- a/README.md +++ b/README.md @@ -1,252 +1,102 @@ -This can be used to create a Docker server cluster for the Aether App using Flask, Aether API using FastAPI. +# Aether Framework - Docker Environment -## Initialize -### Part 1 -* Create directory and clone the Aether environment. +This repository provides the Docker orchestration and configuration for the Aether Platform. It manages the lifecycle of the Aether API (FastAPI), Aether App (SvelteKit/Flask), and supporting infrastructure (MariaDB, Redis, Nginx). + +## 🚀 Quick Start + +### 1. Initialize Directory Structure +Create the base directory and clone this environment: ```bash -sudo mkdir /srv/env -sudo chown -R scott:scott /srv/env/ -git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-container-environment.git /srv/env/test_aether +sudo mkdir -p /srv/env/aether +sudo chown -R $USER:$USER /srv/env/aether +git clone https://bitbucket.org/oneskyit/one-sky-it-container-environment.git /srv/env/aether/container_env ``` -* Using rsync is also an option. The symlinked directories will need to be copied separately. + +### 2. Configure Environment Settings +Copy the template and update it with your local paths and credentials: ```bash -sudo rsync -vhr -progress /srv/env/prod_aether/ /srv/env/bak_aether -rsync -av /srv/http/aether_app/ /srv/apps/prod_aether_app_flask -rsync -av /srv/http/aether_api/ /srv/apps/prod_aether_api_fastapi +cd /srv/env/aether/container_env +cp env.default .env +# Edit .env to match your host system +vim .env ``` -### Part 2 -* Create links to needed for the srv/ directories. See the README.md file under srv/ for details. -* Copy Let's Encrypt certificates to the conf/certs/ directory. See the README.md file under conf/certs/ for details. -* Copy MariaDB database files to the srv/mariadb/ directory using rsync. The original files may need to be copied from the normal Arch Linux location (/var/lib/mysql) first and then possibly reset the root password. +### 3. Setup Persistent Data & Symlinks +The containers expect data and source code to be available in the `srv/` directory via symlinks: ```bash -sudo rsync -vhr -progress /var/lib/mysql/ /srv/env/test_aether/srv/mariadb/ -sudo rsync -vhrz scott@linode.oneskyit.com:/srv/env/test_aether/srv/mariadb/ /srv/env/test_aether/srv/mariadb/ -sudo chown -R scott:scott /srv/env/test_aether/srv/mariadb/ +# Link your local source code +ln -s ~/OSIT_dev/aether_api_fastapi srv/aether_api +ln -s ~/OSIT_dev/aether_app_sveltekit srv/aether_app + +# Link physical file storage +ln -s /mnt/data/aether/hosted_files srv/hosted_files +ln -s /mnt/data/aether/hosted_tmp srv/hosted_tmp ``` -### Part 3 -* Create the environment settings file and place it under the root of the Docker Compose directory. Copy the .env.default file as a template. -```bash -cp /srv/env/test_aether/.env /srv/env/test_aether/.env.bak -cp /srv/env/test_aether/.env.default /srv/env/test_aether/.env -``` -**/.env [very basic example]** -The AE_CFG_ID and AE_APP_CFG_ID can be something generic (dev, test, prod a, prod b) when using Docker Compose. -```cnf -# One Sky IT's Aether Framework and System - -OSIT_ENV=development - -# Aether general shared config options -AE_CFG_ID=0 - -## Aether API access and use -## Aether DB access and use -AE_DB_PASSWORD="the password with $$ escape" -## Aether Redis access and use -## Aether SMTP access and use - -# Aether API specific config options (FastAPI) - -# Aether app specific config (Flask with Svelte) -AE_APP_CFG_ID=0 - -KEY="The Value" -``` - -#### Part 4 - Still figuring things out... -Update /etc/hosts with something like this. -```cnf -127.0.0.1 dev-api.oneskyit.com dev-app.oneskyit.com -127.0.0.1 dev-demo.oneskyit.com dev-businessgroup.oneskyit.com dev-ishlt.oneskyit.com -``` - -* Nothing in aether_flask_gunicorn_conf.py needs to be changed? -* Nothing in aether_fastapi_gunicorn_conf.py needs to be changed? -* Nothing in aether_api_config.py needs to be changed? -* Nothing in aether_app_config.py needs to be changed? - -## Manage Docker Compose Environment -```bash -docker restart ae_api_dev -docker restart ae_app_dev -docker restart ae_mariadb_dev -``` - -## Switch AE Docker branch -```bash -git status -git pull --all -git switch branch_to_change_to -git status -git pull origin branch_to_git -less docker-compose.yml -``` - -## Clone Svelte App(s) - -### One Sky IT NPM library -git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-npm-library.git /srv/env/aether_npm_library - -### Actual Svelte apps (note the branches) -Clone: -* git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-app-svelte.git /srv/env/aether_xxx_yyy_svelte -* git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-app-svelte.git ~/OSIT_dev/aether_xxx_yyy_svelte -Module groups: -* aether_app_core_svelte -* aether_app_mods_svelte -* aether_app_mod_events_svelte -Note: -Will need to change branches. -```bash -git branch --all -git switch ae_core_dev -git pull origin ae_core_dev - -git switch ae_mods_dev -git pull origin ae_mods_dev - -git switch ae_mod_events_dev -git pull origin ae_mod_events_dev - -git branch --all -``` -Also... don't forget to npm update before you npm run build or dev. - -### Native Electron app -git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-app-native.git /srv/env/aether_app_native - - - -sdfasdfasd git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-container-environment.git /srv/env/test_aether - - +### 4. Certificates & Database +* **SSL:** Place your Let's Encrypt certificates in `conf/certs/`. +* **Database:** Use the restoration scripts (see below) to import a MariaDB snapshot. --- +## 🛠️ Management Commands - -## More Notes - - +### Orchestration ```bash -sudo ln -s /srv/apps/prod_aether_app_flask /srv/env/bak_aether/srv/aether_app_ln -sudo ln -s /srv/apps/prod_aether_api_fastapi /srv/env/bak_aether/srv/aether_api_ln -sudo ln -s /mnt/data/speaker_ready/hosted_tmp /srv/env/bak_aether/srv/hosted_tmp_ln -sudo ln -s /mnt/data/speaker_ready/hosted_files /srv/env/bak_aether/srv/hosted_files_ln - -# Asus laptop -sudo ln -s /mnt/data_drive/OSIT/hosted_files /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_ln -sudo ln -s /mnt/data_drive/OSIT/hosted_tmp /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_ln - - -ln -s /srv/http/dev_app.oneskyit.com /srv/env/test_aether/srv/aether_app_ln -ln -s /srv/http/dev_fastapi.oneskyit.com /srv/env/test_aether/srv/aether_api_ln -ln -s /mnt/data/speaker_ready/hosted_tmp /srv/env/test_aether/srv/hosted_tmp_ln -ln -s /mnt/data/speaker_ready/hosted_files /srv/env/test_aether/srv/hosted_files_ln - -ls -lha /srv/env/test_aether/srv/aether_app_ln/ -ls -lha /srv/env/test_aether/srv/aether_api_ln/ -ls -lha /srv/env/test_aether/srv/hosted_tmp_ln/ -ls -lha /srv/env/test_aether/srv/hosted_files_ln/ -```` - -### Part 2 -```bash -sudo ls -lha /var/lib/mysql/ +docker compose up -d # Start all services +docker compose down # Stop all services +docker compose restart ae_api # Restart specific service ``` +### Branch Management +```bash +git pull --all +git switch development +docker compose up -d --build +``` +--- -## Common: -* conf/ = All config files -* logs/ = All log files -* srv/ = All files and data directories that are being served up in some way. - * srv/hosted_files = All hashed hosted files (/mnt/data_drive/srv/data/osit_app/hosted_files_dev/) - * srv/static_files = All static files - * --- - * srv/aether_api = Aether API - * srv/aether_app = Aether App - * --- - * srv/html_php = The default HTML and PHP directory - * srv/oneskyit_site = One Sky IT, LLC main site - * --- - * srv/mailman2 = All Mailman2 files - * srv/mariadb = All MariaDB database files - * srv/nextcloud = All Nextcloud app files +## 🗄️ Database Management (Physical Backups) -## Servics: -nginx -php -gunicorn -mariadb -phpmyadmin - -redis -postfix? -mailman? -Nextcloud??? -Syncthing??? -restic??? -memcached??? - - -## Setup -## Check for in use services and ports -Note that the Aether FastAPI will hang if redis is not found. -sudo systemctl status mariadb.services -sudo systemctl status nginx.service -sudo systemctl status php-fpm.service -sudo systemctl status postfix.service -sudo systemctl status redis.service - - -### Create links to Aether API and app directories -Make sure the docker-compose.yml file is updated with the correct paths. - -ln -s /home/scott/OSIT_dev/aether_api_fastapi /home/scott/OSIT_dev/aether_container_env/srv/aether_api_link -ln -s /home/scott/OSIT_dev/aether_app /home/scott/OSIT_dev/aether_container_env/srv/aether_app_link - -ln -s /mnt/data_drive/srv/data/osit_app/hosted_files /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_link -ln -s /mnt/data_drive/srv/data/osit_app/hosted_files_dev /home/scott/OSIT_dev/aether_container_env/srv/hosted_files_dev_link - -ln -s /mnt/data_drive/srv/data/osit_app/hosted_tmp /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_link -ln -s /mnt/data_drive/srv/data/osit_app/hosted_tmp_dev /home/scott/OSIT_dev/aether_container_env/srv/hosted_tmp_dev_link - -## Database Management (Physical Backups & Restores) - -The system uses physical hot backups via `mariabackup` for speed and data integrity. +The system uses physical hot backups via `mariabackup` for maximum speed and data integrity. ### User-Facing Scripts -These scripts are located in the `aether_container_env/` root directory. +These scripts are located in the root directory: -* **`./backup_db.sh`**: - * Triggers an immediate hot backup of the local MariaDB instance. - * Stores results in `backups/local_backup_YYYYMMDD_HHMM.gz`. -* **`./export_db.sh`**: - * Creates a "Conference Ready" backup. - * Triggers a backup inside the `ae_ops_dev` container and moves it to `backups/conference_export/`. - * Ensures the resulting file is owned by the host user (UID 1000). -* **`./restore_db.sh [backup_file.gz]`**: - * Performs a full "Clean Slate" restoration from a `.gz` stream file. - * **Logic**: Stops MariaDB -> Archives current data (`srv/mariadb_bak_...`) -> Wipes `srv/mariadb` -> Extracts/Prepares data -> Resets `root` and `aether_dev` passwords to match your current `.env`. -* **`./check_and_import.sh`**: - * Watchdog script that monitors `backups/import/` for new `.gz` files. - * If a file is found, it triggers `./restore_db.sh` and then moves the processed file to `backups/imported/`. +* **`./backup_db.sh`**: Triggers an immediate hot backup. Results are stored in `backups/`. +* **`./export_db.sh`**: Creates a "Conference Ready" backup in `backups/conference_export/` with host-user ownership. +* **`./restore_db.sh [backup_file.gz]`**: Performs a full "Clean Slate" restoration. + * *Warning:* This stops MariaDB, archives current data, and resets `root` passwords to match your `.env`. +* **`./check_and_import.sh`**: A watchdog script that monitors `backups/import/` for new snapshots and triggers automated restoration. -### Internal Logic Scripts -These are located in `aether_container_env/scripts/` and are primarily called by the user-facing scripts or the `ae_ops` cron service. +--- -* **`scripts/backup_internal.sh`**: - * Runs inside the `ae_ops` container. - * Connects to MariaDB, streams the backup, and fixes file ownership for the host. -* **`scripts/restore_internal.sh`**: - * Runs inside a temporary restoration container. - * Handles decompression, metadata linking (handling `mariadb_backup_...` vs legacy `xtrabackup_...` filenames), and log application (`--prepare`). +## 📂 Directory Map -### Backup Directory Structure -* `backups/`: General local backups. -* `backups/import/`: Drop files here for `check_and_import.sh`. -* `backups/imported/`: Archive of processed import files. -* `backups/conference_export/`: Specialized manual exports. \ No newline at end of file +* **`conf/`**: Configuration templates for Nginx, Gunicorn, and the API. +* **`logs/`**: Centralized logging for all containers. +* **`srv/`**: Mount points for data and source code (managed via symlinks). +* **`scripts/`**: Internal automation logic. +* **`backups/`**: Storage for MariaDB snapshots. + +--- + +## 📝 Configuration Guidelines + +### Environment Profiles (`COMPOSE_PROFILES`) +* **`database`**: Includes MariaDB, phpMyAdmin, and the `ae_ops` maintenance service. +* **App-Only**: Leave empty if connecting to a remote/shared database server. + +### Aether Config ID (`AE_CFG_ID`) +Specifies the record from the `cfg` table to load during the API bootstrap process: +* `1`: Default / Template +* `5`: Home Development +* `7`: Live Testing + +--- + +## ⚠️ Security Notes +* Never commit the `.env` file (it is ignored by git). +* Ensure `AE_API_JWT_KEY` is a unique, high-entropy string in production. +* The API prioritizes `.env` credentials over DB settings for core infrastructure (SMTP/DB) to prevent accidental lockouts.