Skip to content

dev_environment

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

dev_environment

README.md

Development Environment

Complete Docker Compose setup for local development and testing of the SQL-to-ARC middleware.

Services

1. postgres

PostgreSQL 15 database server with:

  • Default credentials: postgres/postgres
  • Port: 5432
  • Persistent volume: postgres_data
  • Health check enabled

2. db-init

One-time initialization container that:

3. sql_to_arc

The SQL-to-ARC converter that:

  • Builds from ../docker/Dockerfile.sql_to_arc
  • Waits for db-init to complete
  • Connects to PostgreSQL and Middleware API
  • Mounts encrypted secrets via sops
  • Currently set to sleep 3600 (modify compose.dev.yaml to enable converter)

4. middleware-api (Demo Only)

A simple mock service that simulates the Middleware API for local testing.

  • No mTLS required (uses HTTP)
  • Logs all incoming ARC uploads
  • Available via http://localhost:8000

Quick Start (Demo Mode - Recommended for First Time)

If you don't have the mTLS keys yet and just want to see the workflow in action with a local database and a mock API:

./start-demo.sh --build

This starts:

  1. postgres: A local DB.
  2. db-init: Fills the DB with sample data.
  3. middleware-api: A local mock API.
  4. sql-to-arc: The converter, pointing to the local mock.

Quick Start (Standard/External Mode)

Prerequisites

  • Docker and Docker Compose
  • sops for secret management
  • Age or PGP key configured for sops decryption

Start with Decryption

./start-dev.sh

This will:

  1. Start PostgreSQL
  2. Initialize the database with Edaphobase data
  3. Run the SQL-to-ARC converter

With image rebuild:

./start-dev.sh --build

Start with External Middleware API

If you want to run sql_to_arc against an external API server (e.g. production or staging) that requires client certificates:

  1. Copy your client certificate and key to dev_environment/client.crt and dev_environment/client.key.
  2. Edit dev_environment/config-external.yaml and set the api_url to the external endpoint.
  3. Run the external start script:
./start-external.sh

This starts only postgres, db-init, and sql_to_arc.

View Logs

docker compose logs -f
docker compose logs -f postgres
docker compose logs -f sql_to_arc

Stop Services

docker compose down

Clean Everything (including data)

docker compose down -v

Configuration

Environment Variables

Set via .env file or shell environment:

  • POSTGRES_USER - Database user (default: postgres)
  • POSTGRES_PASSWORD - Database password (default: postgres)

Secrets with sops

The client.key file should be encrypted with sops:

# Encrypt (first time)
sops -e -i client.key

# Edit encrypted file
sops client.key

# Decrypt to view
sops -d client.key

The start-dev.sh script uses sops exec-file to temporarily decrypt client.key during container startup.

config.dev.yaml

Application configuration for sql_to_arc:

  • db_host: Set to postgres (Docker service name)
  • api_client.client_cert_path: /run/secrets/client.crt
  • api_client.client_key_path: /run/secrets/client.key

Service Dependencies

postgres (healthcheck)
  ↓
db-init (waits for healthy postgres)
  ↓
sql_to_arc (waits for db-init completion)

Troubleshooting

Database not initializing

Check db-init logs:

docker compose logs db-init

Common issues:

  • Network timeout downloading dump → retry with docker compose up db-init
  • PostgreSQL not ready → check postgres healthcheck

sql_to_arc fails

Check logs:

docker compose logs sql_to_arc

Common issues:

  • Secrets not mounted → verify sops decryption works: sops -d client.key
  • API unreachable → check api_url in config.dev.yaml
  • Database connection → verify db-init completed successfully

Rebuild specific service

docker compose build sql_to_arc
docker compose up sql_to_arc

Manual Usage (without start-dev.sh)

If you don't want to use sops or the start script:

# Start postgres and db-init only
docker compose up -d postgres db-init

# Wait for initialization
docker compose logs -f db-init

# Run sql_to_arc manually (after decrypting secrets)
sops exec-file client.key \
  'docker compose run --rm sql_to_arc'

Development Workflow

  1. Make changes to sql_to_arc code
  2. Rebuild image: ./start-dev.sh --build
  3. View logs: docker compose logs -f sql_to_arc
  4. Iterate

Files

  • compose.dev.yaml - Docker Compose service definitions
  • config.dev.yaml - Application configuration
  • client.crt - Client certificate (plain)
  • client.key - Client private key (encrypted with sops)
  • start-dev.sh - Startup script with sops integration
  • run.sh - DEPRECATED - Old script (kept for reference)