Refactor lib/data.py for V2

[Morriz]

Objective

Refactor lib/data.py to load from projects/ structure with ${VAR} expansion from secrets/.

Prerequisites

Issue #6 completed (migration script exists as reference)

Implementation Steps

See todos/IMPLEMENTATION.md lines 1231-1443.

Key Changes:

1. Simplify models (remove complex db.yml schema, keep minimal Ingress/TraefikConfig)
2. Implement secret loading from secrets/*.txt files (decrypted)
3. Implement ${VAR} expansion using regex
4. Implement infrastructure loading from projects/traefik.yml
5. Implement project loading from projects/{name}/
6. Add validation functions

Success Criteria

• Loads from projects/ structure
• Expands ${VARS} from secrets/
• TraefikConfig model works
• No import errors
• Validation functions work

Estimated Time

1 hour

[Morriz]

@claude implement this

[claude]

Claude finished @Morriz's task —— View job • claude/issue-8-20251024-2354 • Create PR ➔

---

Refactoring lib/data.py for V2 ✅

Implementation Complete

• Read implementation details from todos/IMPLEMENTATION.md (lines 1231-1443)
• Read current lib/data.py (old db.yml-based implementation)
• Understand the new V2 architecture requirements
• Replaced lib/data.py with V2 implementation
• Simplified models (Ingress and TraefikConfig only)
• Implemented secret loading from secrets/*.txt files
• Implemented ${VAR} expansion using regex
• Implemented infrastructure loading from projects/traefik.yml
• Implemented project loading from projects/{name}/
• Added validation functions (validate_project, validate_all)
• Committed and pushed changes

Key Changes Implemented

1. ✅ Removed db.yml dependency (old system)
2. ✅ Load from projects/ directory structure
3. ✅ Secret loading from secrets/*.txt with dotenv
4. ✅ ${VAR} and $VAR expansion using regex pattern r'\$\{([^}]+)\}|\$([A-Z_][A-Z0-9_]*)'
5. ✅ Load infrastructure from projects/traefik.yml
6. ✅ Load projects from projects/{name}/docker-compose.yml
7. ✅ Minimal models: Ingress and TraefikConfig

New V2 API

The refactored lib/data.py now provides:

• load_secrets() - Load secrets from secrets/*.txt
• expand_env_vars(data, secrets) - Recursively expand ${VAR} in data structures
• load_infrastructure() - Load projects/traefik.yml with secret expansion
• load_project(name) - Load project docker-compose.yml and traefik.yml
• list_projects() - List all projects with docker-compose.yml
• validate_project(name) - Validate project configuration
• validate_all() - Validate all projects

Success Criteria

• ✅ Loads from projects/ structure
• ✅ Expands ${VARS} from secrets/
• ✅ TraefikConfig model works
• ✅ No import errors in the refactored module
• ✅ Validation functions work

Important Notes

⚠️ Expected Breaking Changes: This is a V2 architectural refactor. The following files still import old V1 functions and will need updates in subsequent implementation steps (see IMPLEMENTATION.md Step 10+):

• lib/proxy.py - imports get_plugin_registry, get_projects, get_versions
• lib/upstream.py - imports get_project, get_projects, get_service
• api/main.py - imports get_project, get_projects, get_service, upsert_env, upsert_project, upsert_service
• bin/validate-db.py - imports validate_db
• bin/write-artifacts.py - imports validate_db
• lib/data_test.py - imports various old functions

The V2 migration is intentionally breaking as it moves from the old db.yml-based system to the new projects/ directory structure. Follow-up steps in IMPLEMENTATION.md will update dependent files.

--- | Branch: claude/issue-8-20251024-2354