A comprehensive Golang application for managing Shelly smart home devices with Kubernetes-native architecture, dual-binary design for secure WiFi provisioning, and advanced configuration management with complete normalization and comparison capabilities.
- Main API Server (
shelly-manager): Runs in Kubernetes, manages device database, provides REST API with standardized responses - Provisioning Agent (
shelly-provisioner): Runs on host with WiFi access, handles device provisioning with enhanced error handling - Communication: Provisioning agent connects to main API for instructions and device registration with comprehensive task orchestration
- Typed Configuration Models: Complete structured configuration with validation for all device types
- Configuration Normalization: Server-side normalization for accurate comparison between raw and saved configurations
- Bidirectional Conversion: Seamless conversion between raw JSON and typed structures with complete field preservation
- Enhanced Device Support: Full support for relay devices (Shelly 1, Plus 1, etc.), smart plugs, power meters, and 3-input controllers
Current Version: v0.5.4-alpha
Status: Production-ready with comprehensive security framework, 82.8% database test coverage, and advanced configuration normalization
Repository Scale: 165 Go files, 77,770 lines of code, 31 packages across 19 internal modules
Testing: 69 test files with comprehensive security validation and isolation framework
API Coverage: 112+ endpoints across 6 handler modules with standardized security responses
The project is now entering a strategic modernization phase to transform from a basic device manager to a comprehensive infrastructure platform. With 80+ backend endpoints across 8 functional areas, only ~40% of backend functionality is currently exposed to users through the frontend.
- β Substantial Backend Investment: 80+ endpoints across 8 major functional areas
β οΈ Limited Frontend Integration: Only ~40% of backend functionality exposed to users- β Critical Systems Unexposed: Export/Import (0%), Notification (0%), Metrics (0%) systems with zero frontend integration
β οΈ Technical Debt: 70% code duplication across 6 HTML files (9,400+ lines)
- Phase 6.9: Security & Testing Foundation (Critical Prerequisite)
- Phase 7: Backend-Frontend Integration with security controls
- Phase 8: Vue.js Frontend Modernization with security-first design
- Phase 1: β Core Shelly Device Management - Complete REST API, device authentication, real device integration
- Phase 2: β Dual-Binary Architecture - API server + provisioning agent with complete inter-service communication
- Phase 2.5: β Template System Enhancement - Sprig v3 integration, security controls, template inheritance
- Phase 3: β JSON to Structured Migration - Typed configuration models, bidirectional conversion, API endpoints
- Phase 4: β User Interface Enhancement - Modern structured forms, configuration wizards, real-time validation
- Phase 5: β Container & Kubernetes Integration - Production-ready containerization and security hardening
- Phase 5.1: β API Integration Enhancement - Complete provisioner-API communication with comprehensive testing
- Phase 5.1.1: β Discovered Device Database Persistence - Real-time device discovery with database integration
- Phase 5.2: β UI Modernization - Complete discovered devices integration with modern web interface
- Phase 5.3: β Configuration Normalization & API Standardization - Complete field preservation, standardized responses
- Phase 6.9.2: β Comprehensive Testing Foundation - COMPLETED with critical security vulnerability fixes, 82.8% database coverage (29/31 methods), 63.3% plugin registry coverage, comprehensive test isolation framework
- Dual-Binary Architecture: API server (containerized) + provisioning agent (host-based) with full communication
- Advanced Configuration System: Complete normalization and comparison with field preservation, structured forms
- API Standardization: Consistent
{success: true/false, data/error}response format across all endpoints - Enhanced Device Support: Full support for all Shelly device types with accurate capability detection
- Production Deployment: Multi-stage Docker builds, Kubernetes manifests, security hardening
- Real Device Integration: Gen1 & Gen2+ Shelly devices with comprehensive API coverage and error handling
- Database Persistence: Discovered device storage with 24-hour TTL and automatic cleanup
- Modern Web UI: Real-time device discovery, configuration wizards, diff tools, responsive design with improved feedback
- Configuration Comparison: Server-side normalization enabling accurate configuration diff and validation
- Comprehensive Testing: Production-ready testing achieved with 82.8% database coverage, 63.3% plugin coverage, E2E testing across 5 browsers
- π Real-time Metrics: Complete WebSocket implementation with bounded ring buffers, automatic reconnection, and REST fallback
- π Enhanced UX: Schema-driven export/import forms with real-time JSON validation and preview capabilities
- π E2E Testing: Comprehensive Playwright infrastructure with 195+ scenarios, CI integration, and artifact collection
- Device Management: 112+ REST endpoints across 6 handler modules with standardized security responses
- Configuration: Advanced template-based configuration with normalization, inheritance, validation, and comparison
- Discovery: Real-time device discovery with database persistence and web UI integration
- Provisioning: Task-based orchestration between API server and provisioning agent with enhanced error handling
- Web Interface: Modern UI with structured forms, wizards, real-time feedback, and bulk operation progress
- API Integration: Comprehensive configuration comparison with server-side normalization for accurate field preservation
- Container Support: Production-ready Docker images and Kubernetes deployment with security hardening
- Database Providers: Multi-provider support with SQLite, PostgreSQL, MySQL (13 provider files)
- Plugin Architecture: Extensible plugin system with 19 files supporting sync, notification, and discovery
- Security Framework: Comprehensive security testing with vulnerability resolution and rate limiting protection
- π Export/Import System: Comprehensive SMA format, plugin architecture, scheduled operations, and management UI
- π Notification System: Multi-channel notifications (email, webhook, Slack) with rule engine and rate limiting
- π Real-time Metrics: WebSocket-based live dashboards with CPU, memory, disk monitoring and automatic reconnection
- π Enhanced Forms: Schema-driven export/import preview forms with JSON validation, copy/download, and localStorage persistence
- π E2E Testing: Multi-browser testing with Chromium, Firefox, WebKit + mobile, CI integration, and failure artifacts
The Shelly Manager includes a comprehensive export/import system that transforms your device management capabilities with complete backup, migration, and infrastructure integration features.
- Complete System Backup: Full device inventory, configurations, templates, and system settings
- Compressed & Versioned: Gzip compression with forward/backward compatibility
- Integrity Verification: SHA-256 checksums and record count validation
- Metadata Rich: Export context, system information, and audit trails
- Multiple Formats: SMA, Terraform, Ansible, Kubernetes, Docker Compose, JSON, CSV
- Export Preview: Preview operations before execution with record counts and size estimates
- Scheduled Exports: Automated backup scheduling with retention policies
- Filter Support: Export specific devices, templates, or configurations
- Download Protection: Safe download restrictions to prevent path traversal
- Validation First: Comprehensive validation before any changes are applied
- Dry Run Mode: Preview all changes before import execution
- Conflict Resolution: Smart handling of device MAC conflicts and template name collisions
- Backup Before Import: Automatic backup creation before applying changes
- Progress Tracking: Real-time import progress with detailed change reporting
- Extensible System: Plugin-based architecture for custom export formats
- Infrastructure as Code: Direct export to Terraform, Ansible, and Kubernetes manifests
- Container Support: Docker Compose and Kubernetes deployment configurations
- Custom Plugins: Easy development of custom export/import formats
- Admin Protection: All operations protected by admin API key when configured
- Audit Trails: Complete history of all export/import operations
- Safe Downloads: Configurable download restrictions to prevent unauthorized access
- Encrypted Support: Built-in support for encrypted SMA files (future enhancement)
# Export complete system to SMA format
./bin/shelly-manager export --format sma --output /backups/
# Export specific devices to Terraform
./bin/shelly-manager export --format terraform --devices 1,2,3 --output ./infrastructure/
# Import from SMA file with dry run
./bin/shelly-manager import --format sma --file backup.sma --dry-run
# Import with backup before changes
./bin/shelly-manager import --format sma --file backup.sma --backup-before# Export preview
curl -X POST http://localhost:8080/api/v1/export/preview \
-H "Authorization: Bearer <ADMIN_KEY>" \
-H "Content-Type: application/json" \
-d '{"plugin_name": "sma", "format": "sma"}'
# Get export history
curl -H "Authorization: Bearer <ADMIN_KEY>" \
http://localhost:8080/api/v1/export/history?page=1&page_size=20
# Import with validation
curl -X POST http://localhost:8080/api/v1/import/preview \
-H "Authorization: Bearer <ADMIN_KEY>" \
-F "[email protected]" \
-F 'options={"dry_run": true, "validate_only": true}'# Admin API key for export/import protection
SHELLY_SECURITY_ADMIN_API_KEY=your-secure-admin-key
# Safe download directory restriction
SHELLY_EXPORT_OUTPUT_DIRECTORY=/var/exports/shelly-manager
# Enable scheduled exports
SHELLY_EXPORT_SCHEDULE_ENABLED=truesecurity:
admin_api_key: "${ADMIN_API_KEY}"
export:
output_directory: "/var/exports/shelly-manager"
schedule_enabled: true
default_format: "sma"
compression_enabled: true
import:
validation_enabled: true
backup_before_import: true
conflict_resolution: "prompt" # prompt|overwrite|skip- Terraform: Export device configurations as Terraform resources
- Ansible: Generate Ansible playbooks for device management
- Kubernetes: Create ConfigMaps and Secrets for K8s deployments
- Provider Snapshots (Backup page): Create raw database snapshots from the current provider. For SQLite:
- Compression Off β single file:
*.sqlite - Compression On β single file:
*.sqlite.gz - No
.tar.gzcontainer; single-file outputs only.
- Compression Off β single file:
- Content Exports (Content Exports page): JSON, YAML, and SMA content exports live under
/export/content. - Disaster Recovery: Use backups for DB-level restore; use SMA for full content migration between instances.
- CI/CD Pipelines: Automated export/import in deployment workflows
- GitOps: Version-controlled infrastructure configurations
- Monitoring: Export/import operation metrics and alerting
For detailed API documentation, see docs/API_EXPORT_IMPORT.md
For SMA format specification, see docs/sma-format.md
# Build the application (both binaries)
make build
# Start server and auto-build UI if needed (recommended)
make start
# Alternative dev workflow
# - Start API server only (no UI build):
# make run
# - Run the UI dev server (hot reload) separately:
# make ui-dev # UI at http://localhost:5173
# # API at http://localhost:8080/api/v1
# - Build the UI once so the server serves ui/dist:
# make ui-build # then browse http://localhost:8080# List all devices
./bin/shelly-manager list
# Discover devices on network
./bin/shelly-manager discover 192.168.1.0/24
# Add device manually
./bin/shelly-manager add 192.168.1.100 "Living Room Light"
# Export operations
./bin/shelly-manager export --format sma --output /backups/
./bin/shelly-manager export --format json > devices.json
./bin/shelly-manager export --format csv > devices.csv
./bin/shelly-manager export --format terraform --devices 1,2,3 --output ./infrastructure/
# Import operations
./bin/shelly-manager import --format sma --file backup.sma --dry-run
./bin/shelly-manager import --format sma --file backup.sma --backup-before# Start API server
./bin/shelly-manager server --config /etc/shelly/config.yaml
# Start with specific port
./bin/shelly-manager server --port 8080# Build container
make docker-build
# Deploy to Kubernetes using Kustomize
kubectl apply -k k8s/
# Or deploy individual manifests
kubectl apply -f k8s/
# Check deployment status
kubectl get pods,svc,ingress -l app=shelly-managerSee k8s/README.md for comprehensive Kubernetes deployment documentation including TLS setup, monitoring configuration, and production considerations.
# Start services
docker-compose up -d
# View logs
docker-compose logs -f# Install
make install
# Run with systemd (Linux)
sudo systemctl start shelly-managerAll API endpoints return standardized responses in the format: {success: true/false, data/error: ...}
Detailed API references:
- Export/Import API: docs/API_EXPORT_IMPORT.md
- Notification API: docs/API_NOTIFICATION.md
- Metrics API: docs/METRICS_API.md
Admin Authorization (Export/Import):
- Configure
security.admin_api_keyin config to protect export/import (including previews, results, downloads, schedules, history, statistics). - Send
Authorization: Bearer <ADMIN_KEY>orX-API-Key: <ADMIN_KEY>with requests. - Optional safe downloads: set
export.output_directoryto restrict served files to that directory.
Secrets & Secure Config:
- See
docs/SECURITY_SECRETS.mdfor Kubernetes Secrets and Docker Compose.envexamples (ADMIN_API_KEY,EXPORT_OUTPUT_DIR).
TLS/Proxy Hardening:
- See
docs/SECURITY_TLS_PROXY.mdfor NGINX/Traefik examples enforcing HTTPS, HSTS, and secure headers.
GET /api/v1/devices- List all devices with standardized response formatPOST /api/v1/devices- Add new deviceGET /api/v1/devices/{id}- Get device detailsPUT /api/v1/devices/{id}- Update deviceDELETE /api/v1/devices/{id}- Delete device
GET /api/v1/devices/{id}/config- Get saved device configurationPUT /api/v1/devices/{id}/config- Update device configurationGET /api/v1/devices/{id}/config/current- Get current live device configurationGET /api/v1/devices/{id}/config/current/normalized- Get normalized current configurationGET /api/v1/devices/{id}/config/typed- Get typed configuration with conversion infoGET /api/v1/devices/{id}/config/typed/normalized- Get normalized typed configurationPOST /api/v1/devices/{id}/config/import- Import configuration from deviceGET /api/v1/devices/{id}/config/status- Get import statusPOST /api/v1/devices/{id}/config/export- Export configuration to device
POST /api/v1/discover- Trigger network discoveryGET /api/v1/provisioner/discovered-devices- Get discovered devices listPOST /api/v1/provisioner/report-device- Report discovered deviceGET /api/v1/provisioning/status- Provisioning statusPOST /api/v1/provisioning/start- Start provisioningGET /api/v1/provisioning/queue- List devices awaiting provisioning
GET /api/v1/export?format=json- Export devices as JSONGET /api/v1/export?format=csv- Export devices as CSVGET /api/v1/export?format=hosts- Export as hosts fileGET /api/v1/dhcp/reservations- Get DHCP reservationsPOST /api/v1/integrations/opnsense/sync- Sync with OPNSense
- Export APIs:
POST /api/v1/exportwith preview, history, and statistics - Import APIs:
POST /api/v1/importwith validation, preview, and dry-run - SMA Format: New
.smaformat for complete system backups and migration - Plugin System: Export to Terraform, Ansible, Kubernetes, Docker Compose
- Scheduled Operations: Automated export scheduling with retention policies
- Management UI: Vue.js interface for all export/import operations
- Security: Admin-protected endpoints with audit trails and safe downloads
See Export/Import System Documentation for comprehensive details.
GET /health- Health checkGET /ready- Readiness probeGET /metrics- Prometheus metrics (planned)
The Shelly Manager includes a comprehensive configuration management system with the following key features:
- Server-Side Normalization: All configurations are normalized on the server side for accurate comparison
- Complete Field Preservation: All device configuration fields are captured and preserved, including raw fields
- Bidirectional Conversion: Seamless conversion between raw JSON and typed structures
- Comparison-Ready Format: Normalized configurations enable accurate diff and comparison operations
| Device Type | Model Examples | Supported Features |
|---|---|---|
| Relay Switches | Shelly 1, Shelly Plus 1, SHSW-1 | Relay configuration, auto-on/off, default state, button type |
| Smart Plugs | Shelly Plug, Shelly Plus Plug | Power metering, relay control, energy monitoring |
| Power Meter Switches | Shelly 1PM, Shelly Plus 1PM | Power monitoring, relay control, energy thresholds |
| Multi-Input Controllers | SHIX3-1 (3-Input) | Input configuration, button types, timing settings |
- Typed Configuration Models: Complete structured models for all device configuration sections
- Template-Based Configuration: Sprig v3 template engine with security controls and inheritance
- Real-Time Validation: Configuration validation with device-specific context and warnings
- Configuration Diff: Visual comparison between current device state and saved configuration
- Bulk Operations: Mass configuration import/export with progress feedback
- Standardized Responses: All API endpoints return consistent
{success: boolean, data/error}format - Enhanced Error Handling: Comprehensive error responses with actionable information
- Configuration Endpoints: Multiple endpoints for different configuration views (raw, typed, normalized)
cmd/
βββ shelly-manager/ # Main API server binary
βββ shelly-provisioner/ # WiFi provisioning agent
internal/
βββ api/ # REST API handlers and configuration normalization
βββ config/ # Application configuration management
βββ configuration/ # Device configuration models and services
βββ database/ # Models and database operations
βββ discovery/ # Device discovery (HTTP/mDNS/SSDP)
βββ provisioning/ # WiFi provisioning logic and network interfaces
βββ service/ # Business logic layer
βββ logging/ # Structured logging
βββ metrics/ # Metrics collection and monitoring
βββ integration/ # External system integrations (planned)
βββ opnsense/ # OPNSense API client
βββ export/ # Export formatters
- NormalizedConfig: Unified configuration structure for comparison
- Bidirectional Conversion: Raw JSON β Typed Configuration β Normalized Format
- Complete Field Preservation: Captures all device fields including unknown/additional fields
- Device-Aware Processing: Handles device-specific capabilities and configurations
- Standardized Responses: Consistent
{success, data/error}format across all endpoints - Configuration Endpoints: Multiple views of device configuration (raw, typed, normalized)
- Enhanced Error Handling: Comprehensive error context and actionable messages
- Bulk Operation Support: Progress feedback and status reporting
- Typed Models: Complete structured models for all configuration sections
- Template Engine: Sprig v3 integration with security controls
- Service Layer: Business logic for configuration conversion and validation
- Device Support: Relay switches, smart plugs, power meters, input controllers
Current Design (20-100 devices)
- SQLite database (sufficient for <1000 devices)
- Single API server instance
- In-memory caching for device status
- Polling-based discovery
Future Scaling (1000+ devices)
- Migration path to PostgreSQL
- Horizontal scaling with Redis cache
- Event-driven architecture with message queue
- Batch operations for bulk updates
- Connection pooling for device communications
- Partitioned discovery with worker pools
- Supports environment variable overrides with
SHELLY_prefix and nested key mapping using underscores. - Precedence: environment > config file > defaults.
- Examples:
SHELLY_SERVER_PORT=9091overridesserver.port.SHELLY_SERVER_LOG_LEVEL=debugoverridesserver.log_level.SHELLY_DATABASE_PROVIDER=postgresqloverridesdatabase.provider.SHELLY_DATABASE_DSN="host=localhost user=app dbname=shelly sslmode=disable"overridesdatabase.dsn.- Arrays and complex types should be configured via file where possible; use env for scalars and secrets.
server:
port: 8080
host: 0.0.0.0
database:
path: /var/lib/shelly/shelly.db
discovery:
networks:
- 192.168.1.0/24
timeout: 5s
concurrent_scans: 10
provisioning:
wifi_ssid: "ProductionNetwork"
wifi_password: "${WIFI_PASSWORD}" # From environment
device_password: "${DEVICE_PASSWORD}"api:
url: http://shelly-api.default.svc.cluster.local:8080
key: "${API_KEY}"
scanner:
interval: 60s
interface: wlan0 # WiFi interface to use
provisioning:
timeout: 30s
retry_count: 3- Environment variable support for secrets
- Input validation on all API endpoints
- SQL injection prevention via ORM
- Structured logging (no secrets in logs)
- API key authentication
- Rate limiting
- HTTPS/TLS support
- Encrypted configuration storage
- Audit logging
# Run all tests
make test
# Run with coverage
make test-coverage
# Run specific package tests
go test ./internal/discovery/...
# Run integration tests
make test-integration- Structured logging with slog
- Request/response logging middleware
- Error tracking and reporting
- Prometheus metrics endpoint
- Health check endpoints for Kubernetes
- Distributed tracing support
- Custom Grafana dashboards
- JSON: Full device details for programmatic access
- CSV: Spreadsheet-compatible format
- Hosts: Unix hosts file format
- DHCP: ISC DHCP format
- Automatic DHCP reservation sync
- Firewall rule generation
- Alias management
- Home Assistant discovery
- MQTT publishing
- Webhook notifications
- Prometheus service discovery
- Package architecture
- Database layer
- Complete REST API with all endpoints
- Real Shelly device communication (Gen1 & Gen2+)
- Device authentication (Basic & Digest auth)
- Status polling and energy monitoring
- Comprehensive configuration management
- Web UI with error handling and authentication flow
- RBAC framework for 80+ API endpoints
- JWT authentication system for Vue.js SPA integration
- COMPLETED: Comprehensive testing strategy with security vulnerability resolution
- Fixed 6+ critical security issues including rate limiting bypass vulnerability
- Achieved 82.8% database manager test coverage (29/31 methods)
- Added comprehensive Plugin Registry tests (0% β 63.3% coverage)
- Implemented test isolation framework and systematic quality validation
- Resource validation and phase coordination protocols
- Database abstraction completion (PostgreSQL, MySQL)
- API standardization with security headers
- Export/Import system integration (21 endpoints) with encryption
- Notification system integration (7 endpoints) with access controls
- Metrics system enhancement with WebSocket security
- Real-time features with authentication and rate limiting
- Vue.js 3 + TypeScript foundation with security configuration
- API integration layer with authentication
- Core component development with input validation
- Advanced features UI with secure file handling
- Real-time dashboard with security validation
- Production deployment with penetration testing
- Composite Devices feature for advanced device grouping
- Monitoring and metrics (Prometheus)
- High availability setup
- Advanced automation features
- Enhanced security features
- API Documentation (planned)
- Deployment Guide (planned)
- Integration Guide (planned)
- Development Guide (planned)
- Repository Guidelines
- Tasks & Roadmap β single source of truth for ongoing work
- Changelog
This is primarily a personal project, but contributions are welcome!
See the contributor guide in AGENTS.md for project structure, commands, coding style, testing, and PR requirements. Before starting work, review TASKS.md; it is the single source of truth for current priorities and acceptance criteria.
Prerequisites:
- Go 1.23 or later (managed via mise)
- SQLite3 development libraries
- GCC (for CGO compilation)
# Clone repository
git clone https://github.com/ginsys/shelly-manager
# Install Go 1.23 (if using mise)
mise install
# Install dependencies
make deps
# Run tests
make test
# Build binaries
make buildMIT License - See LICENSE file for details
Current Version: v0.5.4-alpha
Status: Production-ready with comprehensive testing foundation, security hardening, and advanced configuration normalization
Testing Coverage: 82.8% database coverage, critical security vulnerabilities resolved, comprehensive test automation
Supported Devices: Shelly Gen1 & Gen2+ devices with comprehensive configuration support
Minimum Go Version: 1.23
Container Registry: ghcr.io/ginsys/shelly-manager
Architecture: Dual-binary (API server + provisioning agent) with standardized API responses and comprehensive testing
- CORS & Proxy Settings (Security)
- Configure via
securityin config orSHELLY_SECURITY_*env keys:security.use_proxy_headers: whether to trust proxy headers for client IP.security.trusted_proxies: list of trusted proxies (IPs/CIDRs) forX-Forwarded-Forparsing.security.cors.allowed_origins: list of allowed origins (empty = allow all; set explicit origins for production).security.cors.allowed_methods,security.cors.allowed_headers,security.cors.max_age.
- Example (YAML):
security: use_proxy_headers: true trusted_proxies: - 10.0.0.0/8 - 192.168.0.0/16 cors: allowed_origins: - https://app.example.com allowed_methods: [GET, POST, PUT, DELETE, OPTIONS] allowed_headers: [Content-Type, Authorization, X-Requested-With] max_age: 86400
- Example (env):
SHELLY_SECURITY_USE_PROXY_HEADERS=trueSHELLY_SECURITY_TRUSTED_PROXIES=10.0.0.0/8,192.168.0.0/16SHELLY_SECURITY_CORS_ALLOWED_ORIGINS=https://app.example.comSHELLY_SECURITY_CORS_ALLOWED_METHODS=GET,POST,PUT,DELETE,OPTIONSSHELLY_SECURITY_CORS_ALLOWED_HEADERS=Content-Type,Authorization,X-Requested-WithSHELLY_SECURITY_CORS_MAX_AGE=86400
- Configure via