Consolidate node configs, update compose files, refactor README

[spreston8]

Summary

• Config consolidation: Replace all role-specific config files with 2 unified configs (default.conf, standalone-dev.conf) shared by both Rust and Scala nodes. Per-role behavior controlled entirely via CLI flags (--ceremony-master-mode, --heartbeat-disabled, --disable-mergeable-channel-gc).
• Compose updates: All compose files use named volumes, unified f1r3fly network name, F1R3_* env vars for Rust shard, ${VAR:-default} fallbacks on all env vars. Bootstrap key unified across Rust and Scala.
• Integration-tests unification: Removed 5 local config files from integration-tests/conf/, deleted duplicate integration-tests/.env.node — tests now use top-level conf/default.conf, conf/standalone-dev.conf, and .env.node directly. F1R3_* env vars added to Rust integration compose and custom shard. Updated conftest.py for docker compose v2 detection.
• Fix: port-wait cleanup race condition: _wait_for_port_free() had a safety net that called _force_cleanup_custom_containers() when ports were stuck in TIME_WAIT. When triggered from add_peer_to_shard (inside a running shard), this destroyed the active shard and Docker network, causing "network f1r3fly-test-custom not found" on the joiner start. Fixed by adding a force_cleanup parameter — set to False when called from within an active shard context.
• Monitoring stack alignment: Prometheus switched from hardcoded static targets to DNS-based service discovery — only running nodes get scraped, no false DOWN targets for light shard or standalone (fixes Prometheus scrape targets are hardcoded for full shard — shows DOWN targets with light shard #32). Recording rules loaded in all repos (was missing rule_files in f1r3node repos). cAdvisor added to all three repos. block-transfer.json dashboard moved into Grafana provisioning directory. f1r3node.json dashboard synced (24KB with cAdvisor panels). Network naming aligned (name: f1r3fly everywhere).
• CI pipeline: Added .github/workflows/smoke-test.yml with 16 parallel jobs — compose validation, 5 topology health checks, 6 integration tests, and 4 monitoring validation jobs (Rust shard/standalone, Scala shard/standalone) with end-to-end assertions (targets UP, 0 DOWN, rules loaded, metric data present, cAdvisor metrics, Grafana dashboards, Grafana->Prometheus connectivity).
• README refactor: Restructured with shardctl-first flow, complete CLI reference (all 20 commands), extracted docs. Monitoring section with DNS discovery info and dashboard list. Removed smoke-test.sh in favor of CI.
• Prerequisites: Python 3.10 first, Rust before Scala, Java 11→17, manual install for Rust (no Nix required), link to COMPOSE_STRUCTURE.md.
• shardctl fixes: Docker Compose v2 plugin detection (fixes docker-compose not found on GitHub runners), fixed shardctl status duplicate --env-file flags.

Related PRs & Issues

• f1r3node#447 — Rust docker config alignment + monitoring
• f1r3node#448 — Scala docker config alignment + monitoring
• f1r3node#441 — Scala GC crash during genesis
• f1r3node#442 — CLI flags for config settings (resolved)
• f1r3node#452 — Scala validator stuck in Initializing on concurrent state sync
• Fixes #32 — Prometheus hardcoded targets show DOWN for light/standalone

Known Issues

• Scala full shard (5 nodes) intermittently fails on 2-core/7GB GitHub runners — one validator gets stuck in Initializing (f1r3node#452). Marked continue-on-error in CI. Scala light shard (3 nodes) and all Rust topologies pass reliably.
• F1R3_SYNCHRONY_* env vars commented out in .env.node to avoid overriding per-validator CLI settings in test_synchrony_constraint. Set in compose YAML environment: sections instead.

CI Status

Job	Status
Validate & CLI	Pass
Rust Shard (10 finalized)	Pass
Rust Standalone (10 finalized)	Pass
Scala Standalone (10 finalized)	Pass
Scala Light Shard (10 finalized)	Pass
Scala Shard (10 finalized)	Flaky (f1r3node#452)
Rust: test_web_api (shard)	Pass
Rust: test_heartbeat (standalone)	Pass
Rust: test_synchrony (custom)	Pass
Scala: test_web_api (shard)	Flaky (f1r3node#452)
Scala: test_heartbeat (standalone)	Pass
Scala: test_synchrony (custom)	Pass
Monitoring: Rust Shard	New
Monitoring: Rust Standalone	New
Monitoring: Scala Standalone	New
Monitoring: Scala Shard	New (allowed to fail)

Test plan

• Compose validation: 16/16 pass
• Rust shard: 5 nodes reach Running, 10+ finalized blocks
• Rust standalone: reaches Running, 10+ finalized blocks
• Scala standalone: reaches Running, 10+ finalized blocks
• Scala light shard: 3 nodes reach Running, 10+ finalized blocks
• Scala full shard: intermittent (f1r3node#452)
• Integration tests (Rust): test_web_api, test_heartbeat, test_synchrony all pass
• Integration tests (Scala): test_heartbeat, test_synchrony pass
• Integration tests (Scala): test_web_api intermittent (f1r3node#452)
• Monitoring: local test — Rust shard 5 UP / 0 DOWN, rules loaded, 2 dashboards
• Monitoring: local test — Rust standalone 2 UP / 0 DOWN (issue Prometheus scrape targets are hardcoded for full shard — shows DOWN targets with light shard #32 regression)

Co-Authored-By: Claude noreply@anthropic.com

Follow-up PRs

• F1R3_ env var handling*: Current approach of inlining env vars in compose YAML and commenting out in .env.node needs cleanup. See docs/TODO.md.
• Validator bonding & additional validators: Support for bonding new validators and adding additional validator nodes to a running shard (validator4, validator5, etc.) via shardctl is planned for a follow-up PR.
• Rust metric dashboard queries (system-integration#22): Phase 1 observability gauges (f1r3node#405) add 16 new metrics that need new dashboard panels.

[spreston8]

Merging. All review items will be addressed in follow-up PR.