Merged
Conversation
Add HMACAlgorithm enum (MD5/SHA256) to support modems with different HNAP authentication requirements. Algorithm is now a required field in modem.yaml, enforcing explicit declaration per modem. Changes: - Add HMACAlgorithm enum to core/auth/types.py - HNAPAuthConfig requires hmac_algorithm (validates in __post_init__) - HNAPJsonRequestBuilder uses configurable _hmac() method - Schema enforces hmac_algorithm as required field - Update S33 and MB8611 modem.yaml with hmac_algorithm: md5 - Update parsers to pass hmac_algorithm when creating builders This enables S34 support (SHA256) while maintaining backwards compatibility with existing MD5-based modems. Related to PR #90 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Architecture Changes: - Extract restart logic from parsers to core/actions layer - ActionFactory.supports() as single source of truth for action capabilities - Data-driven actions configured via modem.yaml actions section - Remove RESTART from Capability enum (actions != capabilities) - Schema strictness with extra='forbid' to catch invalid fields New Modules: - core/actions/ - ActionFactory, HNAPRestartAction, HTMLRestartAction, RESTRestartAction - core/restart_monitor.py - Post-restart polling extracted from button.py - modem_config/capabilities.py - Capability checking utilities Parser Changes: - Remove restart() methods from all parsers (now in action layer) - Parsers are pure data transformers, no network calls Button Changes: - Rename CaptureHtmlButton to CaptureModemDataButton - Use RestartMonitor for post-restart monitoring - Use coordinator.scraper instead of creating new scrapers modem.yaml Changes: - Remove '- restart' from capabilities lists - Add actions.restart configuration for each modem - HNAP modems use pre_fetch_action for preserving settings Tech Debt: - Add item #13: Diagnostics protocol-specific logic Related: #90 (S34 support by @rplancha identified need for configurable HMAC) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Post-release review from v3.12.0 identified gaps between local validation and CI environment that caused release delays: - Version drift between CI and pre-commit tool versions - Missing scripts/ci-check.sh referenced by Makefile - Workflow drift between tests.yml and release.yml - CodeQL pre-commit hook requires local CLI installation - Pre-commit only checks staged files, not full project Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Resolves Tech Debt #5 - duplicate login detection implementations. Created core/auth/detection.py with three primitives: - has_password_field(): Lenient string search (fast) - has_login_form(): Strict DOM parsing (requires <form> tag) - is_login_page(): Alias for session expiry checks Added session expiry handling in modem_scraper._authenticate(): - Detects when fetch returns login page (session expired) - Re-fetches data URL after successful re-authentication - Form-specific but no-op for HNAP (re-auths every poll anyway) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
HA 2025.11 removed home-assistant.log for HAOS/Supervised users, and system_log only captures WARNING/ERROR. This left diagnostics without INFO-level logs needed for debugging auth discovery issues. Adds a circular buffer (200 entries max) that captures INFO+ logs from our integration independently of HA's logging infrastructure. Logs are written to both our buffer and the standard HA logging system. - Add core/log_buffer.py with LogBuffer and BufferingHandler - Initialize buffer at start of async_setup_entry - Update diagnostics to try our buffer first, fall back to system_log/file - Add unit tests for log buffer Related to #83, #93 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement paradigm-aware HNAP authentication in the discovery pipeline: Architecture changes: - AuthDiscovery._handle_hnap_auth() now performs actual authentication using HNAPJsonRequestBuilder, not just detection - Algorithm discovery: tries MD5 first, then SHA256 (no modem.yaml lookup needed since we don't know which modem yet) - hnap_builder field added to AuthResult, DiscoveryResult, and DiscoveryPipelineResult for passing authenticated builder through - validate_parse() accepts optional hnap_builder for HNAP API calls Data flow for HNAP modems: Step 1: Connectivity check -> working_url Step 2: Auth discovery -> authenticated hnap_builder + hnap_config Step 3: Parser detection -> selected_parser (required for HNAP) Step 4: Validation -> uses hnap_builder via ResourceLoaderFactory Key behaviors: - HNAP credentials validated at setup time, not runtime - HNAP without selected_parser returns helpful error message - Working hmac_algorithm stored in hnap_config for runtime use - html=None handled gracefully throughout pipeline Related to #102 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Document the parallel auth paths introduced by the v3.13.0 HNAP fix: - Discovery-time auth in _handle_hnap_auth() creates builder, validates, discards - Runtime auth in handler.py creates new builder from scratch Lists remediation options and affected files. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
E2E Test Framework: - Fix modems_root to use repo modems/ (has fixtures) - Add hmac_algorithm support for HNAP tests - 74 tests now pass across 19 modems Mock Modem CLI (scripts/mock_modem.py): - Standalone server for development testing - Auto-discovers modems from modems/ directory - Supports HNAP, Form, URL Token auth strategies - Usage: .venv/bin/python scripts/mock_modem.py motorola/mb8611 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds TestDiscoveryPipelineE2E that runs the ACTUAL discovery pipeline (run_discovery_pipeline) against MockModemServer. This is the same code path users experience during config_flow setup. Critical addition after issue #102 analysis: existing E2E tests bypassed discovery by using direct auth (HNAPJsonRequestBuilder, form POST). The HNAP pipeline bug (html=None causing AssertionError) was not caught because no test exercised the real discovery flow. New tests: - test_discovery_pipeline_succeeds: Full pipeline with selected parser - test_discovery_pipeline_auto_detection: Auto-detect parser from HTML Supported strategies: FORM, HNAP TODO: Add mock support for NONE, BASIC, URL_TOKEN, REST_API Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Reorder get_url_patterns() to return protected pages before public pages. This prevents false session expiry detection on modems like MB7621 where public pages (e.g., root URL) always show login form regardless of session state. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add "modem.yaml as source of truth" architecture where known modems skip dynamic auth discovery and use verified config directly. Core changes: - Add get_static_auth_config() to ModemConfigAuthAdapter - Add create_authenticated_session() for static auth flow - Update run_discovery_pipeline() with static_auth_config parameter - Add load_static_auth_config() helper for config flow Config flow changes: - Remove "auto" mode - user must select their modem model - Mandatory modem selection enables static auth path Test infrastructure: - Fix HNAPAuthMockHandler to handle POST requests for SOAP auth - Add comprehensive tests for static auth config - Split E2E tests into dynamic vs static auth paths Version bump: 3.12.1 → 3.13.0 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Migrate all modem.yaml files to use auth.types{} schema
- Update adapter.py with convenience methods for discovery (get_hnap_hints,
get_js_auth_hints, get_auth_form_hints)
- Update mock_modem_server.py and mock_handlers to use new schema
- Fix integration tests to use config.auth.types["form"] path
- Fix modem.yaml files using basic: {} to basic: null
- Add noqa comment for workflow.py complexity warning
This consolidates fragmented auth fields (auth.strategy, auth.form, auth.hnap,
auth.url_token) into a unified auth.types{} structure as the single source
of truth for auth configuration.
Related to auth architecture refinement plan in PLAN.md
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The fixture-level README files duplicated information already in modem.yaml (auth config, capabilities, status). Removed 19 fixture READMEs. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Move unique documentation from fixture README.md files into modem.yaml using standardized fields: - `isps:` - List of compatible ISPs - `notes:` - Operational notes, limitations, quirks - `references:` - External links (security advisories, community posts) Affected modems: SB8200, MB8611, S33, SB6190, SB6141, CM820B, G54, MB7621, MB8600, C3700, C7000v2, CM600, CM1200, CM2000, TC4400, CGA2121, XB7, SuperHub5 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Delete 19 fixtures/metadata.yaml files (data now in modem.yaml) - Add source_code references to modem.yaml for sb6141, sb6190, sb8200, mb8611, tc4400 - Update sb8200 and sb6190 status to awaiting_verification (open issues #81, #83) - Remove obsolete metadata.yaml code from generate_fixture_index.py - Update MODEM_DIRECTORY_SPEC.md to remove metadata.yaml references - Make mock-server.md and mock_modem.py use generic placeholders - Remove unused parser_detection_history from diagnostics Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
G54 parser returns channel IDs like "OFDM-0" and "OFDMA-1" but
__init__.py assumed all channel IDs were numeric, causing:
invalid literal for int() with base 10: 'OFDM-0'
Changes:
- Add _extract_channel_id() helper to handle both numeric ("1", "32")
and prefixed ("OFDM-0", "OFDMA-1") channel IDs
- Add comprehensive tests for channel ID extraction and normalization
- Add mock server CLI: make mock MODEM=g54 for manual testing
Bug discovered during mock server testing with G54 fixtures.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Form auth was discarding the HTML response when auth succeeded without a success_indicator. This broke modems like CGA2121 that return data directly in the login response. Fixes: - form_plain.py: Return response.text in AuthResult.ok() - modem_scraper.py: Copy cookies to CapturingSession for capture mode - modem_scraper.py: Use ActionFactory.supports() for restart validation Also: - Change mock server test password from "password" to "pw" (avoids Chrome password manager warnings during manual testing) - Add regression tests for form auth HTML return - Add comprehensive tests for restart validation - Export ActionType from actions module Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
CGA2121 login page uses type="Password" (capital P) which was not matched by the case-sensitive check. This caused auth discovery to fail for modems with non-lowercase type attributes. The original fix (2daeff0) was lost during the v3.12.1 release merge. This restores the fix and adds a regression test. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
On reload, a new LogBuffer was created but the existing BufferingHandler still pointed to the old buffer. This caused diagnostics to show "No logs captured yet" even after logging activity. Fix: Reuse the existing handler's buffer on reload instead of creating a new one. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Use "pw" instead of "password" for mock server credentials to avoid browser password managers flagging these as real credentials during manual testing and development. Updated: - Mock handlers (form, hnap, url_token) - Integration test conftest files - All E2E and auth test files Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds a dropdown to config flow step 1 allowing users to choose an entity naming prefix: None, Model, or IP Address. This prevents entity ID conflicts when adding multiple modems. Options: - none: "Cable Modem" -> sensor.cable_modem_downstream_1_power - model: "Cable Modem MB7621" -> sensor.cable_modem_mb7621_downstream_1_power - ip: "Cable Modem 192_168_100_1" -> sensor.cable_modem_192_168_100_1_downstream_1_power First modem defaults to "none" for backward compatibility. Second+ modem defaults to "model" (no "none" option to force uniqueness). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
CLI tool to exercise mock modem server with the full scraper flow: - Discovery pipeline (config flow simulation) - Scraper polling (HA coordinator simulation) - Auth failure handling Usage: python scripts/mock_modem.py technicolor/cga2121 --port 9080 python scripts/test_mock_modem.py technicolor/cga2121 --port 9080 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
) Add ParsingError and ResourceFetchError exception types with context attributes (field, raw_value, url, status_code) for better diagnostics. Refactor ~36 except blocks across core modules: - discovery.py: requests.RequestException for HTTP, specific tuples for config loading and parsing - modem_scraper.py: specific exceptions for network/config/parsing ops - steps.py: (RequestException, OSError) for connectivity checks Pattern: specific exceptions first, fallback Exception with exc_info=True for truly unexpected errors. Comments mark intentionally broad catches. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- mb7621: update password to "pw" (credential standardization) - sb8200: update status assertion to awaiting_verification - cm1200, cga2121, superhub5: check modem.yaml instead of metadata.yaml - scripts: remove unused re import Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The name "ModemScraper" implied web scraping, but the class is an orchestrator that coordinates loaders, parsers, and actions. Changes: - Renamed core/modem_scraper.py → core/data_orchestrator.py - Renamed test_modem_scraper.py → test_data_orchestrator.py - Updated ~100 class/module references across 22 files - Preserved historical references in CHANGELOG.md Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Extract 982-line __init__.py into three focused modules: - channel_utils.py: Channel normalization and lookup functions - services.py: Dashboard generation and history clearing services - coordinator.py: Health monitor and update function creation Also fixes missing entity_prefix translation in en.json. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Prevent CI failures by running full validation before push: - Added pre-push-lint hook (ruff check .) - Added pre-push-tests hook (pytest) - Updated setup.py to install pre-push hooks automatically - Updated CONTRIBUTING.md with installation instructions - Removed overstated CI sync tech debt item, replaced with focused fix Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add 24 new tests covering previously untested areas: - ValidationProgressHelper state machine (11 tests) - async_step_auth_type flow (2 tests) - Entity prefix conditional logic (3 tests) - Options flow credential preservation (4 tests) - Options flow detection preservation (2 tests) - Exception classification edge cases (2 tests) Discovered HA deprecation: OptionsFlow.config_entry setter now throws RuntimeError. Tests use object.__setattr__ workaround. Also deprioritize AuthDiscovery refactor (#4) since it's only used by fallback modem workflow - known modems bypass it via modem.yaml hints. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add 'set -e' to devcontainer CI script to ensure the workflow fails if any command fails (pytest collection, ruff check, etc.). Previously the script would continue despite errors and report success even when smoke tests failed.
Fixed "Reset Entities" losing user's auth type selection for modems with multiple auth variants (SB6190, SB8200). Root Cause: Options flow didn't preserve CONF_AUTH_TYPE from config entry, causing modems to default to first auth type in modem.yaml. For SB6190: - User selected 'form_nonce' during initial setup ✓ - "Reset Entities" defaulted to 'none' ✗ - All subsequent polls failed (auth response vs channel data) Fix: - Preserve auth_type in _preserve_credentials() like credentials - Add auth_type parameter to load_static_auth_config() Fixes #93 Related #83
Fixed TC4400 parser returning 0 channels in v3.13.0 betas despite successful authentication. Root Cause: Same issue as SB6190 (#93) - orchestrator stores auth response HTML in resources["/"] instead of actual channel data page. Fix: Parser now prioritizes explicit path (/cmconnectionstatus.html) over root path ("/") to avoid receiving auth response HTML. Before: soup = resources.get("/") # Could be auth HTML or data page After: soup = resources.get("/cmconnectionstatus.html") # Explicit data page if soup is None: soup = resources.get("/") # Fallback Fixes #94
Fixed browser lockout and session issues for Netgear C7000v2. Changes: - Corrected session cookie name: XSRF_TOKEN (not 'session') - Corrected logout endpoint: /goform/logout (not /Logout.htm) - Added POST support for /goform/ logout endpoints in orchestrator - Updated mock server to handle /goform/logout correctly - Updated session limit test to use correct endpoint Root Cause: Incorrect session cookie and logout endpoint caused: - Browser lockouts (couldn't release session) - Integration failing to manage single-session modem properly Fixes #61
Upgrade to har-capture 0.3.2 to automatically redact WiFi credentials and device names in diagnostics captures. Changes: - Bump har-capture requirement from >=0.2.4 to >=0.3.2 - Import HeuristicMode enum from har_capture.sanitization - Use heuristics=HeuristicMode.REDACT for auto-redaction: - diagnostics.py: 2 sanitize_html() calls - tools/capture_modem_html.py: 1 sanitize_html() call - tests/lib/test_html_helper.py: 3 test cases - Updated tests for har-capture 0.3.1+ API: - sanitize_har() now returns tuple: (sanitized_data, report) - sanitize_html() still returns string - Removed real password from test data (security fix) - Added har-capture to devcontainer test requirements har-capture 0.3.2 API: HeuristicMode enum controls heuristic behavior: - DISABLED (default) - Only redact known patterns (MACs, IPs, etc.) - FLAG - Flag suspicious values for manual review - REDACT - Auto-redact suspicious values (WiFi creds, device names) Background: har-capture 0.3.1 changed from auto-redact to flag-only for heuristically detected sensitive values, causing PII leaks in diagnostics. Version 0.3.2 adds the HeuristicMode enum to restore auto-redaction with better control. Related: #61 (C7000v2 WiFi credentials in diagnostics) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Updated test_sanitizes_device_names_before_ip to accept any hash-based
redaction prefix (WIFI_, DEVICE_, etc.) instead of requiring specific
DEVICE_ prefix.
har-capture 0.3.2 may categorize device names differently based on
context, but the important validation is that sensitive values are
actually redacted (not present in output).
Test now checks for pattern: [A-Z]+_[a-f0-9]{8}
Example matches: WIFI_91ac6f52, DEVICE_5e3b8999, CRED_abc12345
Updates har-capture from 0.3.2 to 0.3.3 to fix critical sanitization bugs: - IPv4 addresses now produce valid 4-octet format (was producing 5 octets) - Version strings (e.g., "5.7.1.5") are preserved instead of being sanitized as IPs Changes: - manifest.json: har-capture>=0.3.3,<0.4.0 (explicit semver range) - pyproject.toml: har-capture~=0.3.3 (equivalent, cleaner syntax) - Added regression tests to test_html_helper.py (generic, not version-specific) - Updated CHANGELOG.md [Unreleased] section Semver constraints prevent breaking changes from auto-installing: - Patch updates OK: 0.3.4, 0.3.5 will auto-install - Minor bumps blocked: 0.4.0 will NOT auto-install (requires manual review) Investigation documentation archived to ~/projects/personal/journal for reference. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Adds user-facing documentation for har-capture sanitization library that was integrated in v3.13.0-beta.10. Documents the PII sanitization feature in README security section and adds HAR file sanitization instructions to troubleshooting guide. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The success.indicator "CM3500B" was not present on the redirect page, causing auth validation to fail. Related to #73 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Modems with micro_httpd (e.g., TC4400AM) don't support HEAD requests, causing the health monitor's HEAD-then-GET fallback to fail due to aiohttp session poisoning. This adds HEAD detection during setup, persists it in the config entry, and uses only the proven method at runtime — eliminating the fallback chain entirely. Related to #94 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Mock test_http_head in config_flow TestValidateInput (unmocked call left orphaned aiohttp thread causing teardown errors) - Restore inline comments explaining aiohttp async context manager mock pattern in test_health_monitor.py - Document why health_monitor.py uses aiohttp (parallel ping+HTTP for reboot detection) vs requests used elsewhere Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RAW_DATA contains contributor submissions and unreleased future work that should not be part of the build or quality gates. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge 11 beta sections (beta.1 through beta.11) into a single v3.13.0 entry organized by Added/Changed/Fixed/Improved categories. Required for CI release workflow which extracts notes between version headers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…HA event loop Found during dogfood testing — ssl.create_default_context() reads system CA certs from disk, triggering HA's blocking I/O detection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove cookies.txt (accidental libcurl cookie jar) - Remove PLAN.md and ai-skills/ (moved to RAW_DATA/) - Remove deploy_updates.sh (use local HA via Docker instead) - Update docs to reflect Docker-based dogfooding workflow - Add cookie jar patterns to .gitignore Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
kwschulz
force-pushed
the
feature/v3.13.0
branch
from
f419801 to
c5c7d36
Compare
ℹ️ Personal Email Detected in Commits
One or more commits in this PR contain personal email addresses:
Why This MattersPersonal emails in git history are permanently public and can be harvested by spammers. How to Fix (Optional but Recommended)
For Future CommitsOnce configured, all your future commits will use the privacy-safe email automatically. This check helps protect contributor privacy. Feel free to merge if you're okay with the email being public. |
- Drop Python 3.11 from test matrix (HA 2025.x requires 3.12+) - Migrate str,Enum to StrEnum across 11 enum classes (ruff UP042) - Auto-fix str.split() maxsplit warnings (ruff PLC0207) - Replace unnecessary lambdas with method refs (ruff PLW0108) - Add har-capture dependency to PII check and Code Quality jobs - Fix JS syntax error in commit email check workflow by passing data through process.env instead of template literal interpolation - Fix CodeQL alerts: enforce TLS 1.2+ in test server, initialize parser_class before use, remove duplicate re import Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
kwschulz
force-pushed
the
feature/v3.13.0
branch
from
c5c7d36 to
d6bb59e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Promote v3.13.0-beta.11 to stable. Adds 2 new modems, fixes 9 existing modem issues, and refactors core architecture.
Highlights
form_noncestrategy for SB6190 firmware 9.1.103+ssl.create_default_context()moved to executor to eliminate blocking I/O warning during config flowPost-beta.11 changes
fix: move ssl.create_default_context() to executor in test_http_head— found during dogfood, 2-line fix with passing testschore: remove deploy_updates.sh, use local HA for dogfooding— replaced SSH deployment with local Docker HAVerification
Merge instructions
Use "Create a merge commit" (not squash) — per project convention for release branches.
Related to #81, #93, #83, #107, #61, #94, #75, #73, #1, #6, #14, #20
🤖 Generated with Claude Code