feat(cli): add CLI with CI convenience functions #394
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #394 +/- ##
==========================================
+ Coverage 70.84% 75.87% +5.02%
==========================================
Files 9 41 +32
Lines 2024 4738 +2714
==========================================
+ Hits 1434 3595 +2161
- Misses 590 1143 +553 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Chaffelson
force-pushed
the
feature/cli
branch
4 times, most recently
from
712afd5 to
dbabc28
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Collaborator
|
I really like how this is shaping up, let me know when you are out of draft |
- Add nipyapi CLI with 'ci' subcommand for NiFi CI/CD operations - Profile auto-resolution: switch() without args auto-detects env vars or profile file - CI functions: deploy_flow, start_flow, stop_flow, get_status, cleanup, configure_params - New functions: configure_inherited_params, purge_flowfiles, upload_asset - get_status defaults to root process group when no PG specified - Log control via NIFI_LOG_LEVEL and NIFI_LOG_ON_ERROR env vars - Suppress SSL warnings to keep CI output clean - Convert snake_case to kebab-case for GitHub Actions output format - resolve_git_ref to resolve tags/branches to SHAs
…cture Layout Module: - New nipyapi/layout.py with intelligent canvas positioning - Process group grid alignment (align_pg_grid, suggest_pg_position) - Flow layout functions (below, above, right_of, left_of, fork, new_flow) - Component movement with self-loop bend handling - Flow transposition and de-overlap capabilities - Spine/branch detection for flow restructuring (find_flow_spine, get_side_branches) - Automated flow layout (suggest_flow_layout) Canvas Enhancements: - get_flow_components() for graph traversal of connected components - Automatic bend calculation for self-loop connections - Bends parameter support in create_connection() CLI Improvements: - Expose layout functions via CLI (align_pg_grid, suggest_pg_position, etc.) - SafeModule wrapper for structured error handling - Custom serializer for formatted output Client Generation: - Add octet-stream handler to rest.mustache template - Regenerate NiFi and Registry clients with binary upload support Profile System: - Fix get_default_profile_name() to honor NIPYAPI_PROFILES_FILE env var - Improve profile resolution order for testing isolation CI Module: - Fix purge_flowfiles.py to use detail='all' for process group status Test Infrastructure: - Add NIPYAPI_PROFILES_FILE to Makefile for test isolation - Update conftest.py to use env var for profiles path - Add comprehensive tests for layout, CLI, CI, canvas, and parameters - 332 tests passing, lint clean (10.00/10)
- CI functions no longer call profiles.switch() internally - Profile configuration is now solely the responsibility of the caller - CLI main() handles switch() with --profile arg or auto-resolution - Fixes bug where --profile flag was ignored by CI commands Also includes: - Add --profile CLI option for explicit profile selection - Add NIPYAPI_PROFILE env var support in switch() auto-resolution - Update profiles.rst documentation - Add tests for new profile resolution behavior
- Regenerate NiFi and Registry clients from augmented OpenAPI specs - Restore basicAuth in Registry configuration.py and access_api.py - Fix mock patch path in test_ci.py for Python 3.9 compatibility (use @patch('requests.get') instead of module path)
The ensure_registry function was resolving the token before determining the provider, always preferring GH_REGISTRY_TOKEN regardless of whether gitlab was specified. This caused GitLab deployments to fail when both tokens were set. Now resolves provider first, then selects the appropriate token: - gitlab: GL_REGISTRY_TOKEN first, fallback to GH_REGISTRY_TOKEN - github: GH_REGISTRY_TOKEN first, fallback to GL_REGISTRY_TOKEN
Replace string 'in' check with urlparse().hostname for cleaner URL validation.
CLI coverage: 62% -> 81% - Add tests for list serialization, heredoc format, _to_dict methods - Add tests for _parse_profile_arg function - Add tests for SafeModule error handling and log level inclusion Profiles coverage: 68% -> 74% - Add tests for unicode error handling in properties files - Add tests for get_default_profile_name edge cases CI coverage: 50% -> 58% - ensure_registry: 17% -> 95% (validation + token priority tests) - deploy_flow: 17% -> 61% (validation tests) - configure_params: 27% -> 49% (JSON validation tests) Overall coverage: 66% -> 69%
CLI documentation (cli.rst): - Installation options (pip, uv, uvx) - Configuration (env vars, profiles, priority) - Discovering commands with --help - All available modules with API reference links - Output formatting (JSON, GitHub Actions, GitLab) - Note that low-level nifi/registry APIs not exposed CI documentation (ci.rst): - All 14 CI operations with full parameter tables - configure_inherited_params for inheritance-aware updates - resolve_git_ref utility function - Environment variable reference - Complete workflow examples (shell, GitHub, GitLab, Python) Other changes: - Fix test_safe_module_error_handling for CI environments - Fix security.rst title (was incorrectly 'NiPyAPI 2') - Add layout module to core_modules docs - Update dependencies list
…t isolation Add integration tests against real NiFi for better coverage: test_ci.py: - configure_inherited_params: 10 tests (validation + integration) - upload_asset: 8 tests (file path, PG resolution, param linking) test_parameters.py: - get_parameter_context_hierarchy: 3 tests - get_parameter_ownership_map: 2 tests - update_parameter_in_context: 4 tests test_layout.py: - move_port: 3 tests (input, output, no-refresh) - move_component with port: 1 test - snap_position, get_pg_grid_position, left_of, check_overlap: 5 tests test_versioning.py: - Fix test isolation: wait for async revert to complete in test_revert_flow_ver_wait_false before test ends, preventing revision conflicts in subsequent tests using same fixture
New CI functions: - commit_flow: save flow to version control (initial or subsequent) - get_flow_versions: list version history for a flow - detach_flow: remove version control (for forking workflows) - get_flow_diff: get local modifications to a versioned flow Renames for consistency: - change_version -> change_flow_version - get_versions -> list_flows New versioning helper: - get_local_modifications: helper for fetching local changes Improvements: - list_flows: use nipyapi.utils.getenv_bool for env var parsing - list_flows: use nipyapi.canvas.get_flow helper instead of direct API - save_git_flow_ver: accept string process group ID Tests: - Added comprehensive tests for all new CI functions - Fixed flaky versioning tests with proper state handling - Improved test isolation in shared fixtures
- Add export_flow_definition CI function for exporting process groups to files - Add import_flow_definition CI function for importing flow definitions from files - Support JSON and YAML formats for export - Add include_referenced_services option to include external controller services - Use smart positioning (suggest_pg_position) when import location not provided - Add 10 tests covering validation, roundtrip export/import, and file operations
- Add nipyapi.extensions module for NAR upload, delete, download - Add Python processor initialization status and wait functions - Add processor bundle version management for multi-version NARs - Implement defensive delete_nar with initialization guard and cleanup wait - Implement two-phase upload_nar wait (install + processor discovery) - Fix canvas.create_processor to include bundle for multi-version support - Add programmatic NAR generation for testing (create_test_nar) - Add comprehensive tests for extension operations - Add extensions operations guide (docs/extensions.rst) covering NAR lifecycle, Python processor initialization, version management, and troubleshooting
Values with spaces, pipes, brackets, and other shell special characters are now properly quoted in dotenv output format. This prevents export failures when values contain characters like | [ ] that break shell parsing. Embedded double quotes are escaped with backslash.
…gs and optional parameters - Add include_bindings parameter to include bound_process_groups for each context - Add include_parameters parameter to optionally exclude parameter details - Add description field to parameter output for better context - Add tests for new functionality including transitive binding behavior - Useful for cleanup workflows and parameter inspection during deployment
- Add --version/-V flags to display version and exit - Add -v/-vv/-vvv flags to control log verbosity - Refactor _parse_profile_arg to _parse_cli_flags for combined parsing - Add _apply_verbosity to set NIFI_LOG_LEVEL from verbosity count - Add comprehensive unit tests for flag parsing - Add integration tests for version output
- Accept either a string ID or ProcessGroupEntity for parent_pg parameter - Maintains backward compatibility with existing code using entities - Raises TypeError with clear message for invalid types - Supports 'root' as parent_pg for creating PGs on root canvas - Add tests for string ID, UUID string, and invalid type cases
… handling Registry client improvements: - versioning.get_registry_client: add greedy parameter (default True for compat) - ci.deploy_flow: accept registry client name or ID, add greedy param - ci.list_registry_flows: new function to list flows in registry bucket Canvas improvements: - create_process_group: accept string ID or ProcessGroupEntity for parent_pg - delete_remote_process_group: add force param to stop transmission before delete - set_remote_process_group_transmission: wait for state change before returning Test fixes: - conftest: use force=True when cleaning up RPGs - Add tests for greedy parameter and list_registry_flows validation
Previously, lists nested within dict results were converted via str() which produces Python repr format with single quotes. This is not valid JSON and breaks downstream parsing. Now lists and dicts at leaf positions are JSON-serialized with json.dumps() producing valid JSON with double quotes. Fixes output for: list_flows, list_registry_flows, get_flow_versions, get_flow_diff, and any future CI functions returning list values. Added tests for nested list serialization in both GitHub and dotenv formats.
… name - Add is_uuid() function to nipyapi/utils.py for UUID format detection - Update deploy_flow and list_registry_flows to auto-detect if registry_client is UUID (ID) or name - Add comprehensive tests for is_uuid() in test_utils.py - Update docstrings to document auto-detection behavior
- Add get_connection() to fetch connection by ID or entity - Add update_connection() with name/bends parameters (bends=[] clears) - Add clear_flow_bends() to clear bends before flow reorganization - Update get_flow_components() to return FlowSubgraph(components, connections) - Add include_retry parameter to move_component/move_processor - Refactor transpose_flow to handle all bends in single pass - Add comprehensive tests for new functionality
- Add nipyapi.canvas.verify_controller() for controller service verification - Add nipyapi.canvas.verify_processor() for processor verification - Add nipyapi.ci.verify_config() for batch verification of process groups - Add refresh parameter to delete_controller and update_controller - Improve delete_controller to accept ID or object - Add PYTEST_ARGS support to Makefile test target - Add fix_multi_version_nars module-scoped fixture for test optimization - Remove useless @pytest.mark.usefixtures() decorator - Extension tests now ~27% faster with shared NAR fixtures
- Add list_orphaned_contexts() to parameters module for detecting parameter contexts not bound to any process groups - Enhance get_bulletin_board() with pg_id, source_name, message, and limit filters for scoped bulletin retrieval - Add delete_orphaned_contexts option to ci.cleanup() for safe removal of unused parameter contexts after PG deletion - Add comprehensive tests for all new functionality Note: NiFi does not provide an API to clear bulletins in 2.6.0. Clear bulletins API was introduced in 2.7.0 - upgrade planned separately.
Chaffelson
force-pushed
the
feature/cli
branch
from
eb4af78 to
34d0c5f
Compare
…ring New nipyapi.bulletins module: - get_bulletins(): controller-level bulletins - get_bulletin_board(): filtered bulletins returning BulletinDTO directly - clear_*_bulletins(): component-specific clearing (10 functions) - clear_all_bulletins(): high-level helper for CI Additional changes: - nipyapi.profiles: add list_profiles, show, current commands - nipyapi.utils: add format_timestamp utility - nipyapi.ci.get_status: fix to capture controller service bulletins - nipyapi.canvas: bulletin functions now alias to bulletins module - Makefile: intelligent uv/pip detection - docs: updated contributing guide, module documentation
- Add get_processor_state and clear_processor_state for processor state inspection - Add get_controller_state and clear_controller_state for controller service state - All functions accept entity objects or ID strings - Fix CLI --help flag handling in SafeModule wrapper - Remove unused LogCapture.all_records attribute - Add fix_state_flow test fixture with MapCacheServer and ListFile - Add comprehensive tests for state operations
- Add export_parameters CI command for exporting parameter values - Add --parameters_file option to configure_inherited_params for file-based import - Add parameter_context_handling option to deploy_git_registry_flow for VCS - Add deploy_flow CI command parameter_context_handling support - Fix test fixture isolation to prevent shared fixtures being deleted - Fix parameter context cleanup version check logic - Add ACTIVE_PROFILES_PATH for consistent profile file handling - Add parameter inheritance tests for VCS deployments
Add 43 new tests for CLI handling of complex JSON objects: - 19 JSON input parsing tests (quotes, newlines, backslashes, nested structures, Unicode, null/empty values, mixed types) - 4 roundtrip tests (serialize -> parse verification) - 8 CLI subprocess tests (verify JSON passes through CLI correctly) - 12 CI function tests for configure_params complex inputs These tests verify AI agents can use the nipyapi CLI directly for common operations without needing to write Python code for complex parameter structures.
In GitHub Actions CI, GITHUB_ACTIONS=true causes the CLI to auto-detect 'github' output format (key=value), but the tests expect JSON output for parsing. Force NIFI_OUTPUT_FORMAT=json in the subprocess environment to ensure consistent JSON output. Fixes JSONDecodeError in configure_params CLI tests.
- Create AGENTS.md for AI agent guidance (consuming/contributing sections) - Update contributing.rst with docstring standards and discovery patterns - Fix all 51 Sphinx docstring warnings (nested lists, Example formatting) - Standardize Example:: format across 91 instances with required blank lines - Fix RST title underlines in ci.rst, security.rst - Add cross-reference notation guidance for Sphinx - Add module intent comments to __init__.py
New features: - get_processor_docs(processor): retrieve full processor documentation including property_descriptors, tags, use cases. Accepts ProcessorEntity, DocumentedTypeDTO, or processor type name string. - schedule_processor() enhancements: - Now accepts processor ID string or ProcessorEntity object - Supports all processor states: RUNNING, STOPPED, DISABLED, RUN_ONCE - Uses direct processor API for reliable state transitions - Backwards compatible: bool values still work (True=RUNNING, False=STOPPED) Client fixes: - Add nifi_processor_state_enum.py augmentation to fix upstream NiFi OpenAPI bug where ProcessorDTO.state enum was missing transitional states Testing improvements: - Add pytest filterwarnings config to suppress InsecureRequestWarning in tests - Add comprehensive tests for new functionality Docs: - Minor AGENTS.md improvement: venv activation reminder
- nipyapi.canvas: Add list_flowfiles, get_flowfile_details, get_flowfile_content, peek_flowfiles - get_flowfile_content supports decode options (auto/text/bytes) and output_file saving - nipyapi.utils.fs_write: Add binary=False parameter for raw byte writing (backwards compatible) - Tests: Add test_fs_write_binary, expand test_get_flowfile_content coverage
There was a problem hiding this comment.
Pull request overview
This PR adds a comprehensive CLI interface to nipyapi, enabling shell-based automation and CI/CD integration while preserving backward compatibility. The implementation includes 22 high-level CI operations, enhanced canvas utilities, NAR management, and improved bulletin/state handling.
Key Changes:
- Command-line interface with automatic help generation via Google Fire
- CI operations module with 22 automation-focused functions for flow deployment and management
- Enhanced canvas module with FlowFile inspection, state management, and verification capabilities
- New bulletins and layout modules for improved flow organization
- Extensions module for NAR lifecycle management with multi-version support
Reviewed changes
Copilot reviewed 71 out of 76 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_layout.py |
Comprehensive layout module tests covering grid positioning, flow organization, and component alignment |
tests/test_extensions.py |
NAR management tests including upload/delete roundtrips, multi-version workflows, and processor initialization |
tests/test_cli.py |
CLI interface tests covering serialization, argument parsing, and integration with subprocess |
tests/test_ci.py |
CI operations tests validating deployment, version control, and parameter management workflows |
tests/test_canvas.py |
Enhanced canvas tests for process group creation, FlowFile inspection, state management, and config verification |
tests/test_bulletins.py |
Bulletin retrieval and clearing tests with timestamp formatting utilities |
tests/conftest.py |
Enhanced fixtures including NAR builders, state flow setup, and improved cleanup handling |
resources/client_gen/swagger_templates/rest.mustache |
Binary content type support for asset uploads (octet-stream) |
resources/client_gen/augmentations/nifi_processor_state_enum.py |
Processor state enum fix for transitional states |
pyproject.toml |
CLI dependencies and pytest configuration |
pylintrc |
Disabled duplicate-code warnings |
nipyapi/versioning.py |
Git flow operations including save_git_flow_ver and get_local_modifications |
nipyapi/utils.py |
New utilities: is_uuid, format_timestamp, binary file write support |
nipyapi/registry/rest.py |
Binary content support for registry client |
nipyapi/profiles.py |
Auto-resolution of profiles, enhanced environment variable handling |
nipyapi/nifi/rest.py |
Binary content support for NiFi client |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Extract NiFi Registry functions to nipyapi.nifi_registry module with backwards-compatible shims in versioning.py - Replace global cache with @lru_cache in conftest.py - Clarify switch() usage in profiles.py error message - Document UUID format-only validation in utils.py - Fix URL validation in test_ci.py (use urlparse for hostname check)
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 70 out of 77 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Owner
Author
|
@ottobackwards I think this now has nearly all the operational functionality I'm using for flow authorship and operations automation in NiFi 2.7.2. |
ottobackwards
approved these changes
Dec 30, 2025
Collaborator
ottobackwards
left a comment
ottobackwards
left a comment
There was a problem hiding this comment.
This looks good. I cannot think of a reason not to land it and get it out there.
There are a lot of "magic" strings here, and some places where there are if else based operations that could be made into utility functions, but that can all be follow on as necessary.
Owner
Author
Great, thanks - and good point about the redundancy. I'll create an issue to capture it for later work. |
…rings Bug fixes: - canvas.py: Add _resolve_flowfile_cluster_node() helper for clustered NiFi - canvas.py: get_flowfile_details/get_flowfile_content auto-resolve cluster_node_id - canvas.py: peek_flowfiles preserves cluster_node_id from FlowFileSummaryDTO - canvas.py: create_processor accepts string ID or ProcessGroupEntity - verify_config.py: Remove fail_on_error param (always return result dict) CLI improvements: - cli.py: Use functools.wraps for proper signature preservation in fire - canvas.py, layout.py, utils.py: Inline docstring format for CLI compatibility Documentation: - contributing.rst: Add CLI-specific docstring guidance Tests: - test_canvas.py: Add tests for cluster_node_id handling - test_ci.py: Update tests for verify_config changes
…te_process_group Previously, force deletion would delete each controller service individually, which was slow and caused version control to show local modifications. Now uses schedule_all_controllers() to bulk disable, which is faster and non-destructive.
MapCacheServer binds to port 4557 and won't release it until disabled. Without explicit cleanup, subsequent tests fail with 'Address already in use' when trying to enable their own MapCacheServer. Added finalizer to disable both server and client controllers before fix_pg's finalizer runs to delete the process group.
…et state Previously the function returned immediately after the API call, leaving controllers in transitional states (ENABLING/DISABLING). This caused race conditions in slower environments (e.g. Python 3.9) where callers would check controller state before transitions completed. Now uses wait_to_complete to poll until all controllers in the PG reach the target state, consistent with schedule_controller behavior.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
feat: Add CLI with CI operations for NiFi automation
Description
This PR adds a command-line interface to nipyapi, enabling shell scripting, CI/CD pipelines, and interactive exploration of NiFi deployments.
Motivation
nipyapi has always provided excellent Python API coverage for Apache NiFi, but many automation scenarios require shell-based tooling:
This CLI bridges that gap while maintaining nipyapi's design philosophy of sensible defaults and clear interfaces.
Features
Command-Line Interface (
nipyapi[cli])canvas,versioning,nifi_registry,parameters,security,system,profiles,bulletins,extensions~/.nipyapi/profiles.ymlor environment variablesCI Operations Module (
nipyapi.ci)ensure_registry- Create/update Git Flow Registry Client (GitHub/GitLab)deploy_flow- Deploy versioned flow from registry to canvasstart_flow/stop_flow- Manage flow execution with controller service handlingget_status- Comprehensive status including bulletins and invalid processorslist_flows- List process groups with version control stateget_flow_versions- List version history for a versioned flowchange_flow_version- Switch to specific version or latestcommit_flow- Save flow to version control (initial or subsequent)detach_flow- Remove version control (for forking workflows)get_flow_diff- Get local modifications to a versioned flowrevert_flow- Discard uncommitted local changesconfigure_params/configure_inherited_params- Set parameter values with inheritance supportupload_asset- Upload files to parameter contextscleanup- Stop and delete process groupspurge_flowfiles- Clear queued data from connectionsresolve_git_ref- Resolve branch/tag to commit SHAexport_flow_definition- Export process group as JSON/YAML (no registry required)import_flow_definition- Import flow definition file as new process grouplist_nars/upload_nar/delete_nar- Manage custom NAR extensionsCanvas Module Enhancements (
nipyapi.canvas)list_flowfiles,get_flowfile_details,get_flowfile_content,peek_flowfilesget_processor_state,clear_processor_state,get_controller_state,clear_controller_stateschedule_processor: now supportsRUN_ONCEandDISABLEDstates in addition toRUNNING/STOPPEDget_processor_docs,get_controller_service_docsfor programmatic processor discoveryBulletins Module (
nipyapi.bulletins)get_bulletin_boardwith filtering by source name and message patternclear_all_bulletinsfor NiFi 2.7.0+ (clear bulletins between test runs)Layout Module (
nipyapi.layout)align_pg_grid,align_processors_horizontal,stack_process_groupsbelow(),fork(),right_of(),left_of()for flow buildingExtensions Module (
nipyapi.extensions)list_nars/get_nar/get_nar_details- Query installed extensionsupload_nar- Upload with automatic installation wait and processor discoverydelete_nar- Safe deletion with cleanup verification and init-state guardsget_processor_init_status/wait_for_processor_init- Python processor lifecycleget_processor_bundle_versions/change_processor_bundle_version- Multi-version supportNiFi Registry Module (
nipyapi.nifi_registry)versioning.pyfor clearer module organizationversioning.pyretains Git registry functions and provides backwards-compatible aliaseslist_registry_buckets,create_registry_bucket,save_flow_ver,deploy_flow_version, etc.Utilities Enhancements (
nipyapi.utils)fs_writenow supportsbinary=Truefor raw byte content (e.g., FlowFile downloads)is_uuidhelper for detecting UUID strings (format validation only, not RFC 4122)format_timestampfor NiFi-compatible timestamp formattingEnhanced Profile System
--profileoption for explicit profile selectionExample Usage
CI Platform Integration
The CLI auto-detects CI environments and formats output appropriately:
key=valueformat with heredoc syntax for multiline valuesKEY=VALUEdotenv format with proper quoting for special characters|,[,], spaces) are automatically quotedDocumentation
docs/cli.rst- Complete CLI reference with all modules and optionsdocs/ci.rst- CI operations guide with parameter tables and examplesdocs/profiles.rst- Enhanced profile configuration documentationdocs/extensions.rst- NAR management and Python processor lifecycle guideTesting
test_cli.py,test_ci.py,test_layout.py,test_profiles.py,test_extensions.pycreate_test_nar()fixture)Installation
Breaking Changes
None. The CLI is an optional install (
nipyapi[cli]) and all existing Python APIs remain unchanged.