Merge workflow_status into workflow (#300)#302
Conversation
Collapse the 1:1 workflow_status satellite table into workflow itself: - Add run_id, is_canceled to workflow (is_archived already existed). - Backfill from workflow_status using the existing status_id join. - Strip the FOREIGN KEY (status_id) constraint via PRAGMA writable_schema so DROP COLUMN status_id and DROP TABLE workflow_status work without rename-recreating workflow (which would cascade-delete every child table per the CLAUDE.md landmine). Also drop has_detected_need_to_run_completion_script: no code path ever set it to true, so the read paths (TUI badge, IsCompleteResponse) were dead. Completion-script intent now lives in on_workflow_complete workflow actions. Eliminates the orphan class that PR #299's export workaround was papering over: with the data in workflow itself, there is nowhere for a status row to be stranded. Also removes the noisy JOIN from list queries and the status_id field from the API contract. Regenerate Rust OpenAPI spec, Python client, and Julia client. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Removed fields from the API contract: WorkflowModel.status_id, WorkflowStatusModel.has_detected_need_to_run_completion_script, and IsCompleteResponse.needs_to_run_completion_script. Bump the version constant and regenerate Rust, Python, and Julia clients so their doc-comment version strings track. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR removes the workflow_status satellite table by folding its remaining state into workflow, then updates the server, OpenAPI contract, and generated clients to match the simplified schema. It fits into the codebase as a schema/API cleanup that also removes an orphan-prone 1:1 indirection previously worked around in export logic.
Changes:
- Adds a migration to backfill
run_id/is_canceledontoworkflow, removestatus_id, and dropworkflow_status. - Rewrites server queries and API models to read workflow status directly from
workflow, including removing dead completion-script fields. - Regenerates OpenAPI artifacts and Rust/Python/Julia clients for API version
0.15.0, plus updates export/tests/docs around the new schema.
Reviewed changes
Copilot reviewed 147 out of 147 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
torc-server/migrations/20260423000000_merge_workflow_status_into_workflow.up.sql |
Forward migration merging status state into workflow. |
torc-server/migrations/20260423000000_merge_workflow_status_into_workflow.down.sql |
Rollback migration recreating workflow_status and status_id. |
tests/test_workflow_metadata_project.rs |
Removes status_id from workflow creation test payloads. |
tests/test_export.rs |
Updates export seeding/assertions and adds regression coverage for removed table/column. |
src/tui/ui.rs |
Removes the completion-script badge suffix from TUI summary rendering. |
src/tui/app.rs |
Drops needs_completion_script from TUI workflow summary state. |
src/server/http_server/unblock_processing.rs |
Swaps status lookup join from workflow_status to workflow. |
src/server/http_server/runtime_support.rs |
Reads workflow run_id directly from workflow. |
src/server/http_server/lifecycle_support.rs |
Repoints lifecycle dependency query to workflow.run_id. |
src/server/http_server/jobs_transport.rs |
Validates run_id from workflow instead of workflow_status. |
src/server/export.rs |
Removes explicit workflow_status orphan pruning now that the table is gone. |
src/server/authorization.rs |
Deletes the no-longer-needed workflow-status access wrapper. |
src/server/api/workflows.rs |
Main API rewrite to remove status_id/dead fields and query status from workflow. |
src/server/api/ro_crate.rs |
Pulls RO-Crate run_id from workflow. |
src/server/api/jobs.rs |
Updates workflow status/run-id lookups in job APIs. |
src/openapi_spec.rs |
Adjusts parity checks for removed fields. |
src/models.rs |
Removes status_id and completion-script fields from Rust models. |
src/client/apis/workflows_api.rs |
Generated Rust client version header bump. |
src/client/apis/workflow_actions_api.rs |
Generated Rust client version header bump. |
src/client/apis/user_data_api.rs |
Generated Rust client version header bump. |
src/client/apis/tasks_api.rs |
Generated Rust client version header bump. |
src/client/apis/system_api.rs |
Generated Rust client version header bump. |
src/client/apis/slurm_stats_api.rs |
Generated Rust client version header bump. |
src/client/apis/slurm_schedulers_api.rs |
Generated Rust client version header bump. |
src/client/apis/scheduled_compute_nodes_api.rs |
Generated Rust client version header bump. |
src/client/apis/ro_crate_entities_api.rs |
Generated Rust client version header bump. |
src/client/apis/results_api.rs |
Generated Rust client version header bump. |
src/client/apis/resource_requirements_api.rs |
Generated Rust client version header bump. |
src/client/apis/remote_workers_api.rs |
Generated Rust client version header bump. |
src/client/apis/local_schedulers_api.rs |
Generated Rust client version header bump. |
src/client/apis/jobs_api.rs |
Generated Rust client version header bump. |
src/client/apis/files_api.rs |
Generated Rust client version header bump. |
src/client/apis/failure_handlers_api.rs |
Generated Rust client version header bump. |
src/client/apis/events_api.rs |
Generated Rust client version header bump. |
src/client/apis/compute_nodes_api.rs |
Generated Rust client version header bump. |
src/client/apis/access_control_api.rs |
Generated Rust client version header bump. |
src/api_version.rs |
Bumps shared HTTP API version to 0.15.0. |
python_client/src/torc/openapi_client/rest.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/workflow_status_model.py |
Removes dead status field from Python model. |
python_client/src/torc/openapi_client/models/workflow_model.py |
Removes status_id from Python workflow model. |
python_client/src/torc/openapi_client/models/workflow_action_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/workflow_access_group_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/version_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/user_group_membership_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/user_data_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/task_status.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/task_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/slurm_stats_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/slurm_scheduler_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/scheduled_compute_nodes_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/ro_crate_entity_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/result_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/resource_requirements_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/reset_job_status_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/remote_worker_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/reload_auth_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/process_changed_job_inputs_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/ping_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/message_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/local_scheduler_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_workflows_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_user_group_memberships_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_user_data_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_slurm_stats_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_slurm_schedulers_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_scheduled_compute_nodes_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_ro_crate_entities_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_results_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_resource_requirements_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_required_existing_files_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_missing_user_data_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_local_schedulers_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_jobs_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_job_user_data_relationships_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_job_ids_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_job_file_relationships_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_job_dependencies_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_files_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_failure_handlers_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_events_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_compute_nodes_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/list_access_groups_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/jobs_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/job_user_data_relationship_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/job_status.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/job_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/job_file_relationship_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/job_dependency_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/job_completion_error.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/job_completion_entry.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/is_uninitialized_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/is_complete_response.py |
Removes dead completion-script field from Python model. |
python_client/src/torc/openapi_client/models/get_ready_job_requirements_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/file_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/failure_handler_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/event_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/error_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/delete_ro_crate_entities_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/delete_count_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/create_jobs_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/compute_nodes_resources.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/compute_node_schedule.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/compute_node_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/claim_next_jobs_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/claim_jobs_based_on_resources.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/claim_action_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/claim_action_request.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/batch_complete_jobs_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/batch_complete_jobs_request.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/active_task_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/access_group_model.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/access_check_response.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/models/__init__.py |
Refreshes Python model exports/version metadata. |
python_client/src/torc/openapi_client/exceptions.py |
Generated Python client version metadata bump. |
python_client/src/torc/openapi_client/configuration.py |
Bumps Python SDK/API version strings. |
python_client/src/torc/openapi_client/api_client.py |
Bumps Python SDK user-agent/version strings. |
python_client/src/torc/openapi_client/api/workflows_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/workflow_actions_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/user_data_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/tasks_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/system_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/slurm_stats_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/slurm_schedulers_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/scheduled_compute_nodes_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/ro_crate_entities_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/results_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/resource_requirements_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/remote_workers_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/local_schedulers_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/jobs_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/files_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/failure_handlers_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/events_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/compute_nodes_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/api/access_control_api.py |
Generated Python API client version metadata bump. |
python_client/src/torc/openapi_client/__init__.py |
Bumps Python package version to 0.15.0. |
julia_client/julia_client/docs/WorkflowStatusModel.md |
Removes dead field from Julia docs. |
julia_client/julia_client/docs/WorkflowModel.md |
Removes status_id from Julia docs. |
julia_client/julia_client/docs/IsCompleteResponse.md |
Removes completion-script field from Julia docs. |
julia_client/julia_client/README.md |
Bumps Julia client API version. |
julia_client/Torc/src/api/models/model_WorkflowStatusModel.jl |
Removes dead field from Julia model. |
julia_client/Torc/src/api/models/model_WorkflowModel.jl |
Removes status_id from Julia model. |
julia_client/Torc/src/api/models/model_IsCompleteResponse.jl |
Removes completion-script field from Julia model. |
julia_client/Torc/src/api/APIClient.jl |
Bumps Julia client API version constant. |
examples/datasight/schema_description.md |
Updates schema docs for workflow-embedded status fields. |
api/openapi.yaml |
Publishes the breaking API contract changes at version 0.15.0. |
api/openapi.codegen.yaml |
Keeps codegen source in sync with updated OpenAPI contract. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The earlier commit on this branch merged workflow_status into workflow at
the storage layer but left the satellite shape on the API surface — the
server constructed a synthetic WorkflowStatusModel from one row of
workflow, exposed it via /workflows/{id}/status (GET/PUT), and clients
still treated it as a separate object. That kept the indirection issue
#300 was meant to remove, just hidden in a different place.
This change finishes the collapse:
- Add run_id, is_canceled, is_archived to WorkflowModel; populate them
in get_workflow / list_workflows / create_workflow; accept them in
update_workflow via COALESCE.
- Delete WorkflowStatusModel from models.rs and the Rust-owned OpenAPI
spec.
- Delete GET /workflows/{id}/status, PUT /workflows/{id}/status; their
data is reachable via /workflows/{id} now.
- Keep POST /workflows/{id}/reset_status as a meaningful domain
operation, but its response is now a WorkflowModel-shaped object.
cancel_workflow likewise returns post-cancel WorkflowModel.
- Migrate all internal callers (cancel/is_complete/reset, claim_jobs,
workflow_manager, run_jobs_cmd, torc-slurm-job-runner, archive
command, orphan detection) to read from WorkflowModel.
- Update integration tests across test_workflows, test_workflow_manager,
test_results, test_jobs, test_orphaned_jobs, test_slurm_commands,
test_workflow_metadata_project to use the new shape.
- Regenerate Rust, Python, and Julia clients.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI fix: openapi_spec::tests::generated_yaml_contains_scaffold_paths
asserted the now-deleted `/workflows/{id}/status` path is in the
emitted YAML; removed the assertion.
Down migration fixes (Copilot review on PR #302):
- Restore the original `status_id INTEGER NOT NULL` constraint when
rolling back. ALTER TABLE ADD COLUMN with NOT NULL needs a DEFAULT,
so we add `NOT NULL DEFAULT 0`, backfill, then strip the DEFAULT
clause via writable_schema so the rolled-back schema text matches
the original.
- Drop `run_id` and `is_canceled` from `workflow` after backfilling
them into `workflow_status`. Without this, a downgrade leaves
duplicate status state on both tables that can silently drift.
(`workflow.is_archived` predates the up migration and stays.)
Verified with an up→down→inspect roundtrip: the post-rollback
`workflow` schema has `status_id INTEGER NOT NULL, FOREIGN KEY
(status_id) REFERENCES workflow_status(id)` and no leftover
`run_id`/`is_canceled` columns; PRAGMA foreign_key_check passes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 169 out of 169 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…300) Addresses Copilot review concerns that PUT /workflows/{id} could mutate run_id, is_canceled, and is_archived directly, bypassing the dedicated side-effect logic in cancel_workflow/reset_workflow_status. Also fixes the related issue that POST /workflows accepted those fields on the create payload but silently discarded them. API changes: - Mark run_id, is_canceled, is_archived as readOnly in WorkflowModel (utoipa schema attribute). They are returned by GET /workflows and GET /workflows/{id}, but values supplied to create/update endpoints are now ignored. - Drop run_id, is_canceled, is_archived from the COALESCE in update_workflow's UPDATE SQL. - Add POST /workflows/{id}/archive { is_archived: bool } as the only way to set is_archived. Returns the post-archive WorkflowModel. - Fold run_id bump into reset_workflow_status: every reset now also bumps run_id by 1 ("reset = start a new run"). This collapses the workflow_manager.bump_run_id helper away — initialize/reinitialize flows now just call reset_workflow_status, no separate bump. Internal callers migrated: - archive CLI command → archive_workflow endpoint - workflow_manager.{initialize, initialize_async, reinitialize, reinitialize_async} → drop bump_run_id calls - workflow_manager.bump_run_id → removed (no callers left) - tests/test_workflows.rs, tests/test_workflow_manager.rs → archive via archive_workflow; renamed test_bump_run_id to test_reset_workflow_status_bumps_run_id Down migration: documented the AUTOINCREMENT pairing assumption raised in the Copilot review. The pre-merge schema technically allowed status_id != workflow.id (independent AUTOINCREMENT sequences), but in practice every code path in this repo paired the two IDs together. The note flags the edge case for any DB modified by an external tool. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 171 out of 171 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -147,6 +165,11 @@ def to_dict(self) -> Dict[str, Any]: | |||
| if self.resource_monitor_config is None and "resource_monitor_config" in self.model_fields_set: | |||
| _dict['resource_monitor_config'] = None | |||
|
|
|||
| # set to None if run_id (nullable) is None | |||
| # and model_fields_set contains the field | |||
| if self.run_id is None and "run_id" in self.model_fields_set: | |||
| _dict['run_id'] = None | |||
Summary
Closes #300. Collapses the 1:1
workflow_statussatellite table intoworkflow, eliminating the orphan class that PR #299's export workaround was papering over and removing thestatus_idindirection from the API surface.20260423000000: addsrun_idandis_canceledtoworkflow(is_archivedalready existed), backfills fromworkflow_status, strips theFOREIGN KEY (status_id)clause viaPRAGMA writable_schema, then drops the column and the satellite table. The writable_schema trick avoids the rename-recreate landmine onworkflow(parent of 15+ cascading children — see CLAUDE.md).has_detected_need_to_run_completion_scriptin the same migration. Audit confirmed no code path ever wrotetrue; the dead read paths (TUI badge,IsCompleteResponse.needs_to_run_completion_script) are removed. Completion-script intent is now handled exclusively byon_workflow_completeworkflow actions.SELECT … FROM workflow_status WHERE id = ?rewrites to read directly fromworkflow, the unusedcheck_workflow_status_accesswrapper is gone, andexport.rs's orphan-prune workaround is removed.test_export.rspins the structural property: table absent, column absent, status fields onworkflow.HTTP_API_VERSION0.14.0 → 0.15.0 (breaking: removed three fields from the API contract). Rust, Python, and Julia clients regenerated.Test plan
cargo build --workspace --all-featurescargo clippy --all --all-targets --all-features -- -D warningscargo fmt --all -- --checkanddprint checkcargo nextest run --lib— 293/293 passcargo nextest run --features server-bin --test test_export— 18/18 pass, including the newworkflow_status_is_not_reachable_as_orphan_after_workflow_deleteregressioncargo nextest run --all-features --test test_workflow_metadata_project— 8/8 passbash api/sync_openapi.sh check— parity ok🤖 Generated with Claude Code