Skip to content

Merge workflow_status into workflow (#300)#302

Merged
daniel-thom merged 5 commits intomainfrom
issue-300-remove-workflow-status
May 4, 2026
Merged

Merge workflow_status into workflow (#300)#302
daniel-thom merged 5 commits intomainfrom
issue-300-remove-workflow-status

Conversation

@daniel-thom
Copy link
Copy Markdown
Collaborator

Summary

Closes #300. Collapses the 1:1 workflow_status satellite table into workflow, eliminating the orphan class that PR #299's export workaround was papering over and removing the status_id indirection from the API surface.

  • Migration 20260423000000: adds run_id and is_canceled to workflow (is_archived already existed), backfills from workflow_status, strips the FOREIGN KEY (status_id) clause via PRAGMA writable_schema, then drops the column and the satellite table. The writable_schema trick avoids the rename-recreate landmine on workflow (parent of 15+ cascading children — see CLAUDE.md).
  • Drops has_detected_need_to_run_completion_script in the same migration. Audit confirmed no code path ever wrote true; the dead read paths (TUI badge, IsCompleteResponse.needs_to_run_completion_script) are removed. Completion-script intent is now handled exclusively by on_workflow_complete workflow actions.
  • Server cleanups: list_workflows drops the JOIN, delete_workflow drops the trailing workflow_status DELETE, every SELECT … FROM workflow_status WHERE id = ? rewrites to read directly from workflow, the unused check_workflow_status_access wrapper is gone, and export.rs's orphan-prune workaround is removed.
  • Regression test in test_export.rs pins the structural property: table absent, column absent, status fields on workflow.
  • API version bump HTTP_API_VERSION 0.14.0 → 0.15.0 (breaking: removed three fields from the API contract). Rust, Python, and Julia clients regenerated.

Test plan

  • cargo build --workspace --all-features
  • cargo clippy --all --all-targets --all-features -- -D warnings
  • cargo fmt --all -- --check and dprint check
  • cargo nextest run --lib — 293/293 pass
  • cargo nextest run --features server-bin --test test_export — 18/18 pass, including the new workflow_status_is_not_reachable_as_orphan_after_workflow_delete regression
  • cargo nextest run --all-features --test test_workflow_metadata_project — 8/8 pass
  • bash api/sync_openapi.sh check — parity ok
  • Manually verify migration on a database with realistic data before merging
  • Verify Python/Julia client downstream consumers still work after the field removals

🤖 Generated with Claude Code

daniel-thom and others added 2 commits May 3, 2026 15:33
Collapse the 1:1 workflow_status satellite table into workflow itself:
- Add run_id, is_canceled to workflow (is_archived already existed).
- Backfill from workflow_status using the existing status_id join.
- Strip the FOREIGN KEY (status_id) constraint via PRAGMA writable_schema
  so DROP COLUMN status_id and DROP TABLE workflow_status work without
  rename-recreating workflow (which would cascade-delete every child
  table per the CLAUDE.md landmine).

Also drop has_detected_need_to_run_completion_script: no code path ever
set it to true, so the read paths (TUI badge, IsCompleteResponse) were
dead. Completion-script intent now lives in on_workflow_complete
workflow actions.

Eliminates the orphan class that PR #299's export workaround was
papering over: with the data in workflow itself, there is nowhere for a
status row to be stranded. Also removes the noisy JOIN from list
queries and the status_id field from the API contract. Regenerate Rust
OpenAPI spec, Python client, and Julia client.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Removed fields from the API contract: WorkflowModel.status_id,
WorkflowStatusModel.has_detected_need_to_run_completion_script, and
IsCompleteResponse.needs_to_run_completion_script. Bump the version
constant and regenerate Rust, Python, and Julia clients so their
doc-comment version strings track.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR removes the workflow_status satellite table by folding its remaining state into workflow, then updates the server, OpenAPI contract, and generated clients to match the simplified schema. It fits into the codebase as a schema/API cleanup that also removes an orphan-prone 1:1 indirection previously worked around in export logic.

Changes:

  • Adds a migration to backfill run_id/is_canceled onto workflow, remove status_id, and drop workflow_status.
  • Rewrites server queries and API models to read workflow status directly from workflow, including removing dead completion-script fields.
  • Regenerates OpenAPI artifacts and Rust/Python/Julia clients for API version 0.15.0, plus updates export/tests/docs around the new schema.

Reviewed changes

Copilot reviewed 147 out of 147 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
torc-server/migrations/20260423000000_merge_workflow_status_into_workflow.up.sql Forward migration merging status state into workflow.
torc-server/migrations/20260423000000_merge_workflow_status_into_workflow.down.sql Rollback migration recreating workflow_status and status_id.
tests/test_workflow_metadata_project.rs Removes status_id from workflow creation test payloads.
tests/test_export.rs Updates export seeding/assertions and adds regression coverage for removed table/column.
src/tui/ui.rs Removes the completion-script badge suffix from TUI summary rendering.
src/tui/app.rs Drops needs_completion_script from TUI workflow summary state.
src/server/http_server/unblock_processing.rs Swaps status lookup join from workflow_status to workflow.
src/server/http_server/runtime_support.rs Reads workflow run_id directly from workflow.
src/server/http_server/lifecycle_support.rs Repoints lifecycle dependency query to workflow.run_id.
src/server/http_server/jobs_transport.rs Validates run_id from workflow instead of workflow_status.
src/server/export.rs Removes explicit workflow_status orphan pruning now that the table is gone.
src/server/authorization.rs Deletes the no-longer-needed workflow-status access wrapper.
src/server/api/workflows.rs Main API rewrite to remove status_id/dead fields and query status from workflow.
src/server/api/ro_crate.rs Pulls RO-Crate run_id from workflow.
src/server/api/jobs.rs Updates workflow status/run-id lookups in job APIs.
src/openapi_spec.rs Adjusts parity checks for removed fields.
src/models.rs Removes status_id and completion-script fields from Rust models.
src/client/apis/workflows_api.rs Generated Rust client version header bump.
src/client/apis/workflow_actions_api.rs Generated Rust client version header bump.
src/client/apis/user_data_api.rs Generated Rust client version header bump.
src/client/apis/tasks_api.rs Generated Rust client version header bump.
src/client/apis/system_api.rs Generated Rust client version header bump.
src/client/apis/slurm_stats_api.rs Generated Rust client version header bump.
src/client/apis/slurm_schedulers_api.rs Generated Rust client version header bump.
src/client/apis/scheduled_compute_nodes_api.rs Generated Rust client version header bump.
src/client/apis/ro_crate_entities_api.rs Generated Rust client version header bump.
src/client/apis/results_api.rs Generated Rust client version header bump.
src/client/apis/resource_requirements_api.rs Generated Rust client version header bump.
src/client/apis/remote_workers_api.rs Generated Rust client version header bump.
src/client/apis/local_schedulers_api.rs Generated Rust client version header bump.
src/client/apis/jobs_api.rs Generated Rust client version header bump.
src/client/apis/files_api.rs Generated Rust client version header bump.
src/client/apis/failure_handlers_api.rs Generated Rust client version header bump.
src/client/apis/events_api.rs Generated Rust client version header bump.
src/client/apis/compute_nodes_api.rs Generated Rust client version header bump.
src/client/apis/access_control_api.rs Generated Rust client version header bump.
src/api_version.rs Bumps shared HTTP API version to 0.15.0.
python_client/src/torc/openapi_client/rest.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/workflow_status_model.py Removes dead status field from Python model.
python_client/src/torc/openapi_client/models/workflow_model.py Removes status_id from Python workflow model.
python_client/src/torc/openapi_client/models/workflow_action_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/workflow_access_group_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/version_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/user_group_membership_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/user_data_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/task_status.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/task_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/slurm_stats_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/slurm_scheduler_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/scheduled_compute_nodes_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/ro_crate_entity_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/result_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/resource_requirements_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/reset_job_status_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/remote_worker_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/reload_auth_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/process_changed_job_inputs_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/ping_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/message_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/local_scheduler_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_workflows_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_user_group_memberships_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_user_data_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_slurm_stats_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_slurm_schedulers_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_scheduled_compute_nodes_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_ro_crate_entities_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_results_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_resource_requirements_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_required_existing_files_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_missing_user_data_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_local_schedulers_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_jobs_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_job_user_data_relationships_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_job_ids_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_job_file_relationships_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_job_dependencies_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_files_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_failure_handlers_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_events_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_compute_nodes_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/list_access_groups_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/jobs_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/job_user_data_relationship_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/job_status.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/job_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/job_file_relationship_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/job_dependency_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/job_completion_error.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/job_completion_entry.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/is_uninitialized_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/is_complete_response.py Removes dead completion-script field from Python model.
python_client/src/torc/openapi_client/models/get_ready_job_requirements_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/file_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/failure_handler_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/event_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/error_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/delete_ro_crate_entities_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/delete_count_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/create_jobs_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/compute_nodes_resources.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/compute_node_schedule.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/compute_node_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/claim_next_jobs_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/claim_jobs_based_on_resources.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/claim_action_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/claim_action_request.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/batch_complete_jobs_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/batch_complete_jobs_request.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/active_task_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/access_group_model.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/access_check_response.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/models/__init__.py Refreshes Python model exports/version metadata.
python_client/src/torc/openapi_client/exceptions.py Generated Python client version metadata bump.
python_client/src/torc/openapi_client/configuration.py Bumps Python SDK/API version strings.
python_client/src/torc/openapi_client/api_client.py Bumps Python SDK user-agent/version strings.
python_client/src/torc/openapi_client/api/workflows_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/workflow_actions_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/user_data_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/tasks_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/system_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/slurm_stats_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/slurm_schedulers_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/scheduled_compute_nodes_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/ro_crate_entities_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/results_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/resource_requirements_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/remote_workers_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/local_schedulers_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/jobs_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/files_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/failure_handlers_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/events_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/compute_nodes_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/api/access_control_api.py Generated Python API client version metadata bump.
python_client/src/torc/openapi_client/__init__.py Bumps Python package version to 0.15.0.
julia_client/julia_client/docs/WorkflowStatusModel.md Removes dead field from Julia docs.
julia_client/julia_client/docs/WorkflowModel.md Removes status_id from Julia docs.
julia_client/julia_client/docs/IsCompleteResponse.md Removes completion-script field from Julia docs.
julia_client/julia_client/README.md Bumps Julia client API version.
julia_client/Torc/src/api/models/model_WorkflowStatusModel.jl Removes dead field from Julia model.
julia_client/Torc/src/api/models/model_WorkflowModel.jl Removes status_id from Julia model.
julia_client/Torc/src/api/models/model_IsCompleteResponse.jl Removes completion-script field from Julia model.
julia_client/Torc/src/api/APIClient.jl Bumps Julia client API version constant.
examples/datasight/schema_description.md Updates schema docs for workflow-embedded status fields.
api/openapi.yaml Publishes the breaking API contract changes at version 0.15.0.
api/openapi.codegen.yaml Keeps codegen source in sync with updated OpenAPI contract.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

daniel-thom and others added 2 commits May 3, 2026 16:19
The earlier commit on this branch merged workflow_status into workflow at
the storage layer but left the satellite shape on the API surface — the
server constructed a synthetic WorkflowStatusModel from one row of
workflow, exposed it via /workflows/{id}/status (GET/PUT), and clients
still treated it as a separate object. That kept the indirection issue
#300 was meant to remove, just hidden in a different place.

This change finishes the collapse:

- Add run_id, is_canceled, is_archived to WorkflowModel; populate them
  in get_workflow / list_workflows / create_workflow; accept them in
  update_workflow via COALESCE.
- Delete WorkflowStatusModel from models.rs and the Rust-owned OpenAPI
  spec.
- Delete GET /workflows/{id}/status, PUT /workflows/{id}/status; their
  data is reachable via /workflows/{id} now.
- Keep POST /workflows/{id}/reset_status as a meaningful domain
  operation, but its response is now a WorkflowModel-shaped object.
  cancel_workflow likewise returns post-cancel WorkflowModel.
- Migrate all internal callers (cancel/is_complete/reset, claim_jobs,
  workflow_manager, run_jobs_cmd, torc-slurm-job-runner, archive
  command, orphan detection) to read from WorkflowModel.
- Update integration tests across test_workflows, test_workflow_manager,
  test_results, test_jobs, test_orphaned_jobs, test_slurm_commands,
  test_workflow_metadata_project to use the new shape.
- Regenerate Rust, Python, and Julia clients.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI fix: openapi_spec::tests::generated_yaml_contains_scaffold_paths
asserted the now-deleted `/workflows/{id}/status` path is in the
emitted YAML; removed the assertion.

Down migration fixes (Copilot review on PR #302):
- Restore the original `status_id INTEGER NOT NULL` constraint when
  rolling back. ALTER TABLE ADD COLUMN with NOT NULL needs a DEFAULT,
  so we add `NOT NULL DEFAULT 0`, backfill, then strip the DEFAULT
  clause via writable_schema so the rolled-back schema text matches
  the original.
- Drop `run_id` and `is_canceled` from `workflow` after backfilling
  them into `workflow_status`. Without this, a downgrade leaves
  duplicate status state on both tables that can silently drift.
  (`workflow.is_archived` predates the up migration and stays.)

Verified with an up→down→inspect roundtrip: the post-rollback
`workflow` schema has `status_id INTEGER NOT NULL, FOREIGN KEY
(status_id) REFERENCES workflow_status(id)` and no leftover
`run_id`/`is_canceled` columns; PRAGMA foreign_key_check passes.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 169 out of 169 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/server/api/workflows.rs Outdated
Comment thread src/server/api/workflows.rs
…300)

Addresses Copilot review concerns that PUT /workflows/{id} could mutate
run_id, is_canceled, and is_archived directly, bypassing the dedicated
side-effect logic in cancel_workflow/reset_workflow_status. Also fixes
the related issue that POST /workflows accepted those fields on the
create payload but silently discarded them.

API changes:
- Mark run_id, is_canceled, is_archived as readOnly in WorkflowModel
  (utoipa schema attribute). They are returned by GET /workflows and
  GET /workflows/{id}, but values supplied to create/update endpoints
  are now ignored.
- Drop run_id, is_canceled, is_archived from the COALESCE in
  update_workflow's UPDATE SQL.
- Add POST /workflows/{id}/archive { is_archived: bool } as the only
  way to set is_archived. Returns the post-archive WorkflowModel.
- Fold run_id bump into reset_workflow_status: every reset now also
  bumps run_id by 1 ("reset = start a new run"). This collapses the
  workflow_manager.bump_run_id helper away — initialize/reinitialize
  flows now just call reset_workflow_status, no separate bump.

Internal callers migrated:
- archive CLI command → archive_workflow endpoint
- workflow_manager.{initialize, initialize_async, reinitialize,
  reinitialize_async} → drop bump_run_id calls
- workflow_manager.bump_run_id → removed (no callers left)
- tests/test_workflows.rs, tests/test_workflow_manager.rs → archive
  via archive_workflow; renamed test_bump_run_id to
  test_reset_workflow_status_bumps_run_id

Down migration: documented the AUTOINCREMENT pairing assumption raised
in the Copilot review. The pre-merge schema technically allowed
status_id != workflow.id (independent AUTOINCREMENT sequences), but
in practice every code path in this repo paired the two IDs together.
The note flags the edge case for any DB modified by an external tool.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 171 out of 171 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 143 to +171
@@ -147,6 +165,11 @@ def to_dict(self) -> Dict[str, Any]:
if self.resource_monitor_config is None and "resource_monitor_config" in self.model_fields_set:
_dict['resource_monitor_config'] = None

# set to None if run_id (nullable) is None
# and model_fields_set contains the field
if self.run_id is None and "run_id" in self.model_fields_set:
_dict['run_id'] = None
@daniel-thom daniel-thom merged commit 2e5ac7c into main May 4, 2026
13 checks passed
@daniel-thom daniel-thom deleted the issue-300-remove-workflow-status branch May 4, 2026 00:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Merge workflow_status into workflow (eliminate 1:1 indirection and orphan class)

2 participants