Workflow State for Human Consumption and Automatic Validation

[jmchilton]

Overview

Workflow Format 2 can have the... low-level, verbose tool state format of .ga included (with doubly encoded JSON, etc) (the field name is tool_state on a step) or it can have a newer, simple JSON format of the parameters (the field name is just state on a step). When Galaxy exports a format 2 workflow - it just uses the .ga tool state limiting the ability to read and write these workflows. I would feel much more confident about the future of the format and feel we are delivering more if we could export the workflows with the cleaner format of tool state.

Given the mess of things like workflow modules and parameters/basic.py, I think getting this state import/export right is the messiest piece of the whole picture of having a human readable/writable workflow ecosystem going and have it be fun and efficient to develop against.

I'd like to spend server weeks tackling this task really solidly building on the work in #18524. I think the steps look something like...

1. write a tool state/pydantic validator for workflow tool state (using ideas and meta model implemented in Add Tool-Centric APIs to the Tool Shed 2.0 #18524)
2. implement a workflow export mode that exports gxformat2 with a clean state instead of tool_state if the result validates against our tool state model for workflows
3. ingest all of the IWC workflows and convert the tool_state to state and verify the workflow tests still pass
4. once we're confident this works well for a variety of complex workflows - make the default when exporting format 2 workflows

After that is done, we will have the tooling for clean workflow exports and we will have confidence to know it is working well with a variety of complex workflows. Additionally, the models in #18524 can be produces from a toolbox in Galaxy context or from the tool shed outside the context of a Galaxy (e.g. gxformat2, planemo, IDEs, etc...) so we will be able to readily translate this workflow validation we do for our confidentio into CLI tooling (Planemo) and the galaxy-language-server that will let users catch typing issues, connection issues, etc... instantly.

Workflow validation

This is a bit more of a deep dive on how I vaguely think I will implement (1) above. Our meta model has a variety of exports to create tool state validation for different contexts. I've been using a UML diagram to describe them and I have updated it to include two new modes - "workflow_step" and "workflow_step_linked".

The workflow_step validator would validate the "step" state directly in YAML as is. This isn't super helpful as most things can be empty (except conditional discriminators) but it would catch some problems without any additional work. The second model is one that I would apply after I take the state and insert "descriptions" of some form for default and source values defined as step inputs. Maybe the model would expect a {"class": "link", "source": "XXXX", "source_type": "data"} or something like that. So I would validate the literal representation with the first model, preprocess, and validate the "linked" version with second model.

[mvdbeek]

That's an awesome plan, I like it a lot. My only question is, do you think it isn't possible to implement 1 using a workflow step flavored JSON schema instead of a pydantic validator ? It seems like that would be easier to adapt to typescript code (thinking about gls and our own workflow editor).

[mvdbeek]

Also if there's anything I can help with let me know, happy to update the TTS after we merge #18524

[jmchilton]

@mvdbeek - we can export JSON schema models for all of the different tool state scenarios - like we're doing in #18524 for the tool request state. We can do a different endpoint for each kind of tool state or we can parameterize it. It should help bootstrap non-Python validation. That is lovely and I have no issue with.

If we're already in Python though and we have the models - I imagine it would just be extra work to convert it to JSON schema and validate it using a different tool set though - right? Maybe "ToolSource -> Pydantic -> Json Schema" was an odd choice and we should have describe the meta models in Json Schema directly from the ToolSource... especially given how we're torturing Python's typing system to make this all work but this is the path we're down and we have a bunch of work and validation in place and we've got validation for 3 or 4 useful contexts with different representation in place or nearly in place. I think this path will work out though and we will appreciate the nice validation messages Pydantic bings in and I'm super eager to support validation in other languages and I think we have a path forward on that.

As for the anything to help... if I get (1) and (2) in the next couple months could you take over step (3)? I will provide the tooling and Planemo updates to deal with all of it - but I think you'll do much better at the large scale automated testing thing and you're more familiar with the IWC plumbing. I guess step (3) is also blocking on updating the main tool shed - so that may be moot.