Data seeders and tests on synthetic data

[Vlad-Andres]

We want to investigate the possibility of having a standartized way to implement data seeders based on the data structures in the code that can be enabled per project based and that can be useful for automated testing in CI/CD.

[Vlad-Andres]

I investigated this issue and my proposal for solving it can be described as follows:

Idea

We create a workflow in the boilerplate that checks the existence of a schema/data_model.json metasyn GMF file and if it exists, generates a small dataset using metasyn and uses that data in a smoke test (just run the script on this data). The pipeline then fails if the script gives errors.

Advantages

• Easy to set up: Requires minimal actions to set this up, owners of the repository will just have to set up a GMF ones and change it as soon as data structure changes
• Extendability: Can be extended with more sophisticated tests cases on project-basis
• Opt-in: If researcher doesn't need that then they are not obliged to implement it
• Quick sanity check before deploying to main
• Useful for critical or complex projects that can be prone to abstract mistakes

Disadvantages

• Opt-in: It is non-enforceable therefore applicable only for a restricted range of repositories
• Questionable relevance given the complexity of the implementation
• Dependent on metasyn: The complexity of the dataset and in-data relationships are limited by the replication abilities of metasyn package. Also I think not every dataset used in a research project can be easily parsed into a GMF format
• Limited coverage: The smoke test only tests if the script runs and not anything more

Algorithm of actions for a new collaborator

1. Create a project from the template
2. Create a GMF file in the schema directory
3. Done, now when a PR gets created towards main, the job will run and a smoke test will run on the generated dataset.

@fqixiang, @Dutchmen04 Given this plan, what do you think about the relevance of this given the effort or maybe you have other ideas?

[Vlad-Andres]

Blocker: Issue paused until I receive feedback on my plan and further decision regarding cancelation or continuation is made

[fqixiang]

@Vlad-Andres Could you say more about this feature? I have a hard time remembering a discussion we had on this. What are data seeders? Is this about making sure that every research repository has some data to test the code? Or is it about integrating metasync into every project?

[Vlad-Andres]

@fqixiang This is about making sure we can have some minimal data for the repos such that their data processing script runs without errors. This would be mainly used in a workflow action when creating a PR to main to avoid merging into main some code that will break, it's just like a automated sanity check saying that this script works with the intended data type.

• This baseline check then (workflow action) would be inherited from the template when new contributors create their own repos from it. Moreover, if they want to have a more sophisticated pipeline with automated tests, this baseline helps it since it already uses metasyn and they would only have to define some custom tests and extend the workflow file.

• We didn't discuss about using metasyn but I found it useful since it's our product and I believe it fits well for this task and other libraries that I found are less suitable in my opinion.

We also discussed it like something optional and dependent of how hard it would be to implement versus its added value.

• Do you think that most of our projects' datasets are easily configurable via GMF?
• Do you think new contributors would like to use such a feature and extend it for their own use with some automated tests?

If the answer to the questions above are both 'No' then my opinion is that we should abort it and close the issue due to overhead. It is a good thing to have but it won't make sense if everyone will just not use it since to use it you would have to at least create and maintain the schema representations files for your dataset. I didn't find a more convenient way to do this for now.

[fqixiang]

Hi Vlad, thanks for the detailed explanation. I agree this would be an important feature/requirement to have for every (new) research repository. I also really like your suggestion about integrating metasyn.

To your questions:

Do you think that most of our projects' datasets are easily configurable via GMF?

Unfortunately, no. Quite a few of our recent projects focus on analysing text data (and other types of non-structured data), and as far as I know, such types of data isn't something metasyn can handle yet. But for projects focusing exclusively on traditional tabular datasets, metasyn would be exactly the right tool.

Perhaps for these projects, we can set up an automatic metasyn pipeline with instructions for researchers; for other types of projects, we ask researchers to come up with synthetic data themselves (for now, until metasyn or another tool is available for these data).

Do you think new contributors would like to use such a feature and extend it for their own use with some automated tests?

I think this is something we should ask them to do, instead of letting them decide. This will help with making our repositories FAIR and reproducible.

My question:

Do you think there is enough free compute with GitHub Action to run metasyn?

[Vlad-Andres]

@fqixiang, To answer your question:

Do you think there is enough free compute with GitHub Action to run metasyn?

According to documentation on actions resource limits the free tier gives a max of :

• 6h Runtime
• 500 MB artifact storage | 10 GB cache storage

I tested metasyn generating 5000 data points of the Iris dataset type and it took less than 1 second so I assume for most cases there will be no computation limitations reached.

[fqixiang]

@fqixiang, To answer your question:

Do you think there is enough free compute with GitHub Action to run metasyn?

According to documentation on actions resource limits the free tier gives a max of :

• 6h Runtime
• 500 MB artifact storage | 10 GB cache storage

I tested metasyn generating 5000 data points of the Iris dataset type and it took less than 1 second so I assume for most of the cases there will be no computation limitations reached.

Thanks for looking into this Vlad. Sounds good to me.