[discussion]: Consider parallelising on chunks instead of files

[signekb]

Currently, files with many chunks takes longer to convert (naturally), but this can result in one large SAS file - that takes considerably longer to convert - is left on one worker bc we parallelise on files.

One idea to optimise this could be to:

• Add an arg to convert that defines the line the chunk should start with
• Add a step to the pipeline that defines the chunks (w. SAS file path, chunk start, and chunk size)
• Feed the chunk definitions to convert in the pipeline

However, this could mean - if we want to ensure the same schema within the same file - that we also need to add a schema arg to convert 🤔

[lwjohnst86]

I think parallelizing on chunks would be quite difficult to set up/implement and would require a complete rewrite of how we do things. Not sure that would be worth it, considering how infrequently this is (ideally) done.