Implement file bundling

[timcanham]

Deep-space missions without easy ground-in-the-loop can benefit from a file bundle. A file bundle is essentially an archive that has all the files needed for an operational activity. That would include sequences and other files (tables, etc). It is beneficial to have them as a bundle to allow for an atomic start of activities where the whole bundle passes validation or the activity doesn't execute. That avoids scenarios where individual uploads of files are corrupted and it is hard to figure out what can be run or not.

This design would include:

1. Define the bundle format

• validation (CRC, etc)
• bundling and compression format (tar + gzip?)
• manifest file to map file contents to destination

2. Ground tools for generating the file

• fprime-gds Python scripts to create the bundle

3. Component to decode the bundles

• component would take the file name (from Add file validator/dispatcher component #4390), validate it, decompress it, and debundle it. It would then call the input port of Add file validator/dispatcher component #4390 with all the files de-bundled.

[h313]

The compression format will need to be portable to non-POSIX systems like Zephyr or VxWorks as well. This means that using tar + gzip would add an additional dependency on the GNU gzip library, which may be a pain to cross-compile on non-UNIX systems.

If developer workload is not an issue, the zstd algorithm would be a better choice thant gzip, as it achieves very high compression ratios compared to zlib, which uses the same DEFLATE algorithm as gzip, while minimizing the amount of compute needed to decompress. However, using zstd would require porting the algorithm to each of the supported OS targets, or relying on a third-party fork designed for the target.

Instead, it might be a good idea to use a more common compression library like LZ4, which is present on VxWorks, RIOT, and Zephyr as well. This would require some target-specific code to be written to support connecting to the host OS's lz4 library, but is much less work than porting an entire compression library over.

As far as efficiency, LZ4 supports very fast decompression as it doesn't do Huffman coding, but that does mean that the compression ratio is worse than zlib.

[timcanham]

That's a good point. We want to have 2 features: the bundle of files into a single file, and optional compression to save uplink bandwidth. I picked tar + gzip to illustrate the concept, but pairing it with @SterlingPeet's idea below, we could offer different options.

[SterlingPeet]

I would prefer to see compression format as a dependency injection where the mission can choose the compression/decompression. In this formulation, the framework should provide a solution that makes sense as a default. It would be really neat if multiple options could be specified, perhaps in the manifest, but definitely get it to work first.

The same flexibility in the validation strategy would also be preferable.

[timcanham]

I think this is a good idea - we have the concept of a processing port for DpWriter where projects can define their own processors like compression. This could be the same - we could have a processor port for decompression and a debundler port for extracting the files.

[timcanham]

I did find this:

https://github.com/rxi/microtar

It hasn't been maintained in a while, but it still could be usable.