Skip to content
This repository was archived by the owner on Aug 21, 2024. It is now read-only.
This repository was archived by the owner on Aug 21, 2024. It is now read-only.

Evaluate usage of zst compressed files for the raw image data #575

@F-X64

Description

@F-X64

As @miyunari proposed, we should evaluate the usage of zstandard compressed files as delivery method of raw image data.
Currently the retriever creates files that are zst encoded (AWS) as well as non encoded files (google and azure).
This gives the impression that the question of encoding raw image data isn't well enough thought out.

Expected Outcome
If the decision is made to transition to zst encoding, then the transformer needs to be extended to enable decompression of said files.
Implementation example: #574

It also requires that the retriever creates consistent data across the board.

  • All created raw data files need to be zst encoded, not only AWS
  • It needs to serve a purpose

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions