Binary file content idea

[TheSirC]

Hey!

Super library I was researching this space for a solution exactly like this! Thank you for sharing it!

Regular JSON constraints apply. In particular, the input must be UTF-8. Currently, there's no way to represent files containing non-UTF-8 content.

Have you considered encoding binary file content as base64 and decoding them at runtime ?

[alurm]

Thanks! Yeah, I'm considering adding a "verbose syntax" which would support this. I also thought that maybe using a JSON version extended with byte escapes could work. I haven't decided what's better at this point.

[TheSirC]

I have little experience with byte escapes so I would have leaned on the former rather than the latter. I have no idea how they compare in terms of size in the file either.

[alurm]

I'm considering base64, but it seems that unfortunately there isn't a built-in in Nix for converting to base64.

Syntactically this could've looked like this: ["base64", <base64>].

There's a Nix implementation, but I'm not sure how performant is it.

https://gist.github.com/manveru/74eb41d850bc146b7e78c4cb059507e2

[alurm]

Another approach I'm thinking of now is adding a way to reference files in the JSON.

Here's a rough sketch.

printf '
  {"foo": {"bar": ["path", "lol"]}}
' |
json2dir $src

Is like

mkdir foo
cd foo
cp -r $src/foo/lol bar

"path", [...] can be used for paths relative to $src and not the current context in src.

If optional src isn't given, using "path" fails. And even if it is, there will be (at least basic) checks to ensure that you don't read paths from anywhere in your filesystem.

[TheSirC]

There's a Nix implementation, but I'm not sure how performant is it.

https://gist.github.com/manveru/74eb41d850bc146b7e78c4cb059507e2

I will try to see if I can benchmark it to have a point of reference. It is sure that when you are considering computing base64 in Nix you are doing something interesting 😅

Here are the tools / references I am looking at (just to increase the bus factor) :

• https://nixos.wiki/wiki/Nix_Evaluation_Performance
• https://github.com/juliuskoskela/nix-bench
• https://github.com/Mic92/nix-hyperfine

[TheSirC]

Interesting search gave this treasure in the forum ; mostly for decoding : https://discourse.nixos.org/t/decoding-base64-in-the-nix-language/33893/5

The nix binary seems to have base64 encoding done already in C++ I imagine (and I confirm).

[alurm]

Another issue is that Nix strings can't represent binary data in the first place.

NixOS/nix#1307

So, I suppose, in the general case, one would have to manually base64 pre-encode data that needs to be decoded later.

[alurm]

On the topic of binary content (and file names), YAML seems to have some support for this.

https://yaml.org/type/binary.html

Although level of support of this tag in YAML libs is unclear.