Serialization

[hwiesmann]

What is the recommended way to do quantity serialization?

I have to admit that I am currently not deeply familiar with the library as I am currently evaluating which "units" library to use. One requirement is to easily being able to serialize the quantity using cereal, protobuf or others.

Best regards,
Hardy

[mpusz]

Hi @hwiesmann, Thank you for your interest in the library. Serializing the underlying quantity value should not be a problem. Just call quantity.numerical_value_in(Unit) on a given quantity. Deserialization may be a bit more complicated as a quantity can't be created from just a value. It always needs a unit as well. You can either use a multiply syntax or a 2-parameter constructor.

If you want to serialize a unit or a quantity type, please note that the majority of such libraries on the market require this information to be known at compile-time. This means that you can't dynamically create any quantity with any unit from the stream. So, there is probably no sense in serializing a unit and a quantity type at all. If this was your use case, then you might be interested in #483. Please share your use case there so we will have more motivation for this feature if we try to implement it and then possibly "sell" to the ISO C++ Committee.

[hwiesmann]

Hi

in my serialization use case I would not need a dynamic creation (with respect to units) for quantities. I like to serialize something like this:

`class Demo
{
quantitysi::metre,double m_Length;
};

...
Demo demo;

serialize(demo);

deserialize(demo);
`
So, the type is explicitly known. I have more issues on how to serialize the unit itself than the value of the quantity.

[mpusz]

I am afraid that I do not follow. A unit is a part of a type. If you know the type you also know a unit. In such a case the only thing you need to serialize is the numerical value of the quantity.

[hwiesmann]

Hi, Sorry, I was a bit confused. Just forget my last message.

…

Am 27.12.2024 um 13:09 schrieb Mateusz Pusz ***@***.***>: I am afraid that I do not follow. A unit is a part of a type. If you know the type you also know a unit. In such a case the only thing you need to serialize is the numerical value of the quantity. — Reply to this email directly, view it on GitHub <#661 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD6V3GAONNWK332REBWX632HU7QXAVCNFSM6AAAAABUF5RTU6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNRXG4YTCNQ>. You are receiving this because you were mentioned.

[nebkat]

@hwiesmann I hope you have found your answer since then, but I just wanted to add:

As the mp::quantity types are simply compile-time wrappers around the underlying representation type, we are doing direct binary serialization by making structs such as:

class vehicle {
    mp::quantity<mp::si::metre, float> length;
    mp::quantity<mp::si::metre, float> width;
    mp::quantity<mp::si::metre / mp::si::second, float> top_speed;
}
static_assert(sizeof(vehicle) == 12);

vehicle v = { ... };
std::span<const std::byte> bytes = std::as_bytes(std::span { &v, 1 });
// Write bytes ... read bytes ... convert back
std::bit_cast<std::array<std::byte, 12>>(v); // Also an option

This is just a unit-safe annotated version of this, with no overhead!

class vehicle {
    float length_m;
    float width_m;
    float top_speed_mps;
}

@mpusz I believe the above is well known but I can't actually find a place in the documentation where this is truly emphasized (aside from the comparisons to std::chrono::duration). This property is insanely useful - precisely for such low-level serialization, but even just for annotating existing code with units.

With the library being so extensive there are a lot of deep technical details to get through in the documentation but I think beginners would benefit hugely from a super simplified page on why mp::quantity<mp::si::metre, float> length is better than float length; // meters, without worrying about quantities, references, SI, etc.

If you would be open to it I'd love to contribute a documentation page or a blog post on the topic, perhaps with some first hand application experience.

[chiphogg]

The above is a really simple and elegant approach --- thanks for sharing it!

One caveat is that it works well for use cases where you never serialize to persistent storage. Once you've got any long-lived serialized data anywhere, it brings a whole host of gnarly technical problems that require careful handling of versioning.

The most general approach to serialization that I know of is to make data structures in your serialization library of choice (e.g., protobuf), and make the individual fields raw numbers, with suffixes on the field name to indicate the units. Then, convert each C++ quantity field to its corresponding serializable raw-number field via a unit-safe handoff. So, continuing with the protobuf example:

message Vehicle {
    optional double length_m = 1;
    optional double width_m = 2;
    optional double top_speed_mps = 3;
};

then our serialization function is built from unit-safe handoff lines, such as this:

proto::Vehicle serialize(const vehicle &v) {
    proto::Vehicle result{};

    // ...

    result.set_length_m(v.length().in(m));
    //                ^               ^
    //                Unit-safe handoff

    // ...

    return result;
}

It's not zero-overhead, and it is more manual effort. But it has the advantages of being easy to write and to verify by inspection, and of being better able to handle data that is serialized to persistent storage at different versions of the software.

[nebkat]

One caveat is that it works well for use cases where you never serialize to persistent storage. Once you've got any long-lived serialized data anywhere, it brings a whole host of gnarly technical problems that require careful handling of versioning.

Yes, I should be clear this is not something you want to do on a whim, especially if the serialized data is to be shared to external programs or other machines - and that's ignoring the many dangers of undefined behaviors from the reinterpret_cast in std::as_bytes (though std::bit_cast is definitely a move in the right direction).

Such direct mappings are quite common in the embedded world, where the compactness and high performance are of greater importance than might otherwise be on a PC. It also helps knowing your program will only run on a single microcontroller, so the environment is well constrained, making functionality easily verifiable.

---

One of my favorite examples is RTCM (used in GNSS corrections data): it uses the absolute bare minimum number of bits to encode every value, which results in some silly quantities such as:

That is an integer with units = $25 * 10^{-6} degrees$. When I originally wrote a parser in TypeScript it was always a headache normalizing such quantities back to their base unit (many, many mistakes were made), but now it is a breeze: mp::quantity<mp::mag<25> * mp::mag_power<10, -6> * mp::degree>!

While we're on the topic - ImHex is an exceptional tool for analyzing binary data.

[mpusz]

@nebkat, you are so welcome to contribute 😃

We could definitely benefit from pages:

• Benefits over double (somewhere between "Quick Start" and "Look and Feel"),
• Serialization (in "Use Cases" chapter).

If you have other ideas for improvements, please let me know. We really need some help with the library on many aspects (documentation, testing, design, implementation, systems definitions, examples, etc.).

For example, maybe you have some ideas on how to improve our really simple geographic.h? For sure, we could benefit from the wrap-around semantics for the representation type of latitude and longitude.