XML Support

[lschmierer]

Hi, I would like to get a discussion about XML support rolling and potentially contribute to its development.

Unfortunately the serde approach does not map very well to the way FHIR XML and JSON models differ.
I wrote down two different proposals that both have their own trade-offs.
Maybe we can start a discussion on how XML support in HFS could look like.

Motivation

• Add support for parsing and emitting FHIR XML alongside the existing JSON pipeline.
• Preserve FHIR semantics for primitive extensions, choice elements, and resource type dispatch across both encodings.
• Unblock downstream users that need to exchange XML bundles with systems that do not offer JSON.

Current State

• Generated types derive FhirSerde, which emits JSON-specific Serialize/Deserialize impls.
• Primitive metadata lives in underscore companions (e.g., value / _value), matching JSON but not XML.
• No serializer context is threaded through the generated code, so per-format behavior cannot be toggled today.

XML Library Considerations

Both options require custom helios_xml wrapper functions. Unlike JSON (where serde_json is the de facto standard), there is no standard serde XML implementation. The semantics of quick-xml and serde-xml-rs differ significantly, making it impractical to support both interchangeably.

Therefore, regardless of which option is chosen, I would suggest:

• Introducing a helios_xml module/crate with helios_xml::from_str(), helios_xml::to_string(), etc.
• Picking one XML backend (likely quick-xml due to better performance and streaming support)
• Encapsulating all XML-specific behavior behind this interface

Option 1: Context-Driven Serialization (DeserializeSeed Approach)

Extend generated code to support format-aware serialization and deserialization using context types and DeserializeSeed to thread context through the tree.

use helios_fhir::r4::Patient;
use helios_fhir::serde::{SerializationContext, DeserializationContext};
use helios_xml;

let patient = Patient::default();

// Existing JSON behavior (default `Serialize`)
let json = serde_json::to_string(&patient)?;

// JSON with context
let ctx = SerializationContext::json(&patient);
let json = serde_json::to_string(&ctx)?;

// XML with context (using helios_xml wrapper)
let ctx = SerializationContext::xml(&patient);
let xml = helios_xml::to_string(&ctx)?;

// or rather
let xml = helios_xml::to_string(&patient)?; // where helios_xml::to_string wraps SerializationContext::xml(&patient) internally

// Deserialization with context (using helios_xml wrapper)
let patient: Patient = helios_xml::from_str(xml_str)?;

Implementation Notes

• Rebuild all serde impls to operate on SerializationContext<T> / DeserializationContext.
• Implement DeserializeSeed for all types to enable context passing during deserialization
• Context wraps values during serialization and provides seeds during deserialization.
• Default Serialize/Deserialize impls continue to work for JSON without any context.

Context Design

struct DeserializationContext {
  format: Format,
  mode: DeserializationMode,  // Strict/Lax/Compatibility
  config: Config,             // User preferences
}

• Enables strict/lax/compatibility modes and other custom behaviors (in the future)
• See fhirbolt's implementation which implements this model

Pros

• Proper solution using serde's intended mechanisms (DeserializeSeed)
• Efficient: direct struct ↔ format with no semantic mismatch
• Can evolve from minimal to rich context as needs grow
• Preserves current JSON-oriented API as default (opt-in contexts)

Cons

• Requires reworking all serde code to implement DeserializeSeed
• More complex than Option 2 (custom serializer)
• Direct usage with serde_json is less straight-forward

Option 2: Custom XML Serializer/Deserializer Built on JSON Model

Provide dedicated serde::Serializer/Deserializer implementations that internally translate the existing JSON-oriented structs into XML, without changing generated serde impls.

use helios_fhir::r4::Patient;
use helios_xml;

let patient = Patient::default();

// JSON remains unchanged
let json = serde_json::to_string(&patient)?;

// XML without changing struct serde impls
let xml = helios_xml::to_string(&patient)?;        // custom Serializer intercepts JSON model
let parsed: Patient = helios_xml::from_str(&xml)?; // custom Deserializer produces JSON model

Implementation Notes:

• Implement custom serde::Serializer and serde::Deserializer that understand the JSON model's conventions.
• The custom serializer intercepts calls like serialize_struct, examines field names for patterns (value/_value, valueString/valueInteger), and emits proper XML.
• The custom deserializer reads XML and reconstructs the JSON-style struct with underscore extension fields.
• No changes to existing serde impls.
• Encapsulate all XML logic in separate helios-xml crate.

Pros:

• Zero changes to generated structs or existing JSON code
• Clean encapsulation: XML support is opt-in via separate crate
• No breaking changes to current API
• No or only few changes to existing marco-based Serialize/Deserialize implementations

Cons:

• Performance penalty: semantic mismatch between JSON model and XML requires interpretation overhead
  • Basically round-trip: value (struct) -> _value (JSON) model -> value (XML)
• Less extensibility (features like setting strict/lax mode can not be set at runtime)

Discussion

• I think Option 1, although the most work, would be the most future-proof design
  • Is different levels in strictness and validation (an potential further customizations) sth. you would like to support at some point?
  • JSON support can likely be retained in a non-breaking manner

// Non-context API: Just delegate to context with JSON default
impl Serialize for Patient {
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error> {
        // Delegate to context-aware version
        SerializationContext::json(self).serialize(serializer)
    }
}

impl<'de> Deserialize<'de> for Patient {
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error> {
        // Delegate to seed-based version with JSON context
        DeserializationContext<Patient>::json().deserialize(deserializer)
    }
}

• On the other hand, option 2 is way easier to implement and the mental model is easier to understand

I would love to hear your thoughts on this!

[smunini]

Thanks for the detailed proposal on XML support! This is definitely something we want to add to HFS. Here are my thoughts:

Overall Direction

I agree with pursuing Option 1 (Context-Driven Serialization using DeserializeSeed), but with some architectural modifications detailed below.

Clarifications & Decisions

On "Seed"

For those unfamiliar, the DeserializeSeed pattern in Rust's serde framework allows passing contextual information through the deserialization tree. This enables format-specific decisions at each node during parsing - exactly what we need for handling both JSON and XML representations of FHIR resources.

On DeserializationMode

For the initial implementation, I'd prefer to start with strict mode only and throw errors on any issues. We can always add Lax or Compatibility modes in the future if real-world usage demonstrates a need, but I'm not yet convinced of the use case. This keeps the initial implementation simpler and ensures data quality.

Architectural Changes

Instead of creating a helios_xml crate, I'd like to introduce a helios_serde crate that centralizes all serialization/deserialization capabilities across formats (JSON, XML, CSV, NDJSON, Parquet). This provides several benefits:

1. Centralized format handling - One place for all format-specific logic
2. Code consolidation - Currently we have format_json, format_ndjson, format_csv, and format_parquet functions in crates/sof/src/lib.rs that should be moved to this new crate
3. Better abstraction - The SOF crate can depend on helios_serde for all format operations
4. Easier maintenance - Changes to the fhir-macro (which are a bit tricky) can be coordinated in one place

Testing Structure

To support this, we should reorganize the test files:

Rename existing tests:

• crates/fhir/tests/test_examples.rs → test_json_examples.rs
• crates/fhir/tests/test_serde.rs → test_json_serde.rs

Add new XML tests:

• crates/fhir/tests/test_xml_examples.rs
• crates/fhir/tests/test_xml_serde.rs

Implementation Plan

Would you be interested in implementing this with the following phases?

1. Phase 1: Create helios_serde crate and migrate existing format functions from SOF
2. Phase 2: Implement context-driven serialization infrastructure with DeserializeSeed
3. Phase 3: Add XML-specific serialization/deserialization logic
4. Phase 4: Add comprehensive XML test coverage

[lschmierer]

Thank you for the quick response! I hope you were not offended by the use of AI in my initial post - it helped me organize my thoughts after a long day of work.

Context Design

For the initial implementation, I'd prefer to start with strict mode only and throw errors on any issues.

I totally agree. The DeserializationContext struct was more or less taken from my older fhirbolt implementation for demonstration purposes.It is always better to start slim and only add features as need arises.

Default Model

I'd like to introduce a helios_serde crate that centralizes all

This makes sense. Would you still want to support "pure" serde_json::to_string(&patient)?; usage or channel JSON through helios_serde as well?
I see the following options:

• Still have Serialize and Deserialize implementations directly on the resource implementing the JSON model (as outlined in my comment about preserving compatability with the current API). The implication would be that if users would use other serde (de)serializers (like bincode or cbor) directly on the resources, these would serialize the JSON model (with primitive _elements). But maybe that it even desired for formats like MongoDBs BSON.
• Have Serialize and Deserialize implementations directly on the resource implementing the "FHIR model" (aka. XML model). This would break serde_json::to_string(&patient)?; usage, but is the "formally correct" model.
• Don't implement Serialize and Deserialize implementations directly on the resource, but force usage through SerializationContext and DeserializationContext

I am undecided between option 1 and 3. The combination of the prevalence of serde_json in the Rust community and the lack of a de-facto XML lib lets me rule out option 2. Also I feel that JSON is taking over from the XML dominance in the FHIR-space.

What do you think>

Macro

Changes to the fhir-macro (which are a bit tricky) can be coordinated in one place

Finally I would like to point out that I would keep the derive-macro base approach. I like it much more to my original aproach where I had loads of generated files.

[smunini]

Hi Lukas,

I had an initial attempt at laying the groundwork for what you have proposed. The challenge we have is that Rust's serde and serde_json nicely separates the abstraction layer (serde) with a format implementation layer (serde_json) while at the same time FHIR's JSON has custom semantics around extensions, choice types and other areas that blend these two concepts. The helios-fhir-macro currently generates code with tight coupling to serde_json. The FhirSerde procedural macro embeds direct serde_json calls in many locations. This is incorrect - we should only be returning serde semantics from that macro, and leave formatting elsewhere - in helios_serde.

It seems to me that we should try to follow serde's design principles in our code structure as well. Here is an (unfinished) branch that explores this approach:

I'd like to run a performance test to see how the addition of DeserializationSeed and this approach generally might impact performance. It may not, in which case we may continue with this branch. If it does impact performance, I don't think it would be wise to proceed down this path. JSON just dominates everything. It doesn't make a lot of sense to sacrifice performance for some of these other formats that are infrequently used.In that case, I think Option 1 would be preferred.

Have a look and let me know what you think!

[lschmierer]

Hi Steve,

great to see the progress on this! I had a look at that branch and a few ideas came to my mind.

The FhirSerde procedural macro embeds direct serde_json calls in many locations. This is incorrect - we should only be returning serde semantics from that macro, and leave formatting elsewhere - in helios_serde.

If you are fine with ditching direct serde_json usage (like serde_json::to_string(&patient)?;), the macro generated Serialize and Deserialize implementations could just stick to the format agnostic "core" FHIR model (which is e.g. used when traversing with FHIRPath). The XML model ist close, but differs slightly e.g. in nesting of contained resources.
The JSON and XML (de)serializers located in helios_serde could then take care of these translation.
I think for this approach we would not even need the DeserializeSeed?

I am not 100% sure though if it is possible to channel all context the (de)serializers need through the serde model.
In fhirbolt, I had support for a generic element model and for that needed to provide some type information as static lookup tables. I think I used it for the XML deserializer at some point, but was later able to remove it again.

I'd like to run a performance test to see how the addition of DeserializationSeed

I have two thoughts on that:

• If you make the contexts generic, you do not need to declare all these ...Seed structs. This prob. does not have a performance impact though. I have used this approach in fhirbolt
• But what might have a performance impact: Both your current wip implementation and mine (fhirbolt) will have a lot of runtime branching because of the match context.format() { .... }. What do you think about also making the contexts generic over the format? What do you think about something like

  pub struct DeserializationContext<V, F> {
       _type_phantom: PhantomData<V>,
       _format_phantom: PhantomData<F>,
  }
  
  ...
  
  impl DeseserializeSeed for DeserializationContext<Observation, Format::Json> { .. }
  impl DeseserializeSeed for DeserializationContext<Observation, Format::Xml> { .. }

  I think that would have virtually zero overhead to your current (main branch) JSON-only implementations.

[smunini]

Hi Lukas,

You had me really thinking on this, so I implemented it in the same branch. Here is what changed:

1. Removed DeserializeSeed Infrastructure

• Deleted seed.rs and context.rs from helios-serde
• Removed 75+ lines of seed generation code from the macro
• Simplified Element<V,E> deserialization to use standard Deserialize

2. Created Format-Agnostic IR Layer

• New ir.rs module with:
  • IrValue: Format-agnostic value representation (Null, Bool, Number, String, Array, Object)
  • IrElement<V>: Intermediate representation for FHIR Element types
• Provides foundation for future format translators - especially XML

3. Established Translator Pattern

• Created helios_serde::json module as the public API
• All FHIR serialization now goes through this layer:

  use helios_serde::json;
  
  // Serialize
  let json = json::to_string(&patient)?;
  
  // Deserialize
  let patient: Patient = json::from_str(&json)?;

4. API Change (which is perfectly ok)

• Old: serde_json::to_string(&patient)
• New: helios_serde::json::to_string(&patient)
• Updated all crates (fhirpath, sof, etc) to use new API
• This enforces that FHIR serialization goes through the translator layer

Let me know what you think. I think this provides a much cleaner approach to implementing other formats such as XML and others.

Here is the branch. Also, you may wish to read the updated README.md.

[lschmierer]

Hi, on first glance the approach with the ir model sounds reasonable. However, I have a hard time to follow how this would work out in practice. I have the following questions:

• How would the FHIR Struct (macro-generated) <-> Serialize/Deserialize part look in practice?
  • Would the current by-macro generated Serialize Deserialize be changed to emit the values following the ir semantics (through the serializer.serialize_xxx(..) methods)? Currently they still emit primitive extensions like _value.
  • If so, how can we prevent users to still just call serde_json::to_string(&patient), which would then yield wrong JSON? One idea would be to have a runtime typecheck of the passed serializer using TypeId::of()
• The IR (format-agnostic) <-> Translator Format-specific (JSON, XML, etc.) part would probably be implemented as serde Serializer and Deserializer then? The ir module would then actually be just a "mental model" and the structs actually not needed? Maybe I am fundamentally misunderstanding something...
  • If you actually want to serialize into the existing IrElement struct, wouldn't that involve copying the whole resource twice (into the ir model and back out)?

I asked my LLM of trust to create a comparison table

Feature	Option 1: Generic Context (DeserializeSeed)	Option 2: Custom XML Serializer (JSON-Model)	Option 3: IR Translator Pattern
Core Mechanism	Struct <-> Format (Direct)	Struct -> JSON-Model -> XML-Format	Struct <-> IR <-> Format (Two-step)
	(I think) possible compile-time dispatch via generic context (e.g., Context<V, JsonFormat>). This avoids all runtime match branching.	A custom serde::Serializer intercepts the existing JSON-model serde calls and translates them to XML.	1. Macro generates serde impls to turn a Struct into a format-agnostic IrValue. 2. A "Translator" (e.g., helios_serde::json) converts the IrValue to the final format.
Performance	Very High. Direct, one-pass (de)serialization. Monomorphization creates optimized, format-specific code paths with zero branching overhead.	Low. (only for XML) High overhead. Involves a complex, on-the-fly semantic translation (e.g., _value -> attribute).	Good. Involves a two-step process (Struct -> IR, IR -> JSON). This creates an intermediate copy, which is the primary performance trade-off.
Implementation Effort	Very High. Requires rewriting all serde impls for all types to use DeserializeSeed and generic contexts. Extremely complex to implement and maintain.	Low. No changes to any existing macros or serde impls. All logic is in one new (but complex) Serializer/Deserializer.	High. Requires rewriting all serde impls, but the logic is simpler (Just map fields to IrValue). The main effort is in building the IR <-> Format translators.
Extensibility	High. The Context<V, F> pattern is a natural place to add new features (e.g., strict/lax) or new format types (F).	Low. The serde::Serializer trait is a poor fit for passing rich configuration (like strict/lax) from the user.	Very High. This is the key benefit. To add XML, you just write a new IrValue <-> XML translator. The FHIR structs and macros never change.
Code Coupling	Low. The logic for each format is perfectly isolated in its own impl block (e.g., impl ... for Context<V, JsonFormat>). No format knows about any other format.	High. The XML translator is tightly coupled to the implementation details of the FHIR struct's JSON-model serde impl.	Very Low (Ideal). The "Translator" pattern perfectly decouples the concerns: - Structs know only the IR. - Translators know only IR <-> Format.
API Impact	Low. Can be designed to be non-breaking. serde_json::to_string could still work by default (by delegating to the Context<V, JsonFormat>).	None. No changes to existing API.	Breaking (Intentional). serde_json::to_string(&patient) is broken, as it would serialize the IR itself. Users must use helios_serde::json::to_string(&patient).

I might give Option 1 a try (tomorrow or on Sunday) and keep you updated!

[lschmierer]

Update: I implemented the context aproach (with DeserializeSeed) for JSON in #21. It still has some rough edges, but the general approach seems to work. This README explains the design a little bit.

The PR has some remaining TODO items (mostly cleanup & some performance improvements).
Unfortunately, since its such a big change, its quite hard to review. But its very hard to split these in well-isolated small commits.
Please let me know what you think!

[smunini]

Hi Lukas,

Thanks for the PR - I'll check that out. I tried having a quick look, and you're right - GitHub can't display the diffs, so I'll need to use some local tools. Thanks for the README too. Still working to understand what you have done.

I did some work yesterday to better understand any performance implications of these approaches, and now have a test bench we can use.

Relative to your previous two questions:

1. Generic contexts / eliminating Seed structs

I initially implemented the DeserializeSeed pattern (a251f4a4), but then replaced it with a translator pattern
(ea3214fb) that's even simpler - zero seed structs needed.
Before (DeserializeSeed approach):

// Required seed structs for every type
pub struct PatientSeed { context: FhirDeserializeContext }
pub struct HumanNameSeed { context: FhirDeserializeContext }
// ... hundreds more
After (translator pattern):
// Standard Deserialize - no seeds needed!
impl<'de> Deserialize<'de> for Patient {
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error> {
        // Just works with any deserializer
    }
}

The macro now generates standard Serialize/Deserialize implementations, with format-specific logic isolated in helios_serde::json and (future) helios_serde::xml modules.

I too was worried about a copy penalty, but here are the performance results of the main branch vs the refactor/serde-abstraction branch:

Performance Comparison Summary (100 iterations)

Main Branch (baseline):

• R4: 2.730 seconds
• R4B: 2.120 seconds
• R5: 2.140 seconds
• R6: 1.980 seconds

Refactor/serde-abstraction Branch:

• R4: 2.710 seconds
• R4B: 2.110 seconds
• R5: 2.140 seconds
• R6: 1.960 seconds

Performance Impact (Refactor vs Main):

• R4: No significant difference (2.730s → 2.710s, ~0.7% faster)
• R4B: No significant difference (2.120s → 2.110s, ~0.5% faster)
• R5: No significant difference (2.140s → 2.140s, identical)
• R6: 1% faster (IMPROVEMENT) (1.980s → 1.960s)

About IrElement and double copying

You asked if serializing through IrElement would involve copying the whole resource twice. The answer is no - the current implementation doesn't actually go through IrElement for serialization:
Current JSON flow:
FHIR Struct → (serde) → JSON output
The helios_serde::json module is a thin wrapper around serde_json - no intermediate representation in the hot path. IrElement/IrValue exist as infrastructure for future XML support, but aren't used for JSON serialization.

One localized exception: Array fields (Vec<Element>) do convert elements to serde_json::Value temporarily to handle FHIR's parallel array pattern (field / _field), but this is limited to arrays and necessary for correctness. I can look into correcting that.

So there's no double copying - just direct serde serialization to JSON which is why we didn't see any performance penalty with the intermediate approach.

2. Runtime format branching

The translator pattern achieves zero runtime branching. Format selection happens at compile time through API choice:

  // User explicitly chooses format
  use helios_serde::json;
  let patient = json::from_str::<Patient>(data)?;
  // Future:
  use helios_serde::xml;
  let patient = xml::from_str::<Patient>(data)?;

No match context.format() in the generated code or hot paths - the Rust compiler monomorphizes the correct code path at compile time.

Architecture

• Macro generates format-agnostic FHIR structs with standard serde traits
• Format-specific translators in helios_serde::{json, xml}
• Intermediate representation layer for shared abstractions
• See ./crates/serde/README.md for details

Direct serde_json usage

I think maintaining direct serde_json usage is a nice to have but not have "must have" feature. We could document that easily that they need to use helios_serde. I believe your implementation maintains direct serde_json usage, so that's a nice plus.

[smunini]

Clarification/Correction: About "Currently they still emit primitive extensions like _value." - I didn't find that in the code. There localized exception I mentioned above was a json variable name that was left over. I have corrected that. The actual extension names are emitted in helios_serde's utilities.rs.

[lschmierer]

About "Currently they still emit primitive extensions like _value."

What I meant is that the Serialize/Deserialize implementations still exhibit JSON-specific behaviour in at least in these locations:

• hfs/crates/fhir-macro/src/lib.rs

  Line 719 in cc08e34

  let underscore_variant_key = format!("_{}", variant_key);

• hfs/crates/fhir-macro/src/lib.rs

  Line 1579 in cc08e34

  let underscore_variant_key_str = format!("_{}", variant_key); // For error messages

• hfs/crates/fhir-macro/src/lib.rs

  Line 1764 in cc08e34

  let underscore_name = format!("_{}", base_name); // e.g., "_authorString"

• hfs/crates/fhir-macro/src/lib.rs

  Line 1844 in cc08e34

  format_ident!("_{}", effective_field_name_str);

If I understand your approach correctly (Serialize and Deserialize should emit the "plain" FHIR model, whatever the serialization format is), this would have to change? And then an IR <-> JSON translation layer is still to be built?

Direct serde_json usage

I think maintaining direct serde_json usage is a nice to have but not have "must have" feature. We could document that easily that they need to use helios_serde. I believe your implementation maintains direct serde_json usage, so that's a nice plus.

In the current iteration I removed all direct Serialize/Deserialize implementations. The reason is that with still having these, it is too easy to miss a specific case where the code still calls these directly instead of going through SerializationContext/DeserializationContext.
Also I came to the conclusion that I think it is actually better to not have direct serde_json interoperability and favor explicitness over implicitness. The fact is that the data model for JSON and XML is just different and direct serde_json interoperability would just hide that

FhirSerialize and FhirDeserialize

By the use of our own FhirSerialize and FhirDeserialize traits, it also needs zero structs. FhirSerialize and FhirDeserialize are implemented directly on the resources. The implementation is more verbose, because we can not use the serde_derive macros anymore (because we need to propagate the context down).

[smunini]

Thanks Lukas,

You're correct - there is still some JSON-specific behavior in the macro after all. I had overlooked those parts. I'll work on that.

I had a look at your PR, and was curious, so I ran the benchmark, and it turns out it's roughly 60% slower to my surprise. I know you had some performance improvements planned in your PR comments, however I think there is something else more fundamental.

I'm beginning to think that Option 2: Custom XML Serializer/Deserializer Built on JSON Model might be the best option after all :-)

Performance Comparison Report

Timestamp: 20251115_200101

Branch,Version,Time (seconds)
-----,-------,----------------
Extracting timings from ./benchmark_results/main_20251115_200101.txt for branch main
main,R4,2.700
main,R4B,2.100
main,R5,2.140
Extracting timings from ./benchmark_results/refactor_20251115_200101.txt for branch refactor
refactor,R4,4.510
refactor,R4B,3.350
refactor,R5,3.480

Detailed Results:
Main branch: ./benchmark_results/main_20251115_200101.txt
Refactor branch: ./benchmark_results/refactor_20251115_200101.txt

Performance Differences (Refactor vs Main):

R4: 67.00% slower (REGRESSION)
R4B: 59.00% slower (REGRESSION)
R5: 62.00% slower (REGRESSION)

[lschmierer]

You're correct - there is still some JSON-specific behavior in the macro after all. I had overlooked those parts. I'll work on that.

I am still curious on how exactly this works out. As I still think the IR -> JSON and IR -> XML translators will need some context on how to serialize intermediaries (I am thinking nested resources, and how to know if to write a _primitiveExtension).

I had a look at your PR, and was curious, so I ran the benchmark, and it turns out it's roughly 60% slower to my surprise. I know you had some performance improvements planned in your PR comments, however I think there is something else more fundamental.

I pushed a bunch of performance improvements (that were quite fundamental) - maybe you can run the performance comparison again?
However, the serde_derive macros are so heavily optimized (i.e. dispatching of flattened enum variants which is the last major performance problem on my branch) that is impossible or at least very very much effort to apply all these optimizations. Stateful derive macros were discussed but are nowhere on the horizon serde-rs/serde#881.

I'm beginning to think that Option 2: Custom XML Serializer/Deserializer Built on JSON Model might be the best option after all :-)

I tend to think so too 😅 Not sure if the complexity in my branch is justified, just for a potential XML performance gain.
But let’s the what another performance comparison yields. Can you share the code for the test bench?

[smunini]

Latest performance results on your branch below - big improvement!

The benchmark can be found here

I tend to think so too 😅 Not sure if the complexity in my branch is justified, just for a potential XML performance gain.
But let’s the what another performance comparison yields. Can you share the code for the test bench?

I agree - it's getting really large. The other challenge as you point out is that we won't benefit from improvements in serde_derive as they evolve too. With a lot of eyes on serde, it will continue to be enhanced and improved over time from what is already a really strong crate.

You have found some interesting items in there though. I may cherry-pick the fhir-gen code that sorts the structure definitions such that we don't end up with such big change sets every time we generate the code - that is a good idea. If you want to put that in a separate PR, i'd gladly take that one. The commit you pointed out had the bulk of it but I think the actual change was spread across a couple commits in actuality and I didn't have time to sort them out.

I think my plan may be to explore the IR -> JSON approach a bit more, and if I can get that to work well, I'll go with that. If not, I think Option 2 is probably a good option.

We never know how these things work out until we actually implement them.

Performance Comparison Report

Timestamp: 20251116_094257

Branch,Version,Time (seconds)
-----,-------,----------------
Extracting timings from ./benchmark_results/main_20251116_094257.txt for branch main
main,R4,2.730
main,R4B,2.110
main,R5,2.130

Extracting timings from ./benchmark_results/refactor_20251116_094257.txt for branch refactor
refactor,R4,3.070
refactor,R4B,2.270
refactor,R5,2.270

Detailed Results:
Main branch: ./benchmark_results/main_20251116_094257.txt
Refactor branch: ./benchmark_results/refactor_20251116_094257.txt

Performance Differences (Refactor vs Main):

R4: 12.00% slower (REGRESSION)
R4B: 7.00% slower (REGRESSION)
R5: 6.00% slower (REGRESSION)

[lschmierer]

I got curious and quickly did some comparison with my old fhirbolt implementation (the code, helios main depicted as baseline). fhirbolt is quite fast, except for the deserialization through the Resource enum. I suspect similar issues as with the missing serde_derive optimizations as outlined above. But I will probably not have the motivation to look further into this. 😅

I can confirm the regression of the context approach, except for writing plain structs. This is weird, this must be sth. easy to address, because a simple match should not cause this.

read account enum                  fhirbolt                  17.194 us         1.99x
                                   helios main                8.623 us         1.00x
                                   helios serde-context       8.609 us         1.00x

read account struct                fhirbolt                   4.595 us         0.89x
                                   helios main                5.142 us         1.00x
                                   helios serde-context       6.593 us         1.28x

write account enum                 fhirbolt                   1.142 us         0.63x
                                   helios main                1.799 us         1.00x
                                   helios serde-context       7.008 us         3.90x

write account struct               fhirbolt                   1.171 us         0.65x
                                   helios main                1.790 us         1.00x
                                   helios serde-context       1.026 us         0.57x

I suspect the significantly faster write account struct of serde-context compared to main is due to this commit 357e0ac which avoids multiple serializations to serde_json::Value just for the emptiness check. This should be transferrable to the current main branch as well.
It would be interesting to see the effect of that commit applied main and then run the benchmark again.

I agree - it's getting really large. The other challenge as you point out is that we won't benefit from improvements in serde_derive as they evolve too. With a lot of eyes on serde, it will continue to be enhanced and improved over time from what is already a really strong crate.

I agree. The performance advantage of write account struct is nothing inherent to the serde-context aproach. And even if so, the additional maintenance overhead of the serde_derive less implementation does not justify it.

You have found some interesting items in there though. I may cherry-pick the fhir-gen code that sorts the structure definitions such that we don't end up with such big change sets every time we generate the code - that is a good idea. If you want to put that in a separate PR, i'd gladly take that one. The commit you pointed out had the bulk of it but I think the actual change was spread across a couple commits in actuality and I didn't have time to sort them out.

Sure, I will see what makes sense and submit one or a few separate merge request. This will take me a few days though.

Update: I pushed those changes here #22