Feat: Ability to handle multiple occurrences of same datasets and tags

[JoiBlumen]

According to the Visa spec TLV-format fields can contain multiple occurrences of same datasets and tags.
For instance, messages containing itemized statements may contain multiples of same
dataset IDs, each possibly having same TLV elements for line items of statements or
receipts.

Example from composite-fields.md

field125Spec := &field.Spec{
    Length:      255,
    Description: "Extended Transaction Data",
    Pref:        prefix.Binary.L,
    Tag: &field.TagSpec{
        Length: 1,                         // Data Set ID length in bytes
        Enc:    encoding.ASCIIHexToBytes, // Data Set ID below is ASCII hex and will be converted to bytes (1 byte)
        Sort:   sort.StringsByHex,
        // Handle unknown Data Set IDs
        SkipUnknownTLVTags: true,
        PrefUnknownTLV:     prefix.Binary.LL,  // Data Set length format
    },
    Subfields: map[string]field.Field{
        "56": field.NewComposite(&field.Spec{ // Data Set ID                 <-------- This Dataset can occur n times
            Length:      1535,
            Description: "Merchant Information Data",
            Pref:        prefix.Binary.LL,  // Length of the entire Data Set value
            Tag: &field.TagSpec{
                Enc:                encoding.BerTLVTag,
                Sort:               sort.StringsByHex,
                SkipUnknownTLVTags: true,
            },
            Subfields: map[string]field.Field{                                                
                "01": field.NewString(&field.Spec{                           <-------- This Tag can occur n times
                    Length:      11,
                    Description: "Merchant Identifier",
                    Enc:         encoding.EBCDIC,
                    Pref:        prefix.BerTLV,
                }),
            },
        }),
}

If multiple occurrences of Dataset IDs or tags, are in the message, it will be overwritten.

Looking at the bertlv library, it supports multiple tags within a dataset, which is nice and we could use that to parse the dataset. But then we also need to be able to handle multiple datasets with the same ID.

My question is if there is any planned support for this? I'm unsure if this is a Visa specific requirement, or if the library might be silently overriding multiple occurrences in the case of someone defining the whole spec like the example.

[alovak]

Hey, @JoiBlumen!

I have checked Visa spec and while I see duplicate data set IDs in the description of the field 104, when I check data set description it doesn't assume that there are different specs for the data set. What I see is that it has different tags.

Because we use map for Subfields, I don't see how we can support same keys / data set IDs.

The tricky part with designing something like you are looking for - is to know which spec you should apply for data set ID if you have more than one for it. I thought about features that could tell parser / unpacker which subfield it should unpack if there are two similar tags, but it would significantly complicate the package and based on my experience, something like this happens rarely and can be handled using generic field type like Binary and custom code to parse / build values.

[JoiBlumen]

Thanks for looking into it.

I realize this is quite tricky as maps are used for subfields. It might be easier to discuss this if we keep the scope to same spec for a duplicate dataset ID.
An example from visa is in field 56 Dataset ID 05, Account Owner Data. There can be multiple account owners, and then this dataset gets repeated with the same spec.

In the latest Visa spec I have, under "TLV Processing" they state: "all TLV-format fields can contain multiple occurrences of same datasets and tags". So while it's certainly rare, it's possible for all of Visas TLV-format fields. It's easy to miss it if you define the composite field like in the example, as only the last occurrences will be stored, so unpacking and then packing you'll silently loose data, perhaps overriding during unpacking should emit an error or warning?

In any case my current solution is to define the field as Binary and parse it into custom DataSet slices which hold bertlv.TLV. It would be nice to have official support in this library for lists of fields, but I get that it would be very different from the current map implementation throughout.

If you can think of anything else, I'm happy to brainstorm and even contribute to the library.
Thanks!