Add semantic version register

[glopesdev]

Summary

This PR adds a new register to the core device schema called R_VERSION.

Motivation

The current registers 01-07 are used to version hardware, assembly, harp core, and application firmware.

The assembly version has been unused in practice, and we are now moving in a direction where we need to keep track of the implemented protocol version explicitly, since the harp core and its implementation have diverged and now support multiple core implementations (pico and atxmega). The current version registers also do not support specifying a "patch" version, and require multiple read queries to retrieve all version components.

Rather than reshuffling and adding new registers with the missing information, we propose adding a single new register R_VERSION, which will pack all the required information.

Detailed Design

A new register will be added with the following specs and layout:

Name: R_VERSION
Address: 19
Format: U8[32]
Access: Read-only

For the sake of backwards compatibility, we will keep all the existing registers mirroring the contents of the major and minor version components in the corresponding bytes of the new R_VERSION register.

   R_VERSION

   PROTOCOL_VERSION   :protocol     3 bytes
   FIRMWARE_VERSION   :firmware     3 bytes
   HARDWARE_VERSION   :hardware     3 bytes
   SDK_IDENTIFIER     :sdk          3 bytes
   INTERFACE_HASH     :sha1         20 bytes

This gives us an exactly 32-byte register with all the relevant versioning metadata for validation. To summarize:

• PROTOCOL_VERSION: required to know synchronization guarantees and core register capabilities, e.g. devices might not synchronize correctly if not all sharing same protocol version
• FIRMWARE_VERSION: firmware and interface version will now be effectively identical and are expected to validate directly against each other. Following Use a single version number in device repositories #145 we will start enforcing joint releases of firmware and interface, with rebuilds of firmware and interface even if nothing changed
• HARDWARE_VERSION: required to know the board revision number, e.g. Behavior 1.0, 1.1, 2.0
• SDK_IDENTIFIER: which core SDK are we using (core.atxmega, core.pico, microharp, etc)
• INTERFACE_HASH: the last line of defence in validation is a SHA1 hash digest of the device.yml file which is implemented by the device firmware. This will give us a way to automatically and universally validate that the interface knows how to talk to the device, even in the case of hacks, forks and unpublished tweaks.¹²

R_TAG can still optionally encode the git revision hash, but that will remain in a separate register.

We will move to a quick 5-min approve or refuse point in the next Harp SRM, so any last comments or objections, please bring them now.

Design Meetings

• https://github.com/orgs/harp-tech/discussions/45
• https://github.com/orgs/harp-tech/discussions/115
• https://github.com/orgs/harp-tech/discussions/129
• https://github.com/orgs/harp-tech/discussions/147

Closes #44

Footnotes

1. INTERFACE_HASH will be as soon as possible incorporated into the firmware generators so at least .h files can be generated that can be used to automatically update this hash when embedding in the firmware code of both atxmega and pico cores. ↩

2. In the meantime, for backwards compatibility, when R_VERSION does not exist, or when INTERFACE_HASH is zero, this specific validation can be skipped. ↩

[bruno-f-cruz]

I think we should probably decide on what shape the document to keep track of the SDK takes. I think the what makes most sense is to add it to the who_am_i database as yet another field. All devices currently registered are either ATXMEGA or pico so it shouldn't be too hard to manually add this information without breaking things.

[glopesdev]

I think we should probably decide on what shape the document to keep track of the SDK takes.

I was thinking to possibly add it into the JSON schema directly by resolving #43.

[bruno-f-cruz]

From SRM
We need a way to track this SDK as a string;
Need a way to encode this format/parsing logic in the device.yml
Assuming that the payload is of fixed length, the conversion/cast to a type can be implemented as:

  DeviceName:
    address: 12
    type: U8
    length: 25
    interfaceType: string
    access: Write
    description: Stores the user-specified device name.

This requires:

• Updating the c# generators to handle the cast from byte[] to string
• Updating python parsers to handle the conversion

-From the point of view of the schema that defines the device.yml, we also discussed that would be important to keep a list of maintained "interfaceType" in a form of a controlled vocabulary (i.e. OneOf). It is then the responsibility of downstream tooling to maintain a mapping between "interfaceType" and converters.

• In the edge case that a downstream tool does not implement a standard converter, it is suggested to default to the raw payload
• We talked about listing controlled vocabulary of standard supported interface types, but allowing ANY string to be entered as well.

[bruno-f-cruz]

This PR is likely to undergo further modifications given discussions in https://github.com/orgs/harp-tech/discussions/129

[CLAassistant]

All committers have signed the CLA.

[glopesdev]

After consolidating feedback from multiple SRMs (see above) the current proposal will be updated to the following:

   R_VERSION

   HARDWARE_VERSION   :hardware     3 bytes
   CORE_VERSION       :core         3 bytes
   FIRMWARE_VERSION   :firmware     3 bytes
   SDK_IDENTIFIER     :architecture 3 bytes
   DEVICE_YML_HASH    :sha1         20 bytes

This gives us an exactly 32-byte register with all the relevant versioning metadata for validation. To summarize:

• HARDWARE_VERSION: required to know the board revision number, e.g. Behavior 1.0, 1.1, 2.0
• CORE_VERSION: required to know synchronization behavior and trace back core register guarantees, e.g. devices might not synchronize correctly if not all sharing same core version¹
• FIRMWARE_VERSION: firmware and interface version will now be effectively identical and are expected to validate directly against each other. Following Use a single version number in device repositories #145 we will start enforcing joint releases of firmware and interface, with rebuilds of firmware and interface even if nothing changed
• SDK_IDENTIFIER: which core SDK are we using (core.atxmega, core.pico, microharp, etc)
• DEVICE_YML_HASH: the last line of defence in validation is a SHA1 hash digest of the device.yml file which is implemented by the device firmware. This will give us a way to automatically and universally validate that the interface knows how to talk to the device, even in the case of hacks, forks and unpublished tweaks.²³

R_TAG can still optionally encode the git revision hash, but that will remain in a separate register.

We will move to a quick 5-min approve or refuse point in the next Harp SRM, so any last comments or objections, please bring them now.

Footnotes

1. We don't need a PROTOCOL_VERSION since the interface used also tells us exactly which core registers are implemented, etc. ↩

2. DEVICE_YML_HASH will be as soon as possible incorporated into the firmware generators so at least .h files can be generated that can be used to automatically update this hash when embedding in the firmware code of both atxmega and pico cores. ↩

3. In the meantime, for backwards compatibility, when R_VERSION does not exist, or when DEVICE_YML_HASH is zero, this specific validation can be skipped. ↩

[bruno-f-cruz]

@glopesdev A few notes, but otherwise I think this is making sense.

• Consider moving Core version to first position. This way the first byte could in theory be used to know what major version of the core the board implements. Not saying it will be used, but likely to be generally more useful than the verison of the hardware that will likely always be "1"

• Rename DEVICE_YML_HASH to something more generic (e.g. INTERFACE_HASH) just in case we decide to move to hash more/something else than the yml in the future for some reason.

• When making the PR we should consider making it clear when the TAG does not correspond to the hash. E.g. cases when a beta build is made but the interface does not change.

• footnote 1 - This is true assuming that the device.yml has a reference to the protocol version. I dont think this is the case yet, but I may be missing an implicit link somewhere. Can you clarify?

[glopesdev]

Consider moving Core version to first position.

I guess we could do that, but then shouldn't we also equally argue to have the firmware version first, since that is probably what we want to validate? In the end perhaps the order is not so important since we can always index whatever we want.

Perhaps another possibility to discuss is the meaning of the core version? In the end if by "core" we mean the SDK, this is just a dependency of the firmware. If we know the firmware version we must already know the core version from that so how informative would it be really? Instead, perhaps the core "interface" version is what might be most useful, so it could tell us quickly about the capabilities of the board in terms of core operations and core data logging (e.g. can it do heartbeat, tag, etc).

If we do agree that core interface registers are not really "optional", and that they really must all be present as soon as the device implements a specific core interface, then I agree this likely should be moved first.

Rename DEVICE_YML_HASH to something more generic (e.g. INTERFACE_HASH) just in case we decide to move to hash more/something else than the yml in the future for some reason.

I did try to suggest that but @PathogenDavid pointed out that there are advantages to making it very unambiguous what exactly is being hashed. If we did decide to hash something else then this would be a breaking change in validation anyway. If we have a name for what the device.yml is that is equally unambiguous we could change to that I guess, but we have always been calling it "device.yml".

When making the PR we should consider making it clear when the TAG does not correspond to the hash. E.g. cases when a beta build is made but the interface does not change.

I agree, and I would say further that we make it clear that the TAG is expected to never match this hash. This is not a hash of the firmware, or the git revision, but rather of the device.yml contents. You can have in TAG whatever you want but it is not related at all to this number and can be simply left zero.

footnote 1 - This is true assuming that the device.yml has a reference to the protocol version. I dont think this is the case yet, but I may be missing an implicit link somewhere. Can you clarify?

This was merged into the schema in #57. However, it hasn't been rolled out yet to the generators since we haven't resolved all these details, and I would like to make sure we have everything untangled and clear before doing it.

We can always say that, before the rollout, all devices simply implement the first formalised core schema (there were no formalised versions before that, and we still haven't released any other changes).

This does bring up a good point about what constitutes a "release" of the protocol repository. I think there should probably be a unified release as well, as having all these different bits and pieces versioned differently (binary protocol, device core schema, sync protocol, etc, etc) is really not helping anybody. I think we should really simplify and unify here: core_version == protocol release; device_version == device release (firmware+interface).

And then get rid of everything else since we won't be able to really keep track of it other than what we are already doing by keeping it in the repository.

[bruno-f-cruz]

I think that the core "interface" version (tho I think this is just the ProtocolVersion if I understand correctly) is far more useful than the core (sdk) version. As you point out, the interface can easily inform users of available functionality, agnostic to the underlying device family. I agree the sdk version seems largely irrelevant given a firmware version.
It should be noted that ProtocolVersion is only 2 bytes, if we go in this direction, we should consider either dropping a byte or padding to have a nice 32-sized register :P.

core_version == protocol release;

To clarify, by core_version you mean ProtocolVersion, correct? Wondering since core (sdk) version could in theory be released without corresponding protocol update.

[glopesdev]

To clarify, by core_version you mean ProtocolVersion, correct?

That's right, this is exactly protocolVersion, which is a release version of the harp-tech/protocol repository. Given that we are moving to unified full releases it probably makes the most sense to have everything be full semantic version now so I might revise the schemas in this PR as well to be consistent with this.

Thinking more about your point on the name of DEVICE_YML_HASH, I do also wonder if it isn't too restrictive to having the register specification force a specific file name. I think it might be okay to call it INTERFACE_HASH and just make it clear in the protocol spec that this is currently the hash of the device.yml file.

Given this, the proposal would be updated to the following:

   R_VERSION

   PROTOCOL_VERSION   :protocol     3 bytes
   FIRMWARE_VERSION   :firmware     3 bytes
   HARDWARE_VERSION   :hardware     3 bytes
   SDK_IDENTIFIER     :sdk          3 bytes
   INTERFACE_HASH     :sha1         20 bytes

This gives us an exactly 32-byte register with all the relevant versioning metadata for validation. To summarize:

• PROTOCOL_VERSION: required to know synchronization guarantees and core register capabilities, e.g. devices might not synchronize correctly if not all sharing same protocol version¹
• FIRMWARE_VERSION: firmware and interface version will now be effectively identical and are expected to validate directly against each other. Following Use a single version number in device repositories #145 we will start enforcing joint releases of firmware and interface, with rebuilds of firmware and interface even if nothing changed
• HARDWARE_VERSION: required to know the board revision number, e.g. Behavior 1.0, 1.1, 2.0
• SDK_IDENTIFIER: which core SDK are we using (core.atxmega, core.pico, microharp, etc)
• INTERFACE_HASH: the last line of defence in validation is a SHA1 hash digest of the device.yml file which is implemented by the device firmware. This will give us a way to automatically and universally validate that the interface knows how to talk to the device, even in the case of hacks, forks and unpublished tweaks.²³

R_TAG can still optionally encode the git revision hash, but that will remain in a separate register.

We will move to a quick 5-min approve or refuse point in the next Harp SRM, so any last comments or objections, please bring them now.

Footnotes

1. We don't need a PROTOCOL_VERSION since the interface used also tells us exactly which core registers are implemented, etc. ↩

2. INTERFACE_HASH will be as soon as possible incorporated into the firmware generators so at least .h files can be generated that can be used to automatically update this hash when embedding in the firmware code of both atxmega and pico cores. ↩

3. In the meantime, for backwards compatibility, when R_VERSION does not exist, or when INTERFACE_HASH is zero, this specific validation can be skipped. ↩

[bruno-f-cruz]

Still related to the ProtocolVersion (and some of these should probably be issues of their own):

• What triggers a new release? For the most part, from a device specification, only Major and Minor have relevance. However, patches could easily be used to soak typos and documentation changes in the repository. Should any commit to main simply trigger a bump of the release?
• Should we move the changelogs from individual files to a common change log in the root of the repository? This would make the release history much cleaner and in could even be automated from pull request.
• What should the behavior of downstream core repositories be? They will be the ones that must keep track of this version, however doesnt seem relevant to lock-step to each and every patch version.

[glopesdev]

• What triggers a new release? Should any commit to main simply trigger a bump of the release?

I would not consider a merge commit to main to be a release as it would not give us time to revise and test ourselves internally the protocol specification as a whole, and we know from experience this is an important process in assessing the quality of the specification.

For the most part, from a device specification, only Major and Minor have relevance.

This is only true if you are looking exclusively at core register specification changes. However, I am now proposing we unify releases of all contents of this repository into a single version, which will refer to all the different moving parts, including schema changes, synchronization protocol changes, etc.

As we have seen recently, some of these changes can result in subtle behavior differences, e.g. fixes being deployed to resolve clock alignment issues. These did not imply necessarily any change to the core register interface but did result in very important changes to device synchronization behavior with implications all the way to downstream data analysis.

Additionally, if we start using interface hashes as part of the contents of R_VERSION, any changes to the file, including documentation contents, will result in changes to the hash, and therefore I think it is now reasonable to simply make all these be full semantic version numbers.

• Should we move the changelogs from individual files to a common change log in the root of the repository? This would make the release history much cleaner and in could even be automated from pull request.

Yes, I was going to make a separate PR to fix that. We can decide either to pack the old changes into "releases" (versioned according to core atxmega releases for legacy compatibility) and make a change log like that, or alternatively we can leave the changelog close to the files, but with dates instead of version numbers.

• What should the behavior of downstream core repositories be? They will be the ones that must keep track of this version, however doesn't seem relevant to lock-step to each and every patch version.

Yes, this will be a decision to be made by the maintainer of each device. In order to be compliant with this spec, first of all the device needs to implement R_VERSION and the new device.json. If it doesn't, it will be considered to be implementing the legacy protocol spec (and validated accordingly).

From the moment the device implements R_VERSION, then it is up to the device and core SDK maintainers to decide which set of capabilities to commit to. I would strongly recommend that we make the core register releases strictly incremental and non-optional. What I mean by this is that all registers in a given specification must be present in the core of a device implementing a specific protocol version. For "optional" registers this would mean hard-coding the register default value in the implementation.

[bruno-f-cruz]

Probably worth describing the deprecation of the old version registers in favor of the new one. Something like:

R_CORE_VERSION_H (U8) – Major Core Version

Address: 004

Contains the major version of the Harp Protocol specification.

Warning

This register is deprecated in favor of R_VERSION. The value of this register is expected to be equal to the major version of the R_VERSION Protocol version.