Forward Compatibility Policy

[westonpace]

What policy do we want regarding forward compatibility breaks. Note: this is different from API breaks which are covered by #5056. Also see #5100 for a discussion on backwards compatibility.

Background

I'll further divide forwards compatibility breaks into two kinds:

• (Flagged break) A new client writes a dataset using new features, as indicated by a feature flag and an older client cannot read it
• (Unflagged break) A new client writes a dataset but does not use any new features and an older client cannot read it

I don't think "Flagged break" is a problem. This is always allowed and it is the reason we have feature flags.

However, unflagged breaks can be a problem for any kind of data storage solution. Users are not able to upgrade all of their software at once. In addition, users often have less control over what clients their users (our grand-users?) are using. For example, a dataset might be written with version 0.38 and read by a client using version 0.20.

So far, we have done our best to avoid forward compatibility breaks, but this has also been lightly tested, and some of these have snuck in. We would like to increase our testing to prevent these (this is not really under discussion here).

Question

How much effort do we want to make to avoid unflagged breaks? Do we want some kind of "minimum supported client" policy?

Proposals

• All forwards compatibility breaks must be flagged (no minimum supported client)
• Clients must be less than X months old
• Clients must be less than X major versions behind

Specific Example

A specific problem that happened recently is that we started to write the version classifier (e.g. -beta.X) in the writer version field in the dataset manifest. Old clients were not able to parse that and failed. Either we give up on this change, split the proto into two fields (writer_version and writer_version_classifier), or first fix the client and then wait X months/versions before we start writing the classifiers (I'm sure there are other approaches too).

[wkalt]

* Clients must be less than X months old
* Clients must be less than X major versions behind

I can see reasons for wanting either of these. I think probably a major versions-based policy is best if we can also pair it with a committed major release schedule. That way users get a predictable time-based upgrade cycle and they also know from the version string whether two release are guaranteed compatible.

[wkalt]

How much effort do we want to make to avoid unflagged breaks? Do we want some kind of "minimum supported client" policy?

I think there are different kinds of effort to be concerned with:

• a big upgrade compatibility matrix could be a lot of effort or at least slow things down
• maybe we also carry tech debt forward and it's useful to break compatibility to relieve it.

Maybe other kinds of effort as well? I wonder how much we expect to gain from a tighter policy in the second bucket. For the first bucket, I am curious how big the difference will be in practice because for a policy of full forward-compatibility it would be untenable to maintain a full matrix of possible upgrades/downgrades, and even in the limited case we won't test all forward compatibility across releases within all majors (that would also be too many), so in some sense we need to solve that problem no matter what the policy is.

[wjones127]

we started to write the version classifier (e.g. -beta.X) in the writer version field in the dataset manifest. Old clients were not able to parse that and failed.

The thing that worries me about this particular example as well as the proposal of Clients must be less than X months old, is it seems to assume there is only ever going to be one Lance Library. For example, say someone writes a pure Java implementation. They might not have the same versioning scheme as us. That parsing logic that failed might fail on that library's version scheme too. Plus whatever run time checks are made to enforce the "Clients must be less than X months old" logic will be reading a version that isn't baked in.

If we have any sort of rule about what versions are readable, I'd prefer it was based on the format version, not a library version. The feature flags are meant to be a sort of versioning scheme, that help with compatibility checks.

[wjones127]

I'd honestly consider this code to be bug:

https://github.com/lancedb/lance/blob/e3cf56a5e0631230a76dce1efbfabe5bcf1ed44d/rust/lance-table/src/format/manifest.rs#L687-L690

The spec itself states that that version isn't not guaranteed to be present or or semantically versioned:

https://github.com/lancedb/lance/blob/30d34ec4ab3b7adf2aaabce2e8b1ac93713160ba/protos/table.proto#L57-L60

[wjones127]

I do think it's reasonable to think about what we do if we find some sort of forward-compat breakage that isn't covered by the feature flags.

We could either:

1. Simply consider it a bug in the library, that should be fixed in the next version.
2. Consider changing the spec to match the implementations in practice.

Which one we choose we should decide based on the consequences. Which will cause the least amount of breakage.

[wjones127]

We would like to increase our testing to prevent these (this is not really under discussion here).

This discussion does bring up a new idea for testing, which I think is worth discussing: compatibility with the format spec.

For example, we could create a Python program that uses the raw protobufs to generate random datasets that are spec-compliant (but don't necessarily follow the conventions in the Rust program). Then we make sure that pylance can successfully open them and scan them.