Protocol Conformance Test Suite

[ThisIsMissEm]

In order to allow others to create implementations of the FIRES protocol, we need a conformance test suite that can prove that another implementation has the same output.

The key things are:

• should allow configuration with a config file
• should have two modes:
  • read-only shape validation
  • read-write validation
• write validation needs to have a way to add hooks to call out to run arbitrary code to trigger updates on the server under test, since there's no protocol-level API for writing to datasets or labels yet, and this is most likely to vary between implementations.
• the configuration should probably include the ability to test different "features", e.g., you can test just labels or just datasets, since those two features can exist independently of each other.

The imagined way of running these tests would be something like:

$ npx @fedimod/fires-conformance -c config.ts -s https://server-under-test.example

Where:

• -c or --config specifies the configuration file to load (this contains the hooks)
• -s or --server specifies the server to test
• -m or --mode which is either read-only or read-write

Hooks

The hooks that need to exist for write validation (and some read validation) would be something like:

• authenticate(dataset: URL, context: TestContext): Promise<RequestInit>
• createLabel(name: string, properties: LabelProperties, context: TestContext): Promise<URL>
• createDataset(name: string, properties: DatasetProperties, context: TestContext): Promise<URL>
• createDatasetChange(dataset: URL, change: CreateDatasetChange, context: TestContext): Promise<URL>
• deleteLabel(label: URL, context: TestContext): Promise<void> (see Implement soft-deletes for datasets & labels #231 )
• deleteDataset(dataset: URL, context: TestContext): Promise<void> (see Implement soft-deletes for datasets & labels #231)

There is also recommended_filters that will exist in CreateDatasetChange eventually, but the reference server doesn't yet implement filters, so it's not currently present in the input, but would be a list of URIs.

The DatasetProperties could include summary, content, locale properties.

The LabelProperties could include summary, content, locale (BCP-47 string), deprecated (boolean), translations as an array of objects which would include locale, name, summary and content properties.

The hooks should expose a type that allows for a satisfies typescript constrain to be written.

Docker

We may ship a docker image for the conformance test suite runner.

[ThisIsMissEm]

cc @StevenLangbroek I think this is what we discussed yesterday?