Conformance test redesign

[sudo-bmitch]

This is a redesign of the conformance tests with the following goals:

• Run as a command instead of a go test. This will allow unit testing and avoid some of the Go testing wrappers.
• Make the API testing more granular.
• Design the tests to run against different types of input data, including the various updates to image-spec.
• Minimize dependencies to avoid upstream breaking changes.
• Where possible, note unsupported APIs and fallback to alternatives.

This is still in draft for now, with this open PR for better visibility. Various remaining tasks include:

• Sparse index tests.
• Edge cases on blob pushes (out of order, chunked push with data in the final put).
• Pushing content with the wrong digest or an invalid digest algorithm.
• Verify all allowed HTTP status codes are appropriately handled.
• Cache/reuse auth tokens between HTTP requests.
• Rename conformance2 to conformance and delete the old conformance tests.
• Update the GitHub actions to test against multiple registries, and consider if/when failing 3rd party registries should block a merge to the distribution-spec.
• Generate an image in GitHub actions to be used by a conformance action.
• Update the conformance action.
• Update the oci-conformance repo to handle the new input (results will likely be summarized with two columns, API and Data, with green or gray status to show passing tests or unsupported).
• Add some unit tests.

Other wish list items:

• Add ability to run a single test.

Merging this will unblock #543. Closes #548. Closes #501. Closes #416.

[tianon]

These tests appear to be conflating two concepts 😇

For lack of an officially spec-defined term, blobs whose descriptors include the urls key are colloquially known as "foreign layers", and technically any blob could be foreign (with zero change to the mediaType), although most registries probably validate config differently from layers in that respect (I know Docker Hub certainly does 😇).

Non-distributable layers (those with mediaType of application/vnd.oci.image.layer.nondistributable.v1.tar*), on the other hand, may or may not be explicitly "foreign" (ie, they may or may not have urls set).

• https://github.com/opencontainers/image-spec/blob/v1.1.1/descriptor.md#:~:text=This%20OPTIONAL%20property%20specifies%20a%20list%20of%20URIs%20from%20which%20this%20object%20MAY%20be%20downloaded.

  This OPTIONAL property specifies a list of URIs from which this object MAY be downloaded.

• https://github.com/opencontainers/image-spec/blob/v1.1.1/media-types.md#:~:text=The%20following%20media%20types%20identify%20a%20%22Layer%22%20with%20distribution%20restrictions

• https://github.com/opencontainers/image-spec/blob/v1.1.1/layer.md#non-distributable-layers

  Descriptors referencing non-distributable layers MAY include urls for downloading these layers directly; however, the presence of the urls field SHOULD NOT be used to determine whether or not a layer is non-distributable.

(For historical curiosity, this evolved/matured/split in two parts from the application/vnd.docker.image.rootfs.foreign.diff.tar.gzip media type in the previous "Docker Image Manifest Version 2, Schema 2" spec: https://github.com/distribution/distribution/blob/10e4312e62ab3b0c53bc0e31ade82fbe38af1b43/docs/content/spec/manifest-v2-2.md#:~:text=application/vnd.docker.image.rootfs.foreign.diff.tar.gzip%3A%20%22Layer%22%2C%20as%20a%20gzipped%20tar%20that%20should%20never%20be%20pushed -- this is where I've cribbed the term "foreign layer" from, because "blob/descriptor with urls set" is a mouthful.)