Adding web-features metadata

[jugglinmike]

In gh-4567, we discussed the potential of adding a new layer of test classification information to Test262. After receiving a positive signal there, I'm opening this issue to facilitate a technical design process. I'll start with the design goals that I've identified based on the needs of the initial consumers of this data, and I'll follow that with a concrete proposal which I believe satisfies those goals.

Design Goals

There are two high-level design goals: generalizabilty and power.

The design should be generalizable so that it is recognizable to similar projects that support web-features (e.g. the web-platform-tests project--RCF here). That will allow folks with shared interests (e.g. maintainers of web-status data and the audience of the projects which consume that data) to recognize the pattern across projects. This should promote voluntary contributions, and it will also allow folks external to Test262 to build shared tooling and write common documentation. Practically speaking, this means that (1) the solution should have parity with syntax and semantics established within WPT, and (2) any extensions to that syntax should conceivably find application beyond Test262.

The design should be powerful so that the classifications are both correct and maintainable. This will further encourage volunteer contributions and limit the amount of maintenance churn as Test262 evolves and grows. Practically speaking, this means that the design should be (1) adaptable enough to leverage Test262's existing classification system, (2) granular enough to match subsets of directories, and (3) dynamic enough to limit brittleness (e.g. avoiding static lists of filenames).

Proposal

We'll place classification information in YAML-formatted files named WEB_FEATURES.yml. Those files will satisfy the following CDDL schema:

WebFeatures = {
  features: [*WebFeature],
}

WebFeature = {
  name: text,
  WebFeaturePattern,
}

WebFeaturePattern = (
  WebFeatureFilePattern /
  WebFeatureTagPattern
)

WebFeatureFilePattern = (
  files: [*TextPattern] / "**"
)

WebFeatureTagPattern = (
  tags: [*TextPattern]
)

TextPattern = text .regexp "!?[A-Za-z0-9_.-]+"

The WebFeatureFilePattern would be used to match the names of files which are present in the same directory of the YAML file. The string value ”**” would match all files in the current directory and all subdirectories.

The WebFeatureTagPattern would be used to match features frontmatter tag values in each test. It would apply to files which are present in the current directory and in all subdirectories.

File patterns don't apply recursively for parity with the semantics in WPT; tag patterns apply recursively so they can more succinctly match tree-spanning tags like change-array-by-copy.

The semantics of both patterns would be similar to .gitignore files: the * character acts as a "glob" and a leading ! denotes negation.

A few practical examples:

• Contents of file test/WEB_FEATURES.yml:

  features:
  - name: array-at
    tags:
    - Array.prototype.at
    - TypedArray.prototype.at

• Contents of file test/built-ins/Array/prototype/copyWithin/WEB_FEATURES.yml:

  features:
  - name: array-copywithin
    files:
    - "*"

• Contents of file test/language/expressions/object/method-definition/WEB_FEATURES.yml:

  features:
  - name: async-functions
    tags:
    - async-functions
    - "!class-methods-private"

• Contents of file test/language/expressions/await/WEB_FEATURES.yml:

  features:
  - name: async-functions
    tags:
    - "!async-iteration"

Does this design seem necessary and sufficient for the task at hand?

[ljharb]

I would strongly prefer "anything but yaml".

Since conceptual features don't always have all tests in the same directory, it seems like it'd be easier to just have a single file at the root, that's a map of conceptual features to a list of globs, rather than scattering files all over the repo. (in general, this is also the trend for config files - eslint has moved away from multiple nested config files, into a single flat one at the root, using globs to target overrides, for example)

Generalized similarity with web "stuff" is a good goal, but we shouldn't blindly cargo-cult whatever patterns they've chosen, including any particular schemas or file patterns.

[ptomato]

I don't have a problem with YAML but I agree with Jordan's preference for a single config file.

[jugglinmike]

Following discussion here on this thread and in the Test262 maintainers' meeting (thanks, all!), here's a second draft. At a high level:

• All classifications in a single YAML-formatted file at the root of the repository: /WEB_FEATURES.yml
• files patterns are executed against full paths (rather than file names); when they match a directory, then that directory's contents are included recursively (to include contents non-recusively, use a trailing "glob", like directory-name/*)
• files patterns and tags patterns can be specified in tandem; they are applied in conjunction rather than disjunction

@ljharb I hear your dissatisfaction with YAML, and we did our best to speculate about your rationale. The other maintainers and I are hoping that we can find a middle ground by limiting this proposal to a single file and by restricting that file's syntax to a limited subset of the (e.g. arrays, objects, strings, and comments).

CDDL schema:

WebFeatures = {
  features: [*WebFeature],
}

WebFeature = {
  name: text,
  WebFeaturePattern,
}

WebFeaturePattern = (
  { WebFeatuerFilePattern } /
  { WebFeatureTagPattern } /
  {
    WebFeatureFilePattern,
    WebFeatureTagPattern,
  }
)

WebFeatureFilePattern = (
  files: [*TextPattern]
)

WebFeatureTagPattern = (
  tags: [*TextPattern]
)

TextPattern = text .regexp "!?[A-Za-z0-9_*.-]+"

Example classifications for the web features named "array-at" and "async-generators":

features:
- name: array-at
  files:
  - test/built-ins
  keys:
  - Array.prototype.at
  - TypedArray.prototype.at

# (+ more entries)

- name: async-generators
  # (all files in these directories should be included)
  files:
  - test/language/expressions/async-generator
  - test/language/expressions/class/async-gen-method
  - test/language/expressions/class/async-gen-method-static
  - test/language/statements/async-generator
  - test/language/statements/class/async-gen-method
  - test/language/statements/class/async-gen-method-static
- name: async-generators
  # (only some files in these directories should be included--those that
  # satisfy some frontmatter-flag rules)
  files:
  - test/language/expressions/await
  - test/language/block-scope/syntax/redeclaration
  - test/language/expressions/class/dstr
  - test/language/expressions/object/dstr
  - test/language/expressions/object/method-definition
  - test/language/expressions/class/elements/syntax/valid
  - test/language/expressions/class/elements/syntax/early-errors
  tags:
  - async-iteration
  - "!class-methods-private"
  - "!class-static-methods-private"

# (+ more entries)

How does this version look to you folks?

[ljharb]

Sorry I couldn't make the meeting to discuss it. It's not a dealbreaker, but I'd rather use toml over yaml if we can :-)