Web Features in Test262

The WEB_FEATURES.yml file in the root of this repository represents a collaboration with the WebDX W3C Communuity Group to make Test262 more legible to web developers.

The file defines a mapping between Test262 tests and “features” as perceived by web developers and as defined in the web-features project by the WebDX Community Group.

For more on the motivation, please see GitHub issue #4567.

File Structure

CDDL schema:

WebFeatures = {
  features: [*WebFeature],
}

WebFeature = {
  name: text,
  files: [*FilePattern],
  ?tags: [*TagPattern],
)

FilePattern = text .regexp "!?[A-Za-z0-9_*./-]+"
TagPattern = text .regexp "!?[A-Za-z0-9_.-]+"

FilePattern entries describe test files and directories that should be included in a given web feature. Then they match a directory name, then every file in that directory and its subdirectories should be included. When the asterisk character (*) appears in these entires, it will be interpreted as a “wildcard” (or “glob”) that can be satisfied by zero or more characters of any kind.

TagPattern entries describe features metadata tags as defined in the frontmatter of Test262 test files. They are applied as the conjunction of test files matched with FilePattern, serving as a means to refine expansive file patterns.

In both types of pattern, a leading exclamation mark character (!) denotes negation. Any file that would be included according to the patterns that precede a negated pattern should be omitted from the web feature.

A few practical examples:

features:

  # Every test in the `test/built-ins/Promise` directory and in any of seven of
  # the named sub-directories.
  - name: promise
    files:
      - test/built-ins/Promise/*
      - test/built-ins/Promise/Symbol.species
      - test/built-ins/Promise/all
      - test/built-ins/Promise/prototype/catch
      - test/built-ins/Promise/prototype/then
      - test/built-ins/Promise/race
      - test/built-ins/Promise/reject
      - test/built-ins/Promise/resolve

  # Any test in the `test/language/expressions/object/method-definition`
  # directory (and its subdirectories) which bear the frontmatter tag
  # `async-functions` and which do NOT bear the frontmatter tag
  # `class-methods-private`
  - name: async-functions
    files:
    - test/language/expressions/object/method-definition
    tags:
    - async-functions
    - "!class-methods-private"

For details on the design considerations, please see GitHub issue #4571.

Tooling

This directory defines two command-line utilities in the form of executable Python scripts. Their dependencies can be installed with the following command (executed from the root of this repository):

pip install -r tools/web-features/requirements.txt

lint.py

This script validates basic expectations about the contents of the WEB_FEATURES.yml (e.g. that the file conforms to the schema and that every pattern impacts the set of matched test files). If invoked with the argument --manifest, this utility will write a “manifest” of all web-features and their associated test file names.

check-coverage.py

This script compares two web-features manifests to identify changes that likely indicate an unintentional regression. Specifically, any reduction in the number of tests associated with a web-feature will produce an error.