Never retry tearDownAll (#2815)

A retried `tearDownAll` didn't run any of its callbacks again.
`Invoker.runTearDowns` removes each callback from the declarer's list
before running it, so by the time a retry started the list was empty. The
retry always passed, which hid the original failure.

Give the synthetic `(tearDownAll)` test `retry: 0` so a failure fails the
run, whatever `retry` the group, suite, or tags configure.
Platform-specific metadata, such as `on_platform` in `dart_test.yaml`, is
merged into the test's metadata later by `forPlatform`, so clear the retry
in those entries too. `setUpAll` still retries.

We chose not to make `tearDownAll` retries work. Re-running cleanup
correctly means tracking which callbacks failed, including ones that time
out or fail asynchronously, and the complexity cost is higher than the
utility. Teardown callbacks are often not safe to run twice anyway. A
`tearDownAll` callback that can fail on flaky cleanup should handle that
failure itself, for example by retrying within the callback.

This only surfaces failures that a retry was already hiding.

Fixes #2800
5 files changed
tree: e15fa49073cc337a02b8c1c764493539d4d74aa1
  1. .github/
  2. integration_tests/
  3. pkgs/
  4. tool/
  5. .gitattributes
  6. .gitignore
  7. analysis_options.yaml
  8. CODEOWNERS
  9. CONTRIBUTING.md
  10. mono_repo.yaml
  11. pubspec.yaml
  12. README.md
README.md

Dart CI OpenSSF Scorecard

What's here?

Welcome! package:test is the standard testing library for Dart and Flutter. If you have questions about Dart testing, please see the docs for package:test. package:test_api and package:test_core are implementation details and generally not user-facing.

package:checks is a relatively new library for expressing test expectations. It's a more modern version of package:matcher and features a literate API.

Packages

PackageDescriptionIssuesVersion
checksA framework for checking values against expectations and building custom expectations.issuespub package
fake_asyncFake asynchronous events such as timers and microtasks for deterministic testing.issuespub package
matcherSupport for specifying test expectations via an extensible Matcher class. Also includes a number of built-in Matcher implementations for common cases.issuespub package
testA full featured library for writing and running Dart tests across platforms.issuespub package
test_apiThe user facing API for structuring Dart tests and checking expectations.pub package
test_coreA basic library for writing tests and running them on the VM.pub package
test_descriptorAn API for defining and verifying files and directory structures.issuespub package
test_processTest processes: starting; validating stdout and stderr; checking exit code.issuespub package