Docs/Testing.md

# Testing astcenc

The repository contains a small suite of tests which can be used to sanity
check source code changes to the compressor. It must be noted that this test
suite is relatively limited in scope and does not cover every feature or
bitrate of the standard.

# Required software

Running the tests requires Python 3.7 to be installed on the host machine, and
an `astcenc-avx2` release build to have been previously compiled and installed
into an directory called `astcenc` in the root of the git checkout. This
can be achieved by configuring the CMake build using the install prefix
`-DCMAKE_INSTALL_PREFIX=../` and then running a build with the `install` build
target.

# Running C++ unit tests

We support a small (but growing) number of C++ unit tests, which are written
using the `googletest` framework and integrated in the CMake "CTest" test
framework.

To build unit tests pull the `googletest` git submodule and add `-DUNITTEST=ON`
to the CMake command line when configuring.

To run unit tests use the CMake `ctest` utility from your build directory after
you have built the tests.

```shell
cd build
ctest --verbose
```

# Running command line tests

To run the command line tests, which aim to get coverage of the command line
options and core codec stability without testing the compression quality
itself, run the command line:

    python3 -m unittest discover -s Test -p astc_test*.py -v

# Running image tests

To run the image test suite run the following command from the root directory
of the repository:

    python3 ./Test/astc_test_image.py

This will run though a series of image compression tests, comparing the image
PSNR against a set of reference results from the last stable baseline. The test
will fail if any reduction in PSNR above a set threshold is detected. Note that
performance information is reported, but regressions will not flag a failure.

For debug purposes, all decompressed test output images and result CSV files
are stored in the `TestOutput` directory, using the same test set structure as
the `Test/Images` folder.

## Test selection

The runner supports a number of options to filter down what is run, enabling
developers to focus local testing on the parts of the code they are working on.

* `--encoder` selects which encoder to run. By default the `avx2` encoder is
  selected. Note that some out-of-tree reference encoders (older encoders, and
  some third-party encoders) are supported for comparison purposes. These will
  not work without the binaries being manually provided; they are not
  distributed here.
* `--test-set` selects which image set to run. By default the `Small` image
  test set is selected, which aims to provide basic coverage of many different
  color formats and color profiles.
* `--block-size` selects which block size to run. By default a range of
  block sizes (2D and 3D) are used.
* `--color-profile` selects which color profiles from the standard should be
  used (LDR, LDR sRGB, or HDR) to select images. By default all are selected.
* `--color-format` selects which color formats should be used (L, XY, RGB,
  RGBA) to select images. By default all are selected.

## Performance tests

To provide less noisy performance results the test suite supports compressing
each image multiple times and returning the best measured performance. To
enable this mode use the following options:

* `--repeats <M>` : Run M test compression passes which are timed.

**Note:**  The reference CSV contains performance results measured on an Intel
Core i5 9600K running at 4.3GHz, running each test 5 times.

## Updating reference data

The reference PSNR and performance scores are stored in CSVs committed to the
repository. This data is created by running the tests using the last stable
release on a standard test machine we use for performance testing builds.

It can be useful for developers to rebuild the reference results for their
local machine, in particular for measuring performance improvements. To build
new reference CSVs, download the current reference `astcenc` binary (1.7) from
GitHub for your host OS and place it in to the `./Binaries/1.7/` directory.
Once this is done, run the command:

    python3 ./Test/astc_test_image.py --encoder 1.7 --test-set all --repeats 5

... to regenerate the reference CSV files.

**WARNING:** This can take some hours to complete, and it is best done when the
test suite gets exclusive use of the machine to avoid other processing slowing
down the compression and disturbing the performance data. It is recommended to
shutdown or disable any background applications that are running.

## Valgrind memcheck

It is always worth running the Valgrind memcheck tool to validate that we have
not introduced any obvious memory errors. Build a release build with symbols
information with `-DCMAKE_BUILD_TYPE=RelWithDebInfo` and then run:

    valgrind --tool=memcheck --track-origins=yes <command>