| commit | 3572449e2c57d6329673696e8ea6b3a1bcba71e0 | [log] [tgz] |
|---|---|---|
| author | Brian Sheedy <[email protected]> | Thu Sep 25 16:13:04 2025 |
| committer | Copybara-Service <[email protected]> | Tue Nov 04 22:24:50 2025 |
| tree | 97633267ecdb9519105aeb99912a3449288df9cf | |
| parent | 124fe1b0455f5e3a5d9f51cdec339a4ff545f58b [diff] |
[3/?] Move WorkDir to new file This is part of a chain of CLs to add support for multiple parallel workers to //agents/testing/eval_prompts.py. This CL moves the WorkDir class into a new workers.py file. This does not currently provide any real benefit other than making eval_prompts.py less monolithic, but additional worker-related code will be added to this file in follow-up CLs. Also performs two drive-by improvements: 1. Makes the result thread daemonic so that it does not block the Python process from exiting. 2. Updates the moved tests for WorkDir to share mock setup between tests instead of using repeated mock.patch annotations. Bug: 445459870 Change-Id: I37562b2e31577972ec61d8e24dd6e2309aa773ab Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6981947 Auto-Submit: Brian Sheedy <[email protected]> Commit-Queue: Struan Shrimpton <[email protected]> Reviewed-by: Struan Shrimpton <[email protected]> Cr-Commit-Position: refs/heads/main@{#1520664} GitOrigin-RevId: 154dfd9c0e22f8a369b125d98dea5ce219d4b1ff
This directory contains an experimental script for running prompt evaluation tests on extensions and prompts under //agents. It currently only works locally and will make temporary changes to your Chromium repo.
Existing tests can be run via the //agents/testing/eval_prompts.py script. It should handle everything automatically, although it is advised to commit any changes before running this script. It will automatically retrieve a temporary copy of promptfoo, perform repo setup, run configured tests, and perform teardown.
By default, it will build promptfoo from ToT, but specific behavior can be configured via command line arguments, including use of stable releases via npm which will likely result in faster setup.
If you are running eval_prompts.py on a system without a container runtime like Docker or Podman, you will need to pass the --no-sandbox flag. This is because the script uses sandboxing by default to isolate the test environment.
The prompt eval is intended to be run with Chromium in a btrfs file system. The tests should still run in a normal checkout but will be significantly slower and take up significantly more disk space. These steps can be used to fetch a new Chromium solution in a virtual btrfs file system mounted in your home dir.
The following commands can be used to set up the environment:
# 1. Ensure btrfs is installed sudo apt install btrfs-progs # 2. Create the virtual image file truncate -s 500G ~/btrfs_virtual_disk.img # 3. Format the image with btrfs mkfs.btrfs ~/btrfs_virtual_disk.img # 4. Mount the image mkdir ~/btrfs sudo mount -o loop ~/btrfs_virtual_disk.img ~/btrfs # 5. Update owner sudo chown $(whoami):$(id -ng) ~/btrfs # 6. Create a btrfs subvolume for the checkout btrfs subvolume create ~/btrfs/chromium # 7. Fetch a new Chromium checkout into the subvolume. # This will place the 'src' directory inside '~/btrfs/chromium/'. cd ~/btrfs/chromium fetch chromium # For an existing checkout, you would instead move the contents, e.g.: # mv ~/your_old_chromium/* ~/btrfs/chromium/
After Chromium is checked out, agents/testing/eval_prompts.py can then be run from ~/btrfs/chromium/src/.
The script only installs the extensions in the EXTENSIONS_TO_INSTALL list at the top of the file. If an extension should be present for testing, add the extension name to this list.
Each independent test case should have its own promptfoo yaml config file. See the promptfoo documentation for more information on this. If multiple prompts are expected to result in the same behavior, and thus can be tested in the same way, the config file can contain multiple prompts. promptfoo will automatically test each prompt individually.
Config files should be placed in a tests/promptfoo/ subdirectory of the relevant prompt or extension directory. After they exist on disk, new yaml files will need to be added to the PROMPTFOO_CONFIG_COMPONENTS list at the top of the script for the tests to actually be run.
The gemini_provider.py supports several custom options for advanced testing scenarios, such as applying file changes or loading specific templates. Below is an example of a promptfoo.yaml file that demonstrates how to use the changes option to patch and stage files before a test prompt is run.
This example can be used as a template for writing tests that require a specific file state.
custom_options.promptfoo.yamlprompts: - "What is the staged content of the file `path/to/dummy.txt`?" providers: - id: "python:../../../testing/gemini_provider.py" config: extensions: - depot_tools changes: - apply: "path/to/add_dummy_content.patch" - stage: "path/to/dummy.txt" tests: - description: "Test with custom options" assert: # Check that the agent ran git diff and found the new content. - type: icontains value: "dummy content"
The changes field points to standard .patch files. The test runner will apply them.
add_dummy_content.patchdiff --git a/path/to/dummy.txt b/path/to/dummy.txt index e69de29..27332d3 100644 --- a/path/to/dummy.txt +++ b/path/to/dummy.txt @@ -0,0 +1 @@ +dummy content