pytest-fastest¶
pytest-fastest uses coverage data and Git to run just the tests you need.
We all want to write well-tested code, but in large projects that can mean running thousands of tests that can take a lot of time. pytest-fastest identifies tests:
- That test modules that have changed in Git,
- That we don’t already have coverage data for, and
- That we’ve added or changed.
Usage¶
Most large projects usually work off a common Git branch such as dev
or master
. When a developer wants to add a feature or bugfix, they create a new branch to do their work. After making some changes, they run tests to make sure everything still works, fix a few things, run more tests, fix things, and so on until they’re happy with it. However, that “run tests” part can slow things down, as either lots of irrelevant of tests add time and screen clutter,or the developer manually figures out what really needs to be checked.
pytest-fastest automates that processes. It takes a parameter naming a Git commit, fastest-commit
, and compares the current state of the developer’s work to that commit. It then builds a coverage map tracking which of the project’s files have been accessed by each test function. In future runs with the appropriate fastest-mode
set, it compares that coverage map to the git diff
from fastest-commit to see which tests actually need to be executed - the rest are skipped.
Example¶
First, we clone a large Git repo, create a new branch, and run the full unit test suite to gather coverage data:
$ git clone myrepo
$ git checkout dev
$ git checkout -b feature/newstuff
$ pytest tests/unit --fastest-mode=skip --fastest-commit=dev
...
============================ 569 passed in 19.76 seconds ============================
Now let’s run the tests again:
$ pytest tests/unit --fastest-mode=skip --fastest-commit=dev
...
============================ 569 skipped in 2.83 seconds ============================
Whoa! Magic! Now we modify a module in the package and run the tests:
$ vi src/somefile.py
$ pytest tests/unit --fastest-mode=skip --fastest-commit=dev
...
======================= 6 passed, 563 skipped in 3.03 seconds =======================
Only tests that refer to somefile.py
were run, but everything else was skipped. Now let’s add a new test:
$ vi tests/unit/test_module.py
$ pytest tests/unit --fastest-mode=skip --fastest-commit=dev
...
======================= 7 passed, 563 skipped in 2.98 seconds =======================
Our testing time has dropped from nearly twenty seconds to less than three, and we didn’t have to do any work to figure out which tests needed to be run.
Modes¶
pytest-fastest runs in one of several modes:
all
(default): Run all tests without collecting coverage data. This emulates normal pytest behavior and has no effect on performance.skip
: Skip tests that don’t need to be run, but update coverage data on the ones that do run. This is the mode you will use most often in support of the workflow described above. It will usually be faster thanall
, but because collecting coverage information takes some time, as the number of un-skippable tests grows very large it may actually become slower.gather
: Don’t skip any tests, but do gather coverage data. This is slower thanall
but can be used to seed the coverage cache.cache
: This is a fast mode for fixing existing tests as it skips tests but doesn’t update the coverage cache. It will never be slower thanall
and will always be faster thanskip
. However, it might not pick up subtle changes you make to tests’ call chains and could accidentally skip tests that the more conservativeskip
mode would notice.
Configuration¶
While --fastest-commit
may be given from the command line, most projects always make feature branches from the same common dev
or master
branch. In those cases, it’s probably easier to add a setting like:
[pytest]
fastest_commit = dev
to your pytest.ini. You can still override this with --fastest-commit
if needed.
Limitations¶
Only call graphs are examined. If module_a
imports only a constant from module_b
, and you edit that constant, then pytest-fastest won’t notice the change. This could lead to surprises. Running with --fastest-mode=all
(or gather
) will run all tests that pytest would normally run, though.
Code changes are tracked at the module level, not the function level. If you modify module_a
, then any tests that access any functions in module_a
will run. The main complication is that it’s fairly difficult to accurately parse git diff
’s output to see exactly what’s changed. Future versions may address this.
Git is the only SCM tool currently supported, although the design supports adding others.
Notes¶
pytest-fastest stores its cached coverage data in a file named .fastest.coverage
in the pytest rootdir.
History¶
v0.0.11, 2023-06-15: Dependency version bumps. Correct restoration of sys.settrace after completion.
v0.0.10, 2020-03-05: No code changes, just dependency version bumps.
v0.0.9, 2019-04-05: Update the version of Requests. Switched to Poetry for packaging.
v0.0.8, 2018-08-27: Updated import paths for new versions of pytest.
v0.0.7: First generally usable release.