Testing

tox is used to automate testing, with the tests themselves run with pytest.

Start by installing tox and tox-envlist if you don’t already have them:

python3 -m pip install tox tox-envlist

To run tests for a specific Python version, such as Python 3.6:

tox -e py36

or, to test all Python versions:

tox -n test

Type Annotations

Type annotations are checked using mypy:

tox -e mypy

Code Quality

Run flake8 with tox to ensure your changes don’t introduce any issues:

tox -e flake8

Coverage

On POSIX systems with Firefox installed an HTML coverage report can be generated by running:

tox -n cov

Ensure your changes do not cause a significant decrease in the test coverage. If the coverage drops below the level set in repo_helper.yml (or 80% not specified) the tests will fail.

GitHub Actions

Tests are run on pushes to GitHub using GitHub Actions. You can see the results of these test at the bottom of the pull request page:

_images/github-actions-checks.png

A label will be added to the pull request automatically if the tests fail:

_images/pull-request-labels.png

This makes it easy to identify which tests are failing. Once the tests pass the label will be removed automatically.

If you are a first time contributor to a project manual approval is required for GitHub Actions to run. For more information see https://github.blog/2021-04-22-github-actions-update-helping-maintainers-combat-bad-actors/

Branch protection is used to ensure the following tests pass before merging pull requests:

  • Tests on Windows and Linux for all CPython version between 3.6 and 3.9 supported by the project.

  • Tests on Windows and Linux for PyPy 3.6, if supported by the project.

  • mypy type checking on Windows and Linux.

  • Flake8

  • The documentation check, if the project has documentation.

If the project only supports Linux the tests on Windows will not run and are not required to merge the pull request.

Tests on macOS are optional as they take longer than other platforms. CPython 3.10 and PyPy 3.7 are considered experimental and will not block a pull request from being merged if they fail. However, you should still check the results of these runs to ensure your changes have not introduced any errors there.

You should check the Files changed tab of the pull request to see whether any issues have been identified. This can be due to syntax errors in the documentation source or issues identified by flake8 and Codefactor.

Your pull request may be commented on by Coveralls to report any changes to the code coverage.