38  Code Coverage: Python

38.1 Calculating coverage in Python

We use the plugin tool pytest-cov to do this.

Install as a package via conda:

conda install pytest-cov

add it as a package development dependency with poetry:

poetry add --group dev pytest-cov

38.1.1 Calculating coverage in Python

To calculate line coverage and print it to the terminal:

pytest --cov=<directory>

To calculate line coverage and print it to the terminal:

pytest --cov-branch --cov=<directory>

38.1.2 How does coverage in Python actually count line coverage?

  • the output from poetry run pytest --cov=src gives a table that looks like this:
---------- coverage: platform darwin, python 3.7.6-final-0 -----------
Name                  Stmts   Miss  Cover
-----------------------------------------
big_abs/big_abs.py        8      2    75%
-----------------------------------------
TOTAL                     9      2    78%

In the column labelled as “Stmts”, coverage is calculating all possible line jumps that could have been executed (these line jumps are sometimes called arcs). This is essentially covered + uncovered lines of code.

Note

This leads coverage to count two statements on one line that are separated by a “;” (e.g., print(“hello”); print(“there”)) as one statement, as well as calculating a single statement that is spread across two lines as one statement.

In the column labelled as “Miss”, this is the number of line jumps not executed by the tests. So our covered lines of code is “Stmts” - “Miss”.

The coverage percentage in this scenario is calculated by: \[Coverage = \frac{(Stmts - Miss)}{Stmts}\] \[Coverage = \frac{8 - 2}{8} * 100 = 75\%\]

38.1.3 How does coverage in Python actually branch coverage?

  • the output from poetry run pytest --cov-branch --cov=src gives a table that looks like this:
---------- coverage: platform darwin, python 3.7.6-final-0 -----------
Name                  Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------
big_abs/big_abs.py        8      2      6      3    64%
-------------------------------------------------------
TOTAL                     9      2      6      3    67%

In the column labelled as “Branch”, coverage is actually counting the number of possible jumps from branch points. This is essentially covered + uncovered branches of code.

Note

Because coverage is using line jumps to count branches, each if inherently has an else even if its not explicitly written in the code.

In the column labelled as “BrPart”, this is the number of of possible jumps from branch points executed by the tests. This is essentially our covered branches of code.

The branch coverage percentage in this tool is calculated by:

\[Coverage = \frac{(Stmts\:executed + BrPart)}{(Stmts + Branch)}\]

\[Coverage = \frac{((Stmts - Miss) + BrPart)}{(Stmts + Branch)}\]

Note

You can see this formula actually includes both line and branch coverage in this calculation.

So for big_abs/big_abs.py 64% was calculated from: \[Coverage = \frac{((8 - 2) + 3)}{(8 + 6)} * 100 = 64\%\]

38.2 Using our toy package example

Let’s go back to our previous packaging function and tests. For reference this was our function

def count_char(input_string):
    if not isinstance(input_string, str):
        raise TypeError(f"Expected input to be str, got {type(input_string)}")
    len(input_string)

and the tests we wrote

import pytest

from countchar.count_char import count_char


@pytest.fixture
def text_data():
    file_path = "tests/text.txt"
    with open(file_path, 'r') as file:
        lines = file.readlines()
    return lines


def test_line_1(text_data):
    string = text_data[0]
    assert count_char(string) == 34

def test_line_2(text_data):
    string = text_data[1]
    assert count_char(string) == 10

def test_line_3(text_data):
    string = text_data[2]
    assert count_char(string) == 8

We can now look at the line and branch coverage of our current package files

pytest tests/ --cov=countchar # line coverage
pytest --cov=countchar --cov-branch # branch coverage

and generate the document for a coverage report

pytest --cov=countchar --cov-report html

38.3 Reporting coverage badges

https://codecov.io/ is a service that can take our pytest information and give us a visualization of our test coverage and also give us a badge we can display in our repository.

More detail about creating and linking codecov to our repositor is in this section of the textbook: https://py-pkgs.org/08-ci-cd#recording-code-coverage

General steps:

  • log into https://codecov.io/ with your GitHub account
  • install code coverage github app (if you’re using personal account, your organization may have it already installed)
    • Make sure the application can see your repository in the app settings
  • find repo on codecov
  • click configure
  • setup code cov action
    • use github actions
    • select pytest
    • change --cov-report=xml
      • pytest --cov --cov-branch --cov-report=xml
    • get token info for CODECOV_TOKEN (from configure page)
    • when CI runs you will see a link in that particular step results of the coverage with a URL (else you probably didn’t have a token)
    • refresh the codecoverage repo page
  • badge
    • configuration page there’s a badge + graphs section
    • copy the markdown badge (we are using quarto + quartodoc, rst is for sphinx)