Collecting line, branch, and path coverage with PHPUnit

phpunit/phpunit allows the collection of code coverage through phpunit/php-code-coverage, which currently supports two code coverage drivers: PCOV and Xdebug.

With PCOV, you can collect line coverage; with Xdebug, you can collect line, branch, and path coverage.

The PHPUnit documentation explains line, branch, and path coverage as follows:

The Line Coverage software metric measures whether each executable line was executed.

The Branch Coverage software metric measures whether the boolean expression of each control structure evaluated to both true and false while running the test suite.

The Path Coverage software metric measures whether each of the possible execution paths in a function or method has been followed while running the test suite. An execution path is a unique sequence of branches from the entry of the function or method to its exit.

As you can see, line, branch, and path coverage provide insights into different aspects of your safety net of automated tests.

But what are the costs of running tests and collecting line, branch, and path coverage? Let's find out by running tests and collecting code coverage with PHPUnit for PHPUnit!

Preparing to run tests for PHPUnit

I assume you have installed PHP 8.1 (or PHP 8.2) with the PCOV and Xdebug extensions.

First, run the following command to clone PHPUnit:

git clone git@github.com/sebastianbergmann/phpunit.git

Second, run the following command to create a branch with the name 10.0.18 based on the tag 10.0.18:

git checkout -b 10.0.18 10.0.18

Third, run the following command to install dependencies with composer:

composer install

Now you are ready to run tests and collect code coverage with PHPUnit for PHPUnit!

Collecting line, branch, and path coverage with PHPUnit for PHPUnit

First, disable PCOV and Xdebug (if you are on macOS, take a look at Quickly switching between PCOV and Xdebug). Then run the following command to run tests for PHPUnit and take note of time and memory:

./phpunit --no-progress

On my machine, a 2021 Apple MacBook Pro with an Apple M1 Max chip, 64 GB of RAM, and running PHP 8.2.4, the output looks like this:

PHPUnit 10.0.18 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.4
Configuration: /Users/am/Sites/sebastianbergmann/phpunit/phpunit.xml

Time: 00:22.092, Memory: 38.00 MB

OK, but some tests were skipped!
Tests: 2804, Assertions: 7302, Skipped: 3.

Second, enable PCOV and run the following command to collect line coverage for PHPUnit and take note of time and memory:

./phpunit --coverage-clover=clover.xml --no-progress

On my machine, the output looks like this:

PHPUnit 10.0.18 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.4 with PCOV 1.0.11
Configuration: /Users/am/Sites/sebastianbergmann/phpunit/phpunit.xml

Time: 00:34.150, Memory: 96.00 MB

OK, but some tests were skipped!
Tests: 2804, Assertions: 7304, Skipped: 1.

Generating code coverage report in Clover XML format ... done [00:00.131]

Third, disable PCOV and enable Xdebug, then run the following command to collect line coverage for PHPUnit and take note of time and memory:

./phpunit --coverage-clover=clover.xml --no-progress

On my machine, the output looks like this:

PHPUnit 10.0.18 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.4 with Xdebug 3.2.1
Configuration: /Users/am/Sites/sebastianbergmann/phpunit/phpunit.xml

Time: 00:49.498, Memory: 90.00 MB

OK, but some tests were skipped!
Tests: 2804, Assertions: 7300, Skipped: 5.

Generating code coverage report in Clover XML format ... done [00:00.185]

Fourth, keeping Xdebug enabled, run the following command to collect line, branch, and path coverage for PHPUnit and take note of time and memory:

./phpunit --coverage-clover=clover.xml --no-progress --path-coverage

On my machine, the output looks like this:

PHPUnit 10.0.18 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.4 with Xdebug 3.2.1
Configuration: /Users/am/Sites/sebastianbergmann/phpunit/phpunit.xml

Time: 07:30.861, Memory: 451.34 MB
Time: 07:30.861, Memory: 451.34 MB

OK, but some tests were skipped!
Tests: 2804, Assertions: 7300, Skipped: 5.

Generating code coverage report in Clover XML format ... done [00:00.715]

Let's take a look at the results to understand the costs of running tests and collecting code coverage!

Costs of collecting code coverage

As mentioned before, I ran the tests on a 2021 Apple MacBook Pro with an Apple M1 Max chip, 64 GB of RAM, and running PHP 8.2.4. You will probably observe different results on another machine, but the relative results will be similar.

Code Coverage Driver Code Coverage Time (in i:s.u) Memory (in MB)
none none 00:22.092
38.00 MB
PCOV line 00:34.150
96.00 MB
Xdebug line 00:49.498
90.00 MB
Xdebug line, branch, path 07:30.861
451.34 MB

💡 As Sebastian Bergmann points out in Optimizing Your Test Suite (slide 49) , Xdebug does not use PHP's memory manager, so the memory shown by PHPUnit is incorrect when collecting code coverage with Xdebug.

You will probably observe different results, so let's compare the relative results!

Running tests for PHPUnit as a baseline

First, let's consider running tests for PHPUnit as a baseline.

If you drive the development of your PHP projects with tests (Test-Driven Development), you would never write a bit of production code without writing a failing test first. In theory, your code coverage would be 100%. Unless you wanted to report code coverage, for example, because you have external requirements or would like to display a shiny badge in your repository, you would never need to collect code coverage.

Code Coverage Driver Code Coverage Time (in i:s.u) Memory (in MB)
none none 00:22.092
100%
38.00 MB
100%
PCOV line 00:34.150
154.6%
96.00 MB
252.6%
Xdebug line 00:49.498
224.1%
90.00 MB
236.8%
Xdebug line, branch, path 07:30.861
2040.8%
451.34 MB
1187.7%

As you can see above,

  • collecting line coverage with PCOV takes 1.5 times
  • collecting line coverage with Xdebug takes 2.2 times
  • collecting line, branch, and path coverage with Xdebug takes 20.4 times

as long as running tests for PHPUnit without collecting code coverage.

Collecting line coverage for PHPUnit with PCOV as a baseline

Second, as I have yet to meet someone who consistently applies Test-Driven Development, let's consider collecting line coverage for PHPUnit with PCOV as a baseline.

Code Coverage Driver Code Coverage Time (in i:s.u) Memory (in MB)
none none 00:22.092
64.7%
38.00 MB
39.6%
PCOV line 00:34.150
100%
96.00 MB
100%
Xdebug line 00:49.498
144.9%
90.00 MB
93.8%
Xdebug line, branch, path 07:30.861
1320.2%
451.34 MB
470.1%

As you can see above,

  • collecting line coverage with Xdebug takes 1.4 times
  • collecting line, branch, and path coverage with Xdebug takes 13.2 times

as long as collecting line coverage for PHPUnit with PCOV.

Collecting line coverage for PHPUnit with Xdebug as a baseline

Third, since Xdebug also provides a step debugger and you may not want to switch back and forth between PCOV and Xdebug all the time, let's consider collecting line coverage for PHPUnit with Xdebug as a baseline.

Code Coverage Driver Code Coverage Time (in i:s.u) Memory (in MB)
none none 00:22.092
44.6%
38.00 MB
42.2%
PCOV line 00:34.150
69%
96.00 MB
106.7%
Xdebug line 00:49.498
100%
90.00 MB
100%
Xdebug line, branch, path 07:30.861
910.9%
451.34 MB
501.5%

As you can see above,

  • collecting line, branch, and path coverage with Xdebug takes 9.1 times

as long as collecting line coverage for PHPUnit with Xdebug.

Conclusion

Collecting line, branch, and path coverage is costly if you value your time.

While waiting for the line, branch, and path coverage collection to finish (07:30.861!), I zoned out and checked Instagram on my phone. Context switches indirectly affect the costs of your business.

Even if you don't value your time, external services, such as GitHub Actions, put a price on the time you use for collecting code coverage using their infrastructure (currently $0.008 per minute). The invoices you need to pay directly affect the costs of your business.

Last but not least, the consumption of computing resources indirectly affects the environment, increasing social costs that we all have to pay in different terms.

What are your arguments for collecting line, branch, and path coverage when line coverage would suffice?

Do you find this article helpful?

Do you have feedback?

Do you need help with your PHP project?