This post explains how to enable SonarQube to gather test code coverage metrics of individual tests. Code coverage tools typically produce a report showing the code coverage (by line, branch, etc.) for the combined effect of all the tests executed during a given test session. This is case, for example, when you run unit tests in continuous integration. With the help of SonarQube and JaCoCo, it is possible to gather coverage metrics split at the level of the individual test case (test method in JUnit or TestNG). To enable this, there is some special configuration required that we are showing in this post.
The following process has been verified with SonarQube 4.1.2 and 4.3.2 versions, but it should work with SonarQube 3.7.x (latest LTS release), too. The application code we have used to verify the setup is the familiar Spring Pet Clinic application, enhanced to support Tomcat 7 and Spring 3 (see this post here for reference on updates needed in Pet Clinic: https://deors.wordpress.com/2012/09/06/petclinic-tomcat-7/) The code can be downloaded from GitHub in the repository: https://github.com/deors/deors.demos.petclinic
The instructions are really simple, once you’ve figured out how to connect all the dots. All that is required is to add some specific configuration to Maven Surefire plug-in (Surefire is the plug-in that is tasked with the unit test execution, and it supports JUnit and TestNG). As this specific configuration should not impact the regular unit test execution, it is recommended to include the needed configuration in a separate profile that will be executed only when the SonarQube analysis is performed. Let’s describe the required changes in the pom.xml file, section by section.
No changes are needed here. However, you should take note of any customised configuration of Maven Surefire to be sure it is also applied to the profile we are going to create. In the case of Spring Pet Clinic, this is the relevant portion of the POM we are writing down for reference:
This piece of configuration is telling Surefire to: 1) exclude the integration tests for the execution of unit tests (integration tests are covered by Surefire’s twin plug-in, Failsafe); and 2) disable the byte code verifier, preventing runtime errors when classes are instrumented (i.e. when adding mocks, or TopLink enhancements).
Again no changes are needed in this section. We just wanted to note that if your project is already leveraging JaCoCo to gather integration test coverage metrics, and is explicitly referring to JaCoCo artefact in this section, it can be left – no conflicts have been identified so far. Anyway it should not be needed here, so it’s probably safer to remove it from this section.
All the required changes come in this section. And they are very clean to add, as they all require only to add a new profile to the POM. This profile will configure a special listener for Surefire that will ensure that coverage metrics for each individual test case are appropriately gathered. To guarantee a successful test execution, we will maintain here the same configuration that appears in the build section of the POM. Finally, the profile will add a new dependency to the artefact that contains the listener code. The result is this:
<!-- calculate coverage metrics per test with SonarQube and JaCoCo -->
<!-- same configuration as in the regular test execution goal -->
<!-- plus argLine parameter configured by JaCoCo prepare-agent -->
<!-- new configuration needed for coverage per test -->
A piece of warning around the JaCoCo listener artefact version. Although it is unclear in the documentation, it seems that the best results are obtained when the JaCoCo listener version matches that of the Java plug-in installed in SonarQube. In this case, as the Java plug-in that we have installed in SonarQube is version 2.3, we have used the listener artefact version 2.3. We also tested with listener 1.2 with same good results, but to prevent any future conflict, we recommend keeping versions aligned.
Running the Analysis
Once the changes in the project configuration are done, you just need to re-execute a SonarQube analysis to see the new reports.
Depending on which SonarQube Java version you have installed, the configuration differs a bit.
Running the Analysis in Older Versions
When the Java plug-in version in use is 2.1 or an earlier version, the profile should be enabled when the analysis executes, and only when the analysis executes. This means that it is now a requirement to launch the sonar:sonar goal as a separate Maven build (it was recommended to do so, but in many cases you could execute all the targets in one run). In the case of our version of Pet Clinic:
>mvn clean verify -P cargo-tomcat,selenium-tests,jmeter-tests
>mvn sonar:sonar -P coverage-per-test
If your build is triggered by a Jenkins job, then the new profile should be added to the post-build action as can be seen in this screenshot:
Running the Analysis in Newer Versions
When the Java plug-in version in use is 2.2 or newer, code coverage is no longer executed during the analysis. Therefore you should configure the build to gather the code coverage metrics first:
>mvn clean org.jacoco:jacoco-maven-plugin:0.7.0.201403182114:prepare-agentverify -P coverage-per-test,cargo-tomcat,selenium-tests,jmeter-tests
>mvn sonar:sonar -P coverage-per-test
If your build is triggered by a Jenkins job, then the JaCoCo prepare agent goal and the new profile should be added to the build action as can be seen in this screenshot:
Once the analysis is completed, the code coverage reports get some new interesting views. When clicking on any test on the test view, a new column labelled ‘Covered Lines’ shows the individual hits for each test method in the class: When the link on Covered Lines value is followed, a new widget shows containing all the classes hit by that test method, and the touched lines per class: When the link under each of the classes is followed, a new widget appears showing the class source coloured with the actual line/branch hits:
Users can also get to this view if navigating through other views, as components or violations drill-down. Once the class level is reached, users can use the ‘Coverage’ tab to get this information:
By default, the decoration shown is ‘Lines to cover’, showing the code coverage from all tests combined. Use the drop-down list and select ‘Per test -> Covered lines’ and then select the right text case in the new drop-down list that will appear:
Measuring code coverage of individual tests is a very useful feature to have in development projects. Code coverage metrics alone may not be sufficient to identify that the rights tests are being executed and they are touching the right functionality. With the ability to identify which portions of the code are executed by any test case, developers and tester can ensure that the expected code logic is tested, versus what can be obtained with other code coverage tools that only gives a combined coverage report.