Measure Code Coverage of HtmlUnit Based Tests with Sonar and JaCoCo

This blog post is the third one in a series about Integration Tests with HtmlUnit. The first post, titled “Automating Assembly and Integration Tests with HtmlUnit”, showed how to write integration tests of web UI applications using HtmlUnit. That post can be read here. The second post, titled “Integrate HtmlUnit Based Tests with Apache Maven and Cargo”, showed how to automate the execution of HtmlUnit tests using Maven and Cargo plug-in. That post can be read here.

Finally, in this post we are going to show how to measure code coverage of HtmlUnit tests using Sonar, the popular Continuous Quality Assurance tool, and JaCoCo, a very interesting code coverage tool based on JVM agents instead of instrumenting bytecodes.

The Starting Point

Let’s start from where we ended in previous post. We have a web UI application with an integration test written with HtmlUnit. The web UI application build lifecycle is managed with Maven. We have configured the Maven Failsafe plug-in to run the integration test only, and the Maven Cargo plug-in to automatically provide a runtime environment to execute the integration test.

JaCoCo and Sonar

JaCoCo is a relatively new code coverage tool for Java. While other popular tools as Cobertura or Clover are based on code instrumentation (Cobertura instruments binaries while Clover instruments sources), JaCoCo is built as JVM agent. There is no need to instrument artifacts and deploy those instrumented artifacts to the runtime environment.

This approach has some clear advantages. It is possible to instrument the code and run tests over the instrumented artifact, but it requires heavy Maven tweaking. With JaCoCo, the only requirement is to add the agent to command line of the target runtime. This approach is valid for unit tests, integration tests and integration tests on a server. The only different is that the agent should be added to either the Surefire/Failsafe command-line or to the target application server startup command-line.

Other advantage is that with JaCoCo there are no issues with coverage information not being saved on application server shutdown as happens with Cobertura.

Sonar can leverage JaCoCo or unit and for integration tests code coverage. And as of version 2.11, JaCoCo is the only code coverage tool supported for integration tests code coverage inside Sonar.

This post was written using Sonar 2.9  JaCoCo plug-in 0.6 and JaCoCo 0.5.3.

First of all, it is needed to install JaCoCo plug-in in Sonar. This can be easily done through Sonar’s update center. Once the plug-in is selected and downloaded, restart Sonar to enable it.

Second step is to add JaCoCo widget to the project dashboard. In my case I placed the widget near the regular test and coverage widget.

Running HtmlUnit tests with JaCoCo enabled in Maven

To enable JaCoCo in our Maven build lifecycle, we need to do the followin:

  • Add the JaCoCo agent to the command-line arguments of the server startup.
  • Tell JaCoCo where to leave the code coverage information.
  • Tell Sonar where to find the code coverage information.
To add the JaCoCo I have decided to use a build parameter for JaCoCo installation directory. This way the POM is portable and can be reused in my Continuous Integration engine (Jenkins). As we are writing integration tests that run on a web server, the agent is added to the cargo.jvmargs configuration property of the Maven Cargo plug-in:
            <plugin>
                <groupId>org.codehaus.cargo</groupId>
                <artifactId>cargo-maven2-plugin</artifactId>
                <version>1.1.2</version>
                <configuration>
                    <container>
                        <containerId>tomcat7x</containerId>
                        <zipUrlInstaller>
                            <url>http://archive.apache.org/dist/tomcat/tomcat-7/v7.0.6/bin/apache-tomcat-7.0.6.zip</url>
                        </zipUrlInstaller>
                    </container>
                    <configuration>
                        <properties>
                            <cargo.servlet.port>8180</cargo.servlet.port>
                            <cargo.tomcat.ajp.port>8109</cargo.tomcat.ajp.port>
                            <cargo.jvmargs>-javaagent:${jacoco.agent.path}=destfile=${project.build.directory}/itest.jacoco</cargo.jvmargs>
                        </properties>
                    </configuration>
                </configuration>
                <executions>
                    <!-- start server before integration tests -->
                    <execution>
                        <id>start-container</id>
                        <phase>pre-integration-test</phase>
                        <goals>
                            <goal>start</goal>
                        </goals>
                    </execution>
                    <!-- stop server after integration tests -->
                    <execution>
                        <id>stop-container</id>
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>stop</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

The destfile parameter in the agent tells JaCoCo were to write the code coverage report.

To finish the configuration, we will need to add a sonar.jacoco.itReportPath property to the POM. Obviously we use the same path that is specified in the agent command line:

    <properties>
        <sonar.jacoco.itReportPath>${project.build.directory}/itest.jacoco</sonar.jacoco.itReportPath>
    </properties>

Once configuration is finished, we can execute Maven verify and sonar:sonar goals. Don’t forget to add the jacoco.agent.path JVM argument to the Maven command line (e.g. mvn -Djacoco.agent.path=/tools/jacoco-0.5.3/lib/jacocoagent.jar clean verify sonar:sonar).

With verify we will compile, package and run the tests deployed on the test server. This includes the code coverage report collected by JaCoCo. Then, Sonar will analyze the code as usual but this time it will also get JaCoCo report and process it to add code coverage information to Sonar’s project dashboard:

Conclusion

JaCoCo and Sonar have allowed us to incorporate code coverage of integration tests to the set of metrics we manage under continuous integration/continuous quality assurance. In retrospective, we have shown how to write integration tests for web UI projects, how to automate the repeated execution of those tests inside a Maven build lifecycle and how to get the code coverage metrics from those tests.

Some of the techniques shown in this post can also be applied to different tools or types of projects, for example other testing frameworks besides HtmlUnit or integration tests of non-web UI projects, as batch or integration.

But that will be another story.

About these ads

19 thoughts on “Measure Code Coverage of HtmlUnit Based Tests with Sonar and JaCoCo

  1. Thanks for taking the time to post this clear explanation. After scouring the net for hours and finding only old information, I was finally able to get this working.

    One thing that wasn’t obvious to me is that I had to supply -Djacoco.agent.path=/path/to/jacocoagent.jar to mvn, but after that everything worked like a charm. Also, minor point, I changed the jvmArgs to use destfile=${sonar.jacoco.itReportPath}.

    • Thank you very much for pointing out the missing JVM argument. I missed this. And many thanks forthe recommendation to simplify setup a bit!

  2. Pingback: Sonar » Sonar in the news

  3. Pingback: Measure Code Coverage of HtmlUnit Based Tests with Sonar and JaCoCo | Sonar Quality Platofrm | Scoop.it

  4. Good Article!!!.
    You can avoid download jacocoagent. The solution is:
    -Add jacocoagent dependency in pom.xml

    org.jacoco
    org.jacoco.agent
    0.5.4.201111111111
    runtime
    test

    - Configure cargo.jvmargs
    -javaagent:${settings.localRepository}/org/jacoco/org.jacoco.agent/0.5.4.201111111111/org.jacoco.agent-0.5.4.201111111111-runtime.jar=destfile=${project.build.directory}/itest.jacoco

    I test this solution. In http://pimpam.googlecode.com/svn/trunk/web-calculator is implemented this solution.

    Carlos

      • for those that were curious about the approach suggested by carlos, the full dependency entry, using the latest jacoco build, is:

        org.jacoco
        org.jacoco.agent
        0.5.5.201112152213
        jar
        runtime
        test

        and the argline for failsafe to tun the IT is:

        -javaagent:${settings.localRepository}/org/jacoco/org.jacoco.agent/0.5.5.201112152213/org.jacoco.agent-0.5.5.201112152213-runtime.jar=destfile=${project.build.directory}/itest.jacoco

  5. Pingback: Test Automation with Selenium WebDriver and Selenium Grid – part 3: Continuous Integration | dr. macphail's trance

  6. Hi, great post.

    I’ve got a similar architecture – cargo, with junite2 instead of failsafe – and what’s happening to me it’s that sonar tracks coverage in **IT’s classes** instead of tested classes!! What could be the problem?

    Thanks
    Paolo

    • Hi Paolo,

      Do you have your IT classes in a separate module or in the same module with the application classes?

      I’m not familiar with JUnitE2, but assuming that you are using JaCoCo (this discards any instrumentation issue) the symptom is like if you were actually building, launching tests and analyzing with Sonar the IT module, instead of the application module. That is, JaCoCo tracked coverage of application classes but Sonar filtered only the classes under analysis, that is, the IT classes.

      • Hi Jorge,
        (thanks for the response and sorry for the little detail) i have separate modules.

        I’m working on a multimodule project (similar to http://www.sonarsource.org/measure-code-coverage-by-integration-tests-with-sonar/) with
        - A aggregator
        - B/C ejb modules with tested classes within
        - D ear module
        - E it-war module executing cargo/junite2 to test D (thus B/C)
        So Sonar’s agent is being activated by cargo in E with jvmargs, and it produces jacoco-it.exec only for (classes belonging to) that module. But how can i change it to instrument also other modules’ classes? They’re being executed/deployed within JBoss, that’s already being started with “-javaagent”/jacoco parameter working fine.

        Paolo

  7. I think that the problem is not instrumentation. JaCoCo does not instrument source code or bytecodes (as other tools do, e.g. Coberture or Clover).

    Are you executing the IT suite and collecting coverage before Sonar? When you execute Sonar, are you doing it fom A or D? If you execute Sonar from E then Sonar is analyzing E sources only, and filtering coverage only for IT classes.

    I mean, IT module should be executed and code coverage measured, but never analyzed.

    A way to troobleshoot/confirm this is to generate the JaCoCo human-readable report using Ant (http://www.eclemma.org/jacoco/trunk/doc/examples/ant/build.xml). This way you can confirm that you have the measure for B and C modules (the ones you are interested in, I suppose).

    • Useful Ant task, i found the solution. Actually, the parameter raising problem was the “destfile=${project.build.directory}/itest.jacoco”: i was executing IT from E because IT sources were in E, and the only file created was “E/target/itest.jacoco”. But then, when firing sonar:sonar on the aggregator A (thus on B,C,D,E also) the plugin was looking for the itest.jacoco file in every aggregated module’s target dir, but could find it only when building E. Solution was to specify a fixed “destfile=${user.home}/.m2/itest.jacoco”, unrelated from any “project.*” property being set during each module’s build, and let all modules search for statistics on that fixed-path file.

      Thanks for the great support, Paolo.

      • Happy to help. I’m glad that you found and solved the issue. And I’m sure many others will find this useful. Thanks.

  8. How to configure jacoco on sonar for a web based application that uses Ant instead of maven.

    • Hi. Not a single answer, it may depend on how are you running your integrations tests before executing Sonar analysis. No matter if you are using Ant, Maven or command-line analysis, the JaCoCo IT report should exist before running Sonar analysis. How are you running your integration tests with Ant? Do you automate the packaging, deployment and execution of the tests inside the build script? If the answer is yes, you only need to be sure that when test server is started you add the JaCoCo agent to the VM arguments of the test server. For example:
      -javaagent:/tools/jacoco-0.6.0/jacocoagent.jar=destfile=/projects/my-project/jacoco.exec

      Finally, don’t forget to add the sonar.itReportPath property to Sonar analysis, pointing to where the JaCoCo coverage data file is.

  9. How would i send jacoco to a servlet started by cargo.daemon.. and then retieve coverage from the IT’s run on the daemon..?

    • can you add JVM args to the call to cargo daemon? if so, just the javaagent argument pointing to JaCoCo Jar file and location of coverage data. don’t forget that this is valid also for tests executed manually, I.e. deploy to server with JaCoCo activated and let your testers run the test suite before collecting the results.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s