About deors

senior technology architect in accenture, with a passion for java related stuff, celtic music and the best sci-fi, among other thousand things!

Pitest: Measure the Quality of your Unit Tests with Mutation Testing

It is not uncommon among developers to discuss about the quality of automated unit tests: Are they testing enough of application code? And more importantly, are they really verifying the expected behavior?

The first question has a relatively simple answer: use automated code coverage tools that will track which lines of code and which branches in execution flow are being tested. Code coverage reports are very helpful to 1) determine which portions of application code are not being tested; and 2) if measuring code coverage per individual test, determine whether each test is effectively testing the appropriate piece of application code. If interested in techniques for that, you may want to look at this other blog post: http://deors.wordpress.com/2014/07/04/individual-test-coverage-sonarqube-jacoco/

However, no matter how useful is to measure code coverage, these reports will not let you know one fundamental aspect of tests: which behavior is being verified!

Simply put, your test code may be passing through every single line of your code, and not verifying anything. If you are familiar with JUnit framework, your test code may not contain a single assertion!

To overcome this limitation of automated unit testing, one technique that can be of great help is Mutation Testing.

Mutation Testing… Explained

Let’s assume you have your application code and your test code as usual. A mutation testing tool will take your application code and make small surgical changes, one at a time, a so-called “mutation”. It could be changing a logical operator in an if statement (e.g. > is changed to <=), it could be removing some service call, it could be changing some for loop, it could be altering some return value, and so forth.

Mutation testing is, therefore, based on the assumption that if you are testing your code and making the right assertions to verify the behavior, once you re-execute your unit tests with a mutation in application code some of them should fail.

Pitest – A Mutation Testing Framework for Java

Although very interesting, such a technique would be useless without the proper tools. There are some, and for different languages, like Jester, Jumble or NinjaTurtles, but probably the most mature and powerful we’ve seen to date is Pitest (http://pitest.org).

Working with Pitest is very simple and requires minimal effort to start. It can be integrated with build tools like Maven, Ant or Gradle, with IDEs like Eclipse (Pitclipse plug-in) or IntelliJ, and with quality tools like SonarQube.

Regardless of the way you execute it, Pitest will analyze the application byte codes and decide which mutations will be introduced (for a full list and description of available mutators in Pitest, check their site here: http://pitest.org/quickstart/mutators/).

To optimize the test execution as much as possible, Pitest gathers code coverage metrics in a “normal execution” and then re-executes only the matching test cases for a certain mutation. Total execution time is noticeably longer than a normal unit test execution, basically because the incredible test harness that Pitest adds even to the most simple of code bases.

As a result, Pitest generates a fully detail report showing which mutations “lived” after the execution, that is, which mutations where not detected by any existing assertion. These “lived” mutations are your main focus, because they mean that there is some logic, some return value, or some call that is not being verified.

Of course not all of the mutations will be meaningful. Some may produce out of memory errors or infinite loops. For those cases, Pitest does its best to detect them and remove from the resulting reports. These can be fine-tuned if needed, for example by tuning time outs and other parameters, but sensible defaults work really well to start with.

Pitest in Action

Seeing is believing, so we put Pitest to work on a simple 10-classes Java library. We decided to use the Maven plug-in, as this method requires zero configuration to start. We opened a command prompt at the project directory, and just executed this command:

> mvn org.pitest:pitest-maven:1.0.0:mutationCoverage

After a few minutes (5 to 6 for this project) and lots of iterations showing in the console, the build finishes and the reports are generated in target directory:

> target\pit-reports\201408181908\index.html

When the report loaded in the browser, the first fact that caught our attention was that one class, that we worked hard to be fully tested, AbstractContext, although with a 100% code coverage it showed one lived mutation. Oops, something was not properly verified. Was Pitest right?

mutation-1

After clicking the class name, we could see the detail on where the lived mutation was found:

mutation-2

Pitest was right! Although that method is fully tested, and there are test cases for every single execution flow, we were missing the proper assertion for that if statement. Really really hard to catch if not for a good tool helping us to find out more about our unit tests.

Of course, next step was to add the forgotten assertion to the relevant test method. Once done, we re-launched Pitest. After a few minutes, a new set of reports where created and once loaded in the browser… clean result for that class!

mutation-4

Conclusion

Although arguably a bit fortunate to obtain such a fabulous result at the first try, it is true that after a more thorough inspection of the reports we found many other places where assertions were missing.

Our view is that Pitest is a very valuable tool to write really meaningful and truly useful automated unit test suites, and should be standard gear for Java projects going forward. It is simple to use, requires zero or minimal configuration, and produces valuable results that directly impact in the quality of the test we create, and therefore in the quality of our deliverables.

To mutate, or not to mutate: that is the question.
Whether 'tis nobler in the mind to suffer
The slings and arrows of outrageous unit tests.

Script to List Key Job Settings in Jenkins at a Glance

One can get addicted to scripting in Jenkins quickly! ;-)

When you have dozens even hundreds of jobs in Jenkins, it is really important to have a way to review or change job settings in one shot. One of my favorite scripts, that I use when I want to get key settings from all jobs at a glance, is this one:

import hudson.model.*
import hudson.maven.*

for (job in Hudson.instance.items) {
  if (job instanceof MavenModuleSet) {
    mms = (MavenModuleSet) job
    def name = mms.name
    def jdk = "def"
    if (mms.JDK != null) {
      jdk = mms.JDK.name
    }
    def mvn = mms.mavenName
    def goals = mms.goals
    printf("%-50s | %-10s | %-15s | %-50s\n", name, jdk, mvn, goals)
  }
}

And this is a example output. I love it! :-D  (Be sure to scroll right to see full output.)

deors.demos.annotations.base                       | jdk-8      | null            | clean install                                     
deors.demos.annotations.base.client                | jdk-8      | null            | clean test                                        
deors.demos.annotations.base.processors            | jdk-8      | null            | clean install                                     
deors.demos.annotations.beaninfo                   | jdk-8      | null            | clean install                                     
deors.demos.annotations.beaninfo.client            | jdk-8      | null            | clean test                                        
deors.demos.annotations.beaninfo.processors        | jdk-8      | null            | clean install                                     
deors.demos.annotations.velocity.client            | jdk-8      | null            | clean test                                        
deors.demos.annotations.velocity.processors        | jdk-8      | null            | clean install                                     
deors.demos.batch.springbatch2                     | jdk-7      | maven-3.2.1     | clean verify                                      
deors.demos.cloud.gae                              | jdk-7      | maven-3.2.1     | clean verify                                      
deors.demos.cloud.heroku                           | jdk-7      | maven-3.2.1     | clean verify                                      
deors.demos.cloud.rhc                              | jdk-7      | maven-3.2.1     | clean verify                                      
deors.demos.cloud.vmc                              | jdk-7      | maven-3.2.1     | clean verify                                      
deors.demos.java8                                  | jdk-8      | maven-3.2.1     | clean verify                                      
deors.demos.testing.arquillian                     | jdk-7      | maven-3.2.1     | clean verify                                      
deors.demos.testing.arquillian-glassfish-embedded  | jdk-7      | null            | clean verify -Parquillian-glassfish-embedded      
deors.demos.testing.arquillian-glassfish-remote    | jdk-7      | null            | clean verify -Parquillian-glassfish-remote,!arquillian-glassfish-embedded
deors.demos.testing.arquillian-jboss-managed       | jdk-7      | null            | clean verify -Parquillian-jboss-managed,!arquillian-glassfish-embedded
deors.demos.testing.arquillian-jboss-remote        | jdk-7      | null            | clean verify -Parquillian-jboss-remote,!arquillian-glassfish-embedded
deors.demos.testing.arquillian-weld-embedded       | jdk-7      | null            | clean verify -Parquillian-weld-embedded,!arquillian-glassfish-embedded
deors.demos.testing.htmlunit                       | jdk-7      | maven-3.2.1     | clean verify                                      
deors.demos.testing.htmlunit-cargo-glassfish       | jdk-7      | null            | -P glassfish cargo:redeploy                       
deors.demos.testing.htmlunit-cargo-jboss           | jdk-7      | null            | -P jboss cargo:redeploy                           
deors.demos.testing.htmlunit-cargo-tomcat          | jdk-7      | null            | -P tomcat cargo:redeploy                          
deors.demos.testing.htmlunit-deploy-glassfish      | jdk-7      | null            | clean install                                     
deors.demos.testing.htmlunit-deploy-tomcat         | jdk-7      | null            | clean install                                     
deors.demos.testing.mocks                          | jdk-7      | maven-3.2.1     | clean verify                                      
deors.demos.testing.selenium                       | jdk-7      | maven-3.2.1     | clean verify                                      
deors.demos.testing.selenium-cargo-glassfish       | jdk-7      | null            | -P glassfish cargo:redeploy                       
deors.demos.testing.selenium-cargo-jboss           | jdk-7      | null            | -P jboss cargo:redeploy                           
deors.demos.testing.selenium-cargo-tomcat          | jdk-7      | null            | -P tomcat cargo:redeploy                          
deors.demos.testing.selenium-deploy-glassfish      | jdk-7      | null            | clean install                                     
deors.demos.testing.selenium-deploy-tomcat         | jdk-7      | null            | clean install                                     
deors.demos.web.gwt2                               | jdk-7      | maven-3.2.1     | clean verify                                      
deors.demos.web.gwt2spring                         | jdk-7      | maven-3.2.1     | clean verify                                      
deors.demos.web.springmvc3                         | jdk-7      | maven-3.2.1     | clean verify                                      
petclinic-1-build-test                             | jdk-7      | maven-3.2.1     | clean test                                        
petclinic-2-package                                | jdk-7      | null            | package -DskipTests=true                          
petclinic-3-tomcat-run                             | jdk-7      | null            | cargo:run -Pcargo-tomcat                          
petclinic-4-verify-selenium-htmlunit               | jdk-7      | maven-3.2.1     | failsafe:integration-test -P selenium-tests       
petclinic-5-verify-jmeter                          | jdk-7      | maven-3.2.1     | jmeter:jmeter -P jmeter-tests                     
petclinic-6-tomcat-stop                            | jdk-7      | null            | cargo:stop -Pcargo-tomcat                         
petclinic-9a-verify-selenium-openshift             | jdk-7      | maven-3.2.1     | failsafe:integration-test -P selenium-tests       
petclinic-9b-verify-selenium-heroku                | jdk-7      | maven-3.2.1     | failsafe:integration-test -P selenium-tests       
petclinic-full-all-browsers                        | jdk-7      | maven-3.2.1     | clean verify -P cargo-tomcat,selenium-tests       
petclinic-full-htmlunit-sonar                      | jdk-7      | maven-3.2.1     | clean verify -P cargo-tomcat,selenium-tests,jmeter-tests

Script to Update Jenkins Jobs to Use a Different Maven Instance

One of the features of Jenkins that I like a lot, very useful when you need to do bulk changes on your jobs configuration, is the script console.

The script console allows to run Groovy scripts that can read and alter the state of the jobs, or any other piece of configuration or state exposed through Jenkins API or through its plug-ins respective APIs.

Today I decided to take a look to Apache Maven 3.2.1 and I wanted to easily test all my existing jobs with this new version. As I’m “lazy” ;-) and didn’t want to update the jobs one by one, I created this script to do the job for me in no time. Hope you enjoy it!

import hudson.maven.*
import hudson.model.*
import hudson.tasks.*
import hudson.tools.*

oldMavenName = "maven-3.0.4"
newMavenName = "maven-3.2.1"

Maven.MavenInstallation oldMaven
Maven.MavenInstallation newMaven

// look for old and new Maven installations
// useful to detect that something is not well configured
for (ti in ToolInstallation.all()) {
  if (ti instanceof Maven.MavenInstallation.DescriptorImpl) {
    for (i in ti.installations) {
      if (i.name.equals(oldMavenName)) {
        oldMaven = i
      } else if (i.name.equals(newMavenName)) {
        newMaven = i
      }
    }
  }
}

println("migrating jobs from Maven: " + oldMaven.name)
println("to Maven: " + newMaven.name)
println()

// locate the jobs and update the Maven installation
// optionally filter them by some prefix or regex
for (job in Hudson.instance.items) {
  if (job instanceof MavenModuleSet) {
    mms = (MavenModuleSet) job
    if (mms.name.startsWith("some.prefix")) {
      println("job " + mms.name + " currently using: " + mms.mavenName)
      // if name is null, it means the default Maven installation
      if (mms.mavenName == null || mms.mavenName == oldMaven.name) {
        mms.mavenName = newMaven.name
        println(" migrate to: " + mms.mavenName)
      } else {
        println(" no migration needed")
      }
    }
  }
}

Java 8 Lambdas explained #1: Map/Reduce with Fork/Join and the Beauty of a One-Line Lambda

Lambdas are cool. Lambdas are sexy. Lambdas are expressive and let you write less code. Are these all a cliché? No! They are so true…

Recently we run a session locally explaining Lambdas with side-by-side comparisons of some coding patterns expressed in Java 7 vs. Java 8. We used the best we could obtain of each version of the language and APIs, including lambdas, stream processing, the new Date and Time API… It was fun seeing people reactions on how dramatically code can be reduced from Java 7 to Java 8 style.

In this post I want to show one example of the above: how a simple Map/Reduce pattern can be radically simplified if you switch the Fork/Join API from Java 7 to parallel stream processing in Java 8.

Setting up the project

Let’s start with setting up the project. A very simple Maven project configuration that explicitly sets Java 8 as source and target for compilation and adds a dependency to JavaTuples library:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>deors.demos</groupId>
  <artifactId>deors.demos.java8</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>
  <name>deors.demos.java8</name>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.1</version>
        <configuration>
          <verbose>true</verbose>
          <compilerVersion>1.8</compilerVersion>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
    </plugins>
  </build>
  <dependencies>
    <dependency>
      <groupId>org.javatuples</groupId>
      <artifactId>javatuples</artifactId>
      <version>1.2</version>
      <scope>compile</scope>
    </dependency>
  </dependencies>
</project>

The Problem

For this example, this is the problem we want to solve: given a list of integer tuples (pairs of integers), calculate the sum of the product of each pair. As we want the solution to scale, it should be done through parallel computing so the JVM running the code can take advantage of multi-core environments to split the problem in simpler pieces and make the processing faster.

In Java 7 fashion, a clever programmer could choose the Fork/Join API and create a task that, given the list of tuples, split it in halves when its size exceeds some defined threshold, process each half in parallel and recursively, aggregating the results from each piece until the final result is obtained. Not complex but a bit verbose even with the Fork/Join API. This is the code (don’t worry to write it down – this code can be obtained from GitHub here https://github.com/deors/deors.demos.java8):

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ForkJoinPool;
import java.util.concurrent.RecursiveTask;
import org.javatuples.Pair;

public class SumProductCalculationJava7 extends RecursiveTask<Integer> {
  private static final long serialVersionUID = 6939566748704874245L;
  private int threshold = 10;
  List<Pair<Integer, Integer>> pairList;

  public SumProductCalculationJava7(List<Pair<Integer, Integer>> pairList) {
    super();
    this.pairList = pairList;
  }

  @Override
  protected Integer compute() {
    System.out.printf("fragment %s size %d\n", this, pairList.size());
    if (pairList.size() <= threshold) {
       return computeDirect();
    }
    int split = pairList.size() / 2;

    List<Pair<Integer, Integer>> forkedList1 = pairList.subList(0, split);
    SumProductCalculationJava7 forkedTask1 = new SumProductCalculationJava7(forkedList1);
    forkedTask1.fork();

    List<Pair<Integer, Integer>> forkedList2 = pairList.subList(split, pairList.size());
    SumProductCalculationJava7 forkedTask2 = new SumProductCalculationJava7(forkedList2);
    forkedTask2.fork();

    return forkedTask1.join() + forkedTask2.join();
  }

  private Integer computeDirect() {
    Integer sumproduct = 0;
    for (Pair<Integer, Integer> pair : pairList) {
      sumproduct += pair.getValue0() * pair.getValue1();
    }
    System.out.printf("fragment %s total %s\n", this, sumproduct);    return sumproduct;
  }
}

In short, the compute method from RecursiveTask contract checks the size of the list. If it is lower than the threshold it calculates the sum-product using a private method with a for each loop. If not, it splits the list in two pieces and recursively processes the two fragments. The result of processing the two fragments is, as expected, the sum of both fragment results.

It is verbose, yes, but it does a lot of work in the back stage: use the best concurrency patterns, running the threads, waiting for each fragment to finish before aggregating results bottom-up and returning the final glorious result. Try to do the same with threads in Java 2 style and look how your hair will turn grey during the process!

However, Java 8 still can beat this by a large margin.

Testing the Problem

Now let’s run this task with a main method or JUnit test:

  public static void main(String[] args) {
    List<Pair<Integer, Integer>> thePairList = new ArrayList<>();
    thePairList.add(new Pair<Integer, Integer>(10, 1));
    thePairList.add(new Pair<Integer, Integer>(12, 2));
    thePairList.add(new Pair<Integer, Integer>(14, 3));
    thePairList.add(new Pair<Integer, Integer>(16, 4));
    thePairList.add(new Pair<Integer, Integer>(18, 5));
    thePairList.add(new Pair<Integer, Integer>(20, 6));
    thePairList.add(new Pair<Integer, Integer>(22, 7));
    thePairList.add(new Pair<Integer, Integer>(24, 8));
    thePairList.add(new Pair<Integer, Integer>(26, 9));
    thePairList.add(new Pair<Integer, Integer>(28, 10));
    thePairList.add(new Pair<Integer, Integer>(30, 11));
    thePairList.add(new Pair<Integer, Integer>(32, 12));
    thePairList.add(new Pair<Integer, Integer>(34, 13));
    thePairList.add(new Pair<Integer, Integer>(36, 14));

    SumProductCalculationJava7 theTask = new SumProductCalculationJava7(thePairList);
    ForkJoinPool thePool = new ForkJoinPool();
    Integer result = thePool.invoke(theTask);
    System.out.printf("the final result is %s\n", result);
  }

Well done! As expected, the final result is 2870!

Try now with different lists of tuples or just copy and paste to have a very long list in a few seconds. Change the threshold, measure execution times and fine-tune the program until it is as good as it can be.

You still cannot be sure whether it will be as optimal when executed in a different JVM, though, with different CPU and RAM available resources performance may change and it is likely that the best threshold is different depending on the JVM, machine resources, OS, workload, etc., but you can still deliver this with confidence and fine tune it for production later.

Same Problem. Different Approach

Now let’s do the same calculation above with Java 8 parallel streams and lambda expressions. Don’t expect too much – it’s really as simple as it looks. In fact it is too simple that I just coded everything in the main method:

import java.util.ArrayList;
import java.util.List;
import org.javatuples.Pair;

public class SumProductCalculationJava8 {
  public static void main(String[] args) {
    List<Pair<Integer, Integer>> thePairList = new ArrayList<>();
    thePairList.add(new Pair<Integer, Integer>(10, 1));
    thePairList.add(new Pair<Integer, Integer>(12, 2));
    thePairList.add(new Pair<Integer, Integer>(14, 3));
    thePairList.add(new Pair<Integer, Integer>(16, 4));
    thePairList.add(new Pair<Integer, Integer>(18, 5));
    thePairList.add(new Pair<Integer, Integer>(20, 6));
    thePairList.add(new Pair<Integer, Integer>(22, 7));
    thePairList.add(new Pair<Integer, Integer>(24, 8));
    thePairList.add(new Pair<Integer, Integer>(26, 9));
    thePairList.add(new Pair<Integer, Integer>(28, 10));
    thePairList.add(new Pair<Integer, Integer>(30, 11));
    thePairList.add(new Pair<Integer, Integer>(32, 12));
    thePairList.add(new Pair<Integer, Integer>(34, 13));
    thePairList.add(new Pair<Integer, Integer>(36, 14));

    Integer result = thePairList.parallelStream().
      mapToInt(p -> p.getValue0() * p.getValue1()).sum();
    System.out.printf("the final result is %s\n", result);
  }
}

Just one line of code to solve the same problem. Same result, same magic in the back stage, same powerful performance, but simple, expressive and productive!

Moreover, the JVM does all the optimizations for you. Depending on the variables mentioned above, like CPU, RAM or workload, the JVM can apply different approaches at runtime and make sure that your code runs as quick as possible in all conditions.

This is just one example on how Java 8 lets you write better code, so don’t wait for others to ask you to do it. Try to adopt Java 8 today!

Code Coverage of Individual Tests with SonarQube and JaCoCo

This post explains how to enable SonarQube to gather test code coverage metrics of individual tests. Code coverage tools typically produce a report showing the code coverage (by line, branch, etc.) for the combined effect of all the tests executed during a given test session. This is case, for example, when you run unit tests in continuous integration. With the help of SonarQube and JaCoCo, it is possible to gather coverage metrics split at the level of the individual test case (test method in JUnit or TestNG). To enable this, there is some special configuration required that we are showing in this post.

The Environment

The following process has been verified with SonarQube 4.1.2 and 4.3.2 versions, but it should work with SonarQube 3.7.x (latest LTS release), too. The application code we have used to verify the setup is the familiar Spring Pet Clinic application, enhanced to support Tomcat 7 and Spring 3 (see this post here for reference on updates needed in Pet Clinic: http://deors.wordpress.com/2012/09/06/petclinic-tomcat-7/) The code can be downloaded from GitHub in the repository: https://github.com/deors/deors.demos.petclinic

The Instructions

The instructions are really simple, once you’ve figured out how to connect all the dots. All that is required is to add some specific configuration to Maven Surefire plug-in (Surefire is the plug-in that is tasked with the unit test execution, and it supports JUnit and TestNG). As this specific configuration should not impact the regular unit test execution, it is recommended to include the needed configuration in a separate profile that will be executed only when the SonarQube analysis is performed. Let’s describe the required changes in the pom.xml file, section by section.

Build Section

No changes are needed here. However, you should take note of any customised configuration of Maven Surefire to be sure it is also applied to the profile we are going to create. In the case of Spring Pet Clinic, this is the relevant portion of the POM we are writing down for reference:

<build><plugins>
...
 <plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.13</version>
  <configuration>
   <argLine>-XX:-UseSplitVerifier</argLine>
   <includes>
    <include>**/*Test.java</include>
    <include>**/*Tests.java</include>
   </includes>
   <excludes>
    <exclude>**/it/*IT.java</exclude>
   </excludes>
  </configuration>
 </plugin>
...
</plugins></build>

This piece of configuration is telling Surefire to: 1) exclude the integration tests for the execution of unit tests (integration tests are covered by Surefire’s twin plug-in, Failsafe); and 2) disable the byte code verifier, preventing runtime errors when classes are instrumented (i.e. when adding mocks, or TopLink enhancements).

Dependencies Section

Again no changes are needed in this section. We just wanted to note that if your project is already leveraging JaCoCo to gather integration test coverage metrics, and is explicitly referring to JaCoCo artefact in this section, it can be left – no conflicts have been identified so far. Anyway it should not be needed here, so it’s probably safer to remove it from this section.

Profiles Section

All the required changes come in this section. And they are very clean to add, as they all require only to add a new profile to the POM. This profile will configure a special listener for Surefire that will ensure that coverage metrics for each individual test case are appropriately gathered. To guarantee a successful test execution, we will maintain here the same configuration that appears in the build section of the POM. Finally, the profile will add a new dependency to the artefact that contains the listener code. The result is this:

<profile>
 <!-- calculate coverage metrics per test with SonarQube and JaCoCo -->
 <id>coverage-per-test</id>
  <build>
   <plugins>
    <plugin>
     <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-surefire-plugin</artifactId>
     <version>2.13</version>
     <configuration>
      <!-- same configuration as in the regular test execution goal -->
      <argLine>-XX:-UseSplitVerifier</argLine>
      <includes>
       <include>**/*Test.java</include>
       <include>**/*Tests.java</include>
      </includes>
      <excludes>
       <exclude>**/it/*IT.java</exclude>
      </excludes>
      <!-- new configuration needed for coverage per test -->
      <properties>
       <property>
        <name>listener</name>
         <value>org.sonar.java.jacoco.JUnitListener</value>
       </property>
      </properties>
     </configuration>
    </plugin>
   </plugins>
  </build>
 <dependencies>
  <dependency>
   <groupId>org.codehaus.sonar-plugins.java</groupId>
   <artifactId>sonar-jacoco-listeners</artifactId>
   <version>2.3</version>
   <scope>test</scope>
  </dependency>
 </dependencies>
</profile>

A piece of warning around the JaCoCo listener artefact version. Although it is unclear in the documentation, it seems that the best results are obtained when the JaCoCo listener version matches that of the Java plug-in installed in SonarQube. In this case, as the Java plug-in that we have installed in SonarQube is version 2.0, we have used the listener artefact version 2.0. We also tested with listener 1.2 with same good results, but to prevent any future conflict, we recommend keeping versions aligned.

Running the Analysis

Once the changes in the project configuration are done, you just need to re-execute a SonarQube analysis to see the new reports.

Depending on which SonarQube Java version you have installed, the configuration differs a bit.

Running the Analysis in Older Versions

When the Java plug-in version in use is 2.1 or an earlier version, the profile should be enabled when the analysis executes, and only when the analysis executes. This means that it is now a requirement to launch the sonar:sonar goal as a separate Maven build (it was recommended to do so, but in many cases you could execute all the targets in one run). In the case of our version of Pet Clinic:

>mvn clean verify -P cargo-tomcat,selenium-tests,jmeter-tests
>mvn sonar:sonar -P coverage-per-test

If your build is triggered by a Jenkins job, then the new profile should be added to the post-build action as can be seen in this screenshot: sonar-post-build

Running the Analysis in Newer Versions

When the Java plug-in version in use is 2.2 or newer, code coverage is no longer executed during the analysis. Therefore you should configure the build to gather the code coverage metrics first:

>mvn clean org.jacoco:jacoco-maven-plugin:0.7.0.201403182114:prepare-agentverify -P coverage-per-test,cargo-tomcat,selenium-tests,jmeter-tests
>mvn sonar:sonar -P coverage-per-test

If your build is triggered by a Jenkins job, then the JaCoCo prepare agent goal and the new profile should be added to the build action as can be seen in this screenshot:

sonar-maven-modern

Analysis Results

Once the analysis is completed, the code coverage reports get some new interesting views. When clicking on any test on the test view, a new column labelled ‘Covered Lines’ shows the individual hits for each test method in the class: sonar-test-summary When the link on Covered Lines value is followed, a new widget shows containing all the classes hit by that test method, and the touched lines per class: sonar-test-detail When the link under each of the classes is followed, a new widget appears showing the class source coloured with the actual line/branch hits: sonar-test-code Users can also get to this view if navigating through other views, as components or violations drill-down. Once the class level is reached, users can use the ‘Coverage’ tab to get this information:
sonar-class-coverage
By default, the decoration shown is ‘Lines to cover’, showing the code coverage from all tests combined. Use the drop-down list and select ‘Per test -> Covered lines’ and then select the right text case in the new drop-down list that will appear:
sonar-class-select-decoration
sonar-class-select-testcase
sonar-class-final

Conclusion

Measuring code coverage of individual tests is a very useful feature to have in development projects. Code coverage metrics alone may not be sufficient to identify that the rights tests are being executed and they are touching the right functionality. With the ability to identify which portions of the code are executed by any test case, developers and tester can ensure that the expected code logic is tested, versus what can be obtained with other code coverage tools that only gives a combined coverage report.

Writing Your Own SonarQube Plug-ins – Part 4: Testing and Deploying

In part 1 of this series here, I explained the basics of SonarQube plug-ins and how to start writing your own plug-in. In part 2 here, I explained how to define custom metrics and sensor classes to collect metric data (measures). In part 3 here, I showed the basics of how to create custom widgets to present to users the information we are collecting from analysed projects.

Now finally in part 4, I will show how to test and deploy your plug-ins, and some tips to make development and testing workarounds quicker, like configuring JRebel to speed up development.

Part 4 – Testing and Deploying

If you have been following this series of blog posts, you already have a simple structure for a plug-in, hopefully doing something useful and showing valuable insights from your code and configuration.

So what’s next? How can we test and deploy our plug-ins? The simple naïve answer is: package and deploy it to a working SonarQube instance, as you would do with any other plug-in.

Packaging the plug-in is as easy as executing Maven’s package goal. The Maven configuration, governed by the sonar-plugin packaging type, will ensure that the right classes and configuration files are packaged in the right way. The resulting Jar file will be ready for deployment into a SonarQube instance.

Deploying the plug-in is also an easy task. If you are a seasoned SonarQube user, you probably know how to install and update public plug-ins from SonarQube update center. Unfortunately this way is only available for plug-ins listed in the public SonarQube directory, and is not suitable for our own in-house, internal-only plug-ins. In our case, we would need to manually copy the packaged plug-in file to our SonarQube instance, in the folder extensions\downloads, and then restart SonarQube. This is actually what the update center does: downloading a plug-in file from Internet and place it in downloads folder, so it is picked up automatically in next restart.

Although the process above is simple and easy to automate with a script, it is lengthy. Moreover, it needs a local SonarQube instance that you need to manage, and that it’s subject to get polluted by other plug-ins or just by normal user usage. Fortunately, SonarQube provides with a better alternative way to test a plug-in.

SonarQube Development Mode

To prevent developer’s sanity, SonarQube provides with a development mode. This development mode will automate the deploy process as a Maven plug-in, downloading a SonarQube instance from the internet if needed and running it as a child process of the Maven process.

To launch the development mode, just issue this command:

mvn install org.codehaus.sonar:sonar-dev-maven-plugin:1.7:start-war -Dsonar.runtimeVersion=3.7.3

You may setup an alternate version of SonarQube to host and run the plug-in as needed. The SonarQube instance spawned by this command will run on default settings: listening on port 9000, with embedded H2 database.

Once ready, just launch a SonarQube analysis, either using SonarQube Maven plug-in or using SonarQube Runner standalone analysis. The analysis will connect by default to the server in development mode and run your plug-in as part of the analysis process.

Optimized Testing with JRebel

Although SonarQube development mode is nice and very convenient, it has a fundamental problem – it is slow! It takes quite a few minutes to have the environment prepared, and if you find a bug or want to improve something, and you need to change some Java code, you have to drop it and start the process again.

Fortunately, JRebel comes to a rescue and can dramatically speed up the process. The same that JRebel monitors Java web containers and injects changes in Java bytecodes directly into the running JVM that hosts the web container, without loosing state – effectively saving you the few (or many) minutes needed to redeploy and/or restart the web container – JRebel can monitor our plug-in under development.

To configure JRebel, just add JRebel configuration and agent to the developer mode launch script.

JRebel configuration is just a simple rebel.xml file under src/main/resources folder, which content is set to monitor the right folder where Java classes are compiled into:

<?xml version="1.0" encoding="UTF-8"?>
<application xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.zeroturnaround.com" xsi:schemaLocation="http://www.zeroturnaround.com http://www.zeroturnaround.com/alderaan/rebel-2_0.xsd">
    <classpath>
        <dir name="C:/projects/xxxxx/target/classes"/>
    </classpath>
</application>

Then, add the JRebel agent to the development mode launch command:

mvn install org.codehaus.sonar:sonar-dev-maven-plugin:1.7:start-war -Dsonar.runtimeVersion=3.7.3 -Dsonar.containerArgs=-javaagent:C:\java\tools\jrebel-5.5.2\jrebel.jar -Drebel.log=true

This is possible thanks to sonar.containerArgs command-line options, kindly provided by ZeroTurnaround support team as a merge request into open-sourced SonarQube development mode plug-in. Who said vendor support does not work?

Once launched with JRebel, the development mode may remain active during the full development and testing session, without needing to redeploy or restart it again, and picking changes done in plug-in classes.

Evenmore, if the SonarQube analysis is launched with JRebel agent active, each subsequent SonarQube analysis process that is executed will pick changes in the plug-in, effectively making the plug-in code to be ‘reloaded’ automatically, without needing to redeploy it again. To do that, don’t forget to add this to the SonarQube analysis command:

mvn sonar:sonar -javaagent:c:\java\tools\jrebel-5.5.2\jrebel.jar

Conclusion

It is easy to get started with custom plug-in development in SonarQube if you know where to start. Although open-source plug-ins are a good source of useful information, the online documentation lacks of a good plug-in 101 guide. I really hope that this short blog post series has been of help, it allows you to connect all the pieces that confirms a plug-in, and encourages you to conceive and bring to life your own custom plug-ins, either analysers, reporters or integrators with other existing tools.

Happy coding!

Writing Your Own SonarQube Plug-ins – Part 3: Creating Widgets

In part 1 of this series here, I explained the basics of SonarQube plug-ins and how to start writing your own plug-in. In part 2 here, I explained how to define custom metrics and sensor classes to collect metric data (measures).

In part 3, I will show the basics of how to create custom widgets to present to users the information we are collecting from analysed projects.

Widgets in SonarQube are typically written in Ruby on Rails. SonarQube platform comes with a lot of existing widgets, views, parts, etc. that can be combined into dashboards, which will present different set of information based on roles, but also will allow to drilldown measures or issues from top-level projects to source code.

Part 3 – Creating Widgets

The objective for the widget we would like to create is to simply show collected project-level information in project’s dashboard: which type of project we have configured in the IDE, which dependencies exist with other projects, etc.

In part 1, when creating the plug-in class, we instructed SonarQube to find a widget definition contained in IDEMetadataDashboardWidget class.

A widget class is very simple to create:

  • Create a class extending org.sonar.api.web.AbstractRubyTemplate and implementing org.sonar.api.web.RubyRailsWidget.
  • Annotate the class with @org.sonar.api.web.UserRole. Add as parameter the roles that will have access to this widget: Admin, User, CodeViewer.
  • Optionally annotate the class with @org.sonar.api.web.Description. Add as parameter the description for the widget, as it will be showed in SonarQube dashboard configuration view. Tell the users what is your widget meant to do!
  • Implement getId() and getTitle() methods. Simply return a string with the widget id and title. The id should be unique and the title will later be showed in SonarQube dashboard configuration view.
  • Implement getTemplatePath() method. This method will return a string with the path where the Ruby page can be found. The path should be relative to the plug-in project classpath (starting with a forward slash). During development, however, it is useful to set the path as an absolute path in your file system. Doing that will allow for automatic refresh of changes in the widget while you are coding and testing it. But don’t forget to revert back to the relative path when you are ready to deploy the plug-in!

It’s really simple – it takes more time to explain it than to actually code it ;-). See below the content of the widget class, without Javadocs to simplify the post.

@UserRole(UserRole.USER)
@Description("Shows IDE metadata configuration for the project, including project type, language support and configured frameworks")
public class IDEMetadataDashboardWidget extends AbstractRubyTemplate implements RubyRailsWidget {
    public String getId() {
        return "idemetadata";
    }
    public String getTitle() {
        return "IDE Metadata";
    }
    protected String getTemplatePath() {
        // uncomment next line for change reloading during development
        //return "c:/projects/xxxxx/src/main/resources/xxxxx/sonar/idemetadata/idemetadata_widget.html.erb";
        return "/xxxxx/sonar/idemetadata/idemetadata_widget.html.erb";
    }
}

Creating the View

Now it’s time to actually create the view. Views are created as Ruby on Rails pages. They are simple: even if you have not coded in Ruby on Rails before, you are probably familiarised with other templating engines so the jump to Rails should not be a challenge. Evenmore, the large amount of existing views that are available inside SonarQube are excellent to learn and apply patterns to your own views.

At the end, is HTML, styles and JavaScript what you are delivering to the clients, so just think with open mind what you can do with this mix of technologies. Do you want to use a JavaScript library because you like some of its visual components? You can. Are you a fan of Google charts and graphs? Use them. Are you a jQuery ninja? Unleash that power on your benefit.

If you don’t know where to start, pick an existing widget and make changes here and there, look at the existing JavaScript libraries to interact with SonarQube API, and you will be ready in no time.

Some pieces of advice that can help you to ease that journey:

  • You can place your own static resources (like images) in src/main/resources/static folder. Then you can refer to them from the widget with this code (note the reference to the plug-in id):
url_for_static(:plugin => 'idemetadata', :path => 'check-round-yes-small.png')
  • If you want to place an image in any place of the view, use the image_tag() function:
image_tag(url_for_static(:plugin => 'idemetadata', :path => 'check-round-yes-small.png'), :size => '16x16', :style => 'vertical-align:middle')
  • View consistency is very important. Don’t confuse your users and leverage the standard SonarQube styles. They are available to custom widgets as well and will help you to provide a consistent user interface.

Below you can see a short snippet of the view:

<div class="dashbox">
    <h3>Project name (as configured in the IDE):</h3>
    <p>
        <span class="big">
            <%= format_measure('ide_prj_name') %>
        </span>
    </p>
</div>
<div class="dashbox">
    <table class="width100">
        <tbody>
            <tr>
                <td>
                    <h3>Project type and active frameworks:</h3>
                    <table>
                        <tbody>
                            <tr>
                                <td>
                                    <%= check_image('ide_is_java') -%>
                                    <span>Java VM</span>
                                </td>
                            </tr>
...

This image shows how the configuration view looks like when you are going to add a view into a dashboard:

sonar-plugin-1

And the images below are a couple of examples on how the view looks like once placed in a project’s dashboard, showing the results of the plugin IDE metadata files scan and analysis:

sonar-plugin-2

 sonar-plugin-3

What’s next

In the next and final post of this series, the part 4, I will explain how to execute the plug-in in development mode and how to install it in your production SonarQube instance, with the help of JRebel to speed up development.