Maturing the Continuous Delivery Pipeline

deors:

A bit old but a good read anyway!

Originally posted on DevOpsGuys:

The Maturity Model is a useful assessment tool for understanding your organizations level of Continuous Delivery adoption. Many organizations today have achieved what is needed to move from Level-1 (Regressive) to Level-0 (Repeatable), which is a significant accomplishment and as a reader of this blog post, you’re either about to start your journey of improvement or are already underway.

Continuous Delivery Maturity Model Continuous Delivery Maturity Model

The Maturity Model

The advice for organizations wanting to adopt Continuous Delivery is ever more abundant, but for organizations that started adoption some time ago, the guidance on how to mature the process is still sparse. In this article, we explore one continuous improvement methodology that may help your organization mature its’ Continuous Delivery process.

Humble and Farley outline maturity Level 0 (Repeatable) – as one having process which is documented and partly automated.1 For this to be true, an organization must have first classified its’ software delivery maturity, identified…

View original 1,592 more words

Recording available for JavaOne CON2013 session – Code Generation in the Java Compiler: Annotation Processors Do the Hard Work

Directly from the pen of the happy panda, the recording for my JavaOne 2014 session on ‘Code Generation in the Java Compiler: Annotation Processors Do the Hard Work’ is finally available for replay at Parsleys site.

The direct link to the session recording is: https://www.parleys.com/talk/code-generation-java-compiler-annotation-processors-do-hard-work

New Releases: Oracle Java ME 8.1 and Java ME SDK 8.1

Originally posted on Across the Universe:

This is bigAt JavaOne a few weeks ago, Oracle made available the Java ME 8.1 Developer Preview release for the Freescale FRDM-K64F (“Java ME 8.1 in 190 KB RAM”) and announced the upcoming full release of Java ME Embedded 8.1 (press release).

On Monday this week, we followed up as promised and posted the General Availability (GA) releases of Oracle Java ME 8.1 and the Oracle ME SDK 8.1.

Oracle Java ME Embedded 8.1 and ME SDK 8.1 New Features and Enhancements

  • Support for ARM Cortex-M3/-M4 micro-controllers
  • Updated Raspberry Pi support 
  • Updated Developer Preview on FRDM-K64 with mbed
  • Improved support for two additional Qualcomm Gobi device families
  • New communication, security, and networking features
  • New support for Eclipse IDE, including major update of the Eclipse MTJ plugin
  • Developer improvements: Tooling over USB, heap analysis, faster communication
  • A number of smaller enhancements and fixes

Java 8: Truly Scalable

With this release…

View original 275 more words

Slides and code examples from JavaOne CON2013 session – Code Generation in the Java Compiler: Annotation Processors Do the Hard Work

The slides for my conference session CON2013 in JavaOne 2014 can be found online here: http://www.slideshare.net/deors/javaone-2014-con2013-code-generation-in-the-java-compiler-annotation-processors-do-the-hard-work

The coding examples can be found on GitHub: https://github.com/deors/deors.demos.annotations

con2013

Speaking at JavaOne 2014 San Francisco!

This year I have been honored to be selected to speak at JavaOne 2014. Awesome news! I’m excited for this opportunity to speak at possibly the larger Java developer conference in the world.

The session that was selected is CON2013: Code Generation in the Java Compiler: Annotation Processors Do the Hard Work. This session is largely based on my work around Annotation Processors as a driver for code generation.

It is really a pleasure to see that what was an experimental work three years ago, matures and get the attention of the JavaOne commitee.

So, if you happen to be at JavaOne, don’t miss the opportunity and come to the session! See you there!

Pitest: Measure the Quality of your Unit Tests with Mutation Testing

It is not uncommon among developers to discuss about the quality of automated unit tests: Are they testing enough of application code? And more importantly, are they really verifying the expected behavior?

The first question has a relatively simple answer: use automated code coverage tools that will track which lines of code and which branches in execution flow are being tested. Code coverage reports are very helpful to 1) determine which portions of application code are not being tested; and 2) if measuring code coverage per individual test, determine whether each test is effectively testing the appropriate piece of application code. If interested in techniques for that, you may want to look at this other blog post: https://deors.wordpress.com/2014/07/04/individual-test-coverage-sonarqube-jacoco/

However, no matter how useful is to measure code coverage, these reports will not let you know one fundamental aspect of tests: which behavior is being verified!

Simply put, your test code may be passing through every single line of your code, and not verifying anything. If you are familiar with JUnit framework, your test code may not contain a single assertion!

To overcome this limitation of automated unit testing, one technique that can be of great help is Mutation Testing.

Mutation Testing… Explained

Let’s assume you have your application code and your test code as usual. A mutation testing tool will take your application code and make small surgical changes, one at a time, a so-called “mutation”. It could be changing a logical operator in an if statement (e.g. > is changed to <=), it could be removing some service call, it could be changing some for loop, it could be altering some return value, and so forth.

Mutation testing is, therefore, based on the assumption that if you are testing your code and making the right assertions to verify the behavior, once you re-execute your unit tests with a mutation in application code some of them should fail.

Pitest – A Mutation Testing Framework for Java

Although very interesting, such a technique would be useless without the proper tools. There are some, and for different languages, like Jester, Jumble or NinjaTurtles, but probably the most mature and powerful we’ve seen to date is Pitest (http://pitest.org).

Working with Pitest is very simple and requires minimal effort to start. It can be integrated with build tools like Maven, Ant or Gradle, with IDEs like Eclipse (Pitclipse plug-in) or IntelliJ, and with quality tools like SonarQube.

Regardless of the way you execute it, Pitest will analyze the application byte codes and decide which mutations will be introduced (for a full list and description of available mutators in Pitest, check their site here: http://pitest.org/quickstart/mutators/).

To optimize the test execution as much as possible, Pitest gathers code coverage metrics in a “normal execution” and then re-executes only the matching test cases for a certain mutation. Total execution time is noticeably longer than a normal unit test execution, basically because the incredible test harness that Pitest adds even to the most simple of code bases.

As a result, Pitest generates a fully detail report showing which mutations “lived” after the execution, that is, which mutations where not detected by any existing assertion. These “lived” mutations are your main focus, because they mean that there is some logic, some return value, or some call that is not being verified.

Of course not all of the mutations will be meaningful. Some may produce out of memory errors or infinite loops. For those cases, Pitest does its best to detect them and remove from the resulting reports. These can be fine-tuned if needed, for example by tuning time outs and other parameters, but sensible defaults work really well to start with.

Pitest in Action

Seeing is believing, so we put Pitest to work on a simple 10-classes Java library. We decided to use the Maven plug-in, as this method requires zero configuration to start. We opened a command prompt at the project directory, and just executed this command:

> mvn org.pitest:pitest-maven:1.0.0:mutationCoverage

After a few minutes (5 to 6 for this project) and lots of iterations showing in the console, the build finishes and the reports are generated in target directory:

> target\pit-reports\201408181908\index.html

When the report loaded in the browser, the first fact that caught our attention was that one class, that we worked hard to be fully tested, AbstractContext, although with a 100% code coverage it showed one lived mutation. Oops, something was not properly verified. Was Pitest right?

mutation-1

After clicking the class name, we could see the detail on where the lived mutation was found:

mutation-2

Pitest was right! Although that method is fully tested, and there are test cases for every single execution flow, we were missing the proper assertion for that if statement. Really really hard to catch if not for a good tool helping us to find out more about our unit tests.

Of course, next step was to add the forgotten assertion to the relevant test method. Once done, we re-launched Pitest. After a few minutes, a new set of reports where created and once loaded in the browser… clean result for that class!

mutation-4

Conclusion

Although arguably a bit fortunate to obtain such a fabulous result at the first try, it is true that after a more thorough inspection of the reports we found many other places where assertions were missing.

Our view is that Pitest is a very valuable tool to write really meaningful and truly useful automated unit test suites, and should be standard gear for Java projects going forward. It is simple to use, requires zero or minimal configuration, and produces valuable results that directly impact in the quality of the test we create, and therefore in the quality of our deliverables.

To mutate, or not to mutate: that is the question.
Whether 'tis nobler in the mind to suffer
The slings and arrows of outrageous unit tests.
Follow

Get every new post delivered to your Inbox.

Join 61 other followers