Code Generation using Annotation Processors in the Java language – part 3: Generating Source Code

This post is the third and final part in my series about Code Generation using Annotation Processors in the Java language. In part 1 (read it here) we introduced what Annotations are in the Java language and some of their common uses. In part 2 (read it here) we introduced Annotation Processors, how to build them and how to run them.

Now, in part 3, we are going to show how an Annotation Processor can be used to generate source code.

Code Generation using Annotation Processors in the Java language – part 3: Generating Source Code

Generating source code is easy. Generating the right code is not. And doing it in an elegant and efficient way can be a cumbersome task!

Fortunately, in the last years Model-Driven Engineering (1) (MDE, sometimes referred also as Model-Driven Development or Model-Driven Architecture) has helped to evolve a practice that was more art than science – task for ninja coders – into a mature methodology strongly based on proven processes and tools.

Model-Drive Engineering is much more than generating source code, although we can think of it as a natural entry point to MDE methodologies.

Annotation Processors is one of the many tools that we may use to generate source code.

Model and Meta-model in MDE

Before going into the details on how to generate source code using Annotation Processors, there is a couple of concepts that we would like to present, as we will be referring to them in the following slides: models and meta-models.

One of the pillars of MDE is the construction of abstractions. We model the software system that we intent to build at different levels of detail and with different approaches. When one level of abstraction is modeled, we can start to model the next one and the next, ending with a complete, deployable product.

In this context, a model is no more than the abstraction that we use to represent our system, whatever level of detail we are using.

The meta-model, then, are the rules that we use to write our models. You can think of it as the schema or semantics of models.

Generating source code with Annotation Processors

From what we have seen until now, Annotations are an excellent way for defining a meta-model and creating models. Annotation Types will act as meta-models, while a set of Annotations in a given piece of code will act as a model.

We can leverage this model to generate configuration files or to write new source files that are derived from one existing. For example, create a remoting proxy or a data access object from an annotated bean.

The central piece in this approach is the Annotation Processor. A processor will be able to read annotations found in source code – that is, extract the model – and do whatever we need to do with it – open a file, put contents on it. The Java Compiler will take care of validating the model (annotations should match the defined types).

The Filer

As discussed in part 2, every processor have access to a processing environment that points to some interesting utilities. One of them is the Filer.

The javax.annotation.processing.Filer (2) interface contract defines some methods for creating source files, class files or generic resources. By using the Filer we can be sure of using the right directories and that we are not losing valuable generated items in our file system.

This concern is critical if we want to write generators that honors -d and -s options in javac or directories defined in a Maven POM.

The following example shows how to create a Java source file inside an Annotation Processor. The generated class name will be the same as the annotated class name plus the string “BeanInfo”, as if we were generating a Bean information class:

 if (e.getKind() == ElementKind.CLASS) {
     TypeElement classElement = (TypeElement) e;
     PackageElement packageElement =
         (PackageElement) classElement.getEnclosingElement();

     JavaFileObject jfo = processingEnv.getFiler().createSourceFile(
         classElement.getQualifiedName() + "BeanInfo");

     BufferedWriter bw = new BufferedWriter(jfo.openWriter());
     bw.append("package ");
     bw.append(packageElement.getQualifiedName());
     bw.append(";");
     bw.newLine();
     bw.newLine();
     // rest of generated class contents

Don’t Generate Like my Brother

The previous example is simple, interesting, but a mess!

We are mixing the logic of getting the information we need from annotations (the model) with the logic that writes the generated file (the view).

It is very difficult to write a decent generator in that way. If we need something more complex the task can be cumbersome, error prone and hard to maintain.

We need, therefore, a more elegant approach:

  • Separate clearly model from view.
  • Use templates to ease the task of writing down the generated file.

Let’s see as an example of this approach how to leverage Apache Velocity to build generators the way we want.

A Brief History of Velocity

Velocity, a project from the Apache Software Foundation, is a template engine written in Java that produces all types of text files by mixing a template with data from Java objects.

Velocity has been used historically to render views following the popular MVC pattern, or as a substitute to XSLT to transform data in XML files.

Velocity has its own language named the Velocity Template Language (VTL) that is key to produce rich and easy to read templates. In VTL we can define variables, control flow and iterations and access information contained in Java objects in a simple and intuitive way.

Below you can see a fragment of a Velocity Template:

 #foreach($field in $fields)
     /**
      * Returns the ${field.simpleName} property descriptor.
      *
      * @return the property descriptor
      */
     public PropertyDescriptor ${field.simpleName}PropertyDescriptor() {
         PropertyDescriptor theDescriptor = null;
         return theDescriptor;
     }
 #end
 #foreach($method in $methods)
     /**
      * Returns the ${method.simpleName}() method descriptor.
      *
      * @return the method descriptor
      */
     public MethodDescriptor ${method.simpleName}MethodDescriptor() {
         MethodDescriptor descriptor = null;

As you can see, VTL is very straightforward and easy to understand. Highlighted in bold, you can see two typical VTL constructs: iterating a collection of objects and print some property from each element found in the collection.

The Velocity Generator Approach

Now that we have decided to use Velocity to enhance our generator, we have to redesign it to follow this schema:

  • Write the template that will be used to generate the code.
  • The Annotation Processor will read from the round environment the annotated elements and save them in easy to access Java objects – a map for fields, a map for methods, the class and package names and so forth.
  • The Annotation Processor will initialize the Velocity context.
  • The Annotation Processor will load the Velocity template.
  • The Annotation Processor will create the source file (using the Filer) and pass a writer to it to the Velocity Template, along with the Context.
  • The Velocity engine will generate the source code.

By following this approach you will find that the processor/generator code is clear, well structured and easy to understand and maintain.

Let’s proceed step by step.

Step 1: Write the Template

For the sake of simplicity we are not going to show a full BeanInfo generator, just a subset of the fields and methods that are actually required will be built with our Processor.

Let’s create a file named beaninfo.vm and place it under the src/main/resources of the Maven processor artifact. An example of the contents for the template are:

package ${packageName};

import java.beans.MethodDescriptor;
import java.beans.ParameterDescriptor;
import java.beans.PropertyDescriptor;
import java.lang.reflect.Method;

public class ${className}BeanInfo
    extends java.beans.SimpleBeanInfo {

    /**
     * Gets the bean class object.
     *
     * @return the bean class
     */
    public static Class getBeanClass() {

        return ${packageName}.${className}.class;
    }

    /**
     * Gets the bean class name.
     *
     * @return the bean class name
     */
    public static String getBeanClassName() {

        return "${packageName}.${className}";
    }

    /**
     * Finds the right method by comparing name & number of parameters in the class
     * method list.
     *
     * @param classObject the class object
     * @param methodName the method name
     * @param parameterCount the number of parameters
     *
     * @return the method if found, <code>null</code> otherwise
     */
    public static Method findMethod(Class classObject, String methodName, int parameterCount) {

        try {
            // since this method attempts to find a method by getting all
            // methods from the class, this method should only be called if
            // getMethod cannot find the method
            Method[] methods = classObject.getMethods();
            for (Method method : methods) {
                if (method.getParameterTypes().length == parameterCount
                    && method.getName().equals(methodName)) {
                    return method;
                }
            }
        } catch (Throwable t) {
            return null;
        }
        return null;
    }
#foreach($field in $fields)

    /**
     * Returns the ${field.simpleName} property descriptor.
     *
     * @return the property descriptor
     */
    public PropertyDescriptor ${field.simpleName}PropertyDescriptor() {

        PropertyDescriptor theDescriptor = null;
        return theDescriptor;
    }
#end
#foreach($method in $methods)

    /**
     * Returns the ${method.simpleName}() method descriptor.
     *
     * @return the method descriptor
     */
    public MethodDescriptor ${method.simpleName}MethodDescriptor() {

        MethodDescriptor descriptor = null;

        Method method = null;
        try {
            // finds the method using getMethod with parameter types
            // TODO parameterize parameter types
            Class[] parameterTypes = {java.beans.PropertyChangeListener.class};
            method = getBeanClass().getMethod("${method.simpleName}", parameterTypes);

        } catch (Throwable t) {
            // alternative: use findMethod
            // TODO parameterize number of parameters
            method = findMethod(getBeanClass(), "${method.simpleName}", 1);
        }

        try {
            // creates the method descriptor with parameter descriptors
            // TODO parameterize parameter descriptors
            ParameterDescriptor parameterDescriptor1 = new ParameterDescriptor();
            parameterDescriptor1.setName("listener");
            parameterDescriptor1.setDisplayName("listener");
            ParameterDescriptor[] parameterDescriptors = {parameterDescriptor1};
            descriptor = new MethodDescriptor(method, parameterDescriptors);

        } catch (Throwable t) {
            // alternative: create a plain method descriptor
            descriptor = new MethodDescriptor(method);
        }

        // TODO parameterize descriptor properties
        descriptor.setDisplayName("${method.simpleName}(java.beans.PropertyChangeListener)");
        descriptor.setShortDescription("Adds a property change listener.");
        descriptor.setExpert(false);
        descriptor.setHidden(false);
        descriptor.setValue("preferred", false);

        return descriptor;
    }
#end
}

Note that for this template to work, we will need to pass to Velocity the following information:

  • packageName: the full package name of the generated class
  • className: the name of the generated class
  • fields: a collection containing the fields in the source class; for each field we will need:
    • simpleName: the name of the field
    • type: the type (not used in the example)
    • description: self-explanatory (not used in the example)
  • methods: a collection containing the methods in the source class; for each method we will need:
    • simpleName: the name of the method
    • arguments: the method arguments (not used in the example)
    • returnType: the method return type (not used in the example)
    • description: self-explanatory (not used in the example)

All this information (the model) will be extracted from annotations found in the source class and stored in JavaBeans to be passed to Velocity.

Step 2: The Processor Reads the Model

Let’s create the Processor and don’t forget to annotate it to process a BeanInfo annotation type, as explained in part 2:

@SupportedAnnotationTypes("example.annotations.beaninfo.BeanInfo")
@SupportedSourceVersion(SourceVersion.RELEASE_6)
public class BeanInfoProcessor
    extends AbstractProcessor {

The process method will need to extract information needed to build the model from the annotations and the source class itself. You can use JavaBeans to store as much information as needed, but in this example we will leverage javax.lang.model.element types as we are not planning to send too much details to Velocity (but would need, in the case we were going to build a full BeanInfo generator):

            String fqClassName = null;
            String className = null;
            String packageName = null;
            Map<String, VariableElement> fields = new HashMap<String, VariableElement>();
            Map<String, ExecutableElement> methods = new HashMap<String, ExecutableElement>();

            for (Element e : roundEnv.getElementsAnnotatedWith(BeanInfo.class)) {

                if (e.getKind() == ElementKind.CLASS) {

                    TypeElement classElement = (TypeElement) e;
                    PackageElement packageElement = (PackageElement) classElement.getEnclosingElement();

                    processingEnv.getMessager().printMessage(
                        Diagnostic.Kind.NOTE,
                        "annotated class: " + classElement.getQualifiedName(), e);

                    fqClassName = classElement.getQualifiedName().toString();
                    className = classElement.getSimpleName().toString();
                    packageName = packageElement.getQualifiedName().toString();

                } else if (e.getKind() == ElementKind.FIELD) {

                    VariableElement varElement = (VariableElement) e;

                    processingEnv.getMessager().printMessage(
                        Diagnostic.Kind.NOTE,
                        "annotated field: " + varElement.getSimpleName(), e);

                    fields.put(varElement.getSimpleName().toString(), varElement);

                } else if (e.getKind() == ElementKind.METHOD) {

                    ExecutableElement exeElement = (ExecutableElement) e;

                    processingEnv.getMessager().printMessage(
                        Diagnostic.Kind.NOTE,
                        "annotated method: " + exeElement.getSimpleName(), e);

                    methods.put(exeElement.getSimpleName().toString(), exeElement);
                }
            }

Step 3: Initialize the Velocity Context and Load the Template

The following code snippet show how to initialize the Velocity context and load the template:

            if (fqClassName != null) {

                Properties props = new Properties();
                URL url = this.getClass().getClassLoader().getResource("velocity.properties");
                props.load(url.openStream());

                VelocityEngine ve = new VelocityEngine(props);
                ve.init();

                VelocityContext vc = new VelocityContext();

                vc.put("className", className);
                vc.put("packageName", packageName);
                vc.put("fields", fields);
                vc.put("methods", methods);

                Template vt = ve.getTemplate("beaninfo.vm");

The velocity configuration file, named in this example velocity.properties, should be placed under src/main/resources folder. An example of its contents is below:

runtime.log.logsystem.class = org.apache.velocity.runtime.log.SystemLogChute

resource.loader = classpath
classpath.resource.loader.class = org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader

This set of properties configures a log for Velocity and a classpath-based resource loader to find the template.

Step 4: Create the New Source and Generate the Contents

Finally, let’s create a new source file and run the template using this new file as the target. The following code snippet shows how to do it:

                JavaFileObject jfo = processingEnv.getFiler().createSourceFile(
                    fqClassName + "BeanInfo");

                processingEnv.getMessager().printMessage(
                    Diagnostic.Kind.NOTE,
                    "creating source file: " + jfo.toUri());

                Writer writer = jfo.openWriter();

                processingEnv.getMessager().printMessage(
                    Diagnostic.Kind.NOTE,
                    "applying velocity template: " + vt.getName());

                vt.merge(vc, writer);

                writer.close();

Step 5: Package and run

Finally, register the processor (remember the service configuration file shown in part 2) package it and invoke it from a client project compilation command, Eclipse or Maven build.

Assuming the following client class:

package example.velocity.client;
import example.annotations.beaninfo.BeanInfo;
@BeanInfo public class Article {
    @BeanInfo private String id;
    @BeanInfo private int department;
    @BeanInfo private String status;
    public Article() {
        super();
    }
    public String getId() {
        return id;
    }
    public void setId(String id) {
        this.id = id;
    }
    public int getDepartment() {
        return department;
    }
    public void setDepartment(int department) {
        this.department = department;
    }
    public String getStatus() {
        return status;
    }
    public void setStatus(String status) {
        this.status = status;
    }
    @BeanInfo public void activate() {
        setStatus("active");
    }
    @BeanInfo public void deactivate() {
        setStatus("inactive");
    }
}

When we issue the javac command, we can see in the console that annotated elements are found and the BeanInfo class is generated:

Article.java:6: Note: annotated class: example.annotations.velocity.client.Article
public class Article {
       ^
Article.java:9: Note: annotated field: id
    private String id;
                   ^
Article.java:12: Note: annotated field: department
    private int department;
                ^
Article.java:15: Note: annotated field: status
    private String status;
                   ^
Article.java:53: Note: annotated method: activate
    public void activate() {
                ^
Article.java:59: Note: annotated method: deactivate
    public void deactivate() {
                ^
Note: creating source file: file:/c:/projects/example.annotations.velocity.client/src/main/java/example/annotations/velocity/client/ArticleBeanInfo.java
Note: applying velocity template: beaninfo.vm
Note: example\annotations\velocity\client\ArticleBeanInfo.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

If we check in the sources directory we will find there the BeanInfo class as expected. Mission accomplished!

Conclusion

During this series we have learned the basis about how to generate source code by leveraging the Annotation Processor framework in Java 6:

  • We have learned what are Annotations and Annotation Types and what are the common uses for them.
  • We have learned what are Annotation Processors, how to write them and how to execute them from different tools – the Java Compiler, Eclipse and Maven.
  • We have discussed a bit about Model-Drive Engineering and code generation.
  • We have shown how Annotation Processors can be used to create source code generators that are fully integrated with the Java Compiler.
  • We have shown how to leverage existing generation frameworks as Apache Velocity to create elegant, powerful and maintainable source code generators using Annotation Processors.

Now it is time to apply this to your projects. Think Generate!

(1) If you want to know more about MDE, visit this article in Wikipedia and follow the references.

(2) The Filer API documentation can be reviewed online here.

Author: deors

senior technology architect in accenture, with a passion for technology related stuff, celtic music and the best sci-fi, among other thousand things!

21 thoughts on “Code Generation using Annotation Processors in the Java language – part 3: Generating Source Code”

  1. This was great! Thanks. What I’m missing is using this code generation to extend functionality of the annotated class. In my scenario I would like the generated class to be like the implementation of an interface so the annotated class has methods defined by inheritance. That means I would need to make sure the annotated class extends the implementation. Any idea on this?

    I believe something like this is done on @WebService.

    1. With this method you can’t extend functinoality of an existing class, only generate new files (it may be new classes or configuration files). One possible use case is to create an implementation from an annotated interface, for example.
      If you want to modify the bytes of an existing class, I think you should consider using dynamic proxies or a bytecode manipulator as ASM (http://asm.ow2.org/), CGLib (http://cglib.sourceforge.net/) or Javassist (http://www.jboss.org/javassist).

      1. I’m giving dynamic proxies a try. It seems like what I need. Thanks!

  2. Fantastic tutorial series! Thanks.

    I have a code-gen framework I built for automating some tasks, and I am wondering if it worth converting it to use these APIs. With my existing approach I have the limitations of (a) requiring runtime annotations – making the build dependent on running a separate generator application, and (b) I had to implement my own versions of the TypeMirrors and the util functions to go with them (I couldn’t find any information on the javax.lang.model packages) as I need the ability to do comparisons between existing types and to-be-generated types – that was amazingly painful and I would love to get away from having to maintain the code I wrote to deal with stuff like checking assignment between wilcard bounds and union types – ouch!

    In theory using the javax.annotation.processing classes will get rid of the first limitation, and using the javax.lang.model classes will get rid of the second one. However it’s not clear from the examples I have been able to find whether I can implement two critical components of the solution:

    1) Currently I transform the classes scanned into a ‘repository’ as models based on the combination of the class structures and the annotations – and then these are transformed via a series of plugin repository processors and sent to other repositories as derivative models in a processing pipeline. For this to work using the annotation processor I would need at a minimum two stages, the ‘load’, then the ‘check/transform/output’ stage. I can’t see from any examples how to make this split.

    2) Secondly, if I can’t use the annotation processing classes due to the multi-stage approach not being possible with the annotation processor api, I can’t see an approach to get into the model classes from reflection. I also see that this: http://openjdk.java.net/jeps/119 implies that it’s yet not possible.

    I wonder if you have any pointers to some additional information I could use to determine whether my use-case is supportable using theses APIs?

    1. Ok, I can answer the first one.

      Further explorations have shown me that the annotation processor is called once-per-compile (per round), not once-per-class as I first thought. Making the first option directly supportable.

      I will presume that the second use case won’t be directly required and will be possible at some time in the future.

    1. If you are interested in directly manipulating source code I think that annotation processors will not be able to help. If you also discard using a bytecode manipulation tool like ASM, CGLib or Javassist, then probably your best bet is to leverage another ‘code generation’ mechanisms like Eclipse Jet transforms or a ‘generic pre-processor’. If you use Maven, you can create a Mojo (Maven plug-in) and bind it to generate-sources phase, and it will be triggered before compilation occurs. This is for example the mechanism used by ANTLR to create a parser class from an existing grammar file (http://maven.apache.org/guides/mini/guide-generating-sources.html). I think that some JDO frameworks as DataNucleus also use that approach to ‘enhance’ existing sources before compilation (http://www.datanucleus.org/).

  3. I learn a lot of things with your tutorial! I suceed to generate my class with my annotations. Thanks a lot, it’s work well !

  4. I’m having trouble getting the processor to run with a maven build; do you have a source zip or tar ball somewhere?

  5. Thank you for sharing these great articles. They serve a clean concepts and helpful examples about Annotations and Annotation Processors.

    1. Hi, not sure about what do you mean. Do you refer to debug the javac process where the Annotation Processors are running (and generating code)?

  6. Nice tutorial but you shouldn’t have to worry about these types of things anymore… it’s low-level and time consuming. Use the Jigy
    Generator to create your Spring project and spring will be completely configured for you. Jigy Generator also reverse engineers
    your database to create all your DAO’s, domain objects and validators… Plus you have certain things that just work out
    of the box like login, authentication, file upload etc. The project can be downloaded at http://www.getjigy.com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s