This week I deployed a DSL with its corresponding code generator for a client. The DSL and corresponding editor was written in Xtext and the code generator in Xpand/Xtend.
I was thinking through various options for deploying the editor and code-generator and in the process I investigated the options in Xtext. At first I was very disappointed in the options, but then Peter Friese gave me a short lesson (for which I owe him a(nother) beer :)).
Xtext has a default suggestion for how to integrate the editor and the code generator, but I’m not sure if the option is easily obtainable through the documentation. To help others, I thought it would be a good idea to write down what I learned in this blog-entry.

Description of Problem

After you have created your DSL, its editor and the code expansion templates, you need to decide how to deploy your solution to the users of the DSL. The deployment mechanism is the use of Eclipse plug-ins, but there are a few variables. In particular, it is not clear how you connect the editor and the code generator.
There are a few common scenarios for how to perform the code generation. They are not mutual exclusive (that is, you may want to support multiple options):

• Activation of the code generation from the editor, e.g.:
  
  • The user right-click inside the editor and selects Generate Code
  • The user right-click on the DSL file and select Generate Code
  • A wizard
  
  
• You provide a Eclipse Modeling Workflow that then selects the model file and calls the expansion templates
  
  • Ideally we should now have a way for the DSL user to create this file (a new wizard)
  
  
• Incremental build
  
  • Hook the code generation up with the incremental build in Eclipse as for instance the JDT does (When a java file is saved, the incremental build automatically invokes the compiler to produce the class files)
  
  
• Headless invocation of the code generator targeted for build tools such as:
  
  • Maven
  • ANT
  
  

The question is, which one do you choose and what do you have to do to achieve it?

What Xtext Supports!

Xtext provides one integration option for the code generation in its current version. This  option involves hardly any work at all. When you use the Xtext wizard to create a new Xtext project, the mwe (Eclipse Modeling Workflow) that Xtext create for you has an option for generating a '”new wizard” that is commented out (why it is commented out, I don’t know).

<!-- project wizard (optional) -->
<fragment class="org.eclipse.xtext.ui.generator.projectWizard.SimpleProjectWizardFragment" generatorProjectName="${projectName}.generator" modelFileExtension="sdol"/>

By enabling the new project wizard, the Xtext generator will build an extension in the U/I that constructs a “new project” wizard. That is, when the U/I plug-in is installed, you’ll be able to create a new project particularly tailored to generate code. You can get to this “New Project” wizard by selecting:

File => New… => Project…

and then navigate to the Xtext tab. The option will be called “<MYDSL> Project” where MYDSL is the name of your DSL. In the wizard below, you’ll see “SDOL project”, which happens to be the name of a language that I recently created.

After having chosen to create your DSL project type, you’ll be presented with the generated “New Project” dialog that basically just asks you to give the project a name:



The resulting project has been configured to enable code generation. It is a plug-in project and it uses the plug-in dependencies to load the required plug-ins required to execute the Xpand/Xtend templates you created for your DSL using Eclipse Modeling Workflow. The plug-ins required are:

• com.ibm.icu
  
• org.eclipse.xtext.log4
  
• org.eclipse.xtext
  
• org.eclipse.xtext.generator
  
• org.eclipse.xtend
  
• org.eclipse.xtend.typesystem.emf
  
• org.eclipse.xpand
  
• org.apache.commons.logging
  
• de.itemis.xtext.antlr
  
• org.eclipse.emf.codegen.ecore
  
• org.eclipse.xtend.util.stdlib,
  
• <MYDSL>.generator

You can configure the way the way the project is initialized using some expansion templates in the U/I plug-in. In the U/I project you’ll find an Xpand template called <MYDSL>NewProject.xpt. This file determines the content of a new project. The default expansion template is shown below, but you can obviously change this template to be whatever you want it to be.

«IMPORT my::project::dsl::ui::wizard»
«DEFINE main FOR SDOLProjectInfo»
«EXPAND model FOR this»
«EXPAND workflow FOR this»
«ENDDEFINE»
«DEFINE model FOR SDOLProjectInfo»
«FILE "src/Model.sdo"-»
/*

* This is an example model
*/
type String
entity Leaf extends Composable {
property name: String
}
entity Composite extends Composable {
property content: Composable[]
}
entity Composable {
}
«ENDFILE»
«ENDDEFINE»
«DEFINE workflow FOR SDOLProjectInfo»
«FILE "src/Generator.mwe"-»
<workflow>
<property name="modelFile" value="Model.sdo"/>
<property name="targetDir" value="../src-gen"/>
<workflow file="workflow/SDOLGenerator.mwe"/>
</workflow>
«ENDFILE»
«ENDDEFINE»

When the Xtext Approach does not work…

The approach above should work well in most cases. There is one case where the approach above is problematic (which of course happens to be the one I’m currently are working one). The approach above only works if you are targeting the same Eclipse version as the one you’re using as your IDE.
In my case, my client is targeting an older version of eclipse than the one they use as their IDE. This is not an uncommon scenario. When building Rich Client Platforms (RCP) in Eclipse, you typically want to setup the target platform to a '”fixed version”, but at the same time, you need to leverage the latest Eclipse tools. Because the approach mentioned above leverage the Eclipse dependency mechanism, Eclipse will look for the required plug-ins in the target platform (as opposed to the current IDE).
I actually think the current Xtext approach is a bit of kludge because it uses a run-time dependency setup for what should be resolved at compile/build time , but I can see the convenience of the approach. Also, it probably works for most projects.

How to Work around the Kludge?

To work around the dependency problem I have at my client site, I’ll have to ensure that all the code execute using the IDE plug-ins. That means, I cannot setup the code-generation project using the eclipse plug-in dependencies. I have two choices:

1. Manually managed dependencies
   
   
   
   
   • Setup the code generation project as a simple Java project
     
   • Setup the classpath to point to each of the required plug-in jar files mentioned above
   
   
2. Execute the workflow from within the installed IDE plug-ins
   
   
   
   
   • Write one (or more) U/I extensions to <MYDSL>.ui plugin that activates a code generation task from within the installed plug-in
     
   • Write preference pages for configurable options in the plug-ins for options that are typically setup through the workflow file (e.g., target directory for the files, model file)
     

I’m probably going down the option 2 path and I’ll write up something on my blog when I’m done. Option 1 is of course the easier of the options, but it requires a rather fragile setup of the classpath.

Other Common Code Generation Options

Here are some other code-generation options that you may want to consider:

1. Incremental build of Eclipse
   
2. Maven
   
3. ANT
   
4. Integrated customized
   

Incremental Build

It will take too long to write up this option. It requires understanding of how to setup builders, natures and also how to read resource deltas in Eclipse. The good news is that the Xtext team is currently working on this option. You can read about the plans for this here.
In many projects, this will probably be the preferred option as it fits well with the way Eclipse works for most other tools/languages. That is, as soon as you save your DSL file, eclipse would automatically launch your code-generation script to generate code in the background.

Integration with Maven

Many rely on Maven to do their build. I’ll not argue for nor against this approach, but it may very well be that you need to integrate a code generation task with Maven. Karsten Thoms has written up an article of how to integrate Xtext with Maven that you can read here.

Integration with ANT

Another popular language for build scripts is ANT. You can read about the ANT integration here.

Conclusion

When using Xtext, you need to devise a plan for how to integrate code-generation for the users of your DSL. That is, the user must be able to initialize the code-generation step after having produced the DSL file using your editor.
There are several options for generation. You may want to let the users activate the code-generation in their IDE. You may also want to provide a method for them to run their code generation as part of a build script. All options are achievable in Xtext.
Xtext supports a default integration of code-generation using the “New Project” wizard to create a project setup for code generation. This project uses the plug-in dependencies to activate required plug-ins. This will not work if you target a different eclipse run-time version than the one used for your IDE.
The Xtext team is planning an integration using the incremental build which may work around the Eclipse versioning problem.
You can integrate code-generation step into the editor (e.g., context menus etc.). Integration is relatively straight forward if you know how to write U/I plug-ins as the workflow can be executed from memory (no mwe files).