The JvmModelInferrer, the debugger and Xbase in Xtext 2.3

In previous posts, I had blogged about some experiments done with Xtext 2 wonderful mechanisms to access Java types and to reuse Xbase expressions; however, when generating code, I was using the “manual way”. That is to say, I was not using the JvmModelInferrer mechanism; the mechanism of JvmModelInferrer allows to define a mapping from your DSL elements to Java elements. You’ll then have for free a complete integration of Java into your DSL, and viceversa (yes, from Java you can jump directly to your DSL elements starting from generated classes, methods, etc.). And by the way, you’ll also have for free Java code generation (corresponding to the mapping you defined in your implementation of JvmModelInferrer).

When I wrote those posts, I had done some experiments to implement my DSL Xsemantics, ; I was reluctant to use the JvmModelInferrer because the elements of Xsemantics could not be mapped directly into Java concepts (actually into element of the Jvm model of Xtext), and all the examples I had seen around using the JvmModelInferrer (e.g., the Domainmodel example shipped with Xtext itself) had a very direct mapping to Java concepts. Thus I preferred to learn how to manually generate Java code using Xtend and ImportManager functionalities.

Now, Xtext 2.3 is about to be released, and I started to try the latest milestones; the JvmModelInferrer mechanisms have been improved and one of the coolest new features of Xtext 2.3 is that by using JvmModelInferrer you also get for free debugging functionalities for your DSL (yes, you read correctly, you can debug your DSL programs source code while debugging the Java code!!!). So I decided to try to learn how to use JvmModelInferrer; I’ve just started to port Xsemantics to use JvmModelInferrer, but in the meantime I’ve done some experiments with a simple hello DSL, and I’d like to share my experiences here 🙂

This hello DSL we use here has some features that require some customizations in order to coexist with Xbase; in particular, when using the JvmModelInferrer to define the mapping most of scoping and validation of Xbase will work out of the box; however, we will have to customize something for our DSL. In particular we will have to

  1. tweak the XbaseValidator to deal with some situations which are
    1. correct in our language but not in Xbase
    2. correct in Xbase but not in our language
  2. tweak the XbaseScopeProvider to make something visible in our code
  3. implement a non direct mapping to the Jvm model in the JvmModelInferrer

Remember that you need the milestone update site for Xtext 2.3, until it is released on June:

The HelloInferrer language

The idea of this DSL (called HelloInferrer) is that you can define Hello elements (which will correspond to Java classes) with some operations which are NOT directly mapped to Java methods: in particular, each operation consists of

  1. a name
  2. a (possibly empty) list of (input) parameters (which corresponds to Java parameters)
  3. an output parameter
  4. a body

In the body

  1. the output parameter corresponds to a local variable that can be assigned;
  2. no return statement is allowed inside the body;
  3. the return value in the corresponding Java method will be a wrapper around the output parameter;
  4. the wrapper is an instance of a library class HelloResult, with a type parameter which corresponds to the Java type of the output parameter.

It’s probably easier to show an example of an input program…

… and the corresponding expected generated Java code

What’s this DSL for? Nothing! It’s just to experiment with JvmModelInferrer (and the rules you define in Xsemantics have similar functionalities).

I will try to detail the steps I followed to get to the result, of course, in a test driven fashion 🙂 In particular, we will write a class for tests related to validations and another one for tests related to code generation.

First, this is the Xtext grammar for our language

First, let’s start to implement the inferrer at least for the class corresponding to an Hello element. We expect the following Java code for the given Hello definition (we wrote the test in Xtend2, using Xtext unit test functionalities, in particular the new CompilationTestHelper, which provides useful features):

The first version of the inferrer to make this test pass is the following:

Now let’s write a test for the parsing of a HelloInferrer program which also defines an operation with a body accessing the input parameters:

This test will fail since in the body of the operation the parameters cannot be accessed. Instead of defining a custom scoping we use the JvmModelInferrer mechanism to map the operation to a Java method (we map each Operation’s parameter to a parameter in this Java method, and the Operation’s body to the body of the Java method). This will be enough to make the XbaseScopeProvider happy (for the moment) since it will be able to implement scoping correctly!

For the moment we map an Operation to a void method; then later we will deal with the result of the operation.

The parser test now succeeds, since the parameters can be found.

Now let’s see what we get during generation:

Now the generated Java code looks suspicious: we have a void method which returns something… this should not compile.

The new framework for testing code generation in Xtext 2.3 provides easy means to test whether the generated Java code actually compiles in Java; look at the CompilationTestHelper API and you’ll see. We use these API as follows:

The compile method will try to parse and validate the given source and then will invoke the generator on it; the passed closure can then access the result; in particular compiledClass will try to compile (in Java) the corresponding generated code and if it succeeds it will return the Java Class. If you now run this test it will fail since the generated code is not correct Java code. This is due to the fact that in the inferrer we specified null as the return type of the method, while we should specify a type representing void, here’s the correct inferrer:

Now, we fix the test with the corrected expected code; and since we will use this testing strategy for the generated code in other tests, we create a reusable method in the test class

Dealing with output parameter

We would like to assign values to the output parameter; however, since it is not put in the parameters of the inferred method for the operation, this test will fail with a link error, since the output parameter cannot be referred in the body of the operation

We thus need to create a custom scope provider (extending XbaseScopeProvider)

What is going on here? Well, since in our model inferrer we created a method, through the JvmTypesBuilder.toMethod (actually, a JvmOperation), for an Operation, the JvmTypesBuilder associated the Operation in the AST model to the created JvmOperation. The scope provider will not compute the scope for the original Operation: it will compute it for the associated JvmOperation, that is why we redefine the method createLocalVarScopeForJvmOperation. However, in our custom implementation we need the original Operation… we can retrieve it using an injected IJvmModelAssociations!

The previous test now passes.

Now we would like to be able to assign to the output parameter, but this test will fail

Since the Xbase validator considers by default all parameters (JvmFormalParameter) as final. Thus, we must intercept this checking in the validator, and do not generate errors if we are trying to assign to an Operation’s output parameter:

Note that the method we redefine is in XbaseJavaValidator, and our (generated) AbstractHelloInferrerJavaValidator extends XbaseJavaValidator, since our grammar reuses Xbase grammar. Now the above test succeeds.

However, let’s also make sure that we did not break anything, and that standard input parameters cannot be assigned, and that when we assign to an output parameter the type checking still works:

Generation for Output Parameter

Now we want to deal with output parameter in the generation. If we try to generate code from the following source

We’ll get an exception during generation, since the variable for the output parameter cannot be found.

From now on, things start to get more interesting in the JvmModelInferrer 🙂 Here’s how we modify it

Let’s see what’s going on here:

Instead of assigning to the body of the mapped method the body of the operation, we assign a closure; this closure is automatically passed a ITreeAppendable (a new appendable introduced in Xtext 2.3 which is used also for the debugging functionalities). The nice thing is that when this closure will be executed the passed appendable will already be configured with the binding for this (and if our language had superclasses also super would be bound) which will then work out of the box in your Operation’s method and in the generated Java code) – you might want to take a look at JvmModelGenerator.xtend::createAppendable, which is part of xbase plugin. Furthermore, since this is the body of a mapped method, the appendable which is passed to our closure will also have the bindings for the Java method’s parameters (which were associated to our Operation’s input parameters).

Then, we need to use this appendable to generate the Java declaration of the local variable which corresponds to our Operation’s output parameter:

  1. we thus declare a variable in the appendable; this variable is associated (in the appendable) to our output parameter (this will only map our output parameter to a name in the appendable, which will be used internally when the xbase compiler needs to refer to our output parameter: it will not generate anything in the output buffer);
  2. we use the TypeReferenceSerializer to generate in the Java code for the type of the local variable for our output parameter (this will also deal with possible imports in the generated Java code, since it relies on the ImportManager);
  3. we then generate the name of the local variable (using the one declared above into the appendable) and its initialization to null.

Then, we rely on an injected XbaseCompiler to generate the Java code corresponding to the Operation’s body. Since we declared a variable for the output parameter in the appendable, if the original body referred to the output parameter, the xbase compiler will be able to generate code for the use of output parameter!

Note that we initialize the generated variable for the output parameter to null; this works only if the output parameter’s type is NOT a basic (or primitive) type, e.g. int or boolean. To keep things simple we are not dealing with primitive types for output parameters (this will simplify also the generated return type). Thus, we also must make sure to rule out such use, with a validator’s check method:

Note that we rule out primitive types ONLY for output parameters. Of course, we test this as well

Dealing with generated method return type

Let’s add a class that will be used for returning the result of generated Java methods

We now change the return type of the inferred method as follows

Most tests will fail, due to Xbase validation issues; for instance, for this test program

You’ll get this validation error

java.lang.AssertionError: Expected no errors, but got :[ERROR:Incompatible implicit return type. Expected org.xtext.example.helloinferrer.runtime.HelloResult<java.lang.Boolean> but was void (__synthetic0.helloinferrer line : 2)]

Since the block is expected to return a value of type HelloResult<Boolean> (because it is contained in an inferred Java method which declares to return such type) while it does not return anything. We will need to generate the actual return statement in the Java code, but first we must also avoid that Xbase validator complains about that, thus in our validator we redefine the appropriate method:

Note that we also make sure that in our programs explicit return expressions are not allowed (since in our DSL return statements do not make sense), and of course we test it

We now modify the inferrer so that it also generates the final Java return statement (according to the generated method’s return type) creating a new HelloResult (with the type of the Operation’s output parameter as the type argument) and passing as the argument the output parameter:

Note that we check, when generating the return type, that the JvmFormalParameter is not null and also that the parameterType feature is not null; we need to check this because when writing a program in our DSL editor the inferrer is used also with a possibly not finished program and thus the JvmFormalParameter or its features can be null, for instance:

We can now test that Java code is correctly generated

We also do another test:

The result of this test shows an interesting thing: the type system of Xbase (and its generator) automatically performs boxing and unboxing for primitive types!

Another interesting test is the following:

This shows that, since the inferrer provides a “Java view” of your Greeting element and of its operations, you can already use the inferred Java elements in the Operations’ bodies: since the Operation bar’s output parameter is a MyHello, in Java this corresponds to a method returning a HelloResult<MyHello>, thus in

“value” corresponds to invoking “getValue” on the object (of type HelloResult<MyHello>) returned by the Java method inferred for bar.

Running the editor

In the implementation of the DSL there’s also a project wizard (Helloinferrer project) which creates a plugin project with an example MyHello.helloinferrer, and a Java file, that uses the Java code generated from MyHello.helloinferrer. You can see, thanks to the powerful Xbase mechanisms (tightly integrated with Java), that you can use involved Java types, and all Xbase expression syntax, like closures. In the screenshot you can see the example program (on the left), the generated Java code (on the right) and an example of Java main program using it (bottom).

Dealing with Debugging features

One of the coolest new features in Xtext 2.3 is the debugging functionalities which you get when using Xbase expressions: if you debug an application which uses the Java code generated from your DSL programs, then you can debug the original sources instead of the generated Java code! You can try that by setting breakpoints in your DSL program sources (the context menu breakpoint items are still missing, but you can double click the breakpoint ruler):

Setting the breakpoint

DISCLAIMER: what follows is just my interpretation of the debugging mechanisms of the TreeAppendable (I found no explicit documentation); debugging works with the following modified code, but I might have gotten the semantics of the trace API wrong 🙂

For the Java parts that are generated with the XbaseCompiler in our JvmModelInferrer, the debugging features already work! However, if you try to enter into one of the Operations (with Step Into) you will see that in the stack trace you get some “unknown locations” error, and the debugging gets back to work if you further press Step Into (getting to the first line of the Operation). This is due to the generated variable declaration for the output parameter.

We then must modify the inferrer so that it uses the passed appendable to trace the generation of the declaration of the Java local variable corresponding to the output parameter (and also the final generated Java return statement).

Note that when declaring the local variable we specify true as the second trace argument since we want that to be used during debugging. We don’t need that when generating the final return statement (and we pass false).

Now, create a Helloinferrer project with the wizard, and start debugging the as a Java application:

Start debuggingWhen you get to the line invoking printMappedList (which corresponds to an Operation in MyHello.helloinferrer), Step Into and see the magic:

Stepping into MyHello.helloinferrerYou see! You’re debugging the original source!

You can try to Step Over to pass to the next line

Step Overor try Resume so that the debugger stops at the breakpoint which is inside the closure (and note the value of the variable it in the Variables view)

ResumeTry Resume again, and the debugger will hit the breakpoint again (and see the changed value for it in the Variables view):

Resume again

Getting the sources

The sources can be retrieved by this git repository:

and the project is helloinferrer.

2012-06-08T11:12:11+00:00 By |


  1. nomuna Thursday August 23rd, 2012 at 12:37 PM - Reply

    Hello Mr. Bettini,

    To answer your question about the wizard… I used it. Debugging is still not working.
    1. I can set a breakpoint in printMappedList method in the MyHello.helloinferrer.
    2. When I go to the declaration of the method in MyHelloMain class I jump to MyHello.helloinferrer too.

    But when I debug the MyHelloMain class as Java application, I step into the java source… : /
    Any idea why it is not working?

    I am using ubuntu 10.04 (64bit),
    jdk1.5.0_22 ((64bit)), eclipse Indigo with xtext update 2.3.1.

    java -version on the command line gives me:

    Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_22-b03, mixed mode)


  2. Lorenzo Bettini Sunday September 9th, 2012 at 03:01 PM - Reply

    Apparently, it looks like you have to modify at least once the .helloinferrer file and save it (you can simply add a space anywhere and then remove the space, you don’t actually need to add something new). Since then, debugging the .helloinferrer file starts working 🙂

  3. Tommaso De Sica Friday January 4th, 2013 at 12:28 PM - Reply

    Is it possible to debug dsl file and generated java file in parallel?

    Very thank you!

  4. Christian Mauceri Monday October 14th, 2013 at 05:09 PM - Reply

    Hello Lorenzo,
    First of all thanks for your very interesting post and for your book.
    I have two questions.
    First, in the code I downloaded from your github repository related to this post the method returnType(JvmFormalParameter o) does not retrieve the class org.xtext.example.helloinferrer.runtime.HelloResult, what am I doing wrong in my strings ? Any idea is welcome.
    My second question is more general. I have a quite a big grammar which is in fact a super set of Java. Everything works well (more or less)
    but I bump on a problem with the content assist which is unable to make proposals when I try to have suggestions on the content of an object of a given type or on the static methods of a type. My grammar inherits from Xbase, do I have to translate all my rules in Xbase(Xtend) via JvmModelInferrer or is it possible to directly tweak XbaseProposalProvider in overloading methods in MyDSLProposalProvider ?

  5. Christian Mauceri Monday October 14th, 2013 at 05:37 PM - Reply

    Concerning the first question I imported a jar containing org.xtext.example.helloinferrer.runtime.HelloResult in the test project, and it works fine but I thought that as this class is built at the same time than the inferrer it shold be known by it.

Leave A Comment