Xtext 2.1: using Xbase expressions

On a previous post, I started to blog about how Xtext 2.1 made it extremely easy to have a powerful integration with Java in your DSL. In this post I’d like to continue to experiment with this mechanism, and in particular I’ll start to use Xbase, a new expression language library which allows to integrate (Java-like) expressions in your DSL.

If you take a look at the Five simple steps to your JVM language, you’ll see the power of Xbase! Moreover, in Xtext 2.1, using Xbase is even easier since by writing only a AbstractModelInferrer you’ll get a full integration with Java (by mapping your AST model elements to Java concepts) and a generator (which will produce Java code corresponding to this mapping). Thus, you won’t even need a generator at all.

However, in this post, I’d like to inspect how to use only a small part of Xbase and still have the control on the generation part: in particular (for other projects) I would like to retain the control on the generation for my model, while relying on the Xbase generation for the Xbase parts. Thus, in this post I’ll describe:

  • how to integrate Xbase expressions (XExpression) in your DSL
  • write a generator for your DSL and reuse the XbaseCompiler for the code of XExpressions

Following the spirit of the previous post, I won’t use the Domainmodel example, but something really simple, like the Greeting example, i.e., the very basic DSL you will get when creating an Xtext project inside Eclipse. In particular, you might want to go through the previous post before reading this post (since we reuse some of the concepts seen there).

So, first of all, create a new Xtext project; I’ll call this project org.xtext.example.helloxbase (and the files will have extension helloxbase). Now, we can start generate Xtext artifacts. With the defaults, you’ll also get a HelloXbaseGenerator.xtend, which we now to generate Java code. The first implementation of the generator is: for each Greeting element we will obtain a package and a class named according to the Greeting name feature (package all lower case, and class name with the first letter capital). Moreover, the generated main method will print the corresponding greeting:

package org.xtext.example.helloxbase.generator

import org.eclipse.emf.ecore.resource.Resource
import org.eclipse.xtext.generator.IFileSystemAccess
import org.eclipse.xtext.generator.IGenerator
import org.xtext.example.helloxbase.helloXbase.Greeting

import static extension org.eclipse.xtext.xtend2.lib.ResourceExtensions.*

class HelloXbaseGenerator implements IGenerator {

	override void doGenerate(Resource resource, IFileSystemAccess fsa) {
		for(greeting: resource.allContentsIterable.filter(typeof(Greeting))) {
			fsa.generateFile(
				greeting.packageName + "/" + // package
				greeting.className + ".java", // class name
				greeting.compile)
		}
	}

	def compile(Greeting greeting) '''
	package «greeting.packageName»;

	public class «greeting.className» {
		public static void main(String args[]) {
			System.out.println("Hello «greeting.name»");
		}
	}
	'''

	def className(Greeting greeting) {
		greeting.name.toFirstUpper
	}

	def packageName(Greeting greeting) {
		greeting.name.toLowerCase
	}
}

If you’re already familiar with Xtend2 (or if you read the previous post), you should have a clue about what the generator does; let’s try the generator: restart your second eclipse instance, and make sure that in the plugin project you had created with the helloxbase file, you have a folder named src-gen, and that folder is configured as a source folder. If you type something like

Hello foo!
Hello bar!

you should see some Java code generated in the src-gen folder (according to the schema detailed above).

Now, the idea is to enrich the HelloXbase greeting DSL with some Xbase expressions, thus, we modify the grammar of the language (HelloXbase.xtext) as follows (you should be familiar with ‘import’ functionalities of Xtext):

grammar org.xtext.example.helloxbase.HelloXbase with 
	org.eclipse.xtext.xbase.Xbase

generate helloXbase "http://www.xtext.org/example/helloxbase/HelloXbase"

Model:
	imports += Import*
	greetings+=Greeting*;

Import:
  'import' importedNamespace = QualifiedNameWithWildcard
;

QualifiedNameWithWildcard:
  QualifiedName '.*'?
; 

Greeting:
    'Hello' name=ID 'from' expression = XExpression '!'
;

Note that we now derive our grammar from the Xbase grammar, so we can now rely on Xbase grammar rules; the idea is to be able to write in our DSL something like

Try it yourself: regenerate the Xtext artefacts, and restart the runtime eclipse instance. To fully enjoy Xbase syntax, please make sure that in the project you created in the runtime workspace you also have org.eclipse.xtext.xbase.lib as a dependency in the MANIFEST.

You can see lots of cool stuff here! You basically have all the powerful Xbase expression Java-like syntax, access to static methods using ::, closures, and extension methods (the nullOrEmpty from StringExtensions)! All for free!

Now, what happened to our generator? Is it still invoked automatically on resource changes? I’m afraid not: as soon as you derived your grammar from Xbase, a new file appeared in your project, HelloXbaseJvmModelInferrer.xtend, since in Xtext 2.1, as hinted at the beginning, the generation strategy changed. In particular, in the src-gen AbstractHelloXbaseRuntimeModule the IGenerator is now bound to org.eclipse.xtext.xbase.compiler.JvmModelGenerator; since we want our own generator back, we simply override this binding in HelloXbaseRuntimeModule (the one we’re allowed to modify):

public class HelloXbaseRuntimeModule extends org.xtext.example.helloxbase.AbstractHelloXbaseRuntimeModule {

	/**
	 * Avoids to use the default org.eclipse.xtext.xbase.compiler.JvmModelGenerator
	 * when using xbase.
	 * @see org.xtext.example.helloxbase.AbstractHelloXbaseRuntimeModule#bindIGenerator()
	 */
	@Override
	public Class<? extends IGenerator> bindIGenerator() {
		return HelloXbaseGenerator.class;
	}

}

If you restart your runtime instance you’ll see our generator is back!

Now, we want to generate the Java code corresponding to the Xbase XExpression; we then simply reuse the XbaseCompiler in our generator (which, in Xtext, you know, means ‘have it injected’). In the generator we will now use also the ImportManager (which is explained in the previous post) and start to do some experiments with the XbaseCompiler:

package org.xtext.example.helloxbase.generator

import com.google.inject.Inject
import org.eclipse.emf.ecore.resource.Resource
import org.eclipse.xtext.common.types.TypesFactory
import org.eclipse.xtext.generator.IFileSystemAccess
import org.eclipse.xtext.generator.IGenerator
import org.eclipse.xtext.xbase.XExpression
import org.eclipse.xtext.xbase.compiler.ImportManager
import org.eclipse.xtext.xbase.compiler.StringBuilderBasedAppendable
import org.eclipse.xtext.xbase.compiler.XbaseCompiler
import org.xtext.example.helloxbase.helloXbase.Greeting

import static extension org.eclipse.xtext.xtend2.lib.ResourceExtensions.*

class HelloXbaseGenerator implements IGenerator {

	@Inject
	protected XbaseCompiler xbaseCompiler

	override void doGenerate(Resource resource, IFileSystemAccess fsa) {
		for(greeting: resource.allContentsIterable.filter(typeof(Greeting))) {
			fsa.generateFile(
				greeting.packageName + "/" + // package
				greeting.className + ".java", // class name
				greeting.compile)
		}
	}

	def getJvmType(Greeting greeting) {
		val declaredType = TypesFactory::eINSTANCE.createJvmGenericType
		declaredType.setSimpleName(greeting.className)
		declaredType.setPackageName(greeting.packageName)
		declaredType
	}

	def compile(Greeting greeting) '''
	«val importManager = new ImportManager(true, getJvmType(greeting))»
	«val mainMethod = compile(greeting, importManager)»
	package «greeting.packageName»;
	«IF !importManager.imports.empty»

	«FOR i : importManager.imports»
		import «i»;
	«ENDFOR»
	«ENDIF»

	«mainMethod»
	'''

	def compile(Greeting greeting, ImportManager importManager) '''
	public class «greeting.className» {
		public static void main(String args[]) {
			«compile(greeting.expression, importManager)»

			System.out.println("Hello «greeting.name» from ");
		}
	}
	'''

	def compile(XExpression xExpression, ImportManager importManager) {
		val result = new StringBuilderBasedAppendable(importManager)
		xbaseCompiler.toJavaStatement(xExpression, result, true)
		result
	}

	def className(Greeting greeting) {
		greeting.name.toFirstUpper
	}

	def packageName(Greeting greeting) {
		greeting.name.toLowerCase
	}
}

Let’s concentrate on these parts:

	def compile(Greeting greeting, ImportManager importManager) '''
	public class «greeting.className» {
		public static void main(String args[]) {
			«compile(greeting.expression, importManager)»

			System.out.println("Hello «greeting.name» from ");
		}
	}
	'''

	def compile(XExpression xExpression, ImportManager importManager) {
		val result = new StringBuilderBasedAppendable(importManager)
		xbaseCompiler.toJavaStatement(xExpression, result, true)
		result
	}

We tried to use the toJavaStatement of XbaseCompiler, which seems a good candidate for what we want; in particular, by passing true as the last argument we say that the “result” of generation should be assigned to a Java variable, so that we can refer to it; let’s see what happens in the runtime workspace with this generation:

Hey, cool! The XbaseCompiler generates all the Java statements corresponding to the original XExpression (including closures)! Now we would only like to access the last variable generated (in the examples above, _isNullOrEmpty and _apply, respectively), so that we can generate some Java code ourselves using that variable; but how can we know its name?

Well, the StringBuilderBasedAppendable class not only contains the result of the generation; it also contains a map with all the intermediate Java local variables generated by the compiler. Each variable, in the map, has as the key the corresponding Xbase object. So we only need to know the variable name corresponding to our XExpression and we’re done:

result.getName(greeting.expression)

So the final generator looks like this:

package org.xtext.example.helloxbase.generator

import com.google.inject.Inject
import org.eclipse.emf.ecore.resource.Resource
import org.eclipse.xtext.common.types.TypesFactory
import org.eclipse.xtext.generator.IFileSystemAccess
import org.eclipse.xtext.generator.IGenerator
import org.eclipse.xtext.xbase.XExpression
import org.eclipse.xtext.xbase.compiler.ImportManager
import org.eclipse.xtext.xbase.compiler.StringBuilderBasedAppendable
import org.eclipse.xtext.xbase.compiler.XbaseCompiler
import org.xtext.example.helloxbase.helloXbase.Greeting

import static extension org.eclipse.xtext.xtend2.lib.ResourceExtensions.*

class HelloXbaseGenerator implements IGenerator {

	@Inject
	protected XbaseCompiler xbaseCompiler

	override void doGenerate(Resource resource, IFileSystemAccess fsa) {
		for(greeting: resource.allContentsIterable.filter(typeof(Greeting))) {
			fsa.generateFile(
				greeting.packageName + "/" + // package
				greeting.className + ".java", // class name
				greeting.compile)
		}
	}

	def getJvmType(Greeting greeting) {
		val declaredType = TypesFactory::eINSTANCE.createJvmGenericType
		declaredType.setSimpleName(greeting.className)
		declaredType.setPackageName(greeting.packageName)
		declaredType
	}

	def compile(Greeting greeting) '''
	«val importManager = new ImportManager(true, getJvmType(greeting))»
	«val mainMethod = compile(greeting, importManager)»
	package «greeting.packageName»;
	«IF !importManager.imports.empty»

	«FOR i : importManager.imports»
		import «i»;
	«ENDFOR»
	«ENDIF»

	«mainMethod»
	'''

	def compile(Greeting greeting, ImportManager importManager) '''
	«val result = compile(greeting.expression, importManager)»
	public class «greeting.className» {
		public static void main(String args[]) {
			«result»

			Object expression = «result.getName(greeting.expression)»;
			System.out.println("Hello «greeting.name» from " +
				expression.toString());
		}
	}
	'''

	def compile(XExpression xExpression, ImportManager importManager) {
		val result = new StringBuilderBasedAppendable(importManager)
		xbaseCompiler.toJavaStatement(xExpression, result, true)
		result
	}

	def className(Greeting greeting) {
		greeting.name.toFirstUpper
	}

	def packageName(Greeting greeting) {
		greeting.name.toLowerCase
	}
}

And in the runtime instance you’ll get what you wanted:

You can find the sources for the project helloxbase at

https://github.com/LorenzoBettini/Xtext2-experiments

Hope you find this post useful, and stay tuned for new posts about Xtext :)

 

This entry was tagged , . Bookmark the permalink.

7 Responses to Xtext 2.1: using Xbase expressions

  1. Gary thompson says:

    Hi just a couple of comments (maybe 3!)

    1. Thanks for a couple of really nice posts they have been really useful
    2. One thing I hadn’t realised: when you create the helloxbase project in your second eclipse instance it must be a java project, a general project will not do
    3. I couldn’t get all the syntax you have in your acreen shots to work, specifically something like “Hello wobble from ‘test’ + new String(“.extension”) ! Gives me an error ‘couldn’t resolve reference to jvmIdenifiableElement ‘+’

  2. Hi Gary
    actually the project you create in your second eclipse instance should be a plugin project (I made this explicit in my previous post, not in this one, sorry about that, I’ll fix it); as for the other problem, as I said in this very post

    To fully enjoy Xbase syntax, please make sure that in the project you created in the runtime workspace you also have org.eclipse.xtext.xbase.lib as a dependency in the MANIFEST.

    hope it is clearer now :)

  3. gary thompson says:

    Hi Lorenzo

    thanks for the very fast reply! I have had a play and think I understand now, so to rephrase things

    1. your helloxbase project in your second eclipse instance must be a plug-in project
    2. once you have created your helloxbase plug-in project in your second eclipse instance you must add org.eclipse.xtext.xbase.lib as a dependency in it’s MANIFEST.MF in the required plug-ins section.

    now that’s really gnarly, surely the xtext project people should protect us from these sort of problems a bit more?

    anyway many thanks for you help, that’s solved a problem I have been having for weeks….

    regards
    gary

  4. Hi Gary

    actually, you could also create a Java project, but then you wouldn’t be able to use xbase.lib (well, you could, by setting the classpath, but with plugin projects and dependencies it’s much easier ;)
    Moreover, it’s not xtext project people’s fault :) this procedure might seem strange, but it’s quite typical in DSLs using Xbase; but, most of all, it should be the DSL creator who should also provide a specific project wizard for his DSL and add the required dependencies (and Xtext has a generator fragment for this). I didn’t do this because that’s just an example project :)

    Stay tuned for more posts to come about using Xbase :)

    cheers
    Lorenzo

  5. Gary thompson says:

    Hi Gary

    actually, you could also create a Java project, but then you wouldn’t be able to use xbase.lib (well, you could, by setting the classpath, but with plugin projects and dependencies it’s much easier ;)

    Ok it’s a shortcut ;-)

    Moreover, it’s not xtext project people’s fault :) this procedure might seem strange, but it’s quite typical in DSLs using Xbase; but, most of all, it should be the DSL creator who should also provide a specific project wizard for his DSL and add the required dependencies (and Xtext has a generator fragment for this). I didn’t do this because that’s just an example project :)

    Ok that makes sense as well. However, I guess I still feel that the runtime ought to issue some warnings rather than raising Some rather inobvious exceptions? :)

    Stay tuned for more posts to come about using Xbase :)

    I will!

    Ciao
    Gary

    cheers
    Lorenzo

  6. Claudio says:

    Hi Lorenzo,
    thanks for this post. It’s just what I was looking for: a way to use a generator (that I find more simple to edit compared to the inferrer) and still use the xbase expressions.

    I’ve just a question about it: I’m trying this thing on Xtetxt 2.4.1 and it seams to have problems. To be more clear I’ve this error:


    0 [Worker-4] ERROR org.eclipse.xtext.builder.BuilderParticipant – Error during compilation of ‘platform:/resource/MyDsl/src/testfoo.mydsl’.
    java.lang.NullPointerException
    at org.eclipse.xtext.xbase.compiler.TypeReferenceSerializer.serialize(TypeReferenceSerializer.java:90)
    at org.eclipse.xtext.xbase.compiler.AbstractXbaseCompiler.serialize(AbstractXbaseCompiler.java:342)
    at org.eclipse.xtext.xbase.compiler.AbstractXbaseCompiler.serialize(AbstractXbaseCompiler.java:338)
    at org.eclipse.xtext.xbase.compiler.AbstractXbaseCompiler.serialize(AbstractXbaseCompiler.java:334)
    at org.eclipse.xtext.xbase.compiler.AbstractXbaseCompiler.declareFreshLocalVariable(AbstractXbaseCompiler.java:443)
    at org.eclipse.xtext.xbase.compiler.XbaseCompiler._toJavaStatement(XbaseCompiler.java:769)
    at org.eclipse.xtext.xbase.compiler.XbaseCompiler.doInternalToJavaStatement(XbaseCompiler.java:423)
    at org.eclipse.xtext.xbase.compiler.AbstractXbaseCompiler.internalToJavaStatement(AbstractXbaseCompiler.java:297)
    at org.eclipse.xtext.xbase.compiler.FeatureCallCompiler.prepareExpression(FeatureCallCompiler.java:271)
    at org.eclipse.xtext.xbase.compiler.FeatureCallCompiler._toJavaStatement(FeatureCallCompiler.java:140)
    at org.eclipse.xtext.xbase.compiler.FeatureCallCompiler.doInternalToJavaStatement(FeatureCallCompiler.java:120)
    at org.eclipse.xtext.xbase.compiler.XbaseCompiler.doInternalToJavaStatement(XbaseCompiler.java:449)
    at org.eclipse.xtext.xbase.compiler.AbstractXbaseCompiler.internalToJavaStatement(AbstractXbaseCompiler.java:297)
    at org.eclipse.xtext.xbase.compiler.AbstractXbaseCompiler.toJavaStatement(AbstractXbaseCompiler.java:309)
    at org.xtext.example.mydsl.generator.MyDslGenerator.compile(MyDslGenerator.java:145)
    at org.xtext.example.mydsl.generator.MyDslGenerator.compile(MyDslGenerator.java:102)
    at org.xtext.example.mydsl.generator.MyDslGenerator.compile(MyDslGenerator.java:69)
    at org.xtext.example.mydsl.generator.MyDslGenerator.doGenerate(MyDslGenerator.java:45)
    at org.eclipse.xtext.builder.BuilderParticipant.handleChangedContents(BuilderParticipant.java:291)
    at org.eclipse.xtext.builder.BuilderParticipant.build(BuilderParticipant.java:221)
    at org.eclipse.xtext.builder.impl.RegistryBuilderParticipant.build(RegistryBuilderParticipant.java:60)
    at org.eclipse.xtext.builder.impl.XtextBuilder.doBuild(XtextBuilder.java:170)
    at org.eclipse.xtext.builder.impl.XtextBuilder.incrementalBuild(XtextBuilder.java:146)
    at org.eclipse.xtext.builder.impl.XtextBuilder.build(XtextBuilder.java:95)
    at org.eclipse.core.internal.events.BuildManager$2.run(BuildManager.java:726)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:199)
    at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:239)
    at org.eclipse.core.internal.events.BuildManager$1.run(BuildManager.java:292)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:295)
    at org.eclipse.core.internal.events.BuildManager.basicBuildLoop(BuildManager.java:351)
    at org.eclipse.core.internal.events.BuildManager.build(BuildManager.java:374)
    at org.eclipse.core.internal.events.AutoBuildJob.doBuild(AutoBuildJob.java:143)
    at org.eclipse.core.internal.events.AutoBuildJob.run(AutoBuildJob.java:241)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:53)

    Have you got some hints about it ? Maybe there’s a simple change on the syntax on this version ?
    Thaks,
    Claudio

  7. Priya says:

    Hi Lorenzo,
    Thanx for this post. I was looking for a way where I can use the generator with XBase XBlockExpression.

    The code snippet :
    def compile(XExpression xExpression, ImportManager importManager) {
    val result = new StringBuilderBasedAppendable(importManager)
    xbaseCompiler.toJavaStatement(xExpression, result, true)
    result
    }
    has error “Type mismatch: cannot convert from StringBuilderBasedAppendable to ITreeAppendable”

    Can you please help me out here ?

    Thanks in advance.
    Priya

Lascia un Commento

L'indirizzo email non verrà pubblicato. I campi obbligatori sono contrassegnati *


*

È possibile utilizzare questi tag ed attributi XHTML: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>