Questions and issues to Walter Cazzola.
Domain Specific Languages (DSLs) are used to solve several problems, such as typesetting documents and code (TeX/LaTeX, lout, ...), to express and verify constraints in several domains (OCL, iLOG CP, C4J, ...) and to coordinate the computation (Linda, ...) and/or to data query (SQL, ...). In some cases, these are simply a bunch of programming features useless standalone embedded in a general purpose programming language or provided as external libraries (e.g., Linda and SQL). In these case performances and flexibility are often compromised especially when the DSL is realized as program transformation towards another high-level programming language.
A DSL integrating features from different programming languages and paradigms would be the best choice to express a concise and clean solution to a problem avoiding a cumbersome one due to the absence of a specific feature or to a farraginous one because you are using a general-purpose instead of a domain specific programming language. Unfortunately, to develop a domain specific or a general purpose programming language from scratch implies a considerable effort, a lot of time and skills that not always you can invest in a project.
To simplify and speed up the development of domain specific and general purpose programming languages we designed and developed the Neverlang language workbench. The Neverlang approach basically reflects the fact that programming languages have an intrinsically modular decomposition in language features that are implemented by language components that can be easily plugged and unplugged. A complete compiler/interpreter built up with Neverlang is the result of a compositional process involving several building blocks, i.e., the language components.
In this scenario, to design and implement a domain specific language just consists of composing existing―i.e., reusing―language features and components to automatically get all the needed support tooling—compiler, interpreter, debugger, IDE, etc. Each language component―dubbed as slice in the Neverlang parlance―embodies a single language concept—dubbed as language feature—and provides all the necessary code to support it such as its syntax, how it can be type checked, interpreted, translated to assembly code, debugged, how it should be supported by an IDE, etc. The whole structure of the generated tooling is the result of the composition of such a slices, in particular of the code necessary to compile/interpret each single feature.
How does it work?
Neverlang basically provides: a domain specific language for writing the building blocks and a mechanism for composing these blocks together and for generating the corresponding tooling.
The basic units composing a programming language developed by using Neverlang are modules. Each module encapsulates a specific feature of the language, e.g., a module can encapsulate the syntactically aspect of a loop, the type checking code of a comparison, or the code generation for a method call. Roles define how the composed modules form the compiler/interpreter. syntax, type-checking and evaluation are examples of roles, i.e., apart syntax all the other can be changed and there is no limit in the number of roles. Finally, modules regarding the same language structure but with different roles are grouped together in slices.
Let's see these concepts on a small example: a calculator for integers with only the + operator. The following code shows the implementation of the core of the DSL: the language feature (both module and slice) for the + operator. The final DSL is so simple that we only need a role for the evaluation of the operator.
module addlang.v1.PlusOpModule { reference syntax { add: AddExpression ← Integer "+" Expression; } role (evaluation) { add: .{ $add.value = (Integer) $add[1].value + (Integer) $add[2].value; }. } }
The module addlang.v1.PlusOpModule defines the syntax for the + operator and how it should be evaluated. The syntax is given by a rule of an attributed grammar; it defines the non-terminal AddExpression and use the non-terminals Expression that will be provided by other slices/modules.
Each role (in the example only the role evaluation) adds up on a reference syntax by adding semantics action to the non-terminals. Each role implements a visit on the program's parse tree applying at each node the corresponding semantic action.
The non-terminals are identified through their position in the rules numbering with 0 the top leftmost non-terminal and incrementing by one left-to-right and up-to-down all the non-terminals independently of repetitions and for the whole set of productions defined in the slice. As an alternative, a label can be associated to each rule and used to refer the non-terminals locally to a single rule. The heading of the action just figures out where the semantic action is anchored in the productions.
In the example, we have one semantic action. This enriches the head of the first production (labeled with add) and simply sums the value of the operands (stored in the corresponding non-terminal value attribute) and stores the result in the value attribute of the AddExpression non-terminal. Notes that this implementation rely on the fact that the Expression non-terminal defines a value attribute otherwise the composition will fail.
slice addlang.v1.PlusOpSlice { concrete syntax from addlang.v1.PlusOpModule module addlang.v1.PlusOpModule with role evaluation }
These modules are composed in a slice, where we have to define which role is implemented by each imported module. In this example the slice could be omitted but it also shows a second dimension of composition that can be used, for example, to change either the semantics or the syntax of the operator and reusing the remaining part.
To complete the example we need some language features for the integers and for the expressions, whose non-terminal are used by the + but still dangling.
module addlang.v1.IntModule { reference syntax { int: Integer ← /[0-9]+/ ; } role (evaluation) { int: .{ $int.value = Integer.parseInt(#0.text); }. } }
The above module defines the integers through the regular expression (regex) /[0-9]+/. Note that, in Neverlang, a regex is equivalent to a terminal of the language even if it, de facto, defines several different terminals (all integers in this case). In the semantic action, we retrieve the current value for the regex the predefined attribute text associated to the terminal in position 0 (#0 refers to the regex); text is filled by the lexer with the lexeme matched in the input.
module addlang.v1.Expression { reference syntax { expression: Expression ← AddExpression; constant: Expression ← Integer; } role (evaluation) { expression: .{ $expression.value = (Integer) $expression[1].value; }. constant: .{ $constant.value = (Integer) $constant[1].value; }. } }
The addlang.v1.Expression is what we could call glue code. It just makes sure that both integers and the applications of the + operator are expressions and propagates the attribute values.
module addlang.v1.Program { reference syntax { axiom: Program ← Expression; } role (evaluation) { axiom: .{ System.out.println((Integer)$axiom[1].value); }. } }
In Neverlang, the axiom for the grammar is always named Program. In this DSL a program is just a chain of sums of integers and the expected result is its evaluation. The semantic action associated to the axiom just prints the value of the calculated expression; this is possible because, by default, the parse tree visit is post order.
Now, we have all the language components and we have just to tell Neverlang how to compose them in a language.
language addlang.v1.AddLang { slices addlang.v1.Program addlang.v1.PlusOpSlice addlang.v1.IntModule addlang.v1.Expression roles syntax < evaluation left addlang.v1.PlusOpSlice }
In the language construct we give a name to the language (addlang.v1.AddLang), list the full set of slices/modules (in the slices section) that should be composed in the language and which roles we have and in which order they should be applied (in the roles section).
Ok, our first DSL is ready. For your convenience you can download it from here
Neverlang Features
The Neverlang language workbench is though to enhance the support for variability and reuse of the language features. Its peculiar features are:
Getting started with Neverlang
Download the Neverlang package from the Resources section.
unpack the neverlang package.
> mkdir «where you like»/neverlang > cd «where you like»/neverlang > tar xvzf «where you downloaded»/neverlang2-v0.8ß.tgz
create the $NEVERLANG_HOME environment variable that refers to where you unpacked Neverlang; add bin to your path
> export NEVERLANG_HOME=«path to»/neverlang > export PATH=${NEVERLANG_HOME}/bin:$PATH
note that to avoid to do this everytime you can add it to your .bashrc or your .bash_profile.
Note, Neverlang is developed under Linux, it runs under MacOS and since it runs on the JVM it should work without problems in Windows too. Anyway the above instructions are for a Linux system with bash installed.
Note Since version 1.2 Neverlang runs on top of Java 13.
Let's see how to use it with the AddLang example.
Download the archive and unpack it. A suggested structure for the Neverlang projects asks for a nlg-src and java-src: the former will contain the neverlang sources the latter the generated Java files.
> cd «where you like» > mkdir {nlg-src,java-src} > cd nlg-src > tar xvzf addlang.tgz > cd .. > tree . . ├── java-src └─── nlg-src └─── addlang └─── v1 ├── AddLang.nl ├── Expression.nl ├── IntModule.nl ├── PlusOpModule.nl ├── PlusOpSlice.nl └── Program.nl >
to generate the interpreter
> cd nlg-src/ > nlgc -s ../java-src addlang/v1/*.nl ---------------------------------------- Using Neverlang RSD Compiler 0.8.0 Compiled on: Tue Jul 23 09:40:40 CEST 2019 addlang/v1/AddLang.nl addlang/v1/Expression.nl addlang/v1/IntModule.nl addlang/v1/PlusOpModule.nl addlang/v1/PlusOpSlice.nl addlang/v1/Program.nl addlang/v1/AddLang.java addlang/v1/Expression$role$syntax.java addlang/v1/Expression$role$evaluation$0.java addlang/v1/Expression$role$evaluation$2.java addlang/v1/Expression.java addlang/v1/IntModule$role$syntax.java addlang/v1/IntModule$role$evaluation$0.java addlang/v1/IntModule.java addlang/v1/PlusOpModule$role$syntax.java addlang/v1/PlusOpModule$role$evaluation$0.java addlang/v1/PlusOpModule.java addlang/v1/PlusOpSlice.java addlang/v1/Program$role$syntax.java addlang/v1/Program$role$evaluation$0.java addlang/v1/Program.java addlang/v1/addlang/CategoryMapper.java > cd ../java-src/ > javac -cp $(NEVERLANG_HOME)/Neverlang.jar addlang/v1/*java
to execute the REPL
>nlgi addlang.v1.AddLang
this enables the interactive shell for the developed language; it also has some nice feature for the debugging like the dump of the parse tree.
to execute a script
> nlg addlang.v1.addlang «input file»
the input is in «input file» and the output is printed on the shell.
Here you can find some extra resource (more to come):
Neverlang Staff
The Neverlang2 project is led by Walter Cazzola.