Creating an ANTLR Language Target

This document describes how to make ANTLR generate parsers in a new language, X.

Overview

Creating a new target involves the following key elements:

  1. For the tool, create class XTarget as a subclass of class Target in package org.antlr.v4.codegen.target. This class describes language specific details about escape characters and strings and so on. There is very little to do here typically.
  2. Create *X*.stg in directory tool/resources/org/antlr/v4/tool/templates/codegen/*X*/*X*.stg. This is a StringTemplate group file (.stg) that tells ANTLR how to express all the parsing elements needed to generate code. You will see templates called ParserFile, Parser, Lexer, CodeBlockForAlt, AltBlock, etc... Each of these must be described how to build the indicated chunk of code. Your best bet is to find the closest existing target, copy that template file, and tweak to suit.
  3. Create a runtime library to support the parsers generated by ANTLR. Under directory runtime/*X*, you are in complete control of the directory structure as dictated by common usage of that target language. For example, Java has: runtime/Java/lib and runtime/Java/src directories. Under src, you will find a directory structure for package org.antlr.v4.runtime and below.
  4. Create a template file for runtime tests. All you have to do is provide a few templates that indicate how to print values and declare variables. Our runtime test mechanism in dir runtime-testsuite will automatically generate code using these templates for each target and check the test results. It needs to know how to define various class fields, compare members and so on. You must create a *X*.test.stg file underneath runtime-testsuite/resources/org/antlr/v4/test/runtime and Test.*x*.stg underneath runtime-testsuite/resources/org/antlr/v4/test/runtime/helpers. Again, your best bet is to copy the templates from the closest language to your target and tweak it to suit.
  5. Create test files under /runtime-testsuite/test/org/antlr/v4/test/runtime. They will load defined test cases in each test descriptor. Also add the /runtime-testsuite/test/org/antlr/v4/test/runtime/X/BaseXTest.java which defines how test cases will execute and output.
  6. Create/edit shell scripts in /.github to run tests in CI pipelines.

Getting started

  1. Fork the antlr/antlr4 repository at GitHub to your own user so that you have repository username/antlr4.
  2. Clone username/antlr4, the forked repository, to your local disk. Your remote origin will be the forked repository on GitHub. Add a remote upstream to the original antlr/antlr4 repository (URL https://github.com/antlr/antlr4.git). Changes that you would like to contribute back to the project are done with pull requests.
  3. Try to build it before doing anything
$ mvn compile

That should proceed with success. See Building ANTLR for more details.

Comparing your target‘s parsing decisionmaking with Java’s

ANTLR‘s power comes from it’s dynamic parsing strategy, but that means each target must implement that complicated algorithm. You should compare your target‘s debug output for ParserATNSimulator with Java’s.

Run this so we get right jars before trying this script:

cd ANTLR-ROOT-DIR
mvn install -DskipTests=true
cd runtime-tests 
mvn install -DskipTests=true # yes do it again

Run the script from runtime-tests dir with

../scripts/traceatn.sh /tmp/JSON.g4 json -target Go /tmp/foo.json

or whatever your test grammar, start rule, target, test input are.

Debugging the PHP target

Because the PHP target is hosted in a separate repository, you will need to clone the antlr/php-antlr-runtime repository into the runtime/PHP and install the dependencies with composer install before you can run the tests.

git clone -b dev https://github.com/antlr/antlr-php-runtime.git runtime/PHP
cd runtime/PHP
composer install