Repository: antlr/antlr4 Branch: dev Commit: 7d5770395bb7 Files: 2263 Total size: 9.6 MB Directory structure: gitextract_275zyhbd/ ├── .clang-format ├── .editorconfig ├── .gitattributes ├── .github/ │ ├── ISSUE_TEMPLATE.md │ ├── PULL_REQUEST_TEMPLATE.md │ └── workflows/ │ └── hosted.yml ├── .gitignore ├── ANTLR-HOUSE-RULES.md ├── CHANGES.txt ├── CONTRIBUTING.md ├── LICENSE.txt ├── Package.swift ├── README.md ├── antlr4-maven-plugin/ │ ├── nb-configuration.xml │ ├── pom.xml │ ├── resources/ │ │ └── META-INF/ │ │ └── m2e/ │ │ └── lifecycle-mapping-metadata.xml │ └── src/ │ ├── main/ │ │ └── java/ │ │ └── org/ │ │ └── antlr/ │ │ └── mojo/ │ │ └── antlr4/ │ │ ├── Antlr4ErrorLog.java │ │ ├── Antlr4Mojo.java │ │ ├── GrammarDependencies.java │ │ └── MojoUtils.java │ ├── site/ │ │ ├── apt/ │ │ │ ├── examples/ │ │ │ │ ├── import.apt │ │ │ │ ├── libraries.apt.vm │ │ │ │ └── simple.apt.vm │ │ │ ├── faq.apt.vm │ │ │ ├── index.apt │ │ │ └── usage.apt.vm │ │ └── site.xml │ └── test/ │ ├── java/ │ │ └── org/ │ │ └── antlr/ │ │ └── mojo/ │ │ └── antlr4/ │ │ └── Antlr4MojoTest.java │ └── projects/ │ ├── dependencyRemoved/ │ │ ├── pom.xml │ │ └── src/ │ │ └── main/ │ │ └── antlr4/ │ │ ├── imports/ │ │ │ └── HelloBase.g4 │ │ └── test/ │ │ └── Hello.g4 │ ├── importTokens/ │ │ ├── pom.xml │ │ └── src/ │ │ └── main/ │ │ └── antlr4/ │ │ ├── imports/ │ │ │ └── SimpleLexer.tokens │ │ └── test/ │ │ └── SimpleParser.g4 │ ├── importsCustom/ │ │ ├── pom.xml │ │ └── src/ │ │ └── main/ │ │ └── antlr4/ │ │ ├── Hello.g4 │ │ ├── TestLexer.g4 │ │ ├── TestParser.g4 │ │ └── imports/ │ │ └── TestBaseLexer.g4 │ └── importsStandard/ │ ├── pom.xml │ └── src/ │ └── main/ │ └── antlr4/ │ ├── imports/ │ │ ├── TestBaseLexer.g4 │ │ └── TestBaseLexer2.g4 │ └── test/ │ ├── Hello.g4 │ ├── TestLexer.g4 │ └── TestParser.g4 ├── developer-cert-of-origin.txt ├── doc/ │ ├── IDEs.md │ ├── ace-javascript-target.md │ ├── actions.md │ ├── antlr-project-testing.md │ ├── building-antlr.md │ ├── cpp-target.md │ ├── creating-a-language-target.md │ ├── csharp-target.md │ ├── dart-target.md │ ├── faq/ │ │ ├── actions-preds.md │ │ ├── error-handling.md │ │ ├── general.md │ │ ├── getting-started.md │ │ ├── index.md │ │ ├── installation.md │ │ ├── lexical.md │ │ ├── parse-trees.md │ │ └── translation.md │ ├── getting-started.md │ ├── go-changes.md │ ├── go-target.md │ ├── grammars.md │ ├── index.md │ ├── interpreters.md │ ├── java-target.md │ ├── javascript-target.md │ ├── left-recursion.md │ ├── lexer-rules.md │ ├── lexicon.md │ ├── listeners.md │ ├── options.md │ ├── parser-rules.md │ ├── parsing-binary-files.md │ ├── php-target.md │ ├── predicates.md │ ├── python-target.md │ ├── releasing-antlr.md │ ├── resources.md │ ├── swift-target.md │ ├── target-agnostic-grammars.md │ ├── targets.md │ ├── tool-options.md │ ├── tree-matching.md │ ├── typescript-target.md │ ├── unicode.md │ └── wildcard.md ├── docker/ │ ├── .dockerignore │ ├── Dockerfile │ └── README.md ├── historical-contributors-agreement.txt ├── pom.xml ├── runtime/ │ ├── CSharp/ │ │ ├── src/ │ │ │ ├── Antlr4.csproj │ │ │ ├── Antlr4.snk │ │ │ ├── AntlrFileStream.cs │ │ │ ├── AntlrInputStream.cs │ │ │ ├── Atn/ │ │ │ │ ├── ATN.cs │ │ │ │ ├── ATNConfig.cs │ │ │ │ ├── ATNConfigSet.cs │ │ │ │ ├── ATNDeserializationOptions.cs │ │ │ │ ├── ATNDeserializer.cs │ │ │ │ ├── ATNSimulator.cs │ │ │ │ ├── ATNState.cs │ │ │ │ ├── ATNType.cs │ │ │ │ ├── AbstractPredicateTransition.cs │ │ │ │ ├── ActionTransition.cs │ │ │ │ ├── AmbiguityInfo.cs │ │ │ │ ├── ArrayPredictionContext.cs │ │ │ │ ├── AtomTransition.cs │ │ │ │ ├── BasicBlockStartState.cs │ │ │ │ ├── BasicState.cs │ │ │ │ ├── BlockEndState.cs │ │ │ │ ├── BlockStartState.cs │ │ │ │ ├── ConflictInfo.cs │ │ │ │ ├── ContextSensitivityInfo.cs │ │ │ │ ├── DecisionEventInfo.cs │ │ │ │ ├── DecisionInfo.cs │ │ │ │ ├── DecisionState.cs │ │ │ │ ├── EmptyPredictionContext.cs │ │ │ │ ├── EpsilonTransition.cs │ │ │ │ ├── ErrorInfo.cs │ │ │ │ ├── ILexerAction.cs │ │ │ │ ├── LL1Analyzer.cs │ │ │ │ ├── LexerATNConfig.cs │ │ │ │ ├── LexerATNSimulator.cs │ │ │ │ ├── LexerActionExecutor.cs │ │ │ │ ├── LexerActionType.cs │ │ │ │ ├── LexerChannelAction.cs │ │ │ │ ├── LexerCustomAction.cs │ │ │ │ ├── LexerIndexedCustomAction.cs │ │ │ │ ├── LexerModeAction.cs │ │ │ │ ├── LexerMoreAction.cs │ │ │ │ ├── LexerPopModeAction.cs │ │ │ │ ├── LexerPushModeAction.cs │ │ │ │ ├── LexerSkipAction.cs │ │ │ │ ├── LexerTypeAction.cs │ │ │ │ ├── LookaheadEventInfo.cs │ │ │ │ ├── LoopEndState.cs │ │ │ │ ├── MergeCache.cs │ │ │ │ ├── NotSetTransition.cs │ │ │ │ ├── ParseInfo.cs │ │ │ │ ├── ParserATNSimulator.cs │ │ │ │ ├── PlusBlockStartState.cs │ │ │ │ ├── PlusLoopbackState.cs │ │ │ │ ├── PrecedencePredicateTransition.cs │ │ │ │ ├── PredicateEvalInfo.cs │ │ │ │ ├── PredicateTransition.cs │ │ │ │ ├── PredictionContext.cs │ │ │ │ ├── PredictionContextCache.cs │ │ │ │ ├── PredictionMode.cs │ │ │ │ ├── ProfilingATNSimulator.cs │ │ │ │ ├── RangeTransition.cs │ │ │ │ ├── RuleStartState.cs │ │ │ │ ├── RuleStopState.cs │ │ │ │ ├── RuleTransition.cs │ │ │ │ ├── SemanticContext.cs │ │ │ │ ├── SetTransition.cs │ │ │ │ ├── SimulatorState.cs │ │ │ │ ├── SingletonPredictionContext.cs │ │ │ │ ├── StarBlockStartState.cs │ │ │ │ ├── StarLoopEntryState.cs │ │ │ │ ├── StarLoopbackState.cs │ │ │ │ ├── StateType.cs │ │ │ │ ├── TokensStartState.cs │ │ │ │ ├── Transition.cs │ │ │ │ ├── TransitionType.cs │ │ │ │ └── WildcardTransition.cs │ │ │ ├── BailErrorStrategy.cs │ │ │ ├── BaseErrorListener.cs │ │ │ ├── BufferedTokenStream.cs │ │ │ ├── CharStreams.cs │ │ │ ├── CommonToken.cs │ │ │ ├── CommonTokenFactory.cs │ │ │ ├── CommonTokenStream.cs │ │ │ ├── ConsoleErrorListener.cs │ │ │ ├── DefaultErrorStrategy.cs │ │ │ ├── Dependents.cs │ │ │ ├── Dfa/ │ │ │ │ ├── AbstractEdgeMap.cs │ │ │ │ ├── AcceptStateInfo.cs │ │ │ │ ├── ArrayEdgeMap.cs │ │ │ │ ├── DFA.cs │ │ │ │ ├── DFASerializer.cs │ │ │ │ ├── DFAState.cs │ │ │ │ ├── EmptyEdgeMap.cs │ │ │ │ ├── IEdgeMap.cs │ │ │ │ ├── LexerDFASerializer.cs │ │ │ │ ├── SingletonEdgeMap.cs │ │ │ │ └── SparseEdgeMap.cs │ │ │ ├── DiagnosticErrorListener.cs │ │ │ ├── FailedPredicateException.cs │ │ │ ├── IAntlrErrorListener.cs │ │ │ ├── IAntlrErrorStrategy.cs │ │ │ ├── ICharStream.cs │ │ │ ├── IIntStream.cs │ │ │ ├── IParserErrorListener.cs │ │ │ ├── IRecognizer.cs │ │ │ ├── IToken.cs │ │ │ ├── ITokenFactory.cs │ │ │ ├── ITokenSource.cs │ │ │ ├── ITokenStream.cs │ │ │ ├── IVocabulary.cs │ │ │ ├── IWritableToken.cs │ │ │ ├── InputMismatchException.cs │ │ │ ├── InterpreterRuleContext.cs │ │ │ ├── Lexer.cs │ │ │ ├── LexerInterpreter.cs │ │ │ ├── LexerNoViableAltException.cs │ │ │ ├── ListTokenSource.cs │ │ │ ├── Misc/ │ │ │ │ ├── Args.cs │ │ │ │ ├── ArrayList.cs │ │ │ │ ├── IIntSet.cs │ │ │ │ ├── Interval.cs │ │ │ │ ├── IntervalSet.cs │ │ │ │ ├── MultiMap.cs │ │ │ │ ├── MurmurHash.cs │ │ │ │ ├── NotNullAttribute.cs │ │ │ │ ├── NullableAttribute.cs │ │ │ │ ├── Pair.cs │ │ │ │ ├── ParseCanceledException.cs │ │ │ │ ├── RuleDependencyChecker.cs │ │ │ │ └── Utils.cs │ │ │ ├── NoViableAltException.cs │ │ │ ├── Parser.cs │ │ │ ├── ParserInterpreter.cs │ │ │ ├── ParserRuleContext.cs │ │ │ ├── Properties/ │ │ │ │ └── AssemblyInfo.cs │ │ │ ├── ProxyErrorListener.cs │ │ │ ├── ProxyParserErrorListener.cs │ │ │ ├── README.md │ │ │ ├── RecognitionException.cs │ │ │ ├── Recognizer.cs │ │ │ ├── RuleContext.cs │ │ │ ├── RuleDependencyAttribute.cs │ │ │ ├── RuleVersionAttribute.cs │ │ │ ├── Sharpen/ │ │ │ │ ├── Arrays.cs │ │ │ │ ├── AtomicReference.cs │ │ │ │ ├── BitSet.cs │ │ │ │ ├── Collections.cs │ │ │ │ ├── DictionaryExtensions.cs │ │ │ │ ├── ListExtensions.cs │ │ │ │ ├── Runtime.cs │ │ │ │ └── SequenceEqualityComparer.cs │ │ │ ├── TokenStreamRewriter.cs │ │ │ ├── TokenTypes.cs │ │ │ ├── Tree/ │ │ │ │ ├── AbstractParseTreeVisitor.cs │ │ │ │ ├── ErrorNodeImpl.cs │ │ │ │ ├── IErrorNode.cs │ │ │ │ ├── IParseTree.cs │ │ │ │ ├── IParseTreeListener.cs │ │ │ │ ├── IParseTreeVisitor.cs │ │ │ │ ├── IRuleNode.cs │ │ │ │ ├── ISyntaxTree.cs │ │ │ │ ├── ITerminalNode.cs │ │ │ │ ├── ITree.cs │ │ │ │ ├── ParseTreeProperty.cs │ │ │ │ ├── ParseTreeWalker.cs │ │ │ │ ├── Pattern/ │ │ │ │ │ ├── Chunk.cs │ │ │ │ │ ├── ParseTreeMatch.cs │ │ │ │ │ ├── ParseTreePattern.cs │ │ │ │ │ ├── ParseTreePatternMatcher.cs │ │ │ │ │ ├── RuleTagToken.cs │ │ │ │ │ ├── TagChunk.cs │ │ │ │ │ ├── TextChunk.cs │ │ │ │ │ └── TokenTagToken.cs │ │ │ │ ├── TerminalNodeImpl.cs │ │ │ │ ├── Trees.cs │ │ │ │ └── Xpath/ │ │ │ │ ├── XPath.cs │ │ │ │ ├── XPathElement.cs │ │ │ │ ├── XPathLexer.cs │ │ │ │ ├── XPathLexer.g4 │ │ │ │ ├── XPathLexer.tokens │ │ │ │ ├── XPathLexerErrorListener.cs │ │ │ │ ├── XPathRuleAnywhereElement.cs │ │ │ │ ├── XPathRuleElement.cs │ │ │ │ ├── XPathTokenAnywhereElement.cs │ │ │ │ ├── XPathTokenElement.cs │ │ │ │ ├── XPathWildcardAnywhereElement.cs │ │ │ │ └── XPathWildcardElement.cs │ │ │ ├── UnbufferedCharStream.cs │ │ │ ├── UnbufferedTokenStream.cs │ │ │ └── Vocabulary.cs │ │ └── tests/ │ │ ├── issue-2693/ │ │ │ ├── ErrorListener.cs │ │ │ ├── Program.cs │ │ │ ├── Test.csproj │ │ │ ├── Test.sln │ │ │ ├── TreeOutput.cs │ │ │ ├── asm8080.g4 │ │ │ ├── cpm22.asm │ │ │ └── test.sh │ │ └── issue-3079/ │ │ ├── Arithmetic.g4 │ │ ├── ErrorListener.cs │ │ ├── Program.cs │ │ ├── Test.csproj │ │ ├── Test.sln │ │ ├── readme.md │ │ └── test.sh │ ├── Cpp/ │ │ ├── CMakeLists.txt │ │ ├── CMakeSettings.json │ │ ├── README.md │ │ ├── VERSION │ │ ├── cmake/ │ │ │ ├── Antlr4Package.md │ │ │ ├── ExternalAntlr4Cpp.cmake │ │ │ ├── FindANTLR.cmake │ │ │ ├── README.md │ │ │ ├── antlr4-generator.cmake.in │ │ │ └── antlr4-runtime.cmake.in │ │ ├── demo/ │ │ │ ├── CMakeLists.txt │ │ │ ├── Linux/ │ │ │ │ └── main.cpp │ │ │ ├── Mac/ │ │ │ │ ├── antlr4-cpp-demo/ │ │ │ │ │ └── main.cpp │ │ │ │ ├── antlrcpp Tests/ │ │ │ │ │ ├── Info.plist │ │ │ │ │ ├── InputHandlingTests.mm │ │ │ │ │ ├── MiscClassTests.mm │ │ │ │ │ └── antlrcpp_Tests.mm │ │ │ │ ├── antlrcpp-demo.xcodeproj/ │ │ │ │ │ ├── project.pbxproj │ │ │ │ │ ├── project.xcworkspace/ │ │ │ │ │ │ ├── contents.xcworkspacedata │ │ │ │ │ │ └── xcshareddata/ │ │ │ │ │ │ └── IDEWorkspaceChecks.plist │ │ │ │ │ └── xcshareddata/ │ │ │ │ │ └── xcschemes/ │ │ │ │ │ ├── antlr4-cpp-demo.xcscheme │ │ │ │ │ └── antlrcpp Tests.xcscheme │ │ │ │ └── build.sh │ │ │ ├── README.md │ │ │ ├── TLexer.g4 │ │ │ ├── TParser.g4 │ │ │ ├── Windows/ │ │ │ │ ├── antlr4-cpp-demo/ │ │ │ │ │ ├── antlr4-cpp-demo-vs2022.vcxproj │ │ │ │ │ ├── antlr4-cpp-demo-vs2022.vcxproj.filters │ │ │ │ │ └── main.cpp │ │ │ │ └── antlr4cpp-vs2022.sln │ │ │ ├── generate.cmd │ │ │ └── generate.sh │ │ ├── deploy-macos.sh │ │ ├── deploy-source.sh │ │ ├── deploy-windows.cmd │ │ └── runtime/ │ │ ├── CMakeLists.txt │ │ ├── antlr4cpp-vs2019.vcxproj │ │ ├── antlr4cpp-vs2019.vcxproj.filters │ │ ├── antlr4cpp-vs2022.vcxproj │ │ ├── antlr4cpp-vs2022.vcxproj.filters │ │ ├── antlrcpp-ios/ │ │ │ ├── Info.plist │ │ │ └── antlrcpp_ios.h │ │ ├── antlrcpp.xcodeproj/ │ │ │ ├── project.pbxproj │ │ │ ├── project.xcworkspace/ │ │ │ │ ├── contents.xcworkspacedata │ │ │ │ └── xcshareddata/ │ │ │ │ └── IDEWorkspaceChecks.plist │ │ │ └── xcshareddata/ │ │ │ └── xcschemes/ │ │ │ ├── antlr4.xcscheme │ │ │ ├── antlr4_ios.xcscheme │ │ │ └── antlr4_static.xcscheme │ │ ├── nuget/ │ │ │ ├── ANTLR4.Runtime.cpp.noarch.nuspec │ │ │ ├── ANTLR4.Runtime.cpp.noarch.targets │ │ │ ├── ANTLR4.Runtime.cpp.shared.nuspec │ │ │ ├── ANTLR4.Runtime.cpp.shared.props │ │ │ ├── ANTLR4.Runtime.cpp.shared.targets │ │ │ ├── ANTLR4.Runtime.cpp.static.nuspec │ │ │ ├── ANTLR4.Runtime.cpp.static.targets │ │ │ └── pack.cmd │ │ ├── src/ │ │ │ ├── ANTLRErrorListener.cpp │ │ │ ├── ANTLRErrorListener.h │ │ │ ├── ANTLRErrorStrategy.cpp │ │ │ ├── ANTLRErrorStrategy.h │ │ │ ├── ANTLRFileStream.cpp │ │ │ ├── ANTLRFileStream.h │ │ │ ├── ANTLRInputStream.cpp │ │ │ ├── ANTLRInputStream.h │ │ │ ├── BailErrorStrategy.cpp │ │ │ ├── BailErrorStrategy.h │ │ │ ├── BaseErrorListener.cpp │ │ │ ├── BaseErrorListener.h │ │ │ ├── BufferedTokenStream.cpp │ │ │ ├── BufferedTokenStream.h │ │ │ ├── CharStream.cpp │ │ │ ├── CharStream.h │ │ │ ├── CommonToken.cpp │ │ │ ├── CommonToken.h │ │ │ ├── CommonTokenFactory.cpp │ │ │ ├── CommonTokenFactory.h │ │ │ ├── CommonTokenStream.cpp │ │ │ ├── CommonTokenStream.h │ │ │ ├── ConsoleErrorListener.cpp │ │ │ ├── ConsoleErrorListener.h │ │ │ ├── DefaultErrorStrategy.cpp │ │ │ ├── DefaultErrorStrategy.h │ │ │ ├── DiagnosticErrorListener.cpp │ │ │ ├── DiagnosticErrorListener.h │ │ │ ├── Exceptions.cpp │ │ │ ├── Exceptions.h │ │ │ ├── FailedPredicateException.cpp │ │ │ ├── FailedPredicateException.h │ │ │ ├── FlatHashMap.h │ │ │ ├── FlatHashSet.h │ │ │ ├── InputMismatchException.cpp │ │ │ ├── InputMismatchException.h │ │ │ ├── IntStream.cpp │ │ │ ├── IntStream.h │ │ │ ├── InterpreterRuleContext.cpp │ │ │ ├── InterpreterRuleContext.h │ │ │ ├── Lexer.cpp │ │ │ ├── Lexer.h │ │ │ ├── LexerInterpreter.cpp │ │ │ ├── LexerInterpreter.h │ │ │ ├── LexerNoViableAltException.cpp │ │ │ ├── LexerNoViableAltException.h │ │ │ ├── ListTokenSource.cpp │ │ │ ├── ListTokenSource.h │ │ │ ├── NoViableAltException.cpp │ │ │ ├── NoViableAltException.h │ │ │ ├── Parser.cpp │ │ │ ├── Parser.h │ │ │ ├── ParserInterpreter.cpp │ │ │ ├── ParserInterpreter.h │ │ │ ├── ParserRuleContext.cpp │ │ │ ├── ParserRuleContext.h │ │ │ ├── ProxyErrorListener.cpp │ │ │ ├── ProxyErrorListener.h │ │ │ ├── RecognitionException.cpp │ │ │ ├── RecognitionException.h │ │ │ ├── Recognizer.cpp │ │ │ ├── Recognizer.h │ │ │ ├── RuleContext.cpp │ │ │ ├── RuleContext.h │ │ │ ├── RuleContextWithAltNum.cpp │ │ │ ├── RuleContextWithAltNum.h │ │ │ ├── RuntimeMetaData.cpp │ │ │ ├── RuntimeMetaData.h │ │ │ ├── Token.cpp │ │ │ ├── Token.h │ │ │ ├── TokenFactory.h │ │ │ ├── TokenSource.cpp │ │ │ ├── TokenSource.h │ │ │ ├── TokenStream.cpp │ │ │ ├── TokenStream.h │ │ │ ├── TokenStreamRewriter.cpp │ │ │ ├── TokenStreamRewriter.h │ │ │ ├── UnbufferedCharStream.cpp │ │ │ ├── UnbufferedCharStream.h │ │ │ ├── UnbufferedTokenStream.cpp │ │ │ ├── UnbufferedTokenStream.h │ │ │ ├── Version.h │ │ │ ├── Vocabulary.cpp │ │ │ ├── Vocabulary.h │ │ │ ├── WritableToken.cpp │ │ │ ├── WritableToken.h │ │ │ ├── antlr4-common.h │ │ │ ├── antlr4-runtime.h │ │ │ ├── atn/ │ │ │ │ ├── ATN.cpp │ │ │ │ ├── ATN.h │ │ │ │ ├── ATNConfig.cpp │ │ │ │ ├── ATNConfig.h │ │ │ │ ├── ATNConfigSet.cpp │ │ │ │ ├── ATNConfigSet.h │ │ │ │ ├── ATNDeserializationOptions.cpp │ │ │ │ ├── ATNDeserializationOptions.h │ │ │ │ ├── ATNDeserializer.cpp │ │ │ │ ├── ATNDeserializer.h │ │ │ │ ├── ATNSimulator.cpp │ │ │ │ ├── ATNSimulator.h │ │ │ │ ├── ATNState.cpp │ │ │ │ ├── ATNState.h │ │ │ │ ├── ATNStateType.cpp │ │ │ │ ├── ATNStateType.h │ │ │ │ ├── ATNType.h │ │ │ │ ├── ActionTransition.cpp │ │ │ │ ├── ActionTransition.h │ │ │ │ ├── AmbiguityInfo.cpp │ │ │ │ ├── AmbiguityInfo.h │ │ │ │ ├── ArrayPredictionContext.cpp │ │ │ │ ├── ArrayPredictionContext.h │ │ │ │ ├── AtomTransition.cpp │ │ │ │ ├── AtomTransition.h │ │ │ │ ├── BasicBlockStartState.h │ │ │ │ ├── BasicState.h │ │ │ │ ├── BlockEndState.h │ │ │ │ ├── BlockStartState.h │ │ │ │ ├── ContextSensitivityInfo.cpp │ │ │ │ ├── ContextSensitivityInfo.h │ │ │ │ ├── DecisionEventInfo.cpp │ │ │ │ ├── DecisionEventInfo.h │ │ │ │ ├── DecisionInfo.cpp │ │ │ │ ├── DecisionInfo.h │ │ │ │ ├── DecisionState.cpp │ │ │ │ ├── DecisionState.h │ │ │ │ ├── EpsilonTransition.cpp │ │ │ │ ├── EpsilonTransition.h │ │ │ │ ├── ErrorInfo.cpp │ │ │ │ ├── ErrorInfo.h │ │ │ │ ├── HashUtils.h │ │ │ │ ├── LL1Analyzer.cpp │ │ │ │ ├── LL1Analyzer.h │ │ │ │ ├── LexerATNConfig.cpp │ │ │ │ ├── LexerATNConfig.h │ │ │ │ ├── LexerATNSimulator.cpp │ │ │ │ ├── LexerATNSimulator.h │ │ │ │ ├── LexerAction.cpp │ │ │ │ ├── LexerAction.h │ │ │ │ ├── LexerActionExecutor.cpp │ │ │ │ ├── LexerActionExecutor.h │ │ │ │ ├── LexerActionType.h │ │ │ │ ├── LexerChannelAction.cpp │ │ │ │ ├── LexerChannelAction.h │ │ │ │ ├── LexerCustomAction.cpp │ │ │ │ ├── LexerCustomAction.h │ │ │ │ ├── LexerIndexedCustomAction.cpp │ │ │ │ ├── LexerIndexedCustomAction.h │ │ │ │ ├── LexerModeAction.cpp │ │ │ │ ├── LexerModeAction.h │ │ │ │ ├── LexerMoreAction.cpp │ │ │ │ ├── LexerMoreAction.h │ │ │ │ ├── LexerPopModeAction.cpp │ │ │ │ ├── LexerPopModeAction.h │ │ │ │ ├── LexerPushModeAction.cpp │ │ │ │ ├── LexerPushModeAction.h │ │ │ │ ├── LexerSkipAction.cpp │ │ │ │ ├── LexerSkipAction.h │ │ │ │ ├── LexerTypeAction.cpp │ │ │ │ ├── LexerTypeAction.h │ │ │ │ ├── LookaheadEventInfo.cpp │ │ │ │ ├── LookaheadEventInfo.h │ │ │ │ ├── LoopEndState.h │ │ │ │ ├── NotSetTransition.cpp │ │ │ │ ├── NotSetTransition.h │ │ │ │ ├── OrderedATNConfigSet.cpp │ │ │ │ ├── OrderedATNConfigSet.h │ │ │ │ ├── ParseInfo.cpp │ │ │ │ ├── ParseInfo.h │ │ │ │ ├── ParserATNSimulator.cpp │ │ │ │ ├── ParserATNSimulator.h │ │ │ │ ├── ParserATNSimulatorOptions.h │ │ │ │ ├── PlusBlockStartState.h │ │ │ │ ├── PlusLoopbackState.h │ │ │ │ ├── PrecedencePredicateTransition.cpp │ │ │ │ ├── PrecedencePredicateTransition.h │ │ │ │ ├── PredicateEvalInfo.cpp │ │ │ │ ├── PredicateEvalInfo.h │ │ │ │ ├── PredicateTransition.cpp │ │ │ │ ├── PredicateTransition.h │ │ │ │ ├── PredictionContext.cpp │ │ │ │ ├── PredictionContext.h │ │ │ │ ├── PredictionContextCache.cpp │ │ │ │ ├── PredictionContextCache.h │ │ │ │ ├── PredictionContextMergeCache.cpp │ │ │ │ ├── PredictionContextMergeCache.h │ │ │ │ ├── PredictionContextMergeCacheOptions.h │ │ │ │ ├── PredictionContextType.h │ │ │ │ ├── PredictionMode.cpp │ │ │ │ ├── PredictionMode.h │ │ │ │ ├── ProfilingATNSimulator.cpp │ │ │ │ ├── ProfilingATNSimulator.h │ │ │ │ ├── RangeTransition.cpp │ │ │ │ ├── RangeTransition.h │ │ │ │ ├── RuleStartState.h │ │ │ │ ├── RuleStopState.h │ │ │ │ ├── RuleTransition.cpp │ │ │ │ ├── RuleTransition.h │ │ │ │ ├── SemanticContext.cpp │ │ │ │ ├── SemanticContext.h │ │ │ │ ├── SemanticContextType.h │ │ │ │ ├── SerializedATNView.h │ │ │ │ ├── SetTransition.cpp │ │ │ │ ├── SetTransition.h │ │ │ │ ├── SingletonPredictionContext.cpp │ │ │ │ ├── SingletonPredictionContext.h │ │ │ │ ├── StarBlockStartState.h │ │ │ │ ├── StarLoopEntryState.h │ │ │ │ ├── StarLoopbackState.cpp │ │ │ │ ├── StarLoopbackState.h │ │ │ │ ├── TokensStartState.h │ │ │ │ ├── Transition.cpp │ │ │ │ ├── Transition.h │ │ │ │ ├── TransitionType.cpp │ │ │ │ ├── TransitionType.h │ │ │ │ ├── WildcardTransition.cpp │ │ │ │ └── WildcardTransition.h │ │ │ ├── dfa/ │ │ │ │ ├── DFA.cpp │ │ │ │ ├── DFA.h │ │ │ │ ├── DFASerializer.cpp │ │ │ │ ├── DFASerializer.h │ │ │ │ ├── DFAState.cpp │ │ │ │ ├── DFAState.h │ │ │ │ ├── LexerDFASerializer.cpp │ │ │ │ └── LexerDFASerializer.h │ │ │ ├── internal/ │ │ │ │ ├── Synchronization.cpp │ │ │ │ └── Synchronization.h │ │ │ ├── misc/ │ │ │ │ ├── InterpreterDataReader.cpp │ │ │ │ ├── InterpreterDataReader.h │ │ │ │ ├── Interval.cpp │ │ │ │ ├── Interval.h │ │ │ │ ├── IntervalSet.cpp │ │ │ │ ├── IntervalSet.h │ │ │ │ ├── MurmurHash.cpp │ │ │ │ ├── MurmurHash.h │ │ │ │ ├── Predicate.cpp │ │ │ │ └── Predicate.h │ │ │ ├── support/ │ │ │ │ ├── Any.cpp │ │ │ │ ├── Any.h │ │ │ │ ├── Arrays.cpp │ │ │ │ ├── Arrays.h │ │ │ │ ├── BitSet.h │ │ │ │ ├── CPPUtils.cpp │ │ │ │ ├── CPPUtils.h │ │ │ │ ├── Casts.h │ │ │ │ ├── Declarations.h │ │ │ │ ├── StringUtils.cpp │ │ │ │ ├── StringUtils.h │ │ │ │ ├── Unicode.h │ │ │ │ ├── Utf8.cpp │ │ │ │ └── Utf8.h │ │ │ └── tree/ │ │ │ ├── AbstractParseTreeVisitor.h │ │ │ ├── ErrorNode.h │ │ │ ├── ErrorNodeImpl.cpp │ │ │ ├── ErrorNodeImpl.h │ │ │ ├── IterativeParseTreeWalker.cpp │ │ │ ├── IterativeParseTreeWalker.h │ │ │ ├── ParseTree.cpp │ │ │ ├── ParseTree.h │ │ │ ├── ParseTreeListener.cpp │ │ │ ├── ParseTreeListener.h │ │ │ ├── ParseTreeProperty.h │ │ │ ├── ParseTreeType.h │ │ │ ├── ParseTreeVisitor.cpp │ │ │ ├── ParseTreeVisitor.h │ │ │ ├── ParseTreeWalker.cpp │ │ │ ├── ParseTreeWalker.h │ │ │ ├── TerminalNode.h │ │ │ ├── TerminalNodeImpl.cpp │ │ │ ├── TerminalNodeImpl.h │ │ │ ├── Trees.cpp │ │ │ ├── Trees.h │ │ │ ├── pattern/ │ │ │ │ ├── Chunk.cpp │ │ │ │ ├── Chunk.h │ │ │ │ ├── ParseTreeMatch.cpp │ │ │ │ ├── ParseTreeMatch.h │ │ │ │ ├── ParseTreePattern.cpp │ │ │ │ ├── ParseTreePattern.h │ │ │ │ ├── ParseTreePatternMatcher.cpp │ │ │ │ ├── ParseTreePatternMatcher.h │ │ │ │ ├── RuleTagToken.cpp │ │ │ │ ├── RuleTagToken.h │ │ │ │ ├── TagChunk.cpp │ │ │ │ ├── TagChunk.h │ │ │ │ ├── TextChunk.cpp │ │ │ │ ├── TextChunk.h │ │ │ │ ├── TokenTagToken.cpp │ │ │ │ └── TokenTagToken.h │ │ │ └── xpath/ │ │ │ ├── XPath.cpp │ │ │ ├── XPath.h │ │ │ ├── XPathElement.cpp │ │ │ ├── XPathElement.h │ │ │ ├── XPathLexer.cpp │ │ │ ├── XPathLexer.g4 │ │ │ ├── XPathLexer.h │ │ │ ├── XPathLexer.tokens │ │ │ ├── XPathLexerErrorListener.cpp │ │ │ ├── XPathLexerErrorListener.h │ │ │ ├── XPathRuleAnywhereElement.cpp │ │ │ ├── XPathRuleAnywhereElement.h │ │ │ ├── XPathRuleElement.cpp │ │ │ ├── XPathRuleElement.h │ │ │ ├── XPathTokenAnywhereElement.cpp │ │ │ ├── XPathTokenAnywhereElement.h │ │ │ ├── XPathTokenElement.cpp │ │ │ ├── XPathTokenElement.h │ │ │ ├── XPathWildcardAnywhereElement.cpp │ │ │ ├── XPathWildcardAnywhereElement.h │ │ │ ├── XPathWildcardElement.cpp │ │ │ └── XPathWildcardElement.h │ │ └── tests/ │ │ └── Utf8Test.cpp │ ├── Dart/ │ │ ├── .gitignore │ │ ├── LICENSE │ │ ├── README.md │ │ ├── analysis_options.yaml │ │ ├── benchmark/ │ │ │ └── src/ │ │ │ └── util/ │ │ │ └── bit_set.dart │ │ ├── lib/ │ │ │ ├── antlr4.dart │ │ │ └── src/ │ │ │ ├── atn/ │ │ │ │ ├── atn.dart │ │ │ │ └── src/ │ │ │ │ ├── atn.dart │ │ │ │ ├── atn_config.dart │ │ │ │ ├── atn_config_set.dart │ │ │ │ ├── atn_deserializer.dart │ │ │ │ ├── atn_simulator.dart │ │ │ │ ├── atn_state.dart │ │ │ │ ├── atn_type.dart │ │ │ │ ├── info.dart │ │ │ │ ├── lexer_action.dart │ │ │ │ ├── lexer_action_executor.dart │ │ │ │ ├── lexer_atn_simulator.dart │ │ │ │ ├── parser_atn_simulator.dart │ │ │ │ ├── profiling_atn_simulator.dart │ │ │ │ ├── semantic_context.dart │ │ │ │ └── transition.dart │ │ │ ├── dfa/ │ │ │ │ ├── dfa.dart │ │ │ │ └── src/ │ │ │ │ ├── dfa.dart │ │ │ │ ├── dfa_serializer.dart │ │ │ │ └── dfa_state.dart │ │ │ ├── error/ │ │ │ │ ├── error.dart │ │ │ │ └── src/ │ │ │ │ ├── diagnostic_error_listener.dart │ │ │ │ ├── error_listener.dart │ │ │ │ ├── error_strategy.dart │ │ │ │ └── errors.dart │ │ │ ├── input_stream.dart │ │ │ ├── interval_set.dart │ │ │ ├── lexer.dart │ │ │ ├── ll1_analyzer.dart │ │ │ ├── misc/ │ │ │ │ ├── misc.dart │ │ │ │ └── src/ │ │ │ │ ├── multi_map.dart │ │ │ │ └── pair.dart │ │ │ ├── parser.dart │ │ │ ├── parser_interpreter.dart │ │ │ ├── parser_rule_context.dart │ │ │ ├── prediction_context.dart │ │ │ ├── recognizer.dart │ │ │ ├── rule_context.dart │ │ │ ├── runtime_meta_data.dart │ │ │ ├── token.dart │ │ │ ├── token_factory.dart │ │ │ ├── token_source.dart │ │ │ ├── token_stream.dart │ │ │ ├── tree/ │ │ │ │ ├── src/ │ │ │ │ │ ├── pattern/ │ │ │ │ │ │ ├── chunk.dart │ │ │ │ │ │ └── parse_tree_match.dart │ │ │ │ │ ├── tree.dart │ │ │ │ │ └── trees.dart │ │ │ │ └── tree.dart │ │ │ ├── util/ │ │ │ │ ├── bit_operation_util.dart │ │ │ │ ├── bit_operation_util_html.dart │ │ │ │ ├── bit_set.dart │ │ │ │ ├── murmur_hash.dart │ │ │ │ ├── platform_html.dart │ │ │ │ ├── platform_io.dart │ │ │ │ ├── platform_stub.dart │ │ │ │ └── utils.dart │ │ │ └── vocabulary.dart │ │ ├── pubspec.yaml │ │ └── test/ │ │ └── src/ │ │ └── util/ │ │ └── bit_set_test.dart │ ├── Go/ │ │ └── antlr/ │ │ ├── README.adoc │ │ └── v4/ │ │ ├── LICENSE │ │ ├── antlrdoc.go │ │ ├── atn.go │ │ ├── atn_config.go │ │ ├── atn_config_set.go │ │ ├── atn_deserialization_options.go │ │ ├── atn_deserializer.go │ │ ├── atn_simulator.go │ │ ├── atn_state.go │ │ ├── atn_type.go │ │ ├── char_stream.go │ │ ├── common_token_factory.go │ │ ├── common_token_stream.go │ │ ├── comparators.go │ │ ├── configuration.go │ │ ├── dfa.go │ │ ├── dfa_serializer.go │ │ ├── dfa_state.go │ │ ├── diagnostic_error_listener.go │ │ ├── error_listener.go │ │ ├── error_strategy.go │ │ ├── errors.go │ │ ├── file_stream.go │ │ ├── go.mod │ │ ├── go.sum │ │ ├── input_stream.go │ │ ├── int_stream.go │ │ ├── interval_set.go │ │ ├── jcollect.go │ │ ├── lexer.go │ │ ├── lexer_action.go │ │ ├── lexer_action_executor.go │ │ ├── lexer_atn_simulator.go │ │ ├── ll1_analyzer.go │ │ ├── mutex.go │ │ ├── mutex_nomutex.go │ │ ├── nostatistics.go │ │ ├── parser.go │ │ ├── parser_atn_simulator.go │ │ ├── parser_rule_context.go │ │ ├── prediction_context.go │ │ ├── prediction_context_cache.go │ │ ├── prediction_mode.go │ │ ├── recognizer.go │ │ ├── rule_context.go │ │ ├── semantic_context.go │ │ ├── statistics.go │ │ ├── stats_data.go │ │ ├── token.go │ │ ├── token_source.go │ │ ├── token_stream.go │ │ ├── tokenstream_rewriter.go │ │ ├── trace_listener.go │ │ ├── transition.go │ │ ├── tree.go │ │ ├── trees.go │ │ └── utils.go │ ├── Java/ │ │ ├── doxyfile │ │ ├── nb-configuration.xml │ │ ├── pom.xml │ │ └── src/ │ │ ├── main/ │ │ │ └── dot/ │ │ │ └── org/ │ │ │ └── antlr/ │ │ │ └── v4/ │ │ │ └── runtime/ │ │ │ └── atn/ │ │ │ └── images/ │ │ │ ├── ArrayMerge_DiffTopDiffPar.dot │ │ │ ├── ArrayMerge_EqualTop.dot │ │ │ ├── ArrayMerge_ShareTopDiffPar.dot │ │ │ ├── ArrayMerge_ShareTopSamePar.dot │ │ │ ├── ArrayMerge_ShareTopSharePar.dot │ │ │ ├── Block.dot │ │ │ ├── ClosureGreedy.dot │ │ │ ├── ClosureNonGreedy.dot │ │ │ ├── FullMerge_EmptyRoot.dot │ │ │ ├── FullMerge_EmptyRoots.dot │ │ │ ├── FullMerge_SameRoot.dot │ │ │ ├── LocalMerge_DiffRoots.dot │ │ │ ├── LocalMerge_EmptyParent.dot │ │ │ ├── LocalMerge_EmptyRoot.dot │ │ │ ├── OptionalGreedy.dot │ │ │ ├── OptionalNonGreedy.dot │ │ │ ├── PositiveClosureGreedy.dot │ │ │ ├── PositiveClosureNonGreedy.dot │ │ │ ├── Rule.dot │ │ │ ├── SingletonMerge_DiffRootDiffPar.dot │ │ │ ├── SingletonMerge_DiffRootSamePar.dot │ │ │ ├── SingletonMerge_SameRootDiffPar.dot │ │ │ └── SingletonMerge_SameRootSamePar.dot │ │ └── org/ │ │ └── antlr/ │ │ └── v4/ │ │ └── runtime/ │ │ ├── ANTLRErrorListener.java │ │ ├── ANTLRErrorStrategy.java │ │ ├── ANTLRFileStream.java │ │ ├── ANTLRInputStream.java │ │ ├── BailErrorStrategy.java │ │ ├── BaseErrorListener.java │ │ ├── BufferedTokenStream.java │ │ ├── CharStream.java │ │ ├── CharStreams.java │ │ ├── CodePointBuffer.java │ │ ├── CodePointCharStream.java │ │ ├── CommonToken.java │ │ ├── CommonTokenFactory.java │ │ ├── CommonTokenStream.java │ │ ├── ConsoleErrorListener.java │ │ ├── DefaultErrorStrategy.java │ │ ├── DiagnosticErrorListener.java │ │ ├── FailedPredicateException.java │ │ ├── InputMismatchException.java │ │ ├── IntStream.java │ │ ├── InterpreterRuleContext.java │ │ ├── Lexer.java │ │ ├── LexerInterpreter.java │ │ ├── LexerNoViableAltException.java │ │ ├── ListTokenSource.java │ │ ├── NoViableAltException.java │ │ ├── Parser.java │ │ ├── ParserInterpreter.java │ │ ├── ParserRuleContext.java │ │ ├── ProxyErrorListener.java │ │ ├── RecognitionException.java │ │ ├── Recognizer.java │ │ ├── RuleContext.java │ │ ├── RuleContextWithAltNum.java │ │ ├── RuntimeMetaData.java │ │ ├── Token.java │ │ ├── TokenFactory.java │ │ ├── TokenSource.java │ │ ├── TokenStream.java │ │ ├── TokenStreamRewriter.java │ │ ├── UnbufferedCharStream.java │ │ ├── UnbufferedTokenStream.java │ │ ├── Vocabulary.java │ │ ├── VocabularyImpl.java │ │ ├── WritableToken.java │ │ ├── atn/ │ │ │ ├── ATN.java │ │ │ ├── ATNConfig.java │ │ │ ├── ATNConfigSet.java │ │ │ ├── ATNDeserializationOptions.java │ │ │ ├── ATNDeserializer.java │ │ │ ├── ATNSerializer.java │ │ │ ├── ATNSimulator.java │ │ │ ├── ATNState.java │ │ │ ├── ATNType.java │ │ │ ├── AbstractPredicateTransition.java │ │ │ ├── ActionTransition.java │ │ │ ├── AmbiguityInfo.java │ │ │ ├── ArrayPredictionContext.java │ │ │ ├── AtomTransition.java │ │ │ ├── BasicBlockStartState.java │ │ │ ├── BasicState.java │ │ │ ├── BlockEndState.java │ │ │ ├── BlockStartState.java │ │ │ ├── CodePointTransitions.java │ │ │ ├── ContextSensitivityInfo.java │ │ │ ├── DecisionEventInfo.java │ │ │ ├── DecisionInfo.java │ │ │ ├── DecisionState.java │ │ │ ├── EmptyPredictionContext.java │ │ │ ├── EpsilonTransition.java │ │ │ ├── ErrorInfo.java │ │ │ ├── LL1Analyzer.java │ │ │ ├── LexerATNConfig.java │ │ │ ├── LexerATNSimulator.java │ │ │ ├── LexerAction.java │ │ │ ├── LexerActionExecutor.java │ │ │ ├── LexerActionType.java │ │ │ ├── LexerChannelAction.java │ │ │ ├── LexerCustomAction.java │ │ │ ├── LexerIndexedCustomAction.java │ │ │ ├── LexerModeAction.java │ │ │ ├── LexerMoreAction.java │ │ │ ├── LexerPopModeAction.java │ │ │ ├── LexerPushModeAction.java │ │ │ ├── LexerSkipAction.java │ │ │ ├── LexerTypeAction.java │ │ │ ├── LookaheadEventInfo.java │ │ │ ├── LoopEndState.java │ │ │ ├── NotSetTransition.java │ │ │ ├── OrderedATNConfigSet.java │ │ │ ├── ParseInfo.java │ │ │ ├── ParserATNSimulator.java │ │ │ ├── PlusBlockStartState.java │ │ │ ├── PlusLoopbackState.java │ │ │ ├── PrecedencePredicateTransition.java │ │ │ ├── PredicateEvalInfo.java │ │ │ ├── PredicateTransition.java │ │ │ ├── PredictionContext.java │ │ │ ├── PredictionContextCache.java │ │ │ ├── PredictionMode.java │ │ │ ├── ProfilingATNSimulator.java │ │ │ ├── RangeTransition.java │ │ │ ├── RuleStartState.java │ │ │ ├── RuleStopState.java │ │ │ ├── RuleTransition.java │ │ │ ├── SemanticContext.java │ │ │ ├── SetTransition.java │ │ │ ├── SingletonPredictionContext.java │ │ │ ├── StarBlockStartState.java │ │ │ ├── StarLoopEntryState.java │ │ │ ├── StarLoopbackState.java │ │ │ ├── TokensStartState.java │ │ │ ├── Transition.java │ │ │ └── WildcardTransition.java │ │ ├── dfa/ │ │ │ ├── DFA.java │ │ │ ├── DFASerializer.java │ │ │ ├── DFAState.java │ │ │ └── LexerDFASerializer.java │ │ ├── misc/ │ │ │ ├── AbstractEqualityComparator.java │ │ │ ├── Array2DHashSet.java │ │ │ ├── DoubleKeyMap.java │ │ │ ├── EqualityComparator.java │ │ │ ├── FlexibleHashMap.java │ │ │ ├── IntSet.java │ │ │ ├── IntegerList.java │ │ │ ├── IntegerStack.java │ │ │ ├── InterpreterDataReader.java │ │ │ ├── Interval.java │ │ │ ├── IntervalSet.java │ │ │ ├── LogManager.java │ │ │ ├── MultiMap.java │ │ │ ├── MurmurHash.java │ │ │ ├── NotNull.java │ │ │ ├── ObjectEqualityComparator.java │ │ │ ├── OrderedHashSet.java │ │ │ ├── Pair.java │ │ │ ├── ParseCancellationException.java │ │ │ ├── Predicate.java │ │ │ ├── TestRig.java │ │ │ ├── Triple.java │ │ │ └── Utils.java │ │ └── tree/ │ │ ├── AbstractParseTreeVisitor.java │ │ ├── ErrorNode.java │ │ ├── ErrorNodeImpl.java │ │ ├── IterativeParseTreeWalker.java │ │ ├── ParseTree.java │ │ ├── ParseTreeListener.java │ │ ├── ParseTreeProperty.java │ │ ├── ParseTreeVisitor.java │ │ ├── ParseTreeWalker.java │ │ ├── RuleNode.java │ │ ├── SyntaxTree.java │ │ ├── TerminalNode.java │ │ ├── TerminalNodeImpl.java │ │ ├── Tree.java │ │ ├── Trees.java │ │ ├── pattern/ │ │ │ ├── Chunk.java │ │ │ ├── ParseTreeMatch.java │ │ │ ├── ParseTreePattern.java │ │ │ ├── ParseTreePatternMatcher.java │ │ │ ├── RuleTagToken.java │ │ │ ├── TagChunk.java │ │ │ ├── TextChunk.java │ │ │ └── TokenTagToken.java │ │ └── xpath/ │ │ ├── XPath.java │ │ ├── XPathElement.java │ │ ├── XPathLexer.java │ │ ├── XPathLexerErrorListener.java │ │ ├── XPathRuleAnywhereElement.java │ │ ├── XPathRuleElement.java │ │ ├── XPathTokenAnywhereElement.java │ │ ├── XPathTokenElement.java │ │ ├── XPathWildcardAnywhereElement.java │ │ └── XPathWildcardElement.java │ ├── JavaScript/ │ │ ├── .babelrc │ │ ├── .c8rc.json │ │ ├── .eslintignore │ │ ├── .eslintrc.yml │ │ ├── .gitignore │ │ ├── README.md │ │ ├── package.json │ │ ├── spec/ │ │ │ ├── BitSetSpec.js │ │ │ ├── HashMapSpec.js │ │ │ ├── HashSetSpec.js │ │ │ ├── IntervalSetSpec.js │ │ │ ├── helpers/ │ │ │ │ └── Reporter.js │ │ │ ├── imports/ │ │ │ │ ├── NodeCommonJSImportSpec.cjs │ │ │ │ ├── NodeEsmImportSpec.mjs │ │ │ │ └── builds/ │ │ │ │ ├── node-cjs-ts/ │ │ │ │ │ ├── .gitignore │ │ │ │ │ ├── index.ts │ │ │ │ │ ├── package.json │ │ │ │ │ ├── test.sh │ │ │ │ │ └── tsconfig.node.commonjs.json │ │ │ │ └── node-esm-ts/ │ │ │ │ ├── .gitignore │ │ │ │ ├── index.ts │ │ │ │ ├── package.json │ │ │ │ ├── test.sh │ │ │ │ ├── tsconfig.bundler.es2022.json │ │ │ │ └── tsconfig.node16.json │ │ │ ├── rewriter/ │ │ │ │ ├── Makefile │ │ │ │ ├── TokenStreamRewriterSpec.js │ │ │ │ ├── abc.g4 │ │ │ │ ├── calc.g4 │ │ │ │ └── generatedCode/ │ │ │ │ ├── abc.js │ │ │ │ └── calc.js │ │ │ └── support/ │ │ │ └── jasmine.json │ │ ├── src/ │ │ │ └── antlr4/ │ │ │ ├── BufferedTokenStream.d.ts │ │ │ ├── BufferedTokenStream.js │ │ │ ├── CharStream.d.ts │ │ │ ├── CharStream.js │ │ │ ├── CharStreams.d.ts │ │ │ ├── CharStreams.js │ │ │ ├── CommonToken.d.ts │ │ │ ├── CommonToken.js │ │ │ ├── CommonTokenFactory.js │ │ │ ├── CommonTokenStream.d.ts │ │ │ ├── CommonTokenStream.js │ │ │ ├── FileStream.d.ts │ │ │ ├── FileStream.js │ │ │ ├── InputStream.d.ts │ │ │ ├── InputStream.js │ │ │ ├── Lexer.d.ts │ │ │ ├── Lexer.js │ │ │ ├── Parser.d.ts │ │ │ ├── Parser.js │ │ │ ├── Recognizer.d.ts │ │ │ ├── Recognizer.js │ │ │ ├── Token.d.ts │ │ │ ├── Token.js │ │ │ ├── TokenSource.d.ts │ │ │ ├── TokenSource.js │ │ │ ├── TokenStream.d.ts │ │ │ ├── TokenStream.js │ │ │ ├── TokenStreamRewriter.d.ts │ │ │ ├── TokenStreamRewriter.js │ │ │ ├── TraceListener.js │ │ │ ├── action/ │ │ │ │ ├── LexerAction.js │ │ │ │ ├── LexerChannelAction.js │ │ │ │ ├── LexerCustomAction.js │ │ │ │ ├── LexerIndexedCustomAction.js │ │ │ │ ├── LexerModeAction.js │ │ │ │ ├── LexerMoreAction.js │ │ │ │ ├── LexerPopModeAction.js │ │ │ │ ├── LexerPushModeAction.js │ │ │ │ ├── LexerSkipAction.js │ │ │ │ └── LexerTypeAction.js │ │ │ ├── atn/ │ │ │ │ ├── ATN.d.ts │ │ │ │ ├── ATN.js │ │ │ │ ├── ATNConfig.d.ts │ │ │ │ ├── ATNConfig.js │ │ │ │ ├── ATNConfigSet.d.ts │ │ │ │ ├── ATNConfigSet.js │ │ │ │ ├── ATNDeserializationOptions.d.ts │ │ │ │ ├── ATNDeserializationOptions.js │ │ │ │ ├── ATNDeserializer.d.ts │ │ │ │ ├── ATNDeserializer.js │ │ │ │ ├── ATNSimulator.d.ts │ │ │ │ ├── ATNSimulator.js │ │ │ │ ├── ATNType.js │ │ │ │ ├── AbstractPredicateTransition.js │ │ │ │ ├── LL1Analyzer.js │ │ │ │ ├── LexerATNConfig.js │ │ │ │ ├── LexerATNSimulator.d.ts │ │ │ │ ├── LexerATNSimulator.js │ │ │ │ ├── LexerActionExecutor.js │ │ │ │ ├── LexerActionType.js │ │ │ │ ├── OrderedATNConfigSet.js │ │ │ │ ├── ParserATNSimulator.d.ts │ │ │ │ ├── ParserATNSimulator.js │ │ │ │ ├── PrecedencePredicate.js │ │ │ │ ├── Predicate.js │ │ │ │ ├── PredictionContextCache.d.ts │ │ │ │ ├── PredictionContextCache.js │ │ │ │ ├── PredictionMode.d.ts │ │ │ │ ├── PredictionMode.js │ │ │ │ ├── SemanticContext.js │ │ │ │ ├── index.d.ts │ │ │ │ └── index.js │ │ │ ├── context/ │ │ │ │ ├── ArrayPredictionContext.js │ │ │ │ ├── EmptyPredictionContext.js │ │ │ │ ├── InterpreterRuleContext.js │ │ │ │ ├── ParserRuleContext.d.ts │ │ │ │ ├── ParserRuleContext.js │ │ │ │ ├── PredictionContext.js │ │ │ │ ├── PredictionContextUtils.js │ │ │ │ ├── RuleContext.d.ts │ │ │ │ ├── RuleContext.js │ │ │ │ ├── SingletonPredictionContext.js │ │ │ │ ├── index.d.ts │ │ │ │ └── index.js │ │ │ ├── dfa/ │ │ │ │ ├── DFA.d.ts │ │ │ │ ├── DFA.js │ │ │ │ ├── DFASerializer.js │ │ │ │ ├── DFAState.js │ │ │ │ ├── LexerDFASerializer.js │ │ │ │ ├── PredPrediction.js │ │ │ │ ├── index.d.ts │ │ │ │ └── index.js │ │ │ ├── error/ │ │ │ │ ├── BailErrorStrategy.d.ts │ │ │ │ ├── BailErrorStrategy.js │ │ │ │ ├── ConsoleErrorListener.js │ │ │ │ ├── DefaultErrorStrategy.d.ts │ │ │ │ ├── DefaultErrorStrategy.js │ │ │ │ ├── DiagnosticErrorListener.d.ts │ │ │ │ ├── DiagnosticErrorListener.js │ │ │ │ ├── ErrorListener.d.ts │ │ │ │ ├── ErrorListener.js │ │ │ │ ├── ErrorStrategy.d.ts │ │ │ │ ├── ErrorStrategy.js │ │ │ │ ├── FailedPredicateException.d.ts │ │ │ │ ├── FailedPredicateException.js │ │ │ │ ├── InputMismatchException.d.ts │ │ │ │ ├── InputMismatchException.js │ │ │ │ ├── LexerNoViableAltException.d.ts │ │ │ │ ├── LexerNoViableAltException.js │ │ │ │ ├── NoViableAltException.d.ts │ │ │ │ ├── NoViableAltException.js │ │ │ │ ├── ParseCancellationException.js │ │ │ │ ├── ProxyErrorListener.js │ │ │ │ ├── RecognitionException.d.ts │ │ │ │ ├── RecognitionException.js │ │ │ │ ├── index.d.ts │ │ │ │ └── index.js │ │ │ ├── index.d.cts │ │ │ ├── index.node.js │ │ │ ├── index.web.js │ │ │ ├── misc/ │ │ │ │ ├── AltDict.js │ │ │ │ ├── BitSet.d.ts │ │ │ │ ├── BitSet.js │ │ │ │ ├── HashCode.js │ │ │ │ ├── HashMap.js │ │ │ │ ├── HashSet.js │ │ │ │ ├── Interval.d.ts │ │ │ │ ├── Interval.js │ │ │ │ ├── IntervalSet.d.ts │ │ │ │ ├── IntervalSet.js │ │ │ │ ├── index.d.ts │ │ │ │ └── index.js │ │ │ ├── state/ │ │ │ │ ├── ATNState.d.ts │ │ │ │ ├── ATNState.js │ │ │ │ ├── BasicBlockStartState.js │ │ │ │ ├── BasicState.js │ │ │ │ ├── BlockEndState.js │ │ │ │ ├── BlockStartState.js │ │ │ │ ├── DecisionState.d.ts │ │ │ │ ├── DecisionState.js │ │ │ │ ├── LoopEndState.js │ │ │ │ ├── PlusBlockStartState.js │ │ │ │ ├── PlusLoopbackState.js │ │ │ │ ├── RuleStartState.d.ts │ │ │ │ ├── RuleStartState.js │ │ │ │ ├── RuleStopState.d.ts │ │ │ │ ├── RuleStopState.js │ │ │ │ ├── StarBlockStartState.js │ │ │ │ ├── StarLoopEntryState.js │ │ │ │ ├── StarLoopbackState.js │ │ │ │ ├── TokensStartState.js │ │ │ │ └── index.d.ts │ │ │ ├── transition/ │ │ │ │ ├── ActionTransition.js │ │ │ │ ├── AtomTransition.js │ │ │ │ ├── EpsilonTransition.js │ │ │ │ ├── NotSetTransition.js │ │ │ │ ├── PrecedencePredicateTransition.js │ │ │ │ ├── PredicateTransition.js │ │ │ │ ├── RangeTransition.js │ │ │ │ ├── RuleTransition.js │ │ │ │ ├── SetTransition.js │ │ │ │ ├── Transition.js │ │ │ │ └── WildcardTransition.js │ │ │ ├── tree/ │ │ │ │ ├── ErrorNode.d.ts │ │ │ │ ├── ErrorNode.js │ │ │ │ ├── ErrorNodeImpl.js │ │ │ │ ├── ParseTree.d.ts │ │ │ │ ├── ParseTree.js │ │ │ │ ├── ParseTreeListener.d.ts │ │ │ │ ├── ParseTreeListener.js │ │ │ │ ├── ParseTreeVisitor.d.ts │ │ │ │ ├── ParseTreeVisitor.js │ │ │ │ ├── ParseTreeWalker.d.ts │ │ │ │ ├── ParseTreeWalker.js │ │ │ │ ├── RuleNode.d.ts │ │ │ │ ├── RuleNode.js │ │ │ │ ├── SyntaxTree.d.ts │ │ │ │ ├── SyntaxTree.js │ │ │ │ ├── TerminalNode.d.ts │ │ │ │ ├── TerminalNode.js │ │ │ │ ├── TerminalNodeImpl.js │ │ │ │ ├── Tree.d.ts │ │ │ │ ├── Tree.js │ │ │ │ ├── Trees.d.ts │ │ │ │ ├── Trees.js │ │ │ │ ├── index.d.ts │ │ │ │ └── index.js │ │ │ └── utils/ │ │ │ ├── DoubleDict.js │ │ │ ├── Printer.d.ts │ │ │ ├── arrayToString.d.ts │ │ │ ├── arrayToString.js │ │ │ ├── equalArrays.js │ │ │ ├── escapeWhitespace.js │ │ │ ├── index.d.ts │ │ │ ├── index.js │ │ │ ├── standardEqualsFunction.js │ │ │ ├── standardHashCodeFunction.js │ │ │ ├── stringHashCode.js │ │ │ ├── stringToCharArray.d.ts │ │ │ ├── stringToCharArray.js │ │ │ ├── titleCase.js │ │ │ └── valueToString.js │ │ ├── test-builds.sh │ │ ├── tsconfig.json │ │ └── webpack.config.js │ ├── Python3/ │ │ ├── MANIFEST.in │ │ ├── README.txt │ │ ├── RELEASE-4.5.txt │ │ ├── pyproject.toml │ │ ├── src/ │ │ │ └── antlr4/ │ │ │ ├── BufferedTokenStream.py │ │ │ ├── CommonTokenFactory.py │ │ │ ├── CommonTokenStream.py │ │ │ ├── FileStream.py │ │ │ ├── InputStream.py │ │ │ ├── IntervalSet.py │ │ │ ├── LL1Analyzer.py │ │ │ ├── Lexer.py │ │ │ ├── ListTokenSource.py │ │ │ ├── Parser.py │ │ │ ├── ParserInterpreter.py │ │ │ ├── ParserRuleContext.py │ │ │ ├── PredictionContext.py │ │ │ ├── Recognizer.py │ │ │ ├── RuleContext.py │ │ │ ├── StdinStream.py │ │ │ ├── Token.py │ │ │ ├── TokenStreamRewriter.py │ │ │ ├── Utils.py │ │ │ ├── __init__.py │ │ │ ├── _pygrun.py │ │ │ ├── atn/ │ │ │ │ ├── ATN.py │ │ │ │ ├── ATNConfig.py │ │ │ │ ├── ATNConfigSet.py │ │ │ │ ├── ATNDeserializationOptions.py │ │ │ │ ├── ATNDeserializer.py │ │ │ │ ├── ATNSimulator.py │ │ │ │ ├── ATNState.py │ │ │ │ ├── ATNType.py │ │ │ │ ├── LexerATNSimulator.py │ │ │ │ ├── LexerAction.py │ │ │ │ ├── LexerActionExecutor.py │ │ │ │ ├── ParserATNSimulator.py │ │ │ │ ├── PredictionMode.py │ │ │ │ ├── SemanticContext.py │ │ │ │ ├── Transition.py │ │ │ │ └── __init__.py │ │ │ ├── dfa/ │ │ │ │ ├── DFA.py │ │ │ │ ├── DFASerializer.py │ │ │ │ ├── DFAState.py │ │ │ │ └── __init__.py │ │ │ ├── error/ │ │ │ │ ├── DiagnosticErrorListener.py │ │ │ │ ├── ErrorListener.py │ │ │ │ ├── ErrorStrategy.py │ │ │ │ ├── Errors.py │ │ │ │ └── __init__.py │ │ │ ├── tree/ │ │ │ │ ├── Chunk.py │ │ │ │ ├── ParseTreeMatch.py │ │ │ │ ├── ParseTreePattern.py │ │ │ │ ├── ParseTreePatternMatcher.py │ │ │ │ ├── RuleTagToken.py │ │ │ │ ├── TokenTagToken.py │ │ │ │ ├── Tree.py │ │ │ │ ├── Trees.py │ │ │ │ └── __init__.py │ │ │ └── xpath/ │ │ │ ├── XPath.py │ │ │ ├── XPathLexer.g4 │ │ │ ├── XPathLexer.py │ │ │ └── __init__.py │ │ └── tests/ │ │ ├── TestFileStream.py │ │ ├── TestInputStream.py │ │ ├── TestIntervalSet.py │ │ ├── TestRecognizer.py │ │ ├── TestTokenStreamRewriter.py │ │ ├── __init__.py │ │ ├── c.c │ │ ├── ctest.py │ │ ├── expr/ │ │ │ ├── Expr.g4 │ │ │ ├── ExprLexer.py │ │ │ └── ExprParser.py │ │ ├── mocks/ │ │ │ ├── TestLexer.py │ │ │ └── __init__.py │ │ ├── parser/ │ │ │ ├── __init__.py │ │ │ ├── clexer.py │ │ │ └── cparser.py │ │ ├── run.py │ │ └── xpathtest.py │ └── Swift/ │ ├── .gitignore │ ├── Sources/ │ │ ├── Antlr4/ │ │ │ ├── ANTLRErrorListener.swift │ │ │ ├── ANTLRErrorStrategy.swift │ │ │ ├── ANTLRFileStream.swift │ │ │ ├── ANTLRInputStream.swift │ │ │ ├── BailErrorStrategy.swift │ │ │ ├── BaseErrorListener.swift │ │ │ ├── BufferedTokenStream.swift │ │ │ ├── CharStream.swift │ │ │ ├── CommonToken.swift │ │ │ ├── CommonTokenFactory.swift │ │ │ ├── CommonTokenStream.swift │ │ │ ├── ConsoleErrorListener.swift │ │ │ ├── DefaultErrorStrategy.swift │ │ │ ├── DiagnosticErrorListener.swift │ │ │ ├── FailedPredicateException.swift │ │ │ ├── InputMismatchException.swift │ │ │ ├── IntStream.swift │ │ │ ├── InterpreterRuleContext.swift │ │ │ ├── Lexer.swift │ │ │ ├── LexerInterpreter.swift │ │ │ ├── LexerNoViableAltException.swift │ │ │ ├── ListTokenSource.swift │ │ │ ├── NoViableAltException.swift │ │ │ ├── Parser.swift │ │ │ ├── ParserInterpreter.swift │ │ │ ├── ParserRuleContext.swift │ │ │ ├── ProxyErrorListener.swift │ │ │ ├── RecognitionException.swift │ │ │ ├── Recognizer.swift │ │ │ ├── RuleContext.swift │ │ │ ├── RuntimeMetaData.swift │ │ │ ├── Token.swift │ │ │ ├── TokenFactory.swift │ │ │ ├── TokenSource.swift │ │ │ ├── TokenStream.swift │ │ │ ├── TokenStreamRewriter.swift │ │ │ ├── UnbufferedCharStream.swift │ │ │ ├── UnbufferedTokenStream.swift │ │ │ ├── VocabularySingle.swift │ │ │ ├── WritableToken.swift │ │ │ ├── atn/ │ │ │ │ ├── ATN.swift │ │ │ │ ├── ATNConfig.swift │ │ │ │ ├── ATNConfigSet.swift │ │ │ │ ├── ATNDeserializationOptions.swift │ │ │ │ ├── ATNDeserializer.swift │ │ │ │ ├── ATNSimulator.swift │ │ │ │ ├── ATNState.swift │ │ │ │ ├── ATNType.swift │ │ │ │ ├── AbstractPredicateTransition.swift │ │ │ │ ├── ActionTransition.swift │ │ │ │ ├── AmbiguityInfo.swift │ │ │ │ ├── ArrayPredictionContext.swift │ │ │ │ ├── AtomTransition.swift │ │ │ │ ├── BasicBlockStartState.swift │ │ │ │ ├── BasicState.swift │ │ │ │ ├── BlockEndState.swift │ │ │ │ ├── BlockStartState.swift │ │ │ │ ├── ContextSensitivityInfo.swift │ │ │ │ ├── DecisionEventInfo.swift │ │ │ │ ├── DecisionInfo.swift │ │ │ │ ├── DecisionState.swift │ │ │ │ ├── DefaultATNConfig.swift │ │ │ │ ├── EmptyPredictionContext.swift │ │ │ │ ├── EpsilonTransition.swift │ │ │ │ ├── ErrorInfo.swift │ │ │ │ ├── LL1Analyzer.swift │ │ │ │ ├── LexerATNConfig.swift │ │ │ │ ├── LexerATNSimulator.swift │ │ │ │ ├── LexerAction.swift │ │ │ │ ├── LexerActionExecutor.swift │ │ │ │ ├── LexerActionType.swift │ │ │ │ ├── LexerChannelAction.swift │ │ │ │ ├── LexerCustomAction.swift │ │ │ │ ├── LexerIndexedCustomAction.swift │ │ │ │ ├── LexerModeAction.swift │ │ │ │ ├── LexerMoreAction.swift │ │ │ │ ├── LexerPopModeAction.swift │ │ │ │ ├── LexerPushModeAction.swift │ │ │ │ ├── LexerSkipAction.swift │ │ │ │ ├── LexerTypeAction.swift │ │ │ │ ├── LookaheadEventInfo.swift │ │ │ │ ├── LookupATNConfig.swift │ │ │ │ ├── LookupDictionary.swift │ │ │ │ ├── LoopEndState.swift │ │ │ │ ├── NotSetTransition.swift │ │ │ │ ├── ParseInfo.swift │ │ │ │ ├── ParserATNSimulator.swift │ │ │ │ ├── PlusBlockStartState.swift │ │ │ │ ├── PlusLoopbackState.swift │ │ │ │ ├── PrecedencePredicateTransition.swift │ │ │ │ ├── PredicateEvalInfo.swift │ │ │ │ ├── PredicateTransition.swift │ │ │ │ ├── PredictionContext.swift │ │ │ │ ├── PredictionContextCache.swift │ │ │ │ ├── PredictionMode.swift │ │ │ │ ├── ProfilingATNSimulator.swift │ │ │ │ ├── RangeTransition.swift │ │ │ │ ├── RuleStartState.swift │ │ │ │ ├── RuleStopState.swift │ │ │ │ ├── RuleTransition.swift │ │ │ │ ├── SemanticContext.swift │ │ │ │ ├── SetTransition.swift │ │ │ │ ├── SingletonPredictionContext.swift │ │ │ │ ├── StarBlockStartState.swift │ │ │ │ ├── StarLoopEntryState.swift │ │ │ │ ├── StarLoopbackState.swift │ │ │ │ ├── TokensStartState.swift │ │ │ │ ├── Transition.swift │ │ │ │ └── WildcardTransition.swift │ │ │ ├── dfa/ │ │ │ │ ├── DFA.swift │ │ │ │ ├── DFASerializer.swift │ │ │ │ ├── DFAState.swift │ │ │ │ └── LexerDFASerializer.swift │ │ │ ├── misc/ │ │ │ │ ├── BitSet.swift │ │ │ │ ├── DoubleKeyMap.swift │ │ │ │ ├── IntSet.swift │ │ │ │ ├── InterpreterDataReader.swift │ │ │ │ ├── Interval.swift │ │ │ │ ├── IntervalSet.swift │ │ │ │ ├── MultiMap.swift │ │ │ │ ├── MurmurHash.swift │ │ │ │ ├── Utils.swift │ │ │ │ ├── exception/ │ │ │ │ │ ├── ANTLRError.swift │ │ │ │ │ └── ANTLRException.swift │ │ │ │ ├── extension/ │ │ │ │ │ ├── ArrayExtension.swift │ │ │ │ │ ├── CharacterExtension.swift │ │ │ │ │ ├── IntStreamExtension.swift │ │ │ │ │ ├── StringExtension.swift │ │ │ │ │ ├── TokenExtension.swift │ │ │ │ │ └── UUIDExtension.swift │ │ │ │ └── utils/ │ │ │ │ ├── CommonUtil.swift │ │ │ │ ├── Mutex.swift │ │ │ │ └── Stack.swift │ │ │ └── tree/ │ │ │ ├── AbstractParseTreeVisitor.swift │ │ │ ├── ErrorNode.swift │ │ │ ├── ParseTree.swift │ │ │ ├── ParseTreeListener.swift │ │ │ ├── ParseTreeProperty.swift │ │ │ ├── ParseTreeVisitor.swift │ │ │ ├── ParseTreeWalker.swift │ │ │ ├── RuleNode.swift │ │ │ ├── SyntaxTree.swift │ │ │ ├── TerminalNode.swift │ │ │ ├── TerminalNodeImpl.swift │ │ │ ├── Tree.swift │ │ │ ├── Trees.swift │ │ │ └── pattern/ │ │ │ ├── Chunk.swift │ │ │ ├── ParseTreeMatch.swift │ │ │ ├── ParseTreePattern.swift │ │ │ ├── ParseTreePatternMatcher.swift │ │ │ ├── RuleTagToken.swift │ │ │ ├── TagChunk.swift │ │ │ ├── TextChunk.swift │ │ │ └── TokenTagToken.swift │ │ ├── Info-IOS.plist │ │ └── Info-OSX.plist │ ├── Tests/ │ │ ├── Antlr4Tests/ │ │ │ ├── ANTLRInputStreamTests.swift │ │ │ ├── InterpreterDataTests.swift │ │ │ ├── LexerA.g4 │ │ │ ├── LexerB.g4 │ │ │ ├── MurmurHashTests.swift │ │ │ ├── ParseTreePatternMatcherTests.swift │ │ │ ├── RuntimeMetaDataTests.swift │ │ │ ├── StringExtensionTests.swift │ │ │ ├── Threading.g4 │ │ │ ├── ThreadingTests.swift │ │ │ ├── TokenStreamRewriterTests.swift │ │ │ ├── TokenStreamTests.swift │ │ │ ├── VisitorBasic.g4 │ │ │ ├── VisitorCalc.g4 │ │ │ └── VisitorTests.swift │ │ ├── Info.plist │ │ └── LinuxMain.swift │ └── boot.py ├── runtime-testsuite/ │ ├── pom.xml │ ├── resources/ │ │ ├── junit-platform.properties │ │ └── org/ │ │ └── antlr/ │ │ └── v4/ │ │ └── test/ │ │ └── runtime/ │ │ ├── descriptors/ │ │ │ ├── CompositeLexers/ │ │ │ │ ├── LexerDelegatorInvokesDelegateRule.txt │ │ │ │ └── LexerDelegatorRuleOverridesDelegate.txt │ │ │ ├── CompositeParsers/ │ │ │ │ ├── BringInLiteralsFromDelegate.txt │ │ │ │ ├── CombinedImportsCombined.txt │ │ │ │ ├── DelegatesSeeSameTokenType.txt │ │ │ │ ├── DelegatorAccessesDelegateMembers.txt │ │ │ │ ├── DelegatorInvokesDelegateRule.txt │ │ │ │ ├── DelegatorInvokesDelegateRuleWithArgs.txt │ │ │ │ ├── DelegatorInvokesDelegateRuleWithReturnStruct.txt │ │ │ │ ├── DelegatorInvokesFirstVersionOfDelegateRule.txt │ │ │ │ ├── DelegatorRuleOverridesDelegate.txt │ │ │ │ ├── DelegatorRuleOverridesDelegates.txt │ │ │ │ ├── DelegatorRuleOverridesLookaheadInDelegate.txt │ │ │ │ ├── ImportLexerWithOnlyFragmentRules.txt │ │ │ │ ├── ImportedGrammarWithEmptyOptions.txt │ │ │ │ ├── ImportedRuleWithAction.txt │ │ │ │ └── KeywordVSIDOrder.txt │ │ │ ├── FullContextParsing/ │ │ │ │ ├── AmbigYieldsCtxSensitiveDFA.txt │ │ │ │ ├── AmbiguityNoLoop.txt │ │ │ │ ├── CtxSensitiveDFATwoDiffInput.txt │ │ │ │ ├── CtxSensitiveDFA_1.txt │ │ │ │ ├── CtxSensitiveDFA_2.txt │ │ │ │ ├── ExprAmbiguity_1.txt │ │ │ │ ├── ExprAmbiguity_2.txt │ │ │ │ ├── FullContextIF_THEN_ELSEParse_1.txt │ │ │ │ ├── FullContextIF_THEN_ELSEParse_2.txt │ │ │ │ ├── FullContextIF_THEN_ELSEParse_3.txt │ │ │ │ ├── FullContextIF_THEN_ELSEParse_4.txt │ │ │ │ ├── FullContextIF_THEN_ELSEParse_5.txt │ │ │ │ ├── FullContextIF_THEN_ELSEParse_6.txt │ │ │ │ ├── LoopsSimulateTailRecursion.txt │ │ │ │ └── SLLSeesEOFInLLGrammar.txt │ │ │ ├── LeftRecursion/ │ │ │ │ ├── AmbigLR_1.txt │ │ │ │ ├── AmbigLR_2.txt │ │ │ │ ├── AmbigLR_3.txt │ │ │ │ ├── AmbigLR_4.txt │ │ │ │ ├── AmbigLR_5.txt │ │ │ │ ├── Declarations_1.txt │ │ │ │ ├── Declarations_10.txt │ │ │ │ ├── Declarations_2.txt │ │ │ │ ├── Declarations_3.txt │ │ │ │ ├── Declarations_4.txt │ │ │ │ ├── Declarations_5.txt │ │ │ │ ├── Declarations_6.txt │ │ │ │ ├── Declarations_7.txt │ │ │ │ ├── Declarations_8.txt │ │ │ │ ├── Declarations_9.txt │ │ │ │ ├── DirectCallToLeftRecursiveRule_1.txt │ │ │ │ ├── DirectCallToLeftRecursiveRule_2.txt │ │ │ │ ├── DirectCallToLeftRecursiveRule_3.txt │ │ │ │ ├── Expressions_1.txt │ │ │ │ ├── Expressions_2.txt │ │ │ │ ├── Expressions_3.txt │ │ │ │ ├── Expressions_4.txt │ │ │ │ ├── Expressions_5.txt │ │ │ │ ├── Expressions_6.txt │ │ │ │ ├── Expressions_7.txt │ │ │ │ ├── JavaExpressions_1.txt │ │ │ │ ├── JavaExpressions_10.txt │ │ │ │ ├── JavaExpressions_11.txt │ │ │ │ ├── JavaExpressions_12.txt │ │ │ │ ├── JavaExpressions_2.txt │ │ │ │ ├── JavaExpressions_3.txt │ │ │ │ ├── JavaExpressions_4.txt │ │ │ │ ├── JavaExpressions_5.txt │ │ │ │ ├── JavaExpressions_6.txt │ │ │ │ ├── JavaExpressions_7.txt │ │ │ │ ├── JavaExpressions_8.txt │ │ │ │ ├── JavaExpressions_9.txt │ │ │ │ ├── LabelsOnOpSubrule_1.txt │ │ │ │ ├── LabelsOnOpSubrule_2.txt │ │ │ │ ├── LabelsOnOpSubrule_3.txt │ │ │ │ ├── MultipleActionsPredicatesOptions_1.txt │ │ │ │ ├── MultipleActionsPredicatesOptions_2.txt │ │ │ │ ├── MultipleActionsPredicatesOptions_3.txt │ │ │ │ ├── MultipleActions_1.txt │ │ │ │ ├── MultipleActions_2.txt │ │ │ │ ├── MultipleActions_3.txt │ │ │ │ ├── MultipleAlternativesWithCommonLabel_1.txt │ │ │ │ ├── MultipleAlternativesWithCommonLabel_2.txt │ │ │ │ ├── MultipleAlternativesWithCommonLabel_3.txt │ │ │ │ ├── MultipleAlternativesWithCommonLabel_4.txt │ │ │ │ ├── MultipleAlternativesWithCommonLabel_5.txt │ │ │ │ ├── PrecedenceFilterConsidersContext.txt │ │ │ │ ├── PrefixAndOtherAlt_1.txt │ │ │ │ ├── PrefixAndOtherAlt_2.txt │ │ │ │ ├── PrefixOpWithActionAndLabel_1.txt │ │ │ │ ├── PrefixOpWithActionAndLabel_2.txt │ │ │ │ ├── PrefixOpWithActionAndLabel_3.txt │ │ │ │ ├── ReturnValueAndActionsAndLabels_1.txt │ │ │ │ ├── ReturnValueAndActionsAndLabels_2.txt │ │ │ │ ├── ReturnValueAndActionsAndLabels_3.txt │ │ │ │ ├── ReturnValueAndActionsAndLabels_4.txt │ │ │ │ ├── ReturnValueAndActionsList1_1.txt │ │ │ │ ├── ReturnValueAndActionsList1_2.txt │ │ │ │ ├── ReturnValueAndActionsList1_3.txt │ │ │ │ ├── ReturnValueAndActionsList1_4.txt │ │ │ │ ├── ReturnValueAndActionsList2_1.txt │ │ │ │ ├── ReturnValueAndActionsList2_2.txt │ │ │ │ ├── ReturnValueAndActionsList2_3.txt │ │ │ │ ├── ReturnValueAndActionsList2_4.txt │ │ │ │ ├── ReturnValueAndActions_1.txt │ │ │ │ ├── ReturnValueAndActions_2.txt │ │ │ │ ├── ReturnValueAndActions_3.txt │ │ │ │ ├── ReturnValueAndActions_4.txt │ │ │ │ ├── SemPred.txt │ │ │ │ ├── SemPredFailOption.txt │ │ │ │ ├── Simple_1.txt │ │ │ │ ├── Simple_2.txt │ │ │ │ ├── Simple_3.txt │ │ │ │ ├── TernaryExprExplicitAssociativity_1.txt │ │ │ │ ├── TernaryExprExplicitAssociativity_2.txt │ │ │ │ ├── TernaryExprExplicitAssociativity_3.txt │ │ │ │ ├── TernaryExprExplicitAssociativity_4.txt │ │ │ │ ├── TernaryExprExplicitAssociativity_5.txt │ │ │ │ ├── TernaryExprExplicitAssociativity_6.txt │ │ │ │ ├── TernaryExprExplicitAssociativity_7.txt │ │ │ │ ├── TernaryExprExplicitAssociativity_8.txt │ │ │ │ ├── TernaryExprExplicitAssociativity_9.txt │ │ │ │ ├── TernaryExpr_1.txt │ │ │ │ ├── TernaryExpr_2.txt │ │ │ │ ├── TernaryExpr_3.txt │ │ │ │ ├── TernaryExpr_4.txt │ │ │ │ ├── TernaryExpr_5.txt │ │ │ │ ├── TernaryExpr_6.txt │ │ │ │ ├── TernaryExpr_7.txt │ │ │ │ ├── TernaryExpr_8.txt │ │ │ │ ├── TernaryExpr_9.txt │ │ │ │ ├── WhitespaceInfluence_1.txt │ │ │ │ └── WhitespaceInfluence_2.txt │ │ │ ├── LexerErrors/ │ │ │ │ ├── DFAToATNThatFailsBackToDFA.txt │ │ │ │ ├── DFAToATNThatMatchesThenFailsInATN.txt │ │ │ │ ├── EnforcedGreedyNestedBraces_1.txt │ │ │ │ ├── EnforcedGreedyNestedBraces_2.txt │ │ │ │ ├── ErrorInMiddle.txt │ │ │ │ ├── InvalidCharAtStart.txt │ │ │ │ ├── InvalidCharAtStartAfterDFACache.txt │ │ │ │ ├── InvalidCharInToken.txt │ │ │ │ ├── InvalidCharInTokenAfterDFACache.txt │ │ │ │ ├── LexerExecDFA.txt │ │ │ │ ├── StringsEmbeddedInActions_1.txt │ │ │ │ └── StringsEmbeddedInActions_2.txt │ │ │ ├── LexerExec/ │ │ │ │ ├── ActionPlacement.txt │ │ │ │ ├── CharSet.txt │ │ │ │ ├── CharSetInSet.txt │ │ │ │ ├── CharSetNot.txt │ │ │ │ ├── CharSetPlus.txt │ │ │ │ ├── CharSetRange.txt │ │ │ │ ├── CharSetWithEscapedChar.txt │ │ │ │ ├── CharSetWithMissingEscapeChar.txt │ │ │ │ ├── CharSetWithQuote1.txt │ │ │ │ ├── CharSetWithQuote2.txt │ │ │ │ ├── EOFByItself.txt │ │ │ │ ├── EOFSuffixInFirstRule_1.txt │ │ │ │ ├── EOFSuffixInFirstRule_2.txt │ │ │ │ ├── EscapeTargetStringLiteral.txt │ │ │ │ ├── EscapedCharacters.txt │ │ │ │ ├── GreedyClosure.txt │ │ │ │ ├── GreedyConfigs.txt │ │ │ │ ├── GreedyOptional.txt │ │ │ │ ├── GreedyPositiveClosure.txt │ │ │ │ ├── HexVsID.txt │ │ │ │ ├── KeywordID.txt │ │ │ │ ├── NonGreedyClosure.txt │ │ │ │ ├── NonGreedyConfigs.txt │ │ │ │ ├── NonGreedyOptional.txt │ │ │ │ ├── NonGreedyPositiveClosure.txt │ │ │ │ ├── NonGreedyTermination1.txt │ │ │ │ ├── NonGreedyTermination2.txt │ │ │ │ ├── Parentheses.txt │ │ │ │ ├── PositionAdjustingLexer.txt │ │ │ │ ├── QuoteTranslation.txt │ │ │ │ ├── RecursiveLexerRuleRefWithWildcardPlus_1.txt │ │ │ │ ├── RecursiveLexerRuleRefWithWildcardPlus_2.txt │ │ │ │ ├── RecursiveLexerRuleRefWithWildcardStar_1.txt │ │ │ │ ├── RecursiveLexerRuleRefWithWildcardStar_2.txt │ │ │ │ ├── RefToRuleDoesNotSetTokenNorEmitAnother.txt │ │ │ │ ├── ReservedWordsEscaping.txt │ │ │ │ ├── ReservedWordsEscaping_NULL.txt │ │ │ │ ├── Slashes.txt │ │ │ │ ├── StackoverflowDueToNotEscapedHyphen.txt │ │ │ │ ├── TokenType0xFFFF.txt │ │ │ │ ├── UnicodeCharSet.txt │ │ │ │ └── ZeroLengthToken.txt │ │ │ ├── Listeners/ │ │ │ │ ├── Basic.txt │ │ │ │ ├── LR.txt │ │ │ │ ├── LRWithLabels.txt │ │ │ │ ├── RuleGetters_1.txt │ │ │ │ ├── RuleGetters_2.txt │ │ │ │ ├── TokenGetters_1.txt │ │ │ │ └── TokenGetters_2.txt │ │ │ ├── ParseTrees/ │ │ │ │ ├── AltNum.txt │ │ │ │ ├── ExtraToken.txt │ │ │ │ ├── ExtraTokensAndAltLabels.txt │ │ │ │ ├── NoViableAlt.txt │ │ │ │ ├── RuleRef.txt │ │ │ │ ├── Sync.txt │ │ │ │ ├── Token2.txt │ │ │ │ ├── TokenAndRuleContextString.txt │ │ │ │ ├── TwoAltLoop.txt │ │ │ │ └── TwoAlts.txt │ │ │ ├── ParserErrors/ │ │ │ │ ├── ConjuringUpToken.txt │ │ │ │ ├── ConjuringUpTokenFromSet.txt │ │ │ │ ├── ContextListGetters.txt │ │ │ │ ├── DuplicatedLeftRecursiveCall_1.txt │ │ │ │ ├── DuplicatedLeftRecursiveCall_2.txt │ │ │ │ ├── DuplicatedLeftRecursiveCall_3.txt │ │ │ │ ├── DuplicatedLeftRecursiveCall_4.txt │ │ │ │ ├── ExtraneousInput.txt │ │ │ │ ├── InvalidATNStateRemoval.txt │ │ │ │ ├── InvalidEmptyInput.txt │ │ │ │ ├── LL1ErrorInfo.txt │ │ │ │ ├── LL2.txt │ │ │ │ ├── LL3.txt │ │ │ │ ├── LLStar.txt │ │ │ │ ├── MultiTokenDeletionBeforeLoop.txt │ │ │ │ ├── MultiTokenDeletionBeforeLoop2.txt │ │ │ │ ├── MultiTokenDeletionDuringLoop.txt │ │ │ │ ├── MultiTokenDeletionDuringLoop2.txt │ │ │ │ ├── NoViableAltAvoidance.txt │ │ │ │ ├── SingleSetInsertion.txt │ │ │ │ ├── SingleSetInsertionConsumption.txt │ │ │ │ ├── SingleTokenDeletion.txt │ │ │ │ ├── SingleTokenDeletionBeforeAlt.txt │ │ │ │ ├── SingleTokenDeletionBeforeLoop.txt │ │ │ │ ├── SingleTokenDeletionBeforeLoop2.txt │ │ │ │ ├── SingleTokenDeletionBeforePredict.txt │ │ │ │ ├── SingleTokenDeletionConsumption.txt │ │ │ │ ├── SingleTokenDeletionDuringLoop.txt │ │ │ │ ├── SingleTokenDeletionDuringLoop2.txt │ │ │ │ ├── SingleTokenDeletionExpectingSet.txt │ │ │ │ ├── SingleTokenInsertion.txt │ │ │ │ ├── TokenMismatch.txt │ │ │ │ ├── TokenMismatch2.txt │ │ │ │ └── TokenMismatch3.txt │ │ │ ├── ParserExec/ │ │ │ │ ├── APlus.txt │ │ │ │ ├── AStar_1.txt │ │ │ │ ├── AStar_2.txt │ │ │ │ ├── AorAPlus.txt │ │ │ │ ├── AorAStar_1.txt │ │ │ │ ├── AorAStar_2.txt │ │ │ │ ├── AorB.txt │ │ │ │ ├── AorBPlus.txt │ │ │ │ ├── AorBStar_1.txt │ │ │ │ ├── AorBStar_2.txt │ │ │ │ ├── Basic.txt │ │ │ │ ├── BuildParseTree_FALSE.txt │ │ │ │ ├── BuildParseTree_TRUE.txt │ │ │ │ ├── IfIfElseGreedyBinding1.txt │ │ │ │ ├── IfIfElseGreedyBinding2.txt │ │ │ │ ├── IfIfElseNonGreedyBinding1.txt │ │ │ │ ├── IfIfElseNonGreedyBinding2.txt │ │ │ │ ├── Keyword_1.txt │ │ │ │ ├── Keyword_2.txt │ │ │ │ ├── Keyword_3.txt │ │ │ │ ├── Keyword_4.txt │ │ │ │ ├── Keyword_5.txt │ │ │ │ ├── Keyword_6.txt │ │ │ │ ├── LL1OptionalBlock_1.txt │ │ │ │ ├── LL1OptionalBlock_2.txt │ │ │ │ ├── LabelAliasingAcrossLabeledAlternatives.txt │ │ │ │ ├── Labels.txt │ │ │ │ ├── ListLabelForClosureContext.txt │ │ │ │ ├── ListLabelsOnRuleRefStartOfAlt.txt │ │ │ │ ├── ListLabelsOnSet.txt │ │ │ │ ├── MultipleEOFHandling.txt │ │ │ │ ├── OpenDeviceStatement_Case1.txt │ │ │ │ ├── OpenDeviceStatement_Case2.txt │ │ │ │ ├── OpenDeviceStatement_Case3.txt │ │ │ │ ├── Optional_1.txt │ │ │ │ ├── Optional_2.txt │ │ │ │ ├── Optional_3.txt │ │ │ │ ├── Optional_4.txt │ │ │ │ ├── OrderingPredicates.txt │ │ │ │ ├── ParserProperty.txt │ │ │ │ ├── PredicatedIfIfElse.txt │ │ │ │ ├── PredictionIssue334.txt │ │ │ │ ├── PredictionMode_LL.txt │ │ │ │ ├── PredictionMode_SLL.txt │ │ │ │ ├── ReferenceToATN_1.txt │ │ │ │ ├── ReferenceToATN_2.txt │ │ │ │ ├── ReservedWordsEscaping.txt │ │ │ │ ├── TokenOffset.txt │ │ │ │ ├── Wildcard.txt │ │ │ │ └── uStartingCharDoesNotCauseIllegalUnicodeEscape.txt │ │ │ ├── Performance/ │ │ │ │ ├── DropLoopEntryBranchInLRRule_1.txt │ │ │ │ ├── DropLoopEntryBranchInLRRule_2.txt │ │ │ │ ├── DropLoopEntryBranchInLRRule_3.txt │ │ │ │ ├── DropLoopEntryBranchInLRRule_4.txt │ │ │ │ ├── DropLoopEntryBranchInLRRule_5.txt │ │ │ │ ├── ExpressionGrammar_1.txt │ │ │ │ └── ExpressionGrammar_2.txt │ │ │ ├── SemPredEvalLexer/ │ │ │ │ ├── DisableRule.txt │ │ │ │ ├── EnumNotID.txt │ │ │ │ ├── IDnotEnum.txt │ │ │ │ ├── IDvsEnum.txt │ │ │ │ ├── Indent.txt │ │ │ │ ├── LexerInputPositionSensitivePredicates.txt │ │ │ │ ├── PredicatedKeywords.txt │ │ │ │ └── RuleSempredFunction.txt │ │ │ ├── SemPredEvalParser/ │ │ │ │ ├── ActionHidesPreds.txt │ │ │ │ ├── ActionsHidePredsInGlobalFOLLOW.txt │ │ │ │ ├── AtomWithClosureInTranslatedLRRule.txt │ │ │ │ ├── DepedentPredsInGlobalFOLLOW.txt │ │ │ │ ├── DependentPredNotInOuterCtxShouldBeIgnored.txt │ │ │ │ ├── DisabledAlternative.txt │ │ │ │ ├── IndependentPredNotPassedOuterCtxToAvoidCastException.txt │ │ │ │ ├── NoTruePredsThrowsNoViableAlt.txt │ │ │ │ ├── Order.txt │ │ │ │ ├── PredFromAltTestedInLoopBack_1.txt │ │ │ │ ├── PredFromAltTestedInLoopBack_2.txt │ │ │ │ ├── PredTestedEvenWhenUnAmbig_1.txt │ │ │ │ ├── PredTestedEvenWhenUnAmbig_2.txt │ │ │ │ ├── PredicateDependentOnArg.txt │ │ │ │ ├── PredicateDependentOnArg2.txt │ │ │ │ ├── PredsInGlobalFOLLOW.txt │ │ │ │ ├── RewindBeforePredEval.txt │ │ │ │ ├── Simple.txt │ │ │ │ ├── SimpleValidate.txt │ │ │ │ ├── SimpleValidate2.txt │ │ │ │ ├── ToLeft.txt │ │ │ │ ├── ToLeftWithVaryingPredicate.txt │ │ │ │ ├── TwoUnpredicatedAlts.txt │ │ │ │ ├── TwoUnpredicatedAltsAndOneOrthogonalAlt.txt │ │ │ │ ├── UnpredicatedPathsInAlt.txt │ │ │ │ └── ValidateInDFA.txt │ │ │ └── Sets/ │ │ │ ├── CharSetLiteral.txt │ │ │ ├── ComplementSet.txt │ │ │ ├── LexerOptionalSet.txt │ │ │ ├── LexerPlusSet.txt │ │ │ ├── LexerStarSet.txt │ │ │ ├── NotChar.txt │ │ │ ├── NotCharSet.txt │ │ │ ├── NotCharSetWithRuleRef3.txt │ │ │ ├── OptionalLexerSingleElement.txt │ │ │ ├── OptionalSet.txt │ │ │ ├── OptionalSingleElement.txt │ │ │ ├── ParserNotSet.txt │ │ │ ├── ParserNotToken.txt │ │ │ ├── ParserNotTokenWithLabel.txt │ │ │ ├── ParserSet.txt │ │ │ ├── PlusLexerSingleElement.txt │ │ │ ├── PlusSet.txt │ │ │ ├── RuleAsSet.txt │ │ │ ├── SeqDoesNotBecomeSet.txt │ │ │ ├── StarLexerSingleElement_1.txt │ │ │ ├── StarLexerSingleElement_2.txt │ │ │ ├── StarSet.txt │ │ │ ├── UnicodeEscapedBMPRangeSet.txt │ │ │ ├── UnicodeEscapedBMPSet.txt │ │ │ ├── UnicodeEscapedSMPRangeSet.txt │ │ │ ├── UnicodeEscapedSMPRangeSetMismatch.txt │ │ │ ├── UnicodeEscapedSMPSet.txt │ │ │ ├── UnicodeNegatedBMPSetIncludesSMPCodePoints.txt │ │ │ ├── UnicodeNegatedSMPSetIncludesBMPCodePoints.txt │ │ │ ├── UnicodeUnescapedBMPRangeSet.txt │ │ │ └── UnicodeUnescapedBMPSet.txt │ │ ├── helpers/ │ │ │ ├── Antlr4.Test.csproj.stg │ │ │ ├── Package.swift.stg │ │ │ ├── Test.cpp.stg │ │ │ ├── Test.cs.stg │ │ │ ├── Test.dart.stg │ │ │ ├── Test.go.stg │ │ │ ├── Test.js.stg │ │ │ ├── Test.php.stg │ │ │ ├── Test.py.stg │ │ │ ├── Test.ts.stg │ │ │ ├── Test.vcxproj.stg │ │ │ ├── main.swift.stg │ │ │ ├── package_js.json │ │ │ ├── package_ts.json │ │ │ ├── pubspec.yaml.stg │ │ │ └── tsconfig.json │ │ └── templates/ │ │ ├── CSharp.test.stg │ │ ├── Cpp.test.stg │ │ ├── Dart.test.stg │ │ ├── Go.test.stg │ │ ├── Java.test.stg │ │ ├── JavaScript.test.stg │ │ ├── PHP.test.stg │ │ ├── Python3.test.stg │ │ ├── Swift.test.stg │ │ └── TypeScript.test.stg │ └── test/ │ └── org/ │ └── antlr/ │ └── v4/ │ ├── runtime/ │ │ └── TestCodePointCharStream.java │ └── test/ │ └── runtime/ │ ├── CustomDescriptors.java │ ├── ErrorQueue.java │ ├── FileUtils.java │ ├── GeneratedFile.java │ ├── Generator.java │ ├── GrammarType.java │ ├── OSType.java │ ├── Processor.java │ ├── ProcessorResult.java │ ├── README.md │ ├── RunOptions.java │ ├── RuntimeRunner.java │ ├── RuntimeTestDescriptor.java │ ├── RuntimeTestDescriptorParser.java │ ├── RuntimeTestUtils.java │ ├── RuntimeTests.java │ ├── Stage.java │ ├── StreamReader.java │ ├── TraceATN.java │ ├── cpp/ │ │ ├── CppRunner.java │ │ └── CppRuntimeTests.java │ ├── csharp/ │ │ ├── CSharpRunner.java │ │ └── CSharpRuntimeTests.java │ ├── dart/ │ │ ├── DartRunner.java │ │ └── DartRuntimeTests.java │ ├── go/ │ │ ├── GoRunner.java │ │ └── GoRuntimeTests.java │ ├── java/ │ │ ├── JavaRunner.java │ │ ├── JavaRuntimeTests.java │ │ ├── TestCharStreams.java │ │ ├── TestIntegerList.java │ │ ├── TestInterpreterDataReader.java │ │ ├── api/ │ │ │ ├── Java.g4 │ │ │ ├── TestExpectedTokens.java │ │ │ ├── TestTokenStream.java │ │ │ ├── TestTokenStreamRewriter.java │ │ │ ├── TestVisitors.java │ │ │ ├── VisitorBasic.g4 │ │ │ ├── VisitorCalc.g4 │ │ │ └── perf/ │ │ │ ├── Instrumentor.java │ │ │ ├── TimeLexerSpeed.java │ │ │ ├── emoji.txt │ │ │ ├── graphemes.g4 │ │ │ ├── udhr_hin.txt │ │ │ └── udhr_kor.txt │ │ └── helpers/ │ │ ├── CustomStreamErrorListener.java │ │ ├── RuntimeTestLexer.java │ │ ├── RuntimeTestParser.java │ │ └── TreeShapeListener.java │ ├── javascript/ │ │ ├── JavaScriptRuntimeTests.java │ │ └── NodeRunner.java │ ├── php/ │ │ ├── PHPRunner.java │ │ └── PhpRuntimeTests.java │ ├── python/ │ │ └── PythonRunner.java │ ├── python3/ │ │ ├── Python3Runner.java │ │ └── Python3RuntimeTests.java │ ├── states/ │ │ ├── CompiledState.java │ │ ├── ExecutedState.java │ │ ├── GeneratedState.java │ │ ├── JavaCompiledState.java │ │ ├── JavaExecutedState.java │ │ └── State.java │ ├── swift/ │ │ ├── SwiftRunner.java │ │ └── SwiftRuntimeTests.java │ └── typescript/ │ ├── TsNodeRunner.java │ └── TypeScriptRuntimeTests.java ├── scripts/ │ ├── deploy_to_website.py │ ├── files-to-update.txt │ ├── github_release_notes.py │ ├── parse-extended-pictographic/ │ │ ├── ExtendedPictographic-Parsed.txt │ │ ├── ExtendedPictographic.txt │ │ ├── README.md │ │ └── parse.py │ ├── traceatn.sh │ └── update_antlr_version.py ├── tool/ │ ├── nb-configuration.xml │ ├── pom.xml │ ├── resources/ │ │ └── org/ │ │ └── antlr/ │ │ └── v4/ │ │ └── tool/ │ │ └── templates/ │ │ ├── LeftRecursiveRules.stg │ │ ├── codegen/ │ │ │ ├── CSharp/ │ │ │ │ └── CSharp.stg │ │ │ ├── Cpp/ │ │ │ │ ├── Cpp.stg │ │ │ │ └── Files.stg │ │ │ ├── Dart/ │ │ │ │ └── Dart.stg │ │ │ ├── Go/ │ │ │ │ └── Go.stg │ │ │ ├── Java/ │ │ │ │ └── Java.stg │ │ │ ├── JavaScript/ │ │ │ │ └── JavaScript.stg │ │ │ ├── PHP/ │ │ │ │ └── PHP.stg │ │ │ ├── Python3/ │ │ │ │ └── Python3.stg │ │ │ ├── Swift/ │ │ │ │ └── Swift.stg │ │ │ └── TypeScript/ │ │ │ └── TypeScript.stg │ │ ├── depend.stg │ │ ├── dot/ │ │ │ └── graphs.stg │ │ ├── messages/ │ │ │ └── formats/ │ │ │ ├── antlr.stg │ │ │ ├── gnu.stg │ │ │ └── vs2005.stg │ │ └── unicodedata.st │ └── src/ │ └── org/ │ └── antlr/ │ └── v4/ │ ├── Tool.java │ ├── analysis/ │ │ ├── AnalysisPipeline.java │ │ ├── LeftRecursionDetector.java │ │ ├── LeftRecursiveRuleAltInfo.java │ │ ├── LeftRecursiveRuleAnalyzer.java │ │ └── LeftRecursiveRuleTransformer.java │ ├── automata/ │ │ ├── ATNFactory.java │ │ ├── ATNOptimizer.java │ │ ├── ATNPrinter.java │ │ ├── ATNVisitor.java │ │ ├── CharactersDataCheckStatus.java │ │ ├── LexerATNFactory.java │ │ ├── ParserATNFactory.java │ │ ├── RangeBorderCharactersData.java │ │ └── TailEpsilonRemover.java │ ├── codegen/ │ │ ├── ActionTranslator.java │ │ ├── BlankOutputModelFactory.java │ │ ├── CodeGenPipeline.java │ │ ├── CodeGenerator.java │ │ ├── CodeGeneratorExtension.java │ │ ├── DefaultOutputModelFactory.java │ │ ├── LexerFactory.java │ │ ├── OutputModelController.java │ │ ├── OutputModelFactory.java │ │ ├── OutputModelWalker.java │ │ ├── ParserFactory.java │ │ ├── SourceGenTriggers.g │ │ ├── SourceType.java │ │ ├── Target.java │ │ ├── UnicodeEscapes.java │ │ ├── model/ │ │ │ ├── Action.java │ │ │ ├── AddToLabelList.java │ │ │ ├── AltBlock.java │ │ │ ├── ArgAction.java │ │ │ ├── BaseListenerFile.java │ │ │ ├── BaseVisitorFile.java │ │ │ ├── CaptureNextToken.java │ │ │ ├── CaptureNextTokenType.java │ │ │ ├── Choice.java │ │ │ ├── CodeBlockForAlt.java │ │ │ ├── CodeBlockForOuterMostAlt.java │ │ │ ├── DispatchMethod.java │ │ │ ├── ElementFrequenciesVisitor.java │ │ │ ├── ExceptionClause.java │ │ │ ├── InvokeRule.java │ │ │ ├── LL1AltBlock.java │ │ │ ├── LL1Choice.java │ │ │ ├── LL1Loop.java │ │ │ ├── LL1OptionalBlock.java │ │ │ ├── LL1OptionalBlockSingleAlt.java │ │ │ ├── LL1PlusBlockSingleAlt.java │ │ │ ├── LL1StarBlockSingleAlt.java │ │ │ ├── LabeledOp.java │ │ │ ├── LeftRecursiveRuleFunction.java │ │ │ ├── Lexer.java │ │ │ ├── LexerFile.java │ │ │ ├── ListenerDispatchMethod.java │ │ │ ├── ListenerFile.java │ │ │ ├── Loop.java │ │ │ ├── MatchNotSet.java │ │ │ ├── MatchSet.java │ │ │ ├── MatchToken.java │ │ │ ├── ModelElement.java │ │ │ ├── OptionalBlock.java │ │ │ ├── OutputFile.java │ │ │ ├── OutputModelObject.java │ │ │ ├── Parser.java │ │ │ ├── ParserFile.java │ │ │ ├── PlusBlock.java │ │ │ ├── Recognizer.java │ │ │ ├── RuleActionFunction.java │ │ │ ├── RuleElement.java │ │ │ ├── RuleFunction.java │ │ │ ├── RuleSempredFunction.java │ │ │ ├── SemPred.java │ │ │ ├── SerializedATN.java │ │ │ ├── SerializedJavaATN.java │ │ │ ├── SrcOp.java │ │ │ ├── StarBlock.java │ │ │ ├── Sync.java │ │ │ ├── TestSetInline.java │ │ │ ├── ThrowEarlyExitException.java │ │ │ ├── ThrowNoViableAlt.java │ │ │ ├── ThrowRecognitionException.java │ │ │ ├── TokenInfo.java │ │ │ ├── VisitorDispatchMethod.java │ │ │ ├── VisitorFile.java │ │ │ ├── Wildcard.java │ │ │ ├── chunk/ │ │ │ │ ├── ActionChunk.java │ │ │ │ ├── ActionTemplate.java │ │ │ │ ├── ActionText.java │ │ │ │ ├── ArgRef.java │ │ │ │ ├── LabelRef.java │ │ │ │ ├── ListLabelRef.java │ │ │ │ ├── LocalRef.java │ │ │ │ ├── NonLocalAttrRef.java │ │ │ │ ├── QRetValueRef.java │ │ │ │ ├── RetValueRef.java │ │ │ │ ├── RulePropertyRef.java │ │ │ │ ├── RulePropertyRef_ctx.java │ │ │ │ ├── RulePropertyRef_parser.java │ │ │ │ ├── RulePropertyRef_start.java │ │ │ │ ├── RulePropertyRef_stop.java │ │ │ │ ├── RulePropertyRef_text.java │ │ │ │ ├── SetAttr.java │ │ │ │ ├── SetNonLocalAttr.java │ │ │ │ ├── SymbolRefChunk.java │ │ │ │ ├── ThisRulePropertyRef_ctx.java │ │ │ │ ├── ThisRulePropertyRef_parser.java │ │ │ │ ├── ThisRulePropertyRef_start.java │ │ │ │ ├── ThisRulePropertyRef_stop.java │ │ │ │ ├── ThisRulePropertyRef_text.java │ │ │ │ ├── TokenPropertyRef.java │ │ │ │ ├── TokenPropertyRef_channel.java │ │ │ │ ├── TokenPropertyRef_index.java │ │ │ │ ├── TokenPropertyRef_int.java │ │ │ │ ├── TokenPropertyRef_line.java │ │ │ │ ├── TokenPropertyRef_pos.java │ │ │ │ ├── TokenPropertyRef_text.java │ │ │ │ ├── TokenPropertyRef_type.java │ │ │ │ └── TokenRef.java │ │ │ ├── dbg.java │ │ │ └── decl/ │ │ │ ├── AltLabelStructDecl.java │ │ │ ├── AttributeDecl.java │ │ │ ├── CodeBlock.java │ │ │ ├── ContextGetterDecl.java │ │ │ ├── ContextRuleGetterDecl.java │ │ │ ├── ContextRuleListGetterDecl.java │ │ │ ├── ContextRuleListIndexedGetterDecl.java │ │ │ ├── ContextTokenGetterDecl.java │ │ │ ├── ContextTokenListGetterDecl.java │ │ │ ├── ContextTokenListIndexedGetterDecl.java │ │ │ ├── Decl.java │ │ │ ├── ElementListDecl.java │ │ │ ├── RuleContextDecl.java │ │ │ ├── RuleContextListDecl.java │ │ │ ├── StructDecl.java │ │ │ ├── TokenDecl.java │ │ │ ├── TokenListDecl.java │ │ │ └── TokenTypeDecl.java │ │ └── target/ │ │ ├── CSharpTarget.java │ │ ├── CppTarget.java │ │ ├── DartTarget.java │ │ ├── GoTarget.java │ │ ├── JavaScriptTarget.java │ │ ├── JavaTarget.java │ │ ├── PHPTarget.java │ │ ├── Python3Target.java │ │ ├── SwiftTarget.java │ │ └── TypeScriptTarget.java │ ├── gui/ │ │ ├── BasicFontMetrics.java │ │ ├── GraphicsSupport.java │ │ ├── Interpreter.java │ │ ├── JFileChooserConfirmOverwrite.java │ │ ├── PostScriptDocument.java │ │ ├── SystemFontMetrics.java │ │ ├── TestRig.java │ │ ├── TreeLayoutAdaptor.java │ │ ├── TreePostScriptGenerator.java │ │ ├── TreeTextProvider.java │ │ ├── TreeViewer.java │ │ └── Trees.java │ ├── misc/ │ │ ├── CharSupport.java │ │ ├── EscapeSequenceParsing.java │ │ ├── FrequencySet.java │ │ ├── Graph.java │ │ ├── MutableInt.java │ │ ├── OrderedHashMap.java │ │ └── Utils.java │ ├── parse/ │ │ ├── ANTLRLexer.g │ │ ├── ANTLRParser.g │ │ ├── ATNBuilder.g │ │ ├── ActionSplitter.g │ │ ├── ActionSplitterListener.java │ │ ├── BlockSetTransformer.g │ │ ├── GrammarASTAdaptor.java │ │ ├── GrammarToken.java │ │ ├── GrammarTreeVisitor.g │ │ ├── LeftRecursiveRuleWalker.g │ │ ├── ResyncToEndOfRuleBlock.java │ │ ├── ScopeParser.java │ │ ├── TokenVocabParser.java │ │ ├── ToolANTLRLexer.java │ │ ├── ToolANTLRParser.java │ │ └── v4ParserException.java │ ├── semantics/ │ │ ├── ActionSniffer.java │ │ ├── AttributeChecks.java │ │ ├── BasicSemanticChecks.java │ │ ├── BlankActionSplitterListener.java │ │ ├── RuleCollector.java │ │ ├── SemanticPipeline.java │ │ ├── SymbolChecks.java │ │ ├── SymbolCollector.java │ │ └── UseDefAnalyzer.java │ ├── tool/ │ │ ├── ANTLRMessage.java │ │ ├── ANTLRToolListener.java │ │ ├── Alternative.java │ │ ├── Attribute.java │ │ ├── AttributeDict.java │ │ ├── AttributeResolver.java │ │ ├── BuildDependencyGenerator.java │ │ ├── DOTGenerator.java │ │ ├── DefaultToolListener.java │ │ ├── ErrorManager.java │ │ ├── ErrorSeverity.java │ │ ├── ErrorType.java │ │ ├── Grammar.java │ │ ├── GrammarInterpreterRuleContext.java │ │ ├── GrammarParserInterpreter.java │ │ ├── GrammarSemanticsMessage.java │ │ ├── GrammarSyntaxMessage.java │ │ ├── GrammarTransformPipeline.java │ │ ├── LabelElementPair.java │ │ ├── LabelType.java │ │ ├── LeftRecursionCyclesMessage.java │ │ ├── LeftRecursiveRule.java │ │ ├── LexerGrammar.java │ │ ├── Rule.java │ │ ├── ToolMessage.java │ │ └── ast/ │ │ ├── ActionAST.java │ │ ├── AltAST.java │ │ ├── BlockAST.java │ │ ├── GrammarAST.java │ │ ├── GrammarASTErrorNode.java │ │ ├── GrammarASTVisitor.java │ │ ├── GrammarASTWithOptions.java │ │ ├── GrammarRootAST.java │ │ ├── NotAST.java │ │ ├── OptionalBlockAST.java │ │ ├── PlusBlockAST.java │ │ ├── PredAST.java │ │ ├── QuantifierAST.java │ │ ├── RangeAST.java │ │ ├── RuleAST.java │ │ ├── RuleElementAST.java │ │ ├── RuleRefAST.java │ │ ├── SetAST.java │ │ ├── StarBlockAST.java │ │ └── TerminalAST.java │ └── unicode/ │ └── UnicodeDataTemplateController.java └── tool-testsuite/ ├── pom.xml ├── resources/ │ └── junit-platform.properties └── test/ └── org/ └── antlr/ └── v4/ └── test/ └── tool/ ├── ATNDescriber.java ├── InterpreterTreeTextProvider.java ├── Java.g4 ├── JavaLR.g4 ├── JavaUnicodeInputStream.java ├── MockIntTokenStream.java ├── ParserInterpreterForTesting.java ├── PositionAdjustingLexer.g4 ├── Psl.g4 ├── TestASTStructure.java ├── TestATNConstruction.java ├── TestATNDeserialization.java ├── TestATNInterpreter.java ├── TestATNLexerInterpreter.java ├── TestATNParserPrediction.java ├── TestATNSerialization.java ├── TestActionSplitter.java ├── TestActionTranslation.java ├── TestAmbigParseTrees.java ├── TestAttributeChecks.java ├── TestBasicSemanticErrors.java ├── TestBufferedTokenStream.java ├── TestCharSupport.java ├── TestCodeGeneration.java ├── TestCommonTokenStream.java ├── TestCompositeGrammars.java ├── TestDollarParser.java ├── TestErrorSets.java ├── TestEscapeSequenceParsing.java ├── TestFastQueue.java ├── TestGrammarParserInterpreter.java ├── TestGraphNodes.java ├── TestIntervalSet.java ├── TestLeftRecursionToolIssues.java ├── TestLexerActions.java ├── TestLookaheadTrees.java ├── TestParseTreeMatcher.java ├── TestParserExec.java ├── TestParserInterpreter.java ├── TestParserProfiler.java ├── TestPerformance.java ├── TestScopeParsing.java ├── TestSplitParser.java ├── TestSymbolIssues.java ├── TestTokenPositionOptions.java ├── TestTokenTypeAssignment.java ├── TestToolSyntaxErrors.java ├── TestTopologicalSort.java ├── TestUnbufferedCharStream.java ├── TestUnbufferedTokenStream.java ├── TestUnicodeData.java ├── TestUnicodeEscapes.java ├── TestUnicodeGrammar.java ├── TestUtils.java ├── TestVocabulary.java ├── TestXPath.java └── ToolTestUtils.java ================================================ FILE CONTENTS ================================================ ================================================ FILE: .clang-format ================================================ --- # This section defines defaults for all languages. Currently we derive ANTLR style from LLVM. BasedOnStyle: LLVM # Only use clang-format for C++ for now. DisableFormat: true --- # This section configures C++ formatting. Language: Cpp DisableFormat: false Standard: c++17 # Prevent clang-format from attempting to pick the alignment and always use right alignment. DerivePointerAlignment: false # ANTLR existing style is to right align pointers and references. PointerAlignment: Right ReferenceAlignment: Right # Some of ANTLR existing code is longer than the default 80, so use 100 for now. ColumnLimit: 100 # Historically ANTLR has used indentation within namespaces, so replicate it. NamespaceIndentation: Inner ================================================ FILE: .editorconfig ================================================ root = true [*] tab_width = 4 [*.{java,stg}] charset = utf-8 insert_final_newline = true trim_trailing_whitespace = true indent_style = tab ij_java_else_on_new_line = true [*.{h,cpp}] charset = utf-8 insert_final_newline = true trim_trailing_whitespace = true indent_style = space indent_size = 2 ================================================ FILE: .gitattributes ================================================ # This rule applies to all files which don't match another line below * text=auto ================================================ FILE: .github/ISSUE_TEMPLATE.md ================================================ ================================================ FILE: .github/PULL_REQUEST_TEMPLATE.md ================================================ ================================================ FILE: .github/workflows/hosted.yml ================================================ name: antlr4 concurrency: group: ${{ github.repository }}-${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true on: push: branches: [ master, dev, hostedci ] pull_request: branches: [ master, dev ] permissions: contents: read jobs: cpp-lib-build: runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: os: [ macos-15, ubuntu-22.04, windows-2022 ] compiler: [ clang, gcc ] unity_build: [ ON, OFF ] exclude: - os: windows-2022 compiler: gcc include: - os: windows-2022 compiler: cl steps: - name: Install dependencies (Ubuntu) if: startswith(matrix.os, 'ubuntu') run: | sudo apt-get update -qq sudo apt install -y ninja-build - name: Install dependencies (MacOS) if: startswith(matrix.os, 'macos') run: brew install ninja - name: Setup Clang if: (matrix.compiler == 'clang') && !startswith(matrix.os, 'macos') uses: egor-tensin/setup-clang@v1 with: version: 13 platform: x64 cygwin: 0 - name: Check out code uses: actions/checkout@v3 - name: Use ccache if: startswith(matrix.os, 'macos') || startswith(matrix.os, 'ubuntu') uses: hendrikmuhs/ccache-action@v1.2 with: key: ${{ matrix.os }}-${{ matrix.compiler }} - name: Configure shell (Ubuntu) if: startswith(matrix.os, 'ubuntu') run: echo 'PATH=/usr/lib/ccache:'"$PATH" >> $GITHUB_ENV - name: Configure shell (MacOS) if: startswith(matrix.os, 'macos') run: echo "PATH=$(brew --prefix)/opt/ccache/libexec:$PATH" >> $GITHUB_ENV - name: Build (Windows) if: startswith(matrix.os, 'windows') shell: cmd run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvars64.bat" if "${{ matrix.compiler }}" EQU "cl" ( set CC=cl set CXX=cl echo 'CC=cl' >> $GITHUB_ENV echo 'CXX=cl' >> $GITHUB_ENV ) else ( set CC=clang set CXX=clang++ echo 'CC=clang' >> $GITHUB_ENV echo 'CXX=clang++' >> $GITHUB_ENV ) set where cmake && cmake --version where ninja && ninja --version where %CC% && %CC% -version where %CXX% && %CXX% -version cd runtime/Cpp cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug -DANTLR_BUILD_CPP_TESTS=OFF -DCMAKE_UNITY_BUILD=${{ matrix.unity_build }} -DCMAKE_UNITY_BUILD_BATCH_SIZE=20 -S . -B out/Debug if %errorlevel% neq 0 exit /b %errorlevel% cmake --build out/Debug -j %NUMBER_OF_PROCESSORS% if %errorlevel% neq 0 exit /b %errorlevel% cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DANTLR_BUILD_CPP_TESTS=OFF -S . -B out/Release if %errorlevel% neq 0 exit /b %errorlevel% cmake --build out/Release -j %NUMBER_OF_PROCESSORS% if %errorlevel% neq 0 exit /b %errorlevel% - name: Build (non-Windows) if: startswith(matrix.os, 'macos') || startswith(matrix.os, 'ubuntu') run: | if [ "${{matrix.compiler}}" == "clang" ]; then export CC=clang export CXX=clang++ echo 'CC=clang' >> $GITHUB_ENV echo 'CXX=clang++' >> $GITHUB_ENV else export CC=gcc export CXX=g++ echo 'CC=gcc' >> $GITHUB_ENV echo 'CXX=g++' >> $GITHUB_ENV fi env which cmake && cmake --version which ninja && ninja --version which $CC && $CC --version which $CXX && $CXX --version cd runtime/Cpp cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug -DANTLR_BUILD_CPP_TESTS=OFF -DCMAKE_UNITY_BUILD=${{ matrix.unity_build }} -DCMAKE_UNITY_BUILD_BATCH_SIZE=20 -S . -B out/Debug cmake --build out/Debug --parallel cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DANTLR_BUILD_CPP_TESTS=OFF -S . -B out/Release cmake --build out/Release --parallel - name: Prepare artifacts if: always() run: | cd ${{ github.workspace }}/.. tar czfp antlr_${{ matrix.os }}_${{ matrix.compiler }}.tgz --exclude='.git' antlr4 mv antlr_${{ matrix.os }}_${{ matrix.compiler }}.tgz ${{ github.workspace }}/. - name: Archive artifacts if: always() continue-on-error: true uses: actions/upload-artifact@v4 with: name: antlr_${{ matrix.os }}_${{ matrix.compiler }} path: antlr_${{ matrix.os }}_${{ matrix.compiler }}.tgz build: runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: os: [ macos-15, ubuntu-22.04, windows-2022 ] target: [ tool, cpp, csharp, dart, go, java, javascript, typescript, php, python3, # swift, ] exclude: - os: windows-2022 target: swift steps: # Check out the code before setting the environment since some # of the actions actually parse the files to figure out the # dependencies, for instance, the setup-java actually parses # **/pom.xml files to decide what to cache. - name: Check out code uses: actions/checkout@v3 - name: Checkout antlr PHP runtime if: matrix.target == 'php' uses: actions/checkout@v3 with: repository: antlr/antlr-php-runtime path: runtime/PHP - name: Setup PHP 8.2 if: matrix.target == 'php' uses: shivammathur/setup-php@v2 with: php-version: '8.3' extensions: mbstring tools: composer - name: Install dependencies env: COMPOSER_CACHE_DIR: ${{ github.workspace }}/.cache if: matrix.target == 'php' run: |- cd runtime/PHP composer install --no-progress --no-interaction --prefer-dist --optimize-autoloader - name: Install dependencies (Ubuntu) if: startswith(matrix.os, 'ubuntu') run: | sudo apt-get update -qq sudo apt install -y ninja-build - name: Install dependencies (MacOS) if: startswith(matrix.os, 'macos') run: brew install ninja - name: Set up JDK 11 id: setup-java uses: actions/setup-java@v3 with: distribution: 'zulu' java-version: 11 cache: 'maven' - name: Set up Maven if: steps.setup-java.outputs.cache-hit != 'true' uses: stCarolas/setup-maven@v4.5 with: maven-version: 3.8.5 - name: Add msbuild to PATH if: startswith(matrix.os, 'windows') && (matrix.target == 'cpp') uses: microsoft/setup-msbuild@v1.1 - name: Set up Python 3 if: matrix.target == 'python3' uses: actions/setup-python@v4 with: python-version: '3.x' architecture: 'x64' - name: Set up Node 16 if: (matrix.target == 'javascript') || (matrix.target == 'typescript') uses: actions/setup-node@v3.6.0 with: node-version: '16' - name: Setup Dotnet if: matrix.target == 'csharp' uses: actions/setup-dotnet@v3.0.3 with: dotnet-version: '7.0.x' - name: Setup Dart 2.12.1 if: matrix.target == 'dart' uses: dart-lang/setup-dart@v1.3 with: sdk: 2.12.1 - name: Setup Go 1.19 if: matrix.target == 'go' uses: actions/setup-go@v3.3.1 with: go-version: '^1.19' - name: Setup Swift if: matrix.target == 'swift' uses: swift-actions/setup-swift@v1.19.0 with: swift-version: '5.2' - name: Use ccache if: (startswith(matrix.os, 'macos') || startswith(matrix.os, 'ubuntu')) && (matrix.target == 'cpp') uses: hendrikmuhs/ccache-action@v1.2 with: key: ${{ matrix.os }}-${{ matrix.target }} - name: Configure shell (Ubuntu) if: startswith(matrix.os, 'ubuntu') && (matrix.target == 'cpp') run: echo 'PATH=/usr/lib/ccache:'"$PATH" >> $GITHUB_ENV - name: Configure shell (MacOS) if: startswith(matrix.os, 'macos') && (matrix.target == 'cpp') run: echo "PATH=$(brew --prefix)/opt/ccache/libexec:$PATH" >> $GITHUB_ENV - name: Build ANTLR with Maven run: mvn install -DskipTests=true -Darguments="-Dmaven.javadoc.skip=true" -B -V - name: Test tool if: matrix.target == 'tool' run: | cd tool-testsuite mvn test - name: Test runtime (Windows) if: startsWith(matrix.os, 'windows') && (matrix.target != 'tool') run: | gci env:* | sort-object name cd runtime-testsuite switch ("${{ matrix.target }}") { python3 { mvn -X '-Dantlr-python3-exec="${{ env.pythonLocation }}\python.exe"' '-Dtest=python3.**' test } default { mvn -X '-Dtest=${{ matrix.target }}.**' test } } env: CMAKE_GENERATOR: Ninja - name: Test runtime (non-Windows) if: (startsWith(matrix.os, 'ubuntu') || startsWith(matrix.os, 'macos')) && (matrix.target != 'tool') run: | env cd runtime-testsuite case ${{ matrix.target }} in python3) mvn -X '-Dantlr-python3-exec=${{ env.pythonLocation }}/bin/python' '-Dtest=python3.**' test ;; *) mvn -X '-Dtest=${{ matrix.target }}.**' test ;; esac - name: Prepare artifacts if: always() run: | cd ${{ github.workspace }}/.. tar czfp antlr_${{ matrix.os }}_${{ matrix.target }}.tgz --exclude='.git' antlr4 mv antlr_${{ matrix.os }}_${{ matrix.target }}.tgz ${{ github.workspace }}/. - name: Archive artifacts if: always() continue-on-error: true uses: actions/upload-artifact@v4 with: name: antlr_${{ matrix.os }}_${{ matrix.target }} path: antlr_${{ matrix.os }}_${{ matrix.target }}.tgz ================================================ FILE: .gitignore ================================================ # Nuget packages *.nupkg # Maven build folders target/ # ... but not code generation targets !tool/src/org/antlr/v4/codegen/target/ # Node.js (npm and typings) cached dependencies node_modules/ typings/ # Ant build folders build/ dist/ lib/ user.build.properties # MacOSX files .DS_Store ## Python, selected lines from https://raw.githubusercontent.com/github/gitignore/master/Python.gitignore # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # Build results *.egg-info/ ## CSharp and VisualStudio, selected lines from https://raw.githubusercontent.com/github/gitignore/master/VisualStudio.gitignore # User-specific files *.suo *.user *.userosscache *.sln.docstates # User-specific files (MonoDevelop/Xamarin Studio/Visual Studio) *.userprefs .vs/ project.lock.json # Build results [Dd]ebug/ [Dd]ebugPublic/ [Rr]elease/ [Rr]eleases/ x64/ x86/ bld/ [Bb]in/ [Oo]bj/ [Ll]og/ # NetBeans user configuration files nbactions*.xml /nbproject/private/ */nbproject/private/ # IntelliJ projects *.iws *.iml .idea/ # Eclipse projects .classpath .project .settings/ .metadata # Profiler results *.hprof # parrt's bash prompt data .fetch_time_cache # Playground #/tool/playground/ # Generated files /out/ /gen/ /gen3/ /gen4/ /tool/playground/ tmp/ **/generatedCode/*.interp **/generatedCode/*.tokens **/generatedCode/*.bak # Configurable build files bilder.py bilder.pyc bild.log bild_output.txt runtime/Cpp/demo/generated xcuserdata *.jar .vscode # VSCode Java plugin temporary files javac-services.0.log javac-services.0.log.lck # Don't ignore python tests !runtime/Python3/test/ Antlr4.sln runtime/PHP # Swift binaries .build/ # Code coverage reports coverage/ # Cpp generated build files runtime/Cpp/CMakeCache.txt runtime/Cpp/CMakeFiles/ runtime/Cpp/CPackConfig.cmake runtime/Cpp/CPackSourceConfig.cmake runtime/Cpp/CTestTestfile.cmake runtime/Cpp/Makefile runtime/Cpp/_deps/ runtime/Cpp/cmake_install.cmake runtime/Cpp/runtime/CMakeFiles/ runtime/Cpp/runtime/CTestTestfile.cmake runtime/Cpp/runtime/Makefile runtime/Cpp/runtime/antlr4_tests runtime/Cpp/runtime/antlr4_tests\[1]_include.cmake runtime/Cpp/runtime/antlr4_tests\[1]_tests.cmake runtime/Cpp/runtime/cmake_install.cmake runtime/Cpp/runtime/libantlr4-runtime.4.10.1.dylib runtime/Cpp/runtime/libantlr4-runtime.a runtime/Cpp/runtime/libantlr4-runtime.dylib /runtime/Cpp/runtime/libantlr4-runtime.4.12.0.dylib /runtime/Cpp/runtime/libantlr4-runtime.4.13.0.dylib # Go test and performance trace files **/*.pprof ================================================ FILE: ANTLR-HOUSE-RULES.md ================================================ # ANTLR HOUSE RULES *Last updated: Sept 10, 2022* This brief document describes best practices for us to all get along and for the benefit of the project. Collaborating on this project poses a number of difficulties: * different native languages * different time zones * lack of common company or other organization as social glue * we are just github userids without personal connection to most other contributors * those developers able to contribute to such a complex project typically have a lot of experience and, consequently, strong opinions Effective communication is difficult under the circumstances and civil discourse is a requirement to keep the project on track. Over 35 years, in-fighting between contributors has made parrt's job as supreme dictator for life much more difficult. Rules 1. Assume good intentions of the other party. 2. Try to be welcoming and respectful of differing viewpoints experiences. 2. No personal attacks, meaning ideas can be bad in your comments but not people. Replace "You are ..." with "Your idea is ...". 3. Control your anger please. No hate speech, racism, sexism, or ethnocentrism. No trolling or insulting. See rule #1. 2. Be tolerant and understanding of non-native English speakers' word choice and phrasing. This is a huge source of misunderstandings; see rule #1. For example, to a native English speaker "I cannot *approve* this" makes it sound like the writer has control over the readers contribution. Instead, the writer likely meant "I cannot *support* this." See rule #1. 3. Soften word choice to use conditional tenses and helper words. For example, use phrases such as "I'm not sure this is a good idea because ..." or "I wonder if you'd consider this other possibility: ..." etc... Supreme dictator for life parrt has final say. His decisions will not always be correct nor to your liking, but he has a difficult cost-benefit equation to solve for every bug fix, feature, and PR. Any text contrary to these house rules will likely be edited and replaced with an admonishment by parrt. Send concerns to parrt@antlr.org. ================================================ FILE: CHANGES.txt ================================================ **************************************************************************** As of ANTLR 4.2.1, March 25 2014, we are no longer updating this file. Instead, we are using the github release mechanism. For example, here is 4.2.1 release notes: https://github.com/antlr/antlr4/releases/tag/4.2.1 **************************************************************************** ANTLR v4 Honey Badger January 15, 2014 * Unit tests for lexer actions from yesterday. * Refactored TreeView so we can refresh tree externally w/o creating new one. Needed for intellij plugin. January 14, 2014 * Updated serialized ATN representation of lexer actions, allowing the lexer interpreter to execute the majority of lexer commands (#408) January 12, 2014 * Support executing precedence predicates during the SLL phase of adaptivePredict (#401). The result is a massive performance boost for grammars containing direct left-recursion (improvements of 5% to 1000+% have been observed, depending on the grammar and input). December 29, 2013 * Internal change: Tool.loadGrammar() -> parseGrammar(). Tool.load()->parse() * Added Tool.loadGrammar(fileName) that completely parses, extracts implicit lexer, and processes into Grammar object. Does not geneate code. Use Grammar.getImplicitLexer() to get the lexer created during processing of combined grammar. * Added Grammar.load(fileName) that creates Tool object for you. loadGrammar() lets you create your own Tool for setting error handlers etc... final Grammar g = Grammar.load("/tmp/MyGrammar.g4"); December 19, 2013 * Sam: Improved documentation for tree patterns classes Refactored parts of the tree patterns API to simplify classes and improve encapsulation Move ATN serializer to runtime Use ATNDeserializer methods instead of ATNSimulator methods which are now deprecated * parrt: fix null pointer bug with rule "a : a;" November 24, 2013 * Ter adds tree pattern matching. Preferred interface: ParseTree t = parser.expr(); ParseTreePattern p = parser.compileParseTreePattern("+0", MyParser.RULE_expr); ParseTreeMatch m = p.match(t); String id = m.get("ID"); or String xpath = "//blockStatement/*"; String treePattern = "int = ;"; ParseTreePattern p = parser.compileParseTreePattern(treePattern, JavaParser.RULE_localVariableDeclarationStatement); List matches = p.findAll(tree, xpath); November 20, 2013 * Sam added method stuff like expr() that calls expr(0). Makes it possible to call expr rule from TestRig (grun). November 14, 2013 * Added Sam's ParserInterpreter implementation that uses ATN after deserialization. LexerGrammar lg = new LexerGrammar( "lexer grammar L;\n" + "A : 'a' ;\n" + "B : 'b' ;\n" + "C : 'c' ;\n"); Grammar g = new Grammar( "parser grammar T;\n" + "s : (A{;}|B)* C ;\n", lg); LexerInterpreter lexEngine = lg.createLexerInterpreter(new ANTLRInputStream(input)); CommonTokenStream tokens = new CommonTokenStream(lexEngine); ParserInterpreter parser = g.createParserInterpreter(tokens); ParseTree t = parser.parse(g.rules.get(startRule).index); November 13, 2013 * move getChildren() from Tree into Trees (to avoid breaking change) * Notation: /prog/func, -> all funcs under prog at root /prog/*, -> all children of prog at root /*/func, -> all func kids of any root node prog, -> prog must be root node /prog, -> prog must be root node /*, -> any root *, -> any root //ID, -> any ID in tree //expr/primary/ID, -> any ID child of a primary under any expr //body//ID, -> any ID under a body //'return', -> any 'return' literal in tree //primary/*, -> all kids of any primary //func/*/stat, -> all stat nodes grandkids of any func node /prog/func/'def', -> all def literal kids of func kid of prog //stat/';', -> all ';' under any stat node //expr/primary/!ID, -> anything but ID under primary under any expr node //expr/!primary, -> anything but primary under any expr node //!*, -> nothing anywhere /!*, -> nothing at root September 16, 2013 * Updated build.xml to support v4 grammars in v4 itself; compiles XPathLexer.g4 * Add to XPath: Collection findAll(String xpath); September 11, 2013 * Add ! operator to XPath * Use ANTLR v4 XPathLexer.g4 not regex * Copy lots of find node stuff from v3 GrammarAST to Trees class in runtime. September 10, 2013 * Adding in XPath stuff. August 31, 2013 * Lots of little fixes thanks to Coverity Scan August 7, 2013 * [BREAKING CHANGE] Altered left-recursion elimination to be simpler. Now, we use the following patterns: * Binary expressions are expressions which contain a recursive invocation of the rule as the first and last element of the alternative. * Suffix expressions contain a recursive invocation of the rule as the first element of the alternative, but not as the last element. * Prefix expressions contain a recursive invocation of the rule as the last element of the alternative, but not as the first element. There is no such thing as a "ternary" expression--they are just binary expressions in disguise. The right associativity specifiers no longer on the individual tokens because it's done on alternative basis anyway. The option is now on the individual alternative; e.g., e : e '*' e | e '+' e | e '?' e ':' e | e '=' e | INT ; If your language uses a right-associative ternary operator, you will need to update your grammar to include on the alternative operator. This also fixes #245 and fixes #268: https://github.com/antlr/antlr4/issues/245 https://github.com/antlr/antlr4/issues/268 To smooth the transition, is still allowed on token references but it is ignored. June 30, 2013 -- 4.1 release June 24, 2013 * Resize ANTLRInputStream.data after reading a file with fewer characters than bytes * Fix ATN created for non-greedy optional block with multiple alternatives * Support Unicode escape sequences with indirection in JavaUnicodeInputStream (fixes #287) * Remove the ParserRuleContext.altNum field (fixes #288) * PredictionContext no longer implements Iterable * PredictionContext no longer implements Comparable * Add the EPSILON_CLOSURE error and EPSILON_OPTIONAL warning * Optimized usage of closureBusy set (fixes #282) June 9, 2013 * Add regression test for #239 (already passes) June 8, 2013 * Support list labels on a set of tokens (fixes #270) * Fix associativity of XOR in Java LR grammar (fixes #280) June 1, 2013 * DiagnosticErrorListener includes rule names for each decision in its reports * Document ANTLRErrorListener and DiagnosticErrorListener (fixes #265) * Support '\uFFFF' (fixes #267) * Optimize serialized ATN May 26, 2013 * Report errors that occur while lexing a grammar (fixes #262) * Improved error message for unterminated string literals (fixes #243) May 24, 2013 * Significantly improve performance of JavaUnicodeInputStream.LA(1) May 20, 2013 * Generate Javadoc for generated visitor and listener interfaces and classes * Fix unit tests May 18, 2013 * Group terminals in Java grammars so ATN can collapse sets * Improved Java 7 support in Java grammars (numeric literals) * Updated error listener interfaces * Support detailed statistics in TestPerformance May 17, 2013 * Add JavaUnicodeInputStream to handle Unicode escapes in Java code * Proper Unicode identifier handling in Java grammars * Report file names with lexer errors in TestPerformance May 14, 2013 * Use a called rule stack to prevent stack overflow in LL1Analyzer * Use 0-based indexing for several arrays in the tool * Code simplification, assertions, documentation May 13, 2013 * Unit test updates to ensure exceptions are not hidden May 12, 2013 * Updates to TestPerformance May 5, 2013 * Updated several classes to use MurmurHash 3 hashing May 1, 2013 * Added parse tree JTree to TreeViewer (Bart Kiers) April 30, 2013 * Updated TestPerformance to support parallelization across passes April 24, 2013 * Remove unused stub class ParserATNPathFinder * Remove ParserInterpreter.predictATN * Remove DFA.getATNStatesAlongPath * Encapsulate implementation methods in LexerATNSimulator and ParserATNSimulator * Updated documentation * Simplify creation of new DFA edges * Fix handling of previously cached error edges * Fix DFA created during forced-SLL parsing (PredictionMode.SLL) * Extract methods ParserATNSimulator.getExistingTargetState and ParserATNSimulator.computeTargetState. April 22, 2013 * Lazy initialization of ParserATNSimulator.mergeCache * Improved hash code for DFAState * Improved hash code with caching for ATNConfigSet * Add new configuration parameters to TestPerformance * Update Java LR and Java Std to support Java 7 syntax April 21, 2013 * Add new configuration parameters to TestPerformance April 18, 2013 * Must check rule transition follow states before eliminating states in the ATN (fixes #224) * Simplify ParserATNSimulator and improve performance by combining execDFA and execATN and using DFA edges even after edge computation is required April 15, 2013 * Fix code in TestPerformance that clears the DFA April 12, 2013 * Improved initialization and concurrency control in DFA updates * Fix EOF handling in edge case (fixes #218) April 4, 2013 * Improved testing of error reporting * Fix NPE revealed by updated testing method * Strict handling of redefined rules - prevents code generation (fixes #210) * Updated documentation in Tool March 27, 2013 * Avoid creating empty action methods in lexer (fixes #202) * Split serialized ATN when it exceeds Java's 65535 byte limit (fixes #76) * Fix incorrect reports of label type conflicts across separated labeled outer alternatives (fixes #195) * Update Maven plugin site documentation March 26, 2013 * Fix bugs with the closureBusy set in ParserATNSimulator.closure * Fix handling of empty options{} block (fixes #194) * Add error 149 INVALID_LEXER_COMMAND (fixes #190) * Add error 150 MISSING_LEXER_COMMAND_ARGUMENT * Add error 151 UNWANTED_LEXER_COMMAND_ARGUMENT * Updated documentation in the Parser and RecognitionException classes * Refactored and extensively documented the ANTLRErrorStrategy interface and DefaultErrorStrategy default implementation * Track the number of syntax errors in Parser.notifyErrorListeners instead of in the error strategy * Move primary implementation of getExpectedTokens to ATN, fixes #191 * Updated ATN documentation * Use UUID instead of incremented integer for serialized ATN versioning March 7, 2013 * Added export to PNG feature to the parse tree viewer March 6, 2013 * Allow direct calls to left-recursive rules (fixes #161) * Change error type 146 (EPSILON_TOKEN) to a warning (fixes #180) * Specify locale for all format operations (fixes #158) * Fix generation of invalid Unicode escape sequences in Java code (fixes #164) * Do not require escape for $ in action when not followed by an ID start char (fixes #176) February 23, 2013 * Refactoring Target-related classes to improve support for additional language targets February 22, 2013 * Do not allow raw newline characters in literals * Pair and Triple are immutable; Triple is not a Pair February 5, 2013 * Fix IntervalSet.add when multiple merges are required (fixes #153) January 29, 2013 * don't call process() if args aren't specified (Dave Parfitt) January 21, 2013 -- Release 4.0 * Updated PredictionContext Javadocs * Updated Maven site documentation * Minor tweaks in Java.stg January 15, 2013 * Tweak error messages * (Tool) Make TokenVocabParser fields `protected final` * Fix generated escape sequences for literals containing backslashes January 14, 2013 * Relax parser in favor of errors during semantic analysis * Add error 145: lexer mode must contain at least one non-fragment rule * Add error 146: non-fragment lexer rule can match the empty string January 11, 2013 * Updated error 72, 76; added 73-74 and 136-143: detailed errors about name conflicts * Report exact location for parameter/retval/local name conflicts * Add error 144: multi-character literals are not allowed in lexer sets * Error 134 now only applies to rule references in lexer sets * Updated error messages (cleanup) * Reduce size of _serializedATN by adding 2 to each element: new representation avoids embedded values 0 and 0xFFFF which are common and have multi-byte representations in Java's modified UTF-8 January 10, 2013 * Add error 135: cannot assign a value to list label: $label (fixes antlr/antlr4#128) January 2, 2013 * Fix EOF handling (antlr/antlr4#110) * Remove TREE_PARSER reference * Additional validation checks in ATN deserialization * Fix potential NPE in parser predicate evaluation * Fix termination condition detection in full-context parsing January 1, 2013 * Updated documentation * Minor code cleanup * Added the `-XdbgSTWait` command line option for the Tool * Removed method override since bug was fixed in V3 runtime December 31, 2012 * I altered Target.getTargetStringLiteralFromANTLRStringLiteral() so that it converts \uXXXX in an ANTLR string to \\uXXXX, thus, avoiding Java's conversion to a single character before compilation. December 16, 2012 * Encapsulate some fields in ANTLRMessage * Remove ErrorType.INVALID * Update error/warning messages, show all v3 compatibility messages December 12, 2012 * Use arrays instead of HashSet to save memory in SemanticContext.AND/OR * Use arrays instead of HashSet to save memory in cached DFA * Reduce single-operand SemanticContext.and/or operations December 11, 2012 * Add -long-messages option; only show exceptions with errors when set * "warning treated as error" is a one-off error * Listen for issues reported by StringTemplate, report them as warnings * Fix template issues * GrammarASTWithOptions.getOptions never returns null * Use EnumSet instead of HashSet * Use new STGroup.GROUP_FILE_EXTENSION value December 2, 2012 * Remove -Xverbose-dfa option * Create the ParseTreeVisitor interface for all visitors, rename previous base visitor class to AbstractParseTreeVisitor December 1, 2012 * escape [\n\r\t] in lexical error messages; e.g,: line 2:3 token recognition error at: '\t' line 2:4 token recognition error at: '\n' * added error for bad sets in lexer; e.g.: lexer set element A is invalid (either rule ref or literal with > 1 char) some tests in TestSets appeared to allow ~('a'|B) but it was randomly working. ('a'|B) works, though doesn't collapse to a set. * label+='foo' wasn't generating good code. It was generating token type as variable name. Now, I gen "s" for implicit labels on string literals. * tokens now have token and char source to draw from. * remove -Xsave-lexer option; log file as implicit lexer AST. November 30, 2012 * Maven updates (cleanup, unification, and specify Java 6 bootstrap classpath) November 28, 2012 * Maven updates (uber-jar, manifest details) November 27, 2012 * Maven updates (prepare for deploying to Sonatype OSS) * Use efficient bitset tests instead of long chains of operator == November 26, 2012 * Maven updates (include sources and javadocs, fix warnings) * Don't generate action methods for lexer rules not containing an action * Generated action and sempred methods are private * Remove unused / problematic methods: ** (unused) TerminalNodeImpl.isErrorNode ** (unused) RuleContext.conflictsWith, RuleContext.suffix. ** (problematic) RuleContext.hashCode, RuleContext.equals. November 23, 2012 * Updated Maven build (added master POM, cleaned up module POMs) November 22, 2012 * make sure left-recur rule translation uses token stream from correct imported file. * actions like @after in imported rules caused inf loop. * This misidentified scope lexer/parser: @lexer::members { } @parser::members { } November 18, 2012 * fixed: undefined rule refs caused exception * cleanup, rm dead etypes, add check for ids that cause code gen issues * added notion of one-off error * added check for v3 backward incompatibilities: ** tree grammars ** labels in lexer rules ** tokens {A;B;} syntax ** tokens {A='C';} syntax ** {...}?=> gate semantic predicates ** (...)=> syntactic predicates * Detect EOF in lexer rule November 17, 2012 * .tokens files goes in output dir like parser file. * added check: action in lexer rules must be last element of outermost alt * properly check for grammar/filename difference * if labels, don't allow set collapse for a : A # X | B ; * wasn't checking soon enough for rule redef; now it sets a dead flag in AST so no more walking dup. error(51): T.g:7:0: rule s redefinition (ignoring); previous at line 3 November 11, 2012 * Change version to 4.0b4 (btw, forgot to push 4.0b3 in build.properties when I made git tag 4.0b3...ooops). November 4, 2012 * Kill box in tree dialog box makes dialog dispose of itself October 29, 2012 * Sam fixes nongreedy more. * -Werror added. * Sam made speed improvement re preds in lexer. October 20, 2012 * Merged Sam's fix for nongreedy lexer/parser. lots of unit tests. A fix in prediction ctx merge. https://github.com/parrt/antlr4/pull/99 October 14, 2012 * Rebuild how ANTLR detects SLL conflict and failover to full LL. LL is a bit slower but correct now. Added ability to ask for exact ambiguity detection. October 8, 2012 * Fixed a bug where labeling the alternatives of the start rule caused a null pointer exception. October 1, 2012 -- 4.0b2 release September 30, 2012 * Fixed the unbuffered streams, which actually buffered everything up by mistake. tweaked a few comments. * Added a getter to IntStream for the token factory * Added -depend cmd-line option. September 29, 2012 * no nongreedy or wildcard in parser. September 28, 2012 * empty "tokens {}" is ok now. September 22, 2012 * Rule exception handlers weren't passed to the generated code * $ruleattribute.foo weren't handled properly * Added -package option September 18, 2012 -- 4.0b1 release ================================================ FILE: CONTRIBUTING.md ================================================ # Contributing to ANTLR 4 1. [Fork](https://help.github.com/articles/fork-a-repo) the [antlr/antlr4 repo](https://github.com/antlr/antlr4), which will give you both key branches, `master` and `dev` 2. Make sure to `git checkout dev` in your fork so that you are working from the latest development branch 3. Create and work from a branch derived from `dev` such as `git checkout -b your-branch-name` 4. Install and configure [EditorConfig](http://editorconfig.org/) so your text editor or IDE uses the ANTLR 4 coding style 5. [Build ANTLR 4](doc/building-antlr.md) 6. [Run the ANTLR project unit tests](doc/antlr-project-testing.md) 7. Create a [pull request](https://help.github.com/articles/using-pull-requests/) with your changes and make sure you're comparing your `dev`-derived branch in your fork to the `dev` branch from the `antlr/antlr4` repo: **Note:** Each commit requires a "signature", which is simple as using `-s` (not `-S`) to the git commit command: ``` git commit -s -m 'This is my commit message' ``` Github's pull request process enforces the sig and gives instructions on how to fix any commits that lack the sig. See [Github DCO app](https://github.com/apps/dco) for more info. ================================================ FILE: LICENSE.txt ================================================ Copyright (c) 2012-2022 The ANTLR Project. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither name of copyright holders nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ================================================ FILE: Package.swift ================================================ // swift-tools-version:5.6 import PackageDescription let package = Package( name: "Antlr4", products: [ .library( name: "Antlr4", targets: ["Antlr4"]), .library( name: "Antlr4Static", type: .static, targets: ["Antlr4"]), .library( name: "Antlr4Dynamic", type: .dynamic, targets: ["Antlr4"]), ], targets: [ .target( name: "Antlr4", dependencies: [], path: "./runtime/Swift/Sources/Antlr4"), .testTarget( name: "Antlr4Tests", dependencies: ["Antlr4"], path: "./runtime/Swift/Tests/Antlr4Tests", exclude: [ "./runtime/Swift/Tests/VisitorBasic.g4", "./runtime/Swift/Tests/VisitorCalc.g4", "./runtime/Swift/Tests/LexerA.g4", "./runtime/Swift/Tests/LexerB.g4", "./runtime/Swift/Tests/Threading.g4" ] ) ] ) ================================================ FILE: README.md ================================================ # ANTLR v4 [![Java 11+](https://img.shields.io/badge/java-11+-4c7e9f.svg)](http://java.oracle.com) [![License](https://img.shields.io/badge/license-BSD-blue.svg)](https://raw.githubusercontent.com/antlr/antlr4/master/LICENSE.txt) **ANTLR** (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files. It's widely used to build languages, tools, and frameworks. From a grammar, ANTLR generates a parser that can build parse trees and also generates a listener interface (or visitor) that makes it easy to respond to the recognition of phrases of interest. **Dev branch build status** [![MacOSX, Windows, Linux](https://github.com/antlr/antlr4/actions/workflows/hosted.yml/badge.svg)](https://github.com/antlr/antlr4/actions/workflows/hosted.yml) (github actions) ## Versioning ANTLR 4 supports 10 target languages (Cpp, CSharp, Dart, Java, JavaScript, PHP, Python3, Swift, TypeScript, Go), and ensuring consistency across these targets is a unique and highly valuable feature. To ensure proper support of this feature, each release of ANTLR is a complete release of the tool and the 10 runtimes, all with the same version. As such, ANTLR versioning does not strictly follow semver semantics: * a component may be released with the latest version number even though nothing has changed within that component since the previous release * major version is bumped only when ANTLR is rewritten for a totally new "generation", such as ANTLR3 -> ANTLR4 (LL(\*) -> ALL(\*) parsing) * minor version updates may include minor breaking changes, the policy is to regenerate parsers with every release (4.11 -> 4.12) * backwards compatibility is only guaranteed for patch version bumps (4.11.1 -> 4.11.2) If you use a semver verifier in your CI, you probably want to apply special rules for ANTLR, such as treating minor change as a major change. ## Repo branch structure The default branch for this repo is [`master`](https://github.com/antlr/antlr4/tree/master), which is the latest stable release and has tags for the various releases; e.g., see release tag [4.9.3](https://github.com/antlr/antlr4/tree/4.9.3). Branch [`dev`](https://github.com/antlr/antlr4/tree/dev) is where development occurs between releases and all pull requests should be derived from that branch. The `dev` branch is merged back into `master` to cut a release and the release state is tagged (e.g., with `4.10-rc1` or `4.10`.) Visually our process looks roughly like this: The Go target now has its own dedicated repo: ```bash $ go get github.com/antlr4-go/antlr ``` **Note** The dedicated Go repo is for `go get` and `import` only. Go runtime development is still performed in the main `antlr/antlr4` repo. ## Authors and major contributors * [Terence Parr](http://www.cs.usfca.edu/~parrt/), parrt@cs.usfca.edu ANTLR project lead and supreme dictator for life [University of San Francisco](http://www.usfca.edu/) * [Sam Harwell](http://tunnelvisionlabs.com/) (Tool co-author, Java and original C# target) * [Eric Vergnaud](https://github.com/ericvergnaud) (Javascript, TypeScript, Python2, Python3 targets and maintenance of C# target) * [Peter Boyer](https://github.com/pboyer) (Go target) * [Mike Lischke](http://www.soft-gems.net/) (C++ completed target) * Dan McLaughlin (C++ initial target) * David Sisson (C++ initial target and test) * [Janyou](https://github.com/janyou) (Swift target) * [Ewan Mellor](https://github.com/ewanmellor), [Hanzhou Shi](https://github.com/hanjoes) (Swift target merging) * [Ben Hamilton](https://github.com/bhamiltoncx) (Full Unicode support in serialized ATN and all languages' runtimes for code points > U+FFFF) * [Marcos Passos](https://github.com/marcospassos) (PHP target) * [Lingyu Li](https://github.com/lingyv-li) (Dart target) * [Ivan Kochurkin](https://github.com/KvanTTT) has made major contributions to overall quality, error handling, and Target performance. * [Justin King](https://github.com/jcking) has done a huge amount of work across multiple targets, but especially for C++. * [Ken Domino](https://github.com/kaby76) has a knack for finding bugs/issues and analysis; also a major contributor on the [grammars-v4 repo](https://github.com/antlr/grammars-v4). * [Jim Idle](https://github.com/jimidle) has contributed to previous versions of ANTLR and recently jumped back in to solve a major problem with the Go target. ## Useful information * [Release notes](https://github.com/antlr/antlr4/releases) * [Getting started with v4](https://github.com/antlr/antlr4/blob/master/doc/getting-started.md) * [Official site](http://www.antlr.org/) * [Documentation](https://github.com/antlr/antlr4/blob/master/doc/index.md) * [FAQ](https://github.com/antlr/antlr4/blob/master/doc/faq/index.md) * [ANTLR code generation targets](https://github.com/antlr/antlr4/blob/master/doc/targets.md)
(Currently: Java, C#, Python3, JavaScript, TypeScript, Go, C++, Swift, Dart, PHP) * _Note: As of version 4.14, we are dropping support for Python 2. We love the Python community, but Python 2 support was officially halted in Jan 2020. More recently, GitHub also dropped support for Python 2, which has made it impossible for us to maintain a consistent level of quality across targets (we use GitHub for our CI). Long live Python 3!_ * [Java API](http://www.antlr.org/api/Java/index.html) * [ANTLR v3](http://www.antlr3.org/) * [v3 to v4 Migration, differences](https://github.com/antlr/antlr4/blob/master/doc/faq/general.md) You might also find the following pages useful, particularly if you want to mess around with the various target languages. * [How to build ANTLR itself](https://github.com/antlr/antlr4/blob/master/doc/building-antlr.md) * [How we create and deploy an ANTLR release](https://github.com/antlr/antlr4/blob/master/doc/releasing-antlr.md) ## The Definitive ANTLR 4 Reference Programmers run into parsing problems all the time. Whether it’s a data format like JSON, a network protocol like SMTP, a server configuration file for Apache, a PostScript/PDF file, or a simple spreadsheet macro language—ANTLR v4 and this book will demystify the process. ANTLR v4 has been rewritten from scratch to make it easier than ever to build parsers and the language applications built on top. This completely rewritten new edition of the bestselling Definitive ANTLR Reference shows you how to take advantage of these new features. You can buy the book [The Definitive ANTLR 4 Reference](http://amzn.com/dp/1934356999) at amazon or an [electronic version at the publisher's site](https://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference). You will find the [Book source code](http://pragprog.com/titles/tpantlr2/source_code) useful. ## Additional grammars [This repository](https://github.com/antlr/grammars-v4) is a collection of grammars without actions where the root directory name is the all-lowercase name of the language parsed by the grammar. For example, java, cpp, csharp, c, etc... ================================================ FILE: antlr4-maven-plugin/nb-configuration.xml ================================================ project 4 4 4 true 80 none false 4 4 4 false 80 none 4 false false *;java test ================================================ FILE: antlr4-maven-plugin/pom.xml ================================================ 4.0.0 org.antlr antlr4-master 4.13.3-SNAPSHOT antlr4-maven-plugin maven-plugin ANTLR 4 Maven plugin Maven plugin for ANTLR 4 grammars 2009 3.8.5 org.apache.maven maven-plugin-api ${mavenVersion} provided org.codehaus.plexus plexus-compiler-api 2.12.1 org.sonatype.plexus plexus-build-api 0.0.7 org.antlr antlr4 ${project.version} junit junit 4.13.2 test org.apache.maven.plugin-tools maven-plugin-annotations 3.6.4 provided io.takari.maven.plugins takari-plugin-testing 3.0.0 test org.apache.maven maven-core ${mavenVersion} provided org.apache.maven maven-compat ${mavenVersion} test org.codehaus.plexus plexus-utils 3.4.2 provided org.slf4j slf4j-api 2.0.0 org.slf4j slf4j-simple 2.0.0 resources src/test src/test/resources org.apache.maven.plugins maven-plugin-plugin 3.6.2 true mojo-descriptor descriptor help-goal helpmojo io.takari.maven.plugins takari-lifecycle-plugin 2.0.7 true testProperties process-test-resources testProperties org.apache.maven.plugins maven-compiler-plugin 8 org.apache.maven.plugins maven-javadoc-plugin 3.3.1 true ================================================ FILE: antlr4-maven-plugin/resources/META-INF/m2e/lifecycle-mapping-metadata.xml ================================================ antlr4 true true ================================================ FILE: antlr4-maven-plugin/src/main/java/org/antlr/mojo/antlr4/Antlr4ErrorLog.java ================================================ /* * Copyright (c) 2012-2017 The ANTLR Project. All rights reserved. * Use of this file is governed by the BSD 3-clause license that * can be found in the LICENSE.txt file in the project root. */ package org.antlr.mojo.antlr4; import org.antlr.v4.Tool; import org.antlr.v4.tool.ANTLRMessage; import org.antlr.v4.tool.ANTLRToolListener; import org.apache.maven.plugin.logging.Log; import org.sonatype.plexus.build.incremental.BuildContext; import org.stringtemplate.v4.ST; import java.io.File; /** * This implementation of {@link ANTLRToolListener} reports messages to the * {@link Log} instance provided by Maven. * * @author Sam Harwell */ public class Antlr4ErrorLog implements ANTLRToolListener { private final Tool tool; private final BuildContext buildContext; private final Log log; /** * Creates an instance of {@link Antlr4ErrorLog}. * * @param log The Maven log */ public Antlr4ErrorLog(Tool tool, BuildContext buildContext, Log log) { this.tool = tool; this.buildContext = buildContext; this.log = log; } /** * {@inheritDoc} *

* This implementation passes the message to the Maven log. *

* @param message The message to send to Maven */ @Override public void info(String message) { if (tool.errMgr.formatWantsSingleLineMessage()) { message = message.replace('\n', ' '); } log.info(message); } /** * {@inheritDoc} *

* This implementation passes the message to the Maven log. *

* @param message The message to send to Maven. */ @Override public void error(ANTLRMessage message) { ST msgST = tool.errMgr.getMessageTemplate(message); String outputMsg = msgST.render(); if (tool.errMgr.formatWantsSingleLineMessage()) { outputMsg = outputMsg.replace('\n', ' '); } log.error(outputMsg); if (message.fileName != null) { String text = message.getMessageTemplate(false).render(); buildContext.addMessage(new File(message.fileName), message.line, message.charPosition, text, BuildContext.SEVERITY_ERROR, message.getCause()); } } /** * {@inheritDoc} *

* This implementation passes the message to the Maven log. *

* @param message */ @Override public void warning(ANTLRMessage message) { ST msgST = tool.errMgr.getMessageTemplate(message); String outputMsg = msgST.render(); if (tool.errMgr.formatWantsSingleLineMessage()) { outputMsg = outputMsg.replace('\n', ' '); } log.warn(outputMsg); if (message.fileName != null) { String text = message.getMessageTemplate(false).render(); buildContext.addMessage(new File(message.fileName), message.line, message.charPosition, text, BuildContext.SEVERITY_WARNING, message.getCause()); } } } ================================================ FILE: antlr4-maven-plugin/src/main/java/org/antlr/mojo/antlr4/Antlr4Mojo.java ================================================ /* * Copyright (c) 2012-2017 The ANTLR Project. All rights reserved. * Use of this file is governed by the BSD 3-clause license that * can be found in the LICENSE.txt file in the project root. */ package org.antlr.mojo.antlr4; import org.antlr.v4.Tool; import org.antlr.v4.codegen.CodeGenerator; import org.antlr.v4.runtime.misc.MultiMap; import org.antlr.v4.runtime.misc.Utils; import org.antlr.v4.tool.Grammar; import org.apache.maven.plugin.AbstractMojo; import org.apache.maven.plugin.MojoExecutionException; import org.apache.maven.plugin.MojoFailureException; import org.apache.maven.plugin.logging.Log; import org.apache.maven.plugins.annotations.Component; import org.apache.maven.plugins.annotations.LifecyclePhase; import org.apache.maven.plugins.annotations.Mojo; import org.apache.maven.plugins.annotations.Parameter; import org.apache.maven.plugins.annotations.ResolutionScope; import org.apache.maven.project.MavenProject; import org.codehaus.plexus.compiler.util.scan.InclusionScanException; import org.codehaus.plexus.compiler.util.scan.SimpleSourceInclusionScanner; import org.codehaus.plexus.compiler.util.scan.SourceInclusionScanner; import org.codehaus.plexus.compiler.util.scan.mapping.SourceMapping; import org.codehaus.plexus.compiler.util.scan.mapping.SuffixMapping; import org.sonatype.plexus.build.incremental.BuildContext; import java.io.BufferedWriter; import java.io.File; import java.io.IOException; import java.io.OutputStream; import java.io.OutputStreamWriter; import java.io.StringWriter; import java.io.Writer; import java.net.URI; import java.nio.charset.Charset; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Set; /** * Parses ANTLR 4 grammar files {@code *.g4} and transforms them into Java * source files. * * @author Sam Harwell */ @Mojo( name = "antlr4", defaultPhase = LifecyclePhase.GENERATE_SOURCES, requiresDependencyResolution = ResolutionScope.COMPILE, requiresProject = true, threadSafe = true) public class Antlr4Mojo extends AbstractMojo { // First, let's deal with the options that the ANTLR tool itself // can be configured by. // /** * If set to true then the ANTLR tool will generate a description of the ATN * for each rule in Dot format. */ @Parameter(property = "antlr4.atn", defaultValue = "false") protected boolean atn; /** * specify grammar file encoding; e.g., euc-jp */ @Parameter(property = "project.build.sourceEncoding") protected String inputEncoding; /** * specify output file encoding; defaults to source encoding */ @Parameter(property = "project.build.sourceEncoding") protected String outputEncoding; /** * Generate parse tree listener interface and base class. */ @Parameter(property = "antlr4.listener", defaultValue = "true") protected boolean listener; /** * Generate parse tree visitor interface and base class. */ @Parameter(property = "antlr4.visitor", defaultValue = "false") protected boolean visitor; /** * Treat warnings as errors. */ @Parameter(property = "antlr4.treatWarningsAsErrors", defaultValue = "false") protected boolean treatWarningsAsErrors; /** * Use the ATN simulator for all predictions. */ @Parameter(property = "antlr4.forceATN", defaultValue = "false") protected boolean forceATN; /** * A list of grammar options to explicitly specify to the tool. These * options are passed to the tool using the * -D<option>=<value> syntax. */ @Parameter protected Map options; /** * A list of additional command line arguments to pass to the ANTLR tool. */ @Parameter protected List arguments; /* -------------------------------------------------------------------- * The following are Maven specific parameters, rather than specific * options that the ANTLR tool can use. */ /** * Provides an explicit list of all the grammars that should be included in * the generate phase of the plugin. Note that the plugin is smart enough to * realize that imported grammars should be included but not acted upon * directly by the ANTLR Tool. *

* A set of Ant-like inclusion patterns used to select files from the source * directory for processing. By default, the pattern * **/*.g4 is used to select grammar files. *

*/ @Parameter protected Set includes = new HashSet(); /** * A set of Ant-like exclusion patterns used to prevent certain files from * being processed. By default, this set is empty such that no files are * excluded. */ @Parameter protected Set excludes = new HashSet(); /** * The current Maven project. */ @Parameter(property = "project", required = true, readonly = true) protected MavenProject project; /** * Specifies whether sources are added to the {@code compile} or * {@code test} scope. */ @Parameter(property = "antlr4.generateTestSources", defaultValue = "false") private boolean generateTestSources; /** * The directory where the ANTLR grammar files ({@code *.g4}) are located. */ @Parameter(defaultValue = "${basedir}/src/main/antlr4") private File sourceDirectory; /** * Specify output directory where the Java files are generated. */ @Parameter(defaultValue = "${project.build.directory}/generated-sources/antlr4") private File outputDirectory; /** * Specify location of imported grammars and tokens files. */ @Parameter(defaultValue = "${basedir}/src/main/antlr4/imports") private File libDirectory; /** * The directory where build status information is located. */ @Parameter(defaultValue = "${project.build.directory}/maven-status/antlr4", readonly=true) private File statusDirectory; @Component private BuildContext buildContext; public File getSourceDirectory() { return sourceDirectory; } public File getOutputDirectory() { return outputDirectory; } public File getLibDirectory() { return libDirectory; } void addSourceRoot(File outputDir) { if (generateTestSources) { project.addTestCompileSourceRoot(outputDir.getPath()); } else { project.addCompileSourceRoot(outputDir.getPath()); } } /** * An instance of the ANTLR tool build */ protected Tool tool; /** * The main entry point for this Mojo, it is responsible for converting * ANTLR 4.x grammars into the target language specified by the grammar. * * @exception MojoExecutionException if a configuration or grammar error causes * the code generation process to fail * @exception MojoFailureException if an instance of the ANTLR 4 {@link Tool} * cannot be created */ @Override public void execute() throws MojoExecutionException, MojoFailureException { Log log = getLog(); outputEncoding = validateEncoding(outputEncoding); if (log.isDebugEnabled()) { for (String e : excludes) { log.debug("ANTLR: Exclude: " + e); } for (String e : includes) { log.debug("ANTLR: Include: " + e); } log.debug("ANTLR: Output: " + outputDirectory); log.debug("ANTLR: Library: " + libDirectory); } if (!sourceDirectory.isDirectory()) { log.info("No ANTLR 4 grammars to compile in " + sourceDirectory.getAbsolutePath()); return; } // Ensure that the output directory path is all in tact so that // ANTLR can just write into it. // File outputDir = getOutputDirectory(); if (!outputDir.exists()) { outputDir.mkdirs(); } GrammarDependencies dependencies = new GrammarDependencies(sourceDirectory, libDirectory, arguments, getDependenciesStatusFile(), getLog()); // Now pick up all the files and process them with the Tool // List> argumentSets; Set grammarFiles; Set importGrammarFiles; try { List args = getCommandArguments(); grammarFiles = getGrammarFiles(sourceDirectory); importGrammarFiles = getImportFiles(sourceDirectory); argumentSets = processGrammarFiles(args, grammarFiles, dependencies, sourceDirectory); } catch (Exception e) { log.error(e); throw new MojoExecutionException("Fatal error occured while evaluating the names of the grammar files to analyze", e); } log.debug("Output directory base will be " + outputDirectory.getAbsolutePath()); log.info("ANTLR 4: Processing source directory " + sourceDirectory.getAbsolutePath()); for (List args : argumentSets) { try { // Create an instance of the ANTLR 4 build tool tool = new CustomTool(args.toArray(new String[0])); } catch (Exception e) { log.error("The attempt to create the ANTLR 4 build tool failed, see exception report for details", e); throw new MojoFailureException("Error creating an instanceof the ANTLR tool.", e); } try { dependencies.analyze(grammarFiles, importGrammarFiles, tool); } catch (Exception e) { log.error("Dependency analysis failed, see exception report for details", e); throw new MojoFailureException("Dependency analysis failed.", e); } // Set working directory for ANTLR to be the base source directory tool.inputDirectory = sourceDirectory; tool.processGrammarsOnCommandLine(); // If any of the grammar files caused errors but did nto throw exceptions // then we should have accumulated errors in the counts if (tool.getNumErrors() > 0) { throw new MojoExecutionException("ANTLR 4 caught " + tool.getNumErrors() + " build errors."); } } if (project != null) { // Tell Maven that there are some new source files underneath the output directory. addSourceRoot(this.getOutputDirectory()); } try { dependencies.save(); } catch (IOException ex) { log.warn("Could not save grammar dependency status", ex); } } private List getCommandArguments() { List args = new ArrayList(); if (getOutputDirectory() != null) { args.add("-o"); args.add(outputDirectory.getAbsolutePath()); } // Where do we want ANTLR to look for .tokens and import grammars? if (getLibDirectory() != null && getLibDirectory().isDirectory()) { args.add("-lib"); args.add(libDirectory.getAbsolutePath()); } // Next we need to set the options given to us in the pom into the // tool instance we have created. if (atn) { args.add("-atn"); } if ( inputEncoding!=null && !inputEncoding.isEmpty()) { args.add("-encoding"); outputEncoding = inputEncoding; args.add(inputEncoding); } if (listener) { args.add("-listener"); } else { args.add("-no-listener"); } if (visitor) { args.add("-visitor"); } else { args.add("-no-visitor"); } if (treatWarningsAsErrors) { args.add("-Werror"); } if (forceATN) { args.add("-Xforce-atn"); } if (options != null) { for (Map.Entry option : options.entrySet()) { args.add(String.format("-D%s=%s", option.getKey(), option.getValue())); } } if (arguments != null) { args.addAll(arguments); } return args; } /** * * @param sourceDirectory * @exception InclusionScanException */ private List> processGrammarFiles( List args, Set grammarFiles, GrammarDependencies dependencies, File sourceDirectory) throws InclusionScanException, IOException { // We don't want the plugin to run for every grammar, regardless of whether // it's changed since the last compilation. Check the mtime of the tokens vs // the grammar file mtime to determine whether we even need to execute. Set grammarFilesToProcess = new HashSet(); for (File grammarFile : grammarFiles) { String tokensFileName = grammarFile.getName().split("\\.")[0] + ".tokens"; File outputFile = new File(outputDirectory, tokensFileName); if ( (! outputFile.exists()) || outputFile.lastModified() <= grammarFile.lastModified() || dependencies.isDependencyChanged(grammarFile)) { grammarFilesToProcess.add(grammarFile); } } grammarFiles = grammarFilesToProcess; if (grammarFiles.isEmpty()) { getLog().info("No grammars to process"); return Collections.emptyList(); } MultiMap grammarFileByFolder = new MultiMap(); // Iterate each grammar file we were given and add it into the tool's list of // grammars to process. for (File grammarFile : grammarFiles) { buildContext.refresh(grammarFile); buildContext.removeMessages(grammarFile); getLog().debug("Grammar file '" + grammarFile.getPath() + "' detected."); String relPathBase = MojoUtils.findSourceSubdir(sourceDirectory, grammarFile); String relPath = relPathBase + grammarFile.getName(); getLog().debug(" ... relative path is: " + relPath); grammarFileByFolder.map(relPathBase, grammarFile); } List> result = new ArrayList>(); for (Map.Entry> entry : grammarFileByFolder.entrySet()) { List folderArgs = new ArrayList(args); if (!folderArgs.contains("-package") && !entry.getKey().isEmpty()) { folderArgs.add("-package"); folderArgs.add(getPackageName(entry.getKey())); } for (File file : entry.getValue()) { folderArgs.add(entry.getKey() + file.getName()); } result.add(folderArgs); } return result; } private Set getImportFiles(File sourceDirectory) throws InclusionScanException { if (!libDirectory.exists()) return Collections.emptySet(); Set includes = new HashSet(); includes.add("*.g4"); includes.add("*.tokens"); SourceInclusionScanner scan = new SimpleSourceInclusionScanner(includes, Collections.emptySet()); scan.addSourceMapping(new SuffixMapping("G4", "g4")); return scan.getIncludedSources(libDirectory, null); } private Set getGrammarFiles(File sourceDirectory) throws InclusionScanException { // Which files under the source set should we be looking for as grammar files SourceMapping mapping = new SuffixMapping("g4", Collections.emptySet()); // What are the sets of includes (defaulted or otherwise). Set includes = getIncludesPatterns(); // Now, to the excludes, we need to add the imports directory // as this is autoscanned for imported grammars and so is auto-excluded from the // set of grammar fields we should be analyzing. excludes.add("imports/**"); SourceInclusionScanner scan = new SimpleSourceInclusionScanner(includes, excludes); scan.addSourceMapping(mapping); return scan.getIncludedSources(sourceDirectory, null); } private static String getPackageName(String relativeFolderPath) { if (relativeFolderPath.contains("..")) { throw new UnsupportedOperationException("Cannot handle relative paths containing '..'"); } List parts = new ArrayList(Arrays.asList(relativeFolderPath.split("[/\\\\\\.]+"))); while (parts.remove("")) { // intentionally blank } return Utils.join(parts.iterator(), "."); } public Set getIncludesPatterns() { if (includes == null || includes.isEmpty()) { return Collections.singleton("**/*.g4"); } return includes; } private File getDependenciesStatusFile() { File statusFile = new File(statusDirectory, "dependencies.ser"); if (!statusFile.getParentFile().exists()) { statusFile.getParentFile().mkdirs(); } return statusFile; } private final class CustomTool extends Tool { public CustomTool(String[] args) { super(args); addListener(new Antlr4ErrorLog(this, buildContext, getLog())); } @Override public void process(Grammar g, boolean gencode) { getLog().info("Processing grammar: " + g.fileName); super.process(g, gencode); } @Override public Writer getOutputFileWriter(Grammar g, String fileName) throws IOException { if (outputDirectory == null) { return new StringWriter(); } // output directory is a function of where the grammar file lives // for subdir/T.g4, you get subdir here. Well, depends on -o etc... // But, if this is a .tokens file, then we force the output to // be the base output directory (or current directory if there is not a -o) // File outputDir; if ( fileName.endsWith(CodeGenerator.VOCAB_FILE_EXTENSION) ) { outputDir = new File(outputDirectory); } else { outputDir = getOutputDirectory(g.fileName); } File outputFile = new File(outputDir, fileName); if (!outputDir.exists()) { outputDir.mkdirs(); } URI relativePath = project.getBasedir().toURI().relativize(outputFile.toURI()); getLog().debug(" Writing file: " + relativePath); OutputStream outputStream = buildContext.newFileOutputStream(outputFile); if ( outputEncoding!=null && !outputEncoding.isEmpty()) { return new BufferedWriter(new OutputStreamWriter(outputStream, outputEncoding)); } else { return new BufferedWriter(new OutputStreamWriter(outputStream)); } } } /** * Validates the given encoding. * * @return the validated encoding. If {@code null} was provided, returns the platform default encoding. */ private String validateEncoding(String encoding) { return (encoding == null) ? Charset.defaultCharset().name() : Charset.forName(encoding.trim()).name(); } } ================================================ FILE: antlr4-maven-plugin/src/main/java/org/antlr/mojo/antlr4/GrammarDependencies.java ================================================ /* * Copyright (c) 2012-2017 The ANTLR Project. All rights reserved. * Use of this file is governed by the BSD 3-clause license that * can be found in the LICENSE.txt file in the project root. */ package org.antlr.mojo.antlr4; import org.antlr.runtime.tree.Tree; import org.antlr.v4.Tool; import org.antlr.v4.misc.Graph; import org.antlr.v4.parse.ANTLRParser; import org.antlr.v4.tool.ast.GrammarAST; import org.antlr.v4.tool.ast.GrammarRootAST; import org.apache.maven.plugin.logging.Log; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.IOException; import java.io.ObjectInputStream; import java.io.ObjectOutputStream; import java.util.AbstractMap; import java.util.ArrayList; import java.util.Arrays; import java.util.Collection; import java.util.HashMap; import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Set; class GrammarDependencies { private final Graph graph = new Graph(); private final File sourceDirectory; private final File libDirectory; private final File statusFile; private final String packageName; /** Map grammars to their checksum and references. */ private final Map>> grammars; private final Log log; public GrammarDependencies(File sourceDirectory, File libDirectory, List arguments, File status, Log log) { this.log = log; this.sourceDirectory = sourceDirectory; this.libDirectory = libDirectory; this.statusFile = status; this.grammars = loadStatus(status); this.packageName = getPackage(arguments); } /** * Determines the package to use. * * @param arguments the tool arguments. * * @return the package. Returns {@code null} to indicate that no package should be * used. */ private String getPackage(List arguments) { int index = (arguments != null) ? arguments.indexOf("-package") : -1; return (index > -1) ? (arguments.get(index + 1).replace('.', File.separatorChar) + File.separatorChar) : null; } public void save() throws IOException { if (!grammars.isEmpty()) { log.debug("Persisting grammars dependency status: " + statusFile); ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream( statusFile)); try { out.writeObject(grammars); } finally { out.close(); } } } /** * Performs dependency analysis for the given grammar files. * * @param grammarFiles the grammar files. * @param importGrammarFiles the import grammar files. * @param tool the tool to use. * * @return self-reference. */ public GrammarDependencies analyze(Set grammarFiles, Set importGrammarFiles, Tool tool) throws IOException { log.debug("Analysing grammar dependencies " + sourceDirectory); // for dependency analysis we require all grammars Collection grammarsAndTokens = new HashSet(); grammarsAndTokens.addAll(importGrammarFiles); grammarsAndTokens.addAll(grammarFiles); for (File grammarFile : grammarsAndTokens) { // .tokens files must not be parsed, they can just be referenced if (!grammarFile.getName().endsWith(".tokens")) analyse(grammarFile, grammarsAndTokens, tool); } for (File grammarFile : grammarFiles) { Collection usages = findUsages(getRelativePath(grammarFile)); if (!usages.isEmpty()) { grammars.put(grammarFile, new AbstractMap.SimpleImmutableEntry>( MojoUtils.checksum(grammarFile), usages)); log.debug(" " + getRelativePath(grammarFile) + " used by " + usages); } } for (File grammarFile : importGrammarFiles) { // imported files are not allowed to be qualified Collection usages = findUsages(grammarFile.getName()); if (!usages.isEmpty()) { grammars.put(grammarFile, new AbstractMap.SimpleImmutableEntry>( MojoUtils.checksum(grammarFile), usages)); log.debug(" " + grammarFile.getName() + " imported by " + usages); } } return this; } /** * Determines whether a grammar used by the given grammar was modified since the last * build. * * @param grammarFile the grammar. * * @return {@code true} if a grammar used by the given grammar has been modified. */ public boolean isDependencyChanged(File grammarFile) throws IOException { String grammarPath = getRelativePath(grammarFile); for (Map.Entry>> e : grammars.entrySet()) { File depGrammarFile = e.getKey(); byte[] checksum = e.getValue().getKey(); Collection usages = e.getValue().getValue(); if (usages.contains(grammarPath)) { if (!depGrammarFile.exists() || !Arrays.equals(MojoUtils.checksum(depGrammarFile), checksum)) { log.debug(" " + grammarPath + ": dependency " + depGrammarFile.getName() + " changed"); return true; } } } return false; } /** * Determines the relative target path of the given grammar file. * * @param grammarFile the grammar file. * * @return the relative path. */ private String getRelativePath(File grammarFile) { // the library directory does not allow sub-directories if (grammarFile.getPath().startsWith(libDirectory.getPath())) return grammarFile.getName(); // if a package is given, we have to use it if (packageName != null) return packageName + grammarFile.getName(); // otherwise resolve the path relative to the source directory String path = MojoUtils.findSourceSubdir(sourceDirectory, grammarFile); return path + grammarFile.getName(); } /** * Returns the grammar file names that directly or indirectly use the given grammar. * * @param grammarFileName the grammar file name. * * @return the grammar file names that use the given grammar file. */ private Collection findUsages(String grammarFileName) { Collection result = new ArrayList(); explore(grammarFileName, result); return result; } private void explore(String grammarName, Collection result) { for (Graph.Node node : graph.getNode(grammarName).edges) { result.add(node.payload); explore(node.payload, result); } } private void analyse(File grammarFile, Collection grammarFiles, Tool tool) { GrammarRootAST grammar = tool.parseGrammar(grammarFile.getAbsolutePath()); if (grammar == null) return; for (GrammarAST importDecl : grammar.getAllChildrenWithType(ANTLRParser.IMPORT)) { for (Tree id: importDecl.getAllChildrenWithType(ANTLRParser.ID)) { // missing id is not valid, but we don't want to prevent the root cause from // being reported by the ANTLR tool if (id != null) { String grammarPath = getRelativePath(grammarFile); graph.addEdge(id.getText() + ".g4", grammarPath); } } } for (GrammarAST options : grammar.getAllChildrenWithType(ANTLRParser.OPTIONS)) { for (int i = 0, count = options.getChildCount(); i < count; i++) { Tree option = options.getChild(i); if (option.getType() == ANTLRParser.ASSIGN) { String key = option.getChild(0).getText(); String value = option.getChild(1).getText(); if ("tokenVocab".equals(key)) { String name = stripQuotes(value); // the grammar name may be qualified, but we resolve the path anyway String grammarName = stripPath(name); String grammarPath = MojoUtils.findSourceSubdir(sourceDirectory, grammarFile); File depGrammarFile = resolve(grammarName, grammarPath); // if a package has been given, we use it instead of the file directory path // (files probably reside in the root directory anyway with such a configuration ) if (packageName != null) grammarPath = packageName; graph.addEdge(getRelativePath(depGrammarFile), grammarPath + grammarFile.getName()); } } } } } /** * Resolves the given grammar name. * * @param name the name. * @param path the relative path. * * @return the grammar file. */ private File resolve(String name, String path) { File file = new File(sourceDirectory, path + name + ".g4"); if (file.exists()) return file; file = new File(libDirectory, name + ".g4"); if (file.exists()) return file; return new File(libDirectory, name + ".tokens"); } private Map>> loadStatus(File statusFile) { if (statusFile.exists()) { log.debug("Load grammars dependency status: " + statusFile); try { ObjectInputStream in = new ObjectInputStream(new FileInputStream( statusFile)); try { @SuppressWarnings("unchecked") Map>> data = (Map>>) in.readObject(); return data; } finally { in.close(); } } catch (Exception ex) { log.warn("Could not load grammar dependency status information", ex); } } return new HashMap>>(); } private String stripPath(String str) { return str.replaceAll("^.*[/\\\\]", ""); } private String stripQuotes(String str) { return str.replaceAll("\\A'|'\\Z", ""); } } ================================================ FILE: antlr4-maven-plugin/src/main/java/org/antlr/mojo/antlr4/MojoUtils.java ================================================ /* * Copyright (c) 2012-2017 The ANTLR Project. All rights reserved. * Use of this file is governed by the BSD 3-clause license that * can be found in the LICENSE.txt file in the project root. */ package org.antlr.mojo.antlr4; import java.io.File; import java.io.FileInputStream; import java.io.IOException; import java.io.InputStream; import java.security.MessageDigest; import java.security.NoSuchAlgorithmException; class MojoUtils { /** * Creates the MD5 checksum for the given file. * * @param file the file. * * @return the checksum. */ public static byte[] checksum(File file) throws IOException { try { InputStream in = new FileInputStream(file); byte[] buffer = new byte[2048]; MessageDigest complete = MessageDigest.getInstance("MD5"); try { int n; do { n = in.read(buffer); if (n > 0) { complete.update(buffer, 0, n); } } while (n != -1); } finally { in.close(); } return complete.digest(); } catch (NoSuchAlgorithmException ex) { throw new IOException("Could not create checksum " + file, ex); } } /** * Given the source directory File object and the full PATH to a grammar, produce the * path to the named grammar file in relative terms to the {@code sourceDirectory}. * This will then allow ANTLR to produce output relative to the base of the output * directory and reflect the input organization of the grammar files. * * @param sourceDirectory The source directory {@link File} object * @param grammarFileName The full path to the input grammar file * * @return The path to the grammar file relative to the source directory */ public static String findSourceSubdir(File sourceDirectory, File grammarFile) { String srcPath = sourceDirectory.getPath() + File.separator; String path = grammarFile.getPath(); if (!path.startsWith(srcPath)) { throw new IllegalArgumentException("expected " + path + " to be prefixed with " + sourceDirectory); } File unprefixedGrammarFileName = new File(path.substring(srcPath.length())); if (unprefixedGrammarFileName.getParent() == null) { return ""; } return unprefixedGrammarFileName.getParent() + File.separator; } } ================================================ FILE: antlr4-maven-plugin/src/site/apt/examples/import.apt ================================================ Imported Grammar Files In order to have the ANTLR plugin automatically locate and use grammars used as imports in your main <<<.g4>>> files, you need to place the imported grammar files in the <<>> directory beneath the root directory of your grammar files (which is <<>> by default of course). For a default layout, place your import grammars in the directory: <<>> ================================================ FILE: antlr4-maven-plugin/src/site/apt/examples/libraries.apt.vm ================================================ Libraries The introduction of the import directive in a grammar allows reuse of common grammar files as well as the ability to divide up functional components of large grammars. However it has caused some confusion in regard to the fact that generated vocabulary files (<<<*.tokens>>>) can also be searched for with the <<<>>> directive. This has confused two separate functions and imposes a structure upon the layout of your grammar files in certain cases. If you have grammars that both use the import directive and also require the use of a vocabulary file then you will need to locate the grammar that generates the <<<.tokens>>> file alongside the grammar that uses it. This is because you will need to use the <<<>>> directive to specify the location of your imported grammars and ANTLR will not find any vocabulary files in this directory. The <<<.tokens>>> files for any grammars are generated within the same output directory structure as the <<<.java>>> files. So, wherever the <<<.java>>> files are generated, you will also find the <<<.tokens>>> files. ANTLR looks for <<<.tokens>>> files in both the <<<>>> and the output directory where it is placing the generated <<<.java>>> files. Hence when you locate the grammars that generate <<<.tokens>>> files in the same source directory as the ones that use the <<<.tokens>>> files, then the Maven plugin will find the expected <<<.tokens>>> files. The <<<>>> is specified like any other directory parameter in Maven. Here is an example: +-- org.antlr antlr4-maven-plugin ${project.version} antlr4 src/main/antlr4_imports +-- ================================================ FILE: antlr4-maven-plugin/src/site/apt/examples/simple.apt.vm ================================================ Simple configuration If your grammar files are organized into the default locations as described in the {{{../index.html}introduction}}, then configuring the <<>> file for your project is as simple as adding this to it +-- org.antlr antlr4-maven-plugin ${project.version} antlr4 ... +-- When the <<>> command is executed all grammar files under <<>>, except any import grammars under <<>> will be analyzed and converted to Java source code in the output directory <<>>. Your input files under <<>> should be stored in sub directories that reflect the package structure of your java parsers. If your grammar file <<>> contains: +--- @header { package org.jimi.themuss; } +--- Then the <<<.g4>>> file should be stored in: <<>>. This way the generated <<<.java>>> files will correctly reflect the package structure in which they will finally rest as classes. ================================================ FILE: antlr4-maven-plugin/src/site/apt/faq.apt.vm ================================================ FAQ ================================================ FILE: antlr4-maven-plugin/src/site/apt/index.apt ================================================ ------------- ANTLR v4 Maven Plugin ------------- Jim Idle ------------- March 2009 ------------- ANTLR v4 Maven plugin The ANTLR v4 Maven plugin is completely re-written as of version 4.0; if you are familiar with prior versions, you should note that there are some behavioral differences that make it worthwhile reading this documentation. The job of the plugin is essentially to tell the standard ANTLR parser generator where the input grammar files are and where the output files should be generated. As with all Maven plugins, there are defaults, which you are advised to comply to, but are not forced to comply to. This version of the plugin allows full control over ANTLR and allows configuration of all options that are useful for a build system. The code required to calculate dependencies, check the build order, and otherwise work with your grammar files is built into the ANTLR tool as of version 4.0 of ANTLR and this plugin. * Plugin Versioning The plugin version tracks the version of the ANTLR tool that it controls. Hence if you use version 4.0 of the plugin, you will build your grammars using version 4.0 of the ANTLR tool, version 4.2 of the plugin will use version 4.2 of the ANTLR tool and so on. You may also find that there are patch versions of the plugin such as 4.0-1 4.0-2 and so on. Use the latest patch release of the plugin. The current version of the plugin is shown at the top of this page after the <> date. * Default directories As with all Maven plugins, this plugin will automatically default to standard locations for your grammar and import files. Organizing your source code to reflect this standard layout will greatly reduce the configuration effort required. The standard layout lookd like this: +-- src/main/ | +--- antlr4/... .g4 files organized in the required package structure | +--- imports/ .g4 files that are imported by other grammars. +-- If your grammar is intended to be part of a package called <<>> then you would place it in the directory <<>>. The plugin will then produce <<<.java>>> and <<<.tokens>>> files in the output directory <<>> When the Java files are compiled they will be in the correct location for the Javac compiler without any special configuration. The generated java files are automatically submitted for compilation by the plugin. The <<>> directory is treated in a special way. It should contain any grammar files that are imported by other grammar files (do not make subdirectories here.) Such files are never built on their own, but the plugin will automatically tell the ANTLR tool to look in this directory for library files. ================================================ FILE: antlr4-maven-plugin/src/site/apt/usage.apt.vm ================================================ Usage The ANTLR 4 plugin for Maven can generate parsers for any number of grammars in your project. * Compiling Grammars into Parsers By default, the <<<{{{./antlr4-mojo.html}antlr4}}>>> goal will search for grammar files in the directory <<<$\{basedir\}/src/main/antlr4>>> and any additional <<<.tokens>>> files in the directory <<<$\{basedir\}/src/main/antlr4/imports>>>. This can be configured to search other directories using the plugin configuration parameters as described in the <<<{{{./antlr4-mojo.html}antlr4}}>>> goal documentation. The following figure shows the expected layout of files for the default configuration of this plugin. +-- src/main/ | +--- antlr4/... .g4 files organized in the required package structure | +--- imports/ user-created .tokens files and .g4 files that are imported by other grammars +-- The next step is to configure your POM to call the plugin. The goals will normally run during the generate-sources phase of the build. Examples of how to configure your POM can be found on the various examples pages, reachable via the page menu. If you stick with the default values, the snippet below will suffice: +-- ... org.antlr antlr4-maven-plugin ${project.version} antlr antlr4 ... ... +-- Note that you can create multiple executions, and thus build some grammars with different options to others (such as setting the <<>> option for instance). ================================================ FILE: antlr4-maven-plugin/src/site/site.xml ================================================ ================================================ FILE: antlr4-maven-plugin/src/test/java/org/antlr/mojo/antlr4/Antlr4MojoTest.java ================================================ /* * Copyright (c) 2012-2017 The ANTLR Project. All rights reserved. * Use of this file is governed by the BSD 3-clause license that * can be found in the LICENSE.txt file in the project root. */ package org.antlr.mojo.antlr4; import io.takari.maven.testing.TestMavenRuntime; import io.takari.maven.testing.TestResources; import org.apache.maven.execution.MavenSession; import org.apache.maven.plugin.MojoExecution; import org.apache.maven.plugin.MojoExecutionException; import org.apache.maven.project.MavenProject; import org.codehaus.plexus.util.xml.Xpp3Dom; import org.junit.Rule; import org.junit.Test; import org.junit.rules.ExpectedException; import java.io.IOException; import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; import java.util.Arrays; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; public class Antlr4MojoTest { @Rule public ExpectedException thrown = ExpectedException.none(); @Rule public final TestResources resources = new TestResources(); @Rule public final TestMavenRuntime maven = new TestMavenRuntime(); @Test public void importTokens() throws Exception { Path baseDir = resources.getBasedir("importTokens").toPath(); Path antlrDir = baseDir.resolve("src/main/antlr4"); Path generatedSources = baseDir.resolve("target/generated-sources/antlr4"); Path genParser = generatedSources.resolve("test/SimpleParser.java"); Path tokens = antlrDir.resolve("imports/SimpleLexer.tokens"); MavenProject project = maven.readMavenProject(baseDir.toFile()); MavenSession session = maven.newMavenSession(project); MojoExecution exec = maven.newMojoExecution("antlr4"); //////////////////////////////////////////////////////////////////////// // 1st - all grammars have to be processed //////////////////////////////////////////////////////////////////////// assertFalse(Files.exists(genParser)); maven.executeMojo(session, project, exec); assertTrue(Files.exists(genParser)); //////////////////////////////////////////////////////////////////////// // 2nd - nothing has been modified, no grammars have to be processed //////////////////////////////////////////////////////////////////////// { byte[] sum = checksum(genParser); maven.executeMojo(session, project, exec); assertTrue(Arrays.equals(sum, checksum(genParser))); } //////////////////////////////////////////////////////////////////////// // 3rd - the imported grammar changed, every dependency has to be processed //////////////////////////////////////////////////////////////////////// try(Change change = Change.of(tokens, "DOT=4")) { byte[] sum = checksum(genParser); maven.executeMojo(session, project, exec); assertFalse(Arrays.equals(sum, checksum(genParser))); } } @Test public void importsCustomLayout() throws Exception { Path baseDir = resources.getBasedir("importsCustom").toPath(); Path antlrDir = baseDir.resolve("src/main/antlr4"); Path generatedSources = baseDir.resolve("src/main/java"); Path genTestLexer = generatedSources.resolve("foo/TestLexer.java"); Path genTestParser = generatedSources.resolve("foo/TestParser.java"); Path genHello = generatedSources.resolve("foo/HelloParser.java"); Path baseGrammar = antlrDir.resolve("imports/TestBaseLexer.g4"); Path lexerGrammar = antlrDir.resolve("TestLexer.g4"); Path parserGrammar = antlrDir.resolve("TestParser.g4"); Xpp3Dom outputDirectory = TestMavenRuntime.newParameter("outputDirectory", "src/main/java/foo"); Xpp3Dom arguments = new Xpp3Dom("arguments"); arguments.addChild(TestMavenRuntime.newParameter("argument", "-package")); arguments.addChild(TestMavenRuntime.newParameter("argument", "foo")); MavenProject project = maven.readMavenProject(baseDir.toFile()); MavenSession session = maven.newMavenSession(project); MojoExecution exec = maven.newMojoExecution("antlr4", outputDirectory, arguments); //////////////////////////////////////////////////////////////////////// // 1st - all grammars have to be processed //////////////////////////////////////////////////////////////////////// assertFalse(Files.exists(genHello)); assertFalse(Files.exists(genTestParser)); assertFalse(Files.exists(genTestLexer)); maven.executeMojo(session, project, exec); assertTrue(Files.exists(genHello)); assertTrue(Files.exists(genTestParser)); assertTrue(Files.exists(genTestLexer)); //////////////////////////////////////////////////////////////////////// // 2nd - nothing has been modified, no grammars have to be processed //////////////////////////////////////////////////////////////////////// { byte[] testLexerSum = checksum(genTestLexer); byte[] testParserSum = checksum(genTestParser); byte[] helloSum = checksum(genHello); maven.executeMojo(session, project, exec); assertTrue(Arrays.equals(testLexerSum, checksum(genTestLexer))); assertTrue(Arrays.equals(testParserSum, checksum(genTestParser))); assertTrue(Arrays.equals(helloSum, checksum(genHello))); } //////////////////////////////////////////////////////////////////////// // 3rd - the imported grammar changed, every dependency has to be processed //////////////////////////////////////////////////////////////////////// // modify the grammar to make checksum comparison detect a change try(Change change = Change.of(baseGrammar, "DOT: '.' ;")) { byte[] testLexerSum = checksum(genTestLexer); byte[] testParserSum = checksum(genTestParser); byte[] helloSum = checksum(genHello); maven.executeMojo(session, project, exec); assertFalse(Arrays.equals(testLexerSum, checksum(genTestLexer))); assertFalse(Arrays.equals(testParserSum, checksum(genTestParser))); assertTrue(Arrays.equals(helloSum, checksum(genHello))); } //////////////////////////////////////////////////////////////////////// // 4th - the lexer grammar changed, the parser grammar has to be processed as well //////////////////////////////////////////////////////////////////////// // modify the grammar to make checksum comparison detect a change try(Change change = Change.of(lexerGrammar, "fragment DOT : '.';")) { byte[] testLexerSum = checksum(genTestLexer); byte[] testParserSum = checksum(genTestParser); byte[] helloSum = checksum(genHello); maven.executeMojo(session, project, exec); assertFalse(Arrays.equals(testLexerSum, checksum(genTestLexer))); assertFalse(Arrays.equals(testParserSum, checksum(genTestParser))); assertTrue(Arrays.equals(helloSum, checksum(genHello))); } //////////////////////////////////////////////////////////////////////// // 5th - the parser grammar changed, no other grammars have to be processed //////////////////////////////////////////////////////////////////////// // modify the grammar to make checksum comparison detect a change try(Change change = Change.of(parserGrammar, " t : WS* ;")) { byte[] testLexerSum = checksum(genTestLexer); byte[] testParserSum = checksum(genTestParser); byte[] helloSum = checksum(genHello); maven.executeMojo(session, project, exec); assertTrue(Arrays.equals(testLexerSum, checksum(genTestLexer))); assertFalse(Arrays.equals(testParserSum, checksum(genTestParser))); assertTrue(Arrays.equals(helloSum, checksum(genHello))); } } @Test public void importsStandardLayout() throws Exception { Path baseDir = resources.getBasedir("importsStandard").toPath(); Path antlrDir = baseDir.resolve("src/main/antlr4"); Path generatedSources = baseDir.resolve("target/generated-sources/antlr4"); Path genTestLexer = generatedSources.resolve("test/TestLexer.java"); Path genTestParser = generatedSources.resolve("test/TestParser.java"); Path genHello = generatedSources.resolve("test/HelloParser.java"); Path baseGrammar = antlrDir.resolve("imports/TestBaseLexer.g4"); Path baseGrammar2 = antlrDir.resolve("imports/TestBaseLexer2.g4"); Path lexerGrammar = antlrDir.resolve("test/TestLexer.g4"); Path parserGrammar = antlrDir.resolve("test/TestParser.g4"); MavenProject project = maven.readMavenProject(baseDir.toFile()); MavenSession session = maven.newMavenSession(project); MojoExecution exec = maven.newMojoExecution("antlr4"); //////////////////////////////////////////////////////////////////////// // 1st - all grammars have to be processed //////////////////////////////////////////////////////////////////////// assertFalse(Files.exists(genHello)); assertFalse(Files.exists(genTestParser)); assertFalse(Files.exists(genTestLexer)); maven.executeMojo(session, project, exec); assertTrue(Files.exists(genHello)); assertTrue(Files.exists(genTestParser)); assertTrue(Files.exists(genTestLexer)); byte[] origTestLexerSum = checksum(genTestLexer); byte[] origTestParserSum = checksum(genTestParser); byte[] origHelloSum = checksum(genHello); //////////////////////////////////////////////////////////////////////// // 2nd - nothing has been modified, no grammars have to be processed //////////////////////////////////////////////////////////////////////// { maven.executeMojo(session, project, exec); assertTrue(Arrays.equals(origTestLexerSum, checksum(genTestLexer))); assertTrue(Arrays.equals(origTestParserSum, checksum(genTestParser))); assertTrue(Arrays.equals(origHelloSum, checksum(genHello))); } //////////////////////////////////////////////////////////////////////// // 3rd - the imported grammar changed, every dependency has to be processed //////////////////////////////////////////////////////////////////////// // modify the grammar to make checksum comparison detect a change try(Change change = Change.of(baseGrammar, "DOT: '.' ;")) { maven.executeMojo(session, project, exec); assertFalse(Arrays.equals(origTestLexerSum, checksum(genTestLexer))); assertFalse(Arrays.equals(origTestParserSum, checksum(genTestParser))); assertTrue(Arrays.equals(origHelloSum, checksum(genHello))); } // Restore file and confirm it was restored. maven.executeMojo(session, project, exec); assertTrue(Arrays.equals(origTestLexerSum, checksum(genTestLexer))); assertTrue(Arrays.equals(origTestParserSum, checksum(genTestParser))); assertTrue(Arrays.equals(origHelloSum, checksum(genHello))); //////////////////////////////////////////////////////////////////////// // 4th - the second imported grammar changed, every dependency has to be processed //////////////////////////////////////////////////////////////////////// // modify the grammar to make checksum comparison detect a change try(Change change = Change.of(baseGrammar2, "BANG: '!' ;")) { maven.executeMojo(session, project, exec); assertFalse(Arrays.equals(origTestLexerSum, checksum(genTestLexer))); assertFalse(Arrays.equals(origTestParserSum, checksum(genTestParser))); assertTrue(Arrays.equals(origHelloSum, checksum(genHello))); } // Restore file and confirm it was restored. maven.executeMojo(session, project, exec); assertTrue(Arrays.equals(origTestLexerSum, checksum(genTestLexer))); assertTrue(Arrays.equals(origTestParserSum, checksum(genTestParser))); assertTrue(Arrays.equals(origHelloSum, checksum(genHello))); //////////////////////////////////////////////////////////////////////// // 5th - the lexer grammar changed, the parser grammar has to be processed as well //////////////////////////////////////////////////////////////////////// // modify the grammar to make checksum comparison detect a change try(Change change = Change.of(lexerGrammar, "FOO: 'foo' ;")) { maven.executeMojo(session, project, exec); assertFalse(Arrays.equals(origTestLexerSum, checksum(genTestLexer))); assertFalse(Arrays.equals(origTestParserSum, checksum(genTestParser))); assertTrue(Arrays.equals(origHelloSum, checksum(genHello))); } // Restore file and confirm it was restored. maven.executeMojo(session, project, exec); assertTrue(Arrays.equals(origTestLexerSum, checksum(genTestLexer))); assertTrue(Arrays.equals(origTestParserSum, checksum(genTestParser))); assertTrue(Arrays.equals(origHelloSum, checksum(genHello))); //////////////////////////////////////////////////////////////////////// // 6th - the parser grammar changed, no other grammars have to be processed //////////////////////////////////////////////////////////////////////// // modify the grammar to make checksum comparison detect a change try(Change change = Change.of(parserGrammar, " t : WS* ;")) { maven.executeMojo(session, project, exec); assertTrue(Arrays.equals(origTestLexerSum, checksum(genTestLexer))); assertFalse(Arrays.equals(origTestParserSum, checksum(genTestParser))); assertTrue(Arrays.equals(origHelloSum, checksum(genHello))); } // Restore file and confirm it was restored. maven.executeMojo(session, project, exec); assertTrue(Arrays.equals(origTestLexerSum, checksum(genTestLexer))); assertTrue(Arrays.equals(origTestParserSum, checksum(genTestParser))); assertTrue(Arrays.equals(origHelloSum, checksum(genHello))); } @Test public void processWhenDependencyRemoved() throws Exception { Path baseDir = resources.getBasedir("dependencyRemoved").toPath(); Path antlrDir = baseDir.resolve("src/main/antlr4"); Path baseGrammar = antlrDir.resolve("imports/HelloBase.g4"); MavenProject project = maven.readMavenProject(baseDir.toFile()); MavenSession session = maven.newMavenSession(project); MojoExecution exec = maven.newMojoExecution("antlr4"); maven.executeMojo(session, project, exec); try(Change temp = Change.of(baseGrammar)) { // if the base grammar no longer exists, processing must be performed Files.delete(baseGrammar); thrown.expect(MojoExecutionException.class); thrown.expectMessage("ANTLR 4 caught 1 build errors."); maven.executeMojo(session, project, exec); } } private byte[] checksum(Path path) throws IOException { return MojoUtils.checksum(path.toFile()); } private static class Change implements AutoCloseable { final Path file; final byte[] original; public Change(Path file, String change) { this.file = file; try { original = Files.readAllBytes(file); } catch (IOException ex) { throw new RuntimeException("Could not read file " + file); } String text = new String(original, StandardCharsets.UTF_8) + change; write(file, text.getBytes(StandardCharsets.UTF_8)); } private void write(Path file, byte[] data) { try { Files.write(file, data); } catch (IOException ex) { throw new RuntimeException("Could not write file " + file); } } public static Change of(Path file, String change) { return new Change(file, change); } public static Change of(Path file) { return new Change(file, "\n"); } @Override public void close() { write(file, original); } } } ================================================ FILE: antlr4-maven-plugin/src/test/projects/dependencyRemoved/pom.xml ================================================ 4.0.0 deps.removed depRemoved 1.0-SNAPSHOT jar Test processing after dependency removed junit junit 4.13.1 test antlr4-maven-plugin ================================================ FILE: antlr4-maven-plugin/src/test/projects/dependencyRemoved/src/main/antlr4/imports/HelloBase.g4 ================================================ lexer grammar TestBaseLexer; tokens { Name } // Default "mode": Everything OUTSIDE of a tag Comment : '' ; CDSect : '' ; fragment Whitespace : ' ' | '\n' | '\t' | '\r' ; fragment Hexdigit : [a-fA-F0-9] ; fragment Digit : [0-9] ; ================================================ FILE: antlr4-maven-plugin/src/test/projects/dependencyRemoved/src/main/antlr4/test/Hello.g4 ================================================ grammar Hello; import HelloBase; r : 'hello' ID ; ID : [a-z]+ ; WS : [ \r\t\n]+ -> skip ; ================================================ FILE: antlr4-maven-plugin/src/test/projects/importTokens/pom.xml ================================================ 4.0.0 import.tokens importTokens 1.0-SNAPSHOT jar Test importing tokens file junit junit 4.13.1 test antlr4-maven-plugin ================================================ FILE: antlr4-maven-plugin/src/test/projects/importTokens/src/main/antlr4/imports/SimpleLexer.tokens ================================================ ID=1 INT=2 SEMI=3 ================================================ FILE: antlr4-maven-plugin/src/test/projects/importTokens/src/main/antlr4/test/SimpleParser.g4 ================================================ parser grammar SimpleParser; options { // get token types from SimpleLexer.tokens; don't name it // SimpleParser.tokens as ANTLR will overwrite! tokenVocab=SimpleLexer; } s : ( ID | INT )* SEMI ; ================================================ FILE: antlr4-maven-plugin/src/test/projects/importsCustom/pom.xml ================================================ 4.0.0 imports.custom importsCustom 1.0-SNAPSHOT jar Test importing, custom layout junit junit 4.13.1 test antlr4-maven-plugin ${basedir}/src/main/java/com/foo -visitor -no-listener -Xlog -package com.foo antlr4 ================================================ FILE: antlr4-maven-plugin/src/test/projects/importsCustom/src/main/antlr4/Hello.g4 ================================================ grammar Hello; r : 'hello' ID ; ID : [a-z]+ ; WS : [ \r\t\n]+ -> skip ; ================================================ FILE: antlr4-maven-plugin/src/test/projects/importsCustom/src/main/antlr4/TestLexer.g4 ================================================ lexer grammar TestLexer; import TestBaseLexer; WS : Whitespace+ -> skip; TEXT : ~[<&]+ ; // match any 16 bit char other than < and & ================================================ FILE: antlr4-maven-plugin/src/test/projects/importsCustom/src/main/antlr4/TestParser.g4 ================================================ parser grammar TestParser; options { tokenVocab=TestLexer; } document : (Comment | Name) EOF ; ================================================ FILE: antlr4-maven-plugin/src/test/projects/importsCustom/src/main/antlr4/imports/TestBaseLexer.g4 ================================================ lexer grammar TestBaseLexer; tokens { Name } // Default "mode": Everything OUTSIDE of a tag Comment : '' ; CDSect : '' ; fragment Whitespace : ' ' | '\n' | '\t' | '\r' ; fragment Hexdigit : [a-fA-F0-9] ; fragment Digit : [0-9] ; ================================================ FILE: antlr4-maven-plugin/src/test/projects/importsStandard/pom.xml ================================================ 4.0.0 imports.standard importsStandard 1.0-SNAPSHOT jar Test importing, standard layout junit junit 4.13.1 test antlr4-maven-plugin ================================================ FILE: antlr4-maven-plugin/src/test/projects/importsStandard/src/main/antlr4/imports/TestBaseLexer.g4 ================================================ lexer grammar TestBaseLexer; tokens { Name } // Default "mode": Everything OUTSIDE of a tag Comment : '' ; CDSect : '' ; fragment Whitespace : ' ' | '\n' | '\t' | '\r' ; fragment Hexdigit : [a-fA-F0-9] ; ================================================ FILE: antlr4-maven-plugin/src/test/projects/importsStandard/src/main/antlr4/imports/TestBaseLexer2.g4 ================================================ lexer grammar TestBaseLexer2; fragment Digit : [0-9] ; ================================================ FILE: antlr4-maven-plugin/src/test/projects/importsStandard/src/main/antlr4/test/Hello.g4 ================================================ grammar Hello; r : 'hello' ID ; ID : [a-z]+ ; WS : [ \r\t\n]+ -> skip ; ================================================ FILE: antlr4-maven-plugin/src/test/projects/importsStandard/src/main/antlr4/test/TestLexer.g4 ================================================ lexer grammar TestLexer; import TestBaseLexer, TestBaseLexer2; WS : Whitespace+ -> skip; TEXT : ~[<&]+ ; // match any 16 bit char other than < and & ================================================ FILE: antlr4-maven-plugin/src/test/projects/importsStandard/src/main/antlr4/test/TestParser.g4 ================================================ parser grammar TestParser; options { tokenVocab=TestLexer; } document : (Comment | Name) EOF ; ================================================ FILE: developer-cert-of-origin.txt ================================================ As of 4.10, ANTLR uses the Linux Foundation's Developer Certificate of Origin, DCO, version 1.1. See either https://developercertificate.org/ or the text below. Each commit requires a "signature", which is simple as using `-s` (not `-S`) to the git commit command: git commit -s -m 'This is my commit message' Github's pull request process enforces the sig and gives instructions on how to fix any commits that lack the sig. See https://github.com/apps/dco for more info. No signature is required in this file (unlike the previous ANTLR contributor's certificate of origin.) ----- https://developercertificate.org/ ------ Developer Certificate of Origin Version 1.1 Copyright (C) 2004, 2006 The Linux Foundation and its contributors. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Developer's Certificate of Origin 1.1 By making a contribution to this project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. ================================================ FILE: doc/IDEs.md ================================================ # Integrating ANTLR into Development Systems The Java target is the reference implementation mirrored by other targets. The following pages help you integrate ANTLR into development environments and build systems appropriate for your target language. As of December 2016, we have Java, C#, Python 3, JavaScript, Go, C++, and Swift targets. The easiest thing is probably just to use an [ANTLR plug-in](http://www.antlr.org/tools.html) for your favorite development environment. ================================================ FILE: doc/ace-javascript-target.md ================================================ # Integrating ANTLR JavaScript parsers with ACE editor Having the ability to parse code other than JavaScript is great, but nowadays users expect to be able to edit code with nice edit features such as keyword highlighting, indentation and brace matching, and advanced ones such as syntax checking. I have been through the process of integrating an ANTLR parser with ACE, the dominant code editor for web based code editing. Information about ACE can be found on their web site. This page describes my experience, and humbly aims to help you get started. It is not however a reference guide, and no support is provided. ## Architecture The ACE editor is organized as follows 1. The editor itself is a
which once initialized comprises a number of elements. This UI element is responsible for the display, and the generation of edit events. 1. The editor relies on a Session, which manages events and configuration. 1. The code itself is stored in a Document. Any insertion or deletion of text is reflected in the Document. 1. Keyword highlighting, indentation and brace matching are delegated to a mode. There is no direct equivalent of an ACE mode in ANTLR. While keywords are the equivalent of ANTLR lexer tokens, indentation and brace matching are edit tasks, not parsing ones. A given ACE editor can only have one mode, which corresponds to the language being edited. There is no need for ANTLR integration to support keyword highlighting, indentation and brace matching. 1. Syntax checking is delegated to a worker. This is where ANTLR integration is needed. If syntax checking is enabled, ACE asks the mode to create a worker. In JavaScript, workers run in complete isolation i.e. they don't share code or variables with other workers, or with the HTML page itself. 1. The below diagram describes how the whole system works. In green are the components *you* need to provide. You'll notice that there is no need to load ANTLR in the HTML page itself. You'll also notice that ACE maintains a document in each thread. This is done through low level events sent by the ACE session to the worker which describe the delta. Once applied to the worker document, a high level event is triggered, which is easy to handle since at this point the worker document is a perfect copy of the UI document. ## Step-by-step guide The first thing to do is to create an editor in your html page. This is thoroughly described in the ACE documentation, so we'll just sum it up here: ```xml ``` This should give you a working editor. You may want to control its sizing using CSS. I personally load the editor in an iframe and set its style to position: absolute, top: 0, left: 0 etc... but I'm sure you know better than me how to achieve results. The second thing to do is to configure the ACE editor to use your mode i.e. language configuration. A good place to start is to inherit from the built-in TextMode. The following is a very simple example, which only caters for comments, literals, and a limited subset of separators and keywords : ```javascript ace.define('ace/mode/my-mode',["require","exports","module","ace/lib/oop","ace/mode/text","ace/mode/text_highlight_rules", "ace/worker/worker_client" ], function(require, exports, module) { var oop = require("ace/lib/oop"); var TextMode = require("ace/mode/text").Mode; var TextHighlightRules = require("ace/mode/text_highlight_rules").TextHighlightRules; var MyHighlightRules = function() { var keywordMapper = this.createKeywordMapper({ "keyword.control": "if|then|else", "keyword.operator": "and|or|not", "keyword.other": "class", "storage.type": "int|float|text", "storage.modifier": "private|public", "support.function": "print|sort", "constant.language": "true|false" }, "identifier"); this.$rules = { "start": [ { token : "comment", regex : "//" }, { token : "string", regex : '["](?:(?:\\\\.)|(?:[^"\\\\]))*?["]' }, { token : "constant.numeric", regex : "0[xX][0-9a-fA-F]+\\b" }, { token : "constant.numeric", regex: "[+-]?\\d+(?:(?:\\.\\d*)?(?:[eE][+-]?\\d+)?)?\\b" }, { token : "keyword.operator", regex : "!|%|\\\\|/|\\*|\\-|\\+|~=|==|<>|!=|<=|>=|=|<|>|&&|\\|\\|" }, { token : "punctuation.operator", regex : "\\?|\\:|\\,|\\;|\\." }, { token : "paren.lparen", regex : "[[({]" }, { token : "paren.rparen", regex : "[\\])}]" }, { token : "text", regex : "\\s+" }, { token: keywordMapper, regex: "[a-zA-Z_$][a-zA-Z0-9_$]*\\b" } ] }; }; oop.inherits(MyHighlightRules, TextHighlightRules); var MyMode = function() { this.HighlightRules = MyHighlightRules; }; oop.inherits(MyMode, TextMode); (function() { this.$id = "ace/mode/my-mode"; }).call(MyMode.prototype); exports.Mode = MyMode; }); ``` Now if you store the above in a file called "my-mode.js", setting the ACE Editor becomes straightforward: ```xml ``` At this point you should have a working editor, able to highlight keywords. You may wonder why you need to set the tokens when you have already done so in your ANTLR lexer grammar. First, ACE expects a classification (control, operator, type...) which does not exist in ANTLR. Second, there is no need for ANTLR to achieve this, since ACE comes with its own lexer. Ok, now that we have a working editor comes the time where we need syntax validation. This is where the worker comes in the picture. Creating the worker is the responsibility of the mode you provide. So you need to enhance it with something like the following: ```javascript var WorkerClient = require("ace/worker/worker_client").WorkerClient; this.createWorker = function(session) { this.$worker = new WorkerClient(["ace"], "ace/worker/my-worker", "MyWorker", "../js/my-worker.js"); this.$worker.attachToDocument(session.getDocument()); this.$worker.on("errors", function(e) { session.setAnnotations(e.data); }); this.$worker.on("annotate", function(e) { session.setAnnotations(e.data); }); this.$worker.on("terminate", function() { session.clearAnnotations(); }); return this.$worker; }; ``` The above code needs to be placed in the existing worker, after: ```javascript this.$id = "ace/mode/my-mode"; ``` Please note that the mode code runs on the UI side, not the worker side. The event handlers here are for events sent by the worker, not to the worker. Obviously the above won't work out of the box, because you need to provide the "my-worker.js" file. Creating a worker from scratch is not something I've tried. Simply put, your worker needs to handle all messages sent by ACE using the WorkerClient created by the mode. This is not a simple task, and is better delegated to existing ACE code, so we can focus on tasks specific to our language. What I did is I started from "mode-json.js", a rather simple worker which comes with ACE, stripped out all JSON validation related stuff out of it, and saved the remaining code in a file name "worker-base.js" which you can find [here](resources/worker-base.js). Once this done, I was able to create a simple worker, as follows: ```javascript importScripts("worker-base.js"); ace.define('ace/worker/my-worker',["require","exports","module","ace/lib/oop","ace/worker/mirror"], function(require, exports, module) { "use strict"; var oop = require("ace/lib/oop"); var Mirror = require("ace/worker/mirror").Mirror; var MyWorker = function(sender) { Mirror.call(this, sender); this.setTimeout(200); this.$dialect = null; }; oop.inherits(MyWorker, Mirror); (function() { this.onUpdate = function() { var value = this.doc.getValue(); var annotations = validate(value); this.sender.emit("annotate", annotations); }; }).call(MyWorker.prototype); exports.MyWorker = MyWorker; }); var validate = function(input) { return [ { row: 0, column: 0, text: "MyMode says Hello!", type: "error" } ]; }; ``` At this point, you should have an editor which displays an error icon next to the first line. When you hover over the error icon, it should display: MyMode says Hello!. Is that not a friendly worker? Yum. What remains to be done is have our validate function actually validate the input. Finally ANTLR comes in the picture! To start with, let's load ANTLR and your parser, listener etc.. The preferred approach for loading parser code is to bundle your parser, [as described here](javascript-target.md). You can then load it as part of the importScripts instruction at the start of your worker code. Another approach is to load it using 'require'. Easy, since you could write: ```js var antlr4 = require('antlr4/index'); ``` This may work, but it's actually unreliable. The reason is that the 'require' function that comes with ACE uses a different syntax than the 'require' function used by ANTLR, which follows the NodeJS 'require' convention. So we need to bring in a NodeJS compatible 'require' function that conforms to the NodeJS syntax. I personally use one that comes from Torben Haase's Honey project, which you can find in li/require.js. But hey, now we're going to have 2 'require' functions not compatible with each other! Indeed, this is why you need to take special care, as follows: ```js // load nodejs compatible require var ace_require = require; require = undefined; var Honey = { 'requirePath': ['..'] }; // walk up to js folder, see Honey docs importScripts("../lib/require.js"); var antlr4_require = require; require = ace_require; ``` Now it's safe to load antlr and the parsers generated for your language. Assuming that your language files (generated or hand-built) are in a folder with an index.js file that calls require for each file, your parser loading code can be as simple as follows: ```js // load antlr4 and myLanguage var antlr4, mylanguage; try { require = antlr4_require; antlr4 = require('antlr4/index'); mylanguage = require('mylanguage/index'); } finally { require = ace_require; } ``` Please note the try-finally construct. ANTLR uses 'require' synchronously so it's perfectly safe to ignore the ACE 'require' while running ANTLR code. ACE itself does not guarantee synchronous execution, so you are much safer always switching 'require' back to 'ace_require'. Now detecting deep syntax errors in your code is a task for your ANTLR listener or visitor or whatever piece of code you've delegated this to. We're not going to describe this here, since it would require some knowledge of your language. However, detecting grammar syntax errors is something ANTLR does beautifully (isn't that why you went for ANTLR in the first place?). So what we will illustrate here is how to report grammar syntax errors. I have no doubt that from there, you will be able to extend the validator to suit your specific needs. Whenever ANTLR encounters an unexpected token, it fires an error. By default, the error is routed to an error listener which simply writes to the console. What we need to do is replace this listener by our own listener, se we can route errors to the ACE editor. First, let's create such a listener: ```js // class for gathering errors and posting them to ACE editor var AnnotatingErrorListener = function(annotations) { antlr4.error.ErrorListener.call(this); this.annotations = annotations; return this; }; AnnotatingErrorListener.prototype = Object.create(antlr4.error.ErrorListener.prototype); AnnotatingErrorListener.prototype.constructor = AnnotatingErrorListener; AnnotatingErrorListener.prototype.syntaxError = function(recognizer, offendingSymbol, line, column, msg, e) { this.annotations.push({ row: line - 1, column: column, text: msg, type: "error" }); }; ``` With this, all that remains to be done is plug the listener in when we parse the code. Here is how I do it: ```js var validate = function(input) { var stream = CharStreams.fromString(input); var lexer = new mylanguage.MyLexer(stream); var tokens = new antlr4.CommonTokenStream(lexer); var parser = new mylanguage.MyParser(tokens); var annotations = []; var listener = new AnnotatingErrorListener(annotations) parser.removeErrorListeners(); parser.addErrorListener(listener); parser.parseMyRule(); return annotations; }; ``` You know what? That's it! You now have an ACE editor that does syntax validation using ANTLR! I hope you find this useful, and simple enough to get started. Now wait, hey! How do you debug this? Well, as usual, using Chrome, since no other browser is able to debug worker code. What a shame... ================================================ FILE: doc/actions.md ================================================ # Actions and Attributes In Chapter 10, Attributes and Actions, we learned how to embed actions within grammars and looked at the most common token and rule attributes. This section summarizes the important syntax and semantics from that chapter and provides a complete list of all available attributes. (You can learn more about actions in the grammar from the free excerpt on listeners and actions.) Actions are blocks of text written in the target language and enclosed in curly braces. The recognizer triggers them according to their locations within the grammar. For example, the following rule emits "found a decl" after the parser has seen a valid declaration: ``` decl: type ID ';' {System.out.println("found a decl");} ; type: 'int' | 'float' ; ``` Most often, actions access the attributes of tokens and rule references: ``` decl: type ID ';' {System.out.println("var "+$ID.text+":"+$type.text+";");} | t=ID id=ID ';' {System.out.println("var "+$id.text+":"+$t.text+";");} ; ``` ## Token Attributes All tokens have a collection of predefined, read-only attributes. The attributes include useful token properties such as the token type and text matched for a token. Actions can access these attributes via `$label.attribute` where label labels a particular instance of a token reference (`a` and `b` in the example below are used in the action code as `$a` and `$b`). Often, a particular token is only referenced once in the rule, in which case the token name itself can be used unambiguously in the action code (token `INT` can be used as `$INT` in the action). The following example illustrates token attribute expression syntax: ``` r : INT {int x = $INT.line;} ( ID {if ($INT.line == $ID.line) ...;} )? a=FLOAT b=FLOAT {if ($a.line == $b.line) ...;} ; ``` The action within the `(...)?` subrule can see the `INT` token matched before it in the outer level. Because there are two references to the `FLOAT` token, a reference to `$FLOAT` in an action is not unique; you must use labels to specify which token reference you’re interested in. Token references within different alternatives are unique because only one of them can be matched for any invocation of the rule. For example, in the following rule, actions in both alternatives can reference `$ID` directly without using a label: ``` r : ... ID {System.out.println($ID.text);} | ... ID {System.out.println($ID.text);} ; ``` To access the tokens matched for literals, you must use a label: ``` stat: r='return' expr ';' {System.out.println("line="+$r.line);} ; ``` Most of the time you access the attributes of the token, but sometimes it is useful to access the Token object itself because it aggregates all the attributes. Further, you can use it to test whether an optional subrule matched a token: ``` stat: 'if' expr 'then' stat (el='else' stat)? {if ( $el!=null ) System.out.println("found an else");} | ... ; ``` `$T` and `$L` evaluate to `Token` objects for token name `T` and token label `L`. `$ll` evaluates to `List` for list label `ll`. `$T.attr` evaluates to the type and value specified in the following table for attribute `attr`: |Attribute|Type|Description| |---------|----|-----------| |text|String|The text matched for the token; translates to a call to getText. Example: $ID.text.| |type|int|The token type (nonzero positive integer) of the token such as INT; translates to a call to getType. Example: $ID.type.| |line|int|The line number on which the token occurs, counting from 1; translates to a call to getLine. Example: $ID.line.| |pos|int|The character position within the line at which the token’s first character occurs counting from zero; translates to a call to getCharPositionInLine. Example: $ID.pos.| |index|int|The overall index of this token in the token stream, counting from zero; translates to a call to getTokenIndex. Example: $ID.index.| |channel|int|The token’s channel number. The parser tunes to only one channel, effectively ignoring off-channel tokens. The default channel is 0 (Token.DEFAULT_CHANNEL), and the default hidden channel is Token.HIDDEN_CHANNEL. Translates to a call to getChannel. Example: $ID.channel.| |int|int|The integer value of the text held by this token; it assumes that the text is a valid numeric string. Handy for building calculators and so on. Translates to Integer.valueOf(text-of-token). Example: $INT.int.| ## Parser Rule Attributes ANTLR predefines a number of read-only attributes associated with parser rule references that are available to actions. Actions can access rule attributes only for references that precede the action. The syntax is `$r.attr` for rule name `r` or a label assigned to a rule reference. For example, `$expr.text` returns the complete text matched by a preceding invocation of rule `expr`: ``` returnStat : 'return' expr {System.out.println("matched "+$expr.text);} ; ``` Using a rule label looks like this: ``` returnStat : 'return' e=expr {System.out.println("matched "+$e.text);} ; ``` You can also use `$` followed by the name of the attribute to access the value associated with the currently executing rule. For example, `$start` is the starting token of the current rule. ``` returnStat : 'return' expr {System.out.println("first token "+$start.getText());} ; ``` `$r` and `$rl` evaluate to `ParserRuleContext` objects of type `RContext` for rule name `r` and rule label `rl`. `$rll` evaluates to `List` for rule list label `rll`. `$r.attr` evaluates to the type and value specified in the following table for attribute `attr`: |Attribute|Type|Description| |---------|----|-----------| |text|String|The text matched for a rule or the text matched from the start of the rule up until the point of the `$text` expression evaluation. Note that this includes the text for all tokens including those on hidden channels, which is what you want because usually that has all the whitespace and comments. When referring to the current rule, this attribute is available in any action including any exception actions.| |start|Token|The first token to be potentially matched by the rule that is on the main token channel; in other words, this attribute is never a hidden token. For rules that end up matching no tokens, this attribute points at the first token that could have been matched by this rule. When referring to the current rule, this attribute is available to any action within the rule.| |stop|Token|The last nonhidden channel token to be matched by the rule. When referring to the current rule, this attribute is available only to the after and finally actions.| |ctx|ParserRuleContext|The rule context object associated with a rule invocation. All of the other attributes are available through this attribute. For example, `$ctx.start` accesses the start field within the current rules context object. It’s the same as `$start`.| |parser|Parser|The parser itself. This attribute can be used, for example, to invoke a method defined in the parser's `@members` section from a semantic predicate.| ## Dynamically-Scoped Attributes You can pass information to and from rules using parameters and return values, just like functions in a general-purpose programming language. Programming languages don’t allow functions to access the local variables or parameters of invoking functions, however. For example, the following reference to local variable `x` form a nested method call is illegal in Java: ```java void f() { int x = 0; g(); } void g() { h(); } void h() { int y = x; // INVALID reference to f's local variable x } ``` Variable `x` is available only within the scope of `f`, which is the text lexically delimited by curly brackets. For this reason, Java is said to use lexical scoping. Lexical scoping is the norm for most programming languages. Languages that allow methods further down in the call chain to access local variables defined earlier are said to use dynamic scoping. The term dynamic refers to the fact that a compiler cannot statically determine the set of visible variables. This is because the set of variables visible to a method changes depending on who calls that method. It turns out that, in the grammar realm, distant rules sometimes need to communicate with each other, mostly to provide context information to rules matched below in the rule invocation chain. (Naturally, this assumes that you are using actions directly in the grammar instead of the parse-tree listener event mechanism.) ANTLR allows dynamic scoping in that actions can access attributes from invoking rules using syntax `$r::x` where `r` is a rule name and `x` is an attribute within that rule. It is up to the programmer to ensure that `r` is in fact an invoking rule of the current rule. A runtime exception occurs if `r` is not in the current call chain when you access `$r::x`. To illustrate the use of dynamic scoping, consider the real problem of defining variables and ensuring that variables in expressions are defined. The following grammar defines the symbols attribute where it belongs in the block rule but adds variable names to it in rule `decl`. Rule `stat` then consults the list to see whether variables have been defined. ``` grammar DynScope; prog: block ; block /* List of symbols defined within this block */ locals [ List symbols = new ArrayList() ] : '{' decl* stat+ '}' // print out all symbols found in block // $block::symbols evaluates to a List as defined in scope {System.out.println("symbols="+$symbols);} ; /** Match a declaration and add identifier name to list of symbols */ decl: 'int' ID {$block::symbols.add($ID.text);} ';' ; /** Match an assignment then test list of symbols to verify * that it contains the variable on the left side of the assignment. * Method contains() is List.contains() because $block::symbols * is a List. */ stat: ID '=' INT ';' { if ( !$block::symbols.contains($ID.text) ) { System.err.println("undefined variable: "+$ID.text); } } | block ; ID : [a-z]+ ; INT : [0-9]+ ; WS : [ \t\r\n]+ -> skip ; ``` Here’s a simple build and test sequence: ```bash $ antlr4 DynScope.g4 $ javac DynScope*.java $ grun DynScope prog => { => int i; => i = 0; => j = 3; => } => EOF <= undefined variable: j symbols=[i] ``` There’s an important difference between a simple field declaration in a `@members` action and dynamic scoping. symbols is a local variable and so there is a copy for each invocation of rule `block`. That’s exactly what we want for nested blocks so that we can reuse the same input variable name in an inner block. For example, the following nested code block redefines `i` in the inner scope. This new definition must hide the definition in the outer scope. ``` { int i; int j; i = 0; { int i; int x; x = 5; } x = 3; } ``` Here’s the output generated for that input by DynScope: ```bash $ grun DynScope prog nested-input symbols=[i, x] undefined variable: x symbols=[i, j] ``` Referencing `$block::symbols` accesses the `symbols` field of the most recently invoked `block`’s rule context object. If you need access to a symbols instance from a rule invocation farther up the call chain, you can walk backwards starting at the current context, `$ctx`. Use `getParent` to walk up the chain. ================================================ FILE: doc/antlr-project-testing.md ================================================ # ANTLR project unit tests ## Introduction Because ANTLR supports multiple target languages, the unit tests are broken into two groups: the unit tests that test the tool itself (in `tool-testsuite`) and the unit tests that test the parser runtimes (in `antlr4/runtime-testsuite`). The tool tests are straightforward because they are Java code testing Java code; see the section at the bottom of this file. The runtime tests must be specified in a generic fashion to work across language targets. Furthermore, the various targets from Java must be tested. This usually means Java launching processes to compile, say, C++ and run parsers. As of 4.10, a Java descriptor file held as an [RuntimeTestDescriptor.java](../runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) is used to represent each runtime test. Each test is described with a text file with various sections and resides in a group directory; see [directories under descriptors' dir](../runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors). Here is a sample test descriptor: ``` [notes] This is a regression test for blah blah blah... [type] Parser [grammar] grammar T; a : ID* { }; ID : 'a'..'z'+; WS : (' '|'\n') -> skip; [start] a [input] a b c [output] """abc """ ``` The grammars are strings representing StringTemplates (`ST` objects) so `` will get replace when the unit test file is generated (`Test.java`, `Test.cs`, ...). The `writeln` template must be defined per target. Here are all the [Target templates for runtime tests](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates). Use triple-quotes `"""` when whitespace matters (usually input/output sections). ## Requirements In order to perform the tests on all target languages, the following tools should be installed: * dotnet * Node.js * Python 3 * Go * Swift * Clang (Linux, Mac) or MSBuild (Windows) for C++ * Dart * PHP To **install into local repository** `~/.m2/repository/org/antlr`, do this: ```bash $ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux $ mvn install -DskipTests # make sure all artifacts are visible on this machine ``` ## Running the runtime tests A single test rig is sufficient to test all targets against all descriptors using the [junit dynamic tests](https://junit.org/junit5/docs/current/user-guide/#writing-tests-dynamic-tests) mechanism. But it's often convenient to test just a single target or perhaps even just a single test within a single group of a single target. IntelliJ automatically generates a bunch of [Target runtime test rigs](../runtime-testsuite/test/org/antlr/v4/test/runtime) that allows developers such flexibility. For example, here are the Python3 test rigs in IntelliJ: ![testrigs](images/testrigs.png) And the result of testing the entire subdirectory: ![python3-tests](images/python3-tests.png) All test are run in parallel both via maven and via IDE. In IntelliJ, it's very easy to go to source by right-clicking on any test and pressing `Jump to source` (F4). ## Running test subsets From the `runtime-testsuite` dir ### Run all tests for a single target ```bash $ cd runtime-testsuite $ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux $ mvn -Dtest='java.**' test ------------------------------------------------------- T E S T S ------------------------------------------------------- [INFO] Running org.antlr.v4.test.runtime.java.TestIntegerList [INFO] Running org.antlr.v4.test.runtime.java.JavaRuntimeTests ... [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.023 s - in org.antlr.v4.test.runtime.java.TestIntegerList [INFO] Tests run: 348, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.269 s - in org.antlr.v4.test.runtime.java.JavaRuntimeTests ... ``` ## Adding a runtime test To add a new runtime test, first determine which [group (dir) of tests](../runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new descriptor file implementation by filling in one of these (omitting unused sections): ``` [notes] [type] [grammar] [slaveGrammar] [start] [input] [output] [errors] [flags] [skip] ``` Your best bet is to find a similar test in the appropriate group and then copy and paste the descriptor file, creating a new file within the test group dir. Modify the sections to suit your new problem. ### Ignoring tests In order to turn off a test for a particular target, the `skip` section in the descriptor file should be used. For example, the following skips PHP and Dart targets: ``` [skip] PHP Dart ``` ### Target API/library testing Some parts of the runtime API need to be tested with code written specifically in the target language. For example, all the Java runtime API tests are placed here: [runtime-testsuite/test/org/antlr/v4/test/runtime/java/api](../runtime-testsuite/test/org/antlr/v4/test/runtime/java/api) Notice that it is under an `api` dir. The directory above is where all of the `*Test*` files go. ### Cross-language actions embedded within grammars To get: ``` System.out.println($set.stop); ``` Use instead the language-neutral: ``` ``` Template file [Java.test.stg](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates/Java.test.stg) has templates like: ``` writeln(s) ::= <);>> ``` that translate generic operations to target-specific language statements or expressions. ## Adding an ANTLR tool unit test Just go into the appropriate Java test class in dir [antlr4/tool-testsuite/test/org/antlr/v4/test/tool](../tool-testsuite/test/org/antlr/v4/test/tool) and add your unit test. ================================================ FILE: doc/building-antlr.md ================================================ # Building ANTLR Most programmers do not need the information on this page because they will simply download the appropriate jar(s) or use ANTLR through maven (via ANTLR's antlr4-maven-plugin). If you would like to fork the project and fix bugs or tweak the runtime code generation, then you will almost certainly need to build ANTLR itself. There are two components: 1. the tool that compiles grammars down into parsers and lexers in one of the target languages 1. the runtime used by those generated parsers and lexers. I will assume that the root directory is `/tmp` for the purposes of explaining how to build ANTLR in this document. *As of 4.6, ANTLR tool and Java-target runtime requires Java 7. As of 4.10, we have verified that the tool itself builds with Java 8 and 11.* # Get the source The first step is to get the Java source code from the ANTLR 4 repository at github. You can download the repository from github, but the easiest thing to do is simply clone the repository on your local disk: ```bash $ cd /tmp /tmp $ git clone https://github.com/antlr/antlr4.git Cloning into 'antlr4'... remote: Counting objects: 61480, done. remote: Total 61480 (delta 0), reused 0 (delta 0), pack-reused 61480 Receiving objects: 100% (61480/61480), 31.24 MiB | 7.18 MiB/s, done. Resolving deltas: 100% (32970/32970), done. Checking connectivity... done. Checking out files: 100% (1427/1427), done. ``` # Check your environment If you are starting from a clean, minimum Ubuntu OS, check your environment. ```bash $ sudo apt-get update $ # Get Java $ java > /dev/null 2>&1 $ if [[ "$?" != "0" ]]; then sudo apt install -y openjdk-11-jre-headless; fi $ # Get Mvn $ mvn > /dev/null 2>&1 $ if [[ "$?" != "0" ]]; then sudo apt install -y maven; fi ``` # Compile The current maven build seems complicated to me because there is a dependency of the project on itself. The runtime tests naturally depend on the current version being available but it won't compile without the current version. Once you have the generated/installed jar, mvn builds but otherwise there's a dependency on what you are going to build. You will get this error when you try to clean but you can ignore it: ``` [INFO] ANTLR 4 Runtime Tests (4th generation) ............. FAILURE [ 0.073 s] ... [ERROR] Plugin org.antlr:antlr4-maven-plugin:4.10-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.antlr:antlr4-maven-plugin:jar:4.10-SNAPSHOT -> [Help 1] ``` To be super squeaky clean, you can wipe out the repository cache, then do the build: ``` $ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux cd /tmp/antlr4 # or wherever you have the software rm -rf ~/.m2/repository/org/antlr* mvn clean mvn -DskipTests install ``` **NOTE:** We do `install` not `compile` as tool tests and such refer to modules that must be pulled from the maven install local cache. Once you have completed this process once and there is a jar hanging around in the repository cache. # Installing libs to mvn cache locally To skip the tests (which require all the target languages be installed) and **install into local repository** `~/.m2/repository/org/antlr`, do this: ```bash $ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux $ mvn install -DskipTests # make sure all artifacts are visible on this machine ``` You should see these jars (when building 4.6-SNAPSHOT): ```bash /Users/parrt/.m2/repository/org/antlr $ find antlr4* -name '*.jar' antlr4-maven-plugin/4.6-SNAPSHOT/antlr4-maven-plugin-4.6-SNAPSHOT.jar antlr4-runtime-testsuite/4.6-SNAPSHOT/antlr4-runtime-testsuite-4.6-SNAPSHOT-tests.jar antlr4-runtime-testsuite/4.6-SNAPSHOT/antlr4-runtime-testsuite-4.6-SNAPSHOT.jar antlr4-runtime/4.6-SNAPSHOT/antlr4-runtime-4.6-SNAPSHOT.jar antlr4-tool-testsuite/4.6-SNAPSHOT/antlr4-tool-testsuite-4.6-SNAPSHOT.jar antlr4/4.6-SNAPSHOT/antlr4-4.6-SNAPSHOT-tests.jar antlr4/4.6-SNAPSHOT/antlr4-4.6-SNAPSHOT.jar ``` Note that ANTLR is written in itself, which is why maven downloads antlr4-4.5.jar for boostrapping 4.6-SNAPSHOT purposes. # Testing tool and targets See [ANTLR project unit tests](antlr-project-testing.md). # Building without testing To build without running the tests (saves a lot of time), do this: ```bash $ mvn -DskipTests install ``` ## Building ANTLR in Intellij IDE After download ANTLR source, just "import project from existing sources" and click on the "Maven Projects" tab in right gutter of IDE. It should build stuff in the background automatically and look like: ================================================ FILE: doc/cpp-target.md ================================================ # C++ The C++ target supports all platforms that can either run MS Visual Studio 2017 (or newer), XCode 7 (or newer) or CMake (C++17 required). All build tools can either create static or dynamic libraries, both as 64bit or 32bit arch. Additionally, XCode can create an iOS library. Also see [Antlr4 for C++ with CMake: A practical example](http://blorente.me/beyond-the-loop/Antlr-cpp-cmake/). ## How to create a C++ lexer or parser? This is pretty much the same as creating a Java lexer or parser, except you need to specify the language target, for example: ``` $ antlr4 -Dlanguage=Cpp MyGrammar.g4 ``` You will see that there are a whole bunch of files generated by this call. If visitor or listener are not suppressed (which is the default) you'll get: * MyGrammarLexer.h + MyGrammarLexer.cpp * MyGrammarParser.h + MyGrammarParser.cpp * MyGrammarVisitor.h + MyGrammarVisitor.cpp * MyGrammarBaseVisitor.h + MyGrammarBaseVisitor.cpp * MyGrammarListener.h + MyGrammarListener.cpp * MyGrammarBaseListener.h + MyGrammarBaseListener.cpp ## Where can I get the runtime? Once you've generated the lexer and/or parser code, you need to download or build the runtime. Prebuilt C++ runtime binaries for Windows (Visual Studio 2013/2015), OSX/macOS and iOS are available on the ANTLR web site: * http://www.antlr.org Use CMake to build a Linux library (works also on OSX, however not for the iOS library). Instead of downloading a prebuilt binary you can also easily build your own library on OSX or Windows. Just use the provided projects for XCode or Visual Studio and build it. Should work out of the box without any additional dependency. ## How do I run the generated lexer and/or parser? Putting it all together to get a working parser is really easy. Look in the [runtime/Cpp/demo](../runtime/Cpp/demo) folder for a simple example. The [README](../runtime/Cpp/demo/README.md) there describes shortly how to build and run the demo on OSX, Windows or Linux. ## How do I create and run a custom listener? The generation step above created a listener and base listener class for you. The listener class is an abstract interface, which declares enter and exit methods for each of your parser rules. The base listener implements all those abstract methods with an empty body, so you don't have to do it yourself if you just want to implement a single function. Hence use this base listener as the base class for your custom listener: ```c++ #include #include "antlr4-runtime.h" #include "MyGrammarLexer.h" #include "MyGrammarParser.h" #include "MyGrammarBaseListener.h" using namespace antlr4; class TreeShapeListener : public MyGrammarBaseListener { public: void enterKey(ParserRuleContext *ctx) override { // Do something when entering the key rule. } }; int main(int argc, const char* argv[]) { std::ifstream stream; stream.open(argv[1]); ANTLRInputStream input(stream); MyGrammarLexer lexer(&input); CommonTokenStream tokens(&lexer); MyGrammarParser parser(&tokens); tree::ParseTree *tree = parser.key(); TreeShapeListener listener; tree::ParseTreeWalker::DEFAULT.walk(&listener, tree); return 0; } ``` This example assumes your grammar contains a parser rule named `key` for which the `enterKey` function was generated. ## Special cases for this ANTLR target There are a couple of things that only the C++ ANTLR target has to deal with. They are described here. ### Code Generation Aspects The code generation (by running the ANTLR4 jar) allows to specify 2 values you might find useful for better integration of the generated files into your application (both are optional): * A **namespace**: use the **`-package`** parameter to specify the namespace you want. * An **export macro**: especially in VC++ extra work is required to export your classes from a DLL. This is usually accomplished by a macro that has different values depending on whether you are creating the DLL or import it. The ANTLR4 runtime itself also uses one for its classes: ```c++ #ifdef ANTLR4CPP_EXPORTS #define ANTLR4CPP_PUBLIC __declspec(dllexport) #else #ifdef ANTLR4CPP_STATIC #define ANTLR4CPP_PUBLIC #else #define ANTLR4CPP_PUBLIC __declspec(dllimport) #endif #endif ``` Just like the `ANTLR4CPP_PUBLIC` macro here you can specify your own one for the generated classes using the **`-DexportMacro=...`** command-line parameter or grammar option `options {exportMacro='...';}` in your grammar file. In order to create a static lib in Visual Studio define the `ANTLR4CPP_STATIC` macro in addition to the project settings that must be set for a static library (if you compile the runtime yourself). For gcc and clang it is possible to use the `-fvisibility=hidden` setting to hide all symbols except those that are made default-visible (which has been defined for all public classes in the runtime). ### Compile Aspects When compiling generated files, you can configure a compile option according to your needs (also optional): * A **thread local DFA macro**: Add `-DANTLR4_USE_THREAD_LOCAL_CACHE=1` to the compilation options will enable using thread local DFA cache (disabled by default), after that, each thread uses its own DFA. This will increase memory usage to store thread local DFAs and redundant computation to build thread local DFAs (not too much). The benefit is that it can improve the concurrent performance running with multiple threads. In other words, when you find your concurent throughput is not high enough, you should consider turning on this option. ### Memory Management Since C++ has no built-in memory management we need to take extra care. For that we rely mostly on smart pointers, which however might cause time penalties or memory side effects (like cyclic references) if not used with care. Currently however the memory household looks very stable. Generally, when you see a raw pointer in code consider this as being managed elsewhere. You should never try to manage such a pointer (delete, assign to smart pointer etc.). Accordingly a parse tree is only valid for the lifetime of its parser. The parser, in turn, is only valid for the lifetime of its token stream, and so on back to the original `ANTLRInputStream` (or equivalent). To retain a tree across function calls you'll need to create and store all of these and `delete` all but the tree when you no longer need it. ### Unicode Support Encoding is mostly an input issue, i.e. when the lexer converts text input into lexer tokens. The parser is completely encoding unaware. The C++ target always expects UTF-8 input (either in a string or stream) which is then converted to UTF-32 (a char32_t array) and fed to the lexer. ### Named Actions In order to help customizing the generated files there are a number of additional so-called **named actions**. These actions are tight to specific areas in the generated code and allow to add custom (target specific) code. All targets support these actions * @parser::header * @parser::members * @lexer::header * @lexer::members (and their scopeless alternatives `@header` and `@members`) where header doesn't mean a C/C++ header file, but the top of a code file. The content of the header action appears in all generated files at the first line. So it's good for things like license/copyright information. The content of a *members* action is placed in the public section of lexer or parser class declarations. Hence it can be used for public variables or predicate functions used in a grammar predicate. Since all targets support *header* + *members* they are the best place for stuff that should be available also in generated files for other languages. In addition to that the C++ target supports many more such named actions. Unfortunately, it's not possible to define new scopes (e.g. *listener* in addition to *parser*) so they had to be defined as part of the existing scopes (*lexer* or *parser*). The grammar in the demo application contains all of the named actions as well for reference. Here's the list: * **@lexer::preinclude** - Placed right before the first #include (e.g. good for headers that must appear first, for system headers etc.). Appears in both lexer h and cpp file. * **@lexer::postinclude** - Placed right after the last #include, but before any class code (e.g. for additional namespaces). Appears in both lexer h and cpp file. * **@lexer::context** - Placed right before the lexer class declaration. Use for e.g. additional types, aliases, forward declarations and the like. Appears in the lexer h file. * **@lexer::declarations** - Placed in the private section of the lexer declaration (generated sections in all classes strictly follow the pattern: public, protected, private, from top to bottom). Use this for private vars etc. * **@lexer::definitions** - Placed before other implementations in the cpp file (but after *@postinclude*). Use this to implement e.g. private types. For the parser there are the same actions as shown above for the lexer. In addition to that there are even more actions for visitor and listener classes: * **@parser::listenerpreinclude** * **@parser::listenerpostinclude** * **@parser::listenerdeclarations** * **@parser::listenermembers** * **@parser::listenerdefinitions** * * **@parser::baselistenerpreinclude** * **@parser::baselistenerpostinclude** * **@parser::baselistenerdeclarations** * **@parser::baselistenermembers** * **@parser::baselistenerdefinitions** * * **@parser::visitorpreinclude** * **@parser::visitorpostinclude** * **@parser::visitordeclarations** * **@parser::visitormembers** * **@parser::visitordefinitions** * * **@parser::basevisitorpreinclude** * **@parser::basevisitorpostinclude** * **@parser::basevisitordeclarations** * **@parser::basevisitormembers** * **@parser::basevisitordefinitions** and should be self explanatory now. Note: there is no *context* action for listeners or visitors, simply because they would be even less used than the other actions and there are so many already. ================================================ FILE: doc/creating-a-language-target.md ================================================ # Creating an ANTLR Language Target This document describes how to make ANTLR generate parsers in a new language, *X*. ## Overview Creating a new target involves the following key elements: 1. For the tool, create class *X*Target as a subclass of class `Target` in package `org.antlr.v4.codegen.target`. This class describes language specific details about escape characters and strings and so on. There is very little to do here typically. 2. Create `*X*.stg` in directory `tool/resources/org/antlr/v4/tool/templates/codegen/*X*/*X*.stg`. This is a [StringTemplate](http://www.stringtemplate.org/) group file (`.stg`) that tells ANTLR how to express all the parsing elements needed to generate code. You will see templates called `ParserFile`, `Parser`, `Lexer`, `CodeBlockForAlt`, `AltBlock`, etc... Each of these must be described how to build the indicated chunk of code. Your best bet is to find the closest existing target, copy that template file, and tweak to suit. 3. Create a runtime library to support the parsers generated by ANTLR. Under directory `runtime/*X*`, you are in complete control of the directory structure as dictated by common usage of that target language. For example, Java has: `runtime/Java/lib` and `runtime/Java/src` directories. Under `src`, you will find a directory structure for package `org.antlr.v4.runtime` and below. 4. Create a template file for runtime tests. All you have to do is provide a few templates that indicate how to print values and declare variables. Our runtime test mechanism in dir `runtime-testsuite` will automatically generate code using these templates for each target and check the test results. It needs to know how to define various class fields, compare members and so on. You must create a `*X*.test.stg` file underneath [runtime-testsuite/resources/org/antlr/v4/test/runtime](../runtime-testsuite/resources/org/antlr/v4/test/runtime) and `Test.*x*.stg` underneath [runtime-testsuite/resources/org/antlr/v4/test/runtime/helpers](../runtime-testsuite/resources/org/antlr/v4/test/runtime/helpers). Again, your best bet is to copy the templates from the closest language to your target and tweak it to suit. 6. Create test files under [/runtime-testsuite/test/org/antlr/v4/test/runtime](../runtime-testsuite/test/org/antlr/v4/test/runtime). They will load defined test cases in each test descriptor. Also add the `/runtime-testsuite/test/org/antlr/v4/test/runtime/X/BaseXTest.java` which defines how test cases will execute and output. 7. Create/edit shell scripts in [/.github](../.github) to run tests in CI pipelines. ## Getting started 1. Fork the `antlr/antlr4` repository at GitHub to your own user so that you have repository `username/antlr4`. 2. Clone `username/antlr4`, the forked repository, to your local disk. Your remote `origin` will be the forked repository on GitHub. Add a remote `upstream` to the original `antlr/antlr4` repository (URL `https://github.com/antlr/antlr4.git`). Changes that you would like to contribute back to the project are done with [pull requests](https://help.github.com/articles/using-pull-requests/). 3. Try to build it before doing anything ```bash $ mvn compile ``` That should proceed with success. See [Building ANTLR](building-antlr.md) for more details. ## Comparing your target's parsing decisionmaking with Java's ANTLR's power comes from it's dynamic parsing strategy, but that means each target must implement that complicated algorithm. You should compare your target's debug output for ParserATNSimulator with Java's. Run this so we get right jars before trying this script: ``` cd ANTLR-ROOT-DIR mvn install -DskipTests=true cd runtime-tests mvn install -DskipTests=true # yes do it again ``` Run the script from `runtime-tests` dir with ``` ../scripts/traceatn.sh /tmp/JSON.g4 json -target Go /tmp/foo.json ``` or whatever your test grammar, start rule, target, test input are. ### Debugging the PHP target Because the PHP target is hosted in a separate repository, you will need to clone the [antlr/php-antlr-runtime](https://github.com/antlr/antlr-php-runtime) repository into the `runtime/PHP` and install the dependencies with `composer install` before you can run the tests. ``` git clone -b dev https://github.com/antlr/antlr-php-runtime.git runtime/PHP cd runtime/PHP composer install ``` ================================================ FILE: doc/csharp-target.md ================================================ # C♯ ## Which frameworks are supported? The C# runtime is CLS compliant, and only requires a corresponding 3.5 .Net framework. In practice, the runtime has been extensively tested against: * Microsoft .Net 3.5 framework * Mono .Net 3.5 framework No issue was found, so you should find that the runtime works pretty much against any recent .Net framework. ## How do I get started? You will find full instructions on the [Git repo page for ANTLR C# runtime](https://github.com/antlr/antlr4/tree/master/runtime/CSharp). ## How do I use the runtime from my project? (i.e., How do I run the generated lexer and/or parser?) Let's suppose that your grammar is named `MyGrammar`. The tool will generate for you the following files: * MyGrammarLexer.cs * MyGrammarParser.cs * MyGrammarListener.cs (if you have not activated the -no-listener option) * MyGrammarBaseListener.cs (if you have not activated the -no-listener option) * MyGrammarVisitor.cs (if you have activated the -visitor option) * MyGrammarBaseVisitor.cs (if you have activated the -visitor option) Now a fully functioning code might look like the following for start rule `StartRule`: ```csharp using Antlr4.Runtime; using Antlr4.Runtime.Tree; public void MyParseMethod() { String input = "your text to parse here"; ICharStream stream = CharStreams.fromString(input); ITokenSource lexer = new MyGrammarLexer(stream); ITokenStream tokens = new CommonTokenStream(lexer); MyGrammarParser parser = new MyGrammarParser(tokens); IParseTree tree = parser.StartRule(); } ``` This program will work. But it won't be useful unless you do one of the following: * you visit the parse tree using a custom listener * you visit the parse tree using a custom visitor * your grammar comprises production code (like AntLR3) (please note that production code is target specific, so you can't have multi target grammars that include production code) ## How do I create and run a custom listener? Let's suppose your MyGrammar grammar comprises 2 rules: "key" and "value". The antlr4 tool will have generated the following listener (only partial code shown here): ```csharp interface IMyGrammarParserListener : IParseTreeListener { void EnterKey (MyGrammarParser.KeyContext context); void ExitKey (MyGrammarParser.KeyContext context); void EnterValue (MyGrammarParser.ValueContext context); void ExitValue (MyGrammarParser.ValueContext context); } ``` In order to provide custom behavior, you might want to create the following class: ```csharp class KeyPrinter : MyGrammarBaseListener { // override default listener behavior void ExitKey (MyGrammarParser.KeyContext context) { Console.WriteLine("Oh, a key!"); } } ``` In order to execute this listener, you would simply add the following lines to the above code: ```csharp ... IParseTree tree = parser.StartRule() - only repeated here for reference KeyPrinter printer = new KeyPrinter(); ParseTreeWalker.Default.Walk(printer, tree); ``` Further information can be found from The Definitive ANTLR Reference book. The C# implementation of ANTLR is as close as possible to the Java one, so you shouldn't find it difficult to adapt the examples for C#. See also [Sam Harwell's alternative C# target](https://github.com/tunnelvisionlabs/antlr4cs) ================================================ FILE: doc/dart-target.md ================================================ # ANTLR4 Runtime for Dart From version 4.9 onwards antlr's dart generated code is null sound safety compatible and sets the minimum dart sdk version to 2.12.0. ### First steps #### 1. Install ANTLR4 [The getting started guide](https://github.com/antlr/antlr4/blob/master/doc/getting-started.md) should get you started. #### 2. Install the Dart ANTLR runtime Each target language for ANTLR has a runtime package for running parser generated by ANTLR4. The runtime provides a common set of tools for using your parser. Install the runtime with the same version as the main ANTLR tool: Add this to your package's pubspec.yaml file: ```yaml ... dependencies: antlr4: ... ``` #### 3. Generate your parser You use the ANTLR4 "tool" to generate a parser. These will reference the ANTLR runtime, installed above. Suppose you're using a UNIX system and have set up an alias for the ANTLR4 tool as described in [the getting started guide](https://github.com/antlr/antlr4/blob/master/doc/getting-started.md). To generate your Dart parser, run the following command: ```shell script antlr4 -Dlanguage=Dart MyGrammar.g4 ``` For a full list of antlr4 tool options, please visit the [tool documentation page](https://github.com/antlr/antlr4/blob/master/doc/tool-options.md). ### Complete example Suppose you're using the JSON grammar from https://github.com/antlr/grammars-v4/tree/master/json. Then, invoke `antlr4 -Dlanguage=Dart JSON.g4`. The result of this is a collection of `.dart` including: * JsonLexer.dart * JsonParser.dart * JsonBaseListener.dart * JsonListener.dart (if you have not activated the -no-listener option) * JsonVisitor.dart (if you have activated the -visitor option) We'll write a small main func to call the generated parser/lexer (assuming they are separate). This one writes out the encountered `ParseTreeContext`'s: ```dart import 'package:antlr4/antlr4.dart'; import 'package:my_project/JSONParser.dart'; import 'package:my_project/JSONLexer.dart'; class TreeShapeListener implements ParseTreeListener { @override void enterEveryRule(ParserRuleContext ctx) { print(ctx.text); } @override void exitEveryRule(ParserRuleContext node) { } @override void visitErrorNode(ErrorNode node) { } @override void visitTerminal(TerminalNode node) { } } void main(List args) async { JSONLexer.checkVersion(); JSONParser.checkVersion(); final input = await InputStream.fromPath(args[0]); final lexer = JSONLexer(input); final tokens = CommonTokenStream(lexer); final parser = JSONParser(tokens); parser.addErrorListener(DiagnosticErrorListener()); final tree = parser.json(); ParseTreeWalker.DEFAULT.walk(TreeShapeListener(), tree); } ``` Create a `example.json` file: ```json {"a":1} ``` Parse the input file: ```shell script dart bin/main.dart example.json ``` The expected output is: ``` {"a":1} {"a":1} {"a":1} "a":1 1 ``` ### Debug We have some logs in place that can ease the debugging process, in order to turn these logs on you can enable the following environment declarations: - ANTLR_LEXER_DEBUG - ANTLR_LEXER_DFA_DEBUG - ANTLR_PARSER_DEBUG - ANTLR_PARSER_LIST_ATN_DECISIONS_DEBUG - ANTLR_PARSER_DFA_DEBUG - ANTLR_PARSER_RETRY_DEBUG If you're using flutter, you can define these variables by adding an `--dart-define` arguments, eg. `flutter run --dart-define LEXER_DEBUG=false` ================================================ FILE: doc/faq/actions-preds.md ================================================ # Actions and semantic predicates ## How do I test if an optional rule was matched? For optional rule references such as the initialization clause in the following ``` decl : 'var' ID (EQUALS expr)? ; ``` testing to see if that clause was matched can be done using `$EQUALS!=null` or `$expr.ctx!=null` where `$expr.ctx` points to the context or parse tree created for that reference to rule expr. ================================================ FILE: doc/faq/error-handling.md ================================================ # Error handling ## How do I perform semantic checking with ANTLR? See [How to implement error handling in ANTLR4](http://stackoverflow.com/questions/21613421/how-to-implement-error-handling-in-antlr4/21615751#21615751). ================================================ FILE: doc/faq/general.md ================================================ # General ## Why do we need ANTLR v4? *Oliver Zeigermann asked me some questions about v4. Here is our conversation.* *See the [preface from the book](http://media.pragprog.com/titles/tpantlr2/preface.pdf)* **Q: Why is the new version of ANTLR also called “honey badger”?** ANTLR v4 is called the honey badger release after the fearless hero of the YouTube sensation, The Crazy Nastyass Honey Badger. **Q: Why did you create a new version of ANTLR?** Well, I start creating a new version because v3 had gotten very messy on the inside and also relied on grammars written in ANTLR v2. Unfortunately, v2's open-source license was unclear and so projects such as Eclipse could not include v3 because of its dependency on v2. In the end, Sam Harwell converted all of the v2 grammars into v3 so that v3 was written in itself. Because v3 has a very clean BSD license, the Eclipse project okayed for inclusion in that project in the summer of 2011. As I was rewriting ANTLR, I wanted to experiment with a new variation of the LL(\*) parsing algorithm. As luck would have it, I came up with a cool new version called adaptive LL(\*) that pushes all of the grammar analysis effort to runtime. The parser warms up like Java does with its JIT on-the-fly compiler; the code gets faster and faster the longer it runs. The benefit is that the adaptive algorithm is much stronger than the static LL(\*) grammar analysis algorithm in v3. Honey Badger takes any grammar that you give it; it just doesn't give a damn. (v4 accepts even left recursive grammars, except for indirectly left recursive grammars where x calls y which calls x). v4 is the culmination of 25 years of research into parsers and parser generators. I think I finally know what I want to build. :) **Q: What makes you excited about ANTLR4?** The biggest thing is the new adaptive parsing strategy, which lets us accept any grammar we care to write. That gives us a huge productivity boost because we can now write much more natural expression rules (which occur in almost every grammar). For example, bottom-up parser generators such as yacc let you write very natural grammars like this: ``` e : e '*' e | e '+' e | INT ; ``` ANTLR v4 will also take that grammar now, translating it secretly to a non-left recursive version. Another big thing with v4 is that my goal has shifted from performance to ease-of-use. For example, ANTLR automatically can build parse trees for you and generate listeners and visitors. This is not only a huge productivity win, but also an important step forward in building grammars that don't depend on embedded actions. Those embedded actions (raw Java code or whatever) locked the grammar into use with only one language. If we keep all of the actions out of the grammar and put them into external visitors, we can reuse the same grammar to generate code in any language for which we have an ANTLR target. **Q: What do you think are the things people had problems with in ANTLR3?** The biggest problem was figuring out why ANTLR did not like their grammar. The static analysis often could not figure out how to generate a parser for the grammar. This problem totally goes away with the honey badger because it will take just about anything you give it without a whimper. **Q: And what with other compiler generator tools?** The biggest problem for the average practitioner is that most parser generators do not produce code you can load into a debugger and step through. This immediately removes bottom-up parser generators and the really powerful GLR parser generators from consideration by the average programmer. There are a few other tools that generate source code like ANTLR does, but they don't have v4's adaptive LL(\*) parsers. You will be stuck with contorting your grammar to fit the needs of the tool's weaker, say, LL(k) parsing strategy. PEG-based tools have a number of weaknesses, but to mention one, they have essentially no error recovery because they cannot report an error and until they have parsed the entire input. **Q: What are the main design decisions in ANTLR4?** Ease-of-use over performance. I will worry about performance later. Simplicity over complexity. For example, I have taken out explicit/manual AST construction facilities and the tree grammar facilities. For 20 years I've been trying to get people to go that direction, but I've since decided that it was a mistake. It's much better to give people a parser generator that can automatically build trees and then let them use pure code to do whatever tree walking they want. People are extremely familiar and comfortable with visitors, for example. **Q: What do you think people will like most on ANTLR4?** The lack of errors when you run your grammar through ANTLR. The automatic tree construction and listener/visitor generation. **What do you think are the problems people will try to solve with ANTLR4?** In my experience, almost no one uses parser generators to build commercial compilers. So, people are using ANTLR for their everyday work, building everything from configuration files to little scripting languages. In response to a question about this entry from stackoverflow.com: I believe that compiler developers are very concerned with parsing speed, error reporting, and error recovery. For that, they want absolute control over their parser. Also, some languages are so complicated, such as C++, that parser generators might build parsers slower than compiler developers want. The compiler developers also like the control of a recursive-descent parser for predicating the parse to handle context-sensitive constructs such as `T(i)` in C++. There is also likely a sense that parsing is the easy part of building a compiler so they don't immediately jump automatically to parser generators. I think this is also a function of previous generation parser generators. McPeak's Elkhound GLR-based parser generator is powerful enough and fast enough, in the hands of someone that knows what they're doing, to be suitable for compilers. I can also attest to the fact that ANTLR v4 is now powerful enough and fast enough to compete well with handbuilt parsers. E.g., after warm-up, it's now taking just 1s to parse the entire JDK java/\* library. ## What is the difference between ANTLR 3 and 4? The biggest difference between ANTLR 3 and 4 is that ANTLR 4 takes any grammar you give it unless the grammar had indirect left recursion. That means we don't need syntactic predicates or backtracking so ANTLR 4 does not support that syntax; you will get a warning for using it. ANTLR 4 allows direct left recursion so that expressing things like arithmetic expression syntax is very easy and natural: ``` expr : expr '*' expr | expr '+' expr | INT ; ``` ANTLR 4 automatically constructs parse trees for you and abstract syntax tree (AST) construction is no longer an option. See also [What if I need ASTs not parse trees for a compiler, for example?](https://github.com/antlr/antlr4/blob/master/doc/faq/parse-trees.md#what-if-i-need-asts-not-parse-trees-for-a-compiler-for-example). Another big difference is that we discourage the use of actions directly within the grammar because ANTLR 4 automatically generates [listeners and visitors](https://github.com/antlr/antlr4/blob/master/doc/listeners.md) for you to use that trigger method calls when some phrases of interest are recognized during a tree walk after parsing. See also [Parse Tree Matching and XPath](https://github.com/antlr/antlr4/blob/master/doc/tree-matching.md). Semantic predicates are still allowed in both the parser and lexer rules as our actions. For efficiency sake keep semantic predicates to the right edge of lexical rules. There are no tree grammars because we use listeners and visitors instead. ## Why is my expression parser slow? Make sure to use two-stage parsing. See example in [bug report](https://github.com/antlr/antlr4/issues/374). ```Java CharStream input = CharStreams.fromPath(Paths.get(args[0])); ExprLexer lexer = new ExprLexer(input); CommonTokenStream tokens = new CommonTokenStream(lexer); ExprParser parser = new ExprParser(tokens); parser.getInterpreter().setPredictionMode(PredictionMode.SLL); try { parser.stat(); // STAGE 1 } catch (Exception ex) { tokens.reset(); // rewind input stream parser.reset(); parser.getInterpreter().setPredictionMode(PredictionMode.LL); parser.stat(); // STAGE 2 // if we parse ok, it's LL not SLL } ``` ================================================ FILE: doc/faq/getting-started.md ================================================ # Getting started ## How to I install and run a simple grammar? See [Getting Started with ANTLR v4](https://raw.githubusercontent.com/antlr/antlr4/master/doc/getting-started.md). ## Why does my parser test program hang? Your test program is likely not hanging but simply waiting for you to type some input for standard input. Don't forget that you need to type the end of file character, generally on a line by itself, at the end of the input. On a Mac or Linux machine it is ctrl-D, as gawd intended, or ctrl-Z on a Windows machine. See [Getting Started with ANTLR v4](https://raw.githubusercontent.com/antlr/antlr4/master/doc/getting-started.md). ================================================ FILE: doc/faq/index.md ================================================ # Frequently-Asked Questions (FAQ) This is the main landing page for the ANTLR 4 FAQ. The links below will take you to the appropriate file containing all answers for that subcategory. *To add to or improve this FAQ, [fork](https://help.github.com/articles/fork-a-repo/) the [antlr/antlr4 repo](https://github.com/antlr/antlr4) then update this `doc/faq/index.md` or file(s) in that directory. Submit a [pull request](https://help.github.com/articles/creating-a-pull-request/) to get your changes incorporated into the main repository. Do not mix code and FAQ updates in the sample pull request.* **You must sign the contributors.txt certificate of origin with your pull request if you've not done so before.** ## Getting Started * [How to I install and run a simple grammar?](getting-started.md) * [Why does my parser test program hang?](getting-started.md) ## Installation * [Why can't ANTLR (grun) find my lexer or parser?](installation.md) * [Why can't I run the ANTLR tool?](installation.md) * [Why doesn't my parser compile?](installation.md) ## General * [Why do we need ANTLR v4?](general.md) * [What is the difference between ANTLR 3 and 4?](general.md) * [Why is my expression parser slow?](general.md) ## Grammar syntax ## Lexical analysis * [How can I parse non-ASCII text and use characters in token rules?](lexical.md) * [How do I replace escape characters in string tokens?](lexical.md) * [Why are my keywords treated as identifiers?](lexical.md) * [Why are there no whitespace tokens in the token stream?](lexical.md) ## Parse Trees * [How do I get the input text for a parse-tree subtree?](parse-trees.md) * [What if I need ASTs not parse trees for a compiler, for example?](parse-trees.md) * [When do I use listener/visitor vs XPath vs Tree pattern matching?](parse-trees.md) ## Translation * [ASTs vs parse trees](translation.md) * [Decoupling input walking from output generation](translation.md) ## Actions and semantic predicates * [How do I test if an optional rule was matched?](actions-preds.md) ## Error handling * [How do I perform semantic checking with ANTLR?](error-handling.md) ================================================ FILE: doc/faq/installation.md ================================================ # Installation Please read carefully: [Getting Started with ANTLR v4](https://raw.githubusercontent.com/antlr/antlr4/master/doc/getting-started.md). ## Why can't ANTLR (grun) find my lexer or parser? If you see "Can't load Hello as lexer or parser", it's because you don't have '.' (current directory) in your CLASSPATH. ```bash $ alias antlr4='java -jar /usr/local/lib/antlr-4.2.2-complete.jar' $ alias grun='java org.antlr.v4.runtime.misc.TestRig' $ export CLASSPATH="/usr/local/lib/antlr-4.2.2-complete.jar" $ antlr4 Hello.g4 $ javac Hello*.java $ grun Hello r -tree Can't load Hello as lexer or parser $ ``` For mac/linux, use: ```bash export CLASSPATH=".:/usr/local/lib/antlr-4.2.2-complete.jar:$CLASSPATH" ``` or for Windows: ``` SET CLASSPATH=.;C:\Javalib\antlr4-complete.jar;%CLASSPATH% ``` **See the dot at the beginning?** It's critical. ## Why can't I run the ANTLR tool? If you get a no class definition found error, you are missing the ANTLR jar in your `CLASSPATH` (or you might only have the runtime jar): ```bash /tmp $ java org.antlr.v4.Tool Hello.g4 Exception in thread "main" java.lang.NoClassDefFoundError: org/antlr/v4/Tool Caused by: java.lang.ClassNotFoundException: org.antlr.v4.Tool at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) ``` ## Why doesn't my parser compile? If you see these kinds of errors, it's because you don't have the runtime or complete ANTLR library in your CLASSPATH. ```bash /tmp $ javac Hello*.java HelloBaseListener.java:3: package org.antlr.v4.runtime does not exist import org.antlr.v4.runtime.ParserRuleContext; ^ ... ``` ================================================ FILE: doc/faq/lexical.md ================================================ # Lexical analysis ## How can I parse non-ASCII text and use characters in token rules? See [Using non-ASCII characters in token rules](http://stackoverflow.com/questions/28126507/antlr4-using-non-ascii-characters-in-token-rules/28129510#28129510). ## How do I replace escape characters in string tokens? Unfortunately, manipulating the text of the token matched by a lexical rule is cumbersome (as of 4.2). You have to build up a buffer and then set the text at the end. Actions in the lexer execute at the associated position in the input just like they do in the parser. Here's an example that does escape character replacement in strings. It's not pretty but it works. ``` grammar Foo; @members { StringBuilder buf = new StringBuilder(); // can't make locals in lexer rules } STR : '"' ( '\\' ( 'r' {buf.append('\r');} | 'n' {buf.append('\n');} | 't' {buf.append('\t');} | '\\' {buf.append('\\');} | '\"' {buf.append('"');} ) | ~('\\'|'"') {buf.append((char)_input.LA(-1));} )* '"' {setText(buf.toString()); buf.setLength(0); System.out.println(getText());} ; ``` It's easier and more efficient to return original input string and then use a small function to rewrite the string later during a parse tree walk or whatever. But, here's how to do it from within the lexer. Lexer actions don't work in the interpreter, which includes xpath and tree patterns. For more on the argument against doing complicated things in the lexer, see the [related lexer-action issue at github](https://github.com/antlr/antlr4/issues/483#issuecomment-37326067). ## Why are my keywords treated as identifiers? Keywords such as `begin` are also valid identifiers lexically and so that input is ambiguous. To resolve ambiguities, ANTLR gives precedence to the lexical rules specified first. That implies that you must put the identifier rule after all of your keywords: ``` grammar T; decl : DEF 'int' ID ';' DEF : 'def' ; // ambiguous with ID as is 'int' ID : [a-z]+ ; ``` Notice that literal `'int'` is also physically before the ID rule and will also get precedence. ## Why are there no whitespace tokens in the token stream? The lexer is not sending white space to the parser, which means that the rewrite stream doesn't have access to the tokens either. It is because of the skip lexer command: ``` WS : [ \t\r\n\u000C]+ -> skip ; ``` You have to change all those to `-> channel(HIDDEN)` which will send them to the parser on a different channel, making them available in the token stream, but invisible to the parser. ================================================ FILE: doc/faq/parse-trees.md ================================================ # Parse Trees ## How do I get the input text for a parse-tree subtree? In ParseTree, you have this method: ```java /** Return the combined text of all leaf nodes. Does not get any * off-channel tokens (if any) so won't return whitespace and * comments if they are sent to parser on hidden channel. */ String getText(); ``` But, you probably want this method from TokenStream: ```java /** * Return the text of all tokens in the source interval of the specified * context. This method behaves like the following code, including potential * exceptions from the call to {@link #getText(Interval)}, but may be * optimized by the specific implementation. * *

If {@code ctx.getSourceInterval()} does not return a valid interval of * tokens provided by this stream, the behavior is unspecified.

* * 
 * TokenStream stream = ...;
 * String text = stream.getText(ctx.getSourceInterval());
 *
* * @param ctx The context providing the source interval of tokens to get * text for. * @return The text of all tokens within the source interval of {@code ctx}. */ public String getText(RuleContext ctx); ``` That is, do this: ``` mytokens.getText(mySubTree); ``` ## What if I need ASTs not parse trees for a compiler, for example? For writing a compiler, either generate [LLVM-type static-single-assignment](http://llvm.org/docs/LangRef.html) form or construct an AST from the parse tree using a listener or visitor. Or, use actions in grammar, turning off auto-parse-tree construction. ## When do I use listener/visitor vs XPath vs Tree pattern matching? ### XPath XPath works great when you need to find specific nodes, possibly in certain contexts. The context is limited to the parents on the way to the root of the tree. For example, if you want to find all ID nodes, use path `//ID`. If you want all variable declarations, you might use path `//vardecl`. If you only want fields declarations, then you can use some context information via path `/classdef/vardecl`, which would only find vardecls that are children of class definitions. You can merge the results of multiple XPath `findAll()`s simulating a set union for XPath. The only caveat is that the order from the original tree is not preserved when you union multiple `findAll()` sets. ### Tree pattern matching Use tree pattern matching when you want to find specific subtree structures such as all assignments to 0 using pattern `x = 0;`. (Recall that these are very convenient because you specify the tree structure in the concrete syntax of the language described by the grammar.) If you want to find all assignments of any kind, you can use pattern `x = ;` where `` will find any expression. This works great for matching particular substructures and therefore gives you a bit more ability to specify context. I.e., instead of just finding all identifiers, you can find all identifiers on the left hand side of an expression. ### Listeners/Visitors Using the listener or visitor interfaces give you the most power but require implementing more methods. It might be more challenging to discover the emergent behavior of the listener than a simple tree pattern matcher that says *go find me X under node Y*. Listeners are great when you want to visit many nodes in a tree. Listeners allow you to compute and save context information necessary for processing at various nodes. For example, when building a symbol table manager for a compiler or translator, you need to compute symbol scopes such as globals, class, function, and code block. When you enter a class or function, you push a new scope and then pop it when you exit that class or function. When you see a symbol, you need to define it or look it up in the proper scope. By having enter/exit listener functions push and pop scopes, listener functions for defining variables simply say something like: ```java scopeStack.peek().define(new VariableSymbol("foo")) ``` That way each listener function does not have to compute its appropriate scope. Examples: [DefScopesAndSymbols.java](https://github.com/mantra/compiler/blob/master/src/java/mantra/semantics/DefScopesAndSymbols.java) and [SetScopeListener.java](https://github.com/mantra/compiler/blob/master/src/java/mantra/semantics/SetScopeListener.java) and [VerifyListener.java](https://github.com/mantra/compiler/blob/master/src/java/mantra/semantics/VerifyListener.java) ================================================ FILE: doc/faq/translation.md ================================================ # Translation ## ASTs vs parse trees I used to do specialized AST (**abstract** syntax tree) nodes rather than (concrete) parse trees because I used to think more about compilation and generating bytecode/assembly code. When I started thinking more about translation, I started using parse trees. For v4, I realized that I did mostly translation. I guess what I'm saying is that maybe parse trees are not as good as ASTs for generating bytecodes. Personally, I would rather see `(+ 3 4)` rather than `(expr 3 + 4)` for generating byte codes, but it's not the end of the world. (*Can someone fill this in?*) ## Decoupling input walking from output generation I suggest creating an intermediate model that represents your output. You walk the parse tree to collect information and create your model. Then, you could almost certainly automatically walk this internal model to generate output based upon stringtemplates that match the class names of the internal model. In other words, define a special `IFStatement` object that has all of the fields you want and then create them as you walk the parse tree. This decoupling of the input from the output is very powerful. Just because we have a parse tree listener doesn't mean that the parse tree itself is necessarily the best data structure to hold all information necessary to generate code. Imagine a situation where the output is the exact reverse of the input. In that case, you really want to walk the input just to collect data. Generating output should be driven by the internal model not the way it was represented in the input. ================================================ FILE: doc/getting-started.md ================================================ # Getting Started with ANTLR v4 Hi and welcome to the version 4 release of ANTLR! See [Why do we need ANTLR v4?](faq/general.md) and the [preface of the ANTLR v4 book](http://media.pragprog.com/titles/tpantlr2/preface.pdf). ## Getting started the easy way using antlr4-tools To play around with ANTLR without having to worry about installing it and the Java needed to execute it, use [antlr4-tools](https://github.com/antlr/antlr4-tools). The only requirement is Python3, which is typically installed on all developer machines on all operating systems. (See below for Windows issue.) ```bash $ pip install antlr4-tools ``` That command creates `antlr4` and `antlr4-parse` executables that, if necessary, will download and install Java 11 plus the latest ANTLR jar: ```bash $ antlr4 Downloading antlr4-4.13.2-complete.jar ANTLR tool needs Java to run; install Java JRE 11 yes/no (default yes)? y Installed Java in /Users/parrt/.jre/jdk-11.0.15+10-jre; remove that dir to uninstall ANTLR Parser Generator Version 4.13.2 -o ___ specify output directory where all output is generated -lib ___ specify location of grammars, tokens files ... ``` Let's play with a simple grammar: ``` grammar Expr; prog: expr EOF ; expr: expr ('*'|'/') expr | expr ('+'|'-') expr | INT | '(' expr ')' ; NEWLINE : [\r\n]+ -> skip; INT : [0-9]+ ; ``` ### Windows-specific issues On Windows, the `pip` command doesn't just work---you need to add the `...\local-packages\python38\scripts` dir to your `PATH`, which itself might require a fun reboot. If you use WSL on Windows, then the pip install will also properly at the scripts directly (if you run from bash shell). 1. Go to the Microsoft Store 2. Search in Microsoft Store for Python 3. Select the newest version of Python (3.11). 4. Click the "Get" button. Store installs python and pip at "c:\Users...\AppData\Local\Microsoft\WindowsApps\python.exe" and "c:\Users...\AppData\Local\Microsoft\WindowsApps\pip.exe", respectively. And, it updates the search path immediately with the install. 5. Open a "cmd" terminal. 6. You can now type "python" and "pip", and "pip install antlr4-tools". 7. Unfortunately, it does not add that to the search path. 7. Update the search path to contain `c:\Users...\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p8\LocalCache\local-packages\Python310\Scripts`. You may need to install MSYS2, then do a `find /c/ -name antlr4.exe 2> /dev/null` and enter that path. 8. Or, you can set up an alias to antlr4.exe on that path. The good news is that the ANTLR4 Python tool downloads the ANTLR jar in a standard location, and you don't need to do that manually. It's also possible to go in a browser, go to python.org, and download the python package. But, it's likely you will need to update the path for antlr4.exe as before. ### Try parsing with a sample grammar To parse and get the parse tree in text form, use: ```bash $ antlr4-parse Expr.g4 prog -tree 10+20*30 ^D (prog:1 (expr:2 (expr:3 10) + (expr:1 (expr:3 20) * (expr:3 30))) ) ``` (Note: `^D` means control-D and indicates "end of input" on Unix; use `^Z` on Windows.) Here's how to get the tokens and trace through the parse: ```bash $ antlr4-parse Expr.g4 prog -tokens -trace 10+20*30 ^D [@0,0:1='10',,1:0] [@1,2:2='+',<'+'>,1:2] [@2,3:4='20',,1:3] [@3,5:5='*',<'*'>,1:5] [@4,6:7='30',,1:6] [@5,9:8='',,2:0] enter prog, LT(1)=10 enter expr, LT(1)=10 consume [@0,0:1='10',<8>,1:0] rule expr enter expr, LT(1)=+ consume [@1,2:2='+',<3>,1:2] rule expr enter expr, LT(1)=20 consume [@2,3:4='20',<8>,1:3] rule expr enter expr, LT(1)=* consume [@3,5:5='*',<1>,1:5] rule expr enter expr, LT(1)=30 consume [@4,6:7='30',<8>,1:6] rule expr exit expr, LT(1)= exit expr, LT(1)= exit expr, LT(1)= consume [@5,9:8='',<-1>,2:0] rule prog exit prog, LT(1)= ``` Here's how to get a visual tree view: ```bash $ antlr4-parse Expr.g4 prog -gui 10+20*30 ^D ``` The following will pop up in a Java-based GUI window: ### Generating parser code The previous section used a built-in ANTLR interpreter but typically you will ask ANTLR to generate code in the language used by your project (there are about 10 languages to choose from as of 4.11). Here's how to generate Java code from a grammar: ```bash $ antlr4 Expr.g4 $ ls Expr*.java ExprBaseListener.java ExprLexer.java ExprListener.java ExprParser.java ``` And, here's how to generate C++ code from the same grammar: ```bash $ antlr4 -Dlanguage=Cpp Expr.g4 $ ls Expr*.cpp Expr*.h ExprBaseListener.cpp ExprLexer.cpp ExprListener.cpp ExprParser.cpp ExprBaseListener.h ExprLexer.h ExprListener.h ExprParser.h ``` ## Installation ANTLR is really two things: a tool written in Java that translates your grammar to a parser/lexer in Java (or other target language) and the runtime library needed by the generated parsers/lexers. Even if you are using the ANTLR Intellij plug-in or ANTLRWorks to run the ANTLR tool, the generated code will still need the runtime library. The first thing you should do is probably download and install a development tool plug-in. Even if you only use such tools for editing, they are great. Then, follow the instructions below to get the runtime environment available to your system to run generated parsers/lexers. In what follows, I talk about antlr-4.13.2-complete.jar, which has the tool and the runtime and any other support libraries (e.g., ANTLR v4 is written in v3). If you are going to integrate ANTLR into your existing build system using mvn, ant, or want to get ANTLR into your IDE such as eclipse or intellij, see [Integrating ANTLR into Development Systems](https://github.com/antlr/antlr4/blob/master/doc/IDEs.md). ### UNIX 0. Install Java (version 11 or higher) 1. Download ``` $ cd /usr/local/lib $ curl -O https://www.antlr.org/download/antlr-4.13.2-complete.jar ``` Or just download in browser from website: [https://www.antlr.org/download.html](https://www.antlr.org/download.html) and put it somewhere rational like `/usr/local/lib`. if you are using lower version jdk, just download from [website download](https://github.com/antlr/website-antlr4/tree/gh-pages/download) for previous version, and antlr version before 4.13.2 support jdk 1.8 2. Add `antlr-4.13.2-complete.jar` to your `CLASSPATH`: ``` $ export CLASSPATH=".:/usr/local/lib/antlr-4.13.2-complete.jar:$CLASSPATH" ``` It's also a good idea to put this in your `.bash_profile` or whatever your startup script is. 3. Create aliases for the ANTLR Tool, and `TestRig`. ``` $ alias antlr4='java -Xmx500M -cp "/usr/local/lib/antlr-4.13.2-complete.jar:$CLASSPATH" org.antlr.v4.Tool' $ alias grun='java -Xmx500M -cp "/usr/local/lib/antlr-4.13.2-complete.jar:$CLASSPATH" org.antlr.v4.gui.TestRig' ``` ### WINDOWS (*Thanks to Graham Wideman*) 0. Install Java (version 1.7 or higher) 1. Download antlr-4.13.2-complete.jar (or whatever version) from [https://www.antlr.org/download.html](https://www.antlr.org/download.html) Save to your directory for 3rd party Java libraries, say `C:\Javalib` 2. Add `antlr-4.13.2-complete.jar` to CLASSPATH, either: * Permanently: Using System Properties dialog > Environment variables > Create or append to `CLASSPATH` variable * Temporarily, at command line: ``` SET CLASSPATH=.;C:\Javalib\antlr-4.13.2-complete.jar;%CLASSPATH% ``` 3. Create short convenient commands for the ANTLR Tool, and TestRig, using batch files or doskey commands: * Batch files (in directory in system PATH) antlr4.bat and grun.bat ``` java org.antlr.v4.Tool %* ``` ``` @ECHO OFF SET TEST_CURRENT_DIR=%CLASSPATH:.;=% if "%TEST_CURRENT_DIR%" == "%CLASSPATH%" ( SET CLASSPATH=.;%CLASSPATH% ) @ECHO ON java org.antlr.v4.gui.TestRig %* ``` * Or, use doskey commands: ``` doskey antlr4=java org.antlr.v4.Tool $* doskey grun =java org.antlr.v4.gui.TestRig $* ``` ### Testing the installation Either launch org.antlr.v4.Tool directly: ``` $ java org.antlr.v4.Tool ANTLR Parser Generator Version 4.13.2 -o ___ specify output directory where all output is generated -lib ___ specify location of .tokens files ... ``` or use -jar option on java: ``` $ java -jar /usr/local/lib/antlr-4.13.2-complete.jar ANTLR Parser Generator Version 4.13.2 -o ___ specify output directory where all output is generated -lib ___ specify location of .tokens files ... ``` ## A First Example In a temporary directory, put the following grammar inside file Hello.g4: Hello.g4 ``` // Define a grammar called Hello grammar Hello; r : 'hello' ID ; // match keyword hello followed by an identifier ID : [a-z]+ ; // match lower-case identifiers WS : [ \t\r\n]+ -> skip ; // skip spaces, tabs, newlines ``` Then run ANTLR the tool on it: ``` $ cd /tmp $ antlr4 Hello.g4 $ javac Hello*.java ``` Now test it: ``` $ grun Hello r -tree (Now enter something like the string below) hello parrt (now,do:) ^D (The output:) (r hello parrt) (That ^D means EOF on unix; it's ^Z in Windows.) The -tree option prints the parse tree in LISP notation. It's nicer to look at parse trees visually. $ grun Hello r -gui hello parrt ^D ``` That pops up a dialog box showing that rule `r` matched keyword `hello` followed by identifier `parrt`. ![](images/hello-parrt.png) ## Book source code The book has lots and lots of examples that should be useful too. You can download them here for free: [ANTLR reference book examples in Java](https://media.pragprog.com/titles/tpantlr2/code/tpantlr2-code.zip)
[ANTLR reference book examples in C#](https://github.com/Philippe-Laval/tpantlr2) [Language implementation patterns book examples in Java](https://media.pragprog.com/titles/tpdsl/code/tpdsl-code.zip)
[Language implementation patterns book examples in C#](https://github.com/Philippe-Laval/tpdsl) Also, there is a large collection of grammars for v4 at github: [https://github.com/antlr/grammars-v4](https://github.com/antlr/grammars-v4) ================================================ FILE: doc/go-changes.md ================================================ # Changes to the Go Runtime over time ## v4.12.0 to v4.13.0 Strictly speaking, if ANTLR was a go only project following [SemVer](https://semver.org/) release v4.13.0 would be at least a minor version change and arguably a bump to v5. However, we must follow the ANTLR conventions here or the release numbers would quickly become confusing. I apologize for being unable to follow the Go release rules absolutely to the letter. There are a lot of changes and improvements in this release, but only the change of repo holding the runtime code, and possibly the removal of interfaces will cause any code changes. There are no breaking changes to the runtime interfaces. ANTLR Go Maintainer: [Jim Idle](https://github.com/jimidle) - Email: [jimi@idle.ws](mailto:jimi@idle.ws) ### Code Relocation For complicated reasons, including not breaking the builds of some users who use a monorepo and eschew modules, as well as not making substantial changes to the internal test suite, the Go runtime code will continue to be maintained in the main ANTLR4 repo `antlr/antlr4`. If you wish to contribute changes to the Go runtime code, please continue to submit PRs to this main repo, against the `dev` branch. The code located in the main repo at about the depth of the Mariana Trench, means that the go tools cannot reconcile the module correctly. After some debate, it was decided that we would create a dedicated release repo for the Go runtime so that it will behave exactly as the Go tooling expects. This repo is auto-maintained and keeps both the dev and master branches up to date. Henceforth, all future projects using the ANTLR Go runtime, should import as follows: ```go import ( "github.com/antlr4-go/antlr/v4" ) ``` And use the command: ```shell go get github.com/antlr4-go/antlr ``` To get the module - `go mod tidy` is probably the best way once imports have been changed. Please note that there is no longer any source code kept in the ANTLR repo under `github.com/antlr/antlr4/runtime/Go/antlr`. If you are using the code without modules, then sync the code from the new release repo. ### Documentation Prior to this release, the godocs were essentially unusable as the go doc code was essentially copied without change, from teh Java runtime. The godocs are now properly formatted for Go and pkg.dev. Please feel free to raise an issue if you find any remaining mistakes. Or submit a PR (remember - not to the new repo). It is expected that it might take a few iterations to get the docs 100% squeaky clean. ### Removal of Unnecessary Interfaces The Go runtime was originally produced as almost a copy of the Java runtime but with go syntax. This meant that everything had an interface. There is no need to use interfaces in Go if there is only ever going to be one implementation of some struct and its methods. Interfaces cause an extra deference at runtime and are detrimental to performance if you are trying to squeeze out every last nanosecond, which some users will be trying to do. This is 99% an internal refactoring of the runtime with no outside effects to the user. ### Generated Recognizers Return *struct and not Interfaces The generated recognizer code generated an interface for the parsers and lexers. As they can only be implemented by the generated code, the interfaces were removed. This is possibly the only place you may need to make a code change to your driver code. If your code looked like this: ```go var lexer = parser.NewMySqlLexer(nil) var p = parser.NewMySqlParser(nil) ``` Or this: ```go lexer := parser.NewMySqlLexer(nil) p := parser.NewMySqlParser(nil) ``` Then no changes need to be made. However, fi you predeclared the parser and lexer variables with there type, such as like this: ```go var lexer parser.MySqlLexer var p parser.MySqlParser // ... lexer = parser.NewMySqlLexer(nil) p = parser.NewMySqlParser(nil) ``` You will need to change your variable declarations to pointers (note the introduction of the `*` below. ```go var lexer *parser.MySqlLexer var p *parser.MySqlParser // ... lexer = parser.NewMySqlLexer(nil) p = parser.NewMySqlParser(nil) ``` This is the only user facing change that I can see. This change though has a very beneficial side effect in that you no longer need to cast the interface into a struct so that you can access methods and data within it. Any code you had that needed to do that, will be cleaner and faster. The performance improvement is worth the change and there was no tidy way for me to avoid it. ### Parser Error Recovery Does Not Use Panic THe generated parser code was again essentially trying to be Java code in disguise. This meant that every parser rule executed a `defer {}` and a `recover()`, even if there wer no outstanding parser errors. Parser errors were issued by issuing a `panic()`! While some major work has been performed in the go compiler and runtime to make `defer {}` as fast as possible, `recover()` is (relatively) slow as it is not meant to be used as a general error mechanism, but to recover from say an internal library problem if that problem can be recovered to a known state. The generated code now stores a recognition error and a flag in the main parser struct and use `goto` to exit the rule instead of a `panic()`. As might be imagined, this is significantly faster through the happy path. It is also faster at generating errors. The ANTLR runtime tests do check error raising and recovery, but if you find any differences in the error handling behavior of your parsers, please raise an issue. ### Reduction in use of Pointers Certain internal structs, such as interval sets are small and immutable, but were being passed around as pointers anyway. These have been change to use copies, and resulted in significant performance increases in some cases. There is more work to come in this regard. ### ATN Deserialization When the ATN and associated structures are deserialized for the first time, there was a bug that caused a needed optimization to fail to be executed. This could have a significant performance effect on recognizers that were written in a suboptimal way (as in poorly formed grammars). This is now fixed. ### Prediction Context Caching was not Working This has a massive effect when reusing a parser for a second and subsequent run. The PredictionContextCache merely used memory but did not speed up subsequent executions. This is now fixed, and you should see a big difference in performance when reusing a parser. This single paragraph does not do this fix justice ;) ### Cumulative Performance Improvements Though too numerous to mention, there are a lot of small performance improvements, that add up in accumulation. Everything from improvements in collection performance to slightly better algorithms or specific non-generic algorithms. ### Cumulative Memory Improvements The real improvements in memory usage, allocation and garbage collection are saved for the next major release. However, if your grammar is well-formed and does not require almost infinite passes using ALL(*), then both memory and performance will be improved with this release. ### Bug Fixes Other small bug fixes have been addressed, such as potential panics in funcs that did not check input parameters. There are a lot of bug fixes in this release that most people were probably not aware of. All known bugs are fixed at the time of release preparation. ### A Note on Poorly Constructed Grammars Though I have made some significant strides on improving the performance of poorly formed grammars, those that are particularly bad will see much less of an incremental improvement compared to those that are fairly well-formed. This is deliberately so in this release as I felt that those people who have put in effort to optimize the form of their grammar are looking for performance, where those that have grammars that parser in seconds, tens of seconds or even minutes, are presumed to not care about performance. A particularly good (or bad) example is the MySQL grammar in the ANTLR grammar repository (apologies to the Author if you read this note - this isn't an attack). Although I have improved its runtime performance drastically in the Go runtime, it still takes about a minute to parse complex select statements. As it is constructed, there are no magic answers. I will look in more detail at improvements for such parsers, such as not freeing any memory until the parse is finished (improved 100x in experiments). The best advice I can give is to put some effort in to the actual grammar itself. well-formed grammars will potentially see some huge improvements with this release. Badly formed grammars, not so much. ================================================ FILE: doc/go-target.md ================================================ # ANTLR4 Language Target, Runtime for Go ### Changes from ANTLR 4.12.0 Please see [Changes in ANTLR Go runtimes](go-changes.md), but in summary: - The Go runtime is now stored in the repo `antlr4-go/antlr` - change your import, remove the old location from `go.mod` and use `go get github.com/antlr4-go/antlr` - There are some new `@actions` for adding to the generated import statements and recognizer structure - The recognizer rules are no longer called via an interface, for performance reasons - Memory usage improvements - Performance improvements - Documentation in true Go format - Git tags now work correctly with go tools ### Removal of non v4 code Prior to the release of the v4 tagged runtime, the source code for the Go runtime module existed at `runtime/Go/antlr`, which is the pre-v4 version of the code, and also under `runtime/Go/antlr/v4`. If your project was not using modules, you could merely sync to the latest hash in the master branch and use the code. This has changed. As of the current release, the source code for the Go runtime module has been moved to its own repo in its own GitHub organization. As of now, you can still use the code without modules, but you must use the code in the repo at https://github.com/antlr4-go/antlr instead of the code in the main ANTLR repo. This is for historic reasons as the code was originally written before modules were a thing, and the go runtime source was - and the maintainer's version still is - a part of the monorepo that is `antlr/antlr4/...`. Note that I am unable to properly deprecate the go.mod in the non-V4 directory, for hte same reason that I cannot use tag the v4 module at this depth in the source tree. We strongly advise you to use modules, though it is not required. See below for more information. ANTLR Go Maintainer: [Jim Idle](https://github.com/jimidle) - Email: [jimi@idle.ws](mailto:jimi@idle.ws) ### First steps #### 1. Install ANTLR4 See: [The getting started guide](getting-started.md). #### 2. Get the Go ANTLR runtime Each target language for ANTLR has a runtime package for running a recognizer generated by ANTLR4. The runtime provides a common set of tools for using your parser/lexer. Note that if you have existing projects and have yet to replace the `v1.x.x` modules with the `v4` modules, then you can skip ahead to the section *Upgrading to v4 from earlier versions* The Go runtime uses modules and has a version path of `/v4` to stay in sync with the runtime versions of all the other runtimes and the tool itself. Setup is the same as any other module based project: ```bash $ cd mymodproject $ go mod init mymodproject ``` After which, you can use go get, to get the latest release version of the ANTLR v4 runtime using: ```bash go get github.com/antlr4-go/antlr ``` If your project was already using the v4 runtime from the main ANTLR repo, then you can upgrade to the latest release by removing the `github.com/antlr/antlr4/runtime/Go/antlr/v4` reference in your module, and changing the associated import in your project code. The following script may be useful in changing your imports: ```shell find . -type f \ -name '*.go' \ -exec sed -i -e 's,github.com/antlr/antlr4/runtime/Go/antlr/v4,github.com/antlr4-go/antlr/v4,g' {} \; ``` Note that the import package still imports with the final path as `antlr`, so only the import statement itself needs to change. If you are already using the repo and import `github.com/antlr4-go/antlr/v4` then you can upgrade to the latest version using the standard. ```shell go get -u github.com/antlr4-go/antlr ``` If you have not yet upgraded existing projects to the `/v4` module path, consult the section *Upgrading to v4 from earlier versions* The ANTLR runtime has only one external transient dependency, and that is part of the go system itself: ``` golang.org/x/exp ``` A complete list of releases can be found on [the release page](https://github.com/antlr/antlr4/releases). The Go runtime will be tagged using standard Go tags, so release 4.13.2 in the `antlr4-go/antlr` repo, will be tagged with `v4.13.2` and go get will pick that up from the ANTLR repo. #### 3. Configuring `go generate` in your project In order to promote the use of repeatable builds, it is often useful to add the latest tool jar to your project's repo and configure a `generate.sh` and `generate.go` file. You can of course globally alias the java command required to run the tool. Your own CI and dev environment will guide you. Here is how you can configure `go generate` for your project, assuming that you follow the general recommendation to place the ANTLR grammar files in their own package in your project structure. Here is a general template as a starting point: ``` . ├── myproject ├── parser │ ├── mygrammar.g4 │ ├── antlr-4.13.2-complete.jar │ ├── generate.go │ └── generate.sh ├── parsing # Generated code goes here │ └── error_listeners.go ├── go.mod ├── go.sum ├── main.go └── main_test.go ``` Make sure that the package statement in your grammar file(s) reflects the go package the go code will be generated in. The `generate.go` file then looks like this: ```golang package parser //go:generate ./generate.sh ``` And the `generate.sh` file will look similar to this: ```shell #!/bin/sh alias antlr4='java -Xmx500M -cp "./antlr-4.13.2-complete.jar:$CLASSPATH" org.antlr.v4.Tool' antlr4 -Dlanguage=Go -no-visitor -package parsing *.g4 ``` From the command line at the root of your package - the location of the `go.mod` file - you can then simply issue the command: ```shell go generate ./... ``` If you have not yet run a `go get`, you can now run `go mod tidy` and update your #### 4. Generate your parser manually You use the ANTLR4 "tool" to generate a parser. These will reference the ANTLR runtime, installed above. Suppose you're using a UNIX system and have set up an alias for the ANTLR4 tool as described in [the getting started guide](getting-started.md). To generate your go parser, you'll need to invoke: ```shell antlr4 -Dlanguage=Go MyGrammar.g4 ``` For a full list of antlr4 tool options, please visit the [tool documentation page](tool-options.md). ### Upgrading to `/v4` from the default path *NB: While switching to new module path would normally imply that the public interface for the runtime has changed, this is not actually the case - you will not need to change your existing code to upgrade. The main point of the repo change is so that git tagging works with the ANTLR Go runtime and the go tools.* Prior to release v4.11.0 the Go runtime shipped with a module but the module had no version path. This meant that the tags in the ANTLR repo did not work, as any tag above `v1` must refer to a matching module path. So the command `go get github.com/antlr/antlr4/runtime/Go/antlr` would just bring in whatever was the `HEAD` of the master branch. While this *kind of* worked, it is obviously subject to problems and does not fit properly with the idiomatic ways of Go. As of v4.13.0 the runtime code exists in its own repo, `github.com/antlr4-go/antlr`, and is correctly tagged. However, this means you need to perform a few simple actions in order to upgrade to the `/v4` path. - Firstly, make sure that you are using an ANTLR tool jar with a version number of 4.13.0 or greater. - Next you replace any mention of the old (default) path to ANTLR in your go source files. - If using modules, remove any existing reference to the ANTLR Go runtime - Now regenerate your grammar files either manually or using `go generate ./...` (see above) - Consider whether you can move to using modules in your project A quick way to replace the original module path references is to use this script from your module's base directory: ```shell find . -type f \ -name '*.go' \ -exec sed -i -e 's,github.com/antlr/antlr4/runtime/Go/antlr,github.com/antlr4-go/antlr/v4,g' {} \; ``` After performing the steps above, and you are using modules issuing: ```shell go mod tidy ``` Should fix up your `go.mod` file to reference only the `v4` version of the ANTLR Go runtime: ```shell require github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.13.0 ``` From this point on, your go mod commands will work correctly with the ANTLR repo and upgrades and downgrades will work as you expect. As will branch version such as @dev ### Referencing the Go ANTLR runtime You can reference the go ANTLR runtime package like this: ```golang import "github.com/antlr4-go/antlr/v4" ``` ### Complete example Suppose you're using the JSON grammar from https://github.com/antlr/grammars-v4/tree/master/json placed in the parser directory and have initialized your `go mod` file. Then, invoke `antlr4 -Dlanguage=Go JSON.g4`. The result of this is a collection of .go files in the `parser` directory including: ``` json_parser.go json_base_listener.go json_lexer.go json_listener.go ``` Another common option to the ANTLR tool is `-visitor`, which generates a parse tree visitor, but we won't be doing that here. For a full list of antlr4 tool options, please visit the [tool documentation page](tool-options.md). We'll write a small main func to call the generated parser/lexer (assuming they are separate). This one writes out the encountered `ParseTreeContext`'s. Assuming the generated parser code is in the `parser` directory relative to this code: ```golang package main import ( "github.com/antlr4-go/antlr/v4" "./parser" // Note that with modules you may not be able to use a relative immport path "os" "fmt" ) type TreeShapeListener struct { *parser.BaseJSONListener } func NewTreeShapeListener() *TreeShapeListener { return new(TreeShapeListener) } func (this *TreeShapeListener) EnterEveryRule(ctx antlr.ParserRuleContext) { fmt.Println(ctx.GetText()) } func main() { input, _ := antlr.NewFileStream(os.Args[1]) lexer := parser.NewJSONLexer(input) stream := antlr.NewCommonTokenStream(lexer,0) p := parser.NewJSONParser(stream) p.AddErrorListener(antlr.NewDiagnosticErrorListener(true)) tree := p.Json() antlr.ParseTreeWalkerDefault.Walk(NewTreeShapeListener(), tree) } ``` Fix up your `go.mod` file: ```shell go mod tidy ``` This one expects the input to be passed on the command line: ``` go run test.go input ``` The output is: ``` {"a":1} {"a":1} "a":1 1 ``` ================================================ FILE: doc/grammars.md ================================================ # Grammar Structure A grammar is essentially a grammar declaration followed by a list of rules, but has the general form: ``` /** Optional javadoc style comment */ grammar Name; ① options {...} import ... ; tokens {...} channels {...} // lexer only @actionName {...} rule1 // parser and lexer rules, possibly intermingled ... ruleN ``` The file name containing grammar `X` must be called `X.g4`. You can specify options, imports, token specifications, and actions in any order. There can be at most one each of options, imports, and token specifications. All of those elements are optional except for the header ① and at least one rule. Rules take the basic form: ``` ruleName : alternative1 | ... | alternativeN ; ``` Parser rule names must start with a lowercase letter and lexer rules must start with a capital letter. Grammars defined without a prefix on the `grammar` header are combined grammars that can contain both lexical and parser rules. To make a parser grammar that only allows parser rules, use the following header. ``` parser grammar Name; ... ``` And, naturally, a pure lexer grammar looks like this: ``` lexer grammar Name; ... ``` Only lexer grammars can contain `mode` specifications. Only lexer grammars can contain custom channels specifications ``` channels { WHITESPACE_CHANNEL, COMMENTS_CHANNEL } ``` Those channels can then be used like enums within lexer rules: ``` WS : [ \r\t\n]+ -> channel(WHITESPACE_CHANNEL) ; ``` Sections 15.5, [Lexer Rules](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference) and Section 15.3, [Parser Rules](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference) contain details on rule syntax. Section 15.8, Options describes grammar options and Section 15.4, Actions and Attributes has information on grammar-level actions. ## Grammar Imports Grammar `imports` let you break up a grammar into logical and reusable chunks, as we saw in [Importing Grammars](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference). ANTLR treats imported grammars very much like object-oriented programming languages treat superclasses. A grammar inherits all of the rules, tokens specifications, and named actions from the imported grammar. Rules in the “main grammar” override rules from imported grammars to implement inheritance. Think of `import` as more like a smart include statement (which does not include rules that are already defined). The result of all imports is a single combined grammar; the ANTLR code generator sees a complete grammar and has no idea there were imported grammars. To process a main grammar, the ANTLR tool loads all of the imported grammars into subordinate grammar objects. It then merges the rules, token types, and named actions from the imported grammars into the main grammar. In the diagram below, the grammar on the right illustrates the effect of grammar `MyELang` importing grammar `ELang`. `MyELang` inherits rules `stat`, `WS`, and `ID`, but overrides rule `expr` and adds `INT`. Here’s a sample build and test run that shows `MyELang` can recognize integer expressions whereas the original `ELang` can’t. The third, erroneous input statement triggers an error message that also demonstrates the parser was looking for `MyELang`’s expr not `ELang`’s. ``` $ antlr4 MyELang.g4 $ javac MyELang*.java $ grun MyELang stat => 34; => a; => ; => EOF <= line 3:0 extraneous input ';' expecting {INT, ID} ``` If there are modes in the main grammar or any of the imported grammars then the import process will import those modes and merge their rules where they are not overridden. In the event any mode becomes empty as all its rules have been overridden by rules outside the mode this mode will be discarded. If there were any `tokens` specifications, the main grammar would merge the token sets. If there were any `channel` specifications, the main grammar would merge the channel sets. Any named actions such as `@members` would be merged. In general, you should avoid named actions and actions within rules in imported grammars since that limits their reuse. ANTLR also ignores any options in imported grammars. Imported grammars can also import other grammars. ANTLR pursues all imported grammars in a depth-first fashion. If two or more imported grammars define rule `r`, ANTLR chooses the first version of `r` it finds. In the following diagram, ANTLR examines grammars in the following order `Nested`, `G1`, `G3`, `G2`. `Nested` includes the `r` rule from `G3` because it sees that version before the `r` in `G2`. Not every kind of grammar can import every other kind of grammar: * Lexer grammars can import lexers, including lexers containing modes. * Parsers can import parsers. * Combined grammars can import parsers or lexers without modes. ANTLR adds imported rules to the end of the rule list in a main lexer grammar. That means lexer rules in the main grammar get precedence over imported rules. For example, if a main grammar defines rule `IF : 'if' ;` and an imported grammar defines rule `ID : [a-z]+ ;` (which also recognizes `if`), the imported `ID` won’t hide the main grammar’s `IF` token definition. ## Tokens Section The purpose of the `tokens` section is to define token types needed by a grammar for which there is no associated lexical rule. The basic syntax is: ``` tokens { Token1, ..., TokenN } ``` Most of the time, the tokens section is used to define token types needed by actions in the grammar as shown in Section 10.3, [Recognizing Languages whose Keywords Aren’t Fixed](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference): ``` // explicitly define keyword token types to avoid implicit definition warnings tokens { BEGIN, END, IF, THEN, WHILE } @lexer::members { // keywords map used in lexer to assign token types Map keywords = new HashMap() {{ put("begin", KeywordsParser.BEGIN); put("end", KeywordsParser.END); ... }}; } ``` The `tokens` section really just defines a set of tokens to add to the overall set. ``` $ cat Tok.g4 grammar Tok; tokens { A, B, C } a : X ; $ antlr4 Tok.g4 warning(125): Tok.g4:3:4: implicit definition of token X in parser $ cat Tok.tokens A=1 B=2 C=3 X=4 ``` ## Actions at the Grammar Level Currently there are only two defined named actions (for the Java target) used outside of grammar rules: `header` and `members`. The former injects code into the generated recognizer class file, before the recognizer class definition, and the latter injects code into the recognizer class definition, as fields and methods. For combined grammars, ANTLR injects the actions into both the parser and the lexer. To restrict an action to the generated parser or lexer, use `@parser::name` or `@lexer::name`. Here’s an example where the grammar specifies a package for the generated code: ``` grammar Count; @header { package foo; } @members { int count = 0; } list @after {System.out.println(count+" ints");} : INT {count++;} (',' INT {count++;} )* ; INT : [0-9]+ ; WS : [ \r\t\n]+ -> skip ; ``` The grammar itself then should be in directory `foo` so that ANTLR generates code in that same `foo` directory (at least when not using the `-o` ANTLR tool option): ``` $ cd foo $ antlr4 Count.g4 # generates code in the current directory (foo) $ ls Count.g4 CountLexer.java CountParser.java Count.tokens CountLexer.tokens CountBaseListener.java CountListener.java $ javac *.java $ cd .. $ grun foo.Count list => 9, 10, 11 => EOF <= 3 ints ``` The Java compiler expects classes in package `foo` to be in directory `foo`. ================================================ FILE: doc/index.md ================================================ # ANTLR 4 Documentation Please check [Frequently asked questions (FAQ)](faq/index.md) before asking questions on stackoverflow or antlr-discussion list. Notes:
  • To add to or improve this documentation, fork the antlr/antlr4 repo then update this `doc/index.md` or file(s) in that directory. Submit a pull request to get your changes incorporated into the main repository. Do not mix code and documentation updates in the sample pull request. You must sign the contributors.txt certificate of origin with your pull request if you've not done so before.
  • Copyright © 2012, The Pragmatic Bookshelf. Pragmatic Bookshelf grants a nonexclusive, irrevocable, royalty-free, worldwide license to reproduce, distribute, prepare derivative works, and otherwise use this contribution as part of the ANTLR project and associated documentation.
  • Much of this text was copied with permission from the The Definitive ANTLR 4 Reference, though it is being morphed over time as the tool changes.
Links in the documentation refer to various sections of the book but have been redirected to the general book page on the publisher's site. There are two excerpts on the publisher's website that might be useful to you without having to purchase the book: [Let's get Meta](http://media.pragprog.com/titles/tpantlr2/picture.pdf) and [Building a Translator with a Listener](http://media.pragprog.com/titles/tpantlr2/listener.pdf). You should also consider reading the following books (the vid describes the reference book): This documentation is a reference and summarizes grammar syntax and the key semantics of ANTLR grammars. The source code for all examples in the book, not just this chapter, are free at the publisher's website. The following video is a general tour of ANTLR 4 and includes a description of how to use parse tree listeners to process Java files easily: For those using Java, here's a great [set of ANTLR in Intellij notes](https://docs.google.com/document/d/1gQ2lsidvN2cDUUsHEkT05L-wGbX5mROB7d70Aaj3R64/edit#heading=h.xr0jj8vcdsgc) by Andreas Stefik. ## Sections * [Getting Started with ANTLR v4](getting-started.md) * [Grammar Lexicon](lexicon.md) * [Grammar Structure](grammars.md) * [Parser Rules](parser-rules.md) * [Left-recursive rules](left-recursion.md) * [Actions and Attributes](actions.md) * [Lexer Rules](lexer-rules.md) * [Wildcard Operator and Nongreedy Subrules](wildcard.md) * [Parse Tree Listeners](listeners.md) * [Parse Tree Matching and XPath](tree-matching.md) * [Semantic Predicates](predicates.md) * [Options](options.md) * [ANTLR Tool Command Line Options](tool-options.md) * [Runtime Libraries and Code Generation Targets](targets.md) * [Unicode U+FFFF, U+10FFFF character streams](unicode.md) * [Parsing binary streams](parsing-binary-files.md) * [Parser and lexer interpreters](interpreters.md) * [Writing target-agnostic grammars](target-agnostic-grammars.md) * [Resources](resources.md) # Building / releasing ANTLR itself * [Building ANTLR itself](building-antlr.md) * [Contributing to ANTLR](/CONTRIBUTING.md) * [Cutting an ANTLR Release](releasing-antlr.md) * [ANTLR project unit tests](antlr-project-testing.md) * [Creating an ANTLR Language Target](creating-a-language-target.md) ================================================ FILE: doc/interpreters.md ================================================ # Parser and Lexer Interpreters *Since ANTLR 4.2* For small parsing tasks it is sometimes convenient to use ANTLR in interpreted mode, rather than generating a parser in a particular target, compiling it and running it as part of your application. Here's some sample code that creates lexer and parser Grammar objects and then creates interpreters. Once we have a ParserInterpreter, we can use it to parse starting in any rule we like, given a rule index (which the grammar + the parser can provide). ## Action Code Since interpreters don't use generated parsers + lexers they cannot execute any action code (including predicates). That means the interpreter runs as if there were no predicates at all. If your grammar requires action code in order to parse correctly you will not be able to test it using this approach. ## Java Target Interpreter Setup ```java LexerGrammar lg = new LexerGrammar( "lexer grammar L;\n" + "A : 'a' ;\n" + "B : 'b' ;\n" + "C : 'c' ;\n"); Grammar g = new Grammar( "parser grammar T;\n" + "s : (A|B)* C ;\n", lg); LexerInterpreter lexEngine = lg.createLexerInterpreter(new ANTLRInputStream(input)); CommonTokenStream tokens = new CommonTokenStream(lexEngine); ParserInterpreter parser = g.createParserInterpreter(tokens); ParseTree t = parser.parse(g.rules.get(startRule).index); ``` You can also load combined grammars from a file: ```java public static ParseTree parse(String fileName, String combinedGrammarFileName, String startRule) throws IOException { final Grammar g = Grammar.load(combinedGrammarFileName); LexerInterpreter lexEngine = g.createLexerInterpreter(CharStreams.fromPath(Paths.get(fileName))); CommonTokenStream tokens = new CommonTokenStream(lexEngine); ParserInterpreter parser = g.createParserInterpreter(tokens); ParseTree t = parser.parse(g.getRule(startRule).index); System.out.println("parse tree: "+t.toStringTree(parser)); return t; } ``` Then: ```java ParseTree t = parse("T.om", MantraGrammar, "compilationUnit"); ``` To load separate lexer/parser grammars, do this: ```java public static ParseTree parse(String fileNameToParse, String lexerGrammarFileName, String parserGrammarFileName, String startRule) throws IOException { final LexerGrammar lg = (LexerGrammar) Grammar.load(lexerGrammarFileName); final Grammar pg = Grammar.load(parserGrammarFileName, lg); CharStream input = CharStreams.fromPath(Paths.get(fileNameToParse)); LexerInterpreter lexEngine = lg.createLexerInterpreter(input); CommonTokenStream tokens = new CommonTokenStream(lexEngine); ParserInterpreter parser = pg.createParserInterpreter(tokens); ParseTree t = parser.parse(pg.getRule(startRule).index); System.out.println("parse tree: " + t.toStringTree(parser)); return t; } ``` Then: ```java ParseTree t = parse(fileName, XMLLexerGrammar, XMLParserGrammar, "document"); ``` This is also how we will integrate instantaneous parsing into ANTLRWorks2 and development environment plug-ins. See [TestParserInterpreter.java](../tool-testsuite/test/org/antlr/v4/test/tool/TestParserInterpreter.java). ## Non-Java Target Interpreter Setup The ANTLR4 runtimes do not contain any grammar parsing classes (they are in the ANTLR4 tool jar). Hence we cannot use `LexerGrammar` and `Grammar` to parse grammars for the interpreter. Instead we directly instantiate `LexerInterpreter` and `ParserInterpreter` objects. They require some data (namely symbol information and the ATNs) which only the ANTLR4 tool can give us. However, on each generation run ANTLR not only produces your parser + lexer files but also interpreter data files (*.interp) which contain all you need to feed the interpreters. A support class (`InterpreterDataReader`) is used to load the data for your convenience, which makes this very easy to use. Btw. even the Java target go this route instead of using the non-runtime classes `Grammar` and `LexerGrammar`. Sometimes it might not be feasible to use the tool jar for whatever reason. Here's how the setup looks like (C++ example): ```cpp /** * sourceFileName - name of the file with content to parse * lexerName - the name of your lexer (arbitrary, that's what is used in error messages) * parserName - ditto for the parser * lexerDataFileName - the lexer interpeter data file name (e.g. `/ExprLexer.interp`) * parserDataFileName - ditto for the parser (e.g. `/Expr.interp`) * startRule - the name of the rule to start parsing at */ void parse(std::string const& sourceFileName, std::string const& lexerName, std::string const& parserName, std::string const& lexerDataFileName, std::string const& parserDataFileName, std::string const& startRule) { InterpreterData lexerData = InterpreterDataReader::parseFile(lexerDataFileName); InterpreterData parserData = InterpreterDataReader::parseFile(parserDataFileName); ANTLRFileStream input(sourceFileName); LexerInterpreter lexEngine(lexerName, lexerData.vocabulary, lexerData.ruleNames, lexerData.channels, lexerData.modes, lexerData.atn, &input); CommonTokenStream tokens(&lexEngine); /* Remove comment to print the tokens. tokens.fill(); std::cout << "INPUT:" << std::endl; for (auto token : tokens.getTokens()) { std::cout << token->toString() << std::endl; } */ ParserInterpreter parser(parserName, parserData.vocabulary, parserData.ruleNames, parserData.atn, &tokens); tree::ParseTree *tree = parser.parse(parser.getRuleIndex(startRule)); std::cout << "parse tree: " << tree->toStringTree(&parser) << std::endl; } ``` ================================================ FILE: doc/java-target.md ================================================ # Java ## Development environments ### Intellij There is a very complete and useful plug-in for intellij 12-14, you can grab at the [download page](https://plugins.jetbrains.com/plugin/7358?pr=). Check the [plugin readme](https://github.com/antlr/intellij-plugin-v4) for feature set. Just go to the preferences and click on the "Install plug-in from disk..." button from this dialog box: Select the intellij-plugin-1.x.zip (or whatever version) file and hit okay or apply. It will ask you to restart the IDE. If you look at the plug-ins again, you will see: Also, I have prepared a [video](https://youtu.be/eW4WFgRtFeY) that will help you generate grammars and so on using ANTLR v4 in Intellij (w/o the plugin). ### Eclipse Edgar Espina has created an [eclipse plugin for ANTLR v4](https://marketplace.eclipse.org/content/antlr-ide). Features: Advanced Syntax Highlighting, Automatic Code Generation (on save), Manual Code Generation (through External Tools menu), Code Formatter (Ctrl+Shift+F), Syntax Diagrams, Advanced Rule Navigation between files (F3), Quick fixes. ### NetBeans Sam Harwell's [ANTLRWorks2](http://tunnelvisionlabs.com/products/demo/antlrworks) works also as a plug-in, not just a stand-alone tool built on top of NetBeans. ## Build systems ### ant ### mvn *Maven Plugin Reference* The reference pages for the latest version of the Maven plugin for ANTLR 4 can be found here: [http://www.antlr.org/api/maven-plugin/latest/index.html](http://www.antlr.org/api/maven-plugin/latest/index.html) *Walkthrough* This section describes how to create a simple Antlr 4 project and build it using maven. We are going to use the ArrayInit.g4 example from chapter 3 of the book, and bring it under maven. We will need to rename files and modify them. We will conclude by building a portable stand alone application. Generate the skeleton. To generate the maven skeleton, type these commands: ```bash mkdir SimpleAntlrMavenProject cd SimpleAntlrMavenProject mvn archetype:generate -DgroupId=org.abcd.examples -DartifactId=array-example -Dpackage=org.abcd.examples.ArrayInit -Dversion=1.0 # Accept all the default values cd array-example ``` Maven will ask a series of questions, simply accept the default answers by hitting enter. Move into the directory created by maven: ```bash cd array-example ``` We can use the find command to see the files created by maven: ```bash $ find . -type f ./pom.xml ./src/test/java/org/abcd/examples/ArrayInit/AppTest.java ./src/main/java/org/abcd/examples/ArrayInit/App.java ``` We need to edit the pom.xml file extensively. The App.java will be renamed to ArrayInit.java and will contain the main ANTLR java program which we will download from the book examples. The AppTest.java file will be renamed ArrayInitTest.java but will remain the empty test as created by maven. We will also be adding the grammar file ArrayInit.g4 from the book examples in there. Get the examples for the book and put them in the Downloads folder. To obtain the ArrayInit.g4 grammar from the book, simply download it: ```bash pushd ~/Downloads wget http://media.pragprog.com/titles/tpantlr2/code/tpantlr2-code.tgz tar xvfz tpantlr2-code.tgz popd ``` Copy the grammar to the maven project. The grammar file goes into a special folder under the src/ directory. The folder name must match the maven package name org.abcd.examples.ArrayInit. ```bash mkdir -p src/main/antlr4/org/abcd/examples/ArrayInit cp ~/Downloads/code/starter/ArrayInit.g4 src/main/antlr4/org/abcd/examples/ArrayInit ``` Copy the main program to the maven project. We replace the maven App.java file with the main java program from the book. In the book, that main program is called Test.java, we rename it to ArrayInit.java: ```bash # Remove the maven file rm ./src/main/java/org/abcd/examples/ArrayInit/App.java # Copy and rename the example from the book cp ~/Downloads/code/starter/Test.java ./src/main/java/org/abcd/examples/ArrayInit/ArrayInit.java ``` Spend a few minutes to read the main program. Notice that it reads the standard input stream. We need to remember this when we run the application. Edit the ArrayInit.java file. We need to add a package declaration and to rename the class. Edit the file ./src/main/java/org/abcd/examples/ArrayInit/ArrayInit.java in your favorite editor. The head of the file should look like this when you are done: ```java package org.abcd.examples.ArrayInit; import org.antlr.v4.runtime.*; import org.antlr.v4.runtime.tree.*; public class ArrayInit { ... ``` Edit the ArrayInitTest.java file. Maven creates a test file called AppTest.java, we need to rename it to match the name of our application: ```bash pushd src/test/java/org/abcd/examples/ArrayInit mv AppTest.java ArrayInitTest.java sed 's/App/ArrayInit/g' ArrayInitTest.java >ArrayInitTest.java.tmp mv ArrayInitTest.java.tmp ArrayInitTest.java popd ``` Edit the pom.xml file. Now we need to extensively modify the pom.xml file. The final product looks like this: ```xml 4.0.0 org.abcd.examples array-init 1.0 jar array-init http://maven.apache.org UTF-8 org.antlr antlr4-runtime 4.9.3 junit junit 4.13.1 org.apache.maven.plugins maven-compiler-plugin 3.1 1.8 1.8 org.antlr antlr4-maven-plugin 4.9.3 antlr4 maven-assembly-plugin jar-with-dependencies simple-command package attached ``` This concludes the changes we had to make. We can look at the list of files we have with the find command: ```bash $ find . -type f ./pom.xml ./src/test/java/org/abcd/examples/ArrayInit/ArrayInitTest.java ./src/main/antlr4/org/abcd/examples/ArrayInit/ArrayInit.g4 ./src/main/java/org/abcd/examples/ArrayInit/ArrayInit.java ``` Building a stand alone application. With all the files now in place, we can ask maven to create a standalone application. The following command does this: ```bash mvn package ``` Maven creates a self-contained jar file called target/array-init-1.0-jar-with-dependencies.jar. We can execute the jar file, but remember that it expects some input on the command line, which means the command will hang on the command line until we feed it some input: ```bash java -cp target/array-init-1.0-jar-with-dependencies.jar org.abcd.examples.ArrayInit.ArrayInit ``` And let's feed it the following input: ```bash {1,2,3} ^D ``` The ^D signals the end of the input to the standard input stream and gets the rest of the application going. You should see the following output: ```bash (init { (value 1) , (value 2) , (value 3) }) ``` You can also build a jar file without the dependencies, and execute it with a maven command instead: ```bash mvn install mvn exec:java -Dexec.mainClass=org.abcd.examples.ArrayInit.ArrayInit {1,2,3} ^D ``` ================================================ FILE: doc/javascript-target.md ================================================ # JavaScript ## Which browsers are supported? In theory, all browsers supporting ECMAScript 5.1. In practice, this target has been extensively tested against: * Firefox 34.0.5 * Safari 8.0.2 * Chrome 39.0.2171 * Explorer 11.0.3 The above tests were conducted using Selenium. No issue was found, so you should find that the runtime works pretty much against any recent JavaScript engine. ## Is NodeJS supported? The runtime has also been extensively tested against Node.js 14 LTS. No issue was found. NodeJS together with a packaging tool is now the preferred development path, developers are encouraged to follow it. ## What about modules? Starting with version 8.1, Antlr4 JavaScript runtime follows esm semantics (see https://tc39.es/ecma262/#sec-modules for details) Generated lexers, parsers, listeners and visitors also follow this new standard. If you have used previous versions of the runtime, you will need to migrate and make your parser a module. ## How to create a JavaScript lexer or parser? This is pretty much the same as creating a Java lexer or parser, except you need to specify the language target, for example: ```bash $ antlr4 -Dlanguage=JavaScript MyGrammar.g4 ``` For a full list of antlr4 tool options, please visit the [tool documentation page](tool-options.md). ## Where can I get the runtime? Once you've generated the lexer and/or parser code, you need to download the runtime. The JavaScript runtime is [available from npm](https://www.npmjs.com/package/antlr4). We will not document here how to refer to the runtime from your project, since this would differ a lot depending on your project type and IDE. ## How do I get the runtime in my browser? The runtime is quite big and is currently maintained in the form of around 50 scripts, which follow the same structure as the runtimes for other targets (Java, C#, Python...). This structure is key in keeping code maintainable and consistent across targets. However, it would be a bit of a problem when it comes to get it into a browser. Nobody wants to write 50 times: ```