[ { "path": ".gitignore", "content": "*.sw?\n*~\n*.jar\n*.so\n*.class\n*.pyc\n*.tgz\n*.gz\n*.lm\n*.3bo\n*.mod\nvocab.*\n!vocab.flm\n.project\n.classpath\nbin/ccg2xml.py\nbin/lex.py\nbin/yacc.py\nbin/ccg_editor.py\nbin/Tree.py\nccgbank/convert/\nccgbank/data/novel/two-sents.dir/\nccgbank/extract/*\n!ccgbank/extract/grammar.xml\n!ccgbank/extract/*.xsl\nccgbank/feats/hypertagger/\nccgbank/feats/parser/\nccgbank/feats/realizer/\nccgbank/feats/supertagger/\nccgbank/logs/\nccgbank/models/*/*dict*\nccgbank/models/realizer/excl/\nccgbank/original/corpus/\nccgbank/original/feats/postagger/\nccgbank/original/feats/supertagger/\nccgbank/original/logs/\nccgbank/propccgbank/\nccgbank/stanford-nlp/*.jar\nccgbank/stanford-nlp/classifiers/*.prop\ndocs/api/\ndocs/grammars-rough-guide.pdf\ndocs/realizer-manual.pdf\ndocs/guide/guide.*\n!docs/guide/guide.tex\ndocs/realizer/manual.*\n!docs/realizer/manual.tex\nlib/openccg.jar\noutput/\nsrc/ccg2xml/ccg2xml.py\nsrc/srilmbridge/*.h\n" }, { "path": "AUTHORS", "content": "Main Authors:\n Core Java Code:\n Jason Baldridge \n Gann Bierner \n Michael White \n CCG-to-XML:\n Ben Wing \n Hypertagger:\n Dominic Espinosa \n Supertagger:\n Dennis Mehay \n Disjunctivizer:\n Scott Martin \n\nAdditional Contributors:\n Jonathan Barker \n semantic graph visualization tool\n Cem Bozsahin \n grammars from Bozsahin and Steedman (2003)\n Gunes Erkan \n handling of type hierachies\n Dennis Mehay \n KenLM interface\n (in addition to supertagger)\n Scott Martin \n GrammarDoc\n incoporating the SRILM toolkit for scoring\n build process for CCGbank grammar extraction\n (in addition to disjunctivizer)\n Rajakrishnan Rajkumar \n build files and XSLT transforms for CCGbank grammar extraction\n English agreement model\n David Reitter \n command completion and per grammar history\n \"tiny\" grammar\n Alexandros Triantafyllidis \n visualization of derivations via latex\n Ben Wing \n wccg and WebCCG code; CCG-format grammars\n (in addition to ccg2xml)\n\n" }, { "path": "CHANGES", "content": "0.9.6 - ...\r\n-----------\r\n\r\n* Updated .gitignore, CHANGES and docs/index.html for transition to\r\n GitHub\r\n\r\n\r\n0.9.5 - dependency length minimization, disjunctivizer, KenLM\r\n-------------------------------------------------------------\r\n\r\n* Added features for dependency ordering and dependency length\r\n minimization in realization.\r\n\r\n* Added disjunctivizer package, for creating a disjunctive LF XML\r\n structure based on an LF graph difference.\r\n\r\n* Added support for using a very large 5-gram memory-mapped language\r\n model with KenLM on linux.\r\n\r\n* Added n-best parser output.\r\n\r\n* Added option for in-memory perceptron training.\r\n\r\n\r\n0.9.4 - broad coverage paraphrasing, CCGbank training\r\n-----------------------------------------------------\r\n\r\n* Added Hockenmaier-style generative probability model for parsing and\r\n realization.\r\n\r\n* Added supertagger and use of adaptive supertagging in parsing.\r\n\r\n* Added build files for CCGbank training, documented in\r\n docs/ccgbank-README, as well as ones for parsing and realizing\r\n novel text (thereby generating grammatical paraphrases).\r\n\r\n* Added release targets for CCGbank data and pre-built English models.\r\n\r\n* Added use of Stanford tokenizer, morphological analyzer and named\r\n entity recognizer in parsing novel text.\r\n\r\n* Added use of ordinary hashing for lex signs, so that signs that\r\n differ only in the pos tag can be distinguished (for robustness).\r\n\r\n* Added hypertagger input option and derivation history output to ccg-realize.\r\n\r\n* Added n-best realization output to ccg-test.\r\n\r\n* Added tracking of lex heads to signs via modifier attr on slashes.\r\n\r\n* Added gold standard pred info for training with hypertagger.\r\n\r\n* Added initial syntactic feature extractor.\r\n\r\n* Added caching of supertags in cats.\r\n\r\n* Added option to use word positions in converting atoms in the LFs, \r\n which is now the default. Added :nowordpos command in tccg to \r\n change the preference to the lexical naming option.\r\n \r\n* Changed tccg to also update Grammar.theGrammar.prefs, which seems \r\n to have fixed issue with :nosem option not working.\r\n \r\n* Refactored feature extraction to use a trie for representing features \r\n as a sequence of interned string keys, to allow for lazy feature \r\n extraction that more quickly filters features not in the alphabet.\r\n \r\n* Added serialization of signs.\r\n\r\n* Added python script for drawing derivs in .auto files as trees (uses NLTK).\r\n\r\n* Added cell pruning limit in realization.\r\n \r\n* Added support for 'magic tokens' (like numbers) in ccg2xml,\r\n contributed by vanjena@users.sourceforge.net.\r\n\r\n* Turned off caching of category hash codes b/c of problems with stale \r\n values (a method of checking for staleness might be added later).\r\n\r\n* Improved utf8 support (esp. for macs). Note that utf8 support seems\r\n hopelessly broken for the Windows command-line, in that none of the\r\n available terminal apps (including for cygwin) both display\r\n characters correctly and work with tccg. I/O to files works fine\r\n though.\r\n\r\n* Added xml escaping for bleu and nbest output.\r\n\r\n* Added ccg-draw-graph tool for visualizing semantic dependency\r\n graphs.\r\n\r\n \r\n0.9.3 - minor changes\r\n-------------------\r\n\r\n* Added runCommand method in Visualizer so that the latex\r\n visualization works on Linux\r\n\r\n* Added id info to test items and bleu output.\r\n\r\n* Changed default lex licensing feature to be last in Lexicon.loadLicensingFeatures.\r\n\r\n* Added loop for computing closure of licensed no sem edges in EdgeFactory.\r\n\r\n* Changed FeatureLicenser to unify feat strucs instead of cats.\r\n\r\n \r\n0.9.2 - VisCCG release plus initial hypertagger support\r\n-------------------------------------------------------\r\n* Added check for unary rule cycles in parser and realizer.\r\n\r\n* Added initial version of greedy fragment assembly in realization when \r\n a complete realization is not found.\r\n \r\n* Added case for composition of X/Y Y/Z where Y has arity 2.\r\n\r\n* Added option to filter rule apps by observed supercat-rule combos.\r\n\r\n* VisCCG: Please see the list of changes in the archives at http://comp.ling.utexas.edu/wiki/doku.php/openccg/dev\r\n\r\n* Added LexSemOrigin interface for tracking of origin of lexical \r\n predications back to a sign or unary type changing rule.\r\n\r\n* Removed unused LF in DataItem.\r\n\r\n* Added supertagger-based filtering to lexical lookup.\r\n\r\n* Upgraded to JDOM 1.1.\r\n\r\n* Upgraded parser to use ambiguity packing.\r\n\r\n* Added scoring and n-best pruning to parser.\r\n\r\n* Refactored SignScorer to synsem package, for shared use\r\n by the realizer and parser. NB: This may require minor\r\n refactoring of imports and recompilation of realizer clients.\r\n\r\n* Changed realizer to check instantiation of outermost args by \r\n default, thereby improving completness at minor cost to efficiency.\r\n Accordingly, renamed checkInstantiation flag in EdgeFactory to \r\n debugInstantiation, which now controls whether to report such \r\n cats to System.err.\r\n\r\n* Added hypertagger (realizer supertagger) interface and initial \r\n version of beta-best realization using it.\r\n \r\n* Changed Family.deriveSupertag to remove the semantic part of a \r\n cat name following a colon.\r\n\r\n\r\n0.9.1 - New tools: grammardoc, ccg2xml; other misc updates\r\n----------------------------------------------------------\r\n* Changed dateFormatNoYear to \"*.MM.dd\" to avoid ambiguity with \r\n numbers.\r\n \r\n* Changed Grammar.initializeTransformers to set indenting more robustly \r\n by adding try-catch blocks for illegal argument exceptions.\r\n \r\n* Refactored RuleGroup to apply unary and binary rules separately.\r\n\r\n* Refactored Lexicon and RuleGroup to load lex/morph/rule info incrementally, \r\n using a new XmlScanner utility class. These changes avoid the need to \r\n store large XML docs all in memory at once, while keeping the refactoring \r\n to a minimum.\r\n \r\n* Revised LF flattening to propapage the alts, opts & chunks based on \r\n the expression structure, rather than the graph structure. \r\n This change makes the 'shared' attribute (on nominal references) \r\n more transparent in how it works with disjunctions that operate on \r\n different levels of the tree.\r\n\r\n* Revised LF compaction to allow duplicate predications, where an attempt \r\n is made to attach them in different locations if possible.\r\n\r\n* Added GrammarDoc, which generates HTML documentation from a source grammar.\r\n See README, under `Generating Grammar Documentation' for more information.\r\n\r\n* Added initial version of ccg2xml, for specifying grammars in the \r\n more human-friendly .ccg format.\r\n \r\n* Changed build system\r\n - Made separate build files for ccg2xml and documentation\r\n - Made the `release' target of the main build file create a binary for\r\n distribution, instead of just the source\r\n \r\n \r\n0.9.0 - Disjunctive LFs\r\n-----------------------\r\n* Refactored realizer to put all no-sem edges on the agenda, \r\n which requires making an exception for edges with no indices \r\n in the implementation of the index filter, but otherwise \r\n yields a more uniform approach to creating edges.\r\n \r\n* Refactored realizer to use representative edges (one per cat) \r\n instead of edge groups, which ends up being simpler on the whole \r\n and should be easier to explain.\r\n \r\n* Refactored categories to allow for equality checks with and \r\n without taking the LFs into consideration.\r\n \r\n* Refactored edge equiv classes to use coverage bit vector \r\n and cat sans LF to check equality.\r\n \r\n* Refactored lex instantiation to produce all possible instantiations \r\n that respect the alt exclusivity constraints.\r\n \r\n* Changed Sign, DerivationHistory to store rule object.\r\n\r\n* Changed alt edge construction to create new LF from input signs and rule, \r\n since signs in equiv class of alts can now have different LFs.\r\n \r\n* Added active alt tracking and completing of edges with optional bits.\r\n\r\n* Changed HyloVar to check for equal types when checking for equality up \r\n to var renaming.\r\n \r\n* Refactored generics to avoid type warnings in Eclipse.\r\n\r\n* Relaxed LF chunking constraints to allow combinations with edges \r\n (or trackers more generally) that are shared across multiple \r\n alt set options.\r\n \r\n* Added \"shared\" attribute to nominal terms to indicate references \r\n to nodes that are shared across alternatives in a disjunctive LF; \r\n then revamped and reinforced the LF chunking constraints.\r\n \r\n* Fixed problem with signMap not pointing to opt-completed edge.\r\n\r\n* Improved edge printing from realizer chart to show derivations.\r\n\r\n* Updated realizer to keep edges whose signs have the simplest derivation, \r\n among those with the same surface words.\r\n \r\n* Added filter for ungrammatical test cases in ccg-test text output.\r\n\r\n* Added first draft of realizer manual.\r\n\r\n \r\n0.8.6 - Java 1.5 switch, n-gram scoring improvements\r\n----------------------------------------------------\r\n* Added propagation of reverse flag on n-gram models.\r\n\r\n* Refactored LinearNgramScorerCombo and n-gram models to \r\n support interpolation at the word level.\r\n\r\n* Added caching of log probs in NgramScorer, to avoid recomputing \r\n log prob of words for a sign's initial sign.\r\n\r\n* Added n-gram diversity pruning strategy.\r\n\r\n* Changed SignHash to only keep signs that are unique up to surface words, \r\n thereby ignoring different POS or supertags; also changed it to keep \r\n signs with lower derivational complexity during insertion.\r\n \r\n* Added reverse flag for loaded n-gram models with ccg-test, ccg-realize.\r\n\r\n* Fixed sentence delimiter text output for reversible standard n-gram models; \r\n made AAnFilter reversible.\r\n \r\n* Added Xalan 2.6.0 jars, to support Java 1.5 builds.\r\n\r\n* Added support for duration special tokens; note that the implementation has \r\n an unavoidable dependency on Java 1.5.\r\n \r\n \r\n0.8.5 - \"Rough Guide\", sem types, command history/completion, and more\r\n---------------------------------------------------------------------- \r\n* Added initial core-en/types.xml.\r\n\r\n* Generalized feature licensing to allow for selective listing of supertypes \r\n in the also-licensed-by attribute.\r\n \r\n* Fixed bug in unifying two vars with simple types.\r\n\r\n* Removed useless SignHash.values method; clarified intention to \r\n eventually remove this class.\r\n \r\n* Streamlined lexical access for realization.\r\n\r\n* Removed superfluous unique stamps in var classes.\r\n\r\n* Added support for using simple types (aka sorts) with semantic features \r\n and nominals. During category instantiation, a morph item's class is \r\n assigned to the nominal var(s) for the [*DEFAULT*] proposition, and\r\n the types of all nominal vars are then propagated to all other \r\n nominal vars with the same name, throughout the category. \r\n \r\n* Changed tokenizer keep-words-with-sem-classes option in grammar.xsd \r\n to replacement-sem-classes option, where all semantic classes to use \r\n in replacing words with sem classes for language models are listed. \r\n Also changed semantic class replacement routine to uppercase semantic \r\n class names.\r\n \r\n* Added initial sem types to core-en, comic, and flights grammars.\r\n\r\n* Fixed bug in constructing type hierarchies with multiple inheritance.\r\n\r\n* Added ccg-update tool, with initial task to add full words (pre-parsed) \r\n to the testbed file; also updated ccg-test to use the pre-parsed words \r\n when writing training text files.\r\n \r\n* Updated ccg-cvr tool to use full words when present.\r\n Also added filter to remove test item duplicates from \r\n cross-validation training sets.\r\n\r\n* Added reporting of mean reciprocal rank to ccg-test, as well \r\n as residual mean reciprocal rank, based on the cases that do \r\n not match the target exactly.\r\n\r\n* Updated ccg-cvr tool to work with factored language models.\r\n\r\n* Fixed null pointer exception in DefaultTokenizer.format, Word.setW methods.\r\n\r\n* Added timing of lex lookup to realization metrics.\r\n\r\n* Added David's JLine console support to tccg, with command completion and \r\n per grammar history.\r\n\r\n* Added handling of coarticulations in the lexicon.\r\n\r\n* Added caching of lex lookup during realization.\r\n\r\n* Updated to-apml.xsl to handle 'and' in multiword elements.\r\n\r\n* Updated visualizer to handle word lists and to ignore coarts.\r\n\r\n* Added repetition scorer, for discounting repetitive realizations.\r\n\r\n* Added scorer class, pruning strategy class options to ccg-realize.\r\n\r\n* Added workaround for saving command history correctly with Java 1.4 on Linux.\r\n \r\n* Added 'tiny' grammar.\r\n\r\n* Added grammars \"rough guide\".\r\n\r\n* Added supertag as another word attr.\r\n\r\n* Revamped LMs to use trie maps, for better speed & scalability.\r\n\r\n* Improved handling of nulls in FLMs.\r\n\r\n* Cleaned up word representations.\r\n\r\n* Added even/odd selection for scoring too in ccg-test.\r\n\r\n* Added -reverse and -scorer options to ccg-test.\r\n\r\n* Added reverse LM capability.\r\n\r\n* Made supertag attrs configurable.\r\n\r\n* Switched to JDOM 1.0.\r\n\r\n\r\n0.8.4 - Factored language models (initial support), packing/unpacking, and more\r\n--------------------------------------------------------------- \r\n* Added Alex's latex visualization of derivations\r\n (nb: launch of previewer works better on Windows than Linux)\r\n\r\n* Added customizable tokenization and expansion routines for \r\n dates/times/nums/amounts and other named entities.\r\n \r\n* Added -2apml option to ccg-test.\r\n\r\n* Added Word class and many related changes to tokenization.\r\n\r\n* Added -textf|-textfsc options to ccg-test, for writing files in the format \r\n expected by the SRILM toolkit for factored language models.\r\n\r\n* Updated copyright notices.\r\n\r\n* Changed ngram model to use canonical lists of words as keys, \r\n removing size restriction. \r\n \r\n* Added -aanfilter option to ccg-test, with an optional list of\r\n exceptions, which may be culled from bigram counts.\r\n\r\n* Added keep-words-with-sem-classes option to grammar.xsd, to \r\n specify exceptional semantic classes where the word form is also \r\n considered relevant for scoring models.\r\n \r\n NB: Also changed grammar.xsd to specify a custom tokenizer class name \r\n and/or keep-words-with-sem-classes on a separate \r\n tokenizer element.\r\n\r\n* Added support for factored language models with fixed backoff paths,\r\n arranged into families of models for different child variables,\r\n and with the option to have secondary models for shorter available\r\n histories. Also added corresponding -flm|-flmsc options to\r\n ccg-test.\r\n\r\n* Added option to do scoring in a second stage, starting from a packed\r\n representation.\r\n \r\n* Switch from cached combos to collected combos, making the anytime case\r\n more like the packed case.\r\n \r\n* Added compacting of gen forest when unpacking is turned off.\r\n\r\n* Added pretty-printing of regex-like gen forest.\r\n\r\n\r\n0.8.3 - New efficiency methods, Cem-* grammars, and more\r\n--------------------------------------------------------------- \r\n* Added grammars from Bozsahin and Steedman (2003).\r\n\r\n* Improved instantiation of unary rules, ensuring that the \r\n first pred is used for indexing, and fixing a bug whereby \r\n a rule indexed by a lex pred would be missed.\r\n \r\n* Added initial capability to use semantic classes in n-gram scoring, \r\n as shown in ccg-realize.\r\n \r\n* Added LF chunking rules, which yields the most dramatic improvement in\r\n efficiency. \r\n\r\n* Added systematic feature-based licensing and instantiation.\r\n\r\n* Added caching of category combinations.\r\n\r\n* Added labeling of the phrase in the XML output headed by the index\r\n associated with the + semantic feature.\r\n \r\n* Added feature filtering and LF indenting to tccg display options.\r\n\r\n* Added XML configuration of LF relation sorting.\r\n\r\n* Added :2tb (to testbed) command for adding the current parse to the \r\n testbed.\r\n \r\n* Fixed grammar loading so it no longer has to be from the current directory.\r\n\r\n* Made it possible to list a stem as a member of an open class family with a\r\n separate pred, without getting an entry with the default pred too.\r\n\r\n* Enabled indexRel to be declared at the level of entries or families.\r\n\r\n* Added prefs import/export to tccg.\r\n\r\n* Added ccg-cvr tool for cross-validating realizer.\r\n\r\n* Reconfigured ccg-test with various new switches. \r\n\r\n* Put feature licensing on a switch.\r\n\r\n* Made pruning strategy configurable.\r\n\r\n* Changed representation of coord to work better with \r\n chunking (though less concise).\r\n\r\n* Added option to stop realizer after new best time limit (past first \r\n complete realization) is exceeded, via :nbtl N command\r\n\r\n\r\n0.8.2 - Edge pruning during realization, XML/APML I/O, and more\r\n--------------------------------------------------------------- \r\n* Changed build to ccg-build, in bin directory; \r\n also added separate build.xml files to each sample \r\n grammar directory. This way, a call to ccg-build \r\n either builds the system or the current grammar, \r\n depending on what directory you're in.\r\n \r\n* Changed realizer to no longer allow unmatched \r\n attr preds (ie sem features). This way, the presence \r\n of certain sem features can be used to control realization \r\n choices, instead of requiring these features to always \r\n be present. To underspecify these choices, the idea \r\n is to eventually allow for their optional inclusion. \r\n\r\n* Added more options to turn settings off individually in tccg.\r\n\r\n* Enabled realizer to handle type changing rules with \r\n their own semantics in the result category.\r\n \r\n* Added configurable edge pruning per category during \r\n realization, which controls the number of edges with \r\n equivalent categories to keep in the chart.\r\n \r\n* Fixed unification bug by adding occurs checks to Dollar's fill \r\n method, needed at least in part b/c ArgStack doesn't quite \r\n implement Unifiable.\r\n \r\n* Replaced hashString with hashCode and equals up to var names, \r\n yielding a 4-5% improvement in efficiency.\r\n \r\n* Switched to grammar.xml file. If none exists, an attempt is \r\n made to load from the default files lexicon.xml, morph.xml and \r\n rules.xml. See grammar.xsd for format.\r\n \r\n* Added LF load/save from/to XML via a sequence of transformations \r\n specified in the grammar.xml file.\r\n\r\n* Added save-to-xml (:2xml) option for saving LFs to XML \r\n files from tccg.\r\n \r\n* Added save-to-apml (:2apml) option for saving last input string to \r\n APML files from tccg.\r\n \r\n* Updated parser to apply unary rules repeatedly.\r\n\r\n* Various updates to flights grammar, including use of FrameNet roles.\r\n \r\n\r\n0.8.1 - OpenCCG Release with XML Schemas (!)\r\n---------------------------------------------------- \r\nThis release adds XML Schema validation to the grammar build \r\nprocess, where the comments in the XML schemas also \r\nserve as reference documentation for the grammar formats (wahoo!).\r\nThe release also contains several bug fixes to the unification \r\nroutines, and a more substantial \"flights\" grammar with \r\nsemantic control over pitch accents and boundary tones.\r\n\r\n\r\n0.8.0 - First OpenNLP CCG Library Release \r\n---------------------------------------------------- \r\nReorganized directories and renamed packages and tools. \r\nAdded build target for worldcup sample grammar. \r\nRewrote scripts for simplicity and parallelism. \r\nCut out pre-processing components and any classes and \r\nlibraries that looked like dead wood. Started removing \r\nunnecessary interfaces.\r\n\r\n\r\nGrok 0.7.0 - Towards a CCG Realizer\r\n---------------------------------------------------- \r\nMike is taking over Grok development and repurposing it for primary \r\nuse as a CCG Realizer in limited domain dialogue systems. \r\nSee http://www.iccs.informatics.ed.ac.uk/~mwhite/White-Baldridge-ENLG-2003-to-appear.pdf \r\nfor a description of the effort so far.\r\nVersion 0.7.0 will be the last Grok release.\r\nAfter this version, Grok will be split into separately usable \r\nand separately developed OpenNLP components.\r\nTom Morton will be responsible for further development of the \r\npre-processing components.\r\nMike will be responsible for further development of the CCG \r\nparser and realizer.\r\n\r\n\r\nGrok 0.6.0 - Multi-Modal CCG\r\n---------------------------------------------------- \r\nFor more information, see Jason's dissertation available at:\r\nhttp://www.iccs.inf.ed.ac.uk/~jmb/dissertation\r\n\r\n\r\nSee Grok site for further history ...\r\n\r\n" }, { "path": "LICENSE", "content": "\nGNU LESSER GENERAL PUBLIC LICENSE\n\nVersion 2.1, February 1999\n\n(The master copy of this license lives on the GNU website.) \n\nCopyright (C) 1991, 1999 Free Software Foundation, Inc. 59 Temple Place, Suite\n330, Boston, MA 02111-1307 USA Everyone is permitted to copy and distribute\nverbatim copies of this license document, but changing it is not allowed.\n\n[This is the first released version of the Lesser GPL. It also counts as the\n successor of the GNU Library Public License, version 2, hence the version\n number 2.1.]\n\nPreamble\n\nThe licenses for most software are designed to take away your freedom to share\nand change it. By contrast, the GNU General Public Licenses are intended to\nguarantee your freedom to share and change free software--to make sure the\nsoftware is free for all its users.\n\nThis license, the Lesser General Public License, applies to some specially\ndesignated software packages--typically libraries--of the Free Software\nFoundation and other authors who decide to use it. You can use it too, but we\nsuggest you first think carefully about whether this license or the ordinary\nGeneral Public License is the better strategy to use in any particular case,\nbased on the explanations below.\n\nWhen we speak of free software, we are referring to freedom of use, not\nprice. Our General Public Licenses are designed to make sure that you have the\nfreedom to distribute copies of free software (and charge for this service if\nyou wish); that you receive source code or can get it if you want it; that you\ncan change the software and use pieces of it in new free programs; and that you\nare informed that you can do these things.\n\nTo protect your rights, we need to make restrictions that forbid distributors to\ndeny you these rights or to ask you to surrender these rights. These\nrestrictions translate to certain responsibilities for you if you distribute\ncopies of the library or if you modify it.\n\nFor example, if you distribute copies of the library, whether gratis or for a\nfee, you must give the recipients all the rights that we gave ou. You must make\nsure that they, too, receive or can get the source code. If you link other code\nwith the library, you must provide complete object files to the recipients, so\nthat they can relink them with the library after making changes to the library\nand recompiling it. And you must show them these terms so they know their\nrights.\n\nWe protect your rights with a two-step method: (1) we copyright the library, and\n(2) we offer you this license, which gives you legal permission to copy,\ndistribute and/or modify the library.\n\nTo protect each distributor, we want to make it very clear that there is no\nwarranty for the free library. Also, if the library is modified by someone else\nand passed on, the recipients should know that what they have is not the\noriginal version, so that the original author's reputation will not be affected\nby problems that might be introduced by others.\n\nFinally, software patents pose a constant threat to the existence of any free\nprogram. We wish to make sure that a company cannot effectively restrict the\nusers of a free program by obtaining a restrictive license from a patent\nholder. Therefore, we insist that any patent license obtained for a version of\nthe library must be consistent with the full freedom of use specified in this\nlicense.\n\nMost GNU software, including some libraries, is covered by the ordinary GNU\nGeneral Public License. This license, the GNU Lesser General Public License,\napplies to certain designated libraries, and is quite different from the\nordinary General Public License. We use this license for certain libraries in\norder to permit linking those libraries into non-free programs.\n\nWhen a program is linked with a library, whether statically or using a shared\nlibrary, the combination of the two is legally speaking a combined work, a\nderivative of the original library. The ordinary General Public License\ntherefore permits such linking only if the entire combination fits its criteria\nof freedom. The Lesser General Public License permits more lax criteria for\nlinking other code with the library.\n\nWe call this license the \"Lesser\" General Public License because it does Less to\nprotect the user's freedom than the ordinary General Public License. It also\nprovides other free software developers Less of an advantage over competing\nnon-free programs. These disadvantages are the reason we use the ordinary\nGeneral Public License for many libraries. However, the Lesser license provides\nadvantages in certain special circumstances.\n\nFor example, on rare occasions, there may be a special need to encourage the\nwidest possible use of a certain library, so that it becomes a de-facto\nstandard. To achieve this, non-free programs must be allowed to use the\nlibrary. A more frequent case is that a free library does the same job as widely\nused non-free libraries. In this case, there is little to gain by limiting the\nfree library to free software only, so we use the Lesser General Public License.\n\nIn other cases, permission to use a particular library in non-free programs\nenables a greater number of people to use a large body of free software. For\nexample, permission to use the GNU C Library in non-free programs enables many\nmore people to use the whole GNU operating system, as well as its variant, the\nGNU/Linux operating system.\n\nAlthough the Lesser General Public License is Less protective of the users'\nfreedom, it does ensure that the user of a program that is linked with the\nLibrary has the freedom and the wherewithal to run that program using a modified\nversion of the Library.\n\nThe precise terms and conditions for copying, distribution and modification\nfollow. Pay close attention to the difference between a \"work based on the\nlibrary\" and a \"work that uses the library\". The former contains code derived\nfrom the library, whereas the latter must be combined with the library in order\nto run.\n\nTERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION\n\n0. This License Agreement applies to any software library or other program which\ncontains a notice placed by the copyright holder or other authorized party\nsaying it may be distributed under the terms of this Lesser General Public\nLicense (also called \"this License\"). Each licensee is addressed as \"you\".\n\nA \"library\" means a collection of software functions and/or data prepared so as\nto be conveniently linked with application programs (which use some of those\nfunctions and data) to form executables.\n\nThe \"Library\", below, refers to any such software library or work which has been\ndistributed under these terms. A \"work based on the Library\" means either the\nLibrary or any derivative work under copyright law: that is to say, a work\ncontaining the Library or a portion of it, either verbatim or with modifications\nand/or translated straightforwardly into another language. (Hereinafter,\ntranslation is included without limitation in the term \"modification\".)\n\n\"Source code\" for a work means the preferred form of the work for making\nmodifications to it. For a library, complete source code means all the source\ncode for all modules it contains, plus any associated interface definition\nfiles, plus the scripts used to control compilation and installation of the\nlibrary.\n\nActivities other than copying, distribution and modification are not covered by\nthis License; they are outside its scope. The act of running a program using the\nLibrary is not restricted, and output from such a program is covered only if its\ncontents constitute a work based on the Library (independent of the use of the\nLibrary in a tool for writing it). Whether that is true depends on what the\nLibrary does and what the program that uses the Library does.\n\n1. You may copy and distribute verbatim copies of the Library's complete source\ncode as you receive it, in any medium, provided that you conspicuously and\nappropriately publish on each copy an appropriate copyright notice and\ndisclaimer of warranty; keep intact all the notices that refer to this License\nand to the absence of any warranty; and distribute a copy of this License along\nwith the Library.\n\nYou may charge a fee for the physical act of transferring a copy, and you may at\nyour option offer warranty protection in exchange for a fee.\n\n2. You may modify your copy or copies of the Library or any portion of it, thus\nforming a work based on the Library, and copy and distribute such modifications\nor work under the terms of Section 1 above, provided that you also meet all of\nthese conditions:\n\n a) The modified work must itself be a software library. \n b) You must cause the files modified to carry prominent notices stating\n that you changed the files and the date of any change. \n c) You must cause the whole of the work to be licensed at no charge to all\n third parties under the terms of this License. \n d) If a facility in the modified Library refers to a function or a table of\n data to be supplied by an application program that uses the facility, other\n than as an argument passed when the facility is invoked, then you must make\n a good faith effort to ensure that, in the event an application does not\n supply such function or table, the facility still operates, and performs\n whatever part of its purpose remains meaningful.\n\n (For example, a function in a library to compute square roots has a purpose\n that is entirely well-defined independent of the application. Therefore,\n Subsection 2d requires that any application-supplied function or table used\n by this function must be optional: if the application does not supply it,\n the square root function must still compute square roots.)\n\n These requirements apply to the modified work as a whole. If identifiable\n sections of that work are not derived from the Library, and can be\n reasonably considered independent and separate works in themselves, then\n this License, and its terms, do not apply to those sections when you\n distribute them as separate works. But when you distribute the same\n sections as part of a whole which is a work based on the Library, the\n distribution of the whole must be on the terms of this License, whose\n permissions for other licensees extend to the entire whole, and thus to\n each and every part regardless of who wrote it.\n\n Thus, it is not the intent of this section to claim rights or contest your\n rights to work written entirely by you; rather, the intent is to exercise\n the right to control the distribution of derivative or collective works\n based on the Library.\n\n In addition, mere aggregation of another work not based on the Library with\n the Library (or with a work based on the Library) on a volume of a storage\n or distribution medium does not bring the other work under the scope of\n this License.\n\n3. You may opt to apply the terms of the ordinary GNU General Public License\ninstead of this License to a given copy of the Library. To do this, you must\nalter all the notices that refer to this License, so that they refer to the\nordinary GNU General Public License, version 2, instead of to this License. (If\na newer version than version 2 of the ordinary GNU General Public License has\nappeared, then you can specify that version instead if you wish.) Do not make\nany other change in these notices.\n\nOnce this change is made in a given copy, it is irreversible for that copy, so\nthe ordinary GNU General Public License applies to all subsequent copies and\nderivative works made from that copy.\n\nThis option is useful when you wish to copy part of the code of the Library into\na program that is not a library.\n\n4. You may copy and distribute the Library (or a portion or derivative of it,\nunder Section 2) in object code or executable form under the terms of Sections 1\nand 2 above provided that you accompany it with the complete corresponding\nmachine-readable source code, which must be distributed under the terms of\nSections 1 and 2 above on a medium customarily used for software interchange.\n\nIf distribution of object code is made by offering access to copy from a\ndesignated place, then offering equivalent access to copy the source code from\nthe same place satisfies the requirement to distribute the source code, even\nthough third parties are not compelled to copy the source along with the object\ncode.\n\n5. A program that contains no derivative of any portion of the Library, but is\ndesigned to work with the Library by being compiled or linked with it, is called\na \"work that uses the Library\". Such a work, in isolation, is not a derivative\nwork of the Library, and therefore falls outside the scope of this License.\n\nHowever, linking a \"work that uses the Library\" with the Library creates an\nexecutable that is a derivative of the Library (because it contains portions of\nthe Library), rather than a \"work that uses the library\". The executable is\ntherefore covered by this License. Section 6 states terms for distribution of\nsuch executables.\n\nWhen a \"work that uses the Library\" uses material from a header file that is\npart of the Library, the object code for the work may be a derivative work of\nthe Library even though the source code is not. Whether this is true is\nespecially significant if the work can be linked without the Library, or if the\nwork is itself a library. The threshold for this to be true is not precisely\ndefined by law.\n\nIf such an object file uses only numerical parameters, data structure layouts\nand accessors, and small macros and small inline functions (ten lines or less in\nlength), then the use of the object file is unrestricted, regardless of whether\nit is legally a derivative work. (Executables containing this object code plus\nportions of the Library will still fall under Section 6.)\n\nOtherwise, if the work is a derivative of the Library, you may distribute the\nobject code for the work under the terms of Section 6. Any executables\ncontaining that work also fall under Section 6, whether or not they are linked\ndirectly with the Library itself.\n\n6. As an exception to the Sections above, you may also combine or link a \"work\nthat uses the Library\" with the Library to produce a work containing portions of\nthe Library, and distribute that work under terms of your choice, provided that\nthe terms permit modification of the work for the customer's own use and reverse\nengineering for debugging such modifications.\n\nYou must give prominent notice with each copy of the work that the Library is\nused in it and that the Library and its use are covered by this License. You\nmust supply a copy of this License. If the work during execution displays\ncopyright notices, you must include the copyright notice for the Library among\nthem, as well as a reference directing the user to the copy of this\nLicense. Also, you must do one of these things:\n\n a) Accompany the work with the complete corresponding machine-readable\n source code for the Library including whatever changes were used in the\n work (which must be distributed under Sections 1 and 2 above); and, if the\n work is an executable linked with the Library, with the complete\n machine-readable \"work that uses the Library\", as object code and/or source\n code, so that the user can modify the Library and then relink to produce a\n modified executable containing the modified Library. (It is understood that\n the user who changes the contents of definitions files in the Library will\n not necessarily be able to recompile the application to use the modified\n definitions.) \n\n b) Use a suitable shared library mechanism for linking with the Library. A\n suitable mechanism is one that (1) uses at run time a copy of the library\n already present on the user's computer system, rather than copying library\n functions into the executable, and (2) will operate properly with a\n modified version of the library, if the user installs one, as long as the\n modified version is interface-compatible with the version that the work was\n made with. \n\n c) Accompany the work with a written offer, valid for at least three years,\n to give the same user the materials specified in Subsection 6a, above, for\n a charge no more than the cost of performing this distribution. \n\n d) If distribution of the work is made by offering access to copy from a\n designated place, offer equivalent access to copy the above specified\n materials from the same place. \n\n e) Verify that the user has already received a copy of these materials or\n that you have already sent this user a copy.\n\nFor an executable, the required form of the \"work that uses the Library\" must\ninclude any data and utility programs needed for reproducing the executable from\nit. However, as a special exception, the materials to be distributed need not\ninclude anything that is normally distributed (in either source or binary form)\nwith the major components (compiler, kernel, and so on) of the operating system\non which the executable runs, unless that component itself accompanies the\nexecutable.\n\nIt may happen that this requirement contradicts the license restrictions of\nother proprietary libraries that do not normally accompany the operating\nsystem. Such a contradiction means you cannot use both them and the Library\ntogether in an executable that you distribute.\n\n7. You may place library facilities that are a work based on the Library\nside-by-side in a single library together with other library facilities not\ncovered by this License, and distribute such a combined library, provided that\nthe separate distribution of the work based on the Library and of the other\nlibrary facilities is otherwise permitted, and provided that you do these two\nthings:\n\n a) Accompany the combined library with a copy of the same work based on the\n Library, uncombined with any other library facilities. This must be\n distributed under the terms of the Sections above. \n\n b) Give prominent notice with the combined library of the fact that part of\n it is a work based on the Library, and explaining where to find the\n accompanying uncombined form of the same work.\n\n8. You may not copy, modify, sublicense, link with, or distribute the Library\nexcept as expressly provided under this License. Any attempt otherwise to copy,\nmodify, sublicense, link with, or distribute the Library is void, and will\nautomatically terminate your rights under this License. However, parties who\nhave received copies, or rights, from you under this License will not have their\nlicenses terminated so long as such parties remain in full compliance.\n\n9. You are not required to accept this License, since you have not signed\nit. However, nothing else grants you permission to modify or distribute the\nLibrary or its derivative works. These actions are prohibited by law if you do\nnot accept this License. Therefore, by modifying or distributing the Library (or\nany work based on the Library), you indicate your acceptance of this License to\ndo so, and all its terms and conditions for copying, distributing or modifying\nthe Library or works based on it.\n\n10. Each time you redistribute the Library (or any work based on the Library),\nthe recipient automatically receives a license from the original licensor to\ncopy, distribute, link with or modify the Library subject to these terms and\nconditions. You may not impose any further restrictions on the recipients'\nexercise of the rights granted herein. You are not responsible for enforcing\ncompliance by third parties with this License.\n\n11. If, as a consequence of a court judgment or allegation of patent\ninfringement or for any other reason (not limited to patent issues), conditions\nare imposed on you (whether by court order, agreement or otherwise) that\ncontradict the conditions of this License, they do not excuse you from the\nconditions of this License. If you cannot distribute so as to satisfy\nsimultaneously your obligations under this License and any other pertinent\nobligations, then as a consequence you may not distribute the Library at\nall. For example, if a patent license would not permit royalty-free\nredistribution of the Library by all those who receive copies directly or\nindirectly through you, then the only way you could satisfy both it and this\nLicense would be to refrain entirely from distribution of the Library.\n\nIf any portion of this section is held invalid or unenforceable under any\nparticular circumstance, the balance of the section is intended to apply, and\nthe section as a whole is intended to apply in other circumstances.\n\nIt is not the purpose of this section to induce you to infringe any patents or\nother property right claims or to contest validity of any such claims; this\nsection has the sole purpose of protecting the integrity of the free software\ndistribution system which is implemented by public license practices. Many\npeople have made generous contributions to the wide range of software\ndistributed through that system in reliance on consistent application of that\nsystem; it is up to the author/donor to decide if he or she is willing to\ndistribute software through any other system and a licensee cannot impose that\nchoice.\n\nThis section is intended to make thoroughly clear what is believed to be a\nconsequence of the rest of this License.\n\n12. If the distribution and/or use of the Library is restricted in certain\ncountries either by patents or by copyrighted interfaces, the original copyright\nholder who places the Library under this License may add an explicit\ngeographical distribution limitation excluding those countries, so that\ndistribution is permitted only in or among countries not thus excluded. In such\ncase, this License incorporates the limitation as if written in the body of this\nLicense.\n\n13. The Free Software Foundation may publish revised and/or new versions of the\nLesser General Public License from time to time. Such new versions will be\nsimilar in spirit to the present version, but may differ in detail to address\nnew problems or concerns.\n\nEach version is given a distinguishing version number. If the Library specifies\na version number of this License which applies to it and \"any later version\",\nyou have the option of following the terms and conditions either of that version\nor of any later version published by the Free Software Foundation. If the\nLibrary does not specify a license version number, you may choose any version\never published by the Free Software Foundation.\n\n14. If you wish to incorporate parts of the Library into other free programs\nwhose distribution conditions are incompatible with these, write to the author\nto ask for permission. For software which is copyrighted by the Free Software\nFoundation, write to the Free Software Foundation; we sometimes make exceptions\nfor this. Our decision will be guided by the two goals of preserving the free\nstatus of all derivatives of our free software and of promoting the sharing and\nreuse of software generally.\n\n NO WARRANTY \n\n15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE\nLIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED\nIN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE LIBRARY \"AS\nIS\" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT\nNOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A\nPARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE\nLIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME THE COST OF\nALL NECESSARY SERVICING, REPAIR OR CORRECTION.\n\n16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL\nANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE\nLIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL,\nSPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY\nTO USE THE LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING\nRENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF\nTHE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF SUCH HOLDER OR OTHER\nPARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.\n\n END OF TERMS AND CONDITIONS" }, { "path": "README.md", "content": "# OpenCCG\n\nOpenCCG is a system for parsing and generating text using [combinatory categorial grammar](https://en.wikipedia.org/wiki/Combinatory_categorial_grammar) for syntax and [hybrid logic dependency semantics](https://www.aclweb.org/anthology/P02-1041) for, well, the semantic representation.\n\nIf that seems like a mouthful, don't worry too much about the details right now.\nYou can get started [installing OpenCCG](https://davehowcroft.com/post/installing-openccg/) and [working with OpenCCG using the `tccg` utility](https://davehowcroft.com/post/getting-started-with-openccg/) right now.\n\nIf, on the other hand, you want to start understanding what that mouthful means, Johanna Moore at the University of Edinburgh has some [helpful course notes on NLG in general and OpenCCG in particular](https://www.inf.ed.ac.uk/teaching/courses/nlg/).\n\n# Project information\n\nSee CHANGES for a description of the project status. Also see the OpenCCG web site and wiki at UT Austin: \n\n* http://openccg.sf.net\n* http://www.utcompling.com/wiki/openccg\n\nThis `README.md` file contains the configuration and build instructions. Next you'll probably want to look at the tutorial on writing grammars in the human-friendly 'dot ccg' syntax on [the UT Austin OpenCCG wiki](http://www.utcompling.com/wiki/openccg/visccg-tutorial).\n\nAfter that it may be helpful to look at the \"native\" grammar specification in \"Specifying Grammars for OpenCCG: A Rough Guide\" in `docs/grammars-rough-guide.pdf`, as well as the `SAMPLE_GRAMMARS` file for descriptions of the sample grammars that come with the distribution, including ones using the DotCCG syntax. A (somewhat dated) programmer's guide to using the OpenCCG realizer appears in `docs/realizer-manual.pdf`.\n\nThis release also includes a broad English coverage grammar from the CCGBank and associated statistical models; see `docs/ccgbank-README` for details.\n\n\n# Requirements\n\n* Version 1.6 or later of the Java 2 SDK (http://java.sun.com)\n* For ccg2xml and other tools, Python version 2.4 to 2.7 (http://www.python.org)\n\n\n# Libraries\n\nIf you're working with the latest source version from GitHub, you'll need to download the external libraries from the latest release, as GitHub discourages including binaries in their repos:\n\n* Download the [latest release of OpenCCG from sourceforge](https://sourceforge.net/projects/openccg/)\n* Unpack the archive and copy over the files from `openccg/lib/`, as well as `openccg/ccgbank/bin/ner/NERApp.jar`\n* Build the latest source as described further below\n\n\n# Configuring your environment variables\n\nThe easiest thing to do is to set the environment variables `JAVA_HOME` and `OPENCCG_HOME` to the relevant locations on your system. Set `JAVA_HOME` to match the top level directory containing the Java installation you want to use.\n\nFor example, on Windows:\n\n```\nC:\\> set JAVA_HOME=C:\\Program Files\\jdk1.6.0_04\n```\n\nor on Unix:\n\n```\n% setenv JAVA_HOME /usr/local/java\n (csh)\n> export JAVA_HOME=/usr/java\n (ksh, bash)\n```\n\nOn Windows, to get these settings to persist, it's actually easiest to set your environment variables through the System Properties from the Control Panel. For example, under WinXP, go to Control Panel, click on System Properties, choose the Advanced tab, click on Environment Variables, and add your settings in the User variables area.\n\nNext, likewise set `OPENCCG_HOME` to be the top level directory where you unzipped the download. In Unix, type `pwd` in the directory where this file is and use the path given to you by the shell as `OPENCCG_HOME`. You can set this in the same manner as for `JAVA_HOME` above.\n\nNext, add the directory `OPENCCG_HOME/bin` to your path. For example, you can set the path in your `.bashrc` file as follows:\n\n```\n> export PATH=\"$PATH:$OPENCCG_HOME/bin\"\n```\n\nOn Windows, you should also add the python main directory to your path.\n\nFinally, if you are going to use [KenLM](https://kheafield.com/code/kenlm/) with very large language models for realization with CCGbank-extracted grammars on linux, you'll also need to set the library load path:\n\n```\n> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$OPENCCG_HOME/lib\n```\n\nOnce you have taken care of these things, you should be able to build and use the OpenCCG Library.\n\n**Note**: Spaces are allowed in `JAVA_HOME` but not in `OPENCCG_HOME`. To set an environment variable with spaces in it, you need to put quotes around the value when on Unix, but you must *NOT* do this when under Windows.\n\n\n# Increasing Java memory limit\n\nIf you're working with a broad coverage grammar and statistical parsing or realization models, you'll probably need to increase the default memory limit for running OpenCCG's tools. You can do so by editing `bin/ccg-env[.bat]`, increasing the JAVA_MEM environment variable at the end of this script. For training perceptron models in memory, you may need 16g; for realization with the very large gigaword 5-gram model, you may need 8g; otherwise, for parsing and realization with CCGbank-derived models, 4g or possibly even 2g should suffice; finally, for small grammars 512m or 256m should be ok.\n\n# Trying it out\n\nIf you've managed to configure the system, you should be able to change to the directory for the \"tiny\" sample grammar and run `tccg` (for text ccg), the command-line tool for interactively testing grammars:\n\n```\n> cd grammars\n> cd tiny\n> tccg (Windows/Unix)\n```\n\nProvided tccg starts properly, it loads the grammar files, parses them, and shows the command-line interface (at which point you can type `:h` for help or `:q` to quit).\n\nIf you trouble starting up tccg, make sure you have set the environment variables properly, and that the tccg script (located in `openccg/bin`) calls the right shell environment (top-line of the script; to solve the problem, either comment out this line or correct the path).\n\n\n# Visualizing semantic graphs\n\nSemantic dependency graphs in testbed files can be visualized with the help of Graphviz's dot tool. First, download and install [Graphviz](http://www.graphviz.org/). Then, use tccg to create a testbed files with logical forms in it. For example, you can try some examples in the worldcup sample grammar and save them to a file using the command ':2tb tb.xml'. Then make a directory to store the visualized graphs. Finally, run the ccg-draw-graph tool as shown below:\n\n```\n> cd grammars/worldcup\n> tccg (parse examples, save using ':2tb tb.xml')\n> mkdir graphs\n> ccg-draw-graph -i tb.xml -v graphs/g\n```\n\nYou can also show the semantic classes or word indices using the `-c` or `-w` options, respectively. The graphs can be displayed with any PDF display tool.\n\nNote that the graph visualization requires the logical forms to be stored in an xml node-rel format for graphs, as in the worldcup or routes sample grammars. See `SAMPLE_GRAMMARS` for more information.\n\n\n# Creating disjunctive logical forms\n\nThis release includes a new disjunctivizer package, for creating a disjunctive LF XML structure based on an LF graph difference. An LF graph difference is a characterization of the difference between two Hybrid Logic Dependency Semantics graphs and an alignment between them in terms of the edits needed to make one into the other: inserts, deletes, and substitutions. See the build file for junit tests that illustrate how to use the package.\n\n# Generating grammar documentation\n\nOpenCCG includes a tool for generating HTML documentation of the XML files that specify a grammar. It can be run either from the `ccg-grammardoc` script in the `bin/` directory, or as an Ant task. An example of how to incorporate GrammarDoc into an Ant build file is given in the \"tiny\" grammar (`grammars/tiny/build.xml`), in a build target called `document`.\n\n\n# Building the system from source\n\nThe OpenCCG build system is based on Apache Ant. Ant is a little but very handy tool that uses a build file written in XML (`build.xml`) as building instructions. Building the Java portion of OpenCCG is accomplished using the script `ccg-build`; this works under Windows and Unix, but requires that you run it from the top-level directory (where the `build.xml` file is located). If everything is right and all the required packages are visible, this action will generate a file called openccg.jar in the `./lib` directory.\n\nNote that you should *not* build from source by invoking 'ant' directly. Instead, you should use `ccg-build` as shown below (Unix), after ensuring that you've set `OPENCCG_HOME`, `JAVA_HOME` and updated your `PATH` (the `ccg-build` script invokes ant with various parameters that aren't set properly if ant is invoked from the command line):\n\n```\n> cd $OPENCCG_HOME\n> ccg-build\n```\n\n# Working with the Eclipse IDE\n\nThe Eclipse IDE can be used for editing the Java source code, though setup can be a bit tricky. The most reliable method seems to be as follows. First, follow the instructions above for building the source from the command line. Then, in Eclipse, choose File|New|Java Project to create a new Java Project, and give it a name, such as 'openccg'. Leave the default settings as they are, and click Next. Then choose Link Additional Source and browse to the folder `src/` in the directory where you installed OpenCCG (i.e. `$OPENCCG_HOME/src`). You'll need to give this location a new name, such as 'src2' ('src' is already taken by default). The final step is to Add External JARs under the Libraries tab. From OpenCCG's lib directory (i.e. `$OPENCCG_HOME/lib`), choose all of the `.jar` files. At this point, you should be able to hit Finish and the code should compile in Eclipse.\n\nNote that with Eclipse's default settings, the code will compile in your Eclipse workspace, which is separate from your OpenCCG installation (this is a good thing, as Eclipse uses a `bin/` directory for compiled Java classes, whereas OpenCCG uses `bin/` for command-line scripts). Thus, once you have made a round of changes in Eclipse and are ready to try them out in OpenCCG, go back to the command line in `$OPENCCG_HOME` and invoke `ccg-build` to re-build the `openccg.jar` file. This will make your changes available in OpenCCG's programs, such as `tccg`.\n\n# Bug Reports\n\nPlease report bugs at by creating [an issue with a description of the problem](https://github.com/OpenCCG/openccg/issues). \n\n" }, { "path": "SAMPLE_GRAMMARS", "content": " \nThis SAMPLE_GRAMMARS file describes the sample grammars that come \nwith the distribution, and provides an overview of how the grammars \nare organized.\n\nGrammars written directly in the XML format used by OpenCCG appear in\nseparate directories under grammars/. There are currently four small\nEnglish grammars -- tiny, worldcup, flights, and comic -- plus a series of\nrelated grammars, mini-*, for Basque, Dyirbal, English, Inuit, Tagalog and\nTurkish, which are from Bozsahin and Steedman's (2003) study of\nergativity. The worldcup grammar includes the English examples from\nBaldridge (2002). (The Dutch, Turkish, Tagalog, and Toba Batak grammars\nhave not been updated from Grok version 0.6.) The flights and comic\ngrammars (used in the FLIGHTS and COMIC systems) make use of a shared\ngrammar of core English, in the core-en dir, and contain categories for\npitch accents and boundary tones.\n\nGrammars written in the front-end `dot CCG' format, which attempts to provide \na more powerful and easier-to-use format than the raw XML, are in separate\ndirectories under ccg-format-grammars/. There are currently three grammars\nhere -- tiny, tinytiny, and arabic. `tiny' is a grammar originally based on\nthe `tiny' English grammar contained in the grammars/ directory and\ndocumented above. It has been significantly expanded so as to demonstrate\nthe various features of the CCG format. `tinytiny' is a smaller English\ngrammar extracted from `tiny', which attempts to demonstrate a minimal-size\nuseful grammar. `arabic' is a grammar of a large chunk of Classical Arabic,\nwritten by Ben Wing. It was created in particular to demonstrate the power\nof CCG-format macros in handling complex morphology, and contains a nearly\nfull grammar of Arabic verbs. Dot CCG grammars are compiled using ccg2xml; \nrun ccg2xml -h for usage.\n\nThe best place to look for more info on the dot CCG format is in\nccg-format-grammars/tiny/tiny.ccg and in src/ccg2xml/README.\n\nThis release also includes a broad English coverage grammar from the\nCCGBank and associated statistical models; see docs/ccgbank-README for\ndetails.\n\nNote that with all the grammars, there is the option to store logical\nforms in an xml node-rel format for graphs. Conversion to this graph\nformat is done using a couple of XSLT transforms specified in the\ngrammar.xml file; see grammars/worldcup/grammar.xml for an example.\nWhen using this graph format, it is also possible to visualize the\nsemantic graphs, as described in the main README file.\n\nAt present, ccg2xml does not support writing grammar.xml files with\nthe XSLT transforms for the node-rel graph format. As a workaround,\nyou can add these transforms to your own version of the file which you\nthen copy over the generated grammar.xml file, as shown below.\n\n> ccg2xml --prefix= mygram.ccg\n> cp mygram-grammar.xml grammar.xml\n" }, { "path": "TODO", "content": "\nGeneral OpenCCG development:\n----------------------------\n\n- Add check for target LF when adding/writing full words (incl. supertags).\n\n- Look into better handling of optional args.\n\n- Extend feature hierarchy biz to work with category types; \n would make sense to also add category vars.\n\n- Add final bits to grammars rough guide (esp. feature licensing).\n \n- Add option to update testbed.\n\n- Binding theory?\n \n- Get agreement to work with anaphors in appositives.\n\n- Add more dynamic checks, eg for non-existent indexRel values or licensing attrs.\n\n- Add well-formedness check for unique roles -- ie, that role must occur\n only once per semantic head -- and associated method for declaring that\n roles must be unique.\n\n- Interface to morph transducers.\n\n- Make UnifyControl etc thread friendly. Could try tying global vars to\n current thread.\n\n- Improve unification efficiency. Could try indexing, caching across calls\n to parser or realizer, and structure sharing with delayed copying. \n\n\n\nRealizer-oriented development:\n------------------------------\n\n- Try using coarticulations with pitch accents.\n\n- Add orthographic post-processing (capitalization, spacing of punctuation).\n\n- Look into instantiating outermost args. \n\n\n\n\n\n\n\n\nTODO prior to Feb 16:\n---------------------\n\nTabs for testbed, lexicon, features\nNice tree graph for features\nGet Arabic to compile\nIssues of colors, fonts, etc. (Alexis help)\nPNG's of the slash modalities\nOption menu for different magnification (50%, 100%, 200% ...)\nGet pretty buttons from Justin\nMake sure all path, etc. issues are working\nFigure out what's the deal with __init__.py -- importing from another dir\n\nPossibly:\n help-over descriptions of families, provided by Javadoc-style comments in\n the source\n\n\nAlexis -- help with more specific user-interface issues, overall management\ninvolving various people\n\nSudipta --\n -- PNG's of slash modalities; add to editor\n -- TeX/Tk font conversion\n -- find a Tkinter package for displaying tree graphs; use it to add a graph for features to the editor\n -- figure out what's wrong with arabic?\n\nsemantics: RED\nfeatures: BLUE\ncategories: BLACK, sans serif font\nbackground: WHITE\n\n\nBen TODO!!!!!\n-------------\n\nCreate professional-looking web page off of comp.ling.utexas.edu\n" }, { "path": "bin/ccg-build", "content": "#!/bin/sh\n. ccg-env\nANT_HOME=\"$OPENCCG_LIB\"\nPROPS=\"-Dant.home=$ANT_HOME -Dopenccg.home=$OPENCCG_HOME\"\ncase `uname` in\n CYGWIN* ) XALAN_JARS=\"$OPENCCG_LIB/xalan.jar;$OPENCCG_LIB/xercesImpl.jar;$OPENCCG_LIB/xml-apis.jar;$OPENCCG_LIB/xsltc.jar;$OPENCCG_LIB/serializer.jar\"\n ANT_JARS=\"$OPENCCG_LIB/ant.jar;$OPENCCG_LIB/ant-launcher.jar;$OPENCCG_LIB/ant-contrib.jar\"\n ANT_JARS=\"$ANT_JARS;$OPENCCG_LIB/ant-junit.jar;$OPENCCG_LIB/ant-junit4.jar;$OPENCCG_LIB/junit-4.10.jar\"\n CP=\"$JAVA_HOME/lib/tools.jar;$OPENCCG_JAR;$ANT_JARS;$XALAN_JARS;$DIRLIBS;.\"\n PROPS=\"$PROPS -Dcygwin=true\"\n ;;\n * ) XALAN_JARS=\"$OPENCCG_LIB/xalan.jar:$OPENCCG_LIB/xercesImpl.jar:$OPENCCG_LIB/xml-apis.jar:$OPENCCG_LIB/xsltc.jar:$OPENCCG_LIB/serializer.jar\"\n ANT_JARS=\"$OPENCCG_LIB/ant.jar:$OPENCCG_LIB/ant-launcher.jar:$OPENCCG_LIB/ant-contrib.jar\"\n ANT_JARS=\"$ANT_JARS:$OPENCCG_LIB/ant-junit.jar:$OPENCCG_LIB/ant-junit4.jar:$OPENCCG_LIB/junit-4.10.jar\"\n CP=\"$JAVA_HOME/lib/tools.jar:$OPENCCG_JAR:$ANT_JARS:$XALAN_JARS:$DIRLIBS:.\"\n ;;\nesac\n\"$JAVA\" $JAVA_MEM -classpath \"$CP\" $PROPS org.apache.tools.ant.launch.Launcher $@\n" }, { "path": "bin/ccg-build.bat", "content": "@echo off\r\ncall ccg-env\r\nset ANT_HOME=%OPENCCG_LIB%\r\nset PROPS=-Dant.home=%ANT_HOME% -Dopenccg.home=%OPENCCG_HOME%\r\nset XALAN_JARS=%OPENCCG_LIB%\\xalan.jar;%OPENCCG_LIB%\\xercesImpl.jar;%OPENCCG_LIB%\\xml-apis.jar;%OPENCCG_LIB%\\xsltc.jar;%OPENCCG_LIB%\\serializer.jar\r\nset ANT_JARS=%OPENCCG_LIB%\\ant.jar;%OPENCCG_LIB%\\ant-launcher.jar;%OPENCCG_LIB%\\ant-contrib.jar\r\nset ANT_JARS=%ANT_JARS%;%OPENCCG_LIB%\\ant-junit.jar;%OPENCCG_LIB%\\ant-junit4.jar;%OPENCCG_LIB%\\junit-4.10.jar\r\nset CP=\"%JAVA_HOME%\\lib\\tools.jar\";%OPENCCG_JAR%;%ANT_JARS%;%XALAN_JARS%;%DIRLIBS%;.\r\n%JAVA% %JAVA_MEM% -classpath %CP% %PROPS% org.apache.tools.ant.launch.Launcher %*\r\n" }, { "path": "bin/ccg-cvr", "content": "#!/bin/sh\n# For usage, do: ccg-cvr -h\n. ccg-env\n\"$JAVA\" $JAVA_ARGS opennlp.ccg.test.CrossValidateRealizer $@\n\n" }, { "path": "bin/ccg-cvr.bat", "content": "@echo off\r\nrem For usage, do: ccg-cvr -h\r\ncall ccg-env\r\n%JAVA_CMD% opennlp.ccg.test.CrossValidateRealizer %* \r\n\r\n" }, { "path": "bin/ccg-draw-graph", "content": "#!/bin/sh\n\npython \"$OPENCCG_HOME/bin/dlf_parser.py\" \"$@\"\n" }, { "path": "bin/ccg-draw-graph.bat", "content": "@echo off\r\npython %OPENCCG_HOME%/bin/dlf_parser.py %*" }, { "path": "bin/ccg-draw-tree", "content": "#!/bin/sh\n\npython \"$OPENCCG_HOME/bin/ccg_draw_tree.py\" \"$@\"\n" }, { "path": "bin/ccg-draw-tree.bat", "content": "@echo off\r\npython %OPENCCG_HOME%/bin/ccg_draw_tree.py %*" }, { "path": "bin/ccg-env", "content": "#!/bin/sh\n# sets OpenCCG environment variables\n\nif [ \"$JAVA_HOME\" = \"\" ] ; then\n echo\n echo \"Error: JAVA_HOME not found in your environment.\"\n echo\n echo \"Please set the JAVA_HOME variable in your environment to match the\"\n echo \"location of the Java Virtual Machine you want to use.\"\n exit 1\nfi\n\nif [ \"$OPENCCG_HOME\" = \"\" ] ; then\n echo\n echo \"Error: OPENCCG_HOME not found in your environment.\"\n echo\n echo \"Please set the OPENCCG_HOME variable in your environment to match the\"\n echo \"location of your OpenNLP CCG Library distribution.\"\n exit 1\nfi\n\ncase `uname` in\n CYGWIN* ) OPENCCG_HOME=\"`cygpath -w $OPENCCG_HOME`\"\n ;;\nesac\n\nOPENCCG_LIB=\"$OPENCCG_HOME/lib\"\nOPENCCG_SRC=\"$OPENCCG_HOME/src\"\nOPENCCG_CLASSES=\"$OPENCCG_HOME/output/classes\"\nOPENCCG_JAR=\"$OPENCCG_HOME/lib/openccg.jar\"\ncase `uname` in\n CYGWIN* ) DIRLIBS=\"$OPENCCG_LIB/trove.jar;$OPENCCG_LIB/jdom.jar;$OPENCCG_LIB/jline.jar;$OPENCCG_LIB/jopt-simple.jar\"\n CP=\"${OPENCCG_JAR};${DIRLIBS};.\"\n ;;\n * ) DIRLIBS=\"$OPENCCG_LIB/trove.jar:$OPENCCG_LIB/jdom.jar:$OPENCCG_LIB/jline.jar:$OPENCCG_LIB/jopt-simple.jar\"\n CP=\"${OPENCCG_JAR}:${DIRLIBS}:.\"\n ;;\nesac\n# variant for use with 'build compile' option, if desired:\n#CP=\"${OPENCCG_CLASSES}:${OPENCCG_SRC}:${DIRLIBS}\"\nJAVA=\"$JAVA_HOME/bin/java\"\nJAVA_MEM=\"-Xmx256m\"\n#JAVA_MEM=\"-Xmx2048m\"\n#JAVA_MEM=\"-Xmx8g\"\n#JAVA_MEM=\"-Xmx16g\"\nJAVA_ARGS=\"$JAVA_MEM -classpath $CP -Dfile.encoding=UTF8\"\n" }, { "path": "bin/ccg-env.bat", "content": "@echo off\r\nrem sets OpenCCG environment variables\r\n\r\nif not exist \"%JAVA_HOME%\" goto no_JAVA_HOME\r\nif not exist \"%OPENCCG_HOME%\" goto no_OPENCCG_HOME\r\n\r\nset OPENCCG_LIB=%OPENCCG_HOME%\\lib\r\nset DIRLIBS=%OPENCCG_LIB%\\trove.jar;%OPENCCG_LIB%\\jdom.jar;%OPENCCG_LIB%\\jline.jar;%OPENCCG_LIB%\\jopt-simple.jar\r\nset XMLLIBS=%OPENCCG_LIB%\\xml-apis.jar;%OPENCCG_LIB%\\xercesImpl.jar;%OPENCCG_LIB%\\xalan.jar\r\nset OPENCCG_SRC=%OPENCCG_HOME%\\src\r\nset OPENCCG_CLASSES=%OPENCCG_HOME%\\output\\classes\r\nset OPENCCG_JAR=%OPENCCG_HOME%\\lib\\openccg.jar\r\nrem variant without XMLLIBS\r\nrem set CP=%OPENCCG_JAR%;%DIRLIBS%;.\r\nrem variant with XMLLIBS\r\nset CP=%OPENCCG_JAR%;%DIRLIBS%;%XMLLIBS%;.\r\nrem variant for use with 'build compile' option, if desired:\r\nrem set CP=%OPENCCG_CLASSES%;%OPENCCG_SRC%;%DIRLIBS%\r\nset JAVA=\"%JAVA_HOME%\\bin\\java\"\r\nset JAVA_MEM=-Xmx256m\r\nrem set JAVA_MEM=-Xmx2048m\r\nset JAVA_CMD=%JAVA% %JAVA_MEM% -classpath %CP% -Dfile.encoding=UTF8\r\n\r\ngoto end\r\n\r\n:no_JAVA_HOME\r\necho.\r\necho Error: JAVA_HOME not found in your environment.\r\necho.\r\necho Please set the JAVA_HOME variable in your environment to match the\r\necho location of the Java Virtual Machine you want to use.\r\necho.\r\nexit /b 1\r\n\r\n:no_OPENCCG_HOME\r\necho.\r\necho Error: OPENCCG_HOME not found in your environment.\r\necho.\r\necho Please set the OPENCCG_HOME variable in your environment to match the\r\necho location of your OpenNLP CCG Library distribution.\r\necho.\r\nexit /b 1\r\n\r\n:end\r\n" }, { "path": "bin/ccg-grammardoc", "content": "#!/bin/sh\n#\n# $Id: ccg-grammardoc,v 1.2 2006/12/03 17:14:23 mwhite14850 Exp $\n# Script to run grammardoc from the command line.\n# Author: Scott Martin (http://www.ling.osu.edu/~scott/)\n#\n# Usage: ccg-grammardoc [-s|--source sourceDir] [-d|--dest destDir]\n#\n. ccg-env\nANT_HOME=\"$OPENCCG_HOME/lib\"\ncase `uname` in\n CYGWIN* ) CP=\"$CP;$ANT_HOME/ant.jar\"\n ;;\n * ) CP=\"$CP:$ANT_HOME/ant.jar\"\n ;;\nesac\nJAVA_ARGS=\"-Xmx128m -classpath $CP\"\n\"$JAVA\" $JAVA_ARGS opennlp.ccg.grammardoc.GrammarDoc $@\n\n" }, { "path": "bin/ccg-grammardoc.bat", "content": "@echo off\r\nrem Usage: ccg-grammardoc [-s|--source sourceDir] [-d|--dest destDir]\r\ncall ccg-env\r\nset ANT_HOME=%OPENCCG_HOME%\\lib\r\nset CP=%CP%;%ANT_HOME%\\ant.jar\r\nset JAVA_ARGS=-Xmx128m -classpath %CP%\r\n%JAVA% %JAVA_ARGS% opennlp.ccg.grammardoc.GrammarDoc %*\r\n\r\n" }, { "path": "bin/ccg-gt", "content": "#!/bin/sh\n# For usage, do: ccg-gt -h\n. ccg-env\n\"$JAVA\" $JAVA_ARGS opennlp.ccg.test.GenTargets $@\n\n" }, { "path": "bin/ccg-gt.bat", "content": "@echo off\r\nrem For usage, do: ccg-gt -h\r\ncall ccg-env\r\n%JAVA_CMD% opennlp.ccg.test.GenTargets %*\r\n\r\n" }, { "path": "bin/ccg-ht-factors", "content": "#!/bin/sh\n. ccg-env\n#CP=$CP:$OPENCCG_HOME/lib/jopt-simple.jar\n#echo $JAVA_ARGS\n\"$JAVA\" $JAVA_ARGS opennlp.ccg.realize.hypertagger.LMFactorExtractor $@\n\n" }, { "path": "bin/ccg-hypertagger", "content": "#!/bin/sh\n. ccg-env\n#CP=$CP:$OPENCCG_HOME/lib/jopt-simple.jar\n#echo $JAVA_ARGS\n\"$JAVA\" $JAVA_ARGS opennlp.ccg.realize.hypertagger.TagExtract $@\n\n" }, { "path": "bin/ccg-hypertagger.bat", "content": "@echo off\r\ncall ccg-env\r\n%JAVA_CMD% opennlp.ccg.realize.hypertagger.TagExtract %*\r\n\r\n" }, { "path": "bin/ccg-parse", "content": "#!/bin/sh\n# Usage: ccg-parse -h\n. ccg-env\n\"$JAVA\" $JAVA_ARGS opennlp.ccg.Parse $@\n\n" }, { "path": "bin/ccg-parse.bat", "content": "@echo off\r\nrem Usage: ccg-parse -h\r\ncall ccg-env\r\n%JAVA_CMD% opennlp.ccg.Parse %1 %2 %3 %4 %5 %6 %7 %8 %9 \r\n\r\n" }, { "path": "bin/ccg-postagger", "content": "#!/bin/sh\n. ccg-env\n\"$JAVA\" $JAVA_ARGS opennlp.ccg.parse.postagger.BasicPOSTagger $@\n\n" }, { "path": "bin/ccg-postagger.bat", "content": "@echo off\r\ncall ccg-env\r\n%JAVA_CMD% opennlp.ccg.parse.postagger.BasicPOSTagger %*\r\n\r\n" }, { "path": "bin/ccg-realize", "content": "#!/bin/sh\n# Usage: ccg-realize (-g ) ()\n. ccg-env\n\"$JAVA\" $JAVA_ARGS opennlp.ccg.Realize $@\n\n" }, { "path": "bin/ccg-realize.bat", "content": "@echo off\r\nrem Usage: ccg-realize (-g ) ()\r\ncall ccg-env\r\nrem set HPROF=-Xrunhprof:cpu=times,file=hmm-prof.txt\r\n%JAVA_CMD% opennlp.ccg.Realize %1 %2 %3 %4 %5 %6 %7 %8 %9 \r\n\r\n" }, { "path": "bin/ccg-supertagger", "content": "#!/bin/sh\n. ccg-env\n\"$JAVA\" $JAVA_ARGS opennlp.ccg.parse.supertagger.WordAndPOSDictionaryLabellingStrategy $@\n\n" }, { "path": "bin/ccg-supertagger.bat", "content": "@echo off\r\ncall ccg-env\r\n%JAVA_CMD% opennlp.ccg.parse.supertagger.WordAndPOSDictionaryLabellingStrategy %*\r\n\r\n" }, { "path": "bin/ccg-test", "content": "#!/bin/sh\n# For usage, do: ccg-test -h\n. ccg-env\n\"$JAVA\" $JAVA_ARGS opennlp.ccg.test.Regression \"$@\"\n" }, { "path": "bin/ccg-test.bat", "content": "@echo off\r\nrem For usage, do: ccg-test -h\r\ncall ccg-env\r\nrem set HPROF=-Xrunhprof:cpu=times,file=hmm-prof.txt\r\n%JAVA_CMD% opennlp.ccg.test.Regression %*\r\n\r\n" }, { "path": "bin/ccg-update", "content": "#!/bin/sh\n# For usage, do: ccg-update -h\n. ccg-env\n\"$JAVA\" $JAVA_ARGS opennlp.ccg.test.UpdateTestbed $@\n\n" }, { "path": "bin/ccg-update.bat", "content": "@echo off\r\nrem For usage, do: ccg-update -h\r\ncall ccg-env\r\n%JAVA_CMD% opennlp.ccg.test.UpdateTestbed %*\r\n\r\n" }, { "path": "bin/ccg2xml", "content": "#!/bin/sh\n\npython \"$OPENCCG_HOME/bin/ccg2xml.py\" \"$@\"\n" }, { "path": "bin/ccg2xml.bat", "content": "@echo off\r\npython %OPENCCG_HOME%/bin/ccg2xml.py %*" }, { "path": "bin/ccg_draw_tree.py", "content": "#\r\n# ccg_draw_tree uses nltk.Tree to draw a tree from a CCGbank .auto file, \r\n# or to draw two trees from two .auto files\r\n#\r\n\r\nimport sys\r\n\r\nif len(sys.argv) == 1 or sys.argv[1] == '-h' or sys.argv[1] == '--help':\r\n print 'Usage: ccg_draw_tree () ()'\r\n sys.exit(0)\r\n\r\nautofile = sys.argv[1]\r\nderiv_id = sys.argv[2]\r\n\r\nautofile2 = None\r\nderiv_id2 = None\r\n\r\nif len(sys.argv) >= 4: \r\n autofile2 = sys.argv[3]\r\n deriv_id2 = deriv_id\r\n if len(sys.argv) >= 5:\r\n deriv_id2 = sys.argv[4]\r\n\r\ndef get_deriv(autofile, deriv_id):\r\n print 'reading ' + deriv_id + ' from ' + autofile\r\n found_it = False\r\n file = open(autofile, 'rU')\r\n for line in file:\r\n if found_it == True:\r\n return line\r\n if line[0:2] == 'ID':\r\n if line.split()[0].split('=')[1] == deriv_id:\r\n found_it = True\r\n raise NameError('could not find ' + deriv_id + '!')\r\n\r\nderiv = get_deriv(autofile, deriv_id)\r\nderiv2 = None\r\nif autofile2 != None:\r\n deriv2 = get_deriv(autofile2, deriv_id2)\r\n\r\nprint 'importing nltk.Tree'\r\n\r\nfrom nltk import Tree\r\nfrom nltk.draw.tree import draw_trees\r\n\r\nccgbank_node_pattern = r''\r\nccgbank_leaf_pattern = r''\r\n\r\n# nb: the parens around leaves ends up creating blank nodes above leaves\r\ndef parse_ccgbank_node(s):\r\n if s =='': return ''\r\n return s.split(' ')[1]\r\n\r\ndef parse_ccgbank_leaf(s):\r\n tokens = s.split(' ')\r\n return Tree(tokens[1], [tokens[4]])\r\n\r\ndef excise_empty_nodes(t):\r\n if not isinstance(t,Tree): return t\r\n if t.node == '': return excise_empty_nodes(t[0])\r\n return Tree(t.node, [excise_empty_nodes(st) for st in t])\r\n\r\n# nb: returns tree with blank nodes excised\r\ndef parse_ccgbank_tree(s):\r\n t = Tree.parse(s, \r\n parse_node=parse_ccgbank_node, \r\n parse_leaf=parse_ccgbank_leaf, \r\n node_pattern=ccgbank_node_pattern, \r\n leaf_pattern=ccgbank_leaf_pattern)\r\n return excise_empty_nodes(t)\r\n\r\nprint\r\nprint 'parsing: ' + deriv\r\nt = parse_ccgbank_tree(deriv)\r\nprint t\r\n\r\nt2 = None\r\nif deriv2 != None:\r\n print\r\n print 'parsing: ' + deriv2\r\n t2 = parse_ccgbank_tree(deriv2)\r\n print t2\r\n\r\nprint\r\nif t2 == None:\r\n print 'drawing tree'\r\n draw_trees(t)\r\nelse:\r\n print 'drawing trees'\r\n draw_trees(t,t2)\r\n" }, { "path": "bin/dlf_parser.py", "content": "#\n# dlf_parser.py (invoked by ccg-draw-graph) uses graphviz's dot to visualize (D)LF graphs\n#\n# author: Jonathan Barker (with minor contributions by Michael White)\n# license: LGPL\n#\n\nfrom xml.etree.ElementTree import ElementTree\nimport optparse, sys, codecs, xml, os\nfrom collections import defaultdict\n\n# Parse arguments\nop = optparse.OptionParser()\n\nop.add_option(\"-i\", \"--input\", type=\"string\", help=\"input source: file or (default)\", default=sys.stdin)\n\nop.add_option(\"-m\", \"--moses\", type=\"string\", help=\"file/directory prefix for moses output\", default=None)\n\nop.add_option(\"-v\", \"--visualize\", type=\"string\", help=\"file/directory prefix for .pdf output\", default=None)\n\nop.add_option(\"-w\", \"--wordindices\", action=\"store_true\", help=\"include word indices\", default=False)\n\nop.add_option(\"-c\", \"--classnames\", action=\"store_true\", help=\"include semantic class names\", default=False)\n\n(ops, args) = op.parse_args(sys.argv)\n\n# Parse input file\ninput_source = ops.input if ops.input is sys.stdin else open(ops.input, \"rt\")\nraw = xml.etree.ElementTree.XML(input_source.read())\nsnum = \"None\"\natt_id = 0\n\n# Get word number\ndef wordNum(wid):\n if wid.startswith(\"x\"):\n return -1\n else:\n return int(wid[1:].strip(\"f\"))\n\n# Get node span\ndef span(nid, graph, w):\n if wordNum(nid) in w:\n return []\n w.append(wordNum(nid))\n for n, e in graph[nid]:\n if wordNum(n) not in w:\n w.append(wordNum(n))\n w.extend(span(n, graph, w))\n return w\n\n# findall wrapper\ndef findAll(elem, match):\n return max(elem.findall(match), [])\n\n# Class for representing predicates and attributes\nclass Pred:\n def __init__(self):\n self.attrib = []\n self.one_of = []\n self.opt = []\n\n# Class for representing nodes, contains predicate and attribute information\nclass Node:\n def __init__(self):\n self.id = \"\"\n self.className = \"\"\n self.preds = defaultdict(Pred)\n\n def addPred(self, pred, attrib, one_of, opt):\n self.preds[pred].attrib.extend(attrib)\n self.preds[pred].one_of.extend(one_of)\n self.preds[pred].opt.extend(opt)\n\n def moses(self, graph):\n tree = \" \"\n return tree\n\n def dot(self):\n dot_node = self.id+\" [label=<\"\n withClassName = ops.classnames and len(self.className) > 0\n if ops.wordindices:\n dot_node += self.id\n if withClassName:\n dot_node += \":\"\n if withClassName:\n dot_node += self.className\n if len(self.preds) > 0:\n if ops.wordindices or withClassName:\n dot_node += \":\"\n labels = []\n for pname, p in self.preds.items():\n label = \"\"\n # pred\n label += \"\"+pname+\"\"\n # att\n atts = []\n if len(p.attrib) > 0:\n atts.append(\",\".join([\"<\"+k.upper()+\">\"+v for (k, v) in p.attrib]))\n if len(p.one_of) > 0:\n atts.append(\"|\".join([\"<\"+k.upper()+\">\"+v for (k, v) in p.one_of]))\n if len(p.opt) > 0:\n atts.append(\"(\"+\",\".join([\"<\"+k.upper()+\">\"+v for (k, v) in p.opt])+\")?\")\n if len(atts) > 0:\n label += \"\"+\",\".join(atts)+\"\"\n labels.append(label)\n dot_node += \" | \".join(labels)\n dot_node += \">];\\n\"\n return dot_node\n\n def info(self):\n print \"Node id:\",self.id\n for pname, p in self.preds:\n print \"\\tPred:\",self.pred\n print \"\\t\\tAttrib:\",p.attrib\n print \"\\t\\tOne_of:\",p.one_of\n print \"\\t\\tOpt:\",p.opt\n print \"----------------\"\n\n# Returns just the id, stripping the class (if any)\ndef parseId(str):\n colonIndex = str.find(\":\")\n if colonIndex > 0: return str[:colonIndex]\n else: return str\n\n# Returns the class from the id, or the empty string if none\ndef parseClass(str):\n colonIndex = str.find(\":\")\n if colonIndex > 0: return str[colonIndex+1:]\n else: return \"\"\n \n# Method for parsing \ndef parseNode(node, graph, nodes):\n n = nodes[node.get(\"id\")]\n n.id = parseId(node.get(\"id\"))\n n.className = parseClass(node.get(\"id\"))\n attrib = [(k, v) for (k, v) in node.items() if k not in [\"id\", \"pred\"]]\n if node.get(\"pred\") is not None:\n n.addPred(node.get(\"pred\"), attrib, [], [])\n nodes[n.id] = n\n \n for elem in list(node):\n if elem.tag == \"rel\":\n parseRel(elem, n.id, graph, nodes, \"\")\n elif elem.tag == \"one-of\":\n parseOneOf(elem, n, attrib, node.get(\"pred\"), graph, nodes)\n elif elem.tag == \"opt\":\n parseOpt(elem, n, graph, nodes)\n elif elem.tag == \"node\":\n parseNode(elem, graph, nodes)\n else:\n print snum+\": Unexpected tag <\"+elem.tag+\"> after \"\n quit()\n\n# Method for parsing \ndef parseOpt(opt, node, graph, nodes):\n for elem in list(opt):\n if elem.tag == \"atts\":\n for pname, p in node.preds.items():\n node.addPred(pname, [], [], [(k, v) for (k, v) in elem.items() if k not in [\"id\", \"pred\"]])\n elif elem.tag == \"rel\":\n parseRel(elem, node.id, graph, nodes, \"style=dotted, \")\n else:\n print snum+\": Unexpected tag <\"+elem.tag+\"> after \"\n quit()\n\n# Method for parsing \ndef parseOneOf(oneof, node, attrib, pred, graph, nodes):\n global att_id\n num_att = 0\n for elem in list(oneof):\n if elem.tag == \"atts\":\n if pred is not None:\n node.addPred(pred, [], [(k, v) for (k, v) in elem.items() if k not in [\"id\", \"pred\"]], [])\n else:\n node.addPred(elem.get(\"pred\"), [], [(k, v) for (k, v) in elem.items() if k not in [\"id\", \"pred\"]], [])\n if len(list(elem)) > 0:\n num_att += 1\n new_att = Node()\n new_att.id = \"att\"+str(att_id)\n att_id += 1\n new_att.addPred(str(num_att), [], [], [])\n nodes[new_att.id] = new_att\n graph[node.id].append((new_att.id, \" [style=dashed];\\n\"))\n for rel in list(elem):\n parseRel(rel, new_att.id, graph, nodes, \"\")\n elif elem.tag == \"rel\":\n num_att += 1\n new_att = Node()\n new_att.id = \"att\"+str(att_id)\n att_id += 1\n new_att.addPred(str(num_att), [], [], [])\n nodes[new_att.id] = new_att\n graph[node.id].append((new_att.id, \" [style=dashed];\\n\"))\n parseRel(elem, new_att.id, graph, nodes, \"\")\n else:\n print snum+\": Unexpected tag <\"+elem.tag+\"> after \"\n quit()\n\n# Method for parsing \ndef parseRel(rel, nid, graph, nodes, style):\n # \n for subnode in list(rel):\n if subnode.tag == \"node\":\n edge_label = \" [\"+style+\"label = \\\"\"+rel.get(\"name\")+\"\\\"];\\n\"\n if subnode.get(\"id\") is None:\n graph[nid].append((parseId(subnode.get(\"idref\")), edge_label))\n else:\n graph[nid].append((parseId(subnode.get(\"id\")), edge_label))\n parseNode(subnode, graph, nodes)\n elif subnode.tag == \"one-of\":\n subnode.set(\"name\", rel.get(\"name\"))\n parseRel(subnode, nid, graph, nodes, \"style=dashed, \")\n else:\n print snum+\": Unexpected tag <\"+subnode.tag+\"> after \"\n quit()\n# \nitem_no = 0\nfor item in findAll(raw, \"item\"):\n item_no += 1\n if item.get(\"numOfParses\") == \"0\":\n print \"Removing \"+item.get(\"info\")\n else:\n snum = item.get(\"info\")\n # \n lf_num = 0\n for lf in findAll(item, \"lf\"):\n graph = defaultdict(list)\n nodes = defaultdict(Node)\n \n # \n for node in list(lf):\n if node.tag == \"node\":\n parseNode(node, graph, nodes)\n else:\n print snum+\": Unexpected tag <\"+node.tag+\"> after \"\n quit()\n\n # Plot the graph with GraphViz\n if ops.visualize != None:\n viz_name = \"\"\n if type(item.get(\"info\")) != type(\"string\"):\n viz_name = ops.visualize+\".item\"+str(item_no)+\".\"+str(lf_num)\n else:\n viz_name = ops.visualize+\".\"+item.get(\"info\")+\".\"+str(lf_num)\n viz = codecs.open(viz_name+\".dot\", \"w\", \"utf-8\")\n viz.write(\"digraph lf {\\n\")\n for (k, v) in nodes.items():\n viz.write(v.dot())\n for (left, rights) in graph.items():\n for right in rights:\n viz.write(left+\"->\"+right[0]+right[1])\n viz.write(\"}\\n\")\n viz.close()\n os.system(\"dot -Tpdf \"+viz_name+\".dot -o \"+viz_name+\".pdf\")\n os.system(\"rm \"+viz_name+\".dot\")\n lf_num += 1\n" }, { "path": "bin/tccg", "content": "#!/bin/sh\n# For usage, do: tccg -h\n. ccg-env\n\"$JAVA\" $JAVA_ARGS opennlp.ccg.TextCCG \"$@\"\n" }, { "path": "bin/tccg.bat", "content": "@echo off\r\nrem For usage, do: tccg -h\r\ncall ccg-env\r\n%JAVA_CMD% opennlp.ccg.TextCCG %*\r\n\r\n" }, { "path": "bin/visccg", "content": "#!/bin/sh\n\npython \"$OPENCCG_HOME/bin/ccg_editor.py\" \"$@\"\n" }, { "path": "bin/visccg.bat", "content": "@echo off\npython %OPENCCG_HOME%/bin/ccg_editor.py %*\n" }, { "path": "bin/wccg", "content": "#!/bin/sh\n# For usage, do: tccg -h\n. ccg-env\n\"$JAVA\" $JAVA_ARGS opennlp.ccg.WebCCG \"$@\"\n" }, { "path": "build.xml", "content": "\n\n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n generates the openccg.jar file (default)\"/>\n compiles the source code\"/>\n generates the API documentation\"/>\n runs JUnit tests\"/>\n cleans up the compilation directory\"/>\n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n\n\n\n" }, { "path": "ccg-format-grammars/arabic/arabic.ccg", "content": "#############################################################\n# #\n# arabic.ccg #\n# #\n#############################################################\n\n# Author: Ben Wing \n# Date: April 2006\n\n# This is a grammar for a fragment of Arabic. It's particularly\n# useful for demonstrating the extended use of macros to handle\n# complicated morphological inflections.\n\n# See the `tiny' grammar (tiny.ccg) for more info about the format\n# of this file.\n\nfeature {\n CASE<2>: nom, acc, gen;\n NUM<2>: sg, du, pl;\n GEND<2>: m, f;\n STATE<2>: cons, non-cons {indef, def};\n ANIM<2>: hum, nonhum;\n PERS<2>: 1st, 2nd, 3rd;\n RESUMPTIVE<2>: nonres, res;\n SEM-NUM: sg-X, du-X, pl-X;\n SEM-PERS: 1st-X, 2nd-X, 3rd-X;\n TENSE: past, pres;\n MOOD: indic, subj, juss;\n\n # Here's a more complicated hierarchy, from the original tiny grammar.\n ontology: sem-obj {\n phys-obj {\n animate-being {\n person\n },\n thing\n },\n situation {\n change {\n action\n },\n state\n }\n };\n}\n\nrule {\n no typeraise;\n typeraise +: n => s;\n typeraise - $: n => s;\n typeraise - $: pp => s;\n typeraise - $: pp/n => s;\n typechange: s$1 | n[nom] => s$1 ;\n typechange: n<~2>[cons] => n<2>[3rd,def] /* n[gen,def] ;\n typechange: n<~2>[cons] => n<2>[3rd,indef] /* n[gen,indef] ;\n}\n\n##########################################################################\n# Morphological entries #\n# (morph.xml) #\n##########################################################################\n\nword wa:Conj; # \"and\"\nword anna:Comp; # \"that\", introducing sentential complements\nword inna:Comp; # same, but only after the verb qaal \"say\"\n\nword maa:InterrogPro(thing): 3rd; # \"what\"\nword man:InterrogPro(person): 3rd; # \"who\"\n\nword li:Prep; # \"what\"\nword fii:Prep; # \"who\"\n\n# This word means \"this\". \nword haadhaa {\n *: sg, m;\n haadhihi: sg, f;\n ha_ulaahi: pl;\n\n # Bizarrely, this word declines for case only in the dual.\n haadhaani: du, m, nom;\n haadhayni: du, m, acc;\n haadhayni: du, m, gen;\n\n haataani: du, f, nom;\n haatayni: du, f, acc;\n haatayni: du, f, gen;\n}\n\n# This word means \"that\". \nword dhaalik {\n *: sg, m;\n tilka: sg, f;\n ulaa_ika: pl;\n\n # Bizarrely, this word declines for case only in the dual.\n dhaanika: du, m, nom;\n dhaynika: du, m, acc;\n dhaynika: du, m, gen;\n\n taanika: du, f, nom;\n taynika: du, f, acc;\n taynika: du, f, gen;\n}\n\n# This is the relative pronoun.\nword al-ladhii {\n *: sg, m;\n al-latii: sg, f;\n al-ladhiina: pl, m;\n al-laati: pl, f;\n\n # Bizarrely, this word declines for case only in the dual.\n al-ladhaani: du, m, nom;\n al-ladhayni: du, m, acc;\n al-ladhayni: du, m, gen;\n\n al-lataani: du, f, nom;\n al-latayni: du, f, acc;\n al-latayni: du, f, gen;\n}\n\nword pro:Pro {\n ana: 1st, 1st-X, sg, sg-X;\n anta: 2nd, 2nd-X, sg, sg-X, m;\n anti: 2nd, 2nd-X, sg, sg-X, f;\n huwa: 3rd, 3rd-X, sg, sg-X, m;\n hiya: 3rd, 3rd-X, sg, sg-X, f;\n naHnu: 1st, 1st-X, pl, pl-X;\n antun: 2nd, 2nd-X, pl, pl-X, m;\n antunna: 2nd, 2nd-X, pl, pl-X, f;\n hum: 3rd, 3rd-X, pl, pl-X, m;\n hunna: 3rd, 3rd-X, pl, pl-X, f;\n}\n\nword ii:: 1st, 1st-X, sg, sg-X;\nword nii:: 1st, 1st-X, sg, sg-X;\nword ka:: 2nd, 2nd-X, sg, sg-X, m;\nword ki:: 2nd, 2nd-X, sg, sg-X, f;\nword hu:: 3rd, 3rd-X, sg, sg-X, m;\nword haa:: 3rd, 3rd-X, sg, sg-X, f;\nword naa:: 1st, 1st-X, pl, pl-X;\nword kum:: 2nd, 2nd-X, pl, pl-X, m;\nword kunna:: 2nd, 2nd-X, pl, pl-X, f;\nword hum:: 3rd, 3rd-X, pl, pl-X, m;\nword hunna:: 3rd, 3rd-X, pl, pl-X, f;\n\n#############################################\n# Nouns #\n#############################################\n\n# This shows how a reasonably complicated morphology can be accommodated.\n# It is certainly possible that some of this may (and probably should)\n# be offloaded into a separate morphology-processing engine. However,\n# even in that case there is often a good deal more to the lexicon.\n\n# We show a couple examples of complete paradigms, in order to make it\n# easier to understand what's going on below.\n\n# Here is a typical noun (kitaab \"book\") with a broken plural (kutub \"books\").\n# For nouns with broken plurals, the plural is typically declined like\n# the singular. Note that Arabic nouns are conjugated for three numbers\n# (singular, dual, plural), three cases (nominative, accusative, dative),\n# and three states (indefinite, definite, construct). (The construct state\n# is used for nouns that are modified by other nouns -- e.g. \"book\" in\n# \"the book of Mary\".)\n\n# Form\t\tNominative\tAccusative\tDative\n# ---------------------------------------------------------------\n# sg.indef\tkitaabun\tkitaaban\tkitaabin\n# sg.def\tal-kitaabu\tal-kitaaba\tal-kitaabi\n# sg.cons\tkitaabu\t\tkitaaba\t\tkitaabi\n# \n# du.indef\tkitaabaani\tkitaabayni\tkitaabayni\n# du.def\tal-kitaabaani\tal-kitaabayni\tal-kitaabayni\n# du.cons\tkitaabaa\tkitaabay\tkitaabay\n# \n# pl.indef\tkutubun\t\tkutuban\t\tkutubin\n# pl.def\tal-kutubu\tal-kutuba\tal-kutubi\n# pl.cons\tkutubu\t\tkutuba\t\tkutubi\n\n# Here is a typical noun (mudarris \"teacher\") with a different kind of\n# plural, a so-called \"strong masculine plural\", which has its own declension.\n\n# Form\t\tNominative\tAccusative\tDative\n# ---------------------------------------------------------------\n# sg.indef\tmudarrisun\tmudarrisan\tmudarrisin\n# sg.def\tal-mudarrisu\tal-mudarrisa\tal-mudarrisi\n# sg.cons\tmudarrisu\tmudarrisa\tmudarrisi\n#\n# du.indef\tmudarrisaani\tmudarrisayni\tmudarrisayni\n# du.def\tal-mudarrisaani\tal-mudarrisayni\tal-mudarrisayni\n# du.cons\tmudarrisaa\tmudarrisay\tmudarrisay\n#\n# pl.indef\tmudarrisuuna\tmudarrisiina\tmudarrisiina\n# pl.def\tal-mudarrisuuna\tal-mudarrisiina\tal-mudarrisiina\n# pl.cons\tmudarrisuu\tmudarrisii\tmudarrisii\n\n# Here, we make heavy use of macros.\n\n# This macro says: Every time an expression of the form\n# three-form-decl(...) occurs, replace it with the text that comes after.\n# The parameters will be substituted into the text. The braces that\n# denote the macro's text do *NOT* form part of the text that is substituted.\n# Note that macro substitutions are processed recursively: If the text\n# of a macro substitution contains calls to other macros, they will also\n# be processed. This makes \"inheritance\" very easy to implement.\n\n# This macro is used for a particular paradigm corresponding to a\n# particular number of a word.\n\ndef three-different-form-decl(indef-form, def-form, cons-form,\n\t\t\t indef-nom, indef-acc, indef-gen,\n \t\t\t def-nom, def-acc, def-gen,\n\t \t cons-nom, cons-acc, cons-gen,\n\t \t morph-num, sem-num, gend) {\n indef-form.indef-nom: morph-num, sem-num, gend, nom, indef;\n indef-form.indef-acc: morph-num, sem-num, gend, acc, indef;\n indef-form.indef-gen: morph-num, sem-num, gend, gen, indef;\n\n add-al(def-form.def-nom): morph-num, sem-num, gend, nom, def;\n add-al(def-form.def-acc): morph-num, sem-num, gend, acc, def;\n add-al(def-form.def-gen): morph-num, sem-num, gend, gen, def;\n\n cons-form.cons-nom: morph-num, sem-num, gend, nom, cons;\n cons-form.cons-acc: morph-num, sem-num, gend, acc, cons;\n cons-form.cons-gen: morph-num, sem-num, gend, gen, cons;\n}\n\n# It's questionable whether we should do this. This assimilates al-\n# to a following coronal consonant, e.g. ar-rajul, as-sigaara,\n# ath-thalj, an-nuur, aDH-DHuhr, etc.\ndef add-al(form) regsub('^al-([std]h|DH|[tdszrnTDSZL])', 'a\\1-\\1', al-.form)\n\ndef three-form-decl(form, indef-nom, indef-acc, indef-gen,\n\t\t def-nom, def-acc, def-gen,\n\t\t cons-nom, cons-acc, cons-gen,\n\t\t morph-num, sem-num, gend) {\n three-different-form-decl(form, form, form,\n\t\t\t indef-nom, indef-acc, indef-gen,\n \t\t\t def-nom, def-acc, def-gen,\n\t \t cons-nom, cons-acc, cons-gen,\n\t \t morph-num, sem-num, gend)\n}\n\n# Using the above macro, we create two more macros to handle two common\n# paradigm types: Accusative and genitive are the same, and the\n# definite is either the same as the construct (two-form-decl-1) or\n# the same as the indefinite (two-form-decl-2).\n\ndef two-form-decl-1(form, non-cons-nom, non-cons-obl,\n\t\t cons-nom, cons-obl, morph-num, sem-num, gend) {\n three-form-decl(form, non-cons-nom, non-cons-obl, non-cons-obl,\n\t\t non-cons-nom, non-cons-obl, non-cons-obl,\n\t\t cons-nom, cons-obl, cons-obl,\n\t\t morph-num, sem-num, gend)\n}\n\ndef two-form-decl-2(form, indef-nom, indef-obl,\n\t\t non-indef-nom, non-indef-obl, morph-num, sem-num, gend) {\n three-form-decl(form, indef-nom, indef-obl, indef-obl,\n\t\t non-indef-nom, non-indef-obl, non-indef-obl,\n\t\t non-indef-nom, non-indef-obl, non-indef-obl,\n\t\t morph-num, sem-num, gend)\n}\n\n# In turn we create macros for particular paradigms: strong masculine (\"uun\"),\n# strong feminine (\"aat\"), dual, and basic triptote (the paradigm for\n# \"kitaab\" above and, in general, most singulars).\n\n# Note that an alternative to using braces is to put the macro text on\n# the same line as the `def' part of the macro (backslashes can be used\n# to join multiple lines together).\n\ndef uun-plural(form) two-form-decl-1(form, uuna, iina, uu, ii, pl, pl-X, m)\ndef aat-plural(form) \\\n two-form-decl-2(form, aatun, aatin, aatu, aati, pl, pl-X, f)\ndef dual(form, gend) \\\n two-form-decl-1(form, aani, ayni, aa, ay, du, du-X, gend)\ndef triptote(form, morph-num, sem-num, gend) \\\n three-form-decl(form, un, an, in, u, a, i, u, a, i, morph-num, sem-num, gend)\n\n# Here we define macros for full paradigms for words. Note how semicolons\n# are not used, because they are supplied by the macro text itself.\n# (Consult the text for three-form-decl() above, and remember that the\n# braces denoting the macro text are not actually part of the text. This\n# means that if you really want braces as the outermost thing in some\n# macro text, you'll need to supply two levels of braces.)\n\ndef thing(sing, plur) {\n word sing:N(thing) {\n triptote(sing, sg, sg-X, m)\n dual(sing, m)\n triptote(plur, sg, pl-X, f)\n }\n}\n\ndef fem-thing(sing, plur) {\n word sing:N(thing) {\n triptote(sing.t, sg, sg-X, f)\n dual(sing.t, f)\n triptote(plur, sg, pl-X, f)\n }\n}\n\ndef person(sing, plur, gend) {\n word sing:N(person) {\n triptote(sing, sg, sg-X, gend)\n dual(sing, gend)\n triptote(plur, pl, pl-X, gend)\n }\n}\n\ndef male(sing, plur) person(sing, plur, m)\ndef female(sing, plur) person(sing, plur, f)\n\ndef strong-male(sing) {\n word sing:N(person) {\n triptote(sing, sg, sg-X, m)\n dual(sing, m)\n uun-plural(sing)\n }\n}\n\n# Here we define the actual words. Note how short these definitions are,\n# specifying only what's unpredictable.\n\nthing(kitaab, kutub)\nthing(waqt, _awqaat)\nthing(Harf, Huruuf)\nthing(dars, duruus)\nthing(waqt, _awqaat)\nfem-thing(sigaara, sagaayir)\nfem-thing(madiina, mudun)\nmale(rajul, rijaal)\nmale(walad, _awlaad)\nmale(Taalib, Tullaab)\nfemale(bint, _abnaat)\nstrong-male(mudarris)\n\nword imra_a:N(person) {\n three-different-form-decl(imra_at, mar_at, imra_at,\n\t\t\t un, an, in, u, a, i, u, a, i,\n sg, sg-X, f)\n three-different-form-decl(imra_at, mar_at, imra_at,\n\t\t\t aani, ayni, ayni, aani, ayni, ayni, aa, ay, ay,\n du, du-X, f)\n triptote(nisaa_, pl, pl-X, f)\n}\n\ndef extended_construct_word(stem, plur) {\n word stem:N(person) {\n three-form-decl(stem, un, an, in, u, a, i, uu, aa, ii, sg, sg-X, m)\n dual(stem, m)\n triptote(plur, pl, pl-X, m)\n }\n}\n\nextended_construct_word(_ax, _ixwaan)\nextended_construct_word(_ab, _abnaa_)\n\n# Typical paradigms:\n\n# 1sg 'aktaa 'aktaa 'akta |\n# 2sg.m taktaa taktaa takta |\n# 2sg.f taktayna taktay taktay |\n# 3sg.m yaktaa yaktaa yakta |\n# 3sg.f taktaa taktaa takta |\n# 2du taktayaani taktayaa taktayaa |\n# 3du.m yaktayaani yaktayaa yaktayaa |\n# 3du.f taktayaani taktayaa taktayaa |\n# 1pl naktaa naktaa nakta |\n# 2pl.m taktawna taktaw taktaw |\n# 2pl.f taktayna taktayna taktayna |\n# 3pl.m yaktawna yaktaw yaktaw |\n# 3pl.f yaktayna yaktayna yaktayna |\n# \n# 1sg 'aktuu 'aktuwa 'aktu | 'aktii 'aktiya 'akti \n# 2sg.m taktuu taktuwa taktu | taktii taktiya takti \n# 2sg.f taktiina taktii taktii | taktiina taktii taktii \n# 3sg.m yaktuu yaktuwa yaktu | yaktii yaktiya yakti \n# 3sg.f taktuu taktuwa taktu | taktii taktiya takti \n# 2du taktuwaani taktuwaa taktuwaa | taktiyaani taktiyaa taktiyaa \n# 3du.m yaktuwaani yaktuwaa yaktuwaa | yaktiyaani yaktiyaa yaktiyaa \n# 3du.f taktuwaani taktuwaa taktuwaa | taktiyaani taktiyaa taktiyaa \n# 1pl naktuu naktuwa naktu | naktii naktiya nakti \n# 2pl.m taktuuna taktuu taktuu | taktuuna taktuu taktuu \n# 2pl.f taktuuna taktuuna taktuuna | taktiina taktiina taktiina \n# 3pl.m yaktuuna yaktuu yaktuu | yaktuuna yaktuu yaktuu \n# 3pl.f yaktuuna yaktuuna yaktuuna | yaktiina yaktiina yaktiina \n\n\ndef two-form-past(formv, formc) {\n formc.tu: past, 1st, sg;\n formc.ta: past, 2nd, m, sg;\n formc.ti: past, 2nd, f, sg;\n formv.a: past, 3rd, m, sg;\n formv.at: past, 3rd, f, sg;\n\n formc.tumaa: past, 2nd, du;\n formv.aa: past, 3rd, m, du;\n formv.ataa: past, 3rd, f, du;\n\n formc.naa: past, 1st, pl;\n formc.tum: past, 2nd, m, pl;\n formc.tunna: past, 2nd, f, pl;\n formv.uu: past, 3rd, m, pl;\n formc.na: past, 3rd, f, pl;\n}\n\ndef 3rd-weak-past-ay(form) {\n form.ay.tu: past, 1st, sg;\n form.ay.ta: past, 2nd, m, sg;\n form.ay.ti: past, 2nd, f, sg;\n form.aa: past, 3rd, m, sg;\n form.at: past, 3rd, f, sg;\n\n form.ay.tumaa: past, 2nd, du;\n form.ay.aa: past, 3rd, m, du;\n form.ataa: past, 3rd, f, du;\n\n form.ay.naa: past, 1st, pl;\n form.ay.tum: past, 2nd, m, pl;\n form.ay.tunna: past, 2nd, f, pl;\n form.aw: past, 3rd, m, pl;\n form.ay.na: past, 3rd, f, pl;\n}\n\ndef 3rd-weak-past-aw(form) {\n form.aw.tu: past, 1st, sg;\n form.aw.ta: past, 2nd, m, sg;\n form.aw.ti: past, 2nd, f, sg;\n form.aa: past, 3rd, m, sg;\n form.at: past, 3rd, f, sg;\n\n form.aw.tumaa: past, 2nd, du;\n form.aw.aa: past, 3rd, m, du;\n form.ataa: past, 3rd, f, du;\n\n form.aw.naa: past, 1st, pl;\n form.aw.tum: past, 2nd, m, pl;\n form.aw.tunna: past, 2nd, f, pl;\n form.aw: past, 3rd, m, pl;\n form.aw.na: past, 3rd, f, pl;\n}\n\ndef 3rd-weak-past-ii(form) {\n form.ii.tu: past, 1st, sg;\n form.ii.ta: past, 2nd, m, sg;\n form.ii.ti: past, 2nd, f, sg;\n form.iya: past, 3rd, m, sg;\n form.iyat: past, 3rd, f, sg;\n\n form.ii.tumaa: past, 2nd, du;\n form.iy.aa: past, 3rd, m, du;\n form.iy.ataa: past, 3rd, f, du;\n\n form.ii.naa: past, 1st, pl;\n form.ii.tum: past, 2nd, m, pl;\n form.ii.tunna: past, 2nd, f, pl;\n form.uu: past, 3rd, m, pl;\n form.ii.na: past, 3rd, f, pl;\n}\n\ndef strong-past(form) two-form-past(form, form)\n\n# In general, almost all Arabic present-tense verbs of a particular mood\n# can be defined using five forms. Verbs with a hamza in the first radical\n# have a problem in the first-singular; ideally this should be handled\n# automatically using a regexp or something of that sort, but we don't have\n# such support currently, so we use an optional param.\n\ndef gen-pres(mood, fsing, fsing-fem, fdual, fplur-masc, fplur-fem) {\n # This shows how you can use regular expressions if need be.\n # regsub(string, regex, repl) is a special built-in that does regular-\n # expression substitution on STRING, replacing all occurrences of\n # REGEX with REPL. Regular-expression syntax is as in Python.\n # In this case, Arabic verbs have a phonetic rule that eliminates\n # two glottal stops occurring near each other at the beginning of a\n # word. For example, _a_kulu -> _aakulu, and _u_kalu -> _uukalu.\n # (That is, the vowel is lengthened.)\n # _ . regsub(foo, bar, fsing): pres, mood, 1st, sg;\n _ . regsub('^([aiu])_', '\\1\\1', fsing): pres, mood, 1st, sg;\n # _.fsing: pres, mood, 1st, sg;\n t.fsing: pres, mood, 2nd, m, sg;\n t.fsing-fem: pres, mood, 2nd, f, sg;\n y.fsing: pres, mood, 3rd, m, sg;\n t.fsing: pres, mood, 3rd, f, sg;\n\n t.fdual: pres, mood, 2nd, du;\n y.fdual: pres, mood, 3rd, m, du;\n t.fdual: pres, mood, 3rd, f, du;\n\n n.fsing: pres, mood, 1st, pl;\n t.fplur-masc: pres, mood, 2nd, m, pl;\n t.fplur-fem: pres, mood, 2nd, f, pl;\n y.fplur-masc: pres, mood, 3rd, m, pl;\n y.fplur-fem: pres, mood, 3rd, f, pl;\n}\n\n# The \"two-form\" present uses normal (non-3rd-weak) endings but may\n# have two forms of the root, one form vocalic endings (almost all of them)\n# and one for consonant endings (only the feminine plural). This\n# encompasses 2nd-weak verbs and doubled verbs, and (trivially) strong verbs.\n\ndef two-form-pres-indic(formv, formc) {\n gen-pres(indic, formv.u, formv.iina, formv.aani, formv.uuna, formc.na)\n}\n\ndef two-form-pres-subj(formv, formc) {\n gen-pres(subj, formv.a, formv.ii, formv.aa, formv.uu, formc.na)\n}\n\n# The jussive is different because the base form (fsing) has no ending.\n# This means that it may assume the consonant form instead of the vowel\n# form, or may have a number of variants (in particular, for doubled\n# verbs). So the base form needs to be given explicitly.\n\ndef two-form-pres-juss(base, formv, formc) {\n gen-pres(juss, base, formv.ii, formv.aa, formv.uu, formc.na)\n}\n\ndef strong-pres(form) {\n two-form-pres-indic(form, form)\n two-form-pres-subj(form, form)\n two-form-pres-juss(form, form, form)\n}\n\ndef 2nd-weak-pres(formv, formc) {\n two-form-pres-indic(formv, formc)\n two-form-pres-subj(formv, formc)\n two-form-pres-juss(formc, formv, formc)\n}\n\ndef doubled-pres(formv, formc) {\n two-form-pres-indic(formv, formc)\n two-form-pres-subj(formv, formc)\n two-form-pres-juss(formc, formv, formc)\n two-form-pres-juss(formv.a, formv, formc)\n two-form-pres-juss(formv.i, formv, formc)\n}\n\n# Verbs whose third radical is a /w/ or a /y/ have all manner of exceptional\n# forms; easiest just to list them. In general, there are three types,\n# depending on whether the base singular forms end in -aa, -ii, or -uu.\n\ndef 3rd-weak-pres-aa(form) {\n gen-pres(indic, form.aa, form.ayna, form.ayaani, form.awna, form.ayna)\n gen-pres(subj, form.aa, form.ay, form.ayaa, form.aw, form.ayna)\n # Note the shortened vowel here.\n gen-pres(juss, form.a, form.ay, form.ayaa, form.aw, form.ayna)\n}\n\ndef 3rd-weak-pres-ii(form) {\n gen-pres(indic, form.ii, form.iina, form.iyaani, form.uuna, form.iina)\n gen-pres(subj, form.iya, form.ii, form.iyaa, form.uu, form.iina)\n # Note the shortened vowel here.\n gen-pres(juss, form.i, form.ii, form.iyaa, form.uu, form.iina)\n}\n\ndef 3rd-weak-pres-uu(form) {\n gen-pres(indic, form.uu, form.iina, form.uwaani, form.uuna, form.uuna)\n gen-pres(subj, form.uwa, form.ii, form.uwaa, form.uu, form.uuna)\n # Note the shortened vowel here.\n gen-pres(juss, form.u, form.ii, form.uwaa, form.uu, form.uuna)\n}\n\ndef 2nd-weak-verb(pastv, props, pastc, presv, presc) {\n word pastv: props {\n two-form-past(pastv, pastc)\n 2nd-weak-pres(presv, presc)\n }\n}\n\n# Note the way that macro calls can be constructed as well. Here, the\n# value of PAST_TYPE is the suffix at the end of the macro name.\n\ndef 3rd-weak-verb(past_stem, props, past_type, pres_stem, pres_type) {\n word past_stem . past_type: props {\n 3rd-weak-past- . past_type(past_stem)\n 3rd-weak-pres- . pres_type(pres_stem)\n }\n}\n\ndef strong-verb(past, props, pres) {\n word past: props {\n strong-past(past)\n strong-pres(pres)\n }\n}\n\n2nd-weak-verb(kaan, TransV(pred=be), kun, akuun, akun)\n2nd-weak-verb(naam, IntransV(pred=sleep), nim, anaam, anam)\n2nd-weak-verb(qaal, SayV(pred=say), qul, aquul, aqul)\nstrong-verb(katab, TransV(pred=write), aktub)\nstrong-verb(dhahab, IntransV(pred=go), adhhab)\n# Note that the following verb, which begins with a glottal stop,\n# will have a modification made to it in the first-person singular present.\n# (See above.)\nstrong-verb(_akal, IntransV TransV (pred=eat), a_kul)\n3rd-weak-verb(ra_, TransV(pred=see), ay, ar, aa)\n3rd-weak-verb(_a9T, DitransV(pred=give), ay, u9T, ii)\n3rd-weak-verb(laq, TransV(pred=find), ii, alq, aa)\nstrong-verb(9araf, ThinkV(pred=know), a9rif) # see also 9alam\nstrong-verb(tafakkar, ThinkV(pred=think), atafakkar)\nstrong-verb(ta9allam, ThinkV(pred=learn), ata9allam)\n\nfamily N {\n entry: n<2>[X, 3rd, nonres]: X:sem-obj(*);\n}\n\nfamily InterrogPro(Pro) {\n entry: s/*(s/n<2>[res]);\n entry: s/*(s|n<2>[nonres,nom]);\n entry: s/*(s/n<2>[nonres,acc]);\n member: maa, man;\n}\n\nfamily Pro {\n entry: n<2>[X, nom, def, nonres]: X:sem-obj(*);\n member: pro;\n}\n\nfamily Rel {\n entry: (n<~2>[CASE,nonres]\\n<~2>[nonres])/*(s/n<2>[res]);\n entry: (n<~2>[CASE,nonres]\\n<~2>[nonres])/*(s|n<2>[nonres,nom]);\n entry: (n<~2>[CASE,nonres]\\n<~2>[nonres])/*(s/n<2>[nonres,acc]);\n member: al-ladhii;\n}\n\nfamily AndConj(Conj) {\n entry: n[pl, CASE, STATE] \\* n[CASE, STATE] /* n[CASE, STATE];\n entry: s$1 \\* s$1 /* s$1;\n member: wa;\n}\n\nfamily Det(indexRel=det) {\n entry: n<2>[X, def, nonres] /^ n<2>[X]: X:sem-obj(*);\n member: haadhaa, dhaalik;\n}\n\n# good luck on this one! construct even more complicated ones! \n#\n#ar-rajulu al-ladhii kataba al-kutuba ra_aa wa _a9Taa li binti hu al-kilaaba al-latii akalat sagaayira mudarrisii al-waladi\n\n#\"the man that wrote the books saw and gave to his daughter the dogs that ate the cigarettes of the boy's teachers.\"\n\n# Works, correctly:\n\n#ar-rajulu al-ladhii kataba al-kutuba ra_aa wa _a9Taa li binti hu as-sagaayira al-latii qultu inna al-waladu tafakkara anna al-mar_atu _a9Tat haa li ha_ulaahi al-mudarrisiina\n\n#Bad\n\n#ar-rajula al-ladhii katabat al-kutuba ra_aa wa _a9Taa li binti hu as-sagaayira al-latii qultu inna al-waladu tafakkara anna al-mar_atu _a9Tat haa li ha_ulaahi al-mudarrisiina\n\n#\"the man that wrote the books saw and gave to his daughter the cigarettes that I said that the boy thought that the woman gave them to those teachers\"\n\n# Fails, correctly:\n\n#ar-rajulu al-ladhii kataba al-kutuba ra_aa wa _a9Taa li binti hu as-sagaayira al-latii qultu inna al-waladu tafakkara anna al-mar_atu _a9Tat hu li ha-ulaahi al-mudarrisiina\n\n# Works:\n\n# ar-rajulu al-ladhii al-waladu _a9Taa as-sagaayira li binti hu dhahaba\n# ar-rajulu _a9Taa li binti hu as-sagaayira\n# ar-rajulu al-ladhii waladu hu _a9Taa as-sagaayira li al-binti dhahaba\n\n# Won't work:\n\n# ar-rajulu al-ladhii al-waladu _a9Taa li binti hu as-sagaayira dhahaba\n# ar-rajulu al-ladhii waladu hu _a9Taa li al-binti as-sagaayira dhahaba\n\n\n\nfamily PossClitic(Cli, indexRel=poss) {\n entry: n<~1>[X, def, nonres] \\* n<1>[X, cons]: X:sem-obj(*);\n entry: (n<~1>[X, def, nonres] / n<2>[res]) \\* n<1>[X, cons]: X:sem-obj(*);\n member: ii, ka, ki, hu, haa, naa, kum, kunna, hum, hunna;\n}\n\nfamily ObjClitic(Cli, indexRel=poss) {\n entry: (s$1 | n<3>[nonres]) \\ (s$1 / n<2>[nonres] | n<3>[nom,nonres]);\n entry: (s$1 / n<~2>[res] | n<3>[nonres]) \\ (s$1 / n<2>[nonres] | n<3>[nom,nonres]);\n member: nii, ka, ki, hu, haa, naa, kum, kunna, hum, hunna;\n}\n\nfamily Adj(indexRel=adj) {\n entry: n<2>[X, NUM, GEND, CASE, STATE] \\ n<2>[X]:\n X:sem-obj(*);\n}\n\nfamily Prep-Nom(Prep, indexRel=\"*NoSem*\") {\n # The pp<~3> notation generates an 'inheritsFrom' tag rather than\n # an 'id' tag for the feature structure.\n entry: pp<~3>[lex=*] /< n<3>[gen,nonres];\n member: li, fii;\n}\n\nfamily Comp(indexRel=\"*NoSem*\") {\n entry: sbar<~1>[lex=*] / s<1>;\n member: anna, inna;\n}\n\nfamily IntransV(V) {\n entry: s[E] | n<2>[X,NUM,PERS,GEND,non-cons,nonres,nom]:\n\tE:action(* ^ X:animate-being);\n}\n\nfamily SayV(V) {\n entry: s[E] / sbar[Z, lex=inna] | n<2>[X,NUM,PERS,GEND,non-cons,nonres,nom]:\n E:action(* ^ X:animate-being ^ Z);\n}\n\nfamily ThinkV(V) {\n entry: s[E] / sbar[Z, lex=anna] | n<2>[X,NUM,PERS,GEND,non-cons,nonres,nom]:\n E:action(* ^ X:animate-being ^ Z);\n}\n\nfamily TransV(V) {\n entry: s[E] / n[Y,acc,nonres] | n<2>[X,NUM,PERS,GEND,non-cons,nonres,nom]:\n E:action(* ^ X:animate-being ^ Y:sem-obj);\n}\n\nfamily DitransV(V) {\n # The first slash (on the pp) is marked with a mode allowing backward xcomp.\n entry: s[E] /< pp[Z,lex=li] / n[Y,acc,nonres] | n<2>[X,NUM,PERS,GEND,non-cons,nonres,nom]:\n E:action(* ^ X:animate-being ^ Y:sem-obj ^\n\t Z:animate-being) ;\n}\n\ntestbed {\n # different states of subject\n rajulun dhahaba: 1;\n ar-rajulu dhahaba: 1;\n rajulu dhahaba: 0;\n\n # subject-verb agreement\n rajulun dhahabuu: 0;\n ar-rajulu dhahabuu: 0;\n rajulu dhahabuu: 0;\n\n # gender agreement\n ar-rajulu dhahaba: 1;\n ar-rajulu dhahabat: 0;\n al-bintu dhahaba: 0;\n al-bintu dhahabat: 1;\n\n # possession\n rajulun hu dhahaba: 0;\n ar-rajulu hu dhahaba: 0;\n rajulu hu dhahaba: 1;\n\n # subject case\n ar-rajula dhahaba: 0;\n ar-rajuli dhahaba: 0;\n\n # case in construct phrase\n _axuu ar-rajuli dhahaba: 1;\n _axuu ar-rajula dhahaba: 0;\n _axuu ar-rajulu dhahaba: 0;\n\n # construct state in construct phrase\n _axun ar-rajuli dhahaba: 0;\n al-_axu ar-rajuli dhahaba: 0;\n\n # object case\n ar-rajulu ra_aa al-kitaaba: 1;\n ar-rajulu ra_aa al-kitaabi: 0;\n ar-rajulu ra_aa al-kitaabu: 0;\n\n # preposition case\n ar-rajulu _a9Taa al-kitaaba li al-waladi: 1;\n ar-rajulu _a9Taa al-kitaaba li al-waladu: 0;\n ar-rajulu _a9Taa al-kitaaba li al-walada: 0;\n\n # subcategorization\n ar-rajulu ra_aa al-kitaaba li al-waladi: 0;\n\n # backward xcomp\n ar-rajulu _a9Taa li al-waladi al-kitaaba: 1;\n _a9Taa ar-rajulu li al-waladi al-kitaaba: 1;\n\n # object clitics\n ana ra_aytu hu: 1;\n ra_aytu hu ana: 1;\n ra_aytu ana hu: 0;\n hu ra_aytu ana: 0;\n hu ana ra_aytu: 0;\n\n huwa ra_aa nii: 1;\n huwa ra_aa ii: 0;\n huwa ra_aa ana: 0;\n\n ar-rajulu _a9Taa haa li al-waladi: 1;\n _a9Taa haa ar-rajulu li al-waladi: 1;\n\n # relative clauses\n # \"I gave it to the man that the girl saw him\"\n _a9Taytu haa li ar-rajuli al-ladhii al-bintu ra_at hu: 3;\n # \"I gave it to the man that the girl saw her\"\n _a9Taytu haa li ar-rajuli al-ladhii al-bintu ra_at haa: 0;\n # \"I gave it to the man that the girl saw\"\n _a9Taytu haa li ar-rajuli al-ladhii al-bintu ra_at: 0;\n # \"I gave it to the man that the girl saw the boy\"\n _a9Taytu haa li ar-rajuli al-ladhii al-bintu ra_at al-walada: 0;\n}\n" }, { "path": "ccg-format-grammars/inherit/inherit.ccg", "content": "# A minimal grammar which shows inheritance.\n# Jason Baldridge, September 2007\n\n################## Features #################\n\nfeature {\n case: nom acc;\n}\n\n################## Words #################\n\nword John:NP (pred=john);\nword Fido:NP (pred=fido);\nword food:NP (pred=food);\nword to:PP;\nword sleeps:IntransV (pred=sleep);\nword saw:TransV (pred=see);\nword gave:DitransV (pred=give);\n\n################## Categories #################\n\nfamily NP {\n entry: np[X]:X(*); \n}\n\nfamily PP {\n entry: pp[X]/np[X]:X(*); \n}\n\ndef iv_cat (PostSyn, MoreSem) {\n s[E] \\ np[X nom] PostSyn: E(* X MoreSem)\n}\n\ndef tv_cat (PreSyn, PostSyn, MoreSem) {\n iv_cat(PreSyn / np[Y acc] PostSyn, Y MoreSem)\n}\n\nfamily IntransV(V) {\n entry: iv_cat(,);\n}\n\nfamily TransV(V) {\n entry: tv_cat(,,);\n}\n\nfamily DitransV(V) {\n entry: tv_cat( , / np[Z acc] , Z);\n entry: tv_cat(/ pp[Z acc] , , Z);\n}\n\n\n################## Test sentences #################\n\ntestbed {\n John sleeps: 1;\n John saw Fido: 1;\n John gave Fido food: 1;\n John gave food to Fido: 1;\n}\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccg-format-grammars/tiny/tiny.ccg", "content": "#############################################################\n# #\n# tiny.ccg #\n# #\n#############################################################\n\n# Author: Ben Wing \n# Date: April 2006\n\n# This is derived from Geert-Jan M. Kruijff's 'coordination' grammar,\n# which was simplified, revised and extended.\n# \n# David Reitter, dreitter at inf.ed.ac dot uk, 01/2005\n# Michael White, mwhite at inf dot ed.ac.uk, 01/2005\n\n# This is the `tiny' grammar, in .ccg format. There are a bunch\n# of comments below to explain the format. See also the `arabic'\n# grammar for an example of extended macro use to handle a complex\n# morphology.\n\n# To generate a grammar for use with OpenCCG, use `genccg' (or whatever\n# it's currently called). This generates the various XML files needed for\n# OpenCCG (grammar.xml, lexicon.xml, morph.xml, types.xml, and rules.xml).\n# After doing this, you can run `tccg' to load the grammar and use it for\n# parsing.\n\n# Some general notes about this format:\n\n# -- The general feel of the syntax is like C, Java or Perl. Indentation\n# and whitespace is unimportant. (The only exception is in macro\n# definitions, where the text of a macro must either be on the same line\n# as the definition -- possibly extended with backslash\n# line-continuation markers -- or be enclosed in braces.)\n#\n# -- The syntax tries to be very forgiving of the usage of commas and\n# semicolons, for the benefit of macro definitions. In most lists, in\n# fact, commas are optional -- arguments can be separated by no commas,\n# one comma, or many commas, and extra commas can occur at the end of\n# the list. All of this makes macro definitions and macro calls much\n# easier. We usually write such lists below without commas.\n#\n# The main exception where commas matter is in macro calls. You can\n# still put an extra comma at the end of a macro call, but otherwise you\n# must have exactly one comma (no more, no less) between arguments. The\n# reason is that macro arguments can contain pretty much any text\n# whatsoever (including no text at all), so commas are needed to\n# indicate where one argument stops and the next one starts.\n#\n# NOTE: The parser pays attention to parens, brackets, and braces in the\n# text of a macro call argument, and will not get confused by commas\n# inside of matched delimiters. Thus, a macro call `foo(a, bar(b, c))'\n# is correctly interpreted as a call to foo() with two arguments, `a'\n# and `bar(b, c)', and *NOT* a call to foo() with three arguments `a',\n# `bar(b', and `c)'.)\n#\n# Note also how the text of macro definitions and calls can optionally\n# be surrounded by braces, to clearly delimit the text boundaries (see\n# below). It's important to note that the braces do *NOT* form part of\n# the text itself; if you really want your text surrounded by braces,\n# you need to add a second set around the text.\n#\n# -- You can use the `-t' option to see what things look like after macro\n# substitutions have been applied.\n#\n# -- The order of declarations in this file is not significant except\n# that macros and features must be defined before they can be used.\n#\n# -- You can surround literal text (word stems, inflections, etc.) with\n# single or double quotes in order to insert characters into the text\n# that would otherwise be interpreted by the parser, such as spaces,\n# colons, etc. Without such quotes, the only characters that can occur\n# in a word are letters, numbers, '_', '-', '+' and '%'. Note, however,\n# that there is no such thing as \"reserved words\" in this format; words\n# like 'feature', 'word', 'entry', etc. that have a special significance\n# in the right place in the syntax can otherwise be freely used as word\n# stems and inflections, macro variable names, part of speech tags, etc.\n\n#############################################################\n# #\n# FEATURES #\n# #\n#############################################################\n\n# Declaration of features. This is used as follows:\n#\n# (1) So that you can just specify a feature value inside brackets below,\n# and the appropriate feature is substituted. That is, a declaration\n# like n[nom] is equivalent to n[CASE=nom].\n#\n# (2) To specify the types listed in types.xml.\n#\n# (3) To specify the feature values used in word declarations. These\n# compile out into declarations in morph.xml (unrelated to\n# our own use of macros). In order for this to happen, however, you\n# *must* specify a \"macro-tie\", inside of parentheses after the feature\n# type. This is either a number (for a syntactic macro) or a letter\n# (for a semantic macro). See below for exactly how these are used.\n#\n# (A \"strict feature\" mode may be implemented at some point, where all\n# features that are used in category definitions must be declared.)\n\nfeature {\n # A number following the feature type, as follows, is used for syntactic\n # features that can be attached to a word. The number is a feature-\n # structure ID; when an atomic category in a lexical category definition\n # has the corresponding ID, these features will be inserted into that\n # category. Hence, the <2> here corresponds to the <2> that occurs\n # below in the definition of Noun, Det, verbcat(), etc.\n CASE<2>: nom acc;\n NUM<2>: sg pl;\n # You can specify more than one number if you want.\n # NUM<2,3>: sg pl;\n\n # A word in place of a number causes semantic macros to be created; if\n # used in a word {} declaration, the relevant info will be inserted into\n # the hybrid logic part of a lexical category declaration, attached to\n # the argument of the given name. A declaration like (X:NUM) means that\n # 'NUM' is used in the XML declaration in place of\n # the feature's actual name. (FIXME: I don't know what the significance\n # of this is.)\n sem-num: sg-X pl-X;\n TENSE: past pres;\n\n # You can create hierarchical values as shown.\n PERS<2>: non-3rd {1st 2nd} 3rd;\n\n # Alternatively, you can explicitly list the parent(s) of a feature --\n # e.g. to create multiple-inheritance hierarchies.\n #alternate-pers<2>: non-3rd 1st[non-3rd] 2nd[non-3rd] 3rd;\n\n # Here's a more complicated hierarchy, from the original tiny grammar.\n ontology: sem-obj {\n phys-obj {\n animate-being {\n person\n }\n thing\n }\n situation {\n change {\n action\n }\n state\n }\n };\n\n # Here we show how you can mix the two ways of declaring hierarchies,\n # if you have a primarily single-inheritance hierarchy but with certain\n # multiply-inherited values. In this hierarchy, 'werewolf' gets both\n # 'man' and 'wolf' as its parents. 'intersex' will have only 'man' and\n # 'woman' as parents -- 'rational-being' will not be specified as it's\n # redundant.\n entity: being {\n\t # We would call this `thing' but that is used above in\n # the `ontology' hierarchy, and causes a warning to be issued\n thing2\n animate {\n irrational-being {\n fish\n mammal { dog-like {dog wolf} cat horse }\n }\n rational-being {\n man {\n centaur[horse]\n werewolf[wolf]\n }\n woman {\n mermaid[fish]\n }\n intersex[man woman]\n }\n }\n }\n nothingness;\n}\n\n# You can also specify that a feature is \"distributive\" and/or give\n# licensing information for the realizer.\n#\n# Here's the equivalent of the stuff in flights/lexicon.xml and\n# flights/types.xml.\n#\n# A ! before a feature makes it distributive. You can specify\n# licensing-related attributes on either a feature or a feature-value, in\n# the standard way of specifying attributes in .ccg format.\n\nfeature {\n !owner(location=args-only, instantiate=false);\n !info;\n form: dcl-base {dcl, fronted},\n q-base {q(license-marked-cats=true, also-licensed-by=q-base),\n wh(license-marked-cats=true, also-licensed-by=q-base)},\n base,\n emb(location=target-only),\n inf(location=target-only),\n adj(location=target-only),\n ng;\n}\n\n# If you really want the feature value as used in the word {} declarations\n# to be different from the feature value elsewhere, you can do that.\n# This allows you to generate the following XML:\n#\n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n#\n#feature {\n# case<0>: acc0:p-case;\n# case<1>: acc1:p-case;\n# case<2>: acc2:p-case;\n# case<3>: acc3:p-case;\n#}\n\n# NOTE: (1) This doesn't quite work in ccg2xml yet, since only `acc3' gets\n# added to the hierarchy in types.xml. (2) The entire motivation for doing\n# this kind of thing is kludgy, and will be eliminated by allowing features\n# to be explicitly specified for the result of a lexical insertion rule,\n# much like the way that type-raising rules currently work.\n\n# Here's how you specify a relation-sorting order, in case you care.\nrelation-sorting: foo * bar baz;\n\n#############################################################\n# #\n# WORDS #\n# #\n#############################################################\n\n#\n################## Functional words #################\n#\n\n# Some examples of words.\n#\n# The format of word declarations is\n#\n# word STEM:FAMILY ...(ATTRS): FEATURES;\n#\n# or\n#\n# word STEM:FAMILY ...(ATTRS) { INFLECTED-FORM: FEATURES; ...}\n#\n# where STEM is the word's stem, FAMILY is a list of the families that a\n# word is part of, and ATTRS specifies any other attributes associated\n# with the word. FEATURES gives the word's features; these come from the\n# feature {} declarations above. (NOTE: Only feature values whose features\n# specify a \"macro-tie\" value -- something in <> following the\n# feature's name -- can be used. See above.)\n#\n# ATTRS is a list; each attribute is either a specification ATTRIBUTE=VALUE\n# or a single VALUE (equivalent to class=VALUE). The useful attributes are\n#\n# class\t\tSemantic class of a word.\n# pred\t\tSemantic predicate of a word, used in the logical form;\n# if omitted, defaults to the word's stem.\n# excluded List of excluded lexical categories.\n# coart Boolean indicating that this entry is a coarticulation,\n# eg a pitch accent, gesture, or other word-associated element.\n#\n# Any of FAMILY, ATTRS and/or FEATURES can be omitted.\n#\n# The second form above, with braces, is used for words with different\n# inflections. Instead of specifying the features directly after the word,\n# you list the features for each inflection separately. Note that * is\n# shorthand for the stem itself.\n#\n# Note that there can be more than one word {} declaration for a single stem.\n#\n# The families in FAMILY can be either a family name, from a family {}\n# block, or a part of speech. (`genccg' will derive the appropriate parts\n# of speech from any families given when creating the XML file.) Note that\n# the words associated with a particular family can be specified either by\n# tagging each word with its family, by listing a family's words explicitly\n# using the `member' declaration inside of a family {} block, or by a\n# combination of the two.\n\nword the:Det;\nword some:Det;\nword a:Det: sg;\n\nword for;\n\nword pro1:Pro(animate-being) {\n I: 1st sg nom sg-X;\n me: 1st sg acc sg-X;\n we: 1st pl nom pl-X;\n us: 1st pl acc pl-X;\n}\n\nword pro2:Pro(animate-being) {\n you: 2nd;\n}\n\nword pro3f:Pro(animate-being) {\n she: 3rd sg nom sg-X;\n her: 3rd sg acc sg-X;\n}\n\nword pro3m:Pro(animate-being) {\n he: 3rd sg nom sg-X;\n him: 3rd sg acc sg-X;\n}\n\nword pro3n:Pro(thing) {\n it: 3rd sg sg-X;\n}\n\n# If we want the CLASS associated with only some of the inflections,\n# we can use the same stem in more than one decl. (Assigning neuter to\n# \"they\" is strange but that's how the original morph.xml did it!)\n\nword pro3n:Pro {\n they: 3rd pl nom pl-X;\n them: 3rd pl acc pl-X;\n}\n\n#\n################## Nouns #################\n#\n\n# Here, we make use of macros. The basic idea behind macros is simple:\n# They simply do string substitution. However, parameters can be given,\n# making them very powerful.\n\n# This macro says: Every time an expression of the form basic-noun(...) \n# occurs, replace it with the text that comes after. The parameters will\n# be substituted into the text. The braces that denote the macro's text do\n# *NOT* form part of the text that is substituted. (Alternatively, you\n# can put the entire replacement text on the same line as the macro\n# definition, using a backslash at the end of the line if needed in order\n# to continue the definition on the next line.) Likewise, braces can be\n# used to surround text in an argument to a macro call and will not form of\n# the argument's text. This is useful when the argument's text contains\n# commas. In either case, if you really want the text itself to have\n# braces around it, you need to put two sets of braces around it.\n\n# Note that macro substitutions are processed recursively: If the\n# text of a macro substitution contains calls to other macros, they will\n# also be processed. This makes \"inheritance\" very easy to implement.\n\n# Inside of a macro definition's text, the . operator can be used; this\n# concatenates two words together into a single word. See the definition\n# of normal-noun() below for a simple example.\n\n# Substitution of arguments does not occur inside of quoted text.\n\ndef basic-noun(sing, plur, class) {\n word sing:N(class) {\n *: sg sg-X;\n plur: pl pl-X;\n }\n}\n\n#def normal-noun(stem, class) {\n# word stem:N(class) {\n# *: sg sg-X;\n# stem . s: pl pl-X;\n# }\n#}\n\n# or equivalently, using our definition of basic-noun():\n\n# def normal-noun(stem, class) basic-noun(stem, stem . s, class)\n\n# But in fact, we do something more clever to handle pluralization.\n# Here we make use of some built-in macros():\n#\n# regsub(PATTERN, REPLACEMENT, TEXT):\n# Replace all occurrences of regexp PATTERN with REPLACEMENT in TEXT.\n# This follows normal Python conventions for regular expression\n# substitution.\n#\n# ifmatch(PATTERN, TEXT, IF-TEXT, ELSE-TEXT):\n# If TEXT matches PATTERN at its beginning, substitute IF-TEXT; else,\n# substitute ELSE-TEXT.\n#\n# ifmatch-nocase(PATTERN, TEXT, IF-TEXT, ELSE-TEXT):\n# Just like ifmatch() but its pattern-matching is case-insensitive.\n\ndef pluralize(word) {\n # This shows a complicated expression involving the built-ins\n # 'ifmatch' and 'regsub'. If the word ends in a vowel + o or y,\n # we add s. Else, if the word ends in (consonant) + o or y, or s, sh, ch,\n # or x, we change y to i and add es. Finally, in all other cases,\n # just add s. So buy -> buys, boy-> boys, but try -> tries, lady -> ladies.\n # Similarly, go -> goes but goo -> goos. For words like volcano -> volcanos\n # you have to put the forms in manually (or change the rule here, and put\n # forms in -o + es in manually, e.g. does, goes, tomatoes, potatoes).\n #\n ifmatch('^.*[aeiou][oy]$', word, word . s,\n ifmatch('^.*([sxoy]|sh|ch)$', word, regsub('^(.*)y$', '\\1i', word) . es,\n word . s))\n}\n\ndef noun(sing, class) basic-noun(sing, pluralize(sing), class)\n\nnoun(book, thing)\nnoun(DVD, thing)\nnoun(glass, thing)\nnoun(church, thing)\nnoun(flower, thing)\nnoun(bath, thing)\nnoun(teacher, person)\nnoun(lady, person) # Pluralized (correctly) to 'ladies'\nnoun(boy, person) # Pluralized (correctly) to 'boys'\n\n# An irregular noun.\nbasic-noun(policeman, policemen, person)\n\n#\n################## Verbs #################\n#\n\n# The \"props\" parameter specifies families or attributes.\n\ndef basic-verb(stem, props, 3sing, pasttense) {\n word stem:props {\n *: pres non-3rd sg;\n 3sing: pres 3rd sg;\n *: pres pl;\n pasttense: past;\n }\n}\n\ndef verb(stem, props, pasttense) \\\n basic-verb(stem, props, pluralize(stem), pasttense)\n\nverb(buy, TransV DitransBenV, bought)\nverb(rent, TransV DitransBenV, rented)\nverb(go, IntransV, went)\nverb(sleep, IntransV, slept)\n# Here we show how you can specify a predicate or other attribute.\n# Admittedly this is not too useful here, but it can be much more so in\n# the case of a foreign language, where we want the semantic predicates\n# to be in English. (See arabic.ccg.)\nverb(eat, TransV IntransV (pred=manjar), ate)\n\n#############################################################\n# #\n# RULES #\n# #\n#############################################################\n\n# This declaration specifies the contents of rules.xml. Each statement\n# specifies a single rule; it is also possible for statements to cancel\n# some or all rules.\n#\n# Note that some rules are enabled by default; this includes application,\n# composition and crossed composition (forward and backward in each case),\n# as well as forward type-raising from np to s/(s\\np) and backward\n# type-raising from np to s$1\\(s$1/np).\n\nrule {\n # turn off forward cross-composition\n no xcomp +;\n\n # this is how we could turn off all type-raising rules.\n # no typeraise;\n\n # Declare a backward type-raising rule from pp to s$1\\(s$1/pp).\n # The $ causes a dollar-sign raise category to be created, as shown;\n # without it, we'd just get s\\(s/pp).\n typeraise - $: pp => s;\n\n # Declare a type-changing rule to enable pro-drop (not useful in English!)\n # typechange: s[finite]\\np[nom]$1 => s[finite]$1 ;\n}\n\n# This shows how you can turn off all defaults and specify your own\n# properties from scratch, if you want.\n\n# rule {\n# no; # remove all defaults\n# app +-;\n# comp +-; # +- means both forward and backward\n# xcomp -;\n# sub +-;\n# xsub +-;\n# # Defaults for typeraising are np => s, if omitted.\n# typeraise +;\n# typeraise - $;\n# }\n# \n\n#############################################################\n# #\n# CATEGORIES #\n# #\n#############################################################\n\n# Specify lexical families and the lexical insertion rules for each\n# family. Properties of the family can be given in parens after the\n# family name. The family name should either be the same as a part\n# of speech, or the part of speech should be given in parens after\n# the family name.\n\n# Categories can be specified in an expected form, e.g. s\\np[nom]/np[acc].\n# The notation np<3>[acc] corresponds to XML code like this:\n# \n# \n# \n# \n# \n# i.e. the <3> specifies the feature-structure ID, and the [acc]\n# specifies a constraint. In this case, a constraint of the form\n# [CASE=acc] is generated because of the feature {} declaration above.\n# If an unknown value is given, e.g. [foo], it's assumed to be a feature,\n# rather than a feature value, and you get code like\n# \n# \n# You can also write [foo=bar] to explicitly give a feature and value.\n# And an entry like [X] corresponds to\n# \n# Note also that slashes can be followed by a slash mode, e.g. /<, or\n# the mode can be omitted; in this case, a default mode is generated:\n# \\<, />, |.\n\nfamily Det(indexRel=det) {\n entry: np<2>[X PERS=3rd] /^ n<2>[X]:\n X:sem-obj(*);\n}\n\nfamily Prep-Nom(Prep, indexRel=\"*NoSem*\") {\n # You can name your entries, as shown, although it's not clear there's\n # much point.\n #\n # The pp<~3> notation generates an 'inheritsFrom' tag rather than an 'id'\n # tag for the feature structure. This unifies only the properties not\n # explicitly given in the category, i.e. everything but 'lex' will unify.\n #\n # The entry [lex=*] corresponds to \n # and means that a feature 'lex' will be attached, whose value is the\n # word stem.\n entry Nominal: pp<~3>[lex=*] /< np<3>[acc];\n\n # If members are specified, the family defaults to 'closed'.\n member: for;\n}\n\nfamily Conj {\n entry: np[NUM, X0] \\* np[X1] / np[X2]:\n X0(and ^ (L1 ^ elem ^ X1 ^\n\t (L2 ^ elem ^ X2)));\n}\n\nfamily DollarTest {\n entry: s\\np$1\\*(s\\np);\n entry: s\\(np$1)\\*(s\\np);\n}\n\nfamily N {\n entry: n<2>[X NUM]: X:sem-obj(*);\n}\n\nfamily Pro {\n entry: np<2>[X NUM PERS CASE]:\n X:sem-obj(*);\n}\n\n# Here we create a macro to describe the category for a verb, with transitive\n# arguments ARGS (possibly empty) and corresponding semantics SEM.\n# Don't call this macro just \"verb\" since that's used above already!\n\ndef verbcat(args, sem) {\n # A * here corresponds to .\n # Similar use of * appears above in [lex=*] in prepositions,\n # and * in determiners.\n s<1>[E] \\ np<2>[X NUM PERS nom] args:\n E:action(* X:animate-being sem)\n}\nfamily IntransV(V) {\n entry: verbcat(,);\n}\n\n# This shows how we could extend verbcat() to handle transitive verbs.\n# Since the arguments for ditransitive verbs could potentially either\n# before or after the transitive argument, we need two different macros.\n# Or, we could create one macro, with two different parameters for \"before\"\n# and \"after\" arguments.\n\ndef before_transverbcat(args, sem) {\n verbcat(args / np<3>[Y acc], Y:sem-obj sem)\n}\n\ndef after_transverbcat(args, sem) {\n verbcat(/ np<3>[Y acc] args, Y:sem-obj sem)\n}\n\nfamily TransV(V) {\n entry: after_transverbcat(,);\n}\n\nfamily DitransBenV(V) {\n\n # Careful here! Remember that the arguments in a CCG category will be\n # listed in backwards order compared to how they appear in the surface\n # syntax. Hence SUBJ VERB X Y Z corresponds to s\\np/z/y/x.\n\n entry DTV:\n after_transverbcat(/< np[Z acc], Z:animate-being);\n entry NP-PPfor:\n before_transverbcat(/ pp[Z lex=for], Z:animate-being);\n\n # This shows how we could define the previous entry directly in\n # terms of verbcat().\n #entry NP-PPfor: verbcat(/ pp<4>[Z lex=for] /< np<3>[Y acc],\n # Y:sem-obj Z:animate-being);\n}\n\n#############################################################\n# #\n# TESTBED #\n# #\n#############################################################\n\n# Statements to put in testbed.xml. If you omit the number, it will omit\n# the corresponding numOfParses item in the XML; I think this currently\n# is equivalent to specifying 1 as the number. If you put a ! before a\n# line, this indicates a \"known failure\" (known=\"true\" in the XML).\n\ntestbed {\n the teacher buys the policeman a book: 1;\n the teacher buys the policemen some flowers: 1;\n the teachers buy a book: 3;\n she buys it: 3;\n she buy it: 0;\n # Possible example of a known failure, in case we know we don't correctly\n # reject \"she buy buy\":\n ! she buy buy: 0;\n they buys it: 0;\n she buys the flower for him: 1;\n she buys the flower for he: 0;\n her buys the flower for him: 0;\n he rented her a DVD: 1;\n he rented a DVD for her: 1;\n he rented a DVD her: 0;\n}\n\n" }, { "path": "ccg-format-grammars/tinytiny/tinytiny.ccg", "content": "# A truly minimal grammar for CCG.\n# Ben Wing, May 2006\n\n################## Features #################\n\n\n\nfeature {\n CASE<2>: nom acc;\n NUM<2>: sg pl;\n PERS<2>: non-3rd {1st 2nd} 3rd;\n\n TENSE: past pres;\n SEM-NUM: sg-X pl-X;\n\n# Some alternate code that appeared from somewhere, I'm not sure anymore.\n# Enable the following three statements and disable the statement above\n# beginning `NUM<2>: ...'. It looks like it tries to implement something to\n# do with number agreement. (Note that number agreement is already taken care\n# of in the normal system. I'm not sure what the code is trying to do.)\n# num<2,X:num>: sg pl;\n# num<1,2>: sg-agr:sg pl-agr:pl;\n# num<2>: sg-2:sg pl-2:pl;\n\n ontology: sem-obj {\n phys-obj {\n animate-being {\n person\n }\n thing\n }\n situation {\n change {\n action\n }\n state\n }\n };\n}\n\n################## Words #################\n\n# Example of how to have punctuation and other non-word characters in a\n# lexical item.\nword '.':Punc;\nword ',':Punc;\nword '?':Punc;\n\nword the:Det;\nword a:Det: sg;\n\nword pro1:Pro(animate-being) {\n I: 1st sg nom sg-X;\n me: 1st sg acc sg-X;\n we: 1st pl nom pl-X;\n us: 1st pl acc pl-X;\n}\n\ndef noun(sing, plur, class) {\n word sing:N(class) {\n *: sg sg-X;\n plur: pl pl-X;\n }\n}\n\nnoun(book, books, thing)\nnoun(peach, peaches, thing)\nnoun(boy, boys, person)\nnoun(policeman, policemen, person)\n\ndef verb(stem, props, 3sing, pasttense) {\n word stem:props {\n *: pres non-3rd sg;\n 3sing: pres 3rd sg;\n *: pres pl;\n pasttense: past;\n }\n}\n\nverb(go, IntransV, goes, went)\nverb(sleep, IntransV, sleeps, slept)\nverb(eat, TransV IntransV, eats, ate)\nverb(see, TransV, sees, saw)\n# This is a test of set arguments. Not clear it's useful or should be here;\n# maybe move to tiny.ccg?\nverb(setverb, SetargV, setverbs, setverbed)\n\n################## Categories #################\n\nfamily Punc {\n entry: punc;\n}\n\nfamily Det(indexRel=det) {\n entry: np<2>[X PERS=3rd]/^ n<2>[X]:\n X:sem-obj(*);\n}\n\nfamily N {\n entry: n<2>[X]:\n X:sem-obj(*);\n}\n\nfamily Pro {\n entry: np<2>[X]:\n X:sem-obj(*); \n}\n\nfamily IntransV(V) {\n entry: s<1>[E] \\ np<2>[X nom]:\n E:action(* X:animate-being);\n}\n\nfamily TransV(V) {\n entry: s<1>[E] \\ np<2>[X nom] / np<3>[Y acc]:\n E:action(* X:animate-being Y:sem-obj);\n}\n\nfamily SetargV(V) {\n entry: s<1>[E] {\\np<2>[X nom], / np<3>[Y acc]}:\n E:action(* X:animate-being Y:sem-obj);\n}\n\n################## Test sentences #################\n\n\ntestbed {\n\tthe policemen eat:\t2;\n the boys eat: 2;\n the boys eat the peaches: 1;\n the policeman sleeps: 1;\n the policemen sleeps: 0;\n the policemen sleep: 1;\n the policeman sleeps the peach: 0;\n the policeman saw me: 1;\n the policeman saw I: 0;\n I see the book: 1;\n I sees the book: 0;\n I see a book: 1;\n I see a books: 0;\n}\n\n" }, { "path": "ccgbank/bin/american-to-logical-quotes.py", "content": "\"\"\"\nCopryright (c) 2011 Dennis N. Mehay\n\nAssumes tokenized, PTB3-normalized UTF-8 text, one sentence per line.\n\n => \n\nTurns 'American'-style quotations into 'British'/'logical'-style quotations.\n\nSo, e.g.,\n\n`` Hello , '' said John .\n\nbecomes:\n\n`` Hello '' , said John .\n\n[Insert LGPL here]\n\"\"\"\n\nimport sys, codecs, os\n\nstreamReader = codecs.lookup(\"utf-8\")[2]\nstreamWriter = codecs.lookup(\"utf-8\")[-1]\n\nsys.stdin = streamReader(sys.stdin)\nsys.stdout = streamWriter(sys.stdout)\n\nfor ln in sys.stdin:\n # trim off extra whitespace and replace double spaces with single spaces.\n ln = ln.strip().replace(u\" \", u\" \")\n # now replace\n # , ''\n # with\n # ''\n # and\n # . ''\n # with\n # '' .\n ln = ln.replace(u\" , ''\", \" '' ,\").replace(u\" . ''\", \" '' .\")\n # now fix any double-punct messes this might have created.\n ln = ln.replace(u\" '' . ?\", u\" . '' ?\").replace(u\" '' . !\", u\" . '' !\")\n sys.stdout.write(ln + '\\n')\n" }, { "path": "ccgbank/bin/convert-mtc-systems.py", "content": "\"\"\"\nThis program takes a set of documents (all streamed from stdin at once) and formats them in a way suitable for\nuse with the NIST-distributed mteval script. The output is in UTF-8.\n\nUsage: cat [MTC_DIR_FOR_SYSTEM] | python convert-mtc.systems.py [doctype-string {'source', 'target', 'ref'} (default='target')] | [NEW_XML_DOC_TO_STDOUT]\n\"\"\"\nimport sys, os, re, codecs, xml.sax.saxutils, my_unicode\ntry:\n import chardet\nexcept:\n chardet = None\nfrom xml.etree.ElementTree import *\nfrom collections import defaultdict\n\ndef tokenize(t):\n \"\"\"very simple text tokenization:\n n't => n't\n 's => 's\n ' => '\n where '' is not whitespace.\n \"\"\"\n t = t.replace(\"n't\", \" n't\").replace(\"'s\", \" 's\").replace(\"' \", \" ' \")\n if t[-1] == \"'\":\n t = t[:-1] + \" \" + t[-1]\n return t.replace(\" \", \" \")\n\ndef decode_line(ln, encoding):\n res = None\n try:\n res = ln.decode(encoding)\n except:\n try:\n res = ln.decode(\"iso-8859-2\")\n except:\n try:\n res = ln.decode(\"iso-8859-1\")\n except:\n try:\n res = ln.decode(\"utf-8\")\n except:\n try:\n res = ln.decode(\"GB2312\")\n except:\n try:\n res = ln.decode(\"Big5\")\n except:\n try:\n res = ln.decode(\"EUC-TW\")\n except:\n res = ln\n \n return res\n\ndoc_pattern = re.compile(u\"<[Dd][Oo][Cc] docid=\\\"(.*)\\\" sysid=\\\"(.*)\\\">(.*)\")\ndoc_pattern_source = re.compile(u\"<[Dd][Oo][Cc] docid=\\\"(.*)\\\">(.*)\")\nseg_pattern = re.compile(u\"(.*)$\")\n\ndoc_type = \"target\"\nif len(sys.argv) >= 2:\n doc_type = sys.argv[1].lower()\nif not doc_type in [\"target\", \"source\", \"reference\"]:\n doc_type = \"target\"\n \nmtc_in = sys.stdin.readlines()\nmtc_all = (os.linesep).join(mtc_in)\n\nif not chardet is None:\n encoding = chardet.detect(mtc_all)['encoding']\nelse:\n encoding = \"ISO-8859-2\"\n\n# turn stdout into a UTF-8 converting writer.\nstreamWriter = codecs.lookup(\"UTF-8\")[-1]\nsys.stdout = streamWriter(sys.stdout)\noutput = sys.stdout\n\n# map from auto-assigned ID to MTC ID.\nautoid2mtcid = {}\n\nmtc = defaultdict(lambda: [])\nsys = None\nfor l in mtc_in:\n l = decode_line(l, encoding).strip()\n if l.startswith(\"\" + \\\n os.linesep + \"\" + os.linesep +\\\n \"\" + os.linesep)\nif doc_type == \"target\":\n output.write(\" \" % sys + os.linesep)\nelif doc_type == \"source\":\n output.write(\" \" + os.linesep)\nelse:\n output.write(\" \" + os.linesep)\ndocs = mtc.keys()\ndocs.sort()\n\nfor (sy,doc) in docs:\n output.write(\" \" % doc + os.linesep)\n segids_and_texts = mtc[(sy,doc)]\n segids_and_texts.sort(lambda a,b: cmp(int(a[0]),int(b[0])))\n for (segid,text) in segids_and_texts:\n output.write(\" %s \" % (segid,xml.sax.saxutils.escape(my_unicode.removeInvalidChars(tokenize(text)))) + os.linesep)\n output.write(\" \" + os.linesep)\n\nif doc_type == \"target\":\n output.write(\" \" + os.linesep + \"\")\nelif doc_type == \"source\":\n output.write(\" \" + os.linesep + \"\")\nelse:\n output.write(\" \" + os.linesep + \"\")\n\n" }, { "path": "ccgbank/bin/convert-spaces-to-newlines.py", "content": "#\n# converts spaces to newlines, and newlines to special chars,\n# from stdin to stdout\n#\nimport sys, re;\n[sys.stdout.write(re.sub(' ','\\n',re.sub('\\n','',line))) for line in sys.stdin]\n" }, { "path": "ccgbank/bin/convert_all", "content": "#!/bin/bash\nfor i in 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24\ndo\n echo \"Starting conversion for section $i\"\n nice ccg-build -Dsect=$i -Dfile=* convert-puncts-aux-bbn >& logs/log.convert.$i &\ndone\n" }, { "path": "ccgbank/bin/correlate-to-judgments.py", "content": "\"\"\"\nCorrelate human judgments (streamed into sys.stdin -- e.g., from the MTC)\nto the BLEU/NIST scores in the given directories (sys.argv[1] and sys.argv[2]).\n\nDepends on rpy2 having been installed.\n\"\"\"\nimport sys, os\ntry:\n import rpy2\n import rpy2.robjects as robjs\n \nexcept ImportError:\n print >> sys.stderr, \"please install rpy2. exiting...\"\n sys.exit(-1)\n\n\n\ndef num2string(n):\n \"\"\"\n E.g., 0 => '00', 1 => '01' and 10 => '10'.\n \"\"\"\n try:\n numm = int(n)\n except:\n numm = n\n if numm < 10:\n return str(numm)\n else:\n return str(numm)\n \nhuman_judgments = []\nhuman_judgments_fluency = [] # fluency.\nhuman_judgments_acc = [] # accuracy.\nhuman_judgments_ave = [] # average of both.\n \nfor l in sys.stdin:\n l = l.strip() \n if l.startswith(\"#\"):\n continue\n lparts = l.split(\",\")\n # appending (sys,doc,judge,ref_sys,segment,fluency,accuracy)\n human_judgments.append(tuple(lparts[0:7]))\n human_judgments_fluency.append(int(lparts[5]))\n human_judgments_acc.append(int(lparts[6]))\n human_judgments_ave.append((float(lparts[5]) + float(lparts[6]))/2.0)\n\nhuman_judgments_fluency = robjs.FloatVector([float(i) for i in human_judgments_fluency])\nhuman_judgments_acc = robjs.FloatVector([float(i) for i in human_judgments_acc])\nhuman_judgments_ave = robjs.FloatVector([float(i) for i in human_judgments_ave])\n\nbleu_nist_dir1 = sys.argv[1]\nbleu_nist_dir2 = sys.argv[2]\n\n# the next two are maps from: (sys,doc,ref,segID) => score\nbleu_scores = {}\nnist_scores = {}\n\nfor f in [fl for fl in os.listdir(bleu_nist_dir1) if (\"BLEU\" in fl or \"NIST\" in fl)]:\n for l in open(bleu_nist_dir1 + os.sep + f, \"rb\").readlines():\n if l.strip() == \"\":\n continue\n (sys,ref_sys,doc,seg,bleu_or_nist_score) = l.split(\",\")\n if \"BLEU\" in f:\n bleu_scores[(sys,doc,ref_sys,\"S\"+num2string(seg))] = bleu_or_nist_score\n else:\n nist_scores[(sys,doc,ref_sys,\"S\"+num2string(seg))] = bleu_or_nist_score\n \n\nfor f in [fl for fl in os.listdir(bleu_nist_dir2) if (\"BLEU\" in fl or \"NIST\" in fl)]:\n for l in open(bleu_nist_dir2 + os.sep + f, \"rb\").readlines():\n if l.strip() == \"\":\n continue\n (sys,ref_sys,doc,seg,bleu_or_nist_score) = l.split(\",\")\n if \"BLEU\" in f:\n bleu_scores[(sys,doc,ref_sys,\"S\"+num2string(seg))] = float(bleu_or_nist_score)\n else:\n nist_scores[(sys,doc,ref_sys,\"S\"+num2string(seg))] = float(bleu_or_nist_score)\n\n\n# for both BLEU and NIST, compute rpy2 vectors that parallel the seqeuence\n# of human judgments.\n# step through the (sys,doc,judge,ref_sys,segment,fluency,accuracy) tuples.\nbleu_lst = []\nnist_lst = []\nfor (s,d,j,rs,sg,f,a) in human_judgments:\n if (s,d,rs,sg) in bleu_scores:\n bleu_lst.append(bleu_scores.get((s,d,rs,sg)))\n else:\n print \"nope\", (s,d,rs,sg)\n if (s,d,rs,sg) in nist_scores:\n nist_lst.append(nist_scores.get((s,d,rs,sg)))\n else:\n print \"nope\", (s,d,rs,sg)\n\nbleu_vec = robjs.FloatVector(bleu_lst)\nnist_vec = robjs.FloatVector(nist_lst)\n\n# compute correlations\nb_fluency = robjs.r['cor'](bleu_vec, human_judgments_fluency)\nb_accuracy = robjs.r['cor'](bleu_vec, human_judgments_acc)\nb_average = robjs.r['cor'](bleu_vec, human_judgments_ave)\n\nprint \"BLEU's Pearson correlation wrt fluency:\", b_fluency\nprint \"BLEU's Pearson correlation wrt accuracy:\", b_accuracy\nprint \"BLEU's Pearson correlation wrt the average of fluency and accuracy\", b_average\n\nnist_fluency = robjs.r['cor'](nist_vec, human_judgments_fluency)\nnist_accuracy = robjs.r['cor'](nist_vec, human_judgments_acc)\nnist_average = robjs.r['cor'](nist_vec, human_judgments_ave)\n\nprint \"NIST's Pearson correlation wrt fluency:\", nist_fluency\nprint \"NIST's Pearson correlation wrt accuracy:\", nist_accuracy\nprint \"NIST's Pearson correlation wrt the average of fluency and accuracy\", nist_average\n" }, { "path": "ccgbank/bin/filter_feats.py", "content": "#!/usr/bin/env python\n\"\"\"\n(c) 2008 Dennis N. Mehay\nUse this file any way you want, just please give the\nauthor credit if it makes it into any research in any \nmeaningful way. I make no claims whatsoever about the\nfitness or merchantability of this code. Use at\nyour own risk.\n\"\"\"\nimport sys, math\nfrom optparse import OptionParser as OP\n\npr = OP()\npr.add_option(\"-i\",\"--input\",type=\"string\",help=\"input source [default=].\",\\\n default=sys.stdin)\npr.add_option(\"-o\",\"--output\",type=\"string\",help=\"output location [default=].\",\\\n default=sys.stdout)\npr.add_option(\"-n\",\"--number\",type=\"int\",\\\n help=\"number of times a category must have been seen to retain [default=5]\",\\\n default=5)\npr.add_option(\"-f\",\"--feat_freq\",type=\"int\",help=\"feature frequency cutoff\\n\"+\\\n \"(how frequent must a feature be to retain it? [default=1])\",default=1)\n\n(opts,args) = pr.parse_args(sys.argv)\n\ninf = opts.input\nif not inf is sys.stdin:\n inf = open(inf,'r')\n\noutf = opts.output\nif not outf is sys.stdout:\n outf = open(outf,'w')\n\ntry:\n tag_cnt = {}\n ft_cnt = {}\n lines = {}\n \n ln_cnt = -1\n for l in inf:\n l = l.strip()\n ln_cnt += 1\n lines[ln_cnt] = l\n l = l.split()\n tag = l[0]\n tag_cnt[tag] = tag_cnt.get(tag,0) + 1\n feats = l[1:]\n if ':' in feats[0] and not(feats[-1]==':'):\n # real-valued features\n feats = map(lambda a: (a[0:a.rfind(':')],a[a.rfind(':')+1:]), l[1:])\n\n for (f,act) in feats:\n ft_cnt[f] = ft_cnt.get(f,0) + 1 #math.fabs(float(act))\n else:\n # boolean\n for f in feats:\n ft_cnt[f] = ft_cnt.get(f,0) + 1\n\n for i in range(ln_cnt + 1):\n l = lines.get(i)\n l = l.strip().split()\n tag = l[0]\n feats = l[1:]\n if tag_cnt.get(tag) >= opts.number:\n tag_printed = False\n if ':' in feats[0] and not(feats[-1]==':'):\n # real-valued features\n feats = map(lambda a: (a[0:a.rfind(':')],a[a.rfind(':')+1:]), l[1:])\n for (f,act) in feats:\n if ft_cnt.get(f) >= opts.feat_freq:\n if not tag_printed:\n print >> outf, tag,\n tag_printed = True\n print >> outf, f+':'+act,\n else:\n # boolean\n for f in feats:\n if not tag_printed:\n print >> outf, tag,\n tag_printed = True\n print >> outf, f,\n print >> outf, ''\n\n if i%100==0:\n outf.flush()\nfinally:\n outf.flush()\n if not inf is sys.stdin:\n inf.close()\n if not outf is sys.stdout:\n outf.close()\n" }, { "path": "ccgbank/bin/find-betas-no-gold.py", "content": "\"\"\"\nGiven:\n\n(1) a file supertagged words (OpenCCC file output format\nas produced by, e.g., WordAndPOSDictionaryLabellingStrategy),\n\n(2) a list (as a string) of tagging ambiguity levels (e.g.,\n\"1.4 1.6 1.8...\") that represent the desired tag/word levels\n(rounded off at the hundredths place to <=1.41, <=1.61, etc.),\n\n(3) possibly tagging dictionaries (if needed),\nand\n\n(4) the corresponding 'K' parameters (e.g., \"20 150\" as in Clark and\nCurran (2007)),\n\nproduce the list of betas that would produce those ambiguity\nlevels.\n\"\"\"\n\nimport optparse\nimport sys\nimport decimal\nimport math\n\n\"\"\"\nA little on-the-fly class creation for iterating through multi-stag format files.\n\"\"\"\nclass STIterator:\n def __init__(self, f):\n self.f = f\n def next(self):\n l = self.f.readline()\n while l.strip()=='' and l:\n l = self.f.readline()\n if l:\n lines = []\n # not at eof.\n if l.strip() != \"\":\n print >> sys.stderr, \"line=\", l.strip(), \"ill-formed st output file.\"\n raise Exception\n else:\n l = self.f.readline()\n while l.strip() != \"\" and l:\n lines.append(l.strip())\n l = self.f.readline()\n if lines == []:\n print >> sys.stderr, \"line=\", l.strip(), \"ill-formed st output file.\"\n raise Exception\n else:\n res = []\n for ln in lines:\n # each line is: word POS1 ... POSN ST1 ... STM\n # we just want the word, first pos and supertags.\n parts = ln.split()\n wd = parts[0]\n pos = parts[2]\n how_many_poss = int(parts[1])\n stgs = zip(parts[2+(how_many_poss*2)+1::2],map(lambda n: float(n), parts[2+(how_many_poss*2)+2::2]))\n res.append((wd,pos,stgs))\n return res\n else:\n raise StopIteration\n def __iter__(self): return self\n\n\np = optparse.OptionParser()\n\np.add_option(\"-i\", \"--inputf\", type=\"string\", help=\"input source [default=]\", default=sys.stdin)\np.add_option(\"-o\", \"--outputf\", type=\"string\", help=\"output destination [default=]\", default=sys.stdout)\np.add_option(\"-a\", \"--ambiguities\", type=\"string\", help=\"a space delimited string of tagging ambiguity levels [default=\\\"1.2 1.4 1.6 1.8 2.0 2.5 3.0 3.5\\\"]\",\n default=\"1.2 1.4 1.6 1.8 2.0 2.5 3.0 3.5\")\np.add_option(\"-K\", \"--Ks\", type=\"string\", help=\"a space delimited string of K values (only two) [default=\\\"20 150\\\", optional]\", \\\n default=\"20 150\")\np.add_option(\"-w\", \"--wordkeyeddict\", type=\"string\", help=\"word-keyed tagging dict [no default, optional]\",\\\n default=None)\np.add_option(\"-p\", \"--poskeyeddict\", type=\"string\", help=\"POS-keyed tagging dict [no default, optional]\",\\\n default=None)\n\n(ops,args) = p.parse_args()\n\n \ntry:\n # POS-keyed dict must be there if word-keyed one is.\n assert (not (not (ops.wordkeyeddict is None) and (ops.poskeyeddict is None)))\nexcept:\n print >> sys.stderr, \"need POS-keyed dict if using word-keyed dict.\"\n sys.exit(-1)\n\ninf = ops.inputf\nif not inf is sys.stdin:\n inf = open(inf, 'r')\n \noutf = ops.outputf\nif not outf is sys.stdout:\n outf = open(outf, 'w')\n\ninput_sents = [s for s in STIterator(inf)]\n\nwdict = {}\nif not ops.wordkeyeddict is None:\n entries = map(lambda l: l.split(), open(ops.wordkeyeddict, 'r').readlines())\n wdict[entries[0]] = (int(entries[1]), set(entries[2:]))\n\npdict = {}\nif not ops.poskeyeddict is None:\n entries = map(lambda l: l.split(), open(ops.poskeyeddict, 'r').readlines())\n posdict[entries[0]] = set(entries[1:])\n \ntry:\n ambs = map(lambda a: float(a), ops.ambiguities.split())\n betas = []\n current_beta = 1.0\n last_beta_above = None\n last_beta_below = 0.0\n total_tags = 0.0\n total_words = 0.0\n total_right = 0.0\n ks = map(lambda kay: int(kay), ops.Ks.split())\n for a in ambs:\n current_beta = 1.0\n last_beta_above = None\n last_beta_below = 0.0\n k = ks[0] if a!=ambs[-1] else ks[1]\n found = False\n while not found:\n total_tags = 0.0\n total_words = 0.0\n total_right = 0.0 \n for insent in input_sents:\n for lex in insent:\n total_words += 1\n w = lex[0]\n pos = lex[1]\n stags = lex[2]\n if len(wdict)>0:\n # filter with appropriate dictionary.\n (freq,tags) = wdict.get(w,(0,set([])))\n if freq >= k:\n tags = tags\n else:\n tags = pdict.get(pos,set([]))\n if len(tags)>0:\n stags = filter(lambda st: st[0] in tags, stags)\n best = stags[0][1]\n # how many tags are there that made the beta cut-off?\n total_tags += len(filter(lambda st: st[1] >= (current_beta * best), stags))\n # round to the nearest hundredth\n tags_per_word = (float(total_tags)/total_words)\n decimal.getcontext().prec = 4\n as_string = str(decimal.Decimal(str(tags_per_word)))\n # we're looking for 1.40..., or 1.60..., etc. (as the case may be)\n found = tags_per_word == a or ((as_string[:3]==str(a)[:3]) and (as_string[3]=='0'))\n\n if found:\n betas.append(current_beta)\n current_beta = 1.0\n last_beta = None\n else:\n # decide which direction to loosen the beta.\n if tags_per_word > a:\n # get more restrictive (i.e., larger beta).\n if last_beta_above is None:\n print >> sys.stderr, \"error\"\n sys.exit(-1)\n else:\n tempbeta = current_beta\n current_beta += math.fabs(last_beta_above - current_beta)/2.0\n last_beta_below = tempbeta\n\n else:\n # get less restrictive (i.e., smaller beta)\n tempbeta = current_beta\n current_beta -= math.fabs(current_beta - last_beta_below)/2.0\n last_beta_above = tempbeta\n\n print >> outf, \"betas\", ' '.join(map(lambda b: str(b), betas))\nexcept:\n print \"Unexpected error:\", sys.exc_info()[0]\n raise\nfinally:\n # clean up, clean up...\n if not inf is sys.stdin:\n inf.close()\n if not outf is sys.stdout:\n outf.close() \n" }, { "path": "ccgbank/bin/gen_parser_events_a", "content": "#!/bin/bash\nfor i in 02 03 04 05\ndo\n ccg-build -Dsect=$i -f build-ps.xml gen-parser-events &> logs/log.gen.parser.events.$i \ndone\n" }, { "path": "ccgbank/bin/gen_parser_events_b", "content": "#!/bin/bash\nfor i in 06 07 08 09\ndo\n ccg-build -Dsect=$i -f build-ps.xml gen-parser-events &> logs/log.gen.parser.events.$i \ndone\n" }, { "path": "ccgbank/bin/gen_parser_events_c", "content": "#!/bin/bash\nfor i in 10 11 12 13\ndo\n ccg-build -Dsect=$i -f build-ps.xml gen-parser-events &> logs/log.gen.parser.events.$i \ndone\n" }, { "path": "ccgbank/bin/gen_parser_events_d", "content": "#!/bin/bash\nfor i in 14 15 16 17\ndo\n ccg-build -Dsect=$i -f build-ps.xml gen-parser-events &> logs/log.gen.parser.events.$i \ndone\n" }, { "path": "ccgbank/bin/gen_parser_events_e", "content": "#!/bin/bash\nfor i in 18 19 20 21\ndo\n ccg-build -Dsect=$i -f build-ps.xml gen-parser-events &> logs/log.gen.parser.events.$i \ndone\n" }, { "path": "ccgbank/bin/gen_realizer_events_a", "content": "#!/bin/bash\nfor i in 02 03 04 05\ndo\n ccg-build -Dsect=$i -f build-rz.xml gen-realizer-events &> logs/log.gen.realizer.events.$i \ndone\n" }, { "path": "ccgbank/bin/gen_realizer_events_b", "content": "#!/bin/bash\nfor i in 06 07 08 09\ndo\n ccg-build -Dsect=$i -f build-rz.xml gen-realizer-events &> logs/log.gen.realizer.events.$i \ndone\n" }, { "path": "ccgbank/bin/gen_realizer_events_c", "content": "#!/bin/bash\nfor i in 10 11 12 13\ndo\n ccg-build -Dsect=$i -f build-rz.xml gen-realizer-events &> logs/log.gen.realizer.events.$i \ndone\n" }, { "path": "ccgbank/bin/gen_realizer_events_d", "content": "#!/bin/bash\nfor i in 14 15 16 17\ndo\n ccg-build -Dsect=$i -f build-rz.xml gen-realizer-events &> logs/log.gen.realizer.events.$i \ndone\n" }, { "path": "ccgbank/bin/gen_realizer_events_e", "content": "#!/bin/bash\nfor i in 18 19 20 21\ndo\n ccg-build -Dsect=$i -f build-rz.xml gen-realizer-events &> logs/log.gen.realizer.events.$i \ndone\n" }, { "path": "ccgbank/bin/get-text-from-mtc-style.py", "content": "\"\"\"\nGets the text from a MTC-style corpus.\nJust looks for ' ' segments.\n\n => \n\n(c) 2011 Dennis Nolan Mehay\n[Insert LGPL here]\n\"\"\"\nimport re, sys, codecs, os\n\npattern = re.compile(u\"\\\\s*(.*)\\\\s*\")\n\ninput = sys.stdin.read()\n\ntry:\n import chardet\n encoding = chardet.detect(input)['encoding']\nexcept:\n # this is what the original MTC corpus is encoded in.\n encoding = \"iso-8859-2\"\n\ninput = input.decode(encoding)\nstreamWriter = codecs.lookup(encoding)[-1]\nsys.stdout = streamWriter(sys.stdout)\n\nfor seg in pattern.findall(input):\n sys.stdout.write(seg.strip() + os.linesep)\n \n" }, { "path": "ccgbank/bin/get-truecase-list.py", "content": "\"\"\"\nRequires Python >= 2.6x+ but < 3.0.\n\nTakes in a stream (from stdin) or file of *tokenized* plain text (utf-8),\nreturns a list of words that occurred more than twice and were in upper-case\nmore frequently than not.\n\"\"\"\nimport optparse, sys, codecs, os\nfrom collections import defaultdict\n\ndef isAllUpper(st):\n return ( st.upper() == st and st.lower() != st )\n\nop = optparse.OptionParser()\n\nop.add_option(\"-i\", \"--input\", type=\"string\", help=\"input file or stream (default = )\",\n default=sys.stdin)\nop.add_option(\"-o\", \"--output\", type=\"string\", help=\"output file or stream (default = )\",\n default=sys.stdout)\nop.add_option(\"-f\", \"--use_first\", action=\"store_true\", help=\"whether to use the first word of each \"+\\\n \"sentence for counting uppercase vs. lowercase (default = False)\", default=False)\n\n(ops, args) = op.parse_args()\n\ninf = ops.input\nif not inf is sys.stdin:\n inf = open(inf, \"rb\")\n\nuse_first_word = ops.use_first\n\noutf = ops.output\nif not outf is sys.stdout:\n outf = codecs.open(outf, \"wb\", \"utf-8\")\nelse:\n # make stdout code utf-8\n streamWriter = codecs.lookup(\"UTF-8\")[-1]\n outf = streamWriter(outf)\n\n# map from: lowercased_word_key => specific_cased_form => count\nwds2ulcounts = defaultdict(lambda: defaultdict(lambda: 0))\n\ntry:\n l = inf.readline()\n\n while l:\n l = l.strip().decode(\"utf-8\")\n if l == u\"\":\n continue\n \n words = l.split()\n \n if not use_first_word:\n words = words[1:]\n \n for w in words:\n key = w.lower()\n wds2ulcounts[key][w] += 1\n \n l = inf.readline()\n \n for (wdkey,frms) in wds2ulcounts.items():\n wdforms = frms.items()\n # sum all counts. if more than 2, then write out the most frequent, else don't.\n sum_all = sum([cnt for (wf,cnt) in wdforms])\n # if there is only one form, seen more than once and it is a cased form, print it (this last will avoid printing punctuation and\n # always-lowercase words like 'the').\n if len(wdforms) == 1:\n most_freq = wdforms[0] \n if sum_all > 2 and most_freq[0].lower() != most_freq[0]:\n #outf.write(\"wd %s only has one form, seen %d times\" % (wdforms[0][0], wdforms[0][1]) + os.linesep)\n outf.write(most_freq[0] + os.linesep)\n else:\n continue\n else:\n if sum_all == 2:\n #outf.write(\"wd %s only occurred twice. cannot decide which is most frequent.\" % (wdkey) + os.linesep)\n continue\n else:\n wdforms.sort(lambda a,b: -cmp(a[1],b[1]))\n most_freq = wdforms[0]\n second_most_freq = wdforms[1]\n # see whether there is a tie. if so, no dice.\n if most_freq[1] == second_most_freq[1]:\n #outf.write(\"wd %s occurred more than twice, but there was a tie btw forms %s and %s (perhaps others).\" % \\\n # (wdkey, wdforms[0][0], wdforms[1][0]) + os.linesep)\n continue\n else:\n # only mention it if the most freq form is uppercased somewhere.\n if most_freq[0][0].lower() != most_freq[0][0]:\n #outf.write(\"wd %s occurred most with form %s.\" % (wdkey, wdforms[0][0]) + os.linesep)\n outf.write(most_freq[0] + os.linesep)\n \nfinally:\n try:\n outf.close()\n except:\n pass\n\n try:\n inf.close()\n except:\n pass\n \n" }, { "path": "ccgbank/bin/get-uniq-nbest.py", "content": "from BeautifulSoup import BeautifulStoneSoup as BSS\nimport codecs\nimport sys, os\nstreamWriter = codecs.lookup('utf-8')[-1]\nsys.stdout = streamWriter(sys.stdout)\n\ninf = open(sys.argv[1], \"rb\").read()\ntry:\n beta = float(sys.argv[2])\nexcept:\n beta = 0.1\n \nsoup = BSS(inf)\n\nsegs = soup.findAll(lambda t: t.name == u'seg')\n\ntot_paraphrases = 0.0\ntot_segs = 0.0\n\nfor seg in segs:\n tot_segs += 1\n if seg.get('complete') == 'true':\n best = seg.find(lambda p: p.name == 'best')\n ref = seg.ref.find(text=True)\n eye_dee = seg.get(u'id')\n paraphrases = set([p.find(text=True) for p in seg.findAll(lambda e: e.name in [u'best', u'next'])])\n tot_paraphrases += len(paraphrases)\n sys.stdout.write(ref + u' ||| ' + u' <-> '.join(paraphrases))\n sys.stdout.write(os.linesep)\n\nprint \"ave paraphrases/seg\", tot_paraphrases/tot_segs\n" }, { "path": "ccgbank/bin/get_factors_from_parse.py", "content": "#!/usr/bin/env python\n\n\"\"\"\n(c) 2008 Dennis N. Mehay\nUse this file any way you want, just please give the\nauthor credit if it makes it into any research in any \nmeaningful way. I make no claims whatsoever about the\nfitness or merchantability of this code. Use at\nyour own risk.\n\nTake a file of CCGbank-style parses and get the words, \nPOSs and lexical cat's from them.\n\nWe also insert the word as the 'lemma', just as a placeholder.\n\nSo we have the following output form (for each parse\nin the input file):\n\n||| ... |||\n\nPrint out parse IDs (if there) as they are.\n\"\"\"\nimport sys, re\nimport optparse\n\np = optparse.OptionParser()\np.add_option(\"-i\", \"--inputf\", type=\"string\", \\\n help=\"Input file to be postprocessed (one parse per line with IDs preceding them) [defaults to stdin]\",\\\n default=None)\np.add_option(\"-o\", \"--outputf\", type=\"string\", \\\n help=\"The output location [defaults to stdout]\",\\\n default=None)\n\n(opts, args) = p.parse_args(sys.argv)\n\ninf = None\noutf = None\n\nif opts.inputf is None:\n inf = sys.stdin\nelse:\n inf = open(opts.inputf,'r')\n\nif opts.outputf is None:\n outf = sys.stdout\nelse:\n outf = open(opts.outputf,'w')\n\ntry:\n global lexNodePattern\n lexNodePattern = re.compile(r'()+?')\n \n \"\"\"\n A procedure that returns a list of all lexical nodes in a CCGbank-style\n parse tree (in string representation).\n \"\"\"\n def getLexicalNodes(tree):\n matches = re.findall(lexNodePattern, tree)\n return matches\n \n for l in inf:\n if \"ID=\" in l:\n print >> outf, l.strip()\n elif l.strip()!='':\n nodes = getLexicalNodes(l.strip())\n ans = ''\n for n in nodes:\n parts = n.split()\n (w,pos,st) = (parts[4],parts[2],parts[1])\n ans += w+'|'+w+'|'+pos+'|'+st+ ' '\n print >> outf, ans.strip()\n \nfinally:\n if not opts.inputf is None:\n inf.close()\n if not opts.outputf is None:\n outf.close()\n\n \n" }, { "path": "ccgbank/bin/get_just_words_from_ner_text.py", "content": "\"\"\"\nTakes NE tagged text from stdin (assuming utf-8) and does just what it says: prints to stdout only the words.\n\"\"\"\nimport codecs, sys, os\nfrom optparse import OptionParser as OP\n\npr = OP()\npr.add_option(\"-o\",\"--output\",type=\"string\",help=\"output location [default=].\",\\\n default=sys.stdout)\n\n(opts,args) = pr.parse_args(sys.argv)\n\noutf = opts.output\nif not outf is sys.stdout:\n outf = open(outf,'w')\n\nstreamWriter = codecs.lookup(\"utf-8\")[-1]\noutw = streamWriter(outf)\n\nfor l in sys.stdin:\n l = l.decode(\"utf-8\")\n parts = l.split()\n snt = []\n for p in parts:\n if u\"_\" in p:\n subparts = p.split(u\"_\")\n if len(subparts) > 1:\n w = u\"_\".join(subparts[:-1])\n else:\n w = p\n snt.append(w)\n else:\n snt.append(p)\n outw.write(u\" \".join(snt) + '\\n')\n\noutf.flush()\nif not outf is sys.stdout:\n outf.close()\n" }, { "path": "ccgbank/bin/lowercase_tagged_text.py", "content": "#!/usr/bin/env python\n\"\"\"\nTakes a POS-tagged file and writes out the text with tokens lowercased except for\nproper nouns. A file with the list of word-tag pairs can also be written out.\n\n(c) 2010 Michael White\n[insert LGPL here]\n\"\"\"\n\nimport sys\nfrom optparse import OptionParser as OP\n\npr = OP()\npr.add_option(\"-i\",\"--input\",type=\"string\",help=\"input source [default=]\",\\\n default=sys.stdin)\npr.add_option(\"-o\",\"--output\",type=\"string\",help=\"output location [default=]\",\\\n default=sys.stdout)\npr.add_option(\"-p\",\"--pairs\",type=\"string\",help=\"output file for word-tag pairs\",\\\n default=None)\n\n(opts,args) = pr.parse_args(sys.argv)\n\ninf = opts.input\nif not inf is sys.stdin:\n inf = open(inf,'r')\n\noutf = opts.output\nif not outf is sys.stdout:\n outf = open(outf,'w')\n\npairsf = opts.pairs\nif not pairsf is None:\n pairsf = open(pairsf,'w')\n \ntry:\n sent = []\n pairs = []\n for l in inf:\n l = l.strip()\n l = l.split()\n if l[0] == '':\n sent = []\n pairs = []\n elif l[0] == '':\n print >> outf, ' '.join(sent)\n if not pairsf is None:\n for (token,tag) in pairs:\n print >> pairsf, token, tag\n else:\n token,tag = l[0],l[1]\n if tag[:3] != 'NNP' and (len(token) <= 1 or not token[1].isupper()):\n token = token.lower()\n sent.append(token)\n pairs.append((token,tag))\n if len(l) >= 4:\n tag2 = l[3]\n pairs.append((token,tag2))\nfinally:\n if not inf is sys.stdin:\n inf.close()\n if not outf is sys.stdout:\n outf.close()\n if not pairsf is None:\n pairsf.close()\n" }, { "path": "ccgbank/bin/merge-mtc-ids.py", "content": "\"\"\"\nThis program re-inserts the MTC unique IDs (sys+DOC+segment) into an auto-number-ID'ed parse of said\nMTC (or similar) document produced by OpenCCG's 'ccg-parse'.\n\nUsage: python merge-mtc-ids.py [output-of-OpenCCG-parser] [MTC-like-input-file] > [output-of-OpenCCG-parser-with-original-MTC-ids]\n\"\"\"\nimport sys, os, re, codecs\n\ntry:\n import chardet\nexcept:\n chardet = None\nfrom xml.etree.ElementTree import *\n\ndoc_pattern = re.compile(u\"<[Dd][Oo][Cc] docid=\\\"(.*)\\\" sysid=\\\"(.*)\\\">(.*)\")\nseg_pattern = re.compile(u\"(.*)$\")\n\nopenccg_in = sys.argv[1]\n#mtc_in = codecs.open(sys.argv[2], \"rb\", \"utf-8\").read()\nif not chardet is None:\n encoding = chardet.detect(open(sys.argv[2], \"rb\").read())['encoding']\nelse:\n encoding = \"ISO-8859-2\"\nmtc_in = codecs.open(sys.argv[2], \"rb\", encoding).readlines()\n\n# turn stdout into a UTF-8 converting writer.\nstreamWriter = codecs.lookup(encoding)[-1]\nsys.stdout = streamWriter(sys.stdout)\noutput = sys.stdout\n\n# map from auto-assigned ID to MTC ID.\nautoid2mtcid = {}\n\nmtc_ids = []\nfor l in mtc_in:\n l = l.strip()\n if l.startswith(\"\" + os.linesep + \"\" + os.linesep)\nfor event, elem in iterparse(openccg_in):\n if elem.tag.lower() == \"item\":\n next_mtc_id = mtc_ids.pop(0)\n elem.set(\"info\", u\",\".join(next_mtc_id[:-1]))\n output.write(u\"\\t\" + tostring(elem).strip() + os.linesep)\noutput.write(\"\" + os.linesep)\n" }, { "path": "ccgbank/bin/merge-stanford-morpha-with-pos.py", "content": "\"\"\"\nGiven two files: (1) the output of Stanford's re-implementation of 'morpha' and (2) the 'pairs' file of (),\nmerge them into a morph.xml file.\n\"\"\"\nimport sys, codecs, os\nfrom optparse import OptionParser as OP\nfrom xml.sax import saxutils\n\npr = OP()\npr.add_option(\"-m\",\"--morpha_input\",type=\"string\",help=\"morpha input file (required arg)\",\\\n default=None)\npr.add_option(\"-p\",\"--pairs_input\",type=\"string\",help=\"pairs input file (required arg)\",\\\n default=None)\npr.add_option(\"-o\",\"--output\",type=\"string\",help=\"output location [default=]\",\\\n default=sys.stdout)\n\n(opts,args) = pr.parse_args(sys.argv)\n\n# we do not check that you passed in the files (this is intended for internal use only, not as a\n# user-friendly app).\npinf = codecs.open(opts.pairs_input,'rb','utf-8')\nminf = codecs.open(opts.morpha_input,'rb','utf-8')\n\noutf = opts.output\nif not outf is sys.stdout:\n outf = codecs.open(outf,'wb','utf-8')\nelse:\n streamWriter = codecs.lookup(\"utf-8\")[-1]\n outf = streamWriter(sys.stdout)\n\npl = pinf.readline()\nml = minf.readline()\n\noutf.write('' + '\\n')\noutf.write('' + '\\n')\n\nentries = []\n\ntry:\n while pl and ml:\n pl = pl.strip()\n ml = ml.strip()\n pl = pl.split()\n ml = ml.split()\n # skip blank lines from line ending differences\n if len(pl) < 2: \n pl = pinf.readline()\n ml = minf.readline() \n continue\n s = [' 2:\n s.append(' class=\"')\n s.append(saxutils.escape(pl[2]))\n s.append('\"')\n if ml[0].lower() != pl[0].lower() and not (\"^\" in ml[0] or \"*****\" in ml[0]): # add stem only if distinct.\n s.append(' stem=\"')\n s.append(saxutils.escape(ml[0]))\n s.append('\"')\n s.append('/>')\n entries.append(s[:])\n pl = pinf.readline()\n ml = minf.readline() \n\n # sort/uniq\n entries.sort()\n last_one = None\n\n for e in entries:\n if last_one is None or e != last_one:\n outf.write(u''.join(e) + '\\n')\n last_one = e\n \n outf.write('' + '\\n')\nfinally:\n pinf.close()\n minf.close()\n if not outf is sys.stdout:\n outf.close()\n \n" }, { "path": "ccgbank/bin/merge_pos_ne.py", "content": "#!/usr/bin/env python\n\"\"\"\nTakes a POS-tagged file and a file of the original, NE-tagged text and writes out a file of:\n\n...\n[word][POS][SEM_CLASS](if any)[stem](if any)\n...\n\n(c) 2011 Dennis N. Mehay\n[insert LGPL here]\n\"\"\"\n\nclass POSOutputIter:\n def __init__(self, filelikeobj):\n self.f = filelikeobj\n\n def __iter__(self): return self\n \n def next(self):\n try:\n nxtLn = self.f.readline().strip()\n nxtSent = []\n if nxtLn != \"\":\n raise StopIteration\n nxtLn = self.f.readline().strip()\n while nxtLn != \"\":\n nxtSent.append(nxtLn)\n nxtLn = self.f.readline().strip()\n return nxtSent\n except:\n raise StopIteration\n \nimport sys, codecs, os\nfrom optparse import OptionParser as OP\n\npr = OP()\npr.add_option(\"-p\",\"--pos_in\",type=\"string\",help=\"POS-tagged input\",\\\n default=None)\npr.add_option(\"-n\",\"--ner_tagged_in\",type=\"string\",help=\"NE-tagged input (no POS tags yet)\",\\\n default=None)\npr.add_option(\"-o\",\"--output\",type=\"string\",help=\"output location [default=]\",\\\n default=sys.stdout)\n\n(opts,args) = pr.parse_args(sys.argv)\n\npinf = opts.pos_in\nif not pinf is sys.stdin:\n pinf = codecs.open(pinf,'r', 'utf-8')\n\nninf = opts.ner_tagged_in\nif not ninf is sys.stdin:\n ninf = codecs.open(ninf,'r', 'utf-8')\n \noutf = opts.output\nif not outf is sys.stdout:\n outf = codecs.open(outf,'wb', 'utf-8')\nelse:\n streamWriter = codecs.lookup(\"utf-8\")[-1]\n outf = streamWriter(sys.stdout)\n\ntry:\n for posSent in POSOutputIter(pinf):\n origSent = ninf.readline() \n for (posTW,NETagW) in zip(posSent, origSent.split()):\n NETagWParts = NETagW.split(u\"_\")\n\n if len(NETagWParts) > 1:\n NETag = u\"\\t\" + NETagWParts[-1]\n else:\n NETag = \"\"\n\n posTW = posTW.split()\n w = posTW[0]\n \n tgs = posTW[1:][::2][:2]\n for t in tgs:\n outf.write(w + u\"\\t\" + t + NETag + '\\n')\n \n\nfinally:\n pinf.close()\n ninf.close()\n outf.close()\n" }, { "path": "ccgbank/bin/my_unicode.py", "content": "\"\"\"\nUseful functions for dealing with Unicode messiness that arises from dealing with messy\ninput (e.g., gibberish from the Multiple Translation Chinese corpus).\n\"\"\"\nimport re, doctest\neval(r'u\"[\\u0080-\\uffff]+\"')\nRE_XML_ILLEGAL = u'([\\u0000-\\u0008\\u000b-\\u000c\\u000e-\\u001f\\ufffe-\\uffff])' + \\\n u'|' + \\\n u'([%s-%s][^%s-%s])|([^%s-%s][%s-%s])|([%s-%s]$)|(^[%s-%s])' % \\\n (unichr(0xd800),unichr(0xdbff),unichr(0xdc00),unichr(0xdfff),\n unichr(0xd800),unichr(0xdbff),unichr(0xdc00),unichr(0xdfff),\n unichr(0xd800),unichr(0xdbff),unichr(0xdc00),unichr(0xdfff))\ninvalid_char_re = re.compile(\"[^\\u0009\\u000a\\u000d\\u0020-\\uD7FF\\uE000-\\uFFFD]\")\n\ndef removeInvalidChars(text):\n \"\"\"\n Text is a unicode string. All characters that are not valid XML characters are removed.\n \"\"\"\n return re.sub(RE_XML_ILLEGAL, \"?\", text)\n \nif __name__==\"__main__\":\n doctest.testmod()\n" }, { "path": "ccgbank/bin/nbest-mtc-to-bleu-nist.py", "content": "\"\"\"\nThis program takes the n-best realizer output as one file (with sys+DOC+segment IDs -- 'info' attributes),\nthe tb.xml parser output of OpenCCG (for grabbing the strings of those things that did not parse),\nand creates an XML form suitable for use as a reference in the NIST-distributed BLEU script.\n\nUsage: python nbest-mtc-to-bleu-nist.py [nbest-from-realizer] [parser-output(tb.xml)] [max-n] | [NIST/BLEU-compatible-multiref-file]\n\"\"\"\nimport sys, os, re, codecs, tempfile, xml.sax.saxutils\ntry:\n import chardet\nexcept:\n chardet = None\nfrom xml.etree.ElementTree import *\nfrom collections import defaultdict\n\n# hack procedure. remove later.\ndef remove_ne(txt):\n # remove: Time, Location, Organization, Person, Money, Percent, Date\n txt = txt.replace(\" LOCATION\", \"\").replace(\" PERSON\", \"\").replace(\" MONEY\", \"\").replace(\" PERCENT\", \"\").replace(\" DATE\", \"\").replace(\" TIME\", \"\").replace(\" ORGANIZATION\", \"\")\n return txt\n\ndoc_pattern = re.compile(u\"<[Dd][Oo][Cc] docid=\\\"(.*)\\\" sysid=\\\"(.*)\\\">(.*)\")\nseg_pattern = re.compile(u\"(.*)$\")\n\nopenccg_all = open(sys.argv[1], \"rb\").read().replace(\"_&\", \"_&\").replace(\" & \", \"& \")\nparser_all = open(sys.argv[2], \"rb\").read().replace(\"_&\", \"_&\").replace(\" & \", \"& \")\n\nif not chardet is None:\n encoding1 = chardet.detect(openccg_all)['encoding']\nelse:\n encoding1 = \"utf-8\"\n\nif not chardet is None:\n encoding2 = chardet.detect(parser_all)['encoding']\nelse:\n encoding2 = \"utf-8\"\n \nopenccg_src = tempfile.NamedTemporaryFile()\nopenccg_src.write(openccg_all)\nopenccg_src.flush()\nopenccg_all = None\n\nparser_src = tempfile.NamedTemporaryFile()\nparser_src.write(parser_all)\nparser_src.flush()\nparser_all = None\n\ntry:\n n_size = int(sys.argv[3])\nexcept:\n n_size = 4\n \n# turn stdout into a UTF-8 converting writer.\nstreamWriter = codecs.lookup(\"utf-8\")[-1]\nsys.stdout = streamWriter(sys.stdout)\noutput = sys.stdout\n\n# list of list of (ID/ref pairs)\nrefs = []\n\n# max number of unique refs in any (there may not 'n_size' in any of them)\nmax_num_refs = 0\n\n# map from docID -> [(segID, [text])]\ndoc_to_segs = defaultdict(lambda: [])\n\n# get unparsed strings.\nfor event, elem in iterparse(parser_src.name):\n if elem.tag.lower() == \"item\" and elem.get(\"numOfParses\") == \"0\":\n txt = elem.get(\"string\").strip()\n (sys,doc,seg) = elem.get(\"info\").split(\",\")\n doc_to_segs[doc].append((seg, [txt]))\n\nfor event, elem in iterparse(openccg_src.name):\n if elem.tag.lower() == \"seg\":\n (sys,doc,seg) = elem.get(\"id\").split(\",\")\n is_complete = True if elem.get(\"complete\") else False\n nbest_realizations = []\n if not is_complete:\n # just get the original input. \n for child in list(elem):\n if child.tag.lower() == \"ref\":\n nbest_realizations.append(child.text.strip())\n else:\n # get the n-best (only keeping unique strings), so, e.g., 4-best might turn into\n # 1-best if they're all the same.\n how_many = 0\n for child in list(elem):\n if child.tag.lower() in [\"ref\", \"best\", \"next\"] and how_many < n_size:\n txt = child.text.strip()\n if not txt in nbest_realizations:\n nbest_realizations.append(txt)\n how_many += 1\n elif how_many >= n_size:\n break\n if len(nbest_realizations) > max_num_refs:\n max_num_refs = len(nbest_realizations)\n \n doc_to_segs[doc].append((seg, nbest_realizations))\n\n\noutput.write(\"\" + \\\n os.linesep + \"\" + os.linesep +\\\n \"\" + os.linesep)\n\ndocs = doc_to_segs.keys()\ndocs.sort()\n\nfor i in range(max_num_refs):\n output.write(\"\" % (i+1) + os.linesep)\n for doc in docs:\n output.write(\" \" % doc + os.linesep)\n segs = doc_to_segs[doc]\n segs.sort(lambda a,b: cmp(int(a[0]), int(b[0])))\n for (seg, paraphrases) in segs:\n this_one = i\n if i >= len(paraphrases):\n # there aren't as many paraphrases here as there are in the maximum length ref, so we just re-duplicate\n # the last one of this ref. \n this_one = len(paraphrases)-1\n output.write(\" %s \" % (int(seg), xml.sax.saxutils.escape(remove_ne(paraphrases[this_one].replace(\"_\",\" \"))) + os.linesep))\n output.write(\" \" + os.linesep)\n output.write(\"\" + os.linesep)\noutput.write(\"\")\n" }, { "path": "ccgbank/bin/ner/NERApp/src/nerapp/NERApp.java", "content": "package nerapp;\n\nimport java.io.*;\n//import edu.stanford.nlp.ie.crf.*;\nimport edu.stanford.nlp.ie.AbstractSequenceClassifier;\nimport edu.stanford.nlp.ie.NERClassifierCombiner;\nimport edu.stanford.nlp.io.IOUtils;\nimport edu.stanford.nlp.ling.CoreAnnotations;\nimport edu.stanford.nlp.ling.HasWord;\nimport edu.stanford.nlp.sequences.DocumentReaderAndWriter;\nimport edu.stanford.nlp.sequences.PlainTextDocumentReaderAndWriter;\nimport edu.stanford.nlp.util.CoreMap;\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.List;\n\n\n\n/** This tags text using the Stanford NE tagger API.\n *

\n * Usage: java -cp \"stanford-ner.jar:.\" NERApp [serializedClassifier] [fileName]\n *

\n * There are no default arguments.\n * (Created by modifying Jenny Finkel and Chris Manning's example \"NERDemo.java\".)\n *

\n * @author Dennis N. Mehay\n */\n\npublic class NERApp {\n\n @SuppressWarnings(\"unchecked\")\n public static void main(String[] args) throws IOException {\n DocumentReaderAndWriter readerAndWriter = new PlainTextDocumentReaderAndWriter();\n String usageStr = System.getProperty(\"line.separator\") +\n \"java -cp \\\"stanford-core-nlp.jar:.\\\" NERApp [inputFileName] [outputFileName] [classifierModelFile1] (...[classifierModelFile10])\"+\n \tSystem.getProperty(\"line.separator\") + System.getProperty(\"line.separator\") +\n \t\"(I.e., you can specify between one and ten classifiers whose predictions will be combined.\\n\"+\n\t \"Specify the best model first -- it will have precedence in the model combination.)\" +\n System.getProperty(\"line.separator\");\n\n if (args.length < 3) {\n System.out.println(usageStr);\n System.exit(-1);\n }\n\n String[] classifierMods = new String[10];\n\n for(int j = 2; j < args.length; j++) {\n\t classifierMods[j-2] = args[j].trim();\n }\n\n int numClassifiers = 0;\n AbstractSequenceClassifier classifier = null; //CRFClassifier.getClassifierNoExceptions(serializedClassifier);\n for(String classMod : classifierMods) { if(classMod != null) { numClassifiers++; } }\n switch (numClassifiers) {\n case 1:\n\t classifier = new NERClassifierCombiner(true, false, false, classifierMods[0]);\n\t break;\n case 2:\n\t classifier = new NERClassifierCombiner(true, false, false, classifierMods[0], classifierMods[1]);\n\t break;\n case 3:\n\t classifier = new NERClassifierCombiner(true, false, false, classifierMods[0], classifierMods[1], classifierMods[2]);\n\t break;\n case 4:\n\t classifier = new NERClassifierCombiner(true, false, false, classifierMods[0], classifierMods[1], classifierMods[2], classifierMods[3]);\n\t break;\n case 5:\n\t classifier = new NERClassifierCombiner(true, false, false, classifierMods[0], classifierMods[1], classifierMods[2], classifierMods[3], classifierMods[4]);\n\t break;\n case 6:\n\t classifier = new NERClassifierCombiner(true, false, false, classifierMods[0], classifierMods[1], classifierMods[2], classifierMods[3], classifierMods[4],\n\t\t\t\t\t\t classifierMods[5]);\n\t break;\n case 7:\n\t classifier = new NERClassifierCombiner(true, false, false, classifierMods[0], classifierMods[1], classifierMods[2], classifierMods[3], classifierMods[4],\n\t\t\t\t\t\t classifierMods[5], classifierMods[6]);\n\t break;\n case 8:\n\t classifier = new NERClassifierCombiner(true, false, false, classifierMods[0], classifierMods[1], classifierMods[2], classifierMods[3], classifierMods[4],\n\t\t\t\t\t\t classifierMods[5], classifierMods[6], classifierMods[7]);\n\t break;\n case 9:\n\t classifier = new NERClassifierCombiner(true, false, false, classifierMods[0], classifierMods[1], classifierMods[2], classifierMods[3], classifierMods[4],\n\t\t\t\t\t\t classifierMods[5], classifierMods[6], classifierMods[7], classifierMods[8]);\n\t break;\n case 10:\n\t classifier = new NERClassifierCombiner(true, false, false, classifierMods[0], classifierMods[1], classifierMods[2], classifierMods[3], classifierMods[4],\n\t\t\t\t\t\t classifierMods[5], classifierMods[6], classifierMods[7], classifierMods[8], classifierMods[9]);\n\t break;\n default:\n\t System.out.println(usageStr);\n\t System.exit(-1);\n }\n\n\n\n Iterable sents = IOUtils.readLines(args[0]);\n BufferedWriter outf = new BufferedWriter(new FileWriter(new File(args[1])));\n\n for (String sent : sents) {\n String[] parts = sent.split(\"\\\\s+\");\n List wdList = new ArrayList(parts.length);\n for(String w : parts) {\n wdList.add(new MyWord(w));\n }\n List tagging = classifier.classifySentence(wdList);\n String currNE = null;\n StringBuilder res = new StringBuilder();\n\n String wd, annot;\n int cursor = -1;\n for(CoreMap item : tagging) {\n cursor += 1;\n wd = item.get(CoreAnnotations.TextAnnotation.class);\n annot = item.get(CoreAnnotations.AnswerAnnotation.class);\n \n if(annot != null && annot.equals(\"O\") || annot.equals(\"MISC\")) {\n annot = null;\n }\n\n if(currNE != null && !currNE.equals(annot)) {\n res.append(\"\");\n currNE = null;\n }\n\n if(annot != null && !annot.equals(currNE)) {\n currNE = annot;\n res.append(\" \");\n res.append(\"<\"); res.append(currNE); res.append(\">\");\n res.append(wd);\n } else {\n res.append(\" \");\n res.append(wd);\n }\n\n }\n if(null != currNE) {\n res.append(\"\");\n }\n outf.write(res.toString() + System.getProperty(\"line.separator\"));\n outf.flush();\n }\n outf.close();\n }\n\n\n public static String classifyToString(List sentence, DocumentReaderAndWriter readerAndWriter, AbstractSequenceClassifier classif) {\n PlainTextDocumentReaderAndWriter.OutputStyle outFormat =\n PlainTextDocumentReaderAndWriter.OutputStyle.fromShortName(\"inlineXML\");\n\n DocumentReaderAndWriter tmp = readerAndWriter;\n readerAndWriter = new PlainTextDocumentReaderAndWriter();\n readerAndWriter.init(classif.flags);\n\n StringBuilder sb = new StringBuilder();\n sb.append(((PlainTextDocumentReaderAndWriter) readerAndWriter).getAnswers(sentence, outFormat, true));\n return sb.toString();\n }\n}\n\nclass MyWord implements HasWord {\n\n private String wd = null;\n\n public MyWord(String wd) {\n this.wd = wd;\n }\n\n public String word() {\n return wd;\n }\n\n public void setWord(String string) {\n this.wd = string;\n }\n\n}\n" }, { "path": "ccgbank/bin/ner/build-ner-api.properties", "content": "# For compiling against the Stanford API.\nstanford.core.nlp=../../stanford-nlp/stanford-core-nlp.jar" }, { "path": "ccgbank/bin/ner/build-ner-api.xml", "content": "\n\n\n\n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n\n \n\n" }, { "path": "ccgbank/bin/ner/ner-tag.sh", "content": "#!/bin/bash\n#\n# Tag a file $1 using Stanford NER located in base directory $2 (first arg, e.g., \"/home/me/stanford-ner-2.1.0\") \n# with model $3 (second arg, e.g., \"all.3class.distsim.crf.ser.gz\").\n#\n# Output is placed in the file whose path is given in argument $4.\n#\njava -mx700m -cp \"$1/stanford-ner.jar:`dirname $0`/NERApp.jar\" nerapp.NERApp $2/classifiers/$3 $1 2> /dev/null | python `dirname $0`/post-process-stanford-ner.py > $4\n" }, { "path": "ccgbank/bin/ner/ner_word.py", "content": "class NERWord:\n \"\"\"\n A simple wrapper for NER-labelled words.\n \"\"\"\n def __init__(self, wd, label=None, delim=\"/\"):\n self.wd = wd\n self.label = label\n self.delim = delim\n\n @staticmethod\n def parseLineOfWords(ln):\n \"\"\"\n Parses a line of words labelled with NE labels (e.g., \"John Smith entered the United States\").\n \"\"\"\n res = []\n for w in ln:\n if \"\"):\n parts = w.partition(\"\" in wd:\n parts = wd.partition(\">\")\n wd = parts[2]\n lb = lb[2:-1]\n elif w.startswith(\"<\") and \">\" in w:\n parts = w.partition(\">\")\n (wd,lb) = (parts[2], parts[0]+\">\")\n else:\n (wd,lb) = (w,None)\n res.append((wd,lb))\n\n # now distribute the labels to words between within the ... labels.\n final_res = []\n i = len(res) - 1\n while i >= 0:\n (wd,lb) = res[i]\n if lb is None:\n final_res.append(NERWord(wd,lb))\n elif not \"US Dept of Defense example .\" | python post-process-stanford-ner.py\nThis is a US_Dept_of_Defense_ORGANIZATION example .\n\nType: \n\n$ python post-process-stanford-ner.py -h \n\nfor help on the command-line options.\n\"\"\"\n\ndef fuseNERWords(list_of_ner_words):\n return \"_\".join([nerWrd.getWord() for nerWrd in list_of_ner_words] + [list_of_ner_words[0].getLabel()])\n\nop = optparse.OptionParser()\nop.add_option(\"--known_verbs\", type=\"string\", \\\n help=\"file containing known verbs (for split-at-verb-boundary heuristic) [defaults to an empty list of known verbs]\",\\\n default=None)\n\n(ops, args) = op.parse_args()\n\nknown_verbs = set([v.strip() for v in open(ops.known_verbs, \"rb\").readlines()]) if not ops.known_verbs is None else set()\npuncts = set([',', \"'\", '\"', \".\", \"?\", \"!\"])\nheuristic_splitters = (known_verbs | puncts)\n\nfor l in sys.stdin:\n l = l.strip()\n if l == \"\":\n continue\n ner_parts = l.split()\n ner_parts = NERWord.parseLineOfWords(ner_parts)\n\n i = 0\n\n current_NE = None\n current_group = None\n\n res = []\n while i < len(ner_parts):\n prt = ner_parts[i]\n (wd,ne) = (prt.getWord(), prt.getLabel())\n\n if not ne is None:\n # we have a NE label. is it a continuation of what came before (if anything)?\n if ne == current_NE:\n current_group.append(prt)\n else: \n if current_NE is None:\n current_NE = ne\n current_group = [prt]\n else:\n res.append(fuseNERWords(current_group))\n current_NE = ne\n current_group = [prt]\n else:\n if not current_NE is None:\n res.append(fuseNERWords(current_group))\n current_NE = None\n current_group = None\n res.append(wd)\n \n i += 1\n \n if not current_NE is None:\n res.append(fuseNERWords(current_group))\n\n print \" \".join(res)\n\n" }, { "path": "ccgbank/bin/normalize_text.py", "content": "\"\"\"\nAssuming UTF-8 input (defaults to stdin, otherwise, supply a file), normalize plain text in\nvarious ways -- e.g., normalize quotation marks.\n\nCopyright Dennis N. Mehay (2011)\n[Insert LGPL here]\n\"\"\"\nfrom optparse import OptionParser as OP\nimport codecs, sys, os\n\nop = OP()\nop.add_option(\"-i\", \"--input\", type=\"string\", help=\"input stream of text (file, or default=sys.stdin)\", default=sys.stdin)\nop.add_option(\"-o\", \"--output\", type=\"string\", help=\"output stream (file, or default=sys.stdout)\", default=sys.stdout)\n\n(ops,args) = op.parse_args()\n\n\nif not ( ops.input is sys.stdin or ops.input == \"-\" ):\n inf = codecs.open(ops.input, \"rb\", \"utf-8\")\nelse:\n streamReader = codecs.lookup(\"utf-8\")[2]\n inf = streamReader(sys.stdin)\n\nif not ( ops.output is sys.stdout or ops.output == \"-\" ):\n outf = codecs.open(ops.output, \"wb\", \"utf-8\")\nelse:\n streamWriter = codecs.lookup(\"utf-8\")[-1]\n outf = streamWriter(sys.stdout)\n\ntry:\n l = inf.readline().strip()\n while l:\n transformed_line = []\n opening_quotes = True\n for c in l:\n if c == u'\"' and opening_quotes:\n transformed_line.append(u\"``\")\n opening_quotes = False # next double quotes will be closing quotes.\n elif c == u'\"' and not opening_quotes:\n transformed_line.append(u\"''\")\n opening_quotes = True # reset the open-close tracker.\n else:\n transformed_line.append(c)\n\n outf.write(u\"\".join(transformed_line) + os.linesep)\n \n l = inf.readline().strip()\nfinally:\n inf.close()\n outf.close()\n\n\n" }, { "path": "ccgbank/bin/post-process-metricsmatr.py", "content": "\"\"\"\nPipe in a MetricsMATR-style mteval script output and, given (as sys.argv[1]) the name of the\nreference system, produce a file that has lines of the form:\n\n,,,,\n...\n,,,,\n\"\"\"\nimport sys\nrefid = sys.argv[1].strip()\n\nfor l in sys.stdin:\n # e.g., \"multiple_translation_set E09 XIN20020316.0014 1 0.0715856157727753\"\n (setid,sysid,docid,segid,score) = l.strip().split()\n print \",\".join([sysid, refid, docid, segid, score])\n \n" }, { "path": "ccgbank/bin/prepare-for-stanford-morpha.py", "content": "\"\"\"\nTake in a space-delimited file of ...... and turn it into \na form that the Stanford NE recognizer can accept (and from which we can extract\nall the information we need later).\n\n => \n\"\"\"\n\nimport sys, codecs, os\n\nstreamWriter = codecs.lookup(\"utf-8\")[-1]\nsys.stdout = streamWriter(sys.stdout)\n\nstreamReader = codecs.lookup(\"utf-8\")[2]\nsys.stdin = streamReader(sys.stdin)\n\nfor l in sys.stdin:\n l = l.strip()\n parts = l.split()\n if len(parts) > 2:\n # has NE label.\n wordform = u\"*****\".join([parts[0],parts[-1]]).replace(\"_\",\"^\")\n else:\n wordform = parts[0]\n pos = parts[1]\n\n joined = u\"_\".join([wordform, pos])\n sys.stdout.write(joined + '\\n')\n" }, { "path": "ccgbank/bin/reverse-spaces-to-newlines.py", "content": "#\n# reverses the conversion from spaces to newlines, and newlines to special chars,\n# from stdin to stdout\n#\nimport sys, re;\n[sys.stdout.write(re.sub('','\\n',re.sub('\\n',' ',line))) for line in sys.stdin]\n" }, { "path": "ccgbank/bin/run-all-bleu.sh", "content": "#!/bin/bash\n# run Bleu/NIST on all systems sys directory (passed in) with all references in the ref directory\n# (also passed in as an arg).\n# $1 is the mteval script.\n# $2 is the location of the system directory (where all the system files are stored)\n# $3 is the location of the reference directory (where all the reference files are stored).\n# $4 is the location of the source *file*.\n# $5 is the location where the scores will go.\ncurr_dir=`dirname $0`\nfor sys in `ls $2/E*`\ndo\n for ref in `ls $3/*`\n do\n\techo \"Command line: $1 --metricsMATR -t $sys -r $ref -s $4\"\n\tsys_shortname=`basename ${sys} | sed \"s/^\\(E[0-9][0-9]\\).*/\\1/g\"`\n\tref_shortname=`basename ${ref} | sed \"s/^\\(E[0-9][0-9]\\).*/\\1/g\"`\n\t$1 --metricsMATR -t $sys -r $ref -s $4\n\tcat BLEU-seg.scr | python $curr_dir/post-process-metricsmatr.py ${ref_shortname} > $5/BLEU.${sys_shortname}-${ref_shortname}.scr\n\tcat NIST-seg.scr | python $curr_dir/post-process-metricsmatr.py ${ref_shortname} > $5/NIST.${sys_shortname}-${ref_shortname}.scr\n\trm BLEU-*; rm NIST-*;\n done\ndone\n" }, { "path": "ccgbank/bin/stem_nns_vbx", "content": "#/bin/bash\n#\n# produces a file .dir/morph from .dir/pairs\n# that lists the words, pos tags and, for plural nouns and verbs, \n# also stems, derived using morpha\n#\n# write plural nouns and verbs to .dir/nns-vbx\ncat $1.dir/pairs | sort | uniq | grep -E \"NNS|VB.\" > $1.dir/nns-vbx\n# get rest\ncat $1.dir/pairs | sort | uniq | grep -E -v \"NNS|VB.\" | grep -E -v \"s>\" > $1.dir/non-nns-vbx\n# get stems\ncat $1.dir/nns-vbx | tr [:blank:] _ | morpha > $1.dir/nns-vbx.stems\n# merge stems\npaste $1.dir/nns-vbx $1.dir/nns-vbx.stems > $1.dir/nns-vbx-stems\n# merge files\ncat $1.dir/non-nns-vbx $1.dir/nns-vbx-stems | sort | uniq > $1.dir/morph\n# cleanup\nrm -f $1.dir/nns-vbx* $1.dir/non-nns-vbx\n" }, { "path": "ccgbank/bin/toUTF-8.py", "content": "\"\"\"\nCopryright (c) 2011 Dennis N. Mehay\n\nAssumes that 'chardet' is installed.\n\nRe-encodes most known Unicode encodings as UTF-8.\n\n(Provided that there is enough text for chardet to\ncorrectly determine the encoding of the input file.)\n\nIf any file exists by the output file name, it will\nbe overwritten.\n\n[Insert LGPL here]\n\"\"\"\n\nimport sys, codecs, optparse\n\ntry:\n import chardet\nexcept ImportError, ie:\n print >> sys.stderr, \"'chardet' must be installed for this script to work. Exiting...\"\n sys.exit(-1)\n\nop = optparse.OptionParser()\nop.add_option(\"-i\", \"--inputf\", type=\"string\", help=\"input file [required]\", default=None)\nop.add_option(\"-o\", \"--outputf\", type=\"string\", help=\"output file [required, will be overwritten]\", default=None)\n\n(ops,args) = op.parse_args()\n\ntry:\n assert(not (ops.inputf is None or ops.outputf is None))\nexcept AssertionError, ae:\n print >> sys.stderr, \"provide input and output files (type: 'python toUTF-8.py -h' for help)\"\n sys.exit(-1)\n\n# get input file's content and convert to utf-8\ninf = open(ops.inputf, \"rb\")\ninput = inf.read()\noutf = None\ntry:\n encoding = chardet.detect(input).get('encoding')\n input = input.decode(encoding)\n outf = codecs.open(ops.outputf, \"wb\", \"utf-8\")\n outf.write(input)\nexcept Exception, e:\n print >> sys.stderr, \"Something went wrong. Perhaps your input format is too obscure\"\nfinally:\n outf.close()\n inf.close()\n" }, { "path": "ccgbank/bin/write_morph.py", "content": "#!/usr/bin/env python\n\"\"\"\nTakes a file of word-tag pairs or word-tag-stem triples and writes an xml morph file.\n\n(c) 2010 Michael White (modifed by D.N. Mehay 2011)\n[insert LGPL here]\n\"\"\"\n\nimport sys\nfrom optparse import OptionParser as OP\nfrom xml.sax import saxutils\n\nglobal sem_classes\nsem_classes = set([\"PERSON\", \"ORGANIZATION\", \"LOCATION\", \"MONEY\", \"PERCENT\", \"TIME\", \"DATE\"])\n\npr = OP()\npr.add_option(\"-i\",\"--input\",type=\"string\",help=\"input source [default=]\",\\\n default=sys.stdin)\npr.add_option(\"-o\",\"--output\",type=\"string\",help=\"output location [default=]\",\\\n default=sys.stdout)\n\n(opts,args) = pr.parse_args(sys.argv)\n\ninf = opts.input\nif not inf is sys.stdin:\n inf = open(inf,'r')\n\noutf = opts.output\nif not outf is sys.stdout:\n outf = open(outf,'w')\n\ntry:\n print >> outf, ''\n for l in inf:\n l = l.strip()\n l = l.split()\n s = ['= 3:\n if l[2] in sem_classes:\n s.append(' class=\"')\n s.append(saxutils.escape(l[2]))\n s.append('\"')\n else:\n s.append(' stem=\"')\n s.append(saxutils.escape(l[2]))\n s.append('\"')\n if len(l) >= 4:\n s.append(' stem=\"')\n s.append(saxutils.escape(l[3]))\n s.append('\"')\n\nfinally:\n if not inf is sys.stdin:\n inf.close()\n if not outf is sys.stdout:\n outf.close()\n" }, { "path": "ccgbank/build-ht.properties", "content": "\n# nb: info.dir is specified in build.properties\npreds.train=${info.dir}/preds-train\npreds.dev=${info.dir}/preds-00-all\nht.factors.train=${info.dir}/ht.factors.train\nht.factors.dev=${info.dir}/ht.factors.dev\n\ngrammar.dir=${extract.dir}\nht.corpus.dir=${extract.dir}/test\n\nht.feats.dir=${feats.dir}/hypertagger\nht.models.dir=${models.dir}/hypertagger\npos.lm=${ht.models.dir}/pos.lm\nst.lm=${ht.models.dir}/st.lm\nvocab.pos=${ht.models.dir}/vocab.pos\nvocab.st=${ht.models.dir}/vocab.st\nvocab.train=${ht.models.dir}/vocab.train\nposprior.lm=${ht.models.dir}/p_w0.lm\npos.feats=${ht.feats.dir}/pos.feats\npos.dev.feats=${ht.feats.dir}/pos.feats.00\npos.mod=${ht.models.dir}/pos.mod\nhtprior.lm=${ht.models.dir}/t_p0w0.lm\npos.dict=${ht.models.dir}/pos.dict.min10\nword.dict=${ht.models.dir}/word.dict.min10\nht.feats=${ht.feats.dir}/ht.feats\nht2.feats=${ht.feats.dir}/ht2.feats\nht.dev.feats=${ht.feats.dir}/ht.dev.feats\nht.dev.log=${log.dir}/ht.dev.log\nht.mod=${ht.models.dir}/ht.mod\nht2.mod=${ht.models.dir}/ht2.mod\n\nargnames=Arg0:A0 Arg1:A1 Arg1a:A1a Arg1b:A1b Arg2:A2 Arg2a:A2a Arg2b:A2b Arg3:A3 Arg4:A4 Arg5:A5\n" }, { "path": "ccgbank/build-ht.xml", "content": "\n\n\n\n\n\n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n\n\n \n \n \n\t\n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n\t\n\t \n\t\n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n\n \n\n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "ccgbank/build-models.properties", "content": "\n# nb: info.dir is specified in build.properties\nfactors.train=${info.dir}/factors-train\nfactors.dev=${info.dir}/factors-00-all\ntext.train=${info.dir}/text-train\ntext.sc.train=${info.dir}/textsc-train\ntext.dev=${info.dir}/text-00-all\n\ncatfreq.cutoff=10\n\nfeats.dir=./feats\nmodels.dir=./models\nplugins.dir=./plugins\n\noriginal.postagger.models.dir=./original/models/postagger\n\nnovel.dir=${data.dir}/novel\nnovel.file=${novel.dir}/two-sents\n" }, { "path": "ccgbank/build-models.xml", "content": "\n\n\n\n\n\n \n \n \n \n\n\n \n \n \n \n \n \n\n\n \n \n \n\n \n \n \n\n\n \n \n \n\n \n Invoking supertagger all\n \n \n \n Done\n \n\n \n Invoking parser all\n \n \n \n Done\n \n\n \n Invoking hypertagger all\n \n \n \n Done\n \n\n \n Invoking realizer all\n \n \n \n Done\n \n\n\n \n \n \n \n \n\n \n\n\n" }, { "path": "ccgbank/build-original.properties", "content": "\n# nb: original.ccgbank.dir and data.dir are specified in build.properties\n\noriginal.dir=./original\n\ncorpus.dir=${original.dir}/corpus\nfeats.dir=${original.dir}/feats\nmodels.dir=${original.dir}/models\noriginal.log.dir=${original.dir}/logs\n\ntrain.leaves.srilm=${corpus.dir}/train.leaves.srilm\ndev.leaves.srilm=${corpus.dir}/dev.leaves.srilm\n\npostagger.feats.dir=${feats.dir}/postagger\npostagger.models.dir=${models.dir}/postagger\nsupertagger.feats.dir=${feats.dir}/supertagger\nsupertagger.models.dir=${models.dir}/supertagger\n\ntrain.pos.seq=${postagger.feats.dir}/train.pos.seq\npos.lm=${postagger.models.dir}/pos.lm\ntrain.st.seq=${supertagger.feats.dir}/train.st.seq\nst.lm=${supertagger.models.dir}/st.lm\nvocab.pos=${postagger.models.dir}/vocab.pos\nvocab.st=${supertagger.models.dir}/vocab.st\nvocab.train=${supertagger.models.dir}/vocab.train\nposprior.lm=${postagger.models.dir}/p_w0.lm\npos.feats=${postagger.feats.dir}/pos.feats\npos.mod=${postagger.models.dir}/pos.mod\nstprior.lm=${supertagger.models.dir}/t_p0w0.lm\npos.dict=${supertagger.models.dir}/pos.dict.min10\nword.dict=${supertagger.models.dir}/word.dict.min10\nst.feats=${supertagger.feats.dir}/st.feats\nst.mod=${supertagger.models.dir}/st.mod\n\nnovel.dir=${data.dir}/novel\nnovel.file=${novel.dir}/two-sents\n" }, { "path": "ccgbank/build-original.xml", "content": "\n\n\n\n\n\n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n\t\n\t \n\t\n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n\n \n \n \n\n \n \n \n \n\n\n" }, { "path": "ccgbank/build-ps.properties", "content": "# nb: feats.dir and models.dir are specified in build-models.properties\n\nparser.feats.dir=${feats.dir}/parser\nparser.models.dir=${models.dir}/parser\nderiv.factors.train=${parser.feats.dir}/deriv.factors.train\nvocab.parser.train=${parser.models.dir}/vocab.parser.train\nnbest.list.size=5\n\noriginal.postagger.models.dir=./original/models/postagger\n\n# what percentage of first 10 words being cased makes the heuristic say a sentence is in\n# title-case? (change if desired)\ntitlecase.threshold=0.5\n\n# you will need to download and extract ccgbank-data.tgz or english-models.tgz to get the truecase list\ntruecase.list=./aux/aux-files/truecase-list.gz\n\n# we assume you have the stanford core nlp jar file and NE tagging models\n# see docs/ccgbank-README for installation instructions\n# you can also modify the properties below to point to the jar and model files elsewhere\nstanford.core.nlp.dir=./stanford-nlp\nstanford.core.nlp.jar=${stanford.core.nlp.dir}/stanford-core-nlp.jar\n\n# by default, we assume you have placed the Stanford core NLP jar file in ./stanford-nlp, and that\n# there are the following models in a subdirectory there called 'classifiers'\n# (you can get Stanford NE recognition models from: http://nlp.stanford.edu/software/CRF-NER.shtml\n# or from the core NLP download, per docs/ccgbank-README)\nner.model1=${stanford.core.nlp.dir}/classifiers/english.all.3class.distsim.crf.ser.gz \nner.model2=${stanford.core.nlp.dir}/classifiers/english.muc.7class.distsim.crf.ser.gz\nner.model3=${stanford.core.nlp.dir}/classifiers/english.conll.4class.distsim.crf.ser.gz\n\n# if you're using an older version of the Stanford NLP tools, it may\n# be convenient to comment in the older names below\n#ner.model1=${stanford.core.nlp.dir}/classifiers/all.3class.distsim.crf.ser.gz \n#ner.model2=${stanford.core.nlp.dir}/classifiers/muc.distsim.crf.ser.gz\n#ner.model3=${stanford.core.nlp.dir}/classifiers/conll.distsim.crf.ser.gz\n\n" }, { "path": "ccgbank/build-ps.xml", "content": "\n\n\n\n\n\n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n\n\n \n \n \n\n \n \n \n\tExtracting derivation factors for section @{sect}\n\t\n\t \n\t \n\t \n\t\n \n \n Concatenating training derivation factors to ${deriv.factors.train}\n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n\t\n\t \n\t\n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n\t\n\t\n\t \n\t \n\t \n\t \n\t\n \n \n \n \n \n \n\n \n Copying train grammar with cutoffs to ${extract.dir}\n \n \n \n \n\n \n Copying dev morph to ${extract.dir}\n \n \n\n \n Loading parse.prefs\n \n \n \n Parsing dev section to ${log.dir}/parse.dev.log\n \n \n \n \n \n \n \n \n\n\n\n \n \n \n\n \n \n \n\n \n \n \n \n\tConcatenating training derivation factors excluding sect @{sect} to ${deriv.factors.train}.excl@{sect}\n\t\n\t \n\t\n\tCopying flm files to ${parser.models.dir}/excl@{sect}\n\t\n\t \n\t\n\t\n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t\n\t\n\t \n\t \n\t\n \n \n \n\n \n \n \n \n Generating parser training events for sect @{sect}\n Copying train grammar with to ${extract.dir}\n \n\t\n \n Loading gen-events.prefs\n \n\t\n \n Generating events to ${parser.feats.dir}/events-@{sect}.gz\n \n\t\n\t\n\t \n\t \n\t \n\t \n\t \n\t\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n \n\t\n \n \n \n\n \n \n \n\n \n Concatenating event files to ${parser.feats.dir}/events-train.gz\n \n \n\tConcatenating ${parser.feats.dir}/events-@{sect}.gz\n\t\n\t\n\t \n\t\n\t\n \n \n \n \n \n\n \n \n \n\n \n Calculating feature alphabet as ${parser.feats.dir}/alph.gz\n \n \n \n \n \n \n\n \n \n \n\n \n Training perceptron model to ${parser.models.dir}/model.gz\n \n \n \n \n \n \n \n \n \n\n \n Loading parse.prefs\n \n \n \n Parsing dev section to ${log.dir}/parse.perceptron.dev.log\n \n \n \n \n \n \n \n \n\n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n Copying training grammar with cutoffs and other miscellanea to ${novel.file}.dir/extract/\n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n Merging training morph (with cutoffs) and novel morph to ${novel.file}.dir/extract/morph.xml\n \n \n \n \n \n\n \n \n \n\n \n Loading parse.prefs\n \n \n \n Parsing ${novel.file}.dir/nertext-nolabs to ${novel.file}.dir/tb.xml\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n Loading parse.prefs\n \n \n \n Parsing ${novel.file}.dir/nertext-nolabs to ${novel.file}.dir/tb.xml\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n\n\n" }, { "path": "ccgbank/build-release.xml", "content": "\n\n\n\n\n\n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n\n \n \n \n \n\t\n \n \n\t\n\t\n\t\n\t\n\t \n\t\n \n \n \n\n \n \n \n \n\t\n \n \n\t\n\t\n\t\n\t\n\t \n\t\n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n\n\n \n \n \n\n \n \n \n \n \n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n\t\n\t\n\t\n\t\n\t \n\t\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n\n \n \n \n\n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n\n\n \n \n \n\n \n \n \n\n\n" }, { "path": "ccgbank/build-rz.properties", "content": "\n# nb: feats.dir and models.dir is specified in build-models.properties\nrealizer.feats.dir=${feats.dir}/realizer\nrealizer.models.dir=${models.dir}/realizer\nrealizer.model.global=model.global\nrealizer.alph.init=alph.init\nrealizer.alph.name=alph.gz\nrealizer.model.name=model.gz\n\nbig.words.lm=${realizer.models.dir}/gigaword4.5g.kenlm.bin\nwords.lm=${realizer.models.dir}/train.3bo\nwords.sc.lm=${realizer.models.dir}/train-sc.3bo\nstpos.flm=${realizer.models.dir}/stp3.flm\n\n# nb: vocab.train is copied from build-st.properties (which is otherwise not needed)\nsupertagger.models.dir=${models.dir}/supertagger\nvocab.train=${supertagger.models.dir}/vocab.train\n\n# todo: import this property instead\nhypertagger.models.dir=${models.dir}/hypertagger\n\n# nb: parser.models.dir is copied from build-ps.properties (which is otherwise not needed)\nparser.models.dir=${models.dir}/parser\n" }, { "path": "ccgbank/build-rz.xml", "content": "\n\n\n\n\n\n \n \n \n \n \n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n Copying train grammar to ${extract.dir}\n \n \n \n \n\n \n Copying dev morph to ${extract.dir}\n \n \n\n \n Copying test morph to ${extract.dir}\n \n \n\n \n Loading gen-events.prefs\n \n \n \n \n\n \n Loading rz-test.prefs\n \n \n \n \n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n\tConcatenating training text excluding sect @{sect} to ${text.train}.excl@{sect}\n\t\n\t \n\t\n\tConcatenating sem class replaced training text excluding sect @{sect} to ${text.sc.train}.excl@{sect}\n\t\n\t \n\t\n\tBuilding trigram language model to ${realizer.models.dir}/excl/train.excl@{sect}.3bo\n\t\n\t \n\t \n\t \n\t \n\t \n\t\n\tBuilding trigram language model to ${realizer.models.dir}/excl/train-sc.excl@{sect}.3bo\n\t\n\t \n\t \n\t \n\t \n\t \n\t\n\tDeleting concatenated text files\n\t\n\t \n\t\n\t\n\t \n\t\n \n \n \n\n\n \n \n \n\n \n Invoking parser target for generative models excluding each section\n \n \n \n Done\n \n\n \n \n \n \n Generating realizer training events for sect @{sect}\n Copying train grammar to ${extract.dir}\n \n\t\n \n Generating events to ${realizer.feats.dir}/events-@{sect}.gz\n \n\t\n\t\n\t\n\t \n\t \n\t \n\t \n\n\t \n\t\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n \n\t\n \n \n \n\n \n \n \n\n \n Concatenating event files to ${realizer.feats.dir}/events-train.gz\n \n \n\tConcatenating ${realizer.feats.dir}/events-@{sect}.gz\n\t\n\t\n\t \n\t\n\t\n \n \n \n \n \n\n \n \n \n\n \n Calculating feature alphabet as ${realizer.feats.dir}/alph.gz\n \n \n \n \n \n \n\n \n \n \n\n \n Training perceptron model with global feats to ${realizer.models.dir}/${realizer.model.global}\n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n Training perceptron model to ${realizer.models.dir}/${realizer.model.name} \n from ${realizer.feats.dir}/${realizer.alph.name}\n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n\n \n Realizing dev section to ${log.dir}/realize.dev.log\n \n \n \n \n \n \n \n \n\n \n Realizing dev section to ${log.dir}/realize.perceptron.dev.log\n \n \n \n \n \n \n \n \n \n \n \n\n \n Realizing test section to ${log.dir}/realize.perceptron.test.log\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n Realizing ${novel.file}.dir/tb.xml to ${novel.file}.dir/realize.nbest\n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n\n" }, { "path": "ccgbank/build-st.properties", "content": "\n# nb: feats.dir and models.dir are specified in build-models.properties\n\nsupertagger.feats.dir=${feats.dir}/supertagger\nsupertagger.models.dir=${models.dir}/supertagger\ntrain.pos.seq=${supertagger.feats.dir}/train.pos.seq\npos.lm=${supertagger.models.dir}/pos.lm\ntrain.st.seq=${supertagger.feats.dir}/train.st.seq\nst.lm=${supertagger.models.dir}/st.lm\nvocab.pos=${supertagger.models.dir}/vocab.pos\nvocab.st=${supertagger.models.dir}/vocab.st\nvocab.train=${supertagger.models.dir}/vocab.train\nposprior.lm=${supertagger.models.dir}/p_w0.lm\npos.feats=${supertagger.feats.dir}/pos.feats\npos.mod=${supertagger.models.dir}/pos.mod\npos.dict=${supertagger.models.dir}/pos.dict.min10\nword.dict=${supertagger.models.dir}/word.dict.min10\nst.feats=${supertagger.feats.dir}/st.feats\nst.mod=${supertagger.models.dir}/st.mod\n" }, { "path": "ccgbank/build-st.xml", "content": "\n\n\n\n\n\n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\t\n\t \n\t\n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\t\n\t\n \n \n \n\t\n\t\n\t \n\t \n\t \n\t\n \n \n \n\n \n \n \n \n \n \n \n \n\n\n \n \n \n\n \n \n\n\n" }, { "path": "ccgbank/build.properties", "content": "\ndata.dir=./data\ntemplates.dir=./templates\nconvert.dir=./convert\nextract.dir=./extract\ninfo.dir=${extract.dir}/info\ntest.dir=${extract.dir}/test\ngrams.dir=${extract.dir}/grammars\nlog.dir=./logs\ntmp.dir=./tmp\n\ntraining.dirs=02,03,04,05,06,07,08,09,10,11,12,13,14,15,16,17,18,19,20,21\n\n# nb: the \"propccgbank\" and aux files should be unpacked from \n# the ccgbank-data.tgz (see docs/ccgbank-README for details);\n# the original ccgbank dir property below can be changed to \n# point to your copy of the CCGBank, or you can just create a \n# symbolic link from ./ccgbank1.1 to your copy of it\n\nccgbank.dir=./propccgbank/pp_head_verbsAnnotated/AUTO\noriginal.ccgbank.dir=./ccgbank1.1/data/AUTO\n\n#sect=*\nsect=00\n#file=*\nfile=wsj_0001\n\naux.top.dir=./aux\naux.dir=${aux.top.dir}/aux-files\nwords=${data.dir}/wsj-nns-vb\nstems=${data.dir}/wsj-nns-vb-stems\n\n" }, { "path": "ccgbank/build.xml", "content": "\n\n\n\n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n\n \n \n\n \n \n \n\n \n \n\n \n \n\n \n \n \n \n\n \n \n\n \n \n\n \n \n\n \n \n \n\n \n \n \n \n\n \n \n\n \n \n \n \n\n \n\n \n \n \n \n\n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n\n\n \n \n \n\n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n\t\n \n \n \n \n\n\n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n\t\n\t \n\t\n \n \n\t\n \n \n \n \n\n \n \n \n \n\t\n\t \n\t\n \n \n\t\n \n \n \n \n\n \n \n \n \n \n \n\t\n\t \n\t\n \n \n\t\n \n \n \n \n\n \n \n \n \n\t\n\t \n\t\n \n \n\t\n\t\n\t\n \n \n \n\n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n\n\n \n \n \n\n \n \n Extracting grammar with no cutoffs and making LFs\n Section: ${sect} File: ${file}\n \n \n\t\n\t \n\t\n \n \n \n \n \n \n \n\n \n \n\n \n \n \n Concatenating @{type}-train\n \n\t\n \n \n \n\n \n \n \n \n\n \n \n \n \n \n\tStarting extract task for section @{sect}\n\t\n\t \n\t \n\t \n\t\n \n \n Concatenating training section files in ${info.dir}\n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\t\n\t \n\t \n\t \n\t \n\t \n\t \n\t\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\t\n\t \n\t \n\t \n\t \n\t \n\t \n\t\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\t\n\t \n\t\n \n \n \n \n \n \n \n \n \n\n \n \n Extracting grammar with no cutoffs and making LFs with debug derivs true\n Section: ${sect} File: ${file}\n \n \n\t\n\t \n\t\n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n\n \n\n\n\n\n" }, { "path": "ccgbank/data/README", "content": "\nThe files wsj-nns-vb and wsj-nns-vb-stems contain the plural nouns & verbs and\ntheir stems, respectively, from the WSJ part of the Penn Treebank, as computed\nby the morpha tool.\n\nSample commands for creating these files (with directories on the OSU\nLinguistics system) are in get_wsj_nns_vb and stem_wsj_nns_vb.\n\nThere are also some made-up sentences for testing the parser and\nrealizer in novel/two-sents, and an example CCGbank file with Propbank\nroles and corrections in the sample dir.\n" }, { "path": "ccgbank/data/get_wsj_nns_vb", "content": "cat /home/corpora/EN/penn_treebank_3/ims-cwb/wsj/wsj.cqp | egrep \"NNS|VB\" | sort | uniq > wsj-nns-vb\n" }, { "path": "ccgbank/data/novel/two-sents", "content": "Google announced today that it would offer free texting on its Google Voice app for the iPhone.\nThe press release was greeted rapturously.\n" }, { "path": "ccgbank/data/sample/AUTO/00/wsj_0001.auto", "content": "ID=wsj_0001.1 PARSER=GOLD NUMPARSE=1\n( ( ( ( ( ( ( () () ) ) () ) ( ( ( ( () () ) ) () ) ) ) () ) ( () ( ( ( () ( () () ) ) ( () ( () ( () () ) ) ) ) ( () () ) ) ) ) () ) \nID=wsj_0001.2 PARSER=GOLD NUMPARSE=1\n( ( ( ( () () ) ) ( () ( ( () ) ( () ( ( ( () () ) ) ( () ( () ( () ( () () ) ) ) ) ) ) ) ) ) () ) \n" }, { "path": "ccgbank/data/stem_wsj_nns_vb", "content": "cat wsj-nns-vb | tr [:blank:] _ | ./morpha.ix86_linux > wsj-nns-vb-stems\n" }, { "path": "ccgbank/data/wsj-nns-vb", "content": "12-inches\tNNS\n15\tVBN\n16-year-olds\tNNS\n1850s\tNNS\n1890s\tNNS\n18-year-olds\tNNS\n1900s\tNNS\n1920s\tNNS\n1930s\tNNS\n1940s\tNNS\n1950s\tNNS\n1960s\tNNS\n1970s\tNNS\n1980s\tNNS\n1990s\tNNS\n19-year-olds\tNNS\n204s\tNNS\n20s\tNNS\n20-stocks\tNNS\n3090s\tNNS\n'30s\tNNS\n30s\tNNS\n323s\tNNS\n'40s\tNNS\n'50s\tNNS\n'60s\tNNS\n'70s\tNNS\n747-400s\tNNS\n757-200s\tNNS\n75-cents-an-hour\tNNS\n'80s\tNNS\n80s\tNNS\n8300s\tNNS\n'90s\tNNS\nA330-300s\tNNS\nabacuses\tNNS\nabandoned\tVBD\nabandoned\tVBN\nabandoning\tVBG\nAbandoning\tVBG\nabandons\tVBZ\nabandon\tVB\nabandon\tVBP\nabated\tVBD\nabated\tVBN\nabates\tVBZ\nabate\tVB\nabating\tVBG\nabdicate\tVBP\nabetted\tVBN\nabetting\tVBG\nabide\tVB\nabilities\tNNS\nabolished\tVBD\nabolished\tVBN\nabolishing\tVBG\nabolish\tVB\naborted\tVBN\nabortion-rights\tNNS\nAbortion-rights\tNNS\nabortions\tNNS\nabounded\tVBD\nabounding\tVBG\nabounds\tVBZ\nabound\tVBP\nabrasives\tNNS\nabridging\tVBG\nabsences\tNNS\nabsent\tVB\nAbsent\tVB\nabsolve\tVBP\nabsolving\tVBG\nabsorbed\tVBD\nabsorbed\tVBN\nAbsorbed\tVBN\nabsorbers\tNNS\nabsorbing\tVBG\nabsorbs\tVBZ\nabsorb\tVB\nabsorb\tVBP\nabstained\tVBD\nabstentions\tNNS\nabstracts\tNNS\nabused\tVBD\nabused\tVBN\nabuses\tNNS\nabuse\tVB\nabusing\tVBG\nacademics\tNNS\nacceded\tVBD\naccede\tVB\naccelerated\tVBD\naccelerated\tVBN\naccelerates\tVBZ\naccelerate\tVB\naccelerate\tVBP\naccelerating\tVBG\naccents\tNNS\nacceptances\tNNS\nACCEPTANCES\tNNS\naccepted\tVBD\naccepted\tVBN\naccepting\tVBG\naccepts\tVBZ\naccept\tVB\naccept\tVBP\naccessories\tNNS\nAccessories\tNNS\naccess\tVB\naccidents\tNNS\nacclaim\tVB\naccolades\tNNS\naccommodated\tVBN\naccommodate\tVB\naccommodate\tVBP\naccommodating\tVBG\naccommodations\tNNS\naccomodate\tVB\naccompanied\tVBD\naccompanied\tVBN\naccompanies\tVBZ\naccompanying\tVBG\naccompany\tVB\naccomplished\tVBN\naccomplishes\tVBZ\nAccomplishing\tVBG\naccomplishments\tNNS\naccomplish\tVB\naccorded\tVBD\naccorded\tVBN\naccording\tVBG\nAccording\tVBG\naccords\tNNS\naccord\tVB\naccountants\tNNS\naccounted\tVBD\naccounted\tVBN\naccounting\tVBG\naccounts\tNNS\nAccounts\tNNS\naccounts\tVBZ\naccount\tVB\naccount\tVBP\naccreted\tVBN\naccruals\tNNS\naccrued\tVBN\nAccrued\tVBN\naccrues\tVBZ\naccrue\tVB\naccruing\tVBG\naccumulated\tVBN\naccumulate\tVBP\naccumulating\tVBG\naccusations\tNNS\naccused\tVBD\naccused\tVBN\naccusers\tNNS\naccuses\tVBZ\naccuse\tVB\naccuse\tVBP\naccusing\tVBG\naccustomed\tVBN\naces\tVBZ\nachieved\tVBD\nachieved\tVBN\nachievements\tNNS\nachieves\tVBZ\nachieve\tVB\nachieve\tVBP\nachieving\tVBG\naching\tVBG\nacidified\tVBN\nacids\tNNS\nacknowledged\tVBD\nacknowledged\tVBN\nacknowledges\tVBZ\nacknowledge\tVB\nacknowledge\tVBP\nacknowledging\tVBG\nacorns\tNNS\nacquainted\tVBN\nacquiesced\tVBD\nacquiesce\tVB\nacquired\tVBD\nacquired\tVBN\nAcquired\tVBN\nacquirers\tNNS\nAcquirers\tNNS\nacquires\tVBZ\nacquire\tVB\nacquiring\tVBG\nacquisitions\tNNS\nAcquisitions\tNNS\nacquitted\tVBN\nacquit\tVB\nacres\tNNS\nacross-the-board-cuts\tNNS\nacted\tVBD\nacted\tVBN\nacting\tVBG\nActing\tVBG\nactions\tNNS\nactivated\tVBN\nactivate\tVBP\nactives\tNNS\nactivists\tNNS\nActivists\tNNS\nactivities\tNNS\nactors\tNNS\nacts\tNNS\nacts\tVBZ\nactuaries\tNNS\nact\tVB\nact\tVBP\nadapted\tVBD\nadapted\tVBN\nAdapted\tVBN\nadapting\tVBG\nadapt\tVB\nadapt\tVBP\nadded\tVBD\nAdded\tVBD\nadded\tVBN\nAdded\tVBN\naddicted\tVBN\naddicts\tNNS\nadding\tVBG\nAdding\tVBG\nadditions\tNNS\nadditives\tNNS\naddressed\tVBD\naddressed\tVBN\naddresses\tNNS\naddresses\tVBZ\naddressing\tVBG\nAddressing\tVBG\naddress\tVB\naddress\tVBP\nadds\tVBZ\nAdds\tVBZ\nadd\tVB\nAdd\tVB\nadd\tVBP\nadepts\tNNS\nadhered\tVBD\nadhere\tVB\nadhesives\tNNS\nadjourned\tVBN\nadjudicators\tNNS\nadjusted\tVBD\nadjusted\tVBN\nadjusters\tNNS\nAdjusters\tNNS\nadjusting\tVBG\nadjustments\tNNS\nadjusts\tVBZ\nadjust\tVB\nAdjust\tVB\nadjust\tVBP\nadmen\tNNS\nadministered\tVBN\nadministers\tVBZ\nadminister\tVB\nadministrations\tNNS\nadministrators\tNNS\nadmired\tVBD\nadmired\tVBN\nAdmirers\tNNS\nadmires\tVBZ\nadmissions\tNNS\nadmits\tNNS\nadmits\tVBZ\nadmitted\tVBD\nADMITTED\tVBD\nadmitted\tVBN\nadmitting\tVBG\nadmit\tVB\nadmit\tVBP\nadmonishing\tVBG\nadolescents\tNNS\nadopted\tVBD\nADOPTED\tVBD\nadopted\tVBN\nadopting\tVBG\nAdopting\tVBG\nadopts\tVBZ\nadopt\tVB\nadopt\tVBP\nadorned\tVBD\nadorned\tVBN\nadorn\tVB\nADRs\tNNS\nads\tNNS\nAds\tNNS\nADS\tNNS\nadults\tNNS\nAdults\tNNS\nadvanced-ceramics\tNNS\nadvanced\tVBD\nadvanced\tVBN\nadvancements\tNNS\nadvancers\tNNS\nAdvancers\tNNS\nadvances\tNNS\nAdvances\tNNS\nadvance\tVB\nadvance\tVBP\nadvancing\tVBG\nAdvancing\tVBG\nadvantages\tNNS\nadvantage\tVB\nadventures\tNNS\nadversaries\tNNS\nadvertised\tVBD\nadvertised\tVBN\nadvertisements\tNNS\nadvertisers\tNNS\nAdvertisers\tNNS\nadvertises\tVBZ\nadvertise\tVB\nadvertise\tVBP\nadvertising\tVBG\nAdvertising\tVBG\nadvertising\tVBG|NN\nadvised\tVBD\nadvised\tVBN\nadvisers\tNNS\nadvises\tVBZ\nAdvises\tVBZ\nadvise\tVB\nadvise\tVBP\nadvising\tVBG\nadvisories\tNNS\nadvocated\tVBD\nadvocated\tVBN\nadvocates\tNNS\nAdvocates\tNNS\nadvocates\tVBZ\nadvocate\tVB\nadvocate\tVBP\nadvocating\tVBG\naerobics\tNNS\naffairs\tNNS\naffected\tJJ|VBN\naffected\tVBD\naffected\tVBN\naffecting\tVBG\naffections\tNNS\naffects\tVBZ\naffect\tVB\naffect\tVBP\naffidavits\tNNS\naffiliated\tVBN\nAffiliated\tVBN\naffiliates\tNNS\naffiliating\tVBG\naffinities\tNNS\naffirmed\tVBD\naffirming\tVBG\nafflicted\tVBN\nafflicts\tVBZ\nafflict\tVB\nafforded\tVBN\nafford\tVB\nafford\tVBP\naftereffects\tNNS\nAftereffects\tNNS\nafter-hours\tNNS\nafternoons\tNNS\naftershocks\tNNS\nAftershocks\tNNS\nAFTERSHOCKS\tNNS\naged\tVBN\nagencies\tNNS\nagendas\tNNS\nagents\tNNS\nAgents\tNNS\nages\tNNS\nage\tVBP\naggravated\tVBD\naggravated\tVBN\naggravates\tVBZ\naggravate\tVB\naggravate\tVBP\naggravating\tVBG\naggregates\tNNS\naging\tVBG\nagitated\tVBN\nagonize\tVB\nagreed\tVBD\nAgreed\tVBD\nagreed\tVBN\nagreeing\tVBG\nagreements\tNNS\nagrees\tVBZ\nAGREES\tVBZ\nagree\tVB\nagree\tVBP\nagriproducts\tNNS\nAichi\tNNS\naided\tVBD\naided\tVBN\nAided\tVBN\naides\tNNS\nAides\tNNS\naiding\tVBG\naids\tNNS\naids\tVBZ\naid\tVB\nailing\tVBG\nailments\tNNS\nails\tNNS\naimed\tVBD\naimed\tVBN\nAimed\tVBN\naiming\tVBG\naims\tNNS\naims\tVBZ\nAims\tVBZ\naim\tVB\nAim\tVB\naim\tVBP\naircraft\tNNS\naired\tVBD\naired\tVBN\nairfields\tNNS\nairing\tVBG\nairlifted\tVBN\nairlifting\tVBG\nairliners\tNNS\nairlines\tNNS\nAirlines\tNNS\nairmen\tNNS\nairplanes\tNNS\nairports\tNNS\nairs\tNNS\nairs\tVBZ\nair\tVB\nair\tVBP\nairwaves\tNNS\naisles\tNNS\nai\tVBP\nai\tVBZ\nalarmed\tVBN\nAlarmed\tVBN\nalarming\tVBG\nalarms\tNNS\nalarms\tVBZ\nAlbanians\tNNS\nalbums\tNNS\nalchemists\tNNS\nalcoholics\tNNS\nalerted\tVBD\nalerting\tVBG\nalerts\tVBZ\nalert\tVB\nalienated\tVBN\nalienates\tVBZ\nalienate\tVB\nalienating\tVBG\naliens\tNNS\naligned\tVBN\nalign\tVBP\nallayed\tVBN\nallay\tVB\nallegations\tNNS\nalleged\tVBD\nalleged\tVBN\nalleges\tNNS\nalleges\tVBZ\nallege\tVB\nallege\tVBP\nalleging\tVBG\nallergies\tNNS\nalleviate\tVB\nalleviating\tVBG\nalleys\tNNS\nalliances\tNNS\nallied\tVBD\nallies\tNNS\nAllies\tNNS\nalligators\tNNS\nallocated\tVBD\nallocated\tVBN\nallocate\tVB\nallocate\tVBP\nallocating\tVBG\nallocations\tNNS\nallotments\tNNS\nallowances\tNNS\nallowed\tVB\nallowed\tVBD\nallowed\tVBN\nallowing\tVBG\nAllowing\tVBG\nallows\tVBZ\nallow\tVB\nAllow\tVB\nallow\tVBP\nalloys\tNNS\nallrightniks\tNNS\nalluded\tVBD\nallusions\tNNS\nallying\tVBG\nally\tVB\naltered\tVBD\naltered\tVBN\naltering\tVBG\nalternates\tNNS\nalternates\tVBZ\nalternating\tVBG\nalternatives\tNNS\nalter\tVB\nAlter\tVB\nalter\tVBP\naluminum-makers\tNNS\nalumni\tNNS\namahs\tNNS\namalgamate\tVB\namalgamations\tNNS\namassed\tVBD\namassed\tVBN\namasses\tVBZ\namass\tVB\namass\tVBP\namateurs\tNNS\namazed\tVBN\namaze\tVB\nambassadors\tNNS\nambiguities\tNNS\nambitions\tNNS\namble\tVB\nambushed\tVBD\namended\tVBD\namended\tVBN\namending\tVBG\nAmending\tVBG\namendments\tNNS\namend\tVB\namenities\tNNS\nAmenities\tNNS\nA-men\tNNS\nAmericana\tNNS\nAmericanized\tVBD\nAmericans\tNNS\nAmericans\tVBP\namortize\tVB\namounted\tVBD\namounted\tVBN\namounting\tVBG\namounts\tNNS\namounts\tVBZ\namount\tVB\namount\tVBP\namphobiles\tNNS\namplified\tVBN\namplifiers\tNNS\namplifying\tVBG\namplify\tVB\namps\tNNS\namused\tVBN\namusements\tNNS\namuse\tVBP\nam\tVBP\nAm\tVBP\nAM\tVBP\nanachronisms\tNNS\nanalyses\tNNS\nAnalyses\tNNS\nanalysts\tNNS\nAnalysts\tNNS\nanalyzed\tVBD\nanalyzed\tVBN\nanalyze\tVB\nanalyze\tVBP\nanalyzing\tVBG\nanchored\tVBN\nanchors\tNNS\nanchor\tVB\nanchor\tVBP\nand\tVBP\nanemias\tNNS\nanemics\tNNS\nangels\tNNS\nangered\tVBD\nangered\tVBN\nangering\tVBG\nangles\tNNS\nanimals\tNNS\nAnimals\tNNS\nanimated\tVBN\nanimosities\tNNS\nankles\tNNS\nAnne\tVB\nanniversaries\tNNS\nannounced\tVBD\nANNOUNCED\tVBD\nannounced\tVBN\nannouncements\tNNS\nannounces\tVBZ\nannounce\tVB\nannounce\tVBP\nannouncing\tVBG\nannoyed\tVBD\nannoyed\tVBN\nannualized\tVBN\nAnnualized\tVBN\nannuities\tNNS\nAnnuities\tNNS\nANNUITIES\tNNS\nAnointing\tVBG\nanomalies\tNNS\nanswered\tVBD\nanswered\tVBN\nanswering\tVBG\nanswers\tNNS\nAnswers\tNNS\nanswers\tVBZ\nanswer\tVB\nanswer\tVBP\nantagonists\tNNS\nantagonize\tVB\nanteaters\tNNS\nantecedents\tNNS\nantelope\tNNS\nanthers\tNNS\nanti-abortionists\tNNS\nantibodies\tNNS\nanticipated\tVBD\nanticipated\tVBN\nanticipates\tVBZ\nanticipate\tVB\nanticipate\tVBP\nanticipating\tVBG\nAnticipating\tVBG\nantics\tNNS\nanti-heroes\tNNS\nanti-infectives\tNNS\nantipathies\tNNS\nanti-programmers\tNNS\nantiquities\tNNS\nants\tNNS\nanxieties\tNNS\napartments\tNNS\napologies\tNNS\napologists\tNNS\napologized\tVBD\napologizes\tVBZ\napologize\tVB\napologizing\tVBG\nappalled\tVBN\napparitions\tNNS\nappealed\tVBD\nappealed\tVBN\nappealing\tVBG\nAppealing\tVBG\nappeals\tNNS\nappeal\tVB\nappeal\tVBP\nappearances\tNNS\nappeared\tVBD\nappeared\tVBN\nappearing\tVBG\nappears\tNNS\nappears\tVBZ\nAPPEARS\tVBZ\nappear\tVB\nappear\tVBP\nappeased\tVBD\nappease\tVB\nappendages\tNNS\nappended\tVBN\nappend\tVB\napplauded\tVBD\napplauding\tVBG\napplauds\tVBZ\napplaud\tVB\napplaud\tVBP\napples\tNNS\nappliances\tNNS\napplicants\tNNS\napplications\tNNS\nApplications\tNNS\napplied\tVBD\napplied\tVBN\nApplied\tVBN\napplies\tVBZ\napplying\tVBG\nApplying\tVBG\napply\tVB\napply\tVBP\nappointed\tVBD\nappointed\tVBN\nAppointed\tVBN\nappointees\tNNS\nappointments\tNNS\nappoint\tVB\nappraisals\tNNS\nappraised\tVBN\nappraisers\tNNS\nappraise\tVB\nappreciated\tVBD\nappreciated\tVBN\nappreciates\tVBZ\nappreciate\tVB\nappreciate\tVBP\nappreciating\tVBG\napprehensions\tNNS\napprised\tVBN\napproached\tVBD\napproached\tVBN\napproaches\tNNS\napproaches\tVBZ\napproaching\tVBG\napproach\tVB\napproach\tVBP\nappropriated\tVBD\nappropriated\tVBN\nappropriations\tNNS\nAppropriations\tNNS\nappropriators\tNNS\napprovals\tNNS\napproved\tVBD\nAPPROVED\tVBD\napproved\tVBN\napproves\tVBZ\napprove\tVB\napprove\tVBP\napproving\tVBG\napproximates\tVBZ\nArabs\tNNS\narbitragers\tNNS\nArbitragers\tNNS\narbitrageurs\tNNS\narbitraging\tVBG\nArbitraging\tVBG\narbitrates\tVBZ\narbitrating\tVBG\narborists\tNNS\narbs\tNNS\nArbs\tNNS\narcades\tNNS\narched\tVBD\narchitects\tNNS\nArchitects\tNNS\narch\tVBP\nareas\tNNS\nAreas\tNNS\narenas\tNNS\nare\tVBP\nAre\tVBP\nARE\tVBP\nargued\tVBD\nargued\tVBN\nargues\tVBZ\nargue\tVB\nargue\tVBP\narguing\tVBG\nArguing\tVBG\narguments\tNNS\narisen\tVBN\narises\tVBZ\narise\tVB\narise\tVBP\nARISE\tVBP\narising\tVBG\nArkansas\tNNS\narmadillos\tNNS\narmed\tVBN\nArmed\tVBN\narmies\tNNS\narmpits\tNNS\narms\tNNS\nArms\tNNS\narm\tVB\naromas\tNNS\narose\tVBD\naroused\tVBD\naroused\tVBN\narouses\tVBZ\narouse\tVB\narousing\tVBG\narpeggios\tNNS\nArraignments\tNNS\narranged\tVBD\narranged\tVBN\narrangements\tNNS\nArrangements\tNNS\narranges\tVBZ\narrange\tVB\narranging\tVBG\narrears\tNNS\narrested\tVBD\narrested\tVBN\nArrested\tVBN\narrests\tNNS\narrest\tVB\narrivals\tNNS\narrived\tVBD\nARRIVED\tVBD\narrived\tVBN\narrives\tVBZ\narrive\tVB\narrive\tVBP\narriving\tVBG\narrows\tNNS\narsenals\tNNS\narteries\tNNS\narticles\tNNS\narticulate\tVB\nartifacts\tNNS\nartillerists\tNNS\nartists\tNNS\nArtists\tNNS\narts\tNNS\nArts\tNNS\nartworks\tNNS\nAsahi\tNNS\nascending\tVBG\nascertain\tVB\nascribed\tVBN\nascribe\tVBP\nashes\tNNS\nAshland\tVBP\nashtrays\tNNS\nasked\tVBD\nAsked\tVBD\nasked\tVBN\nAsked\tVBN\nasking\tVBG\nasks\tVBZ\nask\tVB\nAsk\tVB\nask\tVBP\naspects\tNNS\naspens\tNNS\naspersions\tNNS\naspirations\tNNS\naspired\tVBD\naspires\tVBZ\naspire\tVBP\naspiring\tVBG\nassailed\tVBD\nassailed\tVBN\nassassinated\tVBD\nassassinated\tVBN\nassassinate\tVB\nassassinating\tVBG\nassassinations\tNNS\nassaults\tNNS\nassault\tVBP\nassemblages\tNNS\nassembled\tVBD\nassembled\tVBN\nassemble\tVB\nassemble\tVBP\nassemblies\tNNS\nassembling\tVBG\nasserted\tVBD\nasserted\tVBN\nasserting\tVBG\nassertions\tNNS\nasserts\tVBZ\nassert\tVB\nassert\tVBP\nasses\tNNS\nassessed\tVBD\nassessed\tVBN\nassessing\tVBG\nassessments\tNNS\nassess\tVB\nassess\tVBP\nassets\tNNS\nAssets\tNNS\nASSETS\tNNS\nassigned\tVBN\nassignments\tNNS\nassigns\tVBZ\nassign\tVB\nassign\tVBP\nassimilate\tVB\nassistants\tNNS\nassisted\tVBN\nassisting\tVBG\nassists\tVBZ\nassist\tVB\nassist\tVBP\nassociated\tVBN\nassociates\tNNS\nAssociates\tNNS\nassociate\tVBP\nassociating\tVBG\nassociations\tNNS\nassuage\tVB\nassumed\tVBD\nassumed\tVBN\nassumes\tVBZ\nassume\tVB\nAssume\tVB\nassume\tVBP\nassuming\tVBG\nAssuming\tVBG\nassumptions\tNNS\nassurances\tNNS\nassured\tVBD\nassured\tVBN\nassures\tVBZ\nassure\tVB\nassuring\tVBG\nasteroids\tNNS\nastounds\tVBZ\nastronauts\tNNS\nate\tVBD\nAte\tVBD\nathletes\tNNS\nathletics\tNNS\nAtlantis\tNNS\natolls\tNNS\natoms\tNNS\nAtone\tVB\natrocities\tNNS\nattached\tVBD\nattached\tVBN\nattaches\tVBZ\nattaching\tVBG\nattach\tVB\nattach\tVBP\nattacked\tVBD\nattacked\tVBN\nattackers\tNNS\nattacking\tVBG\nattacks\tNNS\nAttacks\tNNS\nattacks\tVBZ\nattack\tVB\nattack\tVBP\nattained\tVBN\nattain\tVB\nattarcks\tNNS\nattempted\tVBD\nattempted\tVBN\nattempting\tVBG\nattempts\tNNS\nAttempts\tNNS\nattempts\tVBZ\nattempt\tVB\nattempt\tVBP\nattendants\tNNS\nAttendants\tNNS\nattended\tVBD\nattended\tVBN\nattendees\tNNS\nattending\tVBG\nattends\tVBZ\nattend\tVB\nattend\tVBP\nAttention\tVB\nattests\tVBZ\nattest\tVB\nattics\tNNS\nattitudes\tNNS\nAttitudes\tNNS\nattorneys\tNNS\nAttorneys\tNNS\nattracted\tVBD\nattracted\tVBN\nAttracted\tVBN\nattracting\tVBG\nattractions\tNNS\nattracts\tVBZ\nattract\tVB\nattract\tVBP\nattributed\tVBD\nattributed\tVBN\nattributes\tNNS\nattributes\tVBZ\nattribute\tVB\nattribute\tVBP\nattributing\tVBG\nauctioned\tVBD\nauctioned\tVBN\nauctions\tNNS\nAuctions\tNNS\nauction\tVB\naudiences\tNNS\naudiocassettes\tNNS\naudiophiles\tNNS\naudited\tVBD\naudited\tVBN\nauditing\tVBG\naudition\tVB\nauditors\tNNS\nAuditors\tNNS\naudits\tNNS\nAUDITS\tNNS\naudit\tVB\naugment\tVB\nauspices\tNNS\nauthored\tVBN\nauthorities\tNNS\nAuthorities\tNNS\nauthorizations\tNNS\nauthorized\tVBD\nauthorized\tVBN\nauthorizes\tVBZ\nauthorize\tVB\nauthorizing\tVBG\nauthors\tNNS\nautions\tNNS\nauto-emissions\tNNS\nautographed\tVBN\nautographs\tNNS\nautograph\tVB\nauto\\/homeowners\tNNS\nautomakers\tNNS\nautomated\tVBN\nAutomated\tVBN\nautomates\tVBZ\nautomating\tVBG\nautomobiles\tNNS\nAutomobiles\tNNS\nauto-sales\tNNS\nautos\tNNS\nautumns\tNNS\navenge\tVB\navenues\tNNS\naveraged\tVBD\naveraged\tVBN\naverages\tNNS\naverages\tVBZ\naverage\tVB\naverage\tVBP\naveraging\tVBG\naverred\tVBD\naverted\tVBN\naverting\tVBG\naverts\tVBZ\navert\tVB\naviators\tNNS\navoided\tVBD\nAVOIDED\tVBD\navoided\tVBN\navoiding\tVBG\nAvoiding\tVBG\navoids\tVBZ\navoid\tVB\navoid\tVBP\nawaited\tVBD\nawaiting\tVBG\nawaits\tVBZ\nawait\tVB\nawait\tVBP\nawakened\tVBN\nawarded\tVBD\nawarded\tVBN\nawarding\tVBG\nawards\tNNS\nawards\tVBZ\naward\tVB\nawoke\tVBD\naxioms\tNNS\naxles\tNNS\nB-2s\tNNS\nBabelists\tNNS\nbabies\tNNS\nBabies\tNNS\nbackdated\tVBD\nback-dating\tVBG\nbacked\tVBD\nBACKED\tVBD\nbacked\tVBN\nbackers\tNNS\nbackfired\tVBD\nbackfired\tVBN\nbackfires\tVBZ\nbackfire\tVB\nbackfire\tVBP\nbackflips\tVBZ\nbackgrounds\tNNS\nbacking\tVBG\nbacklogs\tNNS\nbacklots\tNNS\nbackpackers\tNNS\nbackpacks\tNNS\nbackpedaling\tVBG\nbackslapping\tVBG\nbacks\tNNS\nbacks\tVBZ\nBacks\tVBZ\nback-ups\tNNS\nback\tVB\nback\tVBP\nbacteria\tNNS\nbadges\tNNS\nbags\tNNS\nbailed\tVBD\nBailiffs\tNNS\nbailing\tVBG\nbailouts\tNNS\nbail\tVB\nbakeries\tNNS\nbakers\tNNS\nbalanced\tVBN\nbalance-of-payments\tNNS\nbalances\tNNS\nBALANCES\tNNS\nbalances\tVBZ\nbalance\tVB\nbalancing\tVBG\nbalconies\tNNS\nbales\tNNS\nbalked\tVBD\nbalking\tVBG\nBALKS\tVBZ\nbalk\tVB\nbalk\tVBP\nballerinas\tNNS\nballets\tNNS\nballooned\tVBD\nballooned\tVBN\nballooning\tVBG\nballoonists\tNNS\nballoons\tNNS\nballoons\tVBZ\nballoon\tVB\nballots\tNNS\nBALLOTS\tNNS\nballparks\tNNS\nballplayers\tNNS\nballs\tNNS\nballyhooed\tVBN\nbamboozled\tVBN\nbananas\tNNS\nbandages\tNNS\nbanded\tVBN\nbandied\tVBN\nbands\tNNS\nband\tVB\nbangs\tVBZ\nbang\tVB\nbanished\tVBN\nbanish\tVB\nbankers\tNNS\nBankers\tNNS\nBANKERS\tNNS\nbanking\tVBG\nbankrolling\tVBG\nbankroll\tVB\nbankroll\tVBP\nbankruptcies\tNNS\nbankrupt\tVB\nbankrupt\tVBP\nbanks\tNNS\nBanks\tNNS\nbank\tVB\nBank\tVB\nbank\tVBP\nbanned\tVBD\nbanned\tVBN\nbanners\tNNS\nbanning\tVBG\nbanshees\tNNS\nbans\tNNS\nbans\tVBZ\nban\tVB\nban\tVBP\nbarbed\tVBN\nbarbers\tNNS\nbarbs\tNNS\nbargained\tVBD\nbargained\tVBN\nbargain-hunters\tNNS\nbargain-hunt\tVB\nbargaining\tVBG\nbargains\tNNS\nbargain\tVB\nbarges\tNNS\nbarges\tVBZ\nbaring\tVBG\nbarking\tVBG\nbarnacles\tNNS\nbarns\tNNS\nbarons\tNNS\nbarred\tVBD\nbarred\tVBN\nBarred\tVBN\nbarreling\tVBG\nbarrels\tNNS\nBarrels\tNNS\nbarricades\tNNS\nbarriers\tNNS\nbarring\tVBG\nBarring\tVBG\nbars\tNNS\nbars\tVBZ\nbartenders\tNNS\nbartered\tVBN\nbar\tVB\nBar\tVB\nbaseballs\tNNS\nbased\tVBD\nbased\tVBN\nBased\tVBN\nbasements\tNNS\nbases\tNNS\nbases\tVBZ\nbase\tVB\nbase\tVBP\nbashing\tVBG\nBashing\tVBG\nbash\tVB\nbasics\tNNS\nbasing\tVBG\nbaskets\tNNS\nBaskets\tNNS\nbastions\tNNS\nbatches\tNNS\nbathed\tVBN\nbathrooms\tNNS\nbaths\tNNS\nbats\tNNS\nbattalions\tNNS\nbatted\tVBD\nbatted\tVBN\nbattered\tVBD\nbattered\tVBN\nbatteries\tNNS\nbattering\tVBG\nbatter\tVB\nbatting\tVBG\nbattled\tVBD\nBATTLED\tVBD\nbattled\tVBN\nbattlegroups\tNNS\nbattlements\tNNS\nbattles\tNNS\nbattles\tVBZ\nbattle\tVB\nbattle\tVBP\nbattling\tVBG\nbays\tNNS\nbeaches\tNNS\nbeads\tNNS\nbeamed\tVBN\nbeaming\tVBG\nbeams\tVBZ\nbeanballs\tNNS\nbeans\tNNS\nbearings\tNNS\nbearing\tVBG\nbears\tNNS\nBears\tNNS\nbears\tVBZ\nbear\tVB\nbear\tVBP\nbeasties\tNNS\nbeasts\tNNS\nbeaten\tVBN\nbeating\tVBG\nbeats\tNNS\nbeats\tVBZ\nbeat\tVB\nbeat\tVBD\nbeat\tVBP\nbecame\tVBD\nbecomes\tVBZ\nbecome\tVB\nBecome\tVB\nBECOME\tVB\nbecome\tVBD\nbecome\tVBN\nbecome\tVBP\nbecoming\tVBG\nbedeviled\tVBN\nbedevil\tVB\nbedfellows\tNNS\nBedfellows\tNNS\nbedpans\tNNS\nbedrooms\tNNS\nbeds\tNNS\nbeefed\tVBD\nbeefed\tVBN\nbeefing\tVBG\nbeef\tVB\nbeen\tVBN\nBeen\tVBN\nbeen\tVBP\nbeeping\tVBG\nbeeps\tNNS\nbeers\tNNS\nbees\tNNS\nbefallen\tVBN\nbefall\tVBP\nbefell\tVBD\nbefriended\tVBN\nbefuddled\tVBD\nbegan\tVBD\nbeggars\tNNS\nbegged\tVBN\nbegging\tVBG\nbeginnings\tNNS\nbeginning\tVBG\nBeginning\tVBG\nbegins\tVBZ\nBegins\tVBZ\nbegin\tVB\nbegin\tVBP\nbegot\tVBD\nbegs\tVBZ\nbegun\tVBN\nbeg\tVB\nbehaved\tVBD\nbehaved\tVBN\nbehaves\tVBZ\nbehave\tVB\nbehaving\tVBG\nbehaviors\tNNS\nBeheading\tVBG\nbehemoths\tNNS\nbeings\tNNS\nbeing\tVBG\nBeing\tVBG\nBEING\tVBG\nbeing\tVBG|JJ\nbeleaguered\tVBN\nbelfries\tNNS\nbelied\tVBD\nbeliefs\tNNS\nbelie\tVB\nbelie\tVBP\nbelieved\tVBD\nbelieved\tVBN\nbelieves\tVBZ\nbelieve\tVB\nbelieve\tVBP\nbelieving\tVBG\nbelittle\tVB\nbellies\tNNS\nbellow\tVB\nbellringers\tNNS\nbells\tNNS\nbellwethers\tNNS\nbelly-flopped\tVBD\nbelonged\tVBD\nbelongings\tNNS\nbelonging\tVBG\nbelongs\tVBZ\nbelong\tVB\nbelong\tVBP\nbelting\tVBG\nbelts\tNNS\nbemoaning\tVBG\nbenches\tNNS\nbenchmarks\tNNS\nbending\tVBG\nbend\tVB\nbenefactors\tNNS\nbeneficiaries\tNNS\nBeneficiaries\tNNS\nbenefited\tVBD\nbenefited\tVBN\nbenefiting\tVBG\nbenefits\tNNS\nBENEFITS\tNNS\nbenefits\tVBZ\nbenefit\tVB\nBenefit\tVB\nbenefit\tVBP\nbequeathed\tVBD\nbequests\tNNS\nberated\tVBD\nberries\tNNS\nbeset\tVBN\nbesieged\tVBN\nbested\tVBD\nbested\tVBN\nbestirred\tVBN\nbestowed\tVBN\nbetas\tNNS\nbetrayed\tVBN\nbets\tNNS\nBets\tNNS\nbetters\tNNS\nbetter\tVB\nbetting\tVBG\nbet\tVB\nbet\tVBD\nbet\tVBN\nbet\tVBP\nbe\tVB\nBe\tVB\nBE\tVB\nbe\tVBP\nbeverages\tNNS\nbeware\tVB\nBeware\tVB\nBEWARE\tVB\nbewildered\tVBN\nbewildering\tVBG\nbewitched\tVBN\nbiased\tVBN\nbiases\tNNS\nbibles\tNNS\nbickered\tVBN\nbickering\tVBG\nbicycles\tNNS\nbidders\tNNS\nbidding\tVBG\nbids\tNNS\nBids\tNNS\nbids\tVBZ\nbid\tVB\nbid\tVBD\nbid\tVBN\nbid\tVBP\nbifurcate\tVB\nbikers\tNNS\nBikers\tNNS\nbikes\tNNS\nBiking\tVBG\nbilges\tNNS\nbilking\tVBG\nbilled\tVBD\nbilled\tVBN\nbillings\tNNS\nBillings\tNNS\nbilling\tVBG\nbillions\tNNS\nBillions\tNNS\nbillowing\tVBG\nbills\tNNS\nBills\tNNS\nBILLS\tNNS\nbills\tVBZ\nbill\tVB\nbill\tVBP\nbinders\tNNS\nbinding\tVBG\nbinges\tNNS\nbinoculars\tNNS\nbins\tNNS\nBioengineers\tNNS\nbiographers\tNNS\nbiologists\tNNS\nbiomedical-products\tNNS\nbiopsies\tNNS\nbirds\tNNS\nBIRDS\tNNS\nbirthdays\tNNS\nbirths\tNNS\nbiscuits\tNNS\nBishops\tNNS\nbites\tNNS\nbites\tVBZ\nbite\tVB\nbiting\tVBG\nbits\tNNS\nbitten\tVBN\nBitten\tVBN\nblabs\tVBZ\nblacked\tVBN\nblackened\tVBN\nblacklist\tVB\nblackmailed\tVBN\nblackmailers\tNNS\nblackmailing\tVBG\nblackmail\tVB\nblackouts\tNNS\nblacks\tNNS\nBlacks\tNNS\nBlackstone\tVBP\nblades\tNNS\nblamed\tVBD\nblamed\tVBN\nBlamed\tVBN\nblames\tVBZ\nblame\tVB\nblame\tVBP\nblaming\tVBG\nBlaming\tVBG\nBlandings\tNNS\nblanketed\tVBD\nblankets\tNNS\nblanket\tVBP\nblared\tVBD\nblasted\tVBD\nblasting\tVBG\nblasts\tNNS\nblast\tVB\nblaze\tVBP\nblazing\tVBG\nbleachers\tNNS\nbled\tVBD\nbleeding\tVBG\nbleed\tVB\nblemishes\tNNS\nblending\tVBG\nblends\tNNS\nblend\tVB\nblessed\tVBN\nblessings\tNNS\nbless\tVB\nblew\tVBD\nblindfolded\tVBD\nblinds\tNNS\nblini\tNNS\nblinked\tVBD\nblinkers\tNNS\nblinks\tVBZ\nblink\tVB\nblips\tNNS\nblip\tVB\nblip\tVBP\nblitzes\tNNS\nblocked\tVBD\nblocked\tVBN\nBlocked\tVBN\nblocking\tVBG\nblocks\tNNS\nblocks\tVBZ\nblock\tVB\nblock\tVBP\nBlondes\tNNS\nbloodied\tVBN\nbloodletting\tVBG\nbloods\tNNS\nblooming\tVBG\nblossomed\tVBD\nblossomed\tVBN\nblossoms\tNNS\nblotting\tVBG\nblowing\tVBG\nblown\tVBN\nblows\tNNS\nblow\tVB\nbludgeoned\tVBN\nbludgeon\tVB\nbluebloods\tNNS\nblue-chips\tNNS\nblues\tNNS\nBlues\tNNS\nblundered\tVBD\nblunders\tNNS\nblunder\tVB\nblunted\tVBD\nblunted\tVBN\nblunt\tVB\nblurred\tVBD\nblurring\tVBG\nblurting\tVBG\nblurt\tVBP\nblur\tVB\nBMWs\tNNS\nboarding\tVBG\nboardrooms\tNNS\nBoardrooms\tNNS\nboards\tNNS\nboard\tVB\nboasted\tVBD\nboasted\tVBN\nboasts\tVBZ\nboast\tVB\nboast\tVBP\nboaters\tNNS\nboating\tVBG\nboats\tNNS\nbodegas\tNNS\nbodes\tVBZ\nbode\tVB\nbodies\tNNS\nbodyworkers\tNNS\nBoeing\tVBG\nbogged\tVBD\nbogging\tVBG\nbog\tVB\nboiled\tVBD\nboilers\tNNS\nboiling\tVBG\nboils\tVBZ\nboil\tVB\nbolstered\tVBD\nbolstered\tVBN\nBolstered\tVBN\nbolstering\tVBG\nBolstering\tVBG\nbolsters\tVBZ\nbolster\tVB\nbolted\tVBN\nbolts\tNNS\nbombarded\tVBD\nbombarded\tVBN\nbombarding\tVBG\nbombed\tVBD\nbombed\tVBN\nbombers\tNNS\nbombings\tNNS\nbomblets\tNNS\nbombs\tNNS\nbomb\tVB\nbonded\tVBN\nbond-holders\tNNS\nbondholders\tNNS\nbondholdings\tNNS\nbonds\tNNS\nBonds\tNNS\nBONDS\tNNS\nboned\tVBN\nbones\tNNS\nbonnets\tNNS\nbonuses\tNNS\nBonuses\tNNS\nbooed\tVBD\nbooked\tVBD\nbooked\tVBN\nbookers\tNNS\nbookings\tNNS\nBookings\tNNS\nbooking\tVBG\nbooklets\tNNS\nbooks\tNNS\nBooks\tNNS\nbookstores\tNNS\nbook\tVB\nboomed\tVBD\nboomed\tVBN\nboomers\tNNS\nbooming\tVBG\nbooms\tNNS\nboom\tVB\nboosted\tVBD\nboosted\tVBN\nboosters\tNNS\nboosting\tVBG\nboosts\tNNS\nBOOSTS\tNNS\nboosts\tVBZ\nboost\tVB\nBoost\tVB\nboost\tVBP\nbooths\tNNS\nBooths\tNNS\nbootlegged\tVBN\nboots\tNNS\nboozing\tVBG\nbordered\tVBN\nbordering\tVBG\nborders\tNNS\nbored\tVBN\nbore\tVBD\nboring\tVBG\nborne\tVBN\nborn\tVBN\nBorn\tVBN\nborrowed\tVBD\nborrowed\tVBN\nBorrowed\tVBN\nborrowers\tNNS\nBorrowers\tNNS\nborrowings\tNNS\nborrowing\tVBG\nborrowing\tVBG|NN\nborrows\tVBZ\nborrow\tVB\nborrow\tVBP\nbosses\tNNS\nBosses\tNNS\nbotched\tVBN\nbothered\tVBD\nbothered\tVBN\nbothering\tVBG\nbother\tVB\nbother\tVBP\nbottled\tVBN\nbottlenecks\tNNS\nbottlers\tNNS\nbottles\tNNS\nbottle\tVB\nbottling\tVBG\nbottomed\tVBD\nbottomed\tVBN\nbottoming\tVBG\nbottoms\tNNS\nbottom\tVB\nbought\tVBD\nbought\tVBN\nbounced\tVBD\nbounces\tNNS\nbounces\tVBZ\nbounce\tVB\nbounce\tVBP\nbouncing\tVBG\nboundaries\tNNS\nbounding\tVBG\nbounds\tNNS\nbounds\tVBZ\nbound\tVBN\nbourbons\tNNS\nbourses\tNNS\nboutiques\tNNS\nbouts\tNNS\nBouygues\tNNS\nbowed\tVBD\nbowed\tVBN\nBowing\tVBG\nbowling\tVBG\nbowls\tVBZ\nbowl\tVBP\nbow\tVB\nbow\tVBP\nboxes\tNNS\nboycotted\tVBN\nboycott\tVB\nboyfriends\tNNS\nboys\tNNS\nBoys\tNNS\nbraced\tVBD\nbraced\tVBN\nbrace\tVB\nbracing\tVBG\nbrags\tVBZ\nbrag\tVB\nbraids\tNNS\nbrains\tNNS\nbrakes\tNNS\nBrakes\tNNS\nbraking\tVBG\nbranched\tVBN\nbranches\tNNS\nBranches\tNNS\nbranching\tVBG\nbranch\tVB\nbrandished\tVBD\nbrandishing\tVBG\nbrands\tNNS\nBrands\tNNS\nbrands\tVBZ\nbrassieres\tNNS\nbrats\tNNS\nbraved\tVBD\nbrave\tVB\nbraving\tVBG\nBrawls\tNNS\nbrazen\tVB\nBrazilians\tNNS\nbreached\tVBD\nbreached\tVBN\nbreaches\tNNS\nbreach\tVB\nbreaded\tVBN\nbreakdowns\tNNS\nbreakers\tNNS\nbreaking\tVBG\nBreaking\tVBG\nbreaks\tNNS\nbreaks\tVBZ\nbreakthroughs\tNNS\nbreak\tVB\nbreak\tVBP\nbreasts\tNNS\nbreathed\tVBD\nbreathe\tVB\nbreathing\tVBG\nbreaths\tNNS\nbred\tVBD\nbred\tVBN\nbreeders\tNNS\nBreeders\tNNS\nbreeding\tVBG\nbreed\tVB\nbreed\tVBP\nbreezes\tNNS\nbrethren\tNNS\nbrewed\tVBN\nbreweries\tNNS\nBreweries\tNNS\nbrewers\tNNS\nbrewing\tVBG\nbrew\tVB\nbribed\tVBD\nbribed\tVBN\nbribes\tNNS\nbribe\tVB\nbribing\tVBG\nbricks\tNNS\nbridges\tNNS\nbridge\tVB\nbridging\tVBG\nbriefcases\tNNS\nbriefed\tVBD\nbriefed\tVBN\nbriefings\tNNS\nbriefs\tNNS\nBRIEFS\tNNS\nbrief\tVB\nbrigades\tNNS\nbrightened\tVBD\nbrightened\tVBN\nbrightening\tVBG\nbrimmed\tVBD\nbringing\tVBG\nbrings\tVBZ\nbring\tVB\nbring\tVBP\nbristled\tVBD\nbristles\tVBZ\nbristle\tVBP\nBritish\tNNS\nbroadcasters\tNNS\nBroadcasters\tNNS\nbroadcasting\tVBG\nBroadcasting\tVBG\nbroadcasts\tNNS\nBroadcasts\tNNS\nbroadcasts\tVBZ\nbroadcast\tVB\nbroadcast\tVBD\nbroadcast\tVBN\nbroadened\tVBD\nbroadened\tVBN\nbroadening\tVBG\nbroadens\tVBZ\nbroaden\tVB\nbrochures\tNNS\nbroken\tVBN\nbrokerages\tNNS\nbroker-dealers\tNNS\nbrokered\tVBD\nbrokering\tVBG\nbrokers\tNNS\nBrokers\tNNS\nbroke\tVBD\nBroncos\tNNS\nbroncs\tNNS\nbrothers\tNNS\nbrought\tVBD\nbrought\tVBN\nbrowbeat\tVB\nbrowse\tVB\nbrowsing\tVBG\nbruised\tVBN\nbruises\tNNS\nbruising\tVBG\nbrushbacks\tNNS\nbrushed\tVBD\nbrushed\tVBN\nbrushes\tNNS\nbrushing\tVBG\nbrush\tVB\nBrussels\tNNS\nbubbles\tNNS\nbubble\tVB\nBuccaneers\tNNS\nbucked\tVBD\nbucked\tVBN\nbuckets\tNNS\nbucking\tVBG\nBucking\tVBG\nbuckled\tVBD\nbuckle\tVB\nbuckling\tVBG\nbucks\tNNS\nbucks\tVBZ\nbuck\tVB\nbuck\tVBP\nbuddies\tNNS\nbudding\tVBG\nbudged\tVBD\nbudged\tVBN\nbudgeted\tVBN\nbudgeteers\tNNS\nbudgets\tNNS\nbudge\tVB\nbuds\tNNS\nbuffer\tVB\nbuffeted\tVBN\nbuffets\tNNS\nBuffets\tNNS\nbuffet\tVB\nbuffing\tVBG\nbuffs\tNNS\nbugged\tVBN\nbugs\tNNS\nBugs\tNNS\nbuilders\tNNS\nBuilders\tNNS\nbuilding-materials\tNNS\nbuilding-products\tNNS\nbuildings\tNNS\nBuildings\tNNS\nbuilding\tVBG\nBuilding\tVBG\nbuilds\tVBZ\nbuild\tVB\nbuild\tVBN\nbuild\tVBP\nbuilt\tVBD\nbuilt\tVBN\nBuilt\tVBN\nbulbs\tNNS\nbulging\tVBG\nbulkheads\tNNS\nbulldozed\tVBN\nbulldozers\tNNS\nbulletins\tNNS\nbullets\tNNS\nbullhorns\tNNS\nbullied\tVBD\nbullies\tVBZ\nbulls\tNNS\nBulls\tNNS\nbullying\tVBG\nbumble\tVB\nbumble\tVBP\nbumbling\tVBG\nbumped\tVBD\nbumped\tVBN\nbumps\tNNS\nbump\tVB\nBums\tNNS\nbunches\tNNS\nbundled\tVBN\nbundles\tNNS\nbundling\tVBG\nbungled\tVBN\nbuns\tNNS\nbuoyed\tVBD\nbuoyed\tVBN\nBuoyed\tVBN\nbuoying\tVBG\nbuoys\tVBZ\nbuoy\tVB\nburbles\tVBZ\nburdened\tVBD\nburdened\tVBN\nburdens\tNNS\nBurdens\tNNS\nburdens\tVBZ\nbureaucracies\tNNS\nbureaucrats\tNNS\nBureaucrats\tNNS\nbureaus\tNNS\nburgeoning\tVBG\nburgers\tNNS\nburglaries\tNNS\nburglarized\tVBN\nburgs\tNNS\nburials\tNNS\nburied\tVBD\nburied\tVBN\nBuried\tVBN\nburned\tVBD\nburned\tVBN\nBurned\tVBN\nburning\tVBG\nburnishing\tVBG\nburnouts\tNNS\nburns\tNNS\nburns\tVBZ\nburnt\tVBN\nburn\tVB\nburn\tVBP\nbursting\tVBG\nbursts\tNNS\nbursts\tVBZ\nburst\tVBD\nburst\tVBP\nburying\tVBG\nbury\tVB\nbury\tVBP\nbuses\tNNS\nbushels\tNNS\nbushes\tNNS\nbusies\tNNS\nbusiness-communications\tNNS\nbusinesses\tNNS\nBusinesses\tNNS\nbusiness-machines\tNNS\nbusinessmen\tNNS\nBusinessmen\tNNS\nbusiness-partners\tNNS\nbusloads\tNNS\nbusted\tVBD\nbusted\tVBN\nbusting\tVBG\nbusts\tNNS\nbutlers\tNNS\nbutterflies\tNNS\nbuttons\tNNS\nbuttressed\tVBN\nbuttresses\tVBZ\nbuttress\tVB\nbutt\tVB\nbuy-backs\tNNS\nbuy-back\tVB\nbuyers\tNNS\nBuyers\tNNS\nBUYERS\tNNS\nbuyings\tNNS\nbuying\tVBG\nBuying\tVBG\nbuy-outs\tNNS\nbuyouts\tNNS\nbuys\tNNS\nbuys\tVBZ\nbuy\tVB\nBuy\tVB\nbuy\tVB|NN\nbuy\tVBP\nbuzzes\tVBZ\nbuzzing\tVBG\nbuzz\tVB\nbuzzwords\tNNS\nbylaws\tNNS\nbylines\tNNS\nbypass\tVB\nbystanders\tNNS\ncabin-crew\tNNS\ncabinets\tNNS\ncables\tNNS\ncabs\tNNS\ncaches\tNNS\ncadets\tNNS\ncadge\tVBP\ncafes\tNNS\ncafeterias\tNNS\ncaked\tVBD\ncalamities\tNNS\nCalaveras\tNNS\ncalculated\tVBD\ncalculated\tVBN\nCalculated\tVBN\ncalculates\tVBZ\ncalculate\tVB\ncalculate\tVBP\ncalculating\tVBG\nCalculating\tVBG\ncalculations\tNNS\ncalculators\tNNS\ncalibrated\tVBN\nCalifornians\tNNS\ncalled\tVB\ncalled\tVBD\nCalled\tVBD\ncalled\tVBN\nCalled\tVBN\nCALLED\tVBN\ncallers\tNNS\nCallers\tNNS\ncalling\tVBG\nCalling\tVBG\ncalls\tNNS\nCalls\tNNS\ncalls\tVBZ\ncall\tVB\nCall\tVB\nCALL\tVB\ncall\tVBP\ncalmed\tVBN\ncalming\tVBG\ncalm\tVB\ncalories\tNNS\nCaltrans\tNNS\ncalves\tNNS\ncameras\tNNS\ncame\tVBD\ncamouflaged\tVBN\ncampaigned\tVBN\ncampaigning\tVBG\ncampaigns\tNNS\ncampaign\tVB\ncamped\tVBD\ncampers\tNNS\ncamps\tNNS\ncampuses\tNNS\nCampuses\tNNS\ncanals\tNNS\ncanceled\tVBD\ncanceled\tVBN\ncanceled\tVBN|JJ\ncanceling\tVBG\ncancellations\tNNS\ncancels\tVBZ\ncancel\tVB\ncancers\tNNS\ncancer-suppressors\tNNS\ncandidates\tNNS\ncandies\tNNS\ncandles\tNNS\ncanned\tVBN\ncanning\tVBG\ncans\tNNS\ncanvases\tNNS\ncanvassed\tVBN\ncanyons\tNNS\ncapabilities\tNNS\ncapacities\tNNS\ncapacitors\tNNS\ncapital-assets\tNNS\ncapital-draining\tVBG\ncapital-gains\tNNS\ncapitalgains\tNNS\ncapital-goods\tNNS\ncapitalists\tNNS\ncapitalized\tVBD\ncapitalized\tVBN\ncapitalize\tVB\ncapitalizing\tVBG\nCapitalizing\tVBG\ncapitals\tNNS\ncapital-to-assets\tNNS\ncapita\tNNS\ncapitulated\tVBD\ncapped\tVBD\ncapped\tVBN\ncapping\tVBG\ncaps\tNNS\ncapsules\tNNS\ncaptain\tVBP\ncaptioned\tVBD\ncaptivating\tVBG\ncaptives\tNNS\ncaptured\tVBD\ncaptured\tVBN\ncaptured\tVBN|JJ\ncapture\tVB\ncapturing\tVBG\ncap\tVB\ncap\tVBP\ncarats\tNNS\ncarbide-products\tNNS\ncar-dealers\tNNS\ncardholders\tNNS\ncardinals\tNNS\ncards\tNNS\ncared\tVBD\ncared\tVBN\ncareened\tVBD\ncareening\tVBG\ncareen\tVB\ncareers\tNNS\nCareers\tNNS\ncares\tVBZ\ncare\tVB\ncare\tVBP\ncaricatures\tNNS\ncaricature\tVB\ncarillons\tNNS\ncaring\tVBG\ncarnivores\tNNS\ncarpenters\tNNS\ncarpetbaggers\tNNS\ncarpeted\tVBD\ncarpets\tNNS\ncarp\tVBP\ncarried\tVBD\ncarried\tVBN\nCarried\tVBN\ncarriers\tNNS\nCarriers\tNNS\ncarries\tVBZ\ncarry-forwards\tNNS\ncarryforwards\tNNS\ncarrying\tVBG\nCarrying\tVBG\ncarry\tVB\ncarry\tVBP\ncars\tNNS\nCars\tNNS\ncarted\tVBD\ncarting\tVBG\nCartons\tNNS\ncartoonists\tNNS\ncartoons\tNNS\ncartridges\tNNS\ncarts\tNNS\ncart\tVBP\ncarved\tVBD\ncarved\tVBN\ncarvers\tNNS\ncarves\tVBZ\ncarve\tVB\ncarve\tVBP\ncarving\tVBG\ncascaded\tVBD\ncascading\tVBG\ncaseloads\tNNS\ncases\tNNS\nCases\tNNS\ncases\tVBZ\ncashed\tVBD\ncashed\tVBN\ncashing\tVBG\ncash\tVB\ncasings\tNNS\ncasinos\tNNS\nCasinos\tNNS\ncaskets\tNNS\ncassettes\tNNS\ncastigated\tVBN\ncastigate\tVB\ncastigating\tVBG\ncastings\tNNS\ncasting\tVBG\ncastling\tVBG\ncasts\tVBZ\ncast\tVB\ncast\tVBD\ncast\tVBN\nCast\tVBN\ncasualties\tNNS\ncataclysms\tNNS\ncataloging\tVBG\ncatalogs\tNNS\ncatapult\tVB\ncatapult\tVBP\nCataracts\tNNS\ncatastrophes\tNNS\ncatchers\tNNS\ncatches\tVBZ\ncatching\tVBG\nCatching\tVBG\ncatch\tVB\ncatch\tVBP\ncategories\tNNS\ncategorized\tVBN\ncatered\tVBD\ncatering\tVBG\ncaters\tVBZ\ncater\tVB\ncater\tVBP\nCATFISH\tNNS\ncatheters\tNNS\ncathodes\tNNS\ncats\tNNS\ncattle\tNNS\nCattle\tNNS\ncaught\tVBD\ncaught\tVBN\ncaused\tVBD\ncaused\tVBN\ncauses\tNNS\ncauses\tVBZ\ncause\tVB\ncause\tVBP\ncausing\tVBG\ncautioned\tVBD\ncautioning\tVBG\ncautions\tVBZ\ncaution\tVB\ncaution\tVBP\ncaveats\tNNS\ncaved\tVBD\ncaves\tNNS\nC.D.s\tNNS\nCDs\tNNS\nceased\tVBD\nceased\tVBN\nceases\tVBZ\ncease\tVB\ncease\tVBP\nceded\tVBD\nceded\tVBN\ncede\tVB\nceding\tVBG\nceilings\tNNS\ncelebrated\tVBD\ncelebrates\tVBZ\ncelebrate\tVB\ncelebrate\tVBP\ncelebrating\tVBG\ncelebrations\tNNS\ncelebrities\tNNS\ncellars\tNNS\ncellists\tNNS\ncells\tNNS\ncelluloids\tNNS\ncemented\tVBN\nCementing\tVBG\ncement-makers\tNNS\ncement\tVB\ncensored\tVBN\ncensor\tVBP\ncensured\tVBD\ncentenarians\tNNS\ncentered\tVBD\ncentered\tVBN\ncentering\tVBG\ncenters\tNNS\ncenters\tVBZ\nCenters\tVBZ\ncenter\tVBP\ncentimeters\tNNS\ncentralized\tVBN\ncentralize\tVB\ncents\tNNS\nCents\tNNS\ncenturies\tNNS\ncenturions\tNNS\nCEOs\tNNS\nceramics\tNNS\ncereals\tNNS\nceremonies\tNNS\ncertificates\tNNS\nCertificates\tNNS\nCERTIFICATES\tNNS\ncertified\tVBN\ncertify\tVB\nCFCs\tNNS\nchafed\tVBN\nchafe\tVBP\nchains\tNNS\nChains\tNNS\nchain\tVBP\nchaired\tVBD\nchaired\tVBN\nchairmen\tNNS\nchairs\tNNS\nchairs\tVBZ\nchalked\tVBN\nchalking\tVBG\nchallenged\tVBD\nchallenged\tVBN\nchallengers\tNNS\nchallenges\tNNS\nchallenges\tVBZ\nChallenges\tVBZ\nchallenge\tVB\nchallenge\tVBP\nchallenging\tVBG\nchambers\tNNS\nChambers\tNNS\nChampagnes\tNNS\nchampioned\tVBD\nchampioned\tVBN\nchampioning\tVBG\nchampionships\tNNS\nchampions\tNNS\nchampion\tVBP\nchamps\tNNS\nchances\tNNS\nchance\tVBP\nchandeliers\tNNS\nchanged\tVBD\nCHANGED\tVBD\nchanged\tVBN\nCHANGED\tVBN\nchanges\tNNS\nChanges\tNNS\nchanges\tVBZ\nchange\tVB\nChange\tVB\nchange\tVBP\nchanging\tVBG\nChanging\tVBG\nchanneled\tVBD\nchannels\tNNS\nchannel\tVB\nchannel\tVBP\nChans\tNNS\nchanted\tVBD\nchanted\tVBN\nchanting\tVBG\nchants\tNNS\nchaps\tNNS\nchapters\tNNS\ncharacteristics\tNNS\ncharacterized\tVBD\ncharacterized\tVBN\ncharacterizes\tVBZ\ncharacterize\tVB\ncharacterize\tVBP\ncharacterizing\tVBG\ncharacters\tNNS\nCharacters\tNNS\ncharged\tVBD\ncharged\tVBN\ncharge-offs\tNNS\ncharges\tNNS\nCharges\tNNS\ncharges\tVBZ\ncharge\tVB\nCharge\tVB\ncharge\tVBP\ncharging\tVBG\ncharities\tNNS\nCharities\tNNS\ncharlatans\tNNS\nCharlestonians\tNNS\ncharred\tVBN\nchartered\tVBD\nchartered\tVBN\ncharts\tNNS\ncharts\tVBZ\nchart\tVB\nchart\tVBP\nchased\tVBD\nchasers\tNNS\nchase\tVB\nchasing\tVBG\nchassis\tNNS\nchastened\tVBD\nchastened\tVBN\nchastised\tVBD\nchastised\tVBN\nchastises\tVBZ\nchatting\tVBG\nchat\tVB\nchauffeurs\tNNS\nchauffeur\tVB\ncheapens\tVBZ\ncheated\tVBD\ncheaters\tNNS\ncheating\tVBG\ncheat\tVB\ncheat\tVBP\ncheckbooks\tNNS\nchecked\tVBD\nchecked\tVBN\nchecking\tVBG\ncheckpoints\tNNS\nchecks\tNNS\nchecks\tVBZ\ncheck\tVB\nCheck\tVB\ncheck\tVBP\ncheered\tVBD\ncheered\tVBN\ncheering\tVBG\ncheerleaders\tNNS\ncheers\tNNS\ncheer\tVB\ncheeses\tNNS\nchefs\tNNS\nchelicerates\tNNS\nchemicals\tNNS\nChemicals\tNNS\nchemical-weapons\tNNS\nchemists\tNNS\ncherished\tVBN\ncherishes\tVBZ\ncherries\tNNS\ncherubs\tNNS\nchewed\tVBD\nchewing\tVBG\nchews\tNNS\nchew\tVB\nchew\tVBP\nchickens\tNNS\nChickens\tNNS\nchided\tVBD\nchided\tVBN\nchides\tVBZ\nchiefs\tNNS\nchildren\tNNS\nChildren\tNNS\nCHILDREN\tNNS\nchilled\tVBN\nchilling\tVBG\nchill\tVB\nchimes\tVBZ\nchimneys\tNNS\nchimpanzees\tNNS\nChinese\tNNS\nchipped\tVBN\nchipping\tVBG\nchips\tNNS\nchisel\tVB\nchlorofluorocarbons\tNNS\nchoices\tNNS\nChoices\tNNS\nchoked\tVBD\nchoked\tVBN\nchoke\tVB\nchoking\tVBG\nchooses\tVBZ\nchoose\tVB\nChoose\tVB\nchoose\tVBP\nchoosing\tVBG\nchopped\tVBN\nchopping\tVBG\nchops\tNNS\nchopsticks\tNNS\nchop\tVB\nchords\tNNS\nchoreographers\tNNS\nchores\tNNS\nChores\tNNS\nchortled\tVBD\nchoruses\tNNS\nchosen\tVBN\nchose\tVBD\nChristians\tNNS\nchromosomes\tNNS\nChronicles\tNNS\nchronicles\tVBZ\nchucked\tVBD\nchuckles\tNNS\nchuckles\tVBZ\nchuckling\tVBG\nchug\tVBP\nchums\tNNS\nchunks\tNNS\nchurches\tNNS\nchurch-goers\tNNS\nchurn\tVB\nchurn\tVBP\ncials\tNNS\ncigarettes\tNNS\ncigars\tNNS\ncinch\tVB\nCincinnati\tNNS\ncircled\tVBD\ncircles\tNNS\ncircles\tVBZ\ncircuits\tNNS\ncirculars\tNNS\ncirculated\tVBD\ncirculated\tVBN\ncirculate\tVB\ncirculate\tVBP\ncirculating\tVBG\ncirculations\tNNS\ncircumstances\tNNS\ncircumventing\tVBG\ncircumvents\tVBZ\ncircumvent\tVB\ncitations\tNNS\ncited\tVBD\ncited\tVBN\nCited\tVBN\ncites\tVBZ\ncite\tVB\ncite\tVBP\nCiticorp\tVB\ncities\tNNS\nCities\tNNS\nciting\tVBG\nCiting\tVBG\ncitizens\tNNS\nCitizens\tNNS\nCITIZENS\tNNS\ncivics\tNNS\ncivilians\tNNS\ncivil-rights\tNNS\nclad\tVBN\nClad\tVBN\nclaimants\tNNS\nCLAIMANTS\tNNS\nclaimed\tVBD\nclaimed\tVBN\nclaiming\tVBG\nClaiming\tVBG\nclaims\tNNS\nClaims\tNNS\nclaims\tVBZ\nCLAIMS\tVBZ\nclaim\tVB\nclaim\tVBP\nclambered\tVBD\nclamored\tVBD\nclampdowns\tNNS\nclamped\tVBN\nclamping\tVBG\nclamp\tVB\nclanging\tVBG\nclanking\tVBG\nclarifications\tNNS\nclarified\tVBN\nclarifies\tVBZ\nclarifying\tVBG\nclarify\tVB\nclashed\tVBD\nclashed\tVBN\nclashes\tNNS\nclash\tVB\nclassed\tVBN\nclasses\tNNS\nClasses\tNNS\nclassics\tNNS\nClassics\tNNS\nclassifications\tNNS\nclassified\tVBD\nclassified\tVBN\nclassifies\tVBZ\nclassify\tVB\nclassmates\tNNS\nclassrooms\tNNS\nclauses\tNNS\nClaws\tNNS\ncleaned\tVBD\ncleaned\tVBN\ncleaners\tNNS\nclean-fuels\tNNS\ncleaning\tVBG\nCleaning\tVBG\ncleansed\tVBD\ncleansed\tVBN\ncleansers\tNNS\ncleanse\tVB\ncleansing\tVBG\ncleans\tVBZ\nclean\tVB\nclearances\tNNS\ncleared\tVBD\ncleared\tVBN\nclearing\tVBG\nclears\tVBZ\nCLEARS\tVBZ\nclear\tVB\nclear\tVBP\nclergy\tNNS\nclerics\tNNS\nclerks\tNNS\nclicked\tVBD\nclientele\tNNS\nclients\tNNS\nClients\tNNS\nclimbed\tVBD\nclimbed\tVBN\nclimbers\tNNS\nclimbing\tVBG\nclimbs\tVBZ\nclimb\tVB\nclinched\tVBD\nclinching\tVBG\ncling\tVB\ncling\tVBP\nclinical-products\tNNS\nclinics\tNNS\nclinkers\tNNS\nclipped\tVBD\nclipped\tVBN\nclippings\tNNS\nclips\tNNS\nclip\tVB\ncliques\tNNS\ncloak\tVBP\nclobbered\tVBD\nclobbered\tVBN\nclobber\tVB\nclocked\tVBN\nclocks\tNNS\nclocks\tVBZ\nclogged\tVBN\nclogging\tVBG\ncloned\tVBN\nclones\tNNS\nclosed\tVBD\nclosed\tVBN\nClosed\tVBN\nclosed\tVBN|VBD\ncloses\tNNS\ncloses\tVBZ\nCloses\tVBZ\nclose\tVB\nClose\tVB\nclose\tVBP\nclosings\tNNS\nclosing\tVBG\nclosing\tVBG|NN\nclosures\tNNS\nclothed\tVBN\nclothes\tNNS\nclothiers\tNNS\nclouded\tVBN\nclouds\tNNS\nclouds\tVBZ\ncloud\tVB\ncloud\tVBP\nclowns\tNNS\nclubbed\tVBD\nCLUBBING\tVBG\nclubs\tNNS\nClubs\tNNS\nCLUBS\tNNS\nclues\tNNS\nclumps\tNNS\nclustered\tVBN\nclusters\tNNS\nclutching\tVBG\nclutch\tVBP\ncluttered\tVBN\nclutter\tVB\nC'mon\tVB\ncoaches\tNNS\nCoach\tVB\nco-anchored\tVBN\ncoasted\tVBD\ncoasters\tNNS\ncoasts\tNNS\ncoated\tVBN\ncoatings\tNNS\ncoats\tNNS\nCoats\tNNS\ncoattails\tNNS\nco-authored\tVBN\nCo-authors\tNNS\ncoaxing\tVBG\ncoax\tVB\ncobbled\tVBD\ncobbled\tVBN\nCobbs\tNNS\nco-chairmen\tNNS\ncockatoos\tNNS\ncockroaches\tNNS\ncocktails\tNNS\ncoconuts\tNNS\ncoddled\tVBN\ncoded\tVBN\nco-defendants\tNNS\nCODE-NAMED\tVBN\ncodes\tNNS\ncode\tVB\nco-developers\tNNS\ncodified\tVBN\nCodifying\tVBG\nco-edits\tVBZ\ncoerces\tVBZ\nco-exist\tVB\ncoextrude\tVBP\ncoffers\tNNS\nco-founded\tVBD\nco-founders\tNNS\ncognoscenti\tNNS\ncohere\tVB\ncohorts\tNNS\nco-host\tVB\ncoincided\tVBD\ncoincides\tVBZ\ncoincide\tVB\ncoined\tVBN\ncoins\tNNS\ncolas\tNNS\ncold-cuts\tNNS\ncoli\tNNS\ncollaborated\tVBD\ncollaborate\tVB\ncollaborate\tVBP\ncollaborating\tVBG\ncollaborators\tNNS\ncollages\tNNS\ncollapsed\tVBD\ncollapsed\tVBN\ncollapses\tNNS\ncollapses\tVBZ\ncollapse\tVB\ncollapsing\tVBG\ncollars\tNNS\ncollateralized\tVBN\ncolleagues\tNNS\nColleagues\tNNS\ncollected\tVBD\ncollected\tVBN\ncollectibles\tNNS\nCollectibles\tNNS\ncollecting\tVBG\ncollections\tNNS\ncollectives\tNNS\ncollectivizers\tNNS\ncollectors\tNNS\nCollectors\tNNS\ncollects\tVBZ\ncollect\tVB\ncollect\tVBP\ncolleges\tNNS\nColleges\tNNS\ncollege-sports\tNNS\ncolloquies\tNNS\ncolonialists\tNNS\ncolonies\tNNS\ncolonists\tNNS\ncolor-coded\tVBN\ncolor-coding\tVBG\ncolored\tVBN\ncolors\tNNS\ncolor\tVB\ncolumnists\tNNS\ncolumns\tNNS\nco-managed\tVBN\nco-managing\tVBG\ncombatants\tNNS\ncombating\tVBG\nCombatting\tVBG\ncombat\tVB\ncombed\tVBN\ncombinations\tNNS\ncombined\tVBD\ncombined\tVBN\nCombined\tVBN\ncombines\tNNS\ncombines\tVBZ\ncombine\tVB\ncombine\tVBP\ncombing\tVBG\ncombining\tVBG\ncomedies\tNNS\ncomestibles\tNNS\ncomes\tVBZ\nComes\tVBZ\ncomets\tNNS\ncome\tVB\nCome\tVB\ncome\tVBD\ncome\tVBN\nCome\tVBN\ncome\tVBP\ncomforted\tVBN\ncomforting\tVBG\ncomforts\tNNS\ncoming\tVBG\nComing\tVBG\ncommanded\tVBN\ncommanders\tNNS\ncommanding\tVBG\ncommandos\tNNS\ncommand\tVB\ncommand\tVBP\ncommemorated\tVBD\ncommemorate\tVB\ncommemorate\tVBP\ncommenced\tVBD\ncommencing\tVBG\ncommends\tVBZ\ncommend\tVB\ncommentaries\tNNS\ncommentators\tNNS\nCommentators\tNNS\ncommented\tVBD\ncommenting\tVBG\nCommenting\tVBG\ncomments\tNNS\nComments\tNNS\nCOMMENTS\tNNS\ncomments\tVBZ\ncomment\tVB\ncommercialize\tVB\ncommercializing\tVBG\ncommercials\tNNS\nCommercials\tNNS\ncommissioned\tVBD\ncommissioned\tVBN\ncommissioners\tNNS\ncommissioning\tVBG\nCommissioning\tVBG\ncommissions\tNNS\nCommissions\tNNS\ncommissions\tVBZ\ncommitments\tNNS\ncommits\tVBZ\ncommitted\tVBD\ncommitted\tVBN\ncommittees\tNNS\nCommittees\tNNS\ncommittees\tVBZ\ncommittes\tNNS\ncommitting\tVBG\ncommit\tVB\ncommit\tVBP\ncommodities\tNNS\nCommodities\tNNS\ncommoditize\tVB\ncommunicated\tVBD\ncommunicated\tVBN\ncommunicate\tVB\ncommunicating\tVBG\ncommunications\tNNS\ncommuniques\tNNS\ncommunists\tNNS\nCommunists\tNNS\ncommunities\tNNS\ncommuters\tNNS\nCOMMUTERS\tNNS\ncommutes\tNNS\ncommuting\tVBG\ncompacted\tVBN\ncompanies\tNNS\nCompanies\tNNS\nCOMPANIES\tNNS\ncompanions\tNNS\nCompaq\tVB\ncompared\tVBD\ncompared\tVBN\nCompared\tVBN\ncompares\tVBZ\ncompare\tVB\nCompare\tVB\nCOMPARE\tVB\ncompare\tVBP\ncomparing\tVBG\nComparing\tVBG\ncomparisons\tNNS\ncompatriots\tNNS\ncompelled\tVBN\nCompelled\tVBN\ncompels\tVBZ\ncompel\tVB\ncompensated\tVBN\ncompensates\tVBZ\ncompensate\tVB\ncompensations\tNNS\ncompeted\tVBD\ncompeted\tVBN\ncompetes\tVBZ\nCompetes\tVBZ\ncompete\tVB\ncompete\tVBP\ncompeting\tVBG\nCompeting\tVBG\ncompeting\tVBG|JJ\ncompetitions\tNNS\ncompetitors\tNNS\nCompetitors\tNNS\ncompiled\tVBN\nCompiled\tVBN\ncompiles\tVBZ\ncompile\tVB\ncompiling\tVBG\ncomplained\tVBD\ncomplained\tVBN\ncomplaining\tVBG\ncomplains\tVBZ\ncomplaints\tNNS\nComplaints\tNNS\ncomplain\tVB\ncomplain\tVBP\ncomplements\tVBZ\ncomplement\tVB\ncompleted\tVBD\ncompleted\tVBN\ncompletes\tVBZ\ncomplete\tVB\ncompleting\tVBG\ncompletions\tNNS\ncomplexes\tNNS\ncomplexities\tNNS\ncomplicated\tVBD\ncomplicated\tVBN\ncomplicates\tVBZ\ncomplicate\tVB\ncomplications\tNNS\ncomplied\tVBD\ncomplied\tVBN\ncompliments\tNNS\ncompliment\tVBP\ncomplying\tVBG\nComplying\tVBG\ncomply\tVB\ncomponents\tNNS\ncomposed\tVBN\ncomposers\tNNS\ncomposites\tNNS\ncompositions\tNNS\ncompounded\tVBD\ncompounded\tVBN\ncompounding\tVBG\nCompounding\tVBG\ncompounds\tNNS\ncompound\tVB\ncompressed\tVBN\ncompressors\tNNS\ncomprised\tVBN\ncomprises\tVBZ\ncomprise\tVB\ncomprise\tVBP\ncomprising\tVBG\ncompromised\tVBN\ncompromises\tNNS\nCompromises\tNNS\ncompromises\tVBZ\ncompromise\tVB\ncompromising\tVBG\ncompulsions\tNNS\nComputations\tNNS\ncomputerized\tVBN\ncomputerize\tVB\ncomputerizing\tVBG\ncomputer-products\tNNS\ncomputer-services\tNNS\ncomputers\tNNS\nComputers\tNNS\nCOMPUTERS\tNNS\ncomputer-systems\tNNS\ncomputes\tVBZ\ncompute\tVB\ncomputing\tVBG\ncomrades\tNNS\nconcealed\tVBN\nconcealing\tVBG\nconceal\tVB\nconceded\tVBD\nconceded\tVBN\nconcedes\tVBZ\nconcede\tVB\nconcede\tVBP\nconceding\tVBG\nconceived\tVBD\nconceived\tVBN\nconceive\tVB\nconceiving\tVBG\nconcentrated\tVBD\nconcentrated\tVBN\nconcentrates\tVBZ\nconcentrate\tVB\nconcentrate\tVBP\nconcentrating\tVBG\nconcentrations\tNNS\nconceptions\tNNS\nconcepts\tNNS\nconcerned\tVBD\nconcerned\tVBN\nConcerned\tVBN\nconcerning\tVBG\nConcerning\tVBG\nconcerns\tNNS\nConcerns\tNNS\nconcerns\tVBZ\nconcern\tVB\nconcertos\tNNS\nconcerts\tNNS\nconcessions\tNNS\nconcluded\tVBD\nconcluded\tVBN\nconcludes\tVBZ\nconclude\tVB\nconclude\tVBP\nconcluding\tVBG\nconclusions\tNNS\nconcocted\tVBN\nconcoctions\tNNS\nConcocts\tVBZ\nconcurred\tVBD\nconcur\tVB\ncondemned\tVBD\ncondemned\tVBN\ncondemning\tVBG\ncondemns\tVBZ\ncondemn\tVB\ncondensers\tNNS\nconditioned\tVBN\nconditioners\tNNS\nconditions\tNNS\ncondominiums\tNNS\ncondoms\tNNS\ncondoned\tVBD\ncondone\tVB\ncondos\tNNS\nconducted\tVBD\nconducted\tVBN\nconducting\tVBG\nconducts\tVBZ\nconduct\tVB\nconduct\tVBP\nconduits\tNNS\nConduits\tNNS\ncones\tNNS\nconferees\tNNS\nConferees\tNNS\nconferences\tNNS\nconferred\tVBD\nconferring\tVBG\nconfers\tVBZ\nconfer\tVB\nconfer\tVBP\nconfessed\tVBD\nconfessed\tVBN\nconfesses\tVBZ\nconfessing\tVBG\nconfessions\tNNS\nconfidants\tNNS\nconfided\tVBD\nconfides\tVBZ\nConfiding\tVBG\nconfigurations\tNNS\nconfined\tVBN\nconfines\tNNS\nconfirmed\tVBD\nconfirmed\tVBN\nconfirming\tVBG\nConfirming\tVBG\nconfirms\tVBZ\nconfirm\tVB\nconfirm\tVBP\nconfiscated\tVBD\nconfiscated\tVBN\nconfiscate\tVB\nconfiscating\tVBG\nconflicted\tVBN\nconflicting\tVBG\nconflicts\tNNS\nconflict\tVBP\nconforming\tVBG\nconforms\tVBZ\nconform\tVB\nconform\tVBP\nconfrontations\tNNS\nCONFRONTATIONS\tNNS\nconfronted\tVBN\nConfronted\tVBN\nconfronting\tVBG\nconfronts\tVBZ\nconfront\tVB\nconfront\tVBP\nconfused\tVBD\nconfused\tVBN\nconfuses\tVBZ\nconfuse\tVB\nconfusing\tVBG\nconfusions\tNNS\nconglomerates\tNNS\ncongratulated\tVBD\ncongratulated\tVBN\ncongratulate\tVB\ncongressmen\tNNS\nCongressmen\tNNS\nconjures\tVBZ\nconjure\tVBP\nconnected\tVBN\nconnecting\tVBG\nconnections\tNNS\nconnects\tVBZ\nconnect\tVB\nconnect\tVBP\nconnotations\tNNS\nconnote\tVB\nconquer\tVB\nconscripts\tNNS\nconsented\tVBD\nconsented\tVBN\nconsenting\tVBG\nconsents\tNNS\nconsent\tVB\nconsequences\tNNS\nconservationists\tNNS\nConservationists\tNNS\nconservatives\tNNS\nConservatives\tNNS\nCONSERVATIVES\tNNS\nconserve\tVB\nconsiderations\tNNS\nconsidered\tVBD\nconsidered\tVBN\nConsidered\tVBN\nconsidering\tVBG\nConsidering\tVBG\nconsiders\tVBZ\nconsider\tVB\nConsider\tVB\nconsider\tVBP\nconsigned\tVBN\nconsigns\tVBZ\nconsisted\tVBD\nconsisted\tVBN\nconsisting\tVBG\nConsisting\tVBG\nconsists\tVBZ\nconsist\tVB\nconsist\tVBP\ncons\tNNS\nconsoles\tVBZ\nconsole\tVB\nconsolidated\tVBD\nconsolidated\tVBN\nconsolidates\tVBZ\nconsolidate\tVB\nconsolidating\tVBG\nconsolidations\tNNS\nconsonants\tNNS\nconsorting\tVBG\nconspiracies\tNNS\nconspirators\tNNS\nconspired\tVBD\nconspire\tVBP\nconspiring\tVBG\nconstituencies\tNNS\nconstituents\tNNS\nconstituted\tVBD\nconstitutes\tVBZ\nconstitute\tVB\nconstitute\tVBP\nconstrained\tVBN\nconstrains\tVBZ\nconstraints\tNNS\nconstrain\tVB\nconstricting\tVBG\nconstrictors\tNNS\nconstructed\tVBN\nconstructing\tVBG\nconstructions\tNNS\nconstruct\tVB\nconstrued\tVBD\nconstrued\tVBN\nconstrue\tVB\nconstrue\tVBP\nconsultants\tNNS\nConsultants\tNNS\nconsultations\tNNS\nconsulted\tVBN\nconsulting\tVBG\nconsult\tVB\nconsumed\tVBD\nconsumed\tVBN\nconsumer-electronics\tNNS\nconsumer-goods\tNNS\nconsumer-products\tNNS\nconsumers\tNNS\nConsumers\tNNS\nconsume\tVB\nconsume\tVBP\nconsummated\tVBN\ncontacted\tVBD\ncontacted\tVBN\ncontacting\tVBG\ncontacts\tNNS\ncontact\tVB\ncontained\tVBD\ncontained\tVBN\ncontainers\tNNS\ncontaining\tVBG\ncontains\tVBZ\ncontain\tVB\ncontain\tVBP\ncontaminated\tVBN\ncont'd.\tVBN\ncontemplated\tVBD\ncontemplated\tVBN\ncontemplates\tVBZ\ncontemplate\tVB\ncontemplate\tVBP\ncontemplating\tVBG\ncontemporaries\tNNS\ncontemporize\tVB\ncontended\tVBD\ncontended\tVBN\ncontenders\tNNS\ncontending\tVBG\ncontends\tVBZ\ncontend\tVB\ncontend\tVBP\ncontentions\tNNS\ncontents\tNNS\ncontent\tVB\ncontestants\tNNS\ncontested\tVBD\ncontested\tVBN\ncontesting\tVBG\ncontests\tNNS\ncontest\tVB\nContinentals\tNNS\ncontingencies\tNNS\ncontinued\tVBD\ncontinued\tVBN\nContinued\tVBN\ncontinues\tVBZ\ncontinue\tVB\ncontinue\tVBP\ncontinuing\tVBG\nContinuing\tVBG\ncontraceptives\tNNS\ncontract-drilling\tVBG\ncontracted\tVBD\ncontracted\tVBN\ncontracting\tVBG\nContracting\tVBG\ncontractions\tNNS\ncontractors\tNNS\nContractors\tNNS\ncontracts\tNNS\nContracts\tNNS\ncontracts\tVBZ\ncontract\tVB\ncontract\tVBP\ncontradicting\tVBG\ncontradictions\tNNS\ncontradicts\tVBZ\ncontradict\tVB\nContras\tNNS\ncontrasted\tVBD\ncontrasted\tVBN\ncontrasting\tVBG\nContrasts\tNNS\ncontrasts\tVBZ\ncontrast\tVB\ncontrast\tVBP\ncontravened\tVBN\ncontribued\tVBD\ncontributed\tVBD\ncontributed\tVBN\ncontributes\tVBZ\ncontribute\tVB\ncontribute\tVBP\ncontributing\tVBG\nContributing\tVBG\ncontributions\tNNS\ncontributors\tNNS\ncontrolled\tVBD\ncontrolled\tVBN\ncontrollers\tNNS\ncontrolling\tVBG\ncontrols\tNNS\ncontrols\tVBZ\ncontrol\tVB\ncontrol\tVBP\ncontroversies\tNNS\ncon\tVB\nconvenants\tNNS\nconvened\tVBD\nconvened\tVBN\nconvenes\tVBZ\nconvene\tVB\nconvene\tVBP\nconvening\tVBG\nconventional-arms\tNNS\nconventioners\tNNS\nconvention-goers\tNNS\nconventions\tNNS\nconverged\tVBD\nconversations\tNNS\nconversions\tNNS\nconverted\tVBD\nconverted\tVBN\nconverters\tNNS\nconvertibles\tNNS\nconverting\tVBG\nconverts\tNNS\nconverts\tVBZ\nconvert\tVB\nconvert\tVBP\nconveyed\tVBD\nconveys\tVBZ\nconvey\tVB\nconvey\tVBP\nconvicted\tVBD\nconvicted\tVBN\nconvictions\tNNS\nCONVICTS\tVBZ\nconvict\tVB\nconvinced\tVBD\nconvinced\tVBN\nconvinced\tVBN|JJ\nconvinces\tVBZ\nconvince\tVB\nconvince\tVBP\nconvincing\tVBG\nconvoluted\tVBN\nconvolutions\tNNS\nconvoys\tNNS\nconvulsions\tNNS\ncookbooks\tNNS\ncooked\tVBN\ncookies\tNNS\ncooking\tVBG\ncooks\tNNS\ncook\tVB\ncoolants\tNNS\ncooled\tVBD\ncooled\tVBN\ncooling\tVBG\ncools\tVBZ\ncool\tVB\nCool\tVB\ncool\tVBP\ncooperated\tVBD\ncooperated\tVBN\ncooperate\tVB\ncooperate\tVBP\ncooperating\tVBG\ncooperatives\tNNS\ncoordinated\tVBN\ncoordinates\tVBZ\ncoordinate\tVB\ncoordinating\tVBG\nCoors\tNNS\nco-payments\tNNS\ncope\tVB\ncopied\tVBD\ncopied\tVBN\ncopiers\tNNS\ncopies\tNNS\nco-pilot\tNNS\ncoping\tVBG\nco-presidents\tNNS\nco-produce\tVB\ncopycats\tNNS\ncopying\tVBG\ncopyrighted\tVBN\ncopyrights\tNNS\ncopy\tVB\ncopy\tVBP\nCopy\tVBP\ncores\tNNS\ncorkscrews\tNNS\ncornered\tVBN\ncorners\tNNS\ncornerstones\tNNS\ncorner\tVB\ncornices\tNNS\ncorporate-earnings\tNNS\ncorporates\tNNS\ncorporations\tNNS\nCorporations\tNNS\ncorral\tVB\ncorrected\tVBD\ncorrected\tVBN\ncorrecting\tVBG\ncorrections\tNNS\ncorrects\tVBZ\ncorrect\tVB\ncorrect\tVBP\ncorrelate\tVBP\ncorresponded\tVBD\ncorrespondents\tNNS\nCorresponding\tVBG\ncorrespond\tVB\ncorridors\tNNS\ncorroborate\tVBP\nCorvettes\tNNS\ncosmetics\tNNS\nCosmetics\tNNS\ncosmologies\tNNS\nco-sponsored\tVBN\nco-sponsors\tNNS\nco-sponsor\tVB\nCosta\tNNS\ncosting\tVBG\ncosts\tNNS\nCosts\tNNS\ncosts\tVBZ\ncostumed\tVBN\ncostumes\tNNS\ncost\tVB\ncost\tVBD\ncost\tVBN\ncost\tVBP\ncots\tNNS\ncottages\tNNS\ncouched\tVBN\ncouching\tVBG\ncoughed\tVBD\ncoughed\tVBN\ncoughing\tVBG\ncoughs\tNNS\ncough\tVB\ncouncilors\tNNS\ncouncils\tNNS\ncounseled\tVBN\ncounseling\tVBG\ncounselors\tNNS\ncounsels\tVBZ\ncounsel\tVB\ncounted\tVBD\ncounted\tVBN\ncountenance\tVB\ncounteracted\tVBN\ncounteract\tVB\ncounterattack\tVB\ncounterbalanced\tVBN\ncounterbidders\tNNS\ncounterbids\tNNS\ncounter-claims\tNNS\ncounterclaims\tNNS\ncountered\tVBD\nCountered\tVBD\ncountered\tVBN\ncountering\tVBG\ncountermeasures\tNNS\ncounterparts\tNNS\ncounterprogram\tVB\ncounters\tNNS\ncountersued\tVBD\ncountersued\tVBN\ncountersuing\tVBG\ncountervailing\tVBG\ncounter\tVB\ncounter\tVBP\ncounties\tNNS\ncounting\tVBG\nCounting\tVBG\ncountries\tNNS\nCountries\tNNS\ncountrymen\tNNS\ncounts\tNNS\ncounts\tVBZ\ncount\tVB\ncount\tVBP\nCount\tVBP\ncoupled\tVBN\nCoupled\tVBN\ncouples\tNNS\ncouplets\tNNS\ncoup-makers\tNNS\ncoupons\tNNS\nCoupons\tNNS\ncoups\tNNS\ncouriers\tNNS\ncoursed\tVBN\ncourses\tNNS\ncourted\tVBD\ncourted\tVBN\ncourtesies\tNNS\ncourthouses\tNNS\ncourting\tVBG\ncourtrooms\tNNS\ncourts\tNNS\nCourts\tNNS\nCOURTS\tNNS\ncourts\tVBZ\ncourt\tVB\ncousins\tNNS\ncovenants\tNNS\ncoverages\tNNS\ncovered\tVBD\ncovered\tVBN\ncoverings\tNNS\ncovering\tVBG\ncovers\tNNS\ncovers\tVBZ\ncoverts\tNNS\ncover\tVB\ncover\tVBP\ncoveted\tVBN\ncovets\tVBZ\ncowards\tNNS\ncowboys\tNNS\nCowboys\tNNS\ncower\tVBP\nco-workers\tNNS\ncows\tNNS\nCPAs\tNNS\ncrabs\tNNS\ncracked\tVBD\ncracked\tVBN\ncracking\tVBG\nCracking\tVBG\ncrackle\tVBP\ncracks\tNNS\ncrack\tVB\ncrack\tVBP\ncrafted\tVBD\ncrafted\tVBN\ncrafting\tVBG\ncraftsmen\tNNS\ncraft\tVB\ncrammed\tVBD\ncramming\tVBG\ncramps\tNNS\ncrams\tVBZ\ncranes\tNNS\ncraning\tVBG\ncranked\tVBD\ncranked\tVBN\ncranks\tNNS\ncrank\tVB\ncrashed\tVBD\nCRASHED\tVBD\ncrashes\tNNS\ncrashes\tVBZ\ncrashing\tVBG\nCrashing\tVBG\ncrash\tVB\ncratering\tVBG\ncrates\tNNS\ncrave\tVBP\ncrawled\tVBD\ncrawling\tVBG\ncrawls\tVBZ\ncrawl\tVB\ncreak\tVB\ncreamed\tVBN\ncreams\tNNS\ncreated\tVBD\ncreated\tVBN\nCreated\tVBN\ncreates\tVBZ\nCreates\tVBZ\ncreate\tVB\nCreate\tVB\ncreate\tVBP\ncreating\tVBG\ncreations\tNNS\ncreators\tNNS\nCreators\tNNS\ncreatures\tNNS\ncredentials\tNNS\ncredit-data\tNN|NNS\ncredited\tVBD\ncredited\tVBN\ncrediting\tVBG\ncreditors\tNNS\nCreditors\tNNS\ncredit-ratings\tNNS\ncredits\tNNS\ncredits\tVBZ\ncredit\tVB\ncredit\tVBP\ncreeping\tVBG\ncrematoriums\tNNS\ncrept\tVBD\ncrept\tVBN\ncrest\tVB\ncrevasses\tNNS\ncrevices\tNNS\ncrews\tNNS\ncries\tVBZ\ncrimes\tNNS\ncriminalize\tVB\ncriminals\tNNS\ncrimping\tVBG\ncrimp\tVB\ncringed\tVBD\ncrippled\tVBN\ncripples\tNNS\ncrippling\tVBG\ncrises\tNNS\nCrises\tNNS\ncrisscrossing\tVBG\ncriss-cross\tVBP\ncriteria\tNNS\ncriticisms\tNNS\ncriticized\tVBD\ncriticized\tVBN\ncriticizes\tVBZ\ncriticize\tVB\ncriticize\tVBP\ncriticizing\tVBG\ncritics\tNNS\nCritics\tNNS\ncroak\tVBP\ncroissants\tNNS\ncronies\tNNS\ncrooks\tNNS\ncrooned\tVBD\ncroons\tVBZ\ncropped\tVBD\ncropped\tVBN\ncropping\tVBG\ncrops\tNNS\ncrop\tVB\nCROSS-BRED\tVBD\ncrossed\tVBD\ncrossed\tVBN\ncrosses\tVBZ\ncrossing\tVBG\ncross-pollinated\tVBN\ncrossroads\tNNS\ncross-shareholdings\tNNS\ncross\tVB\ncross\tVBP\ncrouched\tVBD\nCrouched\tVBN\ncrowded\tVBD\ncrowded\tVBN\ncrowds\tNNS\ncrowds\tVBZ\ncrowd\tVBP\ncrowed\tVBN\ncrowned\tVBN\ncrows\tVBZ\nCRs\tNNS\ncrudes\tNNS\ncruisers\tNNS\ncruise\tVBP\ncruising\tVBG\ncrumbled\tVBD\ncrumbled\tVBN\ncrumbles\tVBZ\ncrumble\tVB\ncrumbling\tVBG\ncrumpled\tVBD\ncrumpled\tVBN\ncrunched\tVBD\nCrunch\tVB\ncrushed\tVBN\ncrushes\tVBZ\ncrushing\tVBG\ncrush\tVB\ncrying\tVBG\ncryptographers\tNNS\ncrystals\tNNS\ncry\tVB\ncry\tVBP\ncubs\tNNS\ncuckoos\tNNS\ncues\tNNS\ncuff\tVB\nculled\tVBN\nculminated\tVBD\nculminated\tVBN\nculminates\tVBZ\nculminating\tVBG\nculprits\tNNS\ncultivated\tVBN\ncultivates\tVBZ\ncultivating\tVBG\ncultures\tNNS\ncups\tNNS\ncurators\tNNS\ncurbed\tVBD\ncurbed\tVBN\ncurbing\tVBG\nCURBING\tVBG\ncurbs\tNNS\ncurb\tVB\nCurb\tVB\ncured\tVBN\ncures\tNNS\ncure\tVB\ncuring\tVBG\ncurled\tVBD\ncurl\tVB\ncurrencies\tNNS\ncurrents\tNNS\ncurry\tVB\ncursed\tVBD\ncurses\tNNS\ncurtailed\tVBD\ncurtailed\tVBN\ncurtailing\tVBG\ncurtail\tVB\ncurtail\tVBP\ncurtains\tNNS\ncushioned\tVBN\ncushioning\tVBG\ncushion\tVB\ncustomers\tNNS\nCustomers\tNNS\ncustomized\tVBN\ncustoms\tNNS\nCustoms\tNNS\ncutbacks\tNNS\ncutouts\tNNS\ncuts\tNNS\ncuts\tVBZ\ncutters\tNNS\ncutting-tools\tNNS\ncutting\tVBG\nCutting\tVBG\ncutting\tVBG|NN\ncut\tVB\nCut\tVB\ncut\tVBD\ncut\tVBN\ncut\tVBP\ncuvees\tNNS\nCVB\tNNP\ncycads\tNNS\nCycads\tNNS\ncycles\tNNS\ncyclicals\tNNS\ncycling\tVBG\ncyclists\tNNS\nc-Yields\tNNS\nczars\tNNS\ndabbled\tVBD\ndabble\tVB\ndabbling\tVBG\ndabs\tVBZ\ndailies\tNNS\ndalliances\tNNS\ndamaged\tVBD\ndamaged\tVBN\ndamages\tNNS\ndamages\tVBZ\ndamage\tVB\ndamage\tVBP\ndamaging\tVBG\ndamped\tVBN\ndamped\tVBP\ndampened\tVBD\ndampen\tVB\ndamping\tVBG\ndamp\tVB\ndamp\tVBP\ndams\tNNS\ndancers\tNNS\ndances\tNNS\ndance\tVB\ndancing\tVBG\ndangers\tNNS\nDangers\tNNS\ndangled\tVBN\ndangling\tVBG\ndared\tVBD\ndare\tVB\ndare\tVBP\ndarlings\tNNS\ndashed\tVBD\ndashed\tVBN\ndashes\tNNS\ndash\tVB\ndatabases\tNNS\ndata\tNN|NNS\ndata\tNNS\nData\tNNS\ndata\tNNS|NN\ndated\tVBN\ndates\tNNS\ndates\tVBZ\ndate\tVB\ndate\tVBP\ndating\tVBG\ndaughters\tNNS\ndaunted\tVBD\ndaunting\tVBG\ndawdling\tVBG\ndawning\tVBG\ndawns\tVBZ\ndays.\tNNS\ndays\tNNS\nDays\tNNS\ndazzled\tVBN\ndeactivates\tVBZ\ndeadbeats\tNNS\ndeadlines\tNNS\ndeadlocked\tVBN\ndead\tVBN\nDeafening\tVBG\ndealerships\tNNS\ndealers\tNNS\nDealers\tNNS\ndealings\tNNS\ndealing\tVBG\nDealing\tVBG\ndealmakers\tNNS\ndeals\tNNS\ndeals\tVBZ\ndealt\tVBD\ndealt\tVBN\ndeal\tVB\ndeal\tVBP\ndeaths\tNNS\ndebacles\tNNS\ndebated\tVBN\ndebates\tNNS\nDebates\tNNS\ndebate\tVB\ndebating\tVBG\ndebentures\tNNS\ndebtholders\tNNS\ndebtors\tNNS\ndebts\tNNS\ndebunk\tVB\ndebuted\tVBD\ndebut\tVB\ndecades\tNNS\nDecades\tNNS\ndecapitalize\tVBP\ndeceased\tVBN\ndeceived\tVBD\ndeceived\tVBN\ndeceive\tVB\ndecelerated\tVBN\ndecelerating\tVBG\ndecentralized\tVBN\ndecentralizing\tVBG\ndecertified\tVBN\ndecided\tVBD\ndecided\tVBN\ndecides\tVBZ\ndecide\tVB\nDecide\tVB\ndecide\tVBP\ndeciding\tVBG\nDeciding\tVBG\ndecimated\tVBN\ndecision-makers\tNNS\ndecisions\tNNS\nDecisions\tNNS\ndecked\tVBN\ndeckhands\tNNS\ndecking\tVBG\ndecks\tNNS\ndeclarations\tNNS\ndeclared\tVBD\ndeclared\tVBN\ndeclares\tVBZ\ndeclare\tVB\ndeclare\tVBP\ndeclaring\tVBG\nDeclaring\tVBG\ndeclassifying\tVBG\ndeclined\tVBD\ndeclined\tVBN\ndecliners\tNNS\nDecliners\tNNS\ndeclines\tNNS\nDeclines\tNNS\ndeclines\tVBZ\ndecline\tVB\ndecline\tVBP\ndeclining\tVBG\nDeclining\tVBG\ndecontaminated\tVBN\ndecorated\tVBN\nDecorated\tVBN\ndecorators\tNNS\ndecoys\tNNS\ndecreased\tVBD\ndecreased\tVBN\ndecreases\tNNS\ndecreases\tVBZ\ndecrease\tVB\ndecrease\tVBP\ndecreasing\tVBG\ndecreed\tVBN\ndecribed\tVBD\ndecried\tVBD\ndecries\tVBZ\ndecrying\tVBG\nDec.\tVB\ndedicated\tVBD\ndedicated\tVBN\ndeduces\tVBZ\ndeducted\tVBN\ndeductibles\tNNS\ndeducting\tVBG\ndeductions\tNNS\nDeductions\tNNS\ndeduct\tVB\ndeeds\tNNS\ndeemed\tVBD\ndeemed\tVBN\nde-emphasized\tVBN\nde-emphasize\tVB\ndeems\tVBZ\ndeem\tVBP\ndeepened\tVBD\ndeepening\tVBG\ndeer\tNNS\nDeer\tNNS\ndefaulted\tVBD\ndefaulted\tVBN\ndefaulters\tNNS\ndefaulting\tVBG\ndefaults\tNNS\nDefaults\tNNS\ndefault\tVB\ndefeated\tVBD\ndefeated\tVBN\ndefeating\tVBG\ndefeats\tNNS\ndefeats\tVBZ\ndefeat\tVB\ndefected\tVBD\ndefecting\tVBG\ndefections\tNNS\nDefections\tNNS\ndefects\tNNS\ndefect\tVB\ndefendants\tNNS\nDefendants\tNNS\ndefended\tVBD\ndefended\tVBN\ndefenders\tNNS\nDefenders\tNNS\ndefending\tVBG\nDefending\tVBG\ndefends\tVBZ\ndefend\tVB\ndefend\tVBP\ndefense-electronics\tNNS\ndefenses\tNNS\ndeferred\tVBN\ndeferring\tVBG\ndefer\tVB\ndeficiencies\tNNS\ndeficits\tNNS\ndefied\tVBD\ndefies\tVBZ\ndefined\tVBD\ndefined\tVBN\ndefines\tVBZ\ndefine\tVB\ndefining\tVBG\nDefining\tVBG\ndefinitions\tNNS\ndeflated\tVBD\ndeflated\tVBN\ndeflate\tVB\ndeflators\tNNS\ndeflected\tVBD\ndeflecting\tVBG\ndeflect\tVB\ndeformed\tVBN\ndefrauded\tVBD\ndefrauded\tVBN\ndefrauding\tVBG\ndefraud\tVB\ndefunct\tVB\ndefuse\tVB\nDefuse\tVB\ndefying\tVBG\ndefy\tVBP\ndegenerated\tVBD\ndegenerated\tVBN\ndegenerate\tVB\ndegrees\tNNS\ndelayed\tVBD\ndelayed\tVBN\ndelaying\tNN|VBG\ndelaying\tVBG\ndelays\tNNS\ndelays\tVBZ\nDELAYS\tVBZ\ndelay\tVB\ndelay\tVBP\ndelegates\tNNS\nDelegates\tNNS\ndelegate\tVB\ndelegating\tVBG\ndeleted\tVBN\ndelete\tVB\ndeleting\tVBG\ndeletions\tNNS\ndeliberate\tVB\ndeliberating\tVBG\ndeliberations\tNNS\ndelighted\tVBD\ndelighted\tVBN\ndelights\tVBZ\ndelight\tVB\nDELIGHT\tVBP\ndeli\tNNS\ndelinquencies\tNNS\ndelinquents\tNNS\ndelisted\tVBN\ndelivered\tVBD\ndelivered\tVBN\ndeliveries\tNNS\nDeliveries\tNNS\ndelivering\tVBG\ndelivers\tVBZ\ndeliver\tVB\nDeliver\tVB\ndeliver\tVBP\ndelousing\tVBG\ndeluged\tVBD\ndeluged\tVBN\ndelved\tVBN\ndelves\tVBZ\ndemagogues\tNNS\ndemanded\tVBD\ndemanded\tVBN\ndemanding\tVBG\ndemands\tNNS\ndemands\tVBZ\ndemand\tVB\ndemand\tVBP\ndemeaned\tVBN\ndemeanors\tNNS\ndemilitarize\tVB\ndemobilize\tVB\ndemobilizing\tVBG\ndemocracies\tNNS\ndemocratized\tVBN\ndemocratize\tVB\nDemocrats\tNNS\ndemographics\tNNS\nDemographics\tNNS\ndemolished\tVBD\ndemolished\tVBN\ndemolishing\tVBG\ndemolish\tVB\ndemonized\tVBN\ndemons\tNNS\ndemonstrated\tVBD\ndemonstrated\tVBN\ndemonstrates\tVBZ\ndemonstrate\tVB\ndemonstrate\tVBP\ndemonstrating\tVBG\ndemonstrations\tNNS\nDemonstrations\tNNS\ndemonstrators\tNNS\ndemoted\tVBN\ndemurs\tVBZ\ndenationalized\tVBN\ndenials\tNNS\ndenied\tVBD\ndenied\tVBN\nDenied\tVBN\ndenies\tVBZ\ndenims\tNNS\nDenizens\tNNS\ndenominated\tVBN\ndenominations\tNNS\ndenounced\tVBD\ndenounced\tVBN\ndenounce\tVB\ndenounce\tVBP\ndenouncing\tVBG\ndentists\tNNS\ndents\tNNS\ndenuclearized\tVBN\ndenude\tVB\ndenying\tVBG\ndeny\tVB\ndeny\tVBP\ndeparted\tVBD\ndeparting\tVBG\nDeparting\tVBG\ndepartments\tNNS\ndepartures\tNNS\ndepart\tVB\ndepart\tVBP\ndepended\tVBD\ndepended\tVBN\ndependents\tNNS\ndepending\tVBG\nDepending\tVBG\ndepends\tVBZ\ndepend\tVB\ndepend\tVBP\ndepicted\tVBN\ndepicting\tVBG\nDepicting\tVBG\ndepicts\tVBZ\ndepict\tVB\ndepict\tVBP\ndepleted\tVBD\ndepleted\tVBN\ndepletes\tVBZ\ndeplete\tVB\ndeplores\tVBZ\ndeploring\tVBG\ndeployed\tVBD\ndeployed\tVBN\ndeported\tVBD\ndeposed\tVBN\ndeposited\tVBD\ndeposited\tVBN\ndepositing\tVBG\ndepositions\tNNS\ndepositors\tNNS\ndeposits\tNNS\nDeposits\tNNS\ndeposits\tVBZ\ndeposit\tVB\ndepots\tNNS\ndepreciated\tVBD\ndepredations\tNNS\ndepressed\tVBD\ndepressed\tVBN\ndepresses\tVBZ\ndepressing\tVBG\ndepressions\tNNS\ndepress\tVB\ndepress\tVBP\ndeprivations\tNNS\ndeprived\tVBD\ndeprived\tVBN\ndeprived\tVBN|JJ\ndeprives\tVBZ\ndeprive\tVB\ndepriving\tVBG\ndeprogrammings\tNNS\ndepths\tNNS\ndeputies\tNNS\nDeputies\tNNS\nderailed\tVBN\nderailing\tVBG\nderail\tVB\nderegulated\tVBN\nderegulate\tVB\nderided\tVBD\nderided\tVBN\nderivatives\tNNS\nderived\tVBD\nderived\tVBN\nderives\tVBZ\nderive\tVB\nderiving\tVBG\nDescendants\tNNS\ndescended\tVBD\ndescending\tVBG\ndescends\tVBZ\ndescents\tNNS\ndescribed\tVBD\ndescribed\tVBN\ndescribes\tVBZ\ndescribe\tVB\ndescribe\tVBP\ndescribing\tVBG\nDescribing\tVBG\ndescriptions\tNNS\ndeserted\tVBN\ndeserts\tNNS\ndesert\tVB\ndeserved\tVBD\ndeserves\tVBZ\ndeserve\tVB\ndeserve\tVBP\ndesignated\tVBN\ndesignates\tVBZ\ndesignate\tVB\ndesignating\tVBG\ndesignations\tNNS\ndesigned\tVBD\ndesigned\tVBN\ndesignees\tNNS\ndesigners\tNNS\ndesigning\tVBG\ndesigns\tNNS\nDesigns\tNNS\ndesigns\tVBZ\ndesign\tVB\ndesired\tVBD\ndesired\tVBN\ndesires\tNNS\ndesire\tVBP\ndesist\tVB\ndesks\tNNS\ndespairs\tVBZ\ndespised\tVBN\ndespise\tVB\ndespots\tNNS\ndestabilize\tVB\ndestabilizing\tVBG\ndestinations\tNNS\nDestinations\tNNS\ndestined\tVBN\ndestroyed\tVBD\ndestroyed\tVBN\ndestroying\tVBG\ndestroys\tVBZ\ndestroy\tVB\ndestroy\tVBP\ndetached\tVBN\nDetached\tVBN\ndetailed\tVBN\ndetailing\tVBG\ndetails\tNNS\nDetails\tNNS\ndetained\tVBD\ndetained\tVBN\ndetaining\tVBG\ndetected\tVBD\ndetected\tVBN\ndetecting\tVBG\ndetectives\tNNS\ndetectors\tNNS\ndetect\tVB\ndetect\tVBP\ndetergents\tNNS\ndeteriorated\tVBD\ndeteriorated\tVBN\ndeteriorates\tVBZ\ndeteriorate\tVB\ndeteriorate\tVBP\ndeteriorating\tVBG\ndetermined\tVBD\ndetermined\tVBN\ndetermines\tVBZ\ndetermine\tVB\ndetermine\tVBP\ndetermining\tVBG\nDetermining\tVBG\ndeterred\tVBD\ndeterrents\tNNS\ndeterring\tVBG\ndeters\tVBZ\ndeter\tVB\ndeter\tVBP\ndetests\tVBZ\ndethroned\tVBN\ndetractors\tNNS\ndetracts\tVBZ\ndetract\tVB\ndevaluations\tNNS\ndevalued\tVBD\ndevalued\tVBN\ndevastated\tVBD\ndevastated\tVBN\ndevastating\tVBG\ndeveloped\tVBD\ndeveloped\tVBN\nDeveloped\tVBN\ndevelopers\tNNS\nDevelopers\tNNS\ndeveloping\tVBG\nDeveloping\tVBG\ndevelopments\tNNS\nDevelopments\tNNS\ndevelops\tVBZ\ndevelop\tVB\nDevelop\tVB\ndevelop\tVBP\ndeviated\tVBD\ndeviations\tNNS\ndevices\tNNS\ndevils\tNNS\ndevised\tVBD\ndevised\tVBN\ndevises\tVBZ\ndevise\tVB\ndevise\tVBP\ndevoted\tVBD\ndevoted\tVBN\ndevotees\tNNS\nDevotees\tNNS\ndevotes\tVBZ\ndevote\tVB\ndevote\tVBP\ndevoured\tVBN\ndevouring\tVBG\ndewatering\tVBG\ndiabetics\tNNS\ndiagnosed\tVBD\ndiagnosed\tVBN\ndiagnosing\tVBG\ndiagnostics\tNNS\ndiagramming\tVBG\ndialects\tNNS\ndialing\tVBG\nDIALING\tVBG\ndials\tNNS\ndial\tVB\ndiamonds\tNNS\ndiapers\tNNS\ndiaries\tNNS\ndibenzofurans\tNNS\ndice\tNNS\ndickered\tVBD\ndictated\tVBD\ndictated\tVBN\nDictates\tNNS\ndictates\tVBZ\ndictate\tVB\ndictate\tVBP\ndictatorships\tNNS\ndictators\tNNS\ndid\tVBD\nDid\tVBD\ndied\tVBD\nDied\tVBD\nDIED\tVBD\ndied\tVBN\nDied\tVBN\ndie-hards\tNNS\nDiehards\tNNS\ndiesels\tNNS\ndies\tVBZ\nDies\tVBZ\ndie\tVB\nDie\tVB\ndie\tVBP\nDie\tVBP\ndiffered\tVBD\ndifferences\tNNS\nDifferences\tNNS\ndifferentials\tNNS\ndifferentiate\tVB\ndifferentiating\tVBG\ndiffering\tVBG\ndiffers\tVBZ\ndiffer\tVB\ndiffer\tVBP\ndifficulties\tNNS\ndigested\tVBN\ndigesting\tVBG\ndigest\tVB\ndigging\tVBG\ndigits\tNNS\ndignify\tVB\ndignitaries\tNNS\ndigs\tNNS\nDIGS\tNNS\ndigs\tVBZ\ndig\tVB\ndiluted\tVBD\ndiluted\tVBN\ndiluted\tVBN|JJ\ndilute\tVB\ndiluting\tVBG\ndimensions\tNNS\ndimes\tNNS\ndiminished\tVBD\ndiminished\tVBN\ndiminishes\tVBZ\ndiminishing\tVBG\ndiminish\tVB\ndiminish\tVBP\ndimming\tVBG\ndined\tVBD\ndined\tVBN\ndiners\tNNS\ndining\tVBG\ndinners\tNNS\ndinosaurs\tNNS\ndioxins\tNNS\ndiplomats\tNNS\ndipped\tVBD\ndipped\tVBN\ndipping\tVBG\ndips\tNNS\ndip\tVB\nDip\tVB\ndirected\tVBD\ndirected\tVBN\ndirecting\tVBG\ndirections\tNNS\ndirectives\tNNS\ndirectories\tNNS\ndirectors\tNNS\nDirectors\tNNS\ndirects\tVBZ\ndirect\tVB\nDirks\tNNS\ndisabilities\tNNS\ndisabled\tVBN\ndisabled-workers\tNNS\ndisabling\tVBG\ndisadvantages\tNNS\ndisagreed\tVBD\ndisagreed\tVBN\ndisagreements\tNNS\ndisagrees\tVBZ\ndisagree\tVB\ndisagree\tVBP\ndisallowed\tVBD\ndisappeared\tVBD\ndisappeared\tVBN\ndisappears\tVBZ\ndisappear\tVB\ndisappear\tVBP\ndisappointed\tVBD\ndisappointed\tVBN\ndisappointments\tNNS\nDisappointments\tNNS\nDISAPPOINTMENTS\tNNS\ndisappoint\tVB\ndisapproved\tVBD\ndisapproved\tVBN\ndisapproves\tVBZ\ndisapprove\tVBP\ndisarming\tVBG\ndisarm\tVB\ndisassemble\tVB\ndisassociate\tVB\ndisasters\tNNS\nDisasters\tNNS\ndisavowed\tVBD\ndisbanded\tVBN\ndisbanding\tVBG\ndisband\tVB\ndisbursed\tVBN\ndisbursements\tNNS\ndiscarded\tVBD\ndiscarded\tVBN\ndiscard\tVB\ndiscerns\tVBZ\ndiscern\tVB\ndischarged\tVBN\ndischarges\tNNS\ndischarge\tVB\ndisciples\tNNS\ndisciplined\tVBD\ndisciplined\tVBN\ndisciplining\tVBG\ndisclaims\tVBZ\ndisclosed\tVBD\ndisclosed\tVBN\ndiscloses\tVBZ\ndisclose\tVB\ndisclosing\tVBG\ndisclosures\tNNS\nDisclosures\tNNS\ndiscolored\tVBN\ndiscomfit\tVB\ndisconnected\tVBN\ndisconnect\tVB\ndiscontinued\tVBN\ndiscontinue\tVB\ndiscontinuing\tVBG\ndiscos\tNNS\nDiscos\tNNS\ndiscounted\tVBD\ndiscounted\tVBN\nDiscounted\tVBN\ndiscounting\tVBG\ndiscounts\tNNS\ndiscounts\tVBZ\ndiscount\tVB\ndiscouraged\tVBD\ndiscouraged\tVBN\ndiscourages\tVBZ\ndiscourage\tVB\ndiscourage\tVBP\ndiscouraging\tVBG\ndiscovered\tVBD\ndiscovered\tVBN\ndiscoveries\tNNS\ndiscovering\tVBG\ndiscovers\tVBZ\ndiscover\tVB\ndiscover\tVBP\ndiscredited\tVBN\ndiscredit\tVB\ndiscrepancies\tNNS\nDiscrepancies\tNNS\ndiscrepencies\tNNS\ndiscriminating\tVBG\ndiscs\tNNS\ndiscussed\tVBD\ndiscussed\tVBN\ndiscussing\tVBG\nDiscussing\tVBG\ndiscussions\tNNS\ndiscuss\tVB\ndiscuss\tVBP\ndisdaining\tVBG\ndisdain\tVB\ndiseases\tNNS\ndisembark\tVBP\ndisenchanted\tVBN\ndisengage\tVB\ndisgorge\tVB\ndisgruntled\tVBN\ndisguised\tVBN\ndisguises\tNNS\ndisguise\tVB\ndisgusted\tVBN\nDisgusted\tVBN\ndishes\tNNS\ndish\tVB\ndishwashers\tNNS\ndisinclined\tVBN\ndisinfectants\tNNS\ndisintegrated\tVBD\ndisintegrating\tVBG\ndisks\tNNS\ndislikes\tVBZ\ndislike\tVB\ndislike\tVBP\ndislocations\tNNS\ndismantled\tVBN\ndismantle\tVB\nDismantle\tVB\ndismantling\tVBG\ndismayed\tVBN\ndismember\tVB\ndismissed\tVBD\ndismissed\tVBN\ndismisses\tVBZ\ndismissing\tVBG\nDismissing\tVBG\ndismiss\tVB\ndismiss\tVBP\ndisobey\tVB\ndisorders\tNNS\ndisparaged\tVBD\ndisparage\tVB\ndisparaging\tVBG\ndisparities\tNNS\ndispatched\tVBD\ndispatched\tVBN\ndispatchers\tNNS\ndispatches\tNNS\ndispatching\tVBG\ndispatch\tVB\ndispelled\tVBD\ndispel\tVB\ndispensed\tVBD\ndispense\tVB\ndispense\tVBP\ndispensing\tVBG\ndispersants\tNNS\ndispersed\tVBD\ndispersing\tVBG\ndisplaced\tVBN\ndisplace\tVB\ndisplayed\tVBD\ndisplayed\tVBN\ndisplaying\tVBG\ndisplays\tNNS\ndisplays\tVBZ\ndisplay\tVB\ndisplay\tVBP\ndispleases\tVBZ\ndisposables\tNNS\ndisposals\tNNS\ndisposed\tVBD\ndisposed\tVBN\ndisposes\tVBZ\ndispose\tVB\ndisposing\tVBG\ndispositions\tNNS\ndisprove\tVB\ndisputed\tVBD\ndisputed\tVBN\ndisputes\tNNS\ndisputes\tVBZ\ndispute\tVB\ndispute\tVBP\ndisqualified\tVBN\ndisqualify\tVB\ndisregarded\tVBD\ndisregard\tVB\ndisrupted\tVBD\ndisrupted\tVBN\ndisrupting\tVBG\ndisruptions\tNNS\ndisrupt\tVB\ndissatisfied\tVBN\ndissected\tVBN\ndissecting\tVBG\ndisseminated\tVBN\ndisseminate\tVB\ndisseminating\tVBG\ndissented\tVBD\ndissented\tVBN\ndissenters\tNNS\ndissents\tNNS\ndissidents\tNNS\nDissidents\tNNS\ndissipated\tVBN\ndissipates\tVBZ\ndissipate\tVB\ndissociate\tVB\ndissociating\tVBG\ndissolved\tVBN\ndissolves\tVBZ\ndissolve\tVB\ndissolving\tVBG\ndissuade\tVB\ndistances\tNNS\ndistance\tVB\ndistancing\tVBG\ndistilled\tVBN\ndistillers\tNNS\ndistilling\tVBG\ndistinctions\tNNS\ndistinguished\tVBD\ndistinguished\tVBN\ndistinguish\tVB\ndistorted\tVBN\ndistortions\tNNS\ndistorts\tVBZ\ndistort\tVB\ndistort\tVBP\ndistracted\tVBD\ndistracted\tVBN\ndistracting\tVBG\ndistractions\tNNS\ndistract\tVB\ndistributed\tVBD\ndistributed\tVBN\nDistributed\tVBN\ndistributes\tVBZ\ndistribute\tVB\ndistribute\tVBP\ndistributing\tVBG\nDistributing\tVBG\ndistributions\tNNS\ndistributors\tNNS\ndistricts\tNNS\ndistricts\\/states\tNNS\ndisturbances\tNNS\ndisturbed\tVBN\ndisturbing\tVBG\ndisturbs\tVBZ\ndisturb\tVB\ndithering\tVBG\ndived\tVBD\ndiverge\tVB\ndiverging\tVBG\ndiversifed\tVBN\ndiversifications\tNNS\ndiversified\tVBN\ndiversifying\tVBG\ndiversify\tVB\nDiversify\tVB\ndiversions\tNNS\ndivers\tNNS\ndiverted\tVBN\ndiverting\tVBG\ndivert\tVB\ndivert\tVBP\ndivested\tVBD\ndivesting\tVBG\nDivesting\tVBG\ndivestitures\tNNS\ndivest\tVB\ndivest\tVBP\ndives\tVBZ\ndive\tVB\ndive\tVBP\ndivided\tVBD\ndivided\tVBN\ndividends\tNNS\ndivides\tVBZ\ndivide\tVB\nDivide\tVB\ndivide\tVBP\ndividing\tVBG\ndividing\tVBG|NN|JJ\ndiving\tVBG\ndivisions\tNNS\ndivorced\tVBD\ndivorced\tVBN\ndivulge\tVB\ndivvied\tVBN\ndivvying\tVBG\ndizzying\tVBG\ndockets\tNNS\ndock-siders\tNNS\ndoctors\tNNS\nDoctors\tNNS\nDOCTORS\tNNS\ndoctor\tVB\ndoctrines\tNNS\ndocudramas\tNNS\ndocumentaries\tNNS\ndocumented\tVBN\ndocumenting\tVBG\ndocuments\tNNS\nDocuments\tNNS\ndocuments\tVBZ\ndocument\tVB\ndodged\tVBD\ndodge\tVBP\ndoes\tVBZ\nDoes\tVBZ\ndogged\tVBD\ndogged\tVBN\ndogging\tVBG\ndo-gooders\tNNS\ndogs\tNNS\nDogs\tNNS\nDOGS\tNNS\ndogs\tVBZ\ndoing\tVBG\nDoing\tVBG\ndoldrums\tNNS\ndoled\tVBD\ndole\tVB\ndole\tVBP\ndoling\tVBG\ndollars\tNNS\ndolls\tNNS\ndolphins\tNNS\ndomes\tNNS\ndominated\tVBD\ndominated\tVBN\ndominates\tVBZ\ndominate\tVB\ndominate\tVBP\ndominating\tVBG\ndonated\tVBD\ndonated\tVBN\ndonate\tVB\ndonate\tVBP\ndonating\tVBG\nDonating\tVBG\ndonations\tNNS\nDonations\tNNS\ndone\tVBN\ndonned\tVBD\nDonning\tVBG\ndonors\tNNS\ndons\tNNS\ndons\tVBZ\nDON'T\tVB\ndon\tVB\ndoomed\tVBD\ndoomed\tVBN\ndooming\tVBG\ndoomsayers\tNNS\ndoom\tVB\ndoormen\tNNS\ndoors\tNNS\ndoses\tNNS\ndossiers\tNNS\ndoted\tVBN\ndoth\tVBZ\ndotting\tVBG\ndot\tVBP\ndouble-crossed\tVBD\ndoubled\tVBD\nDoubled\tVBD\ndoubled\tVBN\ndoubles\tVBZ\ndouble\tVB\ndouble\tVBP\ndoubling\tVBG\ndoubted\tVBD\ndoubters\tNNS\ndoubts\tNNS\nDoubts\tNNS\ndoubts\tVBZ\ndoubt\tVB\ndoubt\tVBP\ndo\tVB\nDo\tVB\ndo\tVBP\nDo\tVBP\ndovetails\tVBZ\ndove\tVBD\ndowned\tVBD\ndowngraded\tVBD\ndowngraded\tVBN\nDowngraded\tVBN\ndowngrades\tNNS\ndowngrading\tVBG\ndownpayments\tNNS\ndownplayed\tVBD\ndownsized\tVBN\ndownsize\tVB\ndownsizing\tVBG\ndowns\tNNS\ndownturns\tNNS\ndown\tVBP\ndozens\tNNS\nDozens\tNNS\ndrafted\tVBD\ndrafted\tVBN\ndrafting\tVBG\ndraftsmen\tNNS\ndraft\tVB\ndragged\tVBD\ndragged\tVBN\ndragging\tVBG\nDragging\tVBG\ndrags\tVBZ\ndrag\tVB\ndrag\tVBP\ndrained\tVBN\ndraining\tVBG\ndrains\tVBZ\ndrain\tVB\ndramatizations\tNNS\nDRAMs\tNNS\ndrapes\tNNS\ndrape\tVB\ndrawbacks\tNNS\ndrawings\tNNS\ndrawing\tVBG\nDrawing\tVBG\ndrawn\tVBN\ndraws\tNNS\ndraws\tVBZ\ndraw\tVB\ndraw\tVBP\ndreaded\tVBN\ndreamed\tVBD\ndreamed\tVBN\ndreaming\tVBG\ndreams\tNNS\nDreams\tNNS\ndreams\tVBZ\ndreamt\tVBD\ndream\tVB\nDream\tVB\ndream\tVBP\ndressed\tVBD\ndressed\tVBN\ndresses\tNNS\ndresses\tVBZ\ndressing\tVBG\ndress\tVB\ndress\tVBP\ndrew\tVBD\ndried\tVBD\ndried\tVBN\ndrifted\tVBD\ndrifting\tVBG\ndrift\tVB\ndrift\tVBP\ndrilled\tVBD\ndrilled\tVBN\ndrillers\tNNS\ndrilling\tVBG\ndrills\tNNS\ndrill\tVB\ndrinking\tVBG\ndrinks\tNNS\ndrink\tVB\ndrink\tVBP\ndripping\tVBG\ndriven\tVBN\ndrivers\tNNS\ndrives\tNNS\ndrives\tVBZ\ndrive\tVB\ndrive\tVBP\ndriving\tVBG\nDriving\tVBG\ndrooled\tVBD\ndrooling\tVBG\ndroped\tVBD\ndroplets\tNNS\ndropouts\tNNS\ndropped\tVBD\ndropped\tVBN\ndroppers\tNNS\ndropping\tVBG\ndrops\tNNS\ndrops\tVBZ\ndrop\tVB\ndrop\tVBP\ndroughts\tNNS\ndroves\tNNS\ndrove\tVBD\ndrowned\tVBD\ndrowned\tVBN\ndrown\tVB\ndrug-sales\tNNS\ndrugs\tNNS\nDrugs\tNNS\ndrugstores\tNNS\ndrumming\tVBG\ndrums\tNNS\ndrum\tVB\ndrying\tVBG\ndry\tVB\nDs\tNNS\ndubbed\tVBD\ndubbed\tVBN\nDubbed\tVBN\ndubs\tVBZ\nducking\tVBG\nducklings\tNNS\nducks\tNNS\nDucks\tNNS\nducks\tVBZ\nduck\tVB\nducts\tNNS\nduds\tNNS\ndueling\tVBG\nduels\tNNS\ndue\tNNS\ndues\tNNS\nduffers\tNNS\ndug\tVBD\ndulled\tVBN\ndummies\tNNS\ndumped\tVBD\ndumped\tVBN\ndumping\tVBG\ndumps\tNNS\ndumps\tVBZ\ndump\tVB\ndump\tVBP\ndunes\tNNS\ndupes\tVBZ\nduplicated\tVBN\nduplicate\tVB\nDuplicating\tVBG\nduplications\tNNS\ndurable-goods\tNNS\ndurables\tNNS\ndusted\tVBD\ndusting\tVBG\nduties\tNNS\n'd\tVBD\ndwarfed\tVBN\ndwarfs\tVBZ\ndwarf\tVB\ndwarf\tVBP\ndwellers\tNNS\ndwellings\tNNS\ndwindled\tVBD\ndwindling\tVBG\ndyed\tVBN\ndyes\tNNS\ndying\tVBG\ndynamics\tNNS\ndynamos\tNNS\nearmarked\tVBD\nearmarked\tVBN\nearmarking\tVBG\nearmark\tVB\nearned\tVBD\nearned\tVBN\nearners\tNNS\nearnigs\tNNS\nearnings\tNNS\nEarnings\tNNS\nEARNINGS\tNNS\nearning\tVBG\nearns\tVBZ\nEarns\tVBZ\nearn\tVB\nearn\tVBP\nearrings\tNNS\nears\tNNS\nearthlings\tNNS\nearthquakes\tNNS\nearthworms\tNNS\neased\tVBD\neased\tVBN\neases\tVBZ\nease\tVB\nease\tVBP\neasing\tVBG\nEasterners\tNNS\nEast\tNNS\nEAST\tNNS\neaten\tVBN\neaters\tNNS\neating\tVBG\neats\tVBZ\neat\tVB\neat\tVBP\neavesdrop\tVB\nebbs\tVBZ\nebb\tVB\neccentrics\tNNS\nechelons\tNNS\nechoed\tVBD\nechoed\tVBN\nechoes\tNNS\nechoing\tVBG\nEchoing\tVBG\necho\tVB\necho\tVBP\neclairs\tNNS\neclipse\tVB\neclipsing\tVBG\neconomics\tNNS\neconomies\tNNS\neconomists\tNNS\nEconomists\tNNS\neconomize\tVB\nedged\tVBD\nedged\tVBN\nedges\tNNS\nedge\tVB\nedging\tVBG\neditions\tNNS\neditorials\tNNS\nEditorials\tNNS\neditors\tNNS\nedit\tVB\neducated\tVBN\neducate\tVB\neducating\tVBG\neducations\tNNS\neducators\tNNS\nEducators\tNNS\neffects\tNNS\neffects\tVBZ\neffect\tVB\nefficiencies\tNNS\nefforts\tNNS\nEfforts\tNNS\neggs\tNNS\negos\tNNS\nejected\tVBN\neked\tVBD\neke\tVB\nelaborate\tVB\nelaborating\tVBG\nElaborating\tVBG\nelapsed\tVBN\nelbows\tNNS\nelders\tNNS\nelected\tVBD\nelected\tVBN\nElected\tVBN\nelections\tNNS\nElections\tNNS\nELECTIONS\tNNS\nelectrical-products\tNNS\nelectrified\tVBN\nelectrochemicals\tNNS\nelectrodes\tNNS\nelectrogalvanized\tVBN\nelectrogalvanizing\tVBG\nelectromagnets\tNNS\nelectronic-data\tNN|NNS\nelectronics\tNNS\nelectronic-systems\tNNS\nelectro-optics\tNNS\nelect\tVB\nelect\tVBP\nelements\tNNS\nElements\tNNS\nelephants\tNNS\nelevated\tVBD\nelevates\tVBZ\nelevations\tNNS\nelevators\tNNS\neliminated\tVBD\neliminated\tVBN\neliminates\tVBZ\neliminate\tVB\nEliminate\tVB\neliminate\tVBP\neliminating\tVBG\nEliminating\tVBG\nelite\tNNS\nelites\tNNS\nelitists\tNNS\nelongate\tVB\neluded\tVBD\neluding\tVBG\nemasculate\tVB\nembargoed\tVBD\nembargoes\tNNS\nembargos\tNNS\nembarked\tVBD\nembarked\tVBN\nembarking\tVBG\nembark\tVB\nembarrassed\tVBD\nembarrassed\tVBN\nembarrassing\tVBG\nembarrass\tVB\nembassies\tNNS\nembattled\tVBN\nembedded\tVBN\nEmbedded\tVBN\nembellish\tVB\nembezzled\tVBD\nembezzling\tVBG\nemblems\tNNS\nembodied\tVBN\nembodies\tVBZ\nembody\tVBP\nemboldened\tVBN\nEmboldened\tVBN\nembraced\tVBD\nembraced\tVBN\nembraces\tVBZ\nembrace\tVB\nembracing\tVBG\nembroiled\tVBN\nemerged\tVBD\nemerged\tVBN\nemergencies\tNNS\nemerges\tVBZ\nemerge\tVB\nemerge\tVBP\nemerging\tVBG\nEmerging\tVBG\nemigrated\tVBD\nemigrate\tVB\nemigres\tNNS\nemissaries\tNNS\nemissions\tNNS\nemitted\tVBN\nemoted\tVBD\nemote\tVB\nemotions\tNNS\nempathize\tVB\nemphasized\tVBD\nemphasized\tVBN\nemphasizes\tVBZ\nemphasize\tVB\nemphasize\tVBP\nemphasizing\tVBG\nemphaticize\tVB\nempires\tNNS\nemployed\tVBN\nemployees\tNNS\nEmployees\tNNS\nEMPLOYEES\tNNS\nemployees\tVBZ\nemployers\tNNS\nEmployers\tNNS\nemploying\tVBG\nemploys\tVBZ\nemploy\tVB\nemploy\tVBP\nempowered\tVBN\nempowering\tVBG\nempowers\tVBZ\nempower\tVBP\nemptied\tVBN\nemptying\tVBG\nempty\tVB\nempty\tVBP\nemulated\tVBN\nemulate\tVB\nemulating\tVBG\nenabled\tVBD\nenabled\tVBN\nenables\tVBZ\nenable\tVB\nenable\tVBP\nenabling\tVBG\nenacted\tVBD\nenacted\tVBN\nenacting\tVBG\nenact\tVB\nencapsulate\tVB\nencasing\tVBG\nencircling\tVBG\nenclosed\tVBN\nenclosing\tVBG\nencompassed\tVBD\nencompasses\tVBZ\nencompass\tVB\nencountered\tVBD\nencountered\tVBN\nencounters\tNNS\nencounters\tVBZ\nencounter\tVB\nencouraged\tVBD\nencouraged\tVBN\nEncouraged\tVBN\nencourages\tVBZ\nencourage\tVB\nEncourage\tVB\nencourage\tVBP\nencouraging\tVBG\nEncouraging\tVBG\nencroaching\tVBG\nencrusted\tVBN\nencrypting\tVBG\nencumbered\tVBN\nendangered-species\tNNS\nendangered\tVBN\nendanger\tVB\nendeavoring\tVBG\nendeavors\tNNS\nendeavor\tVB\nended\tVBD\nENDED\tVBD\nended\tVBN\nendings\tNNS\nending\tVBG\nEnding\tVBG\nendorsed\tVBD\nendorsed\tVBN\nendorsements\tNNS\nendorsers\tNNS\nendorses\tVBZ\nendorse\tVB\nendorse\tVBP\nendorsing\tVBG\nendowed\tVBD\nendowed\tVBN\nendow\tVB\nends\tNNS\nEnds\tNNS\nends\tVBZ\nEnds\tVBZ\nend-tailed\tVBN\nendured\tVBD\nendured\tVBN\nendure\tVB\nendure\tVBP\nEndure\tVBP\nenduring\tVBG\nend\tVB\nEnd\tVB\nend\tVBP\nenemies\tNNS\nenergies\tNNS\nenergized\tVBN\nenergy-services\tNNS\nenforced\tVBN\nenforcers\tNNS\nEnforcers\tNNS\nenforces\tVBZ\nenforce\tVB\nenforce\tVBP\nenforcing\tVBG\nengaged\tVBD\nengaged\tVBN\nengagements\tNNS\nengages\tVBZ\nengage\tVB\nengage\tVBP\nengaging\tVBG\nengineered\tVBN\nengineering\tVBG\nengineers\tNNS\nEngineers\tNNS\nengineer\tVB\nengines\tNNS\nEnglish\tNNS\nengraved\tVBN\nengulfed\tVBD\nengulfed\tVBN\nenhanced\tVBD\nenhanced\tVBN\nenhancements\tNNS\nenhances\tVBZ\nenhance\tVB\nenhancing\tVBG\nenjoined\tVBD\nenjoined\tVBN\nenjoin\tVB\nenjoyed\tVBD\nenjoyed\tVBN\nenjoying\tVBG\nEnjoying\tVBG\nenjoys\tVBZ\nenjoy\tVB\nenjoy\tVBP\nenlarged\tVBN\nenlargers\tNNS\nenlarge\tVB\nenlarging\tVBG\nenlightened\tVBN\nenlightening\tVBG\nenlighten\tVB\nenlisted\tVBD\nenlisting\tVBG\nenlist\tVB\nenlivening\tVBG\nennumerated\tVBD\nenraged\tVBD\nenriching\tVBG\nenrich\tVBP\nenrolled\tVBD\nenrolled\tVBN\nenrollees\tNNS\nenrollments\tNNS\nenroll\tVB\nenroll\tVBP\nensconced\tVBN\nensembles\tNNS\nensnarled\tVBN\nensued\tVBD\nensue\tVB\nensuing\tVBG\nensures\tVBZ\nensure\tVB\nensure\tVBP\nensuring\tVBG\nentailed\tVBN\nentails\tVBZ\nentail\tVB\nentangled\tVBN\nentered\tVBD\nentered\tVBN\nentering\tVBG\nenterprises\tNNS\nenters\tVBZ\nentertained\tVBD\nentertained\tVBN\nentertainers\tNNS\nentertaining\tVBG\nentertain\tVB\nenter\tVB\nEnter\tVB\nenter\tVBP\nenthusiasms\tNNS\nenthusiasts\tNNS\nEnthusiasts\tNNS\nenticed\tVBD\nentice\tVB\nentice\tVBP\nenticing\tVBG\nentities\tNNS\nentitled\tVBD\nentitled\tVBN\nentitlements\tNNS\nentitles\tVBZ\nentitle\tVB\nentitle\tVBP\nentitling\tVBG\nentombed\tVBN\nentranced\tVBN\nentrants\tNNS\nentrenched\tVBN\nentrench\tVB\nentrepreneurs\tNNS\nEntrepreneurs\tNNS\nentries\tNNS\nentrusted\tVBN\nentrust\tVB\nentrust\tVBP\nentwined\tVBN\nenvelopes\tNNS\nenvironmentalists\tNNS\nEnvironmentalists\tNNS\nenvironments\tNNS\nenvisaged\tVBD\nenvisaged\tVBN\nenvisioned\tVBN\nenvisions\tVBZ\nenvision\tVB\nenvision\tVBP\nenvy\tVBP\neons\tNNS\nepileptics\tNNS\nepisodes\tNNS\nepitomize\tVBP\nequaled\tVBD\nequaled\tVBN\nequaling\tVBG\nequals\tVBZ\nequal\tVB\nequal\tVBP\nequated\tVBN\nequates\tVBZ\nequate\tVB\nequestrians\tNNS\nequipped\tVBD\nequipped\tVBN\nEquipped\tVBN\nequipping\tVBG\nequips\tVBZ\nequip\tVB\nequities\tNNS\nEquities\tNNS\nequivalents\tNNS\nEquivalents\tNNS\neradicate\tVB\nerased\tVBD\nerased\tVBN\nerases\tVBZ\nerase\tVB\nerasing\tVBG\nErasing\tVBG\nerasures\tNNS\nerected\tVBD\nerected\tVBN\nerect\tVB\neroded\tVBD\neroded\tVBN\nerodes\tVBZ\nErodes\tVBZ\nerode\tVB\nerode\tVBP\neroding\tVBG\nerrata\tNNS\nerred\tVBD\nerred\tVBN\nerrors\tNNS\nerrs\tVBZ\nerr\tVB\nErr\tVB\nerr\tVBP\nerupted\tVBD\nerupted\tVBN\nerupts\tVBZ\nerupt\tVB\nescalated\tVBD\nescalated\tVBN\nescalate\tVB\nescalating\tVBG\nescalators\tNNS\nescaped\tVBD\nescaped\tVBN\nescape\tVB\nescaping\tVBG\neschewed\tVBN\nEscorts\tNNS\nescorts\tVBZ\nescrowed\tVBN\nespouse\tVBP\nESPs\tNNS\nessays\tNNS\nessentials\tNNS\nestablished\tVBD\nestablished\tVBN\nEstablished\tVBN\nestablishes\tVBZ\nestablishing\tVBG\nEstablishing\tVBG\nestablishments\tNNS\nestablish\tVB\nEstablish\tVB\nestablshed\tVBN\nestimated\tVBD\nestimated\tVBN\nEstimated\tVBN\nestimates\tNNS\nEstimates\tNNS\nestimates\tVBZ\nestimate\tVB\nestimate\tVBP\nestimating\tVBG\nEstimating\tVBG\nestimators\tNNS\nestranged\tVBN\nethics\tNNS\neuphemisms\tNNS\nEurobonds\tNNS\nEurodebentures\tNNS\nEURODOLLARS\tNNPS|NNS\nEURODOLLARS\tNNS\nEuroissues\tNNS\nEuronotes\tNNS\nEuropeans\tNNS\nevacuated\tVBN\nevacuate\tVB\nevaded\tVBN\nevaders\tNNS\nevades\tVBZ\nevade\tVB\nevaluated\tVBD\nevaluated\tVBN\nevaluates\tVBZ\nevaluate\tVB\nevaluate\tVBP\nevaluating\tVBG\nEvaluating\tVBG\nevaluations\tNNS\nEvaluations\tNNS\nevangelists\tNNS\nevaporated\tVBD\nevaporated\tVBN\nevaporate\tVB\nevened\tVBN\nevenings\tNNS\nevens\tVBZ\nevents\tNNS\nEvents\tNNS\neven\tVB\nEverglades\tNNS\nevidenced\tVBN\nevil-doers\tNNS\nevinced\tVBD\nevinced\tVBN\neviscerating\tVBG\nevokes\tVBZ\nevoke\tVBP\nevoking\tVBG\nevolved\tVBD\nevolved\tVBN\nevolve\tVB\nevolving\tVBG\nexacerbated\tVBD\nexacerbated\tVBN\nexacerbates\tVBZ\nexacerbate\tVB\nexacerbating\tVBG\nexaggerated\tVBN\nexaggerate\tVB\nexaminations\tNNS\nexamined\tVBD\nexamined\tVBN\nexaminers\tNNS\nExaminers\tNNS\nexamines\tVBZ\nexamine\tVB\nEXAMINE\tVB\nexamine\tVBP\nexamining\tVBG\nexamples\tNNS\nExamples\tNNS\nexams\tNNS\nexcavated\tVBN\nexcavating\tVBG\nexcavators\tNNS\nexceeded\tVBD\nexceeded\tVBN\nexceeding\tVBG\nexceeds\tVBZ\nexceed\tVB\nexceed\tVBP\nexcel\tVB\nexcel\tVBP\nexceptions\tNNS\nexcept\tVB\nexcerpts\tNNS\nExcerpts\tNNS\nexcesses\tNNS\nexchanged\tVBD\nexchanged\tVBN\nexchanges\tNNS\nexchange\tVB\nexchange\tVBP\nexchanging\tVBG\nexcised\tVBD\nexcised\tVBN\nexcise\tVB\nexcited\tVBN\nexcite\tVB\nexclaims\tVBZ\nexcluded\tVBD\nexcluded\tVBN\nExcluded\tVBN\nExcludes\tNNS\nexcludes\tVBZ\nexclude\tVB\nexclude\tVBP\nexcluding\tVBG\nExcluding\tVBG\nexclusions\tNNS\nexcorciate\tVB\nexcoriated\tVBN\nexcursions\tNNS\nexcused\tVBN\nexcuses\tNNS\nexcuse\tVB\nexcutives\tNNS\nexecs\tNNS\nexecuted\tVBD\nExecuted\tVBD\nexecuted\tVBN\nexecutes\tVBZ\nexecute\tVB\nexecute\tVBP\nexecuting\tVBG\nexecutions\tNNS\nExecutions\tNNS\nexecutives\tNNS\nExecutives\tNNS\nEXECUTIVES\tNNS\nexecutors\tNNS\nexemplified\tVBN\nexemplifies\tVBZ\nex-employees\tNNS\nexempted\tVBN\nexempting\tVBG\nexemptions\tNNS\nexempt\tVB\nexercised\tVBD\nexercised\tVBN\nexercises\tNNS\nexercises\tVBZ\nexercise\tVB\nexercise\tVBP\nexercising\tVBG\nexerpts\tNNS\nexerted\tVBN\nexerting\tVBG\nexert\tVB\nexhaled\tVBD\nexhausted\tVBD\nexhausted\tVBN\nexhausting\tVBG\nexhaust\tVB\nexhaust\tVBP\nexhibited\tVBN\nexhibiting\tVBG\nexhibitions\tNNS\nexhibitors\tNNS\nexhibits\tNNS\nexhibits\tVBZ\nexhibit\tVB\nexhort\tVB\nexiled\tVBD\nexiled\tVBN\nexisted\tVBD\nexisted\tVBN\nexisting\tVBG\nExisting\tVBG\nexists\tVBZ\nexist\tVB\nexist\tVBP\nexited\tVBD\nexits\tNNS\nexits\tVBZ\nexit\tVB\nexonerated\tVBD\nexonerated\tVBN\nexonerating\tVBG\nexorcise\tVB\nexorcisms\tNNS\nexpanded\tVBD\nexpanded\tVBN\nexpanding\tVBG\nexpands\tVBZ\nEXPANDS\tVBZ\nexpand\tVB\nexpand\tVBP\nexpansionists\tNNS\nexpansions\tNNS\nexpectations\tNNS\nExpectations\tNNS\nexpected\tVB\nexpected\tVBD\nexpected\tVBN\nexpecting\tVBG\nExpects\tNNS\nexpects\tVBZ\nexpect\tVB\nExpect\tVB\nexpect\tVBP\nEXPECT\tVBP\nexpedients\tNNS\nexpedited\tVBN\nexpedite\tVB\nexpediting\tVBG\nexpelled\tVBD\nexpelled\tVBN\nexpel\tVB\nexpended\tVBN\nexpenditures\tNNS\nexpenditures\tVBZ\nexpenses\tNNS\nExpenses\tNNS\nexperienced\tVBD\nexperienced\tVBN\nExperienced\tVBN\nexperiences\tNNS\nexperiences\tVBZ\nexperience\tVB\nexperience\tVBP\nexperiencing\tVBG\nexperimented\tVBD\nexperimented\tVBN\nexperimenting\tVBG\nexperiments\tNNS\nExperiments\tNNS\nexperts\tNNS\nExperts\tNNS\nexpirations\tNNS\nexpired\tVBD\nexpired\tVBN\nexpires\tVBZ\nexpire\tVB\nexpire\tVBP\nexpiring\tVBG\nexplained\tVBD\nexplained\tVBN\nexplaining\tVBG\nExplaining\tVBG\nexplains\tVBZ\nExplains\tVBZ\nexplain\tVB\nexplain\tVBP\nexplanations\tNNS\nexploded\tVBD\nexploded\tVBN\nexplodes\tVBZ\nexplode\tVB\nexploding\tVBG\nexploited\tVBN\nexploiters\tNNS\nexploiting\tVBG\nexploits\tNNS\nexploits\tVBZ\nexploit\tVB\nexploit\tVBP\nexplores\tVBZ\nexplore\tVB\nexplore\tVBP\nexploring\tVBG\nexplosions\tNNS\nExplosions\tNNS\nexplosives\tNNS\nexported\tVBD\nexported\tVBN\nexporters\tNNS\nexporting\tVBG\nexports\tNNS\nExports\tNNS\nexports\tVBZ\nexport\tVB\nexport\tVBP\nexposed\tVBD\nexposed\tVBN\nexpose\tVB\nexpose\tVBP\nexposing\tVBG\nexposures\tNNS\nexpounding\tVBG\nexpressed\tVBD\nexpressed\tVBN\nexpresses\tVBZ\nexpressing\tVBG\nexpressions\tNNS\nexpress\tVB\nexpress\tVBP\nexpunged\tVBN\nexpunge\tVB\nextended\tVBD\nextended\tVBN\nextending\tVBG\nextends\tVBZ\nextend\tVB\nextend\tVBP\nextensions\tNNS\nExtensions\tNNS\nExteriors\tNNS\nextinguish\tVB\nextorted\tVBD\nextorting\tVBG\nextort\tVB\nextracted\tVBD\nextracted\tVBN\nextracting\tVBG\nextracts\tNNS\nextract\tVB\nextradited\tVBN\nextraditions\tNNS\nextrapolated\tVBN\nextras\tNNS\nextremes\tNNS\nextremists\tNNS\nextricate\tVB\nextrusions\tNNS\nexuded\tVBD\nexude\tVBP\neyeballing\tVBG\neyeballs\tNNS\neyebrows\tNNS\neyed\tVBD\neyeglasses\tNNS\neyeing\tVBG\neyes\tNNS\nEyes\tNNS\neyewitnesses\tNNS\neying\tVBG\nF16s\tNNS\nF-18s\tNNS\nF18s\tNNS\nfabricated\tVBN\nfabricate\tVB\nfabrications\tNNS\nfabricators\tNNS\nfabrics\tNNS\nfacades\tNNS\nfaced\tVBD\nfaced\tVBN\nFaced\tVBN\nfacelifts\tNNS\nfaces\tNNS\nfaces\tVBZ\nface\tVB\nface\tVBP\nfacilitate\tVB\nfacilitate\tVBP\nfacilitating\tVBG\nfacilities\tNNS\nfacings\tNNS\nfacing\tVBG\nFACING\tVBG\nfacsimiles\tNNS\nfactions\tNNS\nfactories\tNNS\nFactories\tNNS\nfactoring\tVBG\nfactors\tNNS\nfactors\tVBZ\nfactor\tVB\nfactory-jobs\tNNS\nfacts\tNNS\nfaded\tVBD\nfaded\tVBN\nfades\tNNS\nfade\tVB\nfading\tVBG\nfads\tNNS\nfailed\tVBD\nFAILED\tVBD\nfailed\tVBN\nfailings\tNNS\nfailing\tVBG\nfails\tVBZ\nFails\tVBZ\nfailures\tNNS\nFailures\tNNS\nfail\tVB\nfail\tVBP\nfainting\tVBG\nfaint\tVB\nfairs\tNNS\nfajitas\tNNS\nfaked\tVBD\nfaked\tVBN\nfaking\tVBG\nfallen\tVBN\nfalling\tVBG\nfalls\tNNS\nfalls\tVBZ\nfall\tVB\nfall\tVBP\nfalsified\tVBN\nfalsifying\tVBG\nfalsify\tVB\nfaltered\tVBD\nfaltered\tVBN\nfaltering\tVBG\nfalters\tVBZ\nfalter\tVBP\nfamed\tVBN\nfamiliarize\tVB\nfamilies\tNNS\nFamilies\tNNS\nfamines\tNNS\nfanatics\tNNS\nfancies\tVBZ\nfangs\tNNS\nfanned\tVBD\nfans\tNNS\nFans\tNNS\nfans\tVBZ\nfantasies\tNNS\nfantasize\tVB\nfantasize\tVBP\nfan\tVB\nfared\tVBD\nfared\tVBN\nfares\tNNS\nFares\tNNS\nfares\tVBZ\nfare\tVB\nfare\tVBP\nFare\tVBP\nfarmers\tNNS\nFarmers\tNNS\nFARMERS\tNNS\nfarming\tVBG\nfarms\tNNS\nfarmsteads\tNNS\nfarms\tVBZ\nfarm\tVB\nfarmwives\tNNS\nfascinated\tVBN\nfascists\tNNS\nfashioned\tVBN\nfashions\tNNS\nfashion\tVB\nfastballs\tNNS\nfastened\tVBN\nfasteners\tNNS\nfatalities\tNNS\nFatalities\tNNS\nfathers\tNNS\nFathers\tNNS\nfats\tNNS\nfattened\tVBD\nfattened\tVBN\nfatten\tVBP\nfaulted\tVBN\nfaultlines\tNNS\nfaults\tNNS\nfault\tVBP\nfauna\tNNS\nfavored\tVBD\nfavored\tVBN\nfavoring\tVBG\nfavorites\tNNS\nfavors\tNNS\nfavors\tVBZ\nfavor\tVB\nfavor\tVBP\nfawning\tVBG\nfaxed\tVBD\nfaxes\tNNS\nfeared\tVBD\nfeared\tVBN\nfearing\tVBG\nfears\tNNS\nFears\tNNS\nfears\tVBZ\nfear\tVB\nfear\tVBP\nfeasted\tVBN\nfeasts\tNNS\nfeathers\tNNS\nfeats\tNNS\nfeatured\tVBD\nfeatured\tVBN\nfeatures\tNNS\nfeatures\tVBZ\nfeature\tVB\nfeature\tVBP\nfeaturing\tVBG\nfed\tVBD\nfed\tVBN\nfeeding\tVBG\nfeedlots\tNNS\nFeedlots\tNNS\nfeeds\tVBZ\nfeed\tVB\nfeed\tVBP\nfeelers\tNNS\nfeelings\tNNS\nFeelings\tNNS\nfeeling\tVBG\nFeeling\tVBG\nfeels\tVBZ\nFeels\tVBZ\nfeel\tVB\nfeel\tVBP\nfees\tNNS\nFees\tNNS\nfeet\tNNS\nfellas\tNNS\nFELLED\tVBD\nfelled\tVBN\nfellows\tNNS\nfell\tVBD\nfell\tVBN\nfelonies\tNNS\nfelons\tNNS\nfelt\tVBD\nfelt\tVBN\nfemales\tNNS\nfeminists\tNNS\nfences\tNNS\nfended\tVBD\nfending\tVBG\nFending\tVBG\nfend\tVB\nferreting\tVBG\nferret\tVB\nferries\tNNS\nferrying\tVBG\nferry\tVB\nfertilized\tVBN\nfertilizers\tNNS\nfertilizing\tVBG\nfester\tVB\nfestivals\tNNS\nfestivities\tNNS\nfestooned\tVBN\nfestooning\tVBG\nfetched\tVBD\nfetches\tVBZ\nfetching\tVBG\nfetch\tVB\nfetch\tVBP\nfetuses\tNNS\nfeuded\tVBD\nfeuding\tVBG\nfiberglass\tNNS\nfibers\tNNS\nficials\tNNS\nfidgeting\tVBG\nfiefdoms\tNNS\nfielded\tVBD\nfielded\tVBN\nfielding\tVBG\nfields\tNNS\nfields\tVBZ\nfield\tVB\nfighter-bombers\tNNS\nfighters\tNNS\nfighting\tVBG\nfights\tNNS\nfights\tVBZ\nfight\tVB\nFight\tVB\nfight\tVBP\nfigured\tVBD\nfigured\tVBN\nfigures\tNNS\nFigures\tNNS\nfigures\tVBZ\nfigure\tVB\nfigure\tVBP\nfiguring\tVBG\nFiguring\tVBG\nfilberts\tNNS\nfilched\tVBN\nfiled\tVBD\nfiled\tVBN\nfilers\tNNS\nfiles\tNNS\nfiles\tVBZ\nfile\tVB\nfile\tVBP\nfilings\tNNS\nfiling\tVBG\nFilipinos\tNNS\nfilled\tVBD\nfilled\tVBN\nfilling\tVBG\nFilling\tVBG\nfills\tVBZ\nfill\tVB\nfill\tVBP\nfilmed\tVBD\nfilmed\tVBN\nFilmed\tVBN\nfilming\tVBG\nfilm-makers\tNNS\nfilms\tNNS\nfiltered\tVBN\nFiltered\tVBN\nfiltering\tVBG\nfilters\tNNS\nfilters\tVBZ\nfinagled\tVBN\nfinalists\tNNS\nfinalized\tVBD\nfinalized\tVBN\nfinalizing\tVBG\nfinals\tNNS\nfinanced\tVBD\nfinanced\tVBN\nfinances\tNNS\nFINANCES\tNNS\nfinances\tVBZ\nfinance\tVB\nfinancial-crimes\tNNS\nfinancial-services\tNNS\nfinanciers\tNNS\nfinancings\tNNS\nfinancing\tVBG\nfindings\tNNS\nfinding\tVBG\nFinding\tVBG\nfinds\tNNS\nfinds\tVBZ\nfind\tVB\nfind\tVBP\nFind\tVBP\nfine-arts\tNNS\nfined\tVBD\nfined\tVBN\nfines\tNNS\nfinessed\tVBN\nfine-tuning\tVB\nfine\tVB\nfingered\tVBN\nfingering\tVBG\nfingerlings\tNNS\nfingerprints\tNNS\nfingers\tNNS\nfinger\tVB\nfining\tVBG\nfinished\tVB\nfinished\tVBD\nfinished\tVBN\nfinishes\tNNS\nfinishes\tVBZ\nfinishing\tVBG\nfinish\tVB\nfinish\tVBP\nfireballs\tNNS\nfired\tVBD\nfired\tVBN\nfirefighters\tNNS\nfirehoops\tNNS\nfiremen\tNNS\nfireplaces\tNNS\nfireproofing\tVBG\nfires\tNNS\nfires\tVBZ\nFires\tVBZ\nfire\tVB\nfireworks\tNNS\nfirings\tNNS\nfiring\tVBG\nfirmed\tVBD\nfirmed\tVBN\nfirming\tVBG\nfirms\tNNS\nFirms\tNNS\nFIRMS\tNNS\nfirm\tVB\nfisheries\tNNS\nfishermen\tNNS\nfishing\tVBG\nfissures\tNNS\nfists\tNNS\nfits\tNNS\nfits\tVBZ\nfit\tVB\nfit\tVBN\nfit\tVBP\nfixed\tVBD\nfixed\tVBN\nfixes\tNNS\nfixing\tVBG\nfixtures\tNNS\nfix\tVB\nfizzes\tVBZ\nfizzled\tVBD\nfizzled\tVBN\nflabbergasted\tVBN\nflags\tNNS\nFlags\tNNS\nflamed\tVBD\nflames\tNNS\nFlanked\tVBN\nflapping\tVBG\nflaps\tNNS\nflared\tVBN\nflare\tVBP\nflaring\tVBG\nflashbacks\tNNS\nflashed\tVBD\nflashed\tVBN\nflashes\tNNS\nflashes\tVBZ\nflashing\tVBG\nflashlights\tNNS\nflash\tVB\nflash\tVBP\nflats\tNNS\nflattened\tVBD\nflattened\tVBN\nflattening\tVBG\nflatten\tVB\nflaunts\tVBZ\nflaunt\tVB\nflavors\tNNS\nflawed\tVBN\nflaws\tNNS\nFlaws\tNNS\nflay\tVB\nfleas\tNNS\nfledging\tVBG\nfled\tVBD\nfled\tVBN\nfleeced\tVBN\nfleeing\tVBG\nfleets\tNNS\nfleets\tVBZ\nflee\tVB\nfleshpots\tNNS\nflew\tVBD\nflexing\tVBG\nflickered\tVBD\nflicking\tVBG\nflied\tVBD\nfliers\tNNS\nflies\tNNS\nflies\tVBZ\nflight-attendants\tNNS\nflights\tNNS\nFlights\tNNS\nflinch\tVB\nflinging\tVBG\nflings\tNNS\nflipped\tVBD\nflipping\tVBG\nflips\tVBZ\nflip\tVB\nflirted\tVBD\nflirted\tVBN\nflirting\tVBG\nfloated\tVBD\nfloated\tVBN\nfloating\tVBG\nFloating\tVBG\nfloats\tVBZ\nfloat\tVB\nfloat\tVBP\nflocked\tVBD\nflocked\tVBN\nflocking\tVBG\nflog\tVB\nflooded\tVBD\nflooded\tVBN\nflooding\tVBG\nflood\tVB\nfloors\tNNS\nflopped\tVBD\nflopped\tVBN\nfloppies\tNNS\nflora\tNNS\nflotations\tNNS\nfloundered\tVBN\nfloundering\tVBG\nflounder\tVB\nflourished\tVBD\nflourished\tVBN\nflourishing\tVBG\nflourish\tVB\nflouting\tVBG\nflowed\tVBD\nflowers\tNNS\nFlowers\tNNS\nflowing\tVBG\nflown\tVBN\nflows\tNNS\nflows\tVBZ\nflow\tVB\nflow\tVBP\nfluctuated\tVBD\nfluctuated\tVBN\nfluctuates\tVBZ\nfluctuate\tVB\nfluctuate\tVBP\nfluctuating\tVBG\nfluctuations\tNNS\nfluids\tNNS\nflunking\tVBG\nflunk\tVBP\nfluoropolymers\tNNS\nflush\tVB\nflush\tVBP\nflying\tVBG\nfly\tVB\nFly\tVB\nfly\tVBP\nfoaming\tVBG\nfocused\tVBD\nfocused\tVBN\nfocuses\tVBZ\nfocusing\tVBG\nFocusing\tVBG\nfocus\tVB\nfocus\tVBP\nfoes\tNNS\nFOES\tNNS\nfog\tVB\nfoiled\tVBD\nfoiled\tVBN\nFoiled\tVBN\nfoiling\tVBG\nfoil\tVB\nfolded\tVBD\nfolded\tVBN\nfolders\tNNS\nfolding\tVBG\nfold\tVB\nfold\tVBP\nfolks\tNNS\nFolks\tNNS\nfollies\tNNS\nfollowed\tVBD\nfollowed\tVBN\nfollowers\tNNS\nfollowing\tVBG\nFollowing\tVBG\nfollows\tVBZ\nfollow\tVB\nFollow\tVB\nfollow\tVBP\nfomenting\tVBG\nfood-fish\tNNS\nfood-services\tNNS\nfoods\tNNS\nfoodstuffs\tNNS\nfooled\tVBN\nfooling\tVBG\nfools\tNNS\nfool\tVB\nfoothills\tNNS\nFoothills\tNNS\nfooting\tVBG\nfootnoted\tVBN\nfootnotes\tNNS\nfootsteps\tNNS\nfoot\tVB\nfoot\tVBP\nforays\tNNS\nforbade\tVBD\nforbidden\tVBN\nforbidding\tVBG\nforbids\tVBZ\nforced\tVBD\nforced\tVBN\nforces\tNNS\nForces\tNNS\nforces\tVBZ\nforce\tVB\nforce\tVBP\nFORCE\tVBP\nforcing\tVBG\nforecasters\tNNS\nforecasting\tVBG\nforecasts\tNNS\nForecasts\tNNS\nforecasts\tVBZ\nforecast\tVB\nforecast\tVBD\nforecast\tVBN\nforecast\tVBP\nforeclosed\tVBD\nforeclosed\tVBN\nForeclosed\tVBN\nforeclosed\tVBN|JJ\nforeclosures\tNNS\nForeclosures\tNNS\nforefathers\tNNS\nforeigners\tNNS\nForeigners\tNNS\nforensics\tNNS\nforerunners\tNNS\nforesaw\tVBD\nforeseen\tVBN\nforesees\tVBZ\nforesee\tVB\nforesee\tVBP\nforeshadowed\tVBN\nforest-products\tNNS\nForest-products\tNNS\nforests\tNNS\nforfeitures\tNNS\nforfeit\tVB\nforged\tVBN\nforgeries\tNNS\nforgets\tVBZ\nforgetting\tVBG\nforget\tVB\nForget\tVB\nforget\tVBP\nforge\tVB\nforgings\tNNS\nforging\tVBG\nforgiven\tVBN\nforgive\tVB\nforgiving\tVBG\nforgotten\tVBN\nforgot\tVBD\nforgot\tVBN\nforgo\tVB\nforked\tVBD\nforked\tVBN\nforklifts\tNNS\nfork\tVB\nformalities\tNNS\nformalizes\tVBZ\nformats\tNNS\nformed\tVBD\nformed\tVBN\nFormed\tVBN\nforming\tVBG\nforms\tNNS\nforms\tVBZ\nformulas\tNNS\nformulated\tVBN\nformulates\tVBZ\nformulate\tVB\nformulating\tVBG\nformulations\tNNS\nform\tVB\nform\tVBP\nforsaken\tVBN\nfortified\tVBN\nfortunes\tNNS\nforums\tNNS\nforward\tVB\nfossils\tNNS\nfostered\tVBD\nfostered\tVBN\nfostering\tVBG\nfoster\tVB\nfought\tVBD\nfought\tVBN\nfouled\tVBN\nfoundations\tNNS\nfounded\tVBD\nfounded\tVBN\nFounded\tVBN\nfoundered\tVBD\nfoundering\tVBG\nfounders\tNNS\nFounders\tNNS\nfounding\tVBG\nFounding\tVBG\nfound\tVBD\nfound\tVBN\nfountains\tNNS\nFour-fifths\tNNS\nfoxes\tNNS\nfractioning\tVBG\nfractions\tNNS\nfractured\tVBD\nfractured\tVBN\nfragmented\tVBN\nfragments\tNNS\nfragments\tVBZ\nfrailties\tNNS\nframed\tVBN\nframers\tNNS\nframes\tNNS\nframe\tVB\nframing\tVBG\nfranchised\tVBN\nfranchisees\tNNS\nFranchisees\tNNS\nfranchisers\tNNS\nfranchises\tNNS\nfranchises\tVBZ\nfranchise\tVB\nfranchising\tVBG\nFranciscans\tNNS\nfrancs\tNNS\nfraternities\tNNS\nfrauds\tNNS\nfrayed\tVBN\nfray\tVB\nfreaked\tVBN\nfreaks\tNNS\nfreedoms\tNNS\nfreed\tVBD\nFREED\tVBD\nfreed\tVBN\nFreed\tVBN\nfreeholders\tNNS\nfreeing\tVBG\nfrees\tVBZ\nfree\tVB\nfree\tVBP\nfreeways\tNNS\nfreezers\tNNS\nfreezes\tNNS\nfreezes\tVBZ\nfreeze\tVB\nfreighters\tNNS\nfreight\tVB\nFrench\tNNS\nfrequencies\tNNS\nfrequents\tVBZ\nfreshmen\tNNS\nfrets\tVBZ\nfretted\tVBD\nfretting\tVBG\nfret\tVB\nfret\tVBP\nfrictions\tNNS\nFridays\tNNS\nfried\tVBN\nfriendships\tNNS\nfriends\tNNS\nFriends\tNNS\nfriers\tNNS\nfriezes\tNNS\nfrigates\tNNS\nfrightened\tVBD\nfrightened\tVBN\nfrighten\tVB\nfringes\tNNS\nfripperies\tNNS\nfrittered\tVBN\nfrittering\tVBG\nfrocks\tNNS\nfrogmen\tNNS\nfrogs\tNNS\nfrolicked\tVBN\nfrolic\tVB\nfronds\tNNS\nfronts\tNNS\nfroth\tVB\nfrozen\tVBN\nfroze\tVBD\nfruits\tNNS\nfrustrated\tVBD\nfrustrated\tVBN\nfrustrate\tVB\nfrustrating\tVBG\nfrustrations\tNNS\nfrying\tVBG\nfudge\tVB\nfudge\tVBP\nfueled\tVBD\nfueled\tVBN\nfueling\tVBG\nFueling\tVBG\nfuel-services\tNNS\nfuels\tNNS\nfuel\tVB\nfugitives\tNNS\nFuji\tNNS\nfulfilled\tVBD\nfulfilled\tVBN\nfulfilling\tVBG\nfulfills\tVBZ\nfulfill\tVB\nfulfill\tVBP\nfulminations\tNNS\nfumes\tNNS\nfumes\tVBZ\nfuming\tVBG\nfumpered\tVBD\nfunctionaries\tNNS\nfunctioned\tVBD\nfunctioning\tVBG\nfunctions\tNNS\nfunction\tVB\nfunction\tVBP\nfundamentalists\tNNS\nfundamentals\tNNS\nfunded\tVBD\nfunded\tVBN\nFunded\tVBN\nfunding\tVBG\nfund-raisers\tNNS\nfundraisers\tNNS\nFund-Raisers\tNNS\nfundraising\tVBG\nfunds\tNNS\nFunds\tNNS\nFUNDS\tNNS\nfunds\tVBZ\nfund\tVB\nfund\tVBP\nfungi\tNNS\nfunneled\tVBD\nfunneled\tVBN\nfunneling\tVBG\nfunnel\tVB\nfurloughed\tVBN\nfurloughs\tNNS\nfurnaces\tNNS\nfurnished\tVBN\nfurnishings\tNNS\nfurnishing\tVBG\nfurnish\tVB\nfurriers\tNNS\nfurrows\tVBZ\nfurs\tNNS\nfurthering\tVBG\nfurthers\tVBZ\nfurther\tVB\nfusses\tVBZ\nfutures\tNNS\nFutures\tNNS\nFUTURES\tNNS\ngadgets\tNNS\ngagged\tVBN\ngained\tVB\ngained\tVBD\ngained\tVBN\ngainers\tNNS\nGainers\tNNS\ngaining\tVBG\ngains\tNNS\nGains\tNNS\ngains\tVBZ\ngain\tVB\ngain\tVBP\ngalaxies\tNNS\ngalleries\tNNS\ngallons\tNNS\ngalloping\tVBG\ngallstones\tNNS\ngalvanized\tVBD\ngalvanize\tVB\ngalvanizing\tVBG\ngamblers\tNNS\ngamble\tVB\ngambling\tVBG\ngames\tNNS\ngangbusters\tNNS\ngangs\tNNS\ngangsters\tNNS\ngaped\tVBN\ngarages\tNNS\ngardeners\tNNS\ngardenettes\tNNS\ngardening\tVBG\ngardens\tNNS\ngarden\tVB\ngarments\tNNS\ngarnered\tVBD\ngarnered\tVBN\ngarner\tVB\ngarner\tVBP\ngases\tNNS\nGases\tNNS\ngas-gathering\tVBG\ngasolines\tNNS\ngasped\tVBD\ngasp\tVB\ngates\tNNS\ngathered\tVBD\ngathered\tVBN\nGathered\tVBN\ngatherings\tNNS\ngathering\tVBG\ngathers\tVBZ\ngather\tVB\ngather\tVBP\ngauges\tVBZ\ngauge\tVB\ngauging\tVBG\ngave\tVBD\nGave\tVBD\ngaze\tVBP\ngeared\tVBN\ngearing\tVBG\ngears\tVBZ\ngear\tVB\ngear\tVBP\ngeeks\tNNS\nGeeks\tNNS\ngemsbok\tNNS\ngems\tNNS\ngender\tVB\ngeneralists\tNNS\ngeneralizations\tNNS\nGeneralizations\tNNS\ngeneralize\tVB\ngenerated\tVBD\ngenerated\tVBN\ngenerates\tVBZ\ngenerate\tVB\ngenerate\tVBP\ngenerating\tVBG\ngenerations\tNNS\ngenerators\tNNS\ngenes\tNNS\ngenres\tNNS\ngentleladies\tNNS\ngentlemen\tNNS\ngeosciences\tNNS\nGermans\tNNS\nGERMANS\tNNS\ngerms\tNNS\ngestured\tVBD\ngestures\tNNS\ngets\tVBZ\nGets\tVBZ\ngetting\tVBG\nGetting\tVBG\nget-togethers\tNNS\nget\tVB\nGet\tVB\nGET\tVB\nget\tVBP\nGet\tVBP\ngeysers\tNNS\nghettos\tNNS\nghostbusters\tNNS\nGhostbusters\tNNS\nghosts\tNNS\ngiants\tNNS\nGiants\tNNS\ngifts\tNNS\ngilts\tNNS\nGilts\tNNS\ngimmicks\tNNS\ngirded\tVBD\ngirding\tVBG\ngirls\tNNS\ngiveaways\tNNS\nGiveaways\tNNS\ngivebacks\tNNS\ngiven\tVBN\nGiven\tVBN\ngives\tVBZ\nGives\tVBZ\ngiveth\tVBZ\ngive\tVB\nGive\tVB\ngive\tVBP\nGIVE\tVBP\ngiving\tVBG\nGiving\tVBG\ngizmos\tNNS\nglamorized\tVBN\nglamorize\tVB\nglanced\tVBD\nglares\tVBZ\nglasses\tNNS\nglaze\tVB\nglaze\tVBP\ngleaming\tVBG\ngleaned\tVBN\nglean\tVB\nglide\tVB\ngliding\tVBG\nglimpses\tNNS\nglitches\tNNS\nglitterati\tNNS\ngloated\tVBD\ngloaters\tNNS\ngloating\tVBG\ngloats\tVBZ\ngloat\tVB\nglobalists\tNNS\ngloss\tVB\ngloss\tVBP\ngloves\tNNS\nglowed\tVBD\nglowing\tVBG\nglued\tVBN\nglues\tNNS\ngluts\tNNS\nglutted\tVBN\nglut\tVB\nglycols\tNNS\ngnaw\tVB\ngoals\tNNS\ngoats\tNNS\ngobbled\tVBN\ngobbling\tVBG\ngoblins\tNNS\nGods\tNNS\ngoes\tVBZ\nGoes\tVBZ\ngoings-on\tNNS\ngoing\tVBG\nGoing\tVBG\ngoldbanded\tVBN\ngolds\tNNS\ngolfers\tNNS\ngolfs\tNNS\nGoliaths\tNNS\ngone\tVBN\nGone\tVBN\ngone\tVBN|JJ\ngon\tVB\ngon\tVBG\ngoodies\tNNS\ngoods\tNNS\nGoodyear\tVBP\ngored\tVBN\ngore\tVB\ngorillas\tNNS\ngossiping\tVBG\ngotten\tVBN\ngot\tVB\ngot\tVBD\nGot\tVBD\ngot\tVBN\ngot\tVBP\ngo\tVB\nGo\tVB\ngo\tVBP\ngoverned\tVBD\ngoverned\tVBN\ngoverning\tVBG\ngovernmental-affairs\tNNS\ngovernment-relations\tNNS\ngovernment-securities\tNNS\ngovernment-set\tVBN\ngovernmentset\tVBN\ngovernments\tNNS\nGovernments\tNNS\ngovernors\tNNS\nGovernors\tNNS\ngovern\tVB\nGovern\tVB\ngovern\tVBP\ngrabbed\tVBD\ngrabbed\tVBN\ngrabbing\tVBG\ngrabs\tNNS\ngrab\tVB\ngrab\tVBP\ngrace\tVB\ngraders\tNNS\ngrades\tNNS\nGrads\tNNS\ngraduated\tVBD\ngraduated\tVBN\ngraduates\tNNS\nGraduates\tNNS\ngraduates\tVBZ\ngrafted\tVBN\ngrains\tNNS\nGrains\tNNS\nGRAINS\tNNS\nGrammys\tNNS\ngrams\tNNS\ngrandchildren\tNNS\ngrandees\tNNS\ngrandkids\tNNS\ngrandmasters\tNNS\ngrandmothers\tNNS\ngrandparents\tNNS\nGranges\tNNS\ngranted\tVBD\ngranted\tVBN\ngranting\tVBG\ngrants\tNNS\ngrants\tVBZ\ngrant\tVB\ngrant\tVBP\ngrapes\tNNS\nGrapes\tNNS\ngraphics\tNNS\nGraphics\tNNS\ngraphs\tNNS\ngrappled\tVBD\ngrapples\tVBZ\ngrapple\tVB\ngrasping\tVBG\ngrasp\tVB\ngrasp\tVBP\ngrassroots\tNNS\ngratuities\tNNS\ngraying\tVBG\ngrazed\tVBD\ngrazers\tNNS\ngraze\tVBP\ngreat-grandchildren\tNNS\ngreats\tNNS\ngreenhouses\tNNS\ngreens\tNNS\nGreens\tNNS\ngreeted\tVBD\ngreeted\tVBN\nGreetings\tNNS\ngreeting\tVBG\ngreets\tVBZ\ngreet\tVB\ngrenades\tNNS\ngrew\tVBD\nGREW\tVBD\ngrew\tVBN\ngridlocked\tVBN\ngrievances\tNNS\ngrill\tVBP\ngrimaced\tVBD\ngrimaces\tNNS\ngrimace\tVB\ngrinders\tNNS\ngrinding\tVBG\ngrinds\tVBZ\ngrind\tVB\ngringos\tNNS\nGrinned\tVBD\ngrinning\tVBG\ngrins\tNNS\ngripes\tNNS\ngripes\tVBZ\ngripped\tVBD\ngripping\tVBG\ngrips\tNNS\ngroans\tVBZ\ngroceries\tNNS\ngroped\tVBD\ngrossing\tVBG\ngrottoes\tNNS\ngroundbreakers\tNNS\ngrounded\tVBN\nground-handling\tNNS\ngrounding\tVBG\ngrounds\tNNS\nground\tVB\nground\tVBD\nground\tVBN\ngrouped\tVBN\ngroups\tNNS\nGroups\tNNS\ngroup\tVB\ngroused\tVBD\nGroused\tVBD\ngrouses\tVBZ\ngrovels\tVBZ\ngrowers\tNNS\ngrowing\tVBG\nGrowing\tVBG\nGROWING\tVBG\ngrowls\tVBZ\ngrown\tVBN\ngrows\tVBZ\nGrows\tVBZ\ngrowths\tNNS\ngrowth\tVB\ngrow\tVB\ngrow\tVBP\ngrueling\tVBG\ngrumbled\tVBD\ngrumble\tVBP\nguaranteed\tVBD\nguaranteed\tVBN\nGuaranteed\tVBN\nguaranteeing\tVBG\nguarantees\tNNS\nguarantees\tVBZ\nguarantee\tVB\nguarantee\tVBP\nguarded\tVBD\nguarded\tVBN\nguarding\tVBG\nguards\tNNS\nguards\tVBZ\nguard\tVB\nguard\tVBP\nGuber-Peters\tNNS\nguerrillas\tNNS\nguessed\tVBD\nguessing\tVBG\nguess\tVB\nguess\tVBP\nguests\tNNS\nGuests\tNNS\nguided\tVBD\nguided\tVBN\nguidelines\tNNS\nGuidelines\tNNS\nguideposts\tNNS\nguides\tNNS\nguide\tVB\nguiding\tVBG\nguilders\tNNS\ngunboats\tNNS\ngunmen\tNNS\nGunmen\tNNS\ngunned\tVBN\ngunners\tNNS\ngunslinging\tVBG\nguns\tNNS\nGuns\tNNS\ngurus\tNNS\nGurus\tNNS\ngushes\tVBZ\ngush\tVBP\nguts\tNNS\ngut\tVB\nguys\tNNS\nGuys\tNNS\nguzzle\tVB\ngymnastics\tNNS\ngyrated\tVBD\ngyrate\tVB\ngyrating\tVBG\ngyrations\tNNS\nhabeas\tNNS\nhabitats\tNNS\nhabits\tNNS\nhackers\tNNS\nhackles\tNNS\nhacks\tNNS\nhack\tVB\nhad\tVBD\nHad\tVBD\nhad\tVBN\nhaggle\tVB\nhagglings\tNNS\nhaggling\tVBG\nhailed\tVBD\nhailed\tVBN\nhailing\tVBG\nhails\tVBZ\nhalf-states\tNNS\nhalls\tNNS\nhallways\tNNS\nhalogenated\tVBD\nhalted\tVBD\nhalted\tVBN\nhalting\tVBG\nhalts\tNNS\nhalts\tVBZ\nhalt\tVB\nhalt\tVBP\nhalved\tVBD\nhalved\tVBN\nhalves\tNNS\nhalves\tVBZ\nhalve\tVB\nhamburgers\tNNS\nhammered\tVBN\nhammering\tVBG\nhammer\tVB\nhamming\tVBG\nhampered\tVBD\nhampered\tVBN\nhampering\tVBG\nhampers\tVBZ\nhamper\tVB\nhamstring\tVB\nhamstrung\tVBN\nhamstrung\tVBP\nhandbills\tNNS\nhandbooks\tNNS\nhand-carried\tVBN\nhandcuffed\tVBN\nhandcuffs\tNNS\nhanded\tVBD\nhanded\tVBN\nhandicapped\tNNS\nhandicapped\tVBN\nhandicap\tVB\nhandled\tVBD\nhandled\tVBN\nhandlers\tNNS\nhandles\tVBZ\nhandle\tVB\nhandle\tVBP\nhandling\tVBG\nhandpicked\tVBN\nhands\tNNS\nhandstands\tNNS\nhands\tVBZ\nhand\tVB\nhanged\tVBN\nhanging\tVBG\nhangs\tVBZ\nhang\tVB\nhang\tVBP\nhappened\tVBD\nHappened\tVBD\nhappened\tVBN\nHappened\tVBN\nhappenings\tNNS\nhappening\tVBG\nhappens\tVBZ\nhappen\tVB\nhappen\tVBP\nharangues\tNNS\nharangues\tVBZ\nharassed\tVBD\nharassing\tVBG\nharass\tVB\nharboring\tVBG\nharbors\tNNS\nharbors\tVBZ\nharbor\tVBP\nhardened\tVBN\nhardships\tNNS\nharmed\tVBD\nharmed\tVBN\nharms\tNNS\nharms\tVBZ\nharm\tVB\nharm\tVBP\nharnessing\tVBG\nharped\tVBD\nharping\tVBG\nharp\tVB\nharried\tVBN\nharvested\tVBN\nharvests\tNNS\nharvest\tVBP\nhashing\tVBG\nhassles\tNNS\nhastened\tVBN\nhasten\tVB\nhas\tVBN\nhas\tVBP\nhas\tVBZ\nHas\tVBZ\nHAS\tVBZ\nhatched\tVBN\nhatch\tVB\nhated\tVBN\nhates\tVBZ\nhate\tVB\nhate\tVBP\nhats\tNNS\nhauled\tVBD\nhaulers\tNNS\nhauling\tVBG\nhaul\tVB\nhaunted\tVBN\nhaunting\tVBG\nhaunts\tNNS\nhaunts\tVBZ\nhaunt\tVB\nhaunt\tVBP\nhavens\tNNS\nhave\tVB\nHave\tVB\nhave\tVBD\nhave\tVBN\nhave\tVBP\nHave\tVBP\nhaving\tVBG\nHaving\tVBG\nHawaii\tNNS\nhawkers\tNNS\nhawking\tVBG\nhawks\tNNS\nhawk\tVBP\nhazards\tNNS\nHCFCs\tNNS\nHDTVs\tNNS\nheadaches\tNNS\nheaded\tVBD\nheaded\tVBN\nHeaded\tVBN\nheading\tVBG\nHeading\tVBG\nheadlights\tNNS\nheadlined\tVBD\nheadlined\tVBN\nheadlines\tNNS\nheadphones\tNNS\nheadquarters\tNNS\nheadrests\tNNS\nheadsets\tNNS\nheads\tNNS\nheads\tVBZ\nhead\tVB\nhead\tVBP\nhealed\tVBN\nhealing\tVBG\nhealth-products\tNNS\nheal\tVB\nheaped\tVBD\nheaped\tVBN\nheaping\tVBG\nheard\tVBD\nheard\tVBN\nHeard\tVBN\nhearings\tNNS\nHearings\tNNS\nhearing\tVBG\nHearing\tVBG\nhears\tVBZ\nHEARS\tVBZ\nheartened\tVBN\nhearts\tNNS\nhear\tVB\nhear\tVBP\nheated\tVBN\nheaters\tNNS\nheating\tVBG\nHeating\tVBG\nheats\tVBZ\nheat\tVB\nheaved\tVBD\nheaved\tVBN\nheavens\tNNS\nheaves\tVBD\nheavyweights\tNNS\nheckled\tVBN\nhedgers\tNNS\nhedges\tNNS\nhedge\tVB\nhedge\tVBP\nhedging\tVBG\nhedging\tVBG|JJ\nheebie-jeebies\tNNS\nheeded\tVBD\nheeded\tVBN\nheed\tVB\nheels\tNNS\nheighborhoods\tNNS\nheightened\tVBD\nheightened\tVBN\nheighten\tVB\nheights\tNNS\nheirs\tNNS\nheld\tVBD\nheld\tVBN\nhelicopters\tNNS\nhelped\tVBD\nhelped\tVBN\nHelped\tVBN\nhelped\tVBP\nhelping\tVBG\nHelping\tVBG\nhelps\tVBZ\nHelps\tVBZ\nhelp\tVB\nHelp\tVB\nhelp\tVBP\nhemorrhaged\tVBN\nhemorrhaging\tVBG\nhemorrhoids\tNNS\nhens\tNNS\nHens\tNNS\nheralded\tVBN\nherald\tVB\nherald\tVBP\nherbicides\tNNS\nHercules\tNNS\nherding\tVBG\nherds\tNNS\nherniated\tVBN\nheroes\tNNS\nherons\tNNS\nhesitate\tVB\nhesitate\tVBP\nhesitating\tVBG\nhewed\tVBD\nhewn\tVBN\nhews\tVBZ\nhidden\tVBN\nhideouts\tNNS\nhiders\tNNS\nhides\tNNS\nhide\tVB\nhiding\tVBG\nhid\tVBD\nhighlands\tNNS\nhighlighted\tVBD\nhighlighted\tVBN\nhighlighting\tVBG\nhighlights\tNNS\nHighlights\tNNS\nhighlights\tVBZ\nhighlight\tVB\nhigh-rises\tNNS\nhighs\tNNS\nhightailing\tVBG\nhightops\tNNS\nhighways\tNNS\nHighways\tNNS\nhiked\tVBN\nhikers\tNNS\nhikes\tNNS\nhills\tNNS\nhindered\tVBD\nhindered\tVBN\nhindering\tVBG\nhinders\tVBZ\nhinder\tVB\nhinge\tVB\nhinted\tVBD\nhinted\tVBN\nhinterlands\tNNS\nhinting\tVBG\nhints\tNNS\nhints\tVBZ\nhint\tVB\nhint\tVBP\nhips\tNNS\nhired\tVBD\nhired\tVBN\nHired\tVBN\nhires\tNNS\nhires\tVBZ\nhire\tVB\nhire\tVBP\nhiring\tVBG\nHispanics\tNNS\nhissed\tVBD\nhistorians\tNNS\nhistorical-claims\tNNS\nhistoricized\tVBN\nhitched\tVBN\nhitches\tNNS\nhits\tNNS\nhits\tVBZ\nhitters\tNNS\nhitting\tVBG\nhit\tVB\nhit\tVBD\nhit\tVBN\nHit\tVBN\nhit\tVBP\nHit\tVBP\nhoarding\tVBG\nhoards\tNNS\nhoard\tVBP\nhobbies\tNNS\nhobbled\tVBN\nhobbles\tVBZ\nhobbling\tVBG\nhobbyists\tNNS\nhobos\tNNS\nHoe\tVB\nhog\tNNS\nhogs\tNNS\nHogs\tNNS\nhoisted\tVBN\nholders\tNNS\nHolders\tNNS\nholdings\tNNS\nHoldings\tNNS\nholding\tVBG\nHolding\tVBG\nholdouts\tNNS\nholdovers\tNNS\nholds\tVBZ\nholdups\tNNS\nhold\tVB\nHold\tVB\nHOLD\tVB\nhold\tVB|NN\nhold\tVBP\nHold\tVBP\nholed\tVBN\nholes\tNNS\nhole\tVBP\nholidays\tNNS\nholler\tVB\nHomeFed\tVBN\nhomeowners\tNNS\nhomered\tVBD\nhomers\tNNS\nhomer\tVB\nhomes\tNNS\nhomicides\tNNS\nhomosexuals\tNNS\nHomosexuals\tNNS\nHondurans\tNNS\nHonduras\tNNS\nhoned\tVBN\nhone\tVB\nhonorariums\tNNS\nhonored\tVBN\nhonoring\tVBG\nhonors\tNNS\nhonor\tVB\nhonor\tVBP\nhoods\tNNS\nhoodwinked\tVBN\nhooked\tVBD\nhooked\tVBN\nhooking\tVBG\nhooks\tVBZ\nhookups\tNNS\nhook\tVB\nhooves\tNNS\nhoped\tVBD\nhoped\tVBN\nhopes\tNNS\nHopes\tNNS\nHOPES\tNNS\nhopes\tVBZ\nhope\tVB\nhope\tVBP\nhoping\tVBG\nHoping\tVBG\nhopping\tVBG\nhopscotched\tVBD\nhops\tVBZ\nhorizons\tNNS\nhormones\tNNS\nhorns\tNNS\nhoroscopes\tNNS\nhorribles\tNNS\nhorrors\tNNS\nhorses\tNNS\nHorses\tNNS\nhorticultural-products\tNNS\nhosannas\tNNS\nhoses\tNNS\nHoses\tNNS\nhoses\tVBZ\nhospitalizations\tNNS\nhospitalized\tVBN\nhospitals\tNNS\nHospitals\tNNS\nhostages\tNNS\nhosted\tVBD\nhostilities\tNNS\nhosting\tVBG\nhosts\tNNS\nhosts\tVBZ\nhost\tVB\nhot-cereals\tNNS\nhotel-casinos\tNNS\nhoteliers\tNNS\nhotels\tNNS\nhotlines\tNNS\nhounded\tVBD\nhounding\tVBG\nhours\tNNS\nHours\tNNS\nhoused\tVBN\nhouseholds\tNNS\nhouses\tNNS\nhouses\tVBZ\nhouse\tVB\nhouse\tVBP\nhousewares\tNNS\nhousewives\tNNS\nHousings\tNNS\nhovered\tVBD\nhovered\tVBN\nhovering\tVBG\nhowling\tVBG\nhuckstering\tVBG\nhuddled\tVBD\nhugged\tVBD\nhugging\tVBG\nhugs\tNNS\nhulking\tVBG\nhumanities\tNNS\nhumanizing\tVBG\nhuman-resources\tNNS\nhuman-rights\tNNS\nhumans\tNNS\nhumbled\tVBN\nhum\tVB\nhunched\tVBN\nhundreds\tNNS\nHundreds\tNNS\nhung\tVBD\nhung\tVBN\nhunker\tVB\nhunted\tVBN\nhunter-gatherers\tNNS\nhunters\tNNS\nhunting\tVBG\nhunts\tVBZ\nhunt\tVB\nhunt\tVBP\nhurdles\tNNS\nhurled\tVBN\nhurling\tVBG\nhurl\tVBP\nhurricanes\tNNS\nhurried\tVBN\nhurries\tVBZ\nhurry\tVB\nhurting\tVBG\nhurtling\tVBG\nhurts\tVBZ\nHurts\tVBZ\nhurt\tVB\nhurt\tVBD\nhurt\tVBN\nhurt\tVBP\nhusbands\tNNS\nHUSBANDS\tNNS\nhustlers\tNNS\nhustles\tVBZ\nhybrids\tNNS\nhyenas\tNNS\nhyped\tVBD\nhypermarkets\tNNS\nhyping\tVBG\nhypnotized\tVBN\nhypocrites\tNNS\nhypothesized\tVBN\nice-baggers\tNNS\nideals\tNNS\nideas\tNNS\nIdeas\tNNS\nidentified\tVBD\nidentified\tVBN\nidentifies\tVBZ\nidentifying\tVBG\nidentify\tVB\nidentify\tVBP\nidentities\tNNS\nideologies\tNNS\nideologues\tNNS\nIdeologues\tNNS\nidiots\tNNS\nidled\tVBD\nidled\tVBN\nidle\tVB\nidling\tVBG\nignited\tVBD\nignited\tVBN\nignite\tVB\nignored\tVBD\nignored\tVBN\nignores\tVBZ\nignore\tVB\nIgnore\tVB\nignore\tVBP\nignoring\tVBG\nIgnoring\tVBG\nillegalities\tNNS\nillnesses\tNNS\nills\tNNS\nilluminates\tVBZ\nilluminate\tVB\nillusions\tNNS\nillustrated\tVBD\nillustrated\tVBN\nillustrates\tVBZ\nillustrate\tVB\nillustrate\tVBP\nillustrations\tNNS\nimages\tNNS\nimagined\tVBD\nimagined\tVBN\nimagines\tVBZ\nimagine\tVB\nImagine\tVB\nimagine\tVBP\nimagining\tVBG\nimbalances\tNNS\nimitated\tVBN\nimitate\tVBP\nimitating\tVBG\nimmersed\tVBN\nimmigrants\tNNS\nimmigrated\tVBD\nimmunities\tNNS\nimpacted\tVBN\nimpacts\tNNS\nimpact\tVB\nimpaired\tVBN\nimpair\tVB\nimpart\tVB\nimpeached\tVBN\nimpeded\tVBN\nimpedes\tVBZ\nimpede\tVB\nimpediments\tNNS\nimpeding\tVBG\nimpelled\tVBN\nimpending\tVBG\nimperatives\tNNS\nimperfections\tNNS\nimperialists\tNNS\nimperiled\tVBN\nimpersonations\tNNS\nimplanted\tVBD\nimplanted\tVBN\nimplanting\tVBG\nimplant\tVB\nimplemented\tVBD\nimplemented\tVBN\nimplementing\tVBG\nimplements\tVBZ\nimplement\tVB\nimplicated\tVBN\nimplicate\tVB\nimplications\tNNS\nimplied\tVBD\nimplied\tVBN\nimplies\tVBZ\nimplores\tVBZ\nimploring\tVBG\nimplying\tVBG\nimply\tVB\nimply\tVBP\nimported\tVBD\nimported\tVBN\nimported\tVBN|JJ\nimporters\tNNS\nimporting\tVBG\nimports\tNNS\nImports\tNNS\nimports\tVBZ\nimport\tVB\nimport\tVBP\nimposed\tVBD\nimposed\tVBN\nimposes\tVBZ\nimpose\tVB\nImpose\tVB\nimposing\tVBG\nimpounded\tVBN\nimpound\tVB\nimpoverished\tVBN\nimpressed\tVBD\nimpressed\tVBN\nimpresses\tVBZ\nimpressionists\tNNS\nimpress\tVB\nimprisoned\tVBN\nImprisoned\tVBN\nimprisoning\tVBG\nimprison\tVB\nimproprieties\tNNS\nimproved\tVBD\nimproved\tVBN\nimprovements\tNNS\nimproves\tVBZ\nimprove\tVB\nimprove\tVBP\nimproving\tVBG\nImproving\tVBG\nimprovised\tVBD\nimpugn\tVB\nimpulses\tNNS\ninaugurated\tVBN\nincarcerate\tVB\nincensed\tVBN\nincentives\tNNS\ninched\tVBD\ninched\tVBN\ninches\tNNS\ninching\tVBG\ninch\tVB\nincidents\tNNS\nincisions\tNNS\ninciting\tVBG\ninclined\tVBN\nincluded\tVBD\nincluded\tVBN\nIncluded\tVBN\nincludes\tVBZ\nIncludes\tVBZ\ninclude\tVB\ninclude\tVBP\nincluding\tVBG\nIncluding\tVBG\nincomes\tNNS\nincongruities\tNNS\ninconsistencies\tNNS\nincorporated\tVBD\nincorporated\tVBN\nincorporates\tVBZ\nincorporate\tVB\nincorporating\tVBG\nincreased\tVB\nincreased\tVBD\nincreased\tVBN\nIncreased\tVBN\nincreased\tVBN|JJ\nincreases\tNNS\nIncreases\tNNS\nincreases\tVBZ\nincrease\tVB\nIncrease\tVB\nincrease\tVBP\nincreasing\tVBG\nIncreasing\tVBG\nincrements\tNNS\nincriminating\tVBG\nincumbents\tNNS\nincurred\tVBD\nincurred\tVBN\nincurring\tVBG\nincur\tVB\nincur\tVBP\nindemnify\tVB\nindependents\tNNS\nindexed\tVBN\nindexers\tNNS\nindexes\tNNS\nIndexes\tNNS\nindex-futures\tNNS\nindexing\tVBG\nindex-options\tNNS\nindicated\tVBD\nindicated\tVBN\nindicates\tVBZ\nindicate\tVB\nindicate\tVBP\nindicating\tVBG\nindications\tNNS\nindicators\tNNS\nindices\tNNS\nindicted\tVBD\nindicted\tVBN\nindictments\tNNS\nindict\tVB\nindividuals\tNNS\nIndividuals\tNNS\nindoctrinated\tVBN\ninduced\tVBD\ninduced\tVBN\ninducements\tNNS\ninduces\tVBZ\ninduce\tVB\ninducing\tVBG\nindulgences\tNNS\nindulges\tVBZ\nindulge\tVB\nindulging\tVBG\nindustrialists\tNNS\nindustrialized\tVBN\nindustrialize\tVB\nindustrials\tNNS\nindustrials\tVBZ\nindustries\tNNS\nIndustries\tNNS\ninefficiencies\tNNS\ninequalities\tNNS\ninequities\tNNS\ninfants\tNNS\nInfants\tNNS\ninfected\tVBD\ninfected\tVBN\ninfecting\tVBG\ninfections\tNNS\ninferences\tNNS\ninferred\tVBN\ninfiltrated\tVBN\ninfiltrate\tVB\ninfiltrating\tVBG\ninflame\tVB\ninflated\tVBD\ninflated\tVBN\ninflates\tVBZ\ninflate\tVB\ninflating\tVBG\nInflation-adjusted\tVBN\ninflicted\tVBD\ninflicted\tVBN\ninflict\tVB\ninflows\tNNS\ninfluenced\tVBD\ninfluenced\tVBN\ninfluences\tNNS\ninfluences\tVBZ\ninfluence\tVB\ninfluence\tVBP\ninfluencing\tVBG\ninformation-services\tNNS\ninformation-systems\tNNS\ninformed\tVBD\ninformed\tVBN\ninforming\tVBG\ninforms\tVBZ\ninform\tVB\ninfractions\tNNS\ninfringed\tVBD\ninfringed\tVBN\ninfringes\tVBZ\ninfringe\tVB\ninfringing\tVBG\ninfuriated\tVBD\ninfuriate\tVB\ninfused\tVBN\ninfuse\tVB\ningest\tVB\ningots\tNNS\ningrates\tNNS\ningratiate\tVB\ningredients\tNNS\ninhabited\tVBN\ninhabits\tVBZ\ninhabit\tVBP\ninherited\tVBD\ninherited\tVBN\ninherits\tVBZ\ninherit\tVBP\ninhibited\tVBD\ninhibit\tVB\ninhibit\tVBP\ninitialed\tVBD\ninitialing\tVBG\ninitials\tNNS\ninitiated\tVBD\ninitiated\tVBN\ninitiate\tVB\ninitiate\tVBP\ninitiating\tVBG\ninitiatiors\tNNS\ninitiatives\tNNS\ninitiatives\tVBZ\ninjected\tVBD\ninjected\tVBN\ninjecting\tVBG\ninjections\tNNS\ninjects\tVBZ\ninject\tVB\nInject\tVB\ninjunctions\tNNS\ninjured\tVBD\ninjured\tVBN\ninjure\tVB\ninjuries\tNNS\ninjuring\tVBG\ninjustices\tNNS\ninks\tNNS\ninmates\tNNS\ninnings\tNNS\ninnocents\tNNS\ninnoculating\tVBG\ninnovated\tVBD\ninnovate\tVB\ninnovations\tNNS\ninnovators\tNNS\ni\tNNS\ninns\tNNS\ninnuendoes\tNNS\ninputs\tNNS\ninquired\tVBD\ninquiries\tNNS\ninquiring\tVBG\ninroads\tNNS\ninsects\tNNS\ninserted\tVBD\ninserted\tVBN\ninserting\tVBG\ninserts\tNNS\ninsert\tVB\ninsiders\tNNS\nInsiders\tNNS\ninsights\tNNS\ninsinuating\tVBG\ninsisted\tVBD\ninsisted\tVBN\ninsisting\tVBG\nInsisting\tVBG\ninsists\tVBZ\ninsist\tVB\ninsist\tVBP\ninspected\tVBD\ninspected\tVBN\ninspecting\tVBG\ninspections\tNNS\ninspectors\tNNS\ninspectors\tVBZ\nInspects\tVBZ\ninspect\tVB\ninspect\tVBP\ninspirations\tNNS\ninspired\tVBD\ninspired\tVBN\nInspired\tVBN\ninspire\tVB\ninspire\tVBP\ninstallations\tNNS\ninstalled\tVBD\ninstalled\tVBN\ninstalling\tVBG\ninstallments\tNNS\ninstall\tVB\ninstall\tVBP\ninstances\tNNS\ninstigated\tVBD\ninstill\tVB\ninstincts\tNNS\ninstituted\tVBD\ninstituted\tVBN\ninstitute\tVB\ninstituting\tVBG\ninstitutions\tNNS\nInstitutions\tNNS\ninstructed\tVBD\ninstructed\tVBN\ninstructing\tVBG\ninstructions\tNNS\ninstructors\tNNS\ninstructs\tVBZ\ninstruct\tVB\ninstrumentalists\tNNS\ninstruments\tNNS\nInstruments\tNNS\ninsulated\tVBN\ninsulate\tVB\ninsulate\tVBP\ninsulating\tVBG\ninsulins\tNNS\ninsulting\tVBG\ninsult\tVB\ninsurance-claims\tNNS\ninsured\tVBD\ninsured\tVBN\nInsureres\tNNS\ninsurers\tNNS\nInsurers\tNNS\nINSURERS\tNNS\ninsures\tVBZ\ninsure\tVB\ninsure\tVBP\ninsurgents\tNNS\ninsuring\tVBG\nintegrated-technologies\tNNS\nintegrated\tVBN\nintegrate\tVB\nintegrating\tVBG\nintellectuals\tNNS\nintended\tVBD\nintended\tVBN\nintends\tVBZ\nintend\tVB\nintend\tVBP\nintensified\tVBD\nintensified\tVBN\nintensifying\tVBG\nintensify\tVB\nintentions\tNNS\nintents\tNNS\ninteractions\tNNS\ninterceded\tVBD\nintercepted\tVBN\nIntercepting\tVBG\ninterconnected\tVBN\ninterconnect\tVB\ninterested\tVBN\ninterests\tNNS\ninterests\tVBZ\ninterest\tVB\ninterfered\tVBD\ninterferes\tVBZ\ninterfere\tVB\ninterfere\tVBP\ninterfering\tVBG\ninterior-furnishings\tNNS\ninteriors\tNNS\ninterjects\tVBZ\ninterloping\tVBG\nintermediaries\tNNS\nintermixed\tVBD\ninternationalists\tNNS\ninternational-operations\tNNS\ninterpretations\tNNS\ninterpreted\tVBD\ninterpreted\tVBN\ninterpreting\tVBG\ninterprets\tVBZ\ninterpret\tVB\ninterpret\tVBP\ninterrogated\tVBN\ninterrogators\tNNS\ninterrupted\tVBN\ninterrupting\tVBG\ninterruptions\tNNS\ninterrupt\tVB\nintersections\tNNS\ninterspersed\tVBN\nintersperses\tVBZ\ninterstates\tNNS\nintertitles\tNNS\nintertwined\tVBN\nintertwining\tVBG\nintervals\tNNS\nintervened\tVBD\nintervened\tVBN\nintervene\tVB\nintervening\tVBG\ninterventionists\tNNS\ninterventions\tNNS\ninterviewed\tVBD\ninterviewed\tVBN\ninterviewing\tVBG\ninterviews\tNNS\nInterviews\tNNS\ninterview\tVB\nintimate\tVB\nintimidated\tVBN\nintimidate\tVB\nintimidating\tVBG\nintimidations\tNNS\nintones\tVBZ\nintrigued\tVBN\nintrigues\tNNS\nintroduced\tVBD\nintroduced\tVBN\nIntroduced\tVBN\nintroduces\tVBZ\nintroduce\tVB\nintroducing\tVBG\nIntroducing\tVBG\nintroductions\tNNS\nintrude\tVBP\nintrusions\tNNS\ninundated\tVBN\ninvaded\tVBD\ninvaders\tNNS\ninvades\tVBZ\ninvade\tVB\ninvade\tVBP\ninvading\tVBG\ninvalidated\tVBD\ninvented\tVBD\ninvented\tVBN\ninventing\tVBG\ninventions\tNNS\ninventories\tNNS\nInventories\tNNS\ninventors\tNNS\ninvent\tVB\ninvent\tVBP\ninverted\tVBN\ninvested\tVBD\ninvested\tVBN\ninvestigated\tVBD\ninvestigated\tVBN\ninvestigates\tVBZ\ninvestigate\tVB\ninvestigating\tVBG\nInvestigating\tVBG\ninvestigations\tNNS\ninvestigators\tNNS\nInvestigators\tNNS\ninvesting\tVBG\nInvesting\tVBG\ninvesting\tVBG|NN\ninvestments\tNNS\ninvestor-relations\tNNS\ninvestors\tNNS\nInvestors\tNNS\ninvests\tVBZ\ninvest\tVB\ninvest\tVBP\ninvitations\tNNS\ninvited\tVBD\ninvited\tVBN\ninvites\tVBZ\ninvite\tVB\ninviting\tVBG\ninvoices\tNNS\ninvoked\tVBD\ninvoked\tVBN\ninvokes\tVBZ\ninvoke\tVB\ninvoking\tVBG\ninvolved\tVB\ninvolved\tVBD\ninvolved\tVBN\nInvolved\tVBN\ninvolves\tVBZ\ninvolve\tVB\ninvolve\tVBP\ninvolving\tVBG\nIOUs\tNNS\nIPOs\tNNS\nIRAs\tNNS\nIrises\tNNS\nirked\tVBD\nirked\tVBN\nirks\tVBZ\nirk\tVB\nironies\tNNS\nirons\tNNS\niron\tVB\nirradiated\tVBN\nirregularities\tNNS\nirritated\tVBN\nirritates\tVBZ\nislands\tNNS\nislands\tVBZ\nis\tNNS\nisolated\tVBD\nisolated\tVBN\nisolates\tVBZ\nisolate\tVB\nisolate\tVBP\nissued\tVBD\nissued\tVBN\nissuers\tNNS\nissues\tNNS\nIssues\tNNS\nISSUES\tNNS\nissues\tVBZ\nissue\tVB\nissue\tVBP\nissuing\tVBG\nis\tVBZ\nIs\tVBZ\nIS\tVBZ\nitemize\tVB\nitems\tNNS\nItems\tNNS\njabs\tNNS\nJackals\tNNS\njacked\tVBD\njacked\tVBN\njackets\tNNS\nJackets\tNNS\njackhammers\tNNS\njacking\tVBG\njack\tVB\njailed\tVBD\njailed\tVBN\nJAILED\tVBN\njails\tNNS\njammed\tVBD\njammed\tVBN\njams\tNNS\njam\tVB\nJan.\tVB\nJapanese-Americans\tNNS\nJapanese\tNNS\nJapanese\tVBP\njarring\tVBG\njars\tNNS\njaunts\tNNS\nJAUNTS\tNNS\njeans\tNNS\nJeeps\tNNS\njelled\tVBD\njeopardized\tVBN\njeopardizes\tVBZ\njeopardize\tVB\njeopardizing\tVBG\njerked\tVBN\njetliners\tNNS\njets\tNNS\nJets\tNNS\njettisoning\tVBG\njet\tVBP\njewelers\tNNS\njewels\tNNS\nJews\tNNS\njiggling\tVBG\njillions\tNNS\njingling\tVBG\njinks\tNNS\njinxed\tVBN\njitters\tNNS\njobs\tNNS\nJobs\tNNS\njockeys\tNNS\njocks\tNNS\njogs\tVBZ\njoined\tVBD\njoined\tVBN\njoining\tVBG\nJoining\tVBG\njoins\tVBZ\nJOINS\tVBZ\njoint-implants\tNNS\njoints\tNNS\njoin\tVB\nJOIN\tVB\njoin\tVBP\njoked\tVBD\njokes\tNNS\njokes\tVBZ\njoke\tVB\njolted\tVBD\njolted\tVBN\njolts\tNNS\njolt\tVB\nJolt\tVB\njostle\tVBP\njournalists\tNNS\nJournalists\tNNS\njournals\tNNS\njousting\tVBG\njoys\tNNS\njudged\tVBN\njudgeships\tNNS\njudges\tNNS\nJudges\tNNS\njudge\tVB\nJudge\tVB\njudge\tVBP\njudging\tVBG\nJudging\tVBG\njudgments\tNNS\njugglers\tNNS\njuggle\tVB\njuggling\tVBG\njugs\tNNS\njuices\tNNS\njumbos\tNNS\njumped\tVBD\njumped\tVBN\njumping\tVBG\nJumping\tVBG\njumps\tNNS\njumps\tVBZ\njump\tVB\njump\tVBP\njunctures\tNNS\njuniors\tNNS\njunk-bond\tNNS\njunkets\tNNS\njunk-holders\tNNS\njunkholders\tNNS\nJunk-holders\tNNS\njunkies\tNNS\njunk\tVB\njuries\tNNS\njurisdictions\tNNS\njurists\tNNS\njurors\tNNS\njury-rigged\tVBD\njustices\tNNS\njustified\tVBN\njustifies\tVBZ\njustifying\tVBG\njustify\tVB\njutting\tVBG\nJutting\tVBG\njuxtapose\tVBP\nkayoed\tVBN\nKC-135s\tNNS\nkeen\tVB\nKeepers\tNNS\nkeeping\tVBG\nKeeping\tVBG\nkeeps\tVBZ\nkeep\tVB\nKeep\tVB\nkeep\tVBP\nkept\tVBD\nkept\tVBN\nketchup\tVB|IN\nkeyboards\tNNS\nkeyed\tVBN\nkeys\tNNS\nkey\tVB\nkickbacks\tNNS\nkicked\tVBD\nkicked\tVBN\nkickers\tNNS\nkicking\tVBG\nkicks\tNNS\nkicks\tVBZ\nkick\tVB\nkick\tVBP\nkiddies\tNNS\nkidding\tVBG\nkidnapped\tVBD\nkidnapped\tVBN\nkidnappers\tNNS\nkidnap\tVB\nkids\tNNS\nKids\tNNS\nkilled\tVBD\nkilled\tVBN\nkillers\tNNS\nkillings\tNNS\nkilling\tVBG\nkills\tVBZ\nkill\tVB\nKill\tVB\nkill\tVBP\nkilobytes\tNNS\nkilograms\tNNS\nkilometers\tNNS\nkindled\tVBN\nkinds\tNNS\nkinfolk\tNNS\nkingpins\tNNS\nkings\tNNS\nKissing\tVBG\nKiss\tVB\nkits\tNNS\nkneaded\tVBN\nkneading\tVBG\nknees\tNNS\nknew\tVBD\nknights\tNNS\nknitted\tVBN\nknitting\tVBG\nknit\tVBN\nknocked\tVBD\nknocked\tVBN\nknocking\tVBG\nknocks\tVBZ\nknock\tVB\nknots\tNNS\nknowing\tVBG\nKnowing\tVBG\nknowns\tNNS\nknown\tVBN\nKnown\tVBN\nknows\tVBZ\nknow\tVB\nknow\tVBP\nKnow\tVBP\nknuckles\tNNS\nkowtow\tVB\nkronor\tNNS\nkudos\tNNS\nlabeled\tVBD\nlabeled\tVBN\nlabeling\tVBG\nLabeling\tVBG\nlabels\tNNS\nlabels\tVBZ\nlabel\tVB\nlabel\tVBP\nlaboratories\tNNS\nlaborers\tNNS\nlaboring\tVBG\nLaboring\tVBG\nlabors\tNNS\nlabs\tNNS\nlaced\tVBN\nlacked\tVBD\nlacked\tVBN\nlackeys\tNNS\nlacking\tVBG\nlacks\tVBZ\nlack\tVBP\nladies\tNNS\nlaggards\tNNS\nlagged\tVBD\nlagged\tVBN\nlagging\tVBG\nlagoons\tNNS\nlags\tNNS\nlags\tVBZ\nLags\tVBZ\nlag\tVB\nlag\tVBP\nlaid\tVBD\nlaid\tVBN\nlakes\tNNS\nlambasted\tVBD\nlambastes\tVBZ\nlamented\tVBD\nlaments\tVBZ\nlaminated\tVBN\nlampposts\tNNS\nlamps\tNNS\nlanded\tVBD\nlanded\tVBN\nlandfills\tNNS\nlandholdings\tNNS\nlandings\tNNS\nlanding\tVBG\nlandlords\tNNS\nlandowners\tNNS\nLandowners\tNNS\nlandscapers\tNNS\nlandscapes\tNNS\nlandslides\tNNS\nlands\tNNS\nland\tVB\nLand\tVBP\nlanes\tNNS\nlanguages\tNNS\nlanguished\tVBD\nlanguished\tVBN\nlanguishes\tVBZ\nlanguishing\tVBG\nlanguish\tVB\nlapsed\tVBN\nlapses\tNNS\nlapses\tVBZ\nlaps\tNNS\nlaptops\tNNS\nLaptops\tNNS\nlap\tVBP\nlasers\tNNS\nlashed\tVBD\nlashing\tVBG\nlash\tVB\nlash\tVBP\nlasted\tVBD\nlasted\tVBN\nlasting\tVBG\nlasts\tVBZ\nlast\tVB\nlast\tVBP\nlatched\tVBN\nlatches\tVBZ\nlatching\tVBG\nlatch\tVBP\nlathes\tNNS\nlauded\tVBD\nlauded\tVBN\nlaughed\tVBD\nlaughed\tVBN\nlaughing\tVBG\nlaughs\tNNS\nlaughs\tVBZ\nlaugh\tVB\nlaunched\tVBD\nlaunched\tVBN\nlaunches\tNNS\nlaunches\tVBZ\nlaunching\tVBG\nlaunch\tVB\nlaundered\tVBD\nlaundered\tVBN\nLaundered\tVBN\nlaunderers\tNNS\nlaundering\tVBG\nlaunder\tVB\nlaurels\tNNS\nlavished\tVBN\nlavishing\tVBG\nlawbreakers\tNNS\nlawmakers\tNNS\nLawmakers\tNNS\nLAWMAKERS\tNNS\nlaw-making\tNNS\nlawns\tNNS\nlaws\tNNS\nlawsuits\tNNS\nLawsuits\tNNS\nlawyers\tNNS\nLawyers\tNNS\nLAWYERS\tNNS\nlaxatives\tNNS\nlayers\tNNS\nlaying\tVBG\nlayoffs\tNNS\nLAYOFFS\tNNS\nlays\tVBZ\nlay\tVB\nlay\tVBD\nlay\tVBP\nLBOs\tNNS\nleaders\tNNS\nLEADERS\tNNS\nleading\tVBG\nLeading\tVBG\nleads\tNNS\nleads\tVBZ\nLeads\tVBZ\nlead\tVB\nlead\tVBN\nlead\tVBP\nleafing\tVBG\nleaflets\tNNS\nleaguers\tNNS\nleagues\tNNS\nleaked\tVBD\nleaked\tVBN\nleakers\tNNS\nleaking\tVBG\nleaks\tNNS\nleaned\tVBD\nleaned\tVBN\nleaning\tVBG\nleans\tVBZ\nlean\tVB\nLean\tVB\nlean\tVBP\nleaped\tVBD\nleaped\tVBN\nleapfrog\tVB\nleaping\tVBG\nLeaping\tVBG\nleaps\tNNS\nleapt\tVBD\nleap\tVB\nleap\tVBP\nlearned\tVBD\nlearned\tVBN\nlearning\tVBG\nLearning\tVBG\nlearns\tVBZ\nlearn\tVB\nlearn\tVBP\nleased\tVBD\nleased\tVBN\nleases\tNNS\nleases\tVBZ\nlease\tVB\nlease\tVBP\nleasing\tVBG\nleasing\tVBG|NN\nleathers\tNNS\nleaves\tNNS\nleaves\tVBZ\nleave\tVB\nLeave\tVB\nleave\tVBP\nleaving\tVBG\nLeaving\tVBG\nlectured\tVBD\nlectures\tVBZ\nlecture\tVB\nlecture\tVBP\nledgers\tNNS\nled\tVBD\nled\tVBN\nLed\tVBN\nleeches\tNNS\nleftists\tNNS\nleftovers\tNNS\nleft\tVBD\nLeft\tVBD\nleft\tVBN\nLeft\tVBN\nlegalizing\tVBG\nlegal-services\tNNS\nlegions\tNNS\nlegislate\tVB\nLegislating\tVBG\nlegislators\tNNS\nLegislators\tNNS\nlegislatures\tNNS\nlegitimized\tVBN\nlegitimize\tVB\nlegs\tNNS\nlemmings\tNNS\nlemons\tNNS\nlenders\tNNS\nLenders\tNNS\nlending\tVBG\nlends\tVBZ\nlend\tVB\nLend\tVB\nlend\tVBP\nlengthened\tVBD\nlengthened\tVBN\nlengthens\tVBZ\nlengthen\tVB\nlengthen\tVBP\nlengths\tNNS\nlenses\tNNS\nLENSES\tNNS\nlent\tVBD\nlent\tVBN\nleotards\tNNS\nlesbians\tNNS\nlesions\tNNS\nlessening\tVBG\nlessen\tVB\nlessers\tNNS\nlessons\tNNS\nlets\tVBZ\nletters\tNNS\nLetters\tNNS\nletting\tVBG\nLetting\tVBG\nlet\tVB\nLet\tVB\nlet\tVBD\nLet\tVBD\nlet\tVBN\nlet\tVBP\nleveled\tVBD\nleveled\tVBN\nleveling\tVBG\nlevels\tNNS\nlevel\tVB\nlevel\tVBP\nleveraged\tVBN\nLeveraged\tVBN\nLEVERAGED\tVBN\nleverage\tVB\nleveraging\tVBG\nlevied\tVBD\nlevy\tVB\nliabilities\tNNS\nliaisons\tNNS\nLiaisons\tNNS\nliars\tNNS\nlibeled\tVBN\nliberalizations\tNNS\nliberalized\tVBD\nliberalized\tVBN\nliberalize\tVB\nliberalize\tVBP\nliberalizing\tVBG\nliberals\tNNS\nLiberals\tNNS\nliberated\tVBD\nlibertarians\tNNS\nliberties\tNNS\nlibrarians\tNNS\nlibraries\tNNS\nlice\tNNS\nlicensed\tVBD\nlicensed\tVBN\nlicenses\tNNS\nlicenses\tVBZ\nlicense\tVB\nlicense\tVBP\nlicensing\tVBG\nlicking\tVBG\nLids\tNNS\nlied\tVBD\nlies\tNNS\nlies\tVBZ\nLIES\tVBZ\nlieutenants\tNNS\nlie\tVB\nlie\tVBP\nlifeguards\tNNS\nlifes\tNNS\nlifted\tVBD\nlifted\tVBN\nlifting\tVBG\nlifts\tNNS\nlifts\tVBZ\nlift\tVB\nLift\tVB\nlift\tVBP\nlighted\tVBN\nlightened\tVBD\nlightening\tVBG\nlighten\tVB\nlighting\tVBG\nlights\tNNS\nLights\tNNS\nlight\tVB\nliked\tVBD\nliked\tVBN\nlikened\tVBD\nlikened\tVBN\nlikening\tVBG\nlikes\tNNS\nlikes\tVBZ\nlike\tVB\nlike\tVBP\nlimbs\tNNS\nlimitations\tNNS\nlimited\tVBD\nlimited\tVBN\nlimiting\tVBG\nLimiting\tVBG\nlimits\tNNS\nlimits\tVBZ\nlimit\tVB\nlimit\tVBP\nlimousines\tNNS\nlimping\tVBG\nlinebackers\tNNS\nlined\tVBD\nlined\tVBN\nliners\tNNS\nlines\tNNS\nLines\tNNS\nlineups\tNNS\nline\tVB\nline\tVBP\nlingering\tVBG\nlingers\tVBZ\nlinger\tVB\nLinger\tVB\nlining\tVBG\nlinkages\tNNS\nlinked\tVBD\nlinked\tVBN\nlinking\tVBG\nLinking\tVBG\nlinks\tNNS\nlinks\tVBZ\nlink\tVB\nlink\tVBP\nLink\tVBP\nlions\tNNS\nlipoproteins\tNNS\nlips\tNNS\nlipsticks\tNNS\nliquefied\tVBN\nliquefies\tVBZ\nliquefy\tVB\nliquefy\tVBP\nliquidated\tVBD\nliquidated\tVBN\nliquidate\tVB\nliquidating\tVBG\nLiquidating\tVBG\nliquids\tNNS\nlire\tNNS\nlisted\tVBD\nlisted\tVBN\nlistened\tVBD\nlistened\tVBN\nlisteners\tNNS\nListeners\tNNS\nlistening\tVBG\nlistens\tVBZ\nlisten\tVB\nListen\tVB\nlisten\tVBP\nlistings\tNNS\nlisting\tVBG\nlists\tNNS\nlists\tVBZ\nlist\tVB\nlist\tVBP\nlithographs\tNNS\nlitigants\tNNS\nlitigators\tNNS\nlittered\tVBN\nlitter\tVBP\nlit\tVBD\nlit\tVBN\nlived\tVBD\nlived\tVBN\nlive-hauled\tVBD\nlive-haulers\tNNS\nlives\tNNS\nLives\tNNS\nlives\tVBZ\nLives\tVBZ\nlive\tVB\nLive\tVB\nlive\tVBP\nLive\tVBP\nliving\tVBG\nLiving\tVBG\nloaded\tVBD\nloaded\tVBN\nloadings\tNNS\nloading\tVBG\nloads\tNNS\nload\tVB\nloafers\tNNS\nloaned\tVBD\nloaned\tVBN\nloans\tNNS\nLoans\tNNS\nloans\tVBZ\nloan\tVB\nloathed\tVBD\nloathed\tVBN\nloathes\tVBZ\nloaves\tNNS\nlobbied\tVBD\nlobbied\tVBN\nlobbies\tNNS\nlobbying\tVBG\nlobbyists\tNNS\nlobby\tVB\nlocales\tNNS\nlocalities\tNNS\nlocalized\tVBN\nlocals\tNNS\nlocated\tVBN\nLocated\tVBN\nlocate\tVB\nlocate\tVBP\nlocating\tVBG\nlocations\tNNS\nlocked\tVBD\nlocked\tVBN\nlocking\tVBG\nlocks\tNNS\nlocks\tVBZ\nlock\tVB\nlock\tVBP\nlocutions\tNNS\nlodged\tVBD\nlodged\tVBN\nlodge\tVB\nlodgings\tNNS\nlogged\tVBD\nlogged\tVBN\nloggers\tNNS\nlogging\tVBG\nlogistics\tNNS\nlogos\tNNS\nlog-rolled\tVBD\nlogs\tNNS\nlogs\tVBZ\nlog\tVB\nlog\tVBP\nlong-term\tNNS\nlong\tVBP\nlooked\tVBD\nlooked\tVBN\nlookee-loos\tNNS\nlooking\tVBG\nLooking\tVBG\nlooks\tNNS\nlooks\tVBZ\nlook\tVB\nLook\tVB\nlook\tVBP\nlooming\tVBG\nlooms\tNNS\nlooms\tVBZ\nloom\tVB\nloom\tVBP\nLOOM\tVBZ\nloonies\tNNS\nloopholes\tNNS\nloops\tNNS\nloosened\tVBN\nloosening\tVBG\nloosen\tVB\nloose\tVB\nlooting\tVBG\nloot\tVB\nlopped\tVBD\nlorded\tVBD\nlords\tNNS\nlosers\tNNS\nloses\tNNS\nloses\tVBZ\nlose\tVB\nlose\tVBP\nlosing\tVBG\nlosses\tNNS\nLosses\tNNS\nLOSSES\tNNS\nlost\tVBD\nLost\tVBD\nlost\tVBN\nlotions\tNNS\nlots\tNNS\nLots\tNNS\nlotteries\tNNS\nloudspeakers\tNNS\nlouis\tNNS\nlounges\tNNS\nlovebirds\tNNS\nloved\tVBD\nloved\tVBN\nlovers\tNNS\nloves\tVBZ\nLoves\tVBZ\nlove\tVB\nlove\tVBP\nlowered\tVBD\nlowered\tVBN\nlowering\tVBG\nlowers\tVBZ\nlower\tVB\nlower\tVBP\nlow-lifes\tNNS\nlows\tNNS\nloyalties\tNNS\nlubricants\tNNS\nlucked\tVBD\nlugged\tVBD\nlugging\tVBG\nlugs\tNNS\nlulled\tVBN\nlumber\tVBP\nluminaries\tNNS\nlumped\tVBN\nlumping\tVBG\nlumps\tNNS\nlumps\tVBZ\nlunch\tVB\nlunged\tVBD\nlunging\tVBG\nlungs\tNNS\nlurched\tVBD\nlurching\tVBG\nlurch\tVBP\nlured\tVBD\nlured\tVBN\nlures\tNNS\nlures\tVBZ\nlure\tVB\nLure\tVBP\nluring\tVBG\nlurking\tVBG\nluxuries\tNNS\nlying\tVBG\nlyrics\tNNS\nLyrics\tNNS\nmachetes\tNNS\nMachiguengas\tNNS\nmachines\tNNS\nMachines\tNNS\nmachinists\tNNS\nMachinists\tNNS\nMachinists\tNNS|NNPS\nmade\tVBD\nmade\tVBN\nMade\tVBN\nmafias\tNNS\nmafiosi\tNNS\nmagazines\tNNS\nMagazines\tNNS\nmaggots\tNNS\nmagicians\tNNS\nmagistrates\tNNS\nmagnetized\tVBN\nmagnets\tNNS\nmagnified\tVBD\nmagnified\tVBN\nmagnify\tVB\nMagnolias\tNNS\nmaharajahs\tNNS\nmailed\tVBD\nmailed\tVBN\nmailers\tNNS\nmailings\tNNS\nmailing\tVBG\nmailmen\tNNS\nmail-sorting\tVBG\nmail\tVB\nmail\tVBP\nmainframes\tNNS\nmains\tNNS\nmaintained\tVBD\nmaintained\tVBN\nmaintaining\tVBG\nmaintains\tVBZ\nmaintain\tVB\nmaintain\tVBP\nMaintain\tVBP\nmajoring\tVBG\nmajors\tNNS\nmakers\tNNS\nMakers\tNNS\nmakes\tNNS\nmakes\tVBZ\nmake\tVB\nMake\tVB\nMAKE\tVB\nmake\tVBP\nMake\tVBP\nmaking\tVBG\nMaking\tVBG\nMAKING\tVBG\nMakin\tVBG\nmalefactors\tNNS\nmales\tNNS\nmalfunctions\tNNS\nmaligned\tVBN\nmalls\tNNS\nmammoths\tNNS\nmanacles\tNNS\nmanaged\tVBD\nmanaged\tVBN\nManaged\tVBN\nmanagements\tNNS\nmanagers\tNNS\nManagers\tNNS\nmanages\tVBZ\nmanage\tVB\nmanage\tVBP\nmanaging\tVBG\nManaging\tVBG\nmandated\tVBD\nmandated\tVBN\nmandates\tNNS\nmandates\tVBZ\nmandate\tVB\nmandating\tVBG\nmaneuvered\tVBD\nmaneuvered\tVBN\nmaneuverings\tNNS\nmaneuvering\tVBG\nmaneuvers\tNNS\nMANEUVERS\tNNS\nmaneuvers\tVBZ\nmaneuver\tVB\nmanhandled\tVBN\nmanifestations\tNNS\nmanifestos\tNNS\nmanifest\tVBP\nmaninstays\tNNS\nmanipulated\tVBN\nmanipulates\tVBZ\nmanipulate\tVB\nmanipulate\tVBP\nmanipulating\tVBG\nmanipulations\tNNS\nmanipulators\tNNS\nmanned\tVBD\nmannerisms\tNNS\nmanners\tNNS\nmanning\tVBG\nmansions\tNNS\nmanuals\tNNS\nMANUALS\tNNS\nmanuevering\tVBG\nmanufactured\tVBD\nmanufactured\tVBN\nmanufacturers\tNNS\nManufacturers\tNNS\nmanufactures\tVBZ\nmanufacture\tVB\nmanufacture\tVBP\nmanufacturing\tVBG\nman\tVB\nmany\tNNS\nmany\tVB\nmapped\tVBN\nmapping\tVBG\nmaps\tNNS\nmap\tVB\nmap\tVBP\nmarathons\tNNS\nmarbles\tNNS\nmarched\tVBD\nmarched\tVBN\nmarchers\tNNS\nmarches\tNNS\nmarching\tVBG\nmarch\tVB\nmarch\tVBP\nMargeotes\tNNS\nmarginalia\tNNS\nmarginalizing\tVBG\nmargin-calls\tNNS\nmargined\tVBN\nmargining\tVBG\nmargins\tNNS\nMargins\tNNS\nmarkdowns\tNNS\nmarked\tVBD\nmarked\tVBN\nmarketed\tVBD\nmarketed\tVBN\nmarketeers\tNNS\nmarketers\tNNS\nMarketers\tNNS\nmarketing-communications\tNNS\nmarketing\tVBG\nmarket-makers\tNNS\nmarketplaces\tNNS\nmarkets\tNNS\nMarkets\tNNS\nmarkets\tVBZ\nmarket\tVB\nmarket\tVBP\nmarking\tVBG\nmarkka\tNNS\nmarks\tNNS\nMarks\tNNS\nmarks\tVBZ\nmark\tVB\nmark\tVBP\nmarquees\tNNS\nmarred\tVBN\nmarriages\tNNS\nmarried\tVBD\nmarried\tVBN\nmarrying\tVBG\nmarry\tVB\nmarshes\tNNS\nmarveled\tVBD\nmarvels\tNNS\nmarvel\tVB\nmasked\tVBD\nmasked\tVBN\nmasks\tNNS\nMasks\tVBZ\nmasons\tNNS\nmasquerading\tVBG\nmassacres\tNNS\nmassages\tNNS\nmassage\tVB\nmassaging\tVBG\nmassed\tVBD\nmasses\tNNS\nmasseurs\tNNS\nmasseuses\tNNS\nmass-media\tNNS\nmass-producing\tVBG\nmastered\tVBN\nmasterpieces\tNNS\nmasters\tNNS\nmaster\tVB\nmatched\tVBD\nmatched\tVBN\nmatches\tNNS\nmatches\tVBZ\nmatching\tVBG\nmatch\tVB\nmatch\tVBP\nmated\tVBN\nmaterialized\tVBD\nmaterialized\tVBN\nmaterializes\tVBZ\nmaterialize\tVB\nmaterialize\tVBP\nmaterials\tNNS\nMaterials\tNNS\nmates\tNNS\nmathematics\tNNS\nmating\tVBG\nmattered\tVBN\nmatters\tNNS\nMatters\tNNS|VBZ\nmatters\tVBZ\nMatters\tVBZ\nmatter\tVB\nmatter\tVBP\nmatured\tVBN\nmatures\tVBZ\nmature\tVB\nmature\tVBP\nmaturing\tVBG\nmaturities\tNNS\nMaturities\tNNS\nmaul\tVB\nmavens\tNNS\nmaximize\tVB\nmaximizing\tVBG\nmaxims\tNNS\nmayors\tNNS\nmazes\tNNS\nMcDonald\tVB\nmeadows\tNNS\nmeals\tNNS\nmeandered\tVBD\nmeanders\tVBZ\nmeanings\tNNS\nmeaning\tVBG\nmeans\tNNS\nMeans\tNNS\nmeans\tVBZ\nMeans\tVBZ\nmeant\tVBD\nmeant\tVBN\nmean\tVB\nMean\tVB\nmean\tVBP\nmeasured\tVBD\nmeasured\tVBN\nMeasured\tVBN\nmeasurements\tNNS\nmeasures\tNNS\nMeasures\tNNS\nmeasures\tVBZ\nmeasure\tVB\nmeasuring\tVBG\nMeats\tNNS\nMEATS\tNNS\nmechanics\tNNS\nmechanisms\tNNS\nmedallions\tNNS\nmeddle\tVB\nmeddling\tVBG\nmedia\tNNS\nmediate\tVB\nmediators\tNNS\nmedical-products\tNNS\nmedicines\tNNS\nmeetings\tNNS\nMeetings\tNNS\nmeeting\tVBG\nMeeting\tVBG\nmeets\tNNS\nmeets\tVBZ\nmeet\tVB\nMeet\tVB\nmeet\tVBP\nmegabytes\tNNS\nmega-crashes\tNNS\nMega-hits\tNNS\nmega-issues\tNNS\nmega-mergers\tNNS\nmega-problems\tNNS\nmega-projects\tNNS\nmegaquestions\tNNS\nmega-resorts\tNNS\nmegawatts\tNNS\nmelding\tVBG\nmelds\tVBZ\nmeld\tVB\nmellowed\tVBN\nmelodies\tNNS\nmelting\tVBG\nmelts\tVBZ\nmelt\tVB\nmemberships\tNNS\nmembers\tNNS\nMembers\tNNS\nmementos\tNNS\nmemoirs\tNNS\nmemorabilia\tNNS\nmemoranda\tNNS\nmemorandums\tNNS\nmemorialized\tVBN\nmemories\tNNS\nmemorize\tVBP\nmemos\tNNS\nMEMOS\tNNS\nmend\tVB\nmen\tNNS\nMen\tNNS\nmentioned\tVBD\nmentioned\tVBN\nmentioning\tVBG\nmentions\tVBZ\nmention\tVB\nmention\tVBP\nmentors\tNNS\nmenus\tNNS\nMercantilists\tNNS\nMercedes-Benzes\tNNS\nMercedes\tNNS\nmerchandised\tVBN\nmerchandisers\tNNS\nmerchandising\tVBG\nmerchants\tNNS\nMerchants\tNNS\nmerged\tVBD\nmerged\tVBN\nmergers\tNNS\nmerge\tVB\nmerge\tVBP\nmerging\tVBG\nmeringues\tNNS\nmerits\tNNS\nmerit\tVB\nmerit\tVBP\nmesh\tVB\nmessages\tNNS\nmessengers\tNNS\nmessing\tVBG\nMessrs.\tNNS\nmess\tVB\nmetabolized\tVBN\nmetals\tNNS\nMetals\tNNS\nMETALS\tNNS\nmetal-workers\tNNS\nmetalworkers\tNNS\nmetaphors\tNNS\nmeted\tVBN\nmeters\tNNS\nmethodologies\tNNS\nmethods\tNNS\nmeting\tVBG\nmetrics\tNNS\nmet\tVBD\nmet\tVBN\nmice\tNNS\nMice\tNNS\nmicoprocessors\tNNS\nmicrobes\tNNS\nmicrochips\tNNS\nmicrocomputers\tNNS\nmicroeconomics\tNNS\nmicroelectronics\tNNS\nmicrophones\tNNS\nmicroprocessors\tNNS\nmicrowaves\tNNS\nmid-1940s\tNNS\nmid-1960s\tNNS\nmid-1970s\tNNS\nmid-1980s\tNNS\nmid-1990s\tNNS\nmiddlemen\tNNS\nmidsized\tVBN\nmiffed\tVBD\nmiffed\tVBN\nMiG-29s\tNNS\nmighta\tMD|VB\nmigrate\tVB\nmigrations\tNNS\nmiles\tNNS\nmilestones\tNNS\nmilitary-electronics\tNNS\nmilitate\tVB\nmilitias\tNNS\nmilked\tVBN\nmilks\tNNS\nmilk\tVB\nmilling\tVBG\nmillionaires\tNNS\nmillion-plus\tNNS\nmillions\tNNS\nMillions\tNNS\nmills\tNNS\nmimics\tNNS\nmimics\tVBZ\nmimic\tVB\nmimic\tVBP\nmince\tVB\nminded\tVBD\nminds\tNNS\nmind\tVB\nMind\tVB\nminefields\tNNS\nminerals\tNNS\nminers\tNNS\nmines\tNNS\nmine\tVBP\nmingle\tVBP\nminiaturized\tVBN\nminicars\tNNS\nminicomputers\tNNS\nminimills\tNNS\nminimized\tVBN\nminimize\tVB\nminimizing\tVBG\nminimums\tNNS\nmining\tVBG\nminiseries\tNNS\nministers\tNNS\nministries\tNNS\nminisupercomputers\tNNS\nminivans\tNNS\nMinneapolis\tNNS\nminorities\tNNS\nminors\tNNS\nminted\tVBN\nminting\tVBG\nmints\tNNS\nminuses\tNNS\nMinuses\tNNS\nminutes\tNNS\nminutiae\tNNS\nMIPs\tNNS\nMIPS\tNNS\nmired\tVBN\nmirrored\tVBD\nmirrors\tVBZ\nmirror\tVB\nmirror\tVBP\nmisadventures\tNNS\nmiscalculated\tVBD\nmiscalculated\tVBN\nmiscarriages\tNNS\nmisclassified\tVBN\nmiscreants\tNNS\nmisdeeds\tNNS\nmisdemeanors\tNNS\nmisfortunes\tNNS\nmisguided\tVBN\nmishandled\tVBD\nmishandling\tVBG\nmisinterpreted\tVBN\nmisinterpret\tVB\nmisjudged\tVBD\nmisjudgments\tNNS\nmislaid\tVBN\nmisleading\tVBG\nmisled\tVBD\nmisled\tVBN\nmismeasurements\tNNS\nmisperceptions\tNNS\nmisplaced\tVBN\nmisquoting\tVBG\nmisrepresentations\tNNS\nmisrepresented\tVBD\nmisrepresenting\tVBG\nmisrepresents\tVBZ\nmisrepresent\tVB\nmisrouted\tVBN\nmissed\tVBD\nmissed\tVBN\nmisses\tVBZ\nmissiles\tNNS\nmissing\tVBG\nmissionaries\tNNS\nmissions\tNNS\nmisspent\tVBN\nmisstated\tVBD\nmisstated\tVBN\nmisstatements\tNNS\nmisstates\tVBZ\nmiss\tVB\nmiss\tVBP\nmistaken\tVBN\nmistakes\tNNS\nmistake\tVB\nmistreat\tVB\nmistresses\tNNS\nmistrials\tNNS\nmists\tNNS\nmisunderstandings\tNNS\nMisunderstanding\tVBG\nmisunderstood\tVBN\nmisused\tVBD\nmites\tNNS\nmitigate\tVB\nmitigating\tVBG\nMitsubishi\tNNS\nMitsui\tNNS\nmixed\tVBD\nmixed\tVBN\nMixed\tVBN\nmixers\tNNS\nmixes\tVBZ\nmixing\tVBG\nmixtures\tNNS\nmix\tVB\nmoaning\tVBG\nmoans\tVBZ\nmoan\tVB\nmobilized\tVBD\nmobilized\tVBN\nmobilize\tVB\nmobilizing\tVBG\nmocked\tVBN\nmocking\tVBG\nmodeled\tVBD\nmodeled\tVBN\nmodeling\tVBG\nmodels\tNNS\nmodel\tVB\nmodems\tNNS\nmoderated\tVBN\nmoderates\tNNS\nmoderate\tVB\nmoderate\tVBP\nmoderating\tVBG\nmodernized\tVBD\nmodernized\tVBN\nmodernize\tVB\nmodernizing\tVBG\nmodes\tNNS\nmodifications\tNNS\nModifications\tNNS\nmodified\tVBN\nmodifies\tVBZ\nmodify\tVB\nmodulate\tVBP\nmoisturizers\tNNS\nmolded\tVBN\nmolds\tNNS\nmolecules\tNNS\nmollified\tVBN\nmollify\tVB\nmoments\tNNS\nMondays\tNNS\nmonetarists\tNNS\nmoneymakers\tNNS\nmonitored\tVBD\nMONITORED\tVBD\nmonitored\tVBN\nmonitoring\tVBG\nmonitors\tNNS\nmonitors\tVBZ\nmonitor\tVB\nmonitor\tVBP\nmonkeys\tNNS\nmonoliths\tNNS\nmonologues\tNNS\nmonopolies\tNNS\nmonopolized\tVBD\nmonopolized\tVBN\nmonopolize\tVB\nmonopolizing\tVBG\nmonths\tNNS\nMonths\tNNS\nMoonies\tNNS\nmoonlighting\tVBG\nmoons\tNNS\nmopping\tVBG\nmop\tVB\nmorals\tNNS\nmores\tNNS\nmornings\tNNS\nmorsels\tNNS\nmortgaged\tVBN\nmortgages\tNNS\nmortgage\tVB\nMoslems\tNNS\nmotels\tNNS\nmothers\tNNS\nMothers\tNNS\nmotifs\tNNS\nmotions\tNNS\nmotions\tVBZ\nmotivated\tVBN\nmotivate\tVB\nmotivate\tVBP\nmotivating\tVBG\nmotives\tNNS\nmotorcycles\tNNS\nmotorists\tNNS\nmotorized\tVBN\nmotors\tNNS\nmounds\tNNS\nmountains\tNNS\nmounted\tVBD\nmounted\tVBN\nmounting\tVBG\nmounts\tVBZ\nmount\tVB\nmount\tVBP\nmourning\tVBG\nMourning\tVBG\nmousetraps\tNNS\nmouthed\tVBD\nmouths\tNNS\nmoved\tVBD\nMOVED\tVBD\nmoved\tVBN\nmovements\tNNS\nmoves\tNNS\nMoves\tNNS\nmoves\tVBZ\nMOVES\tVBZ\nmove\tVB\nMove\tVB\nmove\tVBP\nMove\tVBP\nmovies\tNNS\nmoving\tVBG\nMoving\tVBG\nmow\tVB\nmucked\tVBN\nmuddied\tVBN\nmuddled\tVBN\nmuffled\tVBN\nmuffs\tNNS\nmulling\tVBG\nmulls\tVBZ\nmull\tVB\nmultimedia\tNNS\nmultinationals\tNNS\nmultipled\tVBD\nmultiples\tNNS\nmultiplied\tVBN\nmultiplying\tVBG\nmultiply\tVB\nmultiply\tVBP\nmumbled\tVBN\nmummies\tNNS\nMunching\tVBG\nmunicipalities\tNNS\nmunicipals\tNNS\nMunicipals\tNNS\nMUNICIPALS\tNNS\nmuni\tNNS\nmunis\tNNS\nmurals\tNNS\nmurdered\tVBN\nmurderers\tNNS\nmurdering\tVBG\nmurders\tNNS\nmurmuring\tVBG\nmuscled\tVBD\nmuscles\tNNS\nmuscling\tVBG\nMuscovites\tNNS\nmuses\tNNS\nmuses\tVBZ\nmuseums\tNNS\nMuseums\tNNS\nmuse\tVB\nmushroomed\tVBD\nmushroomed\tVBN\nmushrooms\tNNS\nmusicians\tNNS\nmuster\tVB\nmutated\tVBN\nmutate\tVB\nmutations\tNNS\nmuted\tVBN\nmute\tVB\nmutilated\tVBN\nMutinies\tNNS\nmutters\tNNS\nmutts\tNNS\nmuzzles\tVBZ\n'm\tVBP\nmysteries\tNNS\nmyths\tNNS\nnabbing\tVBG\nnaggings\tNNS\nnagging\tVBG\nnags\tNNS\nnailed\tVBN\nnails\tNNS\nnail\tVB\nname-droppers\tNNS\nname-drops\tVBZ\nnamed\tVBD\nnamed\tVBN\nNamed\tVBN\nnameplates\tNNS\nnames\tNNS\nNames\tNNS\nnames\tVBZ\nname\tVB\nname\tVBP\nnaming\tVBG\nnarcotics\tNNS\nnarratives\tNNS\nnarrowed\tVBD\nnarrowed\tVBN\nnarrowing\tVBG\nNarrowing\tVBG\nnarrows\tVBZ\nnarrow\tVB\nnationalists\tNNS\nnationalized\tVBD\nnationalized\tVBN\nnationals\tNNS\nnations\tNNS\nnatives\tNNS\nnatural-foods\tNNS\nnatural-resources\tNNS\nnavies\tNNS\nnavigate\tVB\nnaysayers\tNNS\nnaysay\tVB\nNazis\tNNS\nNeanderthals\tNNS\nneared\tVBD\nnearing\tVBG\nnear-monopolies\tNNS\nnears\tVBZ\nnecessitated\tVBD\nnecessitated\tVBN\nnecessities\tNNS\nnecks\tNNS\nneckties\tNNS\nneeded\tVBD\nneeded\tVBN\nneeded\tVBN|JJ\nneeding\tVBG\nneeds\tNNS\nNeeds\tNNS\nneeds\tVBZ\nneed\tVB\nneed\tVBP\nnegated\tVBN\nnegatives\tNNS\nneglected\tVBD\nneglected\tVBN\nneglecting\tVBG\nnegotiated\tVBD\nnegotiated\tVBN\nnegotiates\tVBZ\nnegotiate\tVB\nnegotiate\tVBP\nnegotiating\tVBG\nnegotiations\tNNS\nNegotiations\tNNS\nnegotiators\tNNS\nNegotiators\tNNS\nneighbhorhoods\tNNS\nneighborhoods\tNNS\nneighboring\tVBG\nneighbors\tNNS\nneighbours\tNNS\nneophytes\tNNS\nnerds\tNNS\nNerds\tNNS\nnerves\tNNS\nNestled\tVBN\nnests\tNNS\nnets\tNNS\nnets\tVBZ\nnetted\tVBD\nnetted\tVBN\nnetting\tVBG\nnet\tVB\nnetworking\tVBG\nnetworks\tNNS\nNetworks\tNNS\nnetwork\tVB\nneurologists\tNNS\nneutralized\tVBN\nneutralizes\tVBZ\nneutrons\tNNS\nnewborns\tNNS\nnewcasts\tNNS\nnewcomers\tNNS\nnewscasts\tNNS\nnewsies\tNNS\nnewsletters\tNNS\nNews\tNNS\nnewspapers\tNNS\nNewspapers\tNNS\nnewsprints\tNNS\nnewsstands\tNNS\nNewsstands\tNNS\nnews-weeklies\tNNS\nnibbling\tVBG\nniches\tNNS\nNicholas\tNNS\nnicked\tVBN\nnicknamed\tVBN\nnicknames\tNNS\nnightclubs\tNNS\nnightmares\tNNS\nnights\tNNS\nnine-months\tNNS\nnine-tenths\tNNS\nnixed\tVBD\nnoblemen\tNNS\nnods\tVBZ\nno-loads\tNNS\nnominated\tVBD\nnominated\tVBN\nnominate\tVB\nnominations\tNNS\nnominees\tNNS\nnon-clients\tNNS\nnon-communists\tNNS\nnonconformists\tNNS\nnon-economists\tNNS\nnon-lawyers\tNNS\nNon-lawyers\tNNS\nnonoperating\tVBG\nNonperformers\tNNS\nnonperforming\tVBG\nNonperforming\tVBG\nnonrecurring\tVBG\nnon-seamen\tNNS\nnonstops\tNNS\nnoodles\tNNS\nnormalize\tVB\nnorms\tNNS\nNorms\tNNS\nNorthrop\tVB\nnose-dived\tVBD\nnosedived\tVBD\nnose-dived\tVBN\nnosediving\tVBG\nnoses\tNNS\nNos.\tNNS\nnotched\tVBD\nnotch\tVB\nnotebooks\tNNS\nnoted\tVBD\nnoted\tVBN\nnoteholders\tNNS\nnotes\tNNS\nNotes\tNNS\nNOTES\tNNS\nnotes\tVBZ\nnote\tVB\nNote\tVB\nNOTE\tVB\nnote\tVBP\nnoticed\tVBD\nnoticed\tVBN\nnotices\tNNS\nnotice\tVB\nnotice\tVBP\nnoticing\tVBG\nnotifications\tNNS\nnotified\tVBD\nnotified\tVBN\nnotifies\tVBZ\nnotifying\tVBG\nnotify\tVB\nnotify\tVBP\nnoting\tVBG\nNoting\tVBG\nnotions\tNNS\nNo.\tVB\nnovels\tNNS\nnovelties\tNNS\nnovitiates\tNNS\nNov.\tVB\nnozzles\tNNS\nnuances\tNNS\nnuclear-arms\tNNS\nnudge\tVB\nnullified\tVBN\nnullify\tVB\nnullify\tVBP\nnumber-crunchers\tNNS\nnumbered\tVBD\nnumbered\tVBN\nnumbering\tVBG\nnumbers\tNNS\nnursed\tVBD\nnurseries\tNNS\nnurses\tNNS\nnursing\tVBG\nnurtured\tVBD\nnurtured\tVBN\nnurture\tVB\nnurturing\tVBG\nnuts\tNNS\noaks\tNNS\nOases\tNNS\noats\tNNS\nobey\tVBP\nobfuscate\tVB\nobjected\tVBD\nobjected\tVBN\nobjecting\tVBG\nobjections\tNNS\nObjections\tNNS\nobjectives\tNNS\nobjects\tNNS\nobject\tVB\nobject\tVBP\nobligated\tVBD\nobligated\tVBN\nobligations\tNNS\nObligations\tNNS\nobliged\tVBN\nobliged\tVBN|JJ\nobliges\tVBZ\nobliterated\tVBN\nobscured\tVBD\nobscures\tVBZ\nobscure\tVB\nobservations\tNNS\nobserved\tVBD\nobserved\tVBN\nobservers\tNNS\nObservers\tNNS\nobserves\tVBZ\nobserve\tVB\nobserve\tVBP\nobserving\tVBG\nObserving\tVBG\nobsessed\tVBN\nobsoleted\tVBN\nobsoleting\tVBG\nobstacles\tNNS\nobstructed\tVBN\nobstructing\tVBG\nobstruct\tVBP\nobtained\tVBD\nobtained\tVBN\nobtaining\tVBG\nobtain\tVB\nobtain\tVBP\nobviate\tVB\noccasions\tNNS\noccasion\tVB\noccupations\tNNS\noccupied\tVBD\noccupied\tVBN\noccupies\tVBZ\noccupying\tVBG\noccupy\tVB\noccurred\tVBD\noccurred\tVBN\noccurrences\tNNS\noccurring\tVBG\noccurs\tVBZ\noccur\tVB\noccur\tVBP\noceans\tNNS\noctaves\tNNS\noctogenarians\tNNS\noddballs\tNNS\noddities\tNNS\nODDITIES\tNNS\nodds\tNNS\nOdds\tNNS\noffenders\tNNS\noffending\tVBG\noffends\tVBZ\noffend\tVB\noffensives\tNNS\noffered\tVBD\noffered\tVBN\nOffered\tVBN\nOFFERED\tVBN\nofferings\tNNS\noffering\tVBG\nOffering\tVBG\noffers\tNNS\noffers\tVBZ\noffer\tVB\nOffer\tVB\noffer\tVBP\noff-hours\tNNS\nofficals\tNNS\nofficers\tNNS\nOfficers\tNNS\noffices\tNNS\nOffices\tNNS\noffice-supplies\tNNS\nofficials\tNNS\nOfficials\tNNS\nOFFICIALS\tNNS\noffi\tNNS\noffsets\tNNS\noffsetting\tVBG\nOffsetting\tVBG\noffset\tVB\noffset\tVBD\noffset\tVBN\noffset\tVBP\noffshoots\tNNS\noffspring\tNNS\nogles\tVBZ\nogling\tVBG\noils\tNNS\nOils\tNNS\nolds\tNNS\nold-timers\tNNS\nolefins\tNNS\nOlympics\tNNS\nomens\tNNS\nomissions\tNNS\nomits\tVBZ\nomitted\tVBD\nomitted\tVBN\nomit\tVB\nomit\tVBP\noncogenes\tNNS\nOncogenes\tNNS\nones\tNNS\nonlookers\tNNS\nOnlookers\tNNS\non-ramps\tNNS\nooze\tVB\noozing\tVBG\nopened\tVBD\nopened\tVBN\nopenended\tVBN\nopeners\tNNS\nopenings\tNNS\nopening\tVBG\nopens\tVBZ\nopen\tVB\nOpen\tVB\nopen\tVBP\noperas\tNNS\noperated\tVBD\noperated\tVBD|VBN\noperated\tVBN\noperates\tVBZ\noperate\tVB\noperate\tVBP\noperating\tVBG\nOperating\tVBG\noperations\tNNS\noperatives\tNNS\noperators\tNNS\nopining\tVBG\nopinion-makers\tNNS\nopinions\tNNS\nOpinions\tNNS\nopponents\tNNS\nOpponents\tNNS\nopportunists\tNNS\nopportunities\tNNS\nopposed\tVBD\nopposed\tVBN\nOpposed\tVBN\nopposes\tVBZ\noppose\tVB\noppose\tVBP\nopposing\tVBG\nopted\tVBD\nopted\tVBN\noptical-products\tNNS\noptimists\tNNS\nopting\tVBG\noptions\tNNS\nOptions\tNNS\nOPTIONS\tNNS\noption\tVBP\nopt\tVB\nopt\tVBP\noranges\tNNS\norchardists\tNNS\norchards\tNNS\norchestras\tNNS\norchestrated\tVBD\norchestrated\tVBN\norchestrating\tVBG\norchids\tNNS\nordained\tVBN\nordered\tVBD\nordered\tVBN\nORDERED\tVBN\nordering\tVBG\norders\tNNS\nOrders\tNNS\norders\tVBZ\norder\tVB\nordinances\tNNS\norganisms\tNNS\norganizations\tNNS\nOrganizations\tNNS\norganized\tVBD\norganized\tVBN\nORGANIZED\tVBN\norganizers\tNNS\norganizes\tVBZ\norganize\tVB\norganizing\tVBG\norgans\tNNS\noriented\tVBN\noriginated\tVBN\noriginated\tVBP\noriginates\tVBZ\noriginate\tVB\noriginating\tVBG\noriginations\tNNS\noriginators\tNNS\norigins\tNNS\nOrkem\tVB\nornaments\tNNS\norphaned\tVBN\norphans\tNNS\nothers\tNNS\nOthers\tNNS\nounces\tNNS\nousted\tVBD\nousted\tVBN\nOUSTED\tVBN\nousting\tVBG\noust\tVB\noutages\tNNS\noutbidding\tVBG\noutbid\tVB\noutbid\tVBP\noutbreaks\tNNS\noutcomes\tNNS\noutdated\tVBN\noutdid\tVBD\noutdistanced\tVBN\noutdone\tVBN\noutfielders\tNNS\noutfits\tNNS\noutfit\tVB\noutflank\tVB\noutflows\tNNS\nOutflows\tNNS\noutfly\tVB\noutgained\tVBD\noutgrew\tVBD\noutgrown\tVBN\noutings\tNNS\noutlanders\tNNS\noutlasted\tVBD\noutlast\tVB\noutlawed\tVBD\noutlawed\tVBN\noutlawing\tVBG\noutlays\tNNS\nOutlays\tNNS\noutleaped\tVBD\noutlets\tNNS\noutlined\tVBD\noutlined\tVBN\noutlines\tVBZ\noutline\tVB\noutlining\tVBG\noutlooks\tNNS\noutmoded\tVBN\noutnumbered\tVBD\nout-of-staters\tNNS\noutpaced\tVBD\noutpaced\tVBN\noutpaced\tVBP\noutpace\tVB\noutpace\tVBP\noutpacing\tVBG\noutperformed\tVBD\noutperformed\tVBN\noutperforming\tVBG\noutperforms\tVBZ\noutperform\tVB\noutperform\tVBP\noutposts\tNNS\noutraged\tVBN\noutranks\tVBZ\noutselling\tVBG\noutsells\tVBZ\noutsell\tVB\noutshines\tVBZ\noutshine\tVB\noutsiders\tNNS\noutskirts\tNNS\nout-smart\tVB\noutsold\tVBD\noutstripped\tVBD\noutstripped\tVBN\noutstripping\tVBG\noutstrips\tVBZ\noutstrip\tVB\nout-trade\tVB\noutweighed\tVBD\noutweighed\tVBN\noutweigh\tVB\noutweigh\tVBP\novens\tNNS\nover-allotments\tNNS\noveralls\tNNS\noverarching\tVBG\noverbid\tVB\noverbid\tVBD\noverbought\tVBN\noverburden\tVB\novercame\tVBD\novercharges\tNNS\novercollateralized\tVBN\novercomes\tVBZ\novercome\tVB\novercome\tVBN\novercome\tVBP\novercommitted\tVBN\noverdoing\tVBG\noverdone\tVBN\noverdosed\tVBN\noverdosing\tVBG\noveremphasize\tVB\noverflowing\tVBG\noverhanging\tVBG\noverhauled\tVBN\noverhauling\tVBG\noverhaul\tVB\noverheated\tVBN\noverheating\tVBG\noverlaid\tVBN\noverlapping\tVBG\noverlap\tVB\noverlap\tVBP\noverlays\tVBZ\noverlooked\tVBD\noverlooked\tVBN\noverlooking\tVBG\noverlooks\tVBZ\noverlook\tVB\noverlook\tVBP\nover-magazined\tVBN\noverpaid\tVBD\noverpaid\tVBN\noverpaying\tVBG\noverpay\tVB\noverplanted\tVBN\noverpower\tVB\noverpriced\tVBN\noverpurchase\tVB\noverreacted\tVBN\noverreacting\tVBG\nOverreacting\tVBG\noverreact\tVB\noverreact\tVBP\noverride\tVB\noverriding\tVBG\noverrode\tVBD\noverruled\tVBD\noverruled\tVBN\noverrule\tVB\noverruling\tVBG\noverruns\tNNS\noversaw\tVB\noversaw\tVBD\noverseeing\tVBG\noverseen\tVBN\noverseers\tNNS\noversees\tVBZ\noversee\tVB\novershadowed\tVBD\novershadowed\tVBN\novershadowing\tVBG\noversimplified\tVBN\noversold\tVB\noversold\tVBN\noverstated\tVBD\noverstated\tVBN\noverstate\tVB\noverstating\tVBG\noverstrained\tVBN\noversubscribed\tVBN\novertaken\tVBN\novertaxed\tVBN\noverthrowing\tVBG\noverthrown\tVBN\noverthrow\tVB\novertures\tNNS\noverturned\tVBD\noverturned\tVBN\noverturning\tVBG\noverturn\tVB\noverused\tVBN\novervalued\tVBD\novervalued\tVBN\noverweighted\tVBN\noverwhelmed\tVBD\noverwhelmed\tVBN\noverwhelming\tVBG\noverwhelm\tVB\noverwhelm\tVBP\noverworking\tVBG\nowed\tVBD\nowed\tVBN\nowes\tVBZ\nowe\tVB\nowe\tVBP\nowing\tVBG\nowned\tVBD\nowned\tVBN\nowners\tNNS\nOwners\tNNS\nowning\tVBG\nOwning\tVBG\nowns\tVBZ\nown\tVB\nown\tVBN\nown\tVBP\npaced\tVBN\npacemakers\tNNS\npace\tVB\npacified\tVBD\npackaged-goods\tNNS\nPackaged-goods\tNNS\npackaged\tVBN\npackages\tNNS\nPackages\tNNS\npackages\tVBZ\npackage\tVB\npackaging\tVBG\npacked\tVBD\npacked\tVBN\npackets\tNNS\npacking\tVBG\npacks\tNNS\npacks\tVBZ\npack\tVB\npack\tVBP\nPACs\tNNS\nPACS\tNNS\npacts\tNNS\npaddles\tNNS\npaeans\tNNS\npages\tNNS\npaid\tVBD\npaid\tVBN\nPaid\tVBN\npaid\tVBN|JJ\nPaineWebber\tVB\npains\tNNS\npainted\tVBD\npainted\tVBN\npainters\tNNS\npaintings\tNNS\nPaintings\tNNS\npainting\tVBG\npaints\tNNS\npaint\tVB\npaint\tVBP\npaired\tVBN\npairs\tNNS\npair\tVB\npalazzi\tNNS\npaled\tVB\npaled\tVBD\npales\tVBZ\npale\tVBP\npalms\tNNS\npalmtops\tNNS\nPalmtops\tNNS\npalm\tVB\npals\tNNS\npampers\tVBZ\npamphlets\tNNS\npancakes\tNNS\npandering\tVBG\npanelists\tNNS\npanels\tNNS\npangs\tNNS\npanicked\tVBD\npanicking\tVBG\npanics\tNNS\npanic\tVB\npanjandrums\tNNS\npanned\tVBD\npanned\tVBN\npanning\tVBG\nPantages\tNNS\npanties\tNNS\npants\tNNS\npan\tVB\npaper-goods\tNNS\nPAPER\tNNS\npaper-products\tNNS\npapers\tNNS\nPAPERS\tNNS\nparachute\tVB\nparachuting\tVBG\nparades\tNNS\nparakeets\tNNS\nparallels\tNNS\nparallels\tVBZ\nparallel\tVB\nparalyzed\tVBN\nparalyzing\tVBG\nParamedics\tNNS\nparameters\tNNS\nparaphernalia\tNNS\nparaphrase\tVBP\nparasites\tNNS\nparastatals\tNNS\nparcels\tNNS\nparcel\tVB\nparcel\tVBP\nparched\tVBN\npardoned\tVBD\npared\tVBD\npared\tVBN\nparents\tNNS\nParents\tNNS\npare\tVB\nParibas\tNNS\nparimutuels\tNNS\nparing\tVBG\nparishes\tNNS\nparishioners\tNNS\nparities\tNNS\nparked\tVBD\nparked\tVBN\nparking\tVBG\nparks\tNNS\npark\tVB\nPark\tVB\nparlors\tNNS\nparried\tVBD\nparry\tVB\npartake\tVB\nparted\tVBD\nparticipants\tNNS\nParticipants\tNNS\nparticipated\tVBD\nparticipated\tVBN\nparticipates\tVBZ\nparticipate\tVB\nparticipate\tVBP\nparticipating\tVBG\nparticipations\tNNS\nparticulars\tNNS\nparties\tNNS\nParties\tNNS\nparties\tVBZ\nparting\tVBG\npartisans\tNNS\nPartisans\tNNS\npartnerships\tNNS\nPartnerships\tNNS\npartners\tNNS\nPartners\tNNS\nPARTNERS\tNNS\nparts\tNNS\nParts\tNNS\npart\tVB\npashas\tNNS\npassages\tNNS\npassed\tVBD\npassed\tVBN\nPassed\tVBN\npassenger-kilometers\tNNS\npassengers\tNNS\npassers-by\tNNS\npasses\tNNS\npasses\tVBZ\npassing\tVBG\npassions\tNNS\npassports\tNNS\npass\tVB\npass\tVBP\npasswords\tNNS\npasted\tVBN\npastels\tNNS\npasteurized\tVBN\npastimes\tNNS\npastors\tNNS\npatched\tVBN\npatch\tVB\npatented\tVBD\npatented\tVBN\npatents\tNNS\npaths\tNNS\npatients\tNNS\nPatients\tNNS\npatrolled\tVBN\npatrols\tNNS\npatronized\tVBN\npatronize\tVB\npatronizing\tVBG\npatrons\tNNS\npatterned\tVBN\npatterns\tNNS\nPatterns\tNNS\npaused\tVBD\npauses\tNNS\npauses\tVBZ\npause\tVB\npause\tVBP\npausing\tVBG\npaved\tVBD\npaved\tVBN\npaves\tVBZ\npave\tVB\npawing\tVBG\npawning\tVBG\npawns\tNNS\npayables\tNNS\npaychecks\tNNS\npayers\tNNS\nPayers\tNNS\npaying\tVBG\nPaying\tVBG\npayments\tNNS\nPayments\tNNS\nPAYMENTS\tNNS\npayoffs\tNNS\npayouts\tNNS\nPayouts\tNNS\npayrolls\tNNS\npays\tVBZ\nPAYS\tVBZ\npay\tVB\nPay\tVB\nPAY\tVB\npay\tVBP\nPCBs\tNNS\nPCs\tNNS\npeacemakers\tNNS\npeaches\tNNS\npeaked\tVBD\npeaked\tVBN\npeaking\tVBG\npeaks\tNNS\npeak\tVB\npealing\tVBG\npeals\tNNS\npeanuts\tNNS\npearls\tNNS\npears\tNNS\npeasants\tNNS\nPeasants\tNNS\npeas\tNNS\npeccadilloes\tNNS\npecks\tNNS\npeck\tVBP\npeculiarities\tNNS\npedaled\tVBN\npedaling\tVBG\npedal\tVB\npeddled\tVBN\npeddles\tVBZ\npeddle\tVB\npeddle\tVBP\npeddling\tVBG\npedestrians\tNNS\nPedigrees\tNNS\nPeeking\tVBG\npeeled\tVBN\npeering\tVBG\npeers\tNNS\npeers\tVBZ\npeer\tVB\npeer\tVBP\npegged\tVBD\npegged\tVBN\npegging\tVBG\npegs\tVBZ\npeg\tVB\npeg\tVBP\npellets\tNNS\npenalized\tVBN\npenalizes\tVBZ\npenalize\tVB\npenalties\tNNS\npence\tNNS\npencils\tNNS\nPencils\tNNS\nPENCILS\tNNS\nPencil\tVB\npending\tVBG\nPending\tVBG\npending\tVBG|JJ\npenetrated\tVBN\npenetrate\tVB\npenetrating\tVBG\npenises\tNNS\npenned\tVBN\npennies\tNNS\npensions\tNNS\npens\tNNS\npeopled\tVBN\npeople\tNNS\nPeople\tNNS\npeoples\tNNS\npeppered\tVBD\npeppering\tVBG\npeppers\tNNS\nPepsi\tNNS\nperceived\tVBD\nperceived\tVBN\nperceives\tVBZ\nperceive\tVBP\npercentages\tNNS\nperceptions\tNNS\nperched\tVBN\nperfected\tVBN\nperformances\tNNS\nperformed\tVBD\nperformed\tVBN\nperformers\tNNS\nperforming-arts\tNNS\nperforming\tVBG\nPerforming\tVBG\nperforms\tVBZ\nperform\tVB\nperform\tVBP\nperils\tNNS\nperiodicals\tNNS\nperiods\tNNS\nPeriods\tNNS\nperipherals\tNNS\nperishables\tNNS\nperished\tVBN\nperked\tVBD\nperked\tVBN\nperks\tNNS\npermeated\tVBD\npermeating\tVBG\npermits\tNNS\npermits\tVBZ\npermitted\tVBD\npermitted\tVBN\npermitting\tVBG\npermit\tVB\npermit\tVBP\nperpetrated\tVBN\nperpetuates\tVBZ\nperpetuate\tVB\nperpetuate\tVBP\nperpetuating\tVBG\npersecuted\tVBN\npersecuting\tVBG\npersisted\tVBD\npersisting\tVBG\npersists\tVBZ\npersist\tVB\npersist\tVBP\npersonalities\tNNS\npersonalized\tVBN\npersonalize\tVB\npersonnel\tNNS\nPersonnel\tNNS\npersons\tNNS\npersuaded\tVBD\npersuaded\tVBN\npersuades\tVBZ\npersuade\tVB\npersuading\tVBG\nPersuading\tVBG\npertains\tVBZ\nperturbed\tVBD\nperuse\tVB\nperuse\tVBP\npervaded\tVBD\npervade\tVBP\nperversities\tNNS\npesatas\tNNS\npesetas\tNNS\npesos\tNNS\npessimists\tNNS\nPestered\tVBN\npesticides\tNNS\npetitioned\tVBD\npetitions\tNNS\npetition\tVB\npetrochemicals\tNNS\npets\tNNS\nPETS\tNNS\npharaohs\tNNS\npharmaceuticals\tNNS\nPharmaceuticals\tNNS\npharmacies\tNNS\npharmacists\tNNS\nphased\tVBD\nphased\tVBN\nphases\tNNS\nphase\tVB\nphasing\tVBG\nphenomena\tNNS\nPhillips\tNNS\nphilosophers\tNNS\nphilosophies\tNNS\nphobias\tNNS\nphoned\tVBD\nphoned\tVBN\nphones\tNNS\nphones\tVBZ\nphone\tVBP\nphoning\tVBG\nphotocopiers\tNNS\nphotocopying\tVBG\nphotocopy\tVB\nphotofinishers\tNNS\nphotographed\tVBN\nphotographers\tNNS\nphotographing\tVBG\nphotographs\tNNS\nphotographs\tVBZ\nphotos\tNNS\nphrases\tNNS\nphysicians\tNNS\nphysics\tNNS\npianos\tNNS\npiasters\tNNS\npicked\tVBD\npicked\tVBN\npickers\tNNS\npicking\tVBG\nPickin\tVBG\npickles\tNNS\npicks\tNNS\npicks\tVBZ\npickups\tNNS\npick\tVB\nPick\tVB\npick\tVBP\npictured\tVBN\npictures\tNNS\nPictures\tNNS\npictures\tVBZ\npicture\tVBP\npicturing\tVBG\npieced\tVBN\npieces\tNNS\nPieces\tNNS\npiece\tVB\nPierce\tVB\npiers\tNNS\npies\tNNS\npiggybacking\tVBG\nPiggybacking\tVBG\npiglets\tNNS\npigments\tNNS\npigs\tNNS\npiled\tVBD\npiled\tVBN\npiles\tNNS\npile\tVB\npile\tVBP\npilings\tNNS\npiling\tVBG\nPILING\tVBG\npillars\tNNS\npilloried\tVBN\npillorying\tVBG\npillowcases\tNNS\npillows\tNNS\npills\tNNS\npilots\tNNS\nPilots\tNNS\npimps\tNNS\npinched\tVBD\npinched\tVBN\npinching\tVBG\npinch\tVB\npine\tVBP\npinging\tVBG\nping\tVB\npinned\tVBN\npinning\tVBG\npin-pointed\tVBN\npinpointed\tVBN\npinpoint\tVB\npins\tNNS\npins\tVBZ\npints\tNNS\npin\tVB\npioneered\tVBD\npioneers\tNNS\npioneer\tVB\npiped\tVBD\npiped\tVBN\npipelines\tNNS\npipes\tNNS\npirated\tVBN\npirates\tNNS\npiroghi\tNNS\npistils\tNNS\npistols\tNNS\npistons\tNNS\npitched\tVBD\npitched\tVBN\npitchers\tNNS\npitches\tNNS\npitches\tVBZ\npitching\tVBG\npitchmen\tNNS\npitch\tVB\npitch\tVBP\npitfalls\tNNS\nPitfalls\tNNS\npits\tNNS\npits\tVBZ\npitted\tVBD\npitted\tVBN\npitting\tVBG\npit\tVB\npivot\tVB\npizzas-with-everything\tNNS\npizzerias\tNNS\nplacated\tVBN\nplacate\tVB\nplaced\tVBD\nplaced\tVBN\nplacements\tNNS\nplaces\tNNS\nplaces\tVBZ\nplace\tVB\nPlace\tVB\nplace\tVBP\nplacing\tVBG\nplagued\tVBD\nplagued\tVBN\nplague\tVB\nplaguing\tVBG\nplainclothes\tNNS\nplains\tNNS\nPlains\tNNS\nplaintiffs\tNNS\nPlaintiffs\tNNS\nplanes\tNNS\nplanets\tNNS\nplanks\tNNS\nplanned\tVBD\nplanned\tVBN\nplanners\tNNS\nPlanners\tNNS\nplanning\tVBG\nplans\tNNS\nPlans\tNNS\nplans\tVBP\nplans\tVBZ\nPlans\tVBZ\nPLANS\tVBZ\nplantations\tNNS\nplanted\tVBD\nplanted\tVBN\nplanting\tVBG\nplants\tNNS\nPLANTS\tNNS\nplant\tVB\nplan\tVB\nplan\tVBP\nplastics\tNNS\nPlastics\tNNS\nplates\tNNS\nplatforms\tNNS\nplatitudes\tNNS\nplaudits\tNNS\nplayed\tVBD\nplayed\tVBN\nplayers\tNNS\nPlayers\tNNS\nplaygrounds\tNNS\nplaying\tVBG\nPlaying\tVBG\nplayoffs\tNNS\nplays\tNNS\nplays\tVBZ\nPlays\tVBZ\nplay\tVB\nplay\tVBP\npleaded\tVBD\npleaded\tVBN\npleadings\tNNS\npleading\tVBG\nplead\tVB\npleasantries\tNNS\npleased\tVBD\npleased\tVBN\npleases\tVBZ\nplease\tVB\nPlease\tVB\nplease\tVBP\npleasing\tVBG\npleas\tNNS\npleasures\tNNS\nplea\tVB\npledged\tVBD\npledged\tVBN\npledges\tNNS\npledges\tVBZ\npledging\tVBG\npliers\tNNS\nplies\tVBZ\nplights\tNNS\nplods\tVBZ\nplots\tNNS\nplotted\tVBD\nplotters\tNNS\nplotting\tVBG\nplot\tVB\nplowed\tVBD\nplowed\tVBN\nplows\tVBZ\nplow\tVB\nploys\tNNS\nPloys\tNNS\nplucked\tVBN\npluck\tVB\nplugged\tVBD\nplugging\tVBG\nPlugging\tVBG\nplug\tVB\nplug\tVBP\nplummeted\tVBD\nplummeted\tVBN\nplummeting\tVBG\nplummet\tVB\nplummet\tVBP\nplunged\tVBD\nplunged\tVBN\nplunges\tNNS\nplunges\tVBZ\nplunge\tVB\nplunge\tVBP\nplunging\tVBG\nplunking\tVBG\npluses\tNNS\nPluses\tNNS\nplying\tVBG\nPoachers\tNNS\npoaching\tVBG\npocketing\tVBG\npockets\tNNS\nPockets\tNNS\npockets\tVBZ\npocket\tVB\npockmarked\tVBN\npointed\tVBD\npointed\tVBN\npointers\tNNS\npointing\tVBG\npoints\tNNS\npoints\tVBZ\npoint\tVB\npoint\tVBP\npoised\tVBN\npoisoned\tVBN\npoisons\tNNS\npoked\tVBD\npokes\tVBZ\npoking\tVBG\npolarized\tVBN\npoles\tNNS\nPoles\tNNS\npolice\tNNS\nPolice\tNNS\npolices\tNNS\npolice\tVB\npolicies\tNNS\nPolicies\tNNS\npolicing\tVBG\npolicyholders\tNNS\npolicy-makers\tNNS\npolicy-making\tVBG\npolished\tVBD\npolished\tVBN\npolishing\tVBG|NN\npolish\tVB\npoliticians\tNNS\nPoliticians\tNNS\npoliticized\tVBN\npolitico-plaintiffs\tNNS\npolitics\tNNS\nPolitics\tNNS\nPOLITICS\tNNS\npolled\tVBD\npolled\tVBN\npollen-producing\tVBG\npollinated\tVBN\npollinate\tVB\npollinate\tVBP\npollinating\tVBG\npolls\tNNS\nPolls\tNNS\npollsters\tNNS\npolls\tVBZ\npollutants\tNNS\npolluters\tNNS\npollute\tVB\npolluting\tVBG\npoll\tVB\npols\tNNS\npoltergeists\tNNS\npolymers\tNNS\npolyols\tNNS\npolyps\tNNS\npolyrhythms\tNNS\npondering\tVBG\nponder\tVB\nponder\tVBP\nponds\tNNS\nponied\tVBD\nponies\tNNS\npontificate\tVBP\nponying\tVBG\npooled\tVBN\npooling\tVBG\npools\tNNS\npool\tVBP\npopping\tVBG\npops\tVBZ\npopularized\tVBD\npopularized\tVBN\npopularize\tVB\npopulated\tVBN\npopulate\tVB\npopulating\tVBG\npopulations\tNNS\npop\tVB\npop\tVBP\nporcelains\tNNS\nporches\tNNS\npored\tVBD\npored\tVBN\npores\tVBZ\npork-barrelers\tNNS\nportables\tNNS\nportends\tVBZ\nportend\tVB\nportfolios\tNNS\nPortfolios\tNNS\nPORTING\tVBG\nportions\tNNS\nportraits\tNNS\nportrayals\tNNS\nportrayed\tVBD\nportrayed\tVBN\nportraying\tVBG\nportrays\tVBZ\nportray\tVB\nportray\tVBP\nports\tNNS\nPorts\tNNS\nposed\tVBD\nposed\tVBN\nposes\tVBZ\npose\tVB\npose\tVBP\nposing\tVBG\npositioned\tVBD\npositioned\tVBN\npositions\tNNS\npositions\tVBZ\nposition\tVB\npossessed\tVBD\npossessed\tVBN\npossesses\tVBZ\npossessing\tVBG\npossessions\tNNS\npossess\tVB\npossess\tVBP\npossibilities\tNNS\npostcards\tNNS\nposted\tVBD\nPosted\tVBD\nposted\tVBN\nPosted\tVBN\nposters\tNNS\npostings\tNNS\nposting\tVBG\npostmarked\tVBN\npostmarks\tNNS\npostponed\tVBD\npostponed\tVBN\npostpone\tVB\npostpone\tVBP\npostponing\tVBG\nposts\tNNS\nposts\tVBZ\nPosts\tVBZ\nposturing\tVBG\npost\tVB\nPOTABLES\tNNS\npotatoes\tNNS\npotentates\tNNS\npotentialities\tNNS\npotholes\tNNS\npots\tNNS\npot\tVB\npouches\tNNS\npounce\tVB\npounded\tVBN\npounding\tVBG\npounds\tNNS\npound\tVB\npoured\tVBD\npoured\tVBN\npouring\tVBG\npours\tVBZ\npour\tVB\npour\tVBP\npowders\tNNS\npowered\tVBN\npowerhouses\tNNS\npowers\tNNS\npower\tVB\npracticed\tVBD\npracticed\tVBN\npractices\tNNS\npractice\tVB\npracticing\tVBG\npractitioners\tNNS\npragmatists\tNNS\nprairies\tNNS\npraised\tVBD\npraised\tVBN\npraises\tNNS\npraises\tVBZ\npraise\tVB\npraise\tVBP\npraising\tVBG\nprancing\tVBG\nprayers\tNNS\npraying\tVBG\npreaching\tVBG\npreach\tVB\npreach\tVBP\npre-approved\tVBN\npreapproved\tVBN\nprearranged\tVBN\nprecautions\tNNS\npreceded\tVBN\nprecedents\tNNS\nprecedes\tVBZ\nprecede\tVB\npreceding\tVBG\nprecincts\tNNS\nprecious-metals\tNNS\nPrecious-metals\tNNS\nprecipices\tNNS\nprecipitated\tVBD\nprecipitating\tVBG\nprecluded\tVBN\nprecludes\tVBZ\npreclude\tVB\npredates\tVBZ\npredators\tNNS\npredecessors\tNNS\npredetermined\tVBN\npredicated\tVBN\npredict\\/advocate\tVBP\npredicted\tVBD\npredicted\tVBN\npredicting\tVBG\nPredicting\tVBG\npredictions\tNNS\nPredictions\tNNS\npredicts\tVBZ\npredict\tVB\npredict\tVBP\npredispose\tVB\npre-empted\tVBD\npre-empt\tVB\npreferences\tNNS\npreferred\tVBD\npreferred\tVBG\npreferred\tVBN\npreferring\tVBG\nprefers\tVBZ\nprefer\tVB\nprefer\tVBP\nprejudiced\tVBN\nprejudices\tNNS\npremiered\tVBD\npremieres\tNNS\npremiere\tVB\npremiering\tVBG\npremises\tNNS\npremiums\tNNS\npreoccupied\tVBN\nprepaid\tVB\nprepaid\tVBN\npreparations\tNNS\npreparatives\tNNS\nprepared\tVBD\nprepared\tVBN\npreparers\tNNS\nprepares\tVBZ\nprepare\tVB\nprepare\tVBP\npreparing\tVBG\nprepaying\tVBG\nprepayments\tNNS\nPrepayments\tNNS\nprepay\tVB\nprepping\tVBG\npre-registered\tVBN\npre-register\tVB\nprerogatives\tNNS\npresages\tVBZ\npresage\tVB\npreschoolers\tNNS\nprescribed\tVBN\nprescribes\tVBZ\nprescribe\tVB\nprescribe\tVBP\nprescriptions\tNNS\npresentations\tNNS\npresented\tVBD\npresented\tVBN\npresenters\tNNS\npresenting\tVBG\npresents\tNNS\npresents\tVBZ\npresent\tVB\nPresent\tVB\npresent\tVBP\npreserved\tVBD\npreserved\tVBN\npreserves\tVBZ\npreserve\tVB\npreserving\tVBG\npresided\tVBD\npresided\tVBN\npresidents\tNNS\nPresidents\tNNS\npresides\tVBZ\npresiding\tVBG\npre-signed\tVBN\npressed\tVBD\npressed\tVBN\nPressed\tVBN\npresses\tNNS\npressing\tVBG\npressured\tVBD\npressured\tVBN\npressures\tNNS\nPressures\tNNS\npressure\tVB\npressure\tVBP\npressuring\tVBG\npress\tVB\nPress\tVB\npress\tVBP\npresumed\tVBN\npresumes\tVBZ\npresuming\tVBG\npretending\tVBG\npretend\tVB\nPretend\tVB\npretensions\tNNS\npre-tested\tVBN\npre-try\tVB\nprevailed\tVBD\nprevailed\tVBN\nprevailing\tVBG\nprevails\tVBZ\nprevail\tVB\nprevail\tVBP\nprevented\tVBD\nprevented\tVBN\nPrevented\tVBN\npreventing\tVBG\nprevents\tVBZ\nprevent\tVB\nPrevent\tVB\nprevent\tVBP\npreviewing\tVBG\npreviews\tNNS\nprey\tVBP\npriced\tVBD\npriced\tVBN\nprice-earnings\tNNS\nprice\\/earnings\tNNS\nprices\tNNS\nPrices\tNNS\nPRICES\tNNS\nprices\tVBZ\nprice\tVB\nprice\tVBP\npricings\tNNS\npricing\tVBG\npricked\tVBN\npriests\tNNS\nprimed\tVBN\nprimitives\tNNS\nprinces\tNNS\nprincipals\tNNS\nprinciples\tNNS\nprinted\tVBD\nprinted\tVBN\nprinters\tNNS\nprinting\tVBG\nprinting\tVBG|NN\nprintouts\tNNS\nprints\tNNS\nprints\tVBZ\nprint\tVB\nprint\tVBP\npriorities\tNNS\nprisoners\tNNS\nprisons\tNNS\nprivatized\tVBN\nprivatize\tVB\nprivatizing\tVBG\nprivileges\tNNS\nprized\tVBN\nprizes\tNNS\nprobabilities\tNNS\nprobes\tNNS\nprobe\tVB\nprobe\tVBP\nprobing\tVBG\nProbing\tVBG\nproblematics\tNNS\nproblems\tNNS\nprocedures\tNNS\nproceeded\tVBD\nproceeded\tVBN\nproceedings\tNNS\nPROCEEDINGS\tNNS\nproceeding\tVBG\nproceeds\tNNS\nProceeds\tNNS\nproceeds\tVBZ\nproceed\tVB\nproceed\tVBP\nprocessed\tVBD\nprocessed\tVBN\nprocesses\tNNS\nprocesses\tVBZ\nprocessing\tVBG\nProcessing\tVBG\nprocessors\tNNS\nprocess\tVB\nprocess\tVBP\nproclaimed\tVBD\nproclaiming\tVBG\nproclaims\tVBZ\nproclaim\tVB\nproclaim\tVBP\nproclamations\tNNS\nprocure\tVB\nprodded\tVBN\nprodding\tVBG\nprods\tVBZ\nproduced\tVBD\nproduced\tVBN\nproducers\tNNS\nProducers\tNNS\nproduces\tVBZ\nproduce\tVB\nProduce\tVB\nproduce\tVBP\nproducing\tVBG\nproductions\tNNS\nproducts\tNNS\nPRODUCTS\tNNS\nprod\tVB\nprofessed\tVBD\nprofessed\tVBN\nprofesses\tVBZ\nprofessionals\tNNS\nProfessionals\tNNS\nprofessions\tNNS\nprofessors\tNNS\nprofess\tVBP\nproffered\tVBD\nproffered\tVBN\nprofferred\tVBN\nprofiled\tVBN\nprofiles\tNNS\nprofile\tVB\nprofited\tVBD\nprofited\tVBN\nprofiteering\tVBG\nprofiteers\tNNS\nprofiting\tVBG\nprofit-sharing\tNNS\nprofits\tNNS\nProfits\tNNS\nPROFITS\tNNS\nprofits\tVBZ\nprofit-taking\tNNS\nprofitting\tVBG\nprofit\tVB\nprofit\tVBP\nprogenitors\tNNS\nprognosticators\tNNS\nprogrammed\tVBN\nprogrammers\tNNS\nprogramming\tVBG\nprograms\tNNS\nPrograms\tNNS\nprogram\tVB\nprogram\tVBP\nprogressed\tVBD\nprogressed\tVBN\nprogresses\tVBZ\nprogressing\tVBG\nprogressions\tNNS\nprogressives\tNNS\nprogress\tVB\nprohibited\tVBD\nprohibited\tVBN\nprohibiting\tVBG\nprohibitions\tNNS\nprohibits\tVBZ\nprohibit\tVB\nprohibit\tVBP\nprojected\tVBD\nprojected\tVBN\nProjected\tVBN\nprojecting\tVBG\nProjecting\tVBG\nprojections\tNNS\nprojectors\tNNS\nprojects\tNNS\nProjects\tNNS\nprojects\tVBZ\nproject\tVB\nproject\tVBP\nproliferated\tVBN\nproliferate\tVBP\nproliferating\tVBG\nprolonged\tVBN\nprolong\tVB\npromised\tVBD\npromised\tVBN\npromises\tNNS\npromises\tVBZ\nPromises\tVBZ\npromise\tVB\npromise\tVBP\npromising\tVBG\npromoted\tVBD\npromoted\tVBN\npromoters\tNNS\npromotes\tVBZ\npromote\tVB\npromote\tVBP\npromoting\tVBG\npromotions\tNNS\nprompted\tVBD\nprompted\tVBN\nPrompted\tVBN\nprompting\tVBG\nprompts\tVBZ\nprompt\tVB\npromulgated\tVBD\nprongs\tNNS\npronounced\tVBD\npronounced\tVBN\npronouncements\tNNS\npronounces\tVBZ\nproof-of-purchases\tNNS\nproofreading\tVBG\npropagandists\tNNS\npropagandizes\tVBZ\npropagandize\tVB\npropelled\tVBD\npropelled\tVBN\npropelling\tVBG\npropel\tVB\nproperties\tNNS\nproponents\tNNS\nProponents\tNNS\nproportions\tNNS\nproposals\tNNS\nProposals\tNNS\nPROPOSALS\tNNS\nproposed\tVBD\nproposed\tVBN\nproposes\tVBZ\npropose\tVB\npropose\tVBP\nproposing\tVBG\npropositions\tNNS\npropped\tVBD\npropped\tVBN\npropping\tVBG\nproprietorships\tNNS\nproprietors\tNNS\npropsed\tVBN\nprops\tNNS\nprop\tVB\nproscribed\tVBN\nproscribes\tVBZ\nprosecuted\tVBD\nprosecuted\tVBN\nprosecute\tVB\nprosecuting\tVBG\nprosecutions\tNNS\nprosecutors\tNNS\nProsecutors\tNNS\nPROSECUTORS\tNNS\npros\tNNS\nPros\tNNS\nprospects\tNNS\nProspects\tNNS\nPROSPECTS\tNNS\nprospectuses\tNNS\nprospered\tVBN\nprosper\tVB\nprostitutes\tNNS\nprotected\tVBD\nprotected\tVBN\nprotecting\tVBG\nprotections\tNNS\nprotectors\tNNS\nprotects\tVBZ\nprotect\tVB\nprotect\tVBP\nproteges\tNNS\nproteins\tNNS\nproteins\tVBZ\nprotested\tVBD\nprotested\tVBN\nprotesters\tNNS\nprotesting\tVBG\nprotestors\tNNS\nprotests\tNNS\nprotests\tVBZ\nprotest\tVB\nprotocols\tNNS\nprototypes\tNNS\nproved\tVBD\nproved\tVBN\nproven\tVBN\nProverbs\tNNS\nproves\tVBZ\nProves\tVBZ\nprove\tVB\nprove\tVBP\nprovided\tVBD\nprovided\tVBN\nProvided\tVBN\nproviders\tNNS\nprovides\tVBZ\nprovide\tVB\nProvide\tVB\nprovide\tVBP\nProvide\tVBP\nproviding\tVBG\nprovinces\tNNS\nproving\tVBG\nprovisioning\tVBG\nprovisions\tNNS\nprovoked\tVBD\nprovoked\tVBN\nprovoke\tVB\nprovoking\tVBG\nproxies\tNNS\npruned\tVBN\nprune\tVB\npseudo-lobbyists\tNNS\npsychics\tNNS\npsychologists\tNNS\nPsychologists\tNNS\npublications\tNNS\npublicized\tVBN\npublicize\tVB\npublic-relations\tNNS\npublic-works\tNNS\nPublic-works\tNNS\npublished\tVBD\npublished\tVBN\nPublished\tVBN\npublishers\tNNS\nPublishers\tNNS\npublishes\tVBZ\npublishing\tVBG\npublish\tVB\npubs\tNNS\npuffers\tNNS\npull-backs\tNNS\npullbacks\tNNS\npulled\tVBD\npulled\tVBN\npulling\tVBG\npullouts\tNNS\npulls\tNNS\npulls\tVBZ\npull\tVB\nPull\tVB\npull\tVBP\npulverizing\tVBG\npummeled\tVBD\npummeled\tVBN\npummel\tVB\npumped\tVBD\npumped\tVBN\npumping\tVBG\npumps\tNNS\npump\tVB\npump\tVBP\npunched\tVBD\npunched\tVBN\npunchers\tNNS\npunching\tVBG\nPunching\tVBG\npunch\tVB\npunch\tVBP\npundits\tNNS\npunished\tVBN\npunishing\tVBG\npunish\tVB\npuns\tNNS\npunts\tNNS\npupils\tNNS\npuppets\tNNS\npuppies\tNNS\npurchased\tVBD\npurchased\tVBN\npurchasers\tNNS\npurchases\tNNS\npurchases\tVBZ\npurchase\tVB\npurchase\tVBP\npurchasing\tVBG\npurchasing\tVBG|NN\npurged\tVBD\npurged\tVBN\npurges\tVBZ\npurge\tVB\npurging\tVBG\npurists\tNNS\npur-poises\tNNS\npurports\tVBZ\npurport\tVBP\npurposes\tNNS\npurrs\tVBZ\npurse-snatchings\tNNS\npurses\tNNS\npursued\tVBD\npursued\tVBN\npursuers\tNNS\npursues\tVBZ\npursue\tVB\npursue\tVBP\npursuing\tVBG\npursuits\tNNS\npushed\tVBD\npushed\tVBN\npushers\tNNS\npushes\tVBZ\npushing\tVBG\nPushing\tVBG\npush\tVB\npush\tVBP\nPush\tVBP\nputs\tNNS\nputs\tVBZ\nPuts\tVBZ\nputting\tVBG\nPutting\tVBG\nput\tVB\nPut\tVB\nput\tVBD\nPut\tVBD\nput\tVBN\nPut\tVBN\nput\tVBP\nput\tVBP|VB\npuzzled\tVBD\npuzzled\tVBN\npuzzles\tNNS\npuzzle\tVB\npyramiding\tVBG\npyramids\tNNS\nquacks\tNNS\nquacks\tVBZ\nquadrupeds\tNNS\nquadrupled\tVBD\nquadrupled\tVBN\nquadruples\tVBZ\nquadrupling\tVBG\nquakes\tNNS\nqualifications\tNNS\nqualified\tVBD\nqualified\tVBN\nqualifies\tVBZ\nqualifying\tVBG\nqualify\tVB\nqualify\tVBP\nqualities\tNNS\nqualms\tNNS\nquantified\tVBN\nquantify\tVB\nquantities\tNNS\nquarreling\tVBG\nquarrel\tVB\nquarterbacks\tNNS\nquarters\tNNS\nquartets\tNNS\nquashed\tVBD\nquashed\tVBN\nquashing\tVBG\nqueers\tNNS\nquell\tVB\nqueried\tVBN\nqueries\tNNS\nqueries\tVBZ\nquestioned\tVBD\nquestioned\tVBN\nQuestioned\tVBN\nquestioning\tVBG\nquestions\tNNS\nQuestions\tNNS\nquestions\tVBZ\nquestion\tVB\nquestion\tVBP\nqueues\tNNS\nqueuing\tVBG\nquibbling\tVBG\nquicken\tVB\nquieted\tVBD\nquieted\tVBN\nquieting\tVBG\nquiet\tVB\nquipped\tVBD\nquips\tNNS\nquips\tVBZ\nQuips\tVBZ\nquirks\tNNS\nquits\tVBZ\nquitting\tVBG\nquit\tVB\nquit\tVBD\nquit\tVBN\nquit\tVBP\nquivers\tNNS\nquiz\tVB\nquota-cheaters\tNNS\nquotas\tNNS\nquotations\tNNS\nquoted\tVBD\nquoted\tVBN\nquotes\tNNS\nquotes\tVBZ\nquote\tVB\nquote\tVBP\nquoting\tVBG\nQuoting\tVBG\nraced\tVBD\nracehorses\tNNS\nraces\tNNS\nracetracks\tNNS\nracing\tVBG\nRacing\tVBG\nracked\tVBD\nracked\tVBN\nracketeering\tVBG\nrackets\tNNS\nracking\tVBG\nracks\tNNS\nradar-eluding\tVBG\nradicals\tNNS\nradioing\tVBG\nradios\tNNS\nrafters\tNNS\nraged\tVBD\nraged\tVBN\nrages\tNNS\nrage\tVB\nraging\tVBG\nraided\tVBD\nraided\tVBN\nraiders\tNNS\nraiding\tVBG\nraids\tNNS\nraid\tVB\nrailbikes\tNNS\nrailcars\tNNS\nrailings\tNNS\nrailroads\tNNS\nrails\tNNS\nrails\tVBZ\nrailways\tNNS\nrained\tVBD\nrains\tNNS\nrain\tVB\nraised\tVBD\nraised\tVBN\nraisers\tNNS\nraises\tNNS\nraises\tVBZ\nraise\tVB\nRaise\tVB\nraise\tVBP\nraising\tVBG\nRaising\tVBG\nraked\tVBD\nraking\tVBG\nrallied\tVBD\nRALLIED\tVBD\nrallied\tVBN\nrallies\tNNS\nrallies\tVBZ\nrallying\tVBG\nrally\tVB\nRally\tVB\nrally\tVBP\nrambled\tVBD\nramifications\tNNS\nrammed\tVBD\nramparts\tNNS\nramps\tNNS\nramp\tVB\nranchers\tNNS\nranches\tNNS\nrand\tNNS\nranged\tVBD\nranged\tVBN\nrangers\tNNS\nranges\tNNS\nranges\tVBZ\nrange\tVB\nrange\tVBP\nranging\tVBG\nrang\tVBD\nranked\tVBD\nranked\tVBN\nrankings\tNNS\nranking\tVBG\nrankled\tVBN\nranks\tNNS\nranks\tVBZ\nrank\tVB\nrank\tVBP\nran\tVBD\nraped\tVBD\nraped\tVBN\nrapeseeds\tNNS\nraping\tVBG\nrapists\tNNS\nraptors\tNNS\nrarefied\tVBN\nratcheting\tVBG\nrated\tVBD\nrated\tVBN\nRated\tVBN\nratepayers\tNNS\nrates\tNNS\nRates\tNNS\nRATES\tNNS\nrates\tVBZ\nrate\tVB\nrate\tVBP\nratified\tVBD\nratified\tVBN\nratifying\tVBG\nratify\tVB\nratings\tNNS\nRatings\tNNS\nrating\tVBG\nrationalizations\tNNS\nrationalize\tVB\nrationalizing\tVBG\nrationed\tVBN\nratios\tNNS\nRATIOS\tNNS\nrats\tNNS\nrattled\tVBD\nRATTLED\tVBD\nrattled\tVBN\nrattle\tVB\nrattling\tVBG\nrat\tVB\nravaged\tVBN\nravages\tNNS\nRAVAGES\tNNS\nraves\tVBZ\nrave\tVB\nravines\tNNS\nraw-materials\tNNS\nrays\tNNS\nrazed\tVBN\nrazing\tVBG\nreached\tVBD\nreached\tVBN\nReached\tVBN\nreaches\tNNS\nreaches\tVBZ\nreaching\tVBG\nReaching\tVBG\nreach\tVB\nreach\tVBP\nreacted\tVBD\nreacted\tVBN\nreacting\tVBG\nreactions\tNNS\nreactivated\tVBD\nreactivated\tVBN\nreactors\tNNS\nreacts\tVBZ\nreact\tVB\nreact\tVBP\nreaders\tNNS\nReaders\tNNS\nreadied\tVBD\nreadings\tNNS\nreading\tVBG\nReading\tVBG\nreadmit\tVB\nreads\tVBZ\nread\tVB\nRead\tVB\nread\tVBD\nread\tVBD|VBP\nread\tVBN\nread\tVBP\nread\tVBP|VBD\nreaffirmed\tVBD\nreaffirming\tVBG\nreaffirms\tVBZ\nreaffirm\tVB\nReaganauts\tNNS\nrealestate\tVB\nrealigned\tVBD\nrealigning\tVBG\nrealignments\tNNS\nrealign\tVB\nrealists\tNNS\nrealities\tNNS\nrealized\tVBD\nrealized\tVBN\nrealizes\tVBZ\nrealize\tVB\nrealize\tVBP\nrealizing\tVBG\nreallocated\tVBN\nreallocate\tVB\nrealms\tNNS\nreams\tNNS\nreaped\tVBD\nreaped\tVBN\nreaping\tVBG\nreappointed\tVBN\nreapportion\tVBP\nreappraised\tVBD\nreappraised\tVBN\nreappraise\tVB\nreap\tVB\nREAP\tVBP\nRearding\tVBG\nreared\tVBN\nrearing\tVBG\nrearm\tVB\nrearranges\tVBZ\nrearrange\tVB\nreasoned\tVBD\nreasoned\tVBN\nreasons\tNNS\nReasons\tNNS\nreasons\tVBZ\nreason\tVB\nreason\tVBP\nreasserting\tVBG\nreasserts\tVBZ\nreassert\tVB\nreassessing\tVBG\nreassess\tVB\nreassigned\tVBD\nreassigned\tVBN\nreassignments\tNNS\nreassume\tVB\nreassurances\tNNS\nreassured\tVBD\nreassured\tVBN\nreassure\tVB\nreassuring\tVBG\nreauthorize\tVB\nreawakening\tVBG\nrebates\tNNS\nrebelled\tVBD\nrebels\tNNS\nrebounded\tVB\nrebounded\tVBD\nrebounded\tVBN\nrebounding\tVBG\nRebounding\tVBG\nrebounds\tNNS\nrebounds\tVBZ\nrebound\tVB\nrebuffed\tVBD\nrebuffed\tVBN\nrebuilding\tVBG\nRebuilding\tVBG\nrebuild\tVB\nrebuilt\tVBN\nrebuked\tVBD\nrebuts\tVBZ\nrebutted\tVBN\nrebut\tVB\nrecalculated\tVBD\nrecalculating\tVBG\nrecalculations\tNNS\nrecalled\tVBD\nrecalled\tVBN\nrecalling\tVBG\nrecalls\tNNS\nrecalls\tVBZ\nRecalls\tVBZ\nrecall\tVB\nRecall\tVB\nrecall\tVBP\nrecanted\tVBD\nrecanted\tVBN\nrecapitalizations\tNNS\nrecapitalized\tVBN\nrecapture\tVB\nrecede\tVBP\nreceding\tVBG\nreceipts\tNNS\nreceivables\tNNS\nreceived\tVBD\nreceived\tVBN\nreceivers\tNNS\nreceives\tVBZ\nreceive\tVB\nreceive\tVBP\nreceiving\tVBG\nrecentralized\tVBN\nreceptionists\tNNS\nreceptors\tNNS\nrecessed\tVBN\nrecessions\tNNS\nrecharging\tVBG\nrecipes\tNNS\nrecipients\tNNS\nrecites\tVBZ\nreciting\tVBG\nreckoned\tVBN\nreckoning\tVBG\nreckons\tVBZ\nreckon\tVB\nreckon\tVBP\nreclaimed\tVBN\nreclaiming\tVBG\nreclaims\tVBZ\nreclaim\tVB\nreclaim\tVBP\nreclassified\tVBD\nreclining\tVBG\nrecognized\tVBD\nrecognized\tVBN\nrecognizes\tVBZ\nrecognize\tVB\nrecognize\tVBP\nrecognizing\tVBG\nrecommendations\tNNS\nrecommendatons\tNNS\nrecommended\tVBD\nrecommended\tVBN\nrecommending\tVBG\nrecommends\tVBZ\nrecommend\tVB\nrecommend\tVBP\nreconciles\tVBZ\nreconcile\tVB\nreconciling\tVBG\nreconnect\tVB\nreconsidered\tVBN\nreconsider\tVB\nreconstructed\tVBD\nreconstructed\tVBN\nreconstructing\tVBG\nreconstruct\tVB\nrecorded\tVBD\nrecorded\tVBN\nrecorders\tNNS\nrecordings\tNNS\nrecording\tVBG\nrecords\tNNS\nRECORDS\tNNS\nrecords\tVBZ\nrecord\tVB\nRecord\tVB\nrecounted\tVBD\nrecounted\tVBN\nrecounting\tVBG\nrecounts\tVBZ\nrecouped\tVBD\nrecouped\tVBN\nrecoup\tVB\nrecovered\tVBD\nrecovered\tVBN\nrecoveries\tNNS\nrecovering\tVBG\nRecovering\tVBG\nrecovers\tVBZ\nrecover\tVB\nrecover\tVBP\nrecraft\tVB\nre-creactions\tNNS\nrecreate\tVB\nRe-creating\tVBG\nre-creations\tNNS\nrecruited\tVBD\nrecruited\tVBN\nRecruited\tVBN\nrecruiting\tVBG\nrecruits\tNNS\nrecruits\tVBZ\nrecruit\tVB\nrectangles\tNNS\nrectified\tVBN\nrectifying\tVBG\nrecuperate\tVB\nrecurring\tVBG\nrecused\tVBN\nrecycled\tVBN\nrecycles\tVBZ\nrecycle\tVB\nrecycling\tVBG\nreddened\tVB\nredeemed\tVBN\nredeeming\tVBG\nredeem\tVB\nredefined\tVBD\nredefine\tVB\nredefining\tVBG\nredemptions\tNNS\nredeploy\tVB\nredesigned\tVBD\nredesigned\tVBN\nredesigning\tVBG\nredesign\tVB\nredevelop\tVB\nred-flag\tVB\nredial\tVB\nredirected\tVBN\nrediscover\tVB\nredistribute\tVB\nredistributing\tVBG\nredlining\tVBG\nredoing\tVBG\nredoubling\tVBG\nredound\tVB\nredo\tVB\nredraw\tVB\nredress\tVB\nreds\tNNS\nReds\tNNS\nreduced\tVBD\nreduced\tVBN\nreduces\tVBZ\nreduce\tVB\nreduce\tVBP\nreducing\tVBG\nReducing\tVBG\nreductions\tNNS\nreefs\tNNS\nre-elected\tVBD\nre-elected\tVBN\nreeled\tVBD\nreeled\tVBN\nreeling\tVBG\nre-emerge\tVB\nre-emphasize\tVBP\nre-enacting\tVBG\nre-enactments\tNNS\nRe-enactments\tNNS\nre-entered\tVBD\nre-entering\tVBG\nre-enter\tVB\nre-establishing\tVBG\nre-establish\tVB\nreestablish\tVB\nre-evaluate\tVB\nre-evaluate\tVBP\nre-evaluating\tVBG\nre-examine\tVB\nreexamining\tVBG\nre-exports\tNNS\nrefashioning\tVBG\nreferees\tNNS\nreferences\tNNS\nreferrals\tNNS\nreferred\tVBD\nreferred\tVBN\nreferring\tVBG\nReferring\tVBG\nrefers\tVBZ\nrefer\tVB\nrefer\tVBP\nrefile\tVB\nrefinanced\tVBD\nrefinanced\tVBN\nrefinance\tVB\nrefinancing\tVBG\nrefined\tVBN\nrefineries\tNNS\nrefiners\tNNS\nRefiners\tNNS\nrefine\tVB\nrefining\tVBG\nrefitting\tVBG\nreflected\tVBD\nreflected\tVBN\nreflecting\tVBG\nReflecting\tVBG\nreflects\tVBZ\nreflect\tVB\nreflect\tVBP\nrefocused\tVBD\nrefocused\tVBN\nrefocuses\tVB\nrefocusing\tVBG\nrefocus\tVB\nreformed\tVBN\nreformers\tNNS\nreforming\tVBG\nreforms\tNNS\nreformulated\tVBN\nreform\tVB\nrefrained\tVBN\nrefrain\tVB\nrefreshing\tVBG\nrefrigerators\tNNS\nrefueling\tVBG\nrefugees\tNNS\nrefunded\tVBN\nrefunding\tVBG\nrefunding\tVBG|NN\nrefunds\tNNS\nrefund\tVB\nrefurbished\tVBN\nrefurbishing\tVBG\nrefurbish\tVB\nrefused\tVBD\nrefused\tVBN\nrefusers\tNNS\nrefuses\tVBZ\nrefuse\tVB\nrefuse\tVBP\nrefusing\tVBG\nrefuted\tVBD\nrefute\tVB\nregained\tVBD\nregained\tVBN\nregaining\tVBG\nregains\tVBZ\nRegains\tVBZ\nregain\tVB\nregain\tVBP\nregarded\tVBD\nregarded\tVBN\nRegarded\tVBN\nregarding\tVBG\nRegarding\tVBG\nregards\tVBZ\nregard\tVB\nregard\tVBP\nregenerate\tVB\nregions\tNNS\nregistered\tVBD\nregistered\tVBN\nregistering\tVBG\nregisters\tNNS\nregister\tVB\nregister\tVBP\nregistrants\tNNS\nregistrations\tNNS\nregrets\tNNS\nregrets\tVBZ\nregretted\tVBD\nregretted\tVBN\nregret\tVB\nregret\tVBP\nregroup\tVB\nregroup\tVBP\nregulated\tVBN\nregulates\tVBZ\nregulate\tVB\nregulating\tVBG\nregulations\tNNS\nREGULATIONS\tNNS\nregulators\tNNS\nRegulators\tNNS\nregulators\tVBZ\nregummed\tVBD\nrehabilitated\tVBN\nrehabilitate\tVB\nrehashing\tVBG\nreigned\tVBD\nreigning\tVBG\nreignited\tVBD\nreignited\tVBN\nreignite\tVB\nreigniting\tVBG\nreimbursed\tVBD\nreimbursed\tVBN\nreimbursements\tNNS\nreimburses\tVBZ\nreimburse\tVB\nreimburse\tVBP\nreimpose\tVB\nreincorporated\tVBN\nreincorporating\tVBG\nreindicting\tVBG\nreinforced\tVBD\nreinforced\tVBN\nreinforcements\tNNS\nreinforces\tVBZ\nreinforce\tVB\nreinforce\tVBP\nreinforcing\tVBG\nReinforcing\tVBG\nreining\tVBG\nreins\tNNS\nreinstalled\tVBN\nreinstated\tVBD\nreinstated\tVBN\nreinstate\tVB\nreinstating\tVBG\nreinstituting\tVBG\nreinsurers\tNNS\nreintegrated\tVBN\nreintroduced\tVBN\nrein\tVB\nreinvented\tVBD\nreinvent\tVB\nreinvested\tVBD\nreinvested\tVBN\nreinvesting\tVBG\nreinvest\tVB\nreinvest\tVBP\nreinvigorated\tVBN\nreinvigorate\tVB\nreinvigorating\tVBG\nreiterated\tVBD\nreiterates\tVBZ\nreiterating\tVBG\nREITs\tNNS\nrejected\tVBD\nrejected\tVBN\nrejecting\tVBG\nrejections\tNNS\nrejects\tVBZ\nreject\tVB\nreject\tVBP\nrejoice\tVBP\nrejoined\tVBD\nrejoining\tVBG\nRejoins\tVBZ\nrejoin\tVB\nrejuvenate\tVB\nrekindled\tVBN\nRekindled\tVBN\nrekindle\tVB\nrekindling\tVBG\nrelabeling\tVBG\nrelated\tVBD\nrelated\tVBN\nRelated\tVBN\nrelates\tVBZ\nrelate\tVB\nrelate\tVBP\nrelating\tVBG\nrelationships\tNNS\nrelations\tNNS\nRelations\tNNS\nrelatives\tNNS\nrelaunched\tVBN\nrelaunch\tVB\nrelaxed\tVBD\nrelaxed\tVBN\nrelaxing\tVBG\nrelax\tVB\nrelax\tVBP\nrelayed\tVBD\nreleased\tVBD\nreleased\tVBN\nreleases\tNNS\nreleases\tVBZ\nrelease\tVB\nRelease\tVB\nrelease\tVBP\nreleasing\tVBG\nrelegated\tVBN\nrelented\tVBD\nrelenting\tVBG\nrelent\tVBP\nrelics\tNNS\nrelied\tVB\nrelied\tVBD\nrelied\tVBN\nrelies\tVBZ\nrelieved\tVBD\nrelieved\tVBN\nrelieve\tVB\nreligions\tNNS\nrelinquished\tVBD\nrelinquished\tVBN\nrelinquishing\tVBG\nrelinquish\tVB\nrelished\tVBD\nrelishes\tVBZ\nrelish\tVB\nrelive\tVBP\nrelocated\tVBD\nrelocated\tVBN\nrelocate\tVB\nrelocating\tVBG\nrelocations\tNNS\nrelying\tVBG\nRelying\tVBG\nrely\tVB\nrely\tVBP\nremade\tVB\nremained\tVBD\nremained\tVBN\nremaining\tVBG\nRemaining\tVBG\nremains\tNNS\nRemains\tNNS\nremains\tVBZ\nremain\tVB\nremain\tVBP\nremake\tVB\nremanded\tVBD\nremarked\tVBD\nremarked\tVBN\nRemarketers\tNNS\nremarketings\tNNS\nremarks\tNNS\nremarks\tVBZ\nremark\tVB\nremedied\tVBN\nremedies\tNNS\nremedy\tVB\nremembered\tVBD\nremembered\tVBN\nremembering\tVBG\nremembers\tVBZ\nremember\tVB\nRemember\tVB\nremember\tVBP\nRemics\tNNS\nREMICs\tNNS\nreminded\tVBD\nreminded\tVBN\nreminders\tNNS\nreminding\tVBG\nreminds\tVBZ\nremind\tVB\nremind\tVBP\nremittances\tNNS\nremnants\tNNS\nremodeled\tVBN\nremodeling\tVBG\nremora\tNNS\nremoved\tVBD\nremoved\tVBN\nRemoved\tVBN\nremoves\tVBZ\nremove\tVB\nremoving\tVBG\nremunerated\tVBN\nrenamed\tVBD\nrenamed\tVBN\nrename\tVB\nrendered\tVBD\nrendered\tVBN\nrenderings\tNNS\nrendering\tVBG\nrender\tVB\nrendezvoused\tVBD\nrenege\tVB\nreneging\tVBG\nrenegotiated\tVBN\nrenegotiate\tVB\nrenegotiating\tVBG\nrenewals\tNNS\nrenewed\tVBD\nrenewed\tVBN\nRenewed\tVBN\nrenewing\tVBG\nrenews\tVBZ\nrenew\tVB\nrenounced\tVBD\nrenounce\tVB\nrenouncing\tVBG\nrenovated\tVBN\nrenovate\tVB\nrenovating\tVBG\nrenowned\tVBN\nrentals\tNNS\nrented\tVBN\nrenters\tNNS\nrenting\tVBG\nrents\tNNS\nRents\tNNS\nrents\tVBZ\nrent\tVB\nreoffered\tVBN\nreopened\tVBD\nreopened\tVBN\nreopening\tVBG\nreopens\tVBZ\nreopen\tVB\nreorganized\tVBN\nreorganizes\tVBZ\nreorganize\tVB\nreoriented\tVBN\nrepackaged\tVBN\nrepackage\tVB\nrepackaging\tVBG\nrepaid\tVBD\nrepaid\tVBN\nrepainted\tVBN\nrepaired\tVBD\nrepaired\tVBN\nrepairing\tVBG\nrepairs\tNNS\nrepairs\tVBZ\nrepair\tVB\nrepair\tVBP\nreparations\tNNS\nrepassed\tVBN\nrepatriate\tVB\nrepaying\tVBG\nrepayments\tNNS\nrepay\tVB\nrepealed\tVBN\nrepeals\tVBZ\nrepeal\tVB\nrepeated\tVBD\nrepeated\tVBN\nrepeaters\tNNS\nrepeating\tVBG\nrepeats\tNNS\nrepeats\tVBZ\nrepeat\tVB\nrepel\tVB\nrepel\tVBP\nrepercussions\tNNS\nreplaced\tVBD\nreplaced\tVBN\nreplacements\tNNS\nreplaces\tVBZ\nreplace\tVB\nreplace\tVBP\nreplacing\tVBG\nReplacing\tVBG\nreplaster\tVB\nreplays\tNNS\nreplenished\tVBN\nreplenish\tVB\nreplicated\tVBN\nreplicate\tVB\nreplicating\tVBG\nreplied\tVBD\nReplied\tVBD\nreplied\tVBN\nreplies\tNNS\nreplies\tVBZ\nreply\tVB\nrepond\tVB\nreported\tVBD\nreported\tVBN\nreporters\tNNS\nreporting\tVBG\nreports\tNNS\nReports\tNNS\nREPORTS\tNNS\nreports\tVBZ\nReports\tVBZ\nreport\tVB\nReport\tVB\nreport\tVBP\nreposed\tVBN\nreposition\tVB\nrepositories\tNNS\nrepossess\tVB\nrepresentations\tNNS\nrepresentatives\tNNS\nRepresentatives\tNNS\nrepresented\tVBD\nrepresented\tVBN\nrepresenting\tVBG\nrepresentives\tNNS\nrepresents\tVBZ\nrepresent\tVB\nrepresent\tVBP\nrepressed\tVBN\nrepressing\tVBG\nrepriced\tVBN\nreprinted\tVBN\nReprinted\tVBN\nreprints\tVBZ\nreprint\tVB\nreprisals\tNNS\nreproduced\tVBD\nReproduced\tVBN\nreproduce\tVB\nreprove\tVB\nreps\tNNS\nRepublicans\tNNS\nrepublics\tNNS\nrepudiate\tVB\nrepurchased\tVBD\nrepurchased\tVBN\nrepurchases\tNNS\nrepurchase\tVB\nrepurchase\tVBD\nrepurchase\tVBN\nrepurchasing\tVBG\nreputations\tNNS\nreputed\tVBN\nrequested\tVBD\nrequested\tVBN\nrequesting\tVBG\nrequests\tNNS\nrequest\tVB\nrequest\tVBP\nrequired\tVBD\nrequired\tVBN\nrequirements\tNNS\nrequires\tVBZ\nrequire\tVB\nRequire\tVB\nrequire\tVBP\nrequiring\tVBG\nrequisitioned\tVBD\nrerouted\tVBN\nrerouting\tVBG\nreruns\tNNS\nresales\tNNS\nrescheduled\tVBD\nrescheduled\tVBN\nreschedule\tVB\nrescinded\tVBD\nrescinding\tVBG\nrescind\tVB\nrescissions\tNNS\nrescued\tVBD\nrescuers\tNNS\nrescues\tVBZ\nrescue\tVB\nresearched\tVBN\nresearchers\tNNS\nResearchers\tNNS\nRESEARCHERS\tNNS\nresearches\tVBZ\nresearching\tVBG\nresearch\tVB\nresellers\tNNS\nreselling\tVBG\nresells\tVBZ\nresell\tVB\nresell\tVBP\nresemblances\tNNS\nresembles\tVBZ\nresemble\tVB\nresemble\tVBP\nresembling\tVBG\nresented\tVBD\nresent\tVB\nresent\tVBP\nreservations\tNNS\nreserved\tVBD\nreserved\tVBN\nReserved\tVBN\nreserves\tNNS\nReserves\tNNS\nreserves\tVBZ\nreserve\tVB\nreserve\tVBP\nreserving\tVBG\nreservoirs\tNNS\nreset\tVB\nreshaped\tVBD\nreshaped\tVBN\nreshape\tVB\nreshaping\tVBG\nreshuffled\tVBD\nreshuffle\tVB\nreshufflings\tNNS\nreshuffling\tVBG\nresided\tVBN\nresidences\tNNS\nresidents\tNNS\nResidents\tNNS\nresides\tVBZ\nreside\tVB\nresiding\tVBG\nresidues\tNNS\nresignations\tNNS\nRESIGNATIONS\tNNS\nresigned\tVBD\nRESIGNED\tVBD\nresigned\tVBN\nresigning\tVBG\nresign\tVB\nresins\tNNS\nresisted\tVBD\nresisted\tVBN\nresisting\tVBG\nresists\tVBZ\nresist\tVB\nResist\tVB\nresist\tVBP\nresold\tVB\nresold\tVBN\nresold\tVBP\nresolutions\tNNS\nresolved\tVBD\nresolved\tVBN\nresolve\tVB\nresolving\tVBG\nresonated\tVBD\nresonates\tVBZ\nresonate\tVB\nresorts\tNNS\nresorts\tVBZ\nresort\tVB\nresort\tVBP\nresources\tNNS\nrespected\tVBD\nrespected\tVBN\nrespects\tNNS\nrespects\tVBZ\nrespect\tVB\nresponded\tVBD\nresponded\tVBN\nrespondents\tNNS\nresponding\tVBG\nResponding\tVBG\nresponds\tVBZ\nrespond\tVB\nrespond\tVBP\nresponses\tNNS\nResponses\tNNS\nresponsibilities\tNNS\nrestarted\tVBD\nrestarted\tVBN\nrestarters\tNNS\nrestarting\tVBG\nrestart\tVB\nrestated\tVBD\nrestated\tVBN\nrestate\tVB\nrestating\tVBG\nrestaurants\tNNS\nRestaurants\tNNS\nrested\tVBD\nrestored\tVBD\nrestored\tVBN\nrestore\tVB\nrestoring\tVBG\nrestrained\tVBD\nrestrained\tVBN\nrestraining\tVBG\nrestraints\tNNS\nrestrain\tVB\nrestricted\tVBD\nrestricted\tVBN\nrestricting\tVBG\nrestrictions\tNNS\nrestricts\tVBZ\nrestrict\tVB\nRestrict\tVB\nrestrict\tVBP\nrestructured\tVBD\nrestructured\tVBN\nrestructures\tVBZ\nrestructure\tVB\nrestructure\tVBP\nrestructurings\tNNS\nrestructuring\tVBG\nrests\tVBZ\nrest\tVB\nRest\tVB\nrest\tVB|NN\nrest\tVBP\nrestyled\tVBN\nresubmit\tVB\nresulted\tVBD\nresulted\tVBN\nresulting\tVBG\nresulting\tVBG|JJ\nresults\tNNS\nResults\tNNS\nresults\tVBZ\nresult\tVB\nresult\tVBP\nresumed\tVBD\nresumed\tVBN\nresumes\tNNS\nresumes\tVBZ\nresume\tVB\nresume\tVBP\nresuming\tVBG\nre-supplied\tVBN\nresurfaced\tVBD\nresurfaced\tVBN\nresurging\tVBG\nresurrected\tVBD\nresurrected\tVBN\nresurrects\tVBZ\nresurrect\tVB\nresurrect\tVBP\nresuscitate\tVB\nresuscitating\tVBG\nretailers\tNNS\nRetailers\tNNS\nretailing\tVBG\nretail-sales\tNNS\nretails\tVBZ\nretail\tVB\nretail\tVBP\nretained\tVBD\nretained\tVBN\nretaining\tVBG\nretains\tVBZ\nretain\tVB\nretain\tVBP\nretaking\tVBG\nretaliating\tVBG\nretardants\tNNS\nretard\tVB\nrethinking\tVBG\nrethink\tVB\nretired\tVBD\nretired\tVBN\nRetired\tVBN\nretirees\tNNS\nretirements\tNNS\nretires\tVBZ\nretire\tVB\nretire\tVBP\nretiring\tVBG\nretooling\tVBG\nretools\tVBZ\nretorts\tNNS\nretorts\tVBZ\nretraced\tVBD\nretracted\tVBD\nretraining\tVBG\nretreated\tVBD\nretreated\tVBN\nretreating\tVBG\nretreats\tNNS\nretreat\tVB\nretrench\tVBP\nretrieved\tVBD\nretrieved\tVBN\nretrieve\tVB\nretrieve\tVBP\nretrofit\tVB\nretry\tVB\nreturned\tVBD\nreturned\tVBN\nreturning\tVBG\nreturns\tNNS\nReturns\tNNS\nreturns\tVBZ\nreturn\tVB\nreturn\tVBP\nreunions\tNNS\nreunited\tVBN\nreunite\tVB\nreused\tVBN\nrevalued\tVBN\nrevamped\tVBD\nrevamped\tVBN\nrevamping\tVBG\nrevamp\tVB\n're\tVBP\nrevealed\tVBD\nrevealed\tVBN\nrevealing\tVBG\nreveals\tVBZ\nReveals\tVBZ\nreveal\tVB\nreveal\tVBP\nrevelations\tNNS\nrevelers\tNNS\nreveling\tVBG\nrevels\tNNS\nrevel\tVBP\nrevenues\tNNS\nRevenues\tNNS\nreverberated\tVBN\nreverberate\tVB\nreverberating\tVBG\nreverberations\tNNS\nreversals\tNNS\nreversed\tVBD\nreversed\tVBN\nreverses\tVBZ\nreverse\tVB\nReverse\tVBP\nreversing\tVBG\nReversing\tVBG\nreverted\tVBN\nreverts\tVBZ\nreviewed\tVBD\nreviewed\tVBN\nreviewing\tVBG\nReviewing\tVBG\nreviews\tNNS\nreviews\tVBZ\nreview\tVB\nreview\tVBP\nrevised\tVBD\nrevised\tVBN\nRevised\tVBN\nREVISED\tVBN\nrevise\tVB\nrevising\tVBG\nRevising\tVBG\nrevisionists\tNNS\nrevisions\tNNS\nrevisited\tVBN\nrevisits\tVBZ\nrevisit\tVB\nRevitalized\tVBN\nrevitalizing\tVBG\nrevivals\tNNS\nRevivals\tNNS\nrevived\tVBD\nrevived\tVBN\nrevive\tVB\nrevive\tVBP\nreviving\tVBG\nrevoked\tVBN\nrevoke\tVB\nrevoking\tVBG\nrevolutionaries\tNNS\nrevolutionized\tVBD\nrevolutionize\tVB\nrevolves\tVBZ\nrevolve\tVB\nrevolving\tVBG\nrevved\tVBN\nrewarded\tVBN\nrewarding\tVBG\nrewards\tNNS\nRewards\tNNS\nrewards\tVBZ\nreward\tVB\nReward\tVB\nreworked\tVBN\nrewrite\tVB\nrewriting\tVBG\nrewritten\tVBN\nrhymed\tVBD\nrhymes\tVBZ\nrhyming\tVBG\nribbies\tNNS\nribbons\tNNS\nribs\tNNS\nRicans\tNNS\nriches\tNNS\nriders\tNNS\nrides\tNNS\nride\tVB\nride\tVBP\nridges\tNNS\nridiculed\tVBN\nridicules\tVBZ\nriding\tVBG\nrid\tVB\nRID\tVB\nrid\tVBD\nrid\tVBN\nrid\tVBN|JJ\nriffing\tVBG\nrifles\tNNS\nrigged\tVBD\nrigged\tVBN\nrighted\tVBN\nrights\tNNS\nRights\tNNS\nRIGHTS\tNNS\nright-to-lifers\tNNS\nright-wingers\tNNS\nrigors\tNNS\nrigs\tNNS\nriles\tVBZ\nrile\tVBP\nrim\tVBP\nringers\tNNS\nRingers\tNNS\nringing\tVBG\nrings\tNNS\nrings\tVBZ\nring\tVB\nring\tVBP\nriots\tNNS\nripens\tVBZ\nripen\tVBP\nripoffs\tNNS\nripped\tVBD\nripped\tVBN\nRipples\tNNS\nripple\tVBP\nrippling\tVBG\nrisen\tVBN\nrises\tNNS\nrises\tVBZ\nrise\tVB\nrise\tVBP\nrising\tVBG\nRising\tVBG\nrisked\tVBD\nrisking\tVBG\nrisks\tNNS\nRisks\tNNS\nrisks\tVBZ\nrisk\tVB\nrisk\tVBP\nrites\tNNS\nrituals\tNNS\nrivaling\tVBG\nrivalries\tNNS\nrivals\tNNS\nRivals\tNNS\nrivals\tVBZ\nrival\tVB\nrivers\tNNS\nriveted\tVBD\nriveted\tVBN\nrivets\tNNS\nrivets\tVBZ\nroadblocks\tNNS\nroads\tNNS\nRoads\tNNS\nroadways\tNNS\nroamed\tVBD\nroam\tVBP\nroaring\tVBG\nroars\tVBZ\nrobbed\tVBN\nrobberies\tNNS\nrobbers\tNNS\nRobbers\tNNS\nrobbing\tVBG\nrobes\tNNS\nrobots\tNNS\nrocked\tVBD\nrocked\tVBN\nrockers\tNNS\nrocketed\tVBD\nrocketing\tVBG\nrockets\tNNS\nrocking\tVBG\nrocks\tNNS\nrock\tVB\nrodents\tNNS\nrode\tVBD\nRODE\tVBD\nrods\tNNS\nrogues\tNNS\nroiling\tVBG\nroil\tVB\nroles\tNNS\nrollbacks\tNNS\nrolled\tVBD\nrolled\tVBN\nRolled\tVBN\nrollers\tNNS\nrolling\tVBG\nrollovers\tNNS\nrolls\tNNS\nrolls\tVBZ\nroll\tVB\nroll\tVBP\nromancing\tVBG\nromanticized\tVBN\nromps\tNNS\nromp\tVBP\nroofers\tNNS\nroofs\tNNS\nrooftops\tNNS\nRoommates\tNNS\nrooms\tNNS\nroost\tVB\nrooted\tVBN\nrooters\tNNS\nrooting\tVBG\nroots\tNNS\nroot\tVB\nroot\tVBP\nropes\tNNS\nrosarians\tNNS\nroses\tNNS\nrose\tVBD\nrose\tVBP\nrotated\tVBD\nrotate\tVB\nrotating\tVBG\nrotted\tVBN\nrotting\tVBG\nroughed\tVBD\nroughnecks\tNNS\nrounded\tVBD\nrounded\tVBN\nrounds\tNNS\nround\tVB\nround\tVBP\nroustabouts\tNNS\nrouted\tVBN\nroutes\tNNS\nroutes\tVBZ\nroutines\tNNS\nrouting\tVBG\nrowed\tVBD\nrows\tNNS\nrow\tVB\nroyalties\tNNS\nrubbed\tVBD\nrubber-necking\tVBG\nrubber-stamp\tVB\nrubdowns\tNNS\nrubfests\tNNS\nRubins\tNNS\nrubles\tNNS\nrubs\tNNS\nrubs\tVBZ\nrub\tVB\nrub\tVBP\nruffled\tVBN\nRuffled\tVBN\nrugs\tNNS\nruined\tVBN\nRuined\tVBN\nruining\tVBG\nruins\tNNS\nruin\tVB\nruin\tVBP\nruled\tVBD\nruled\tVBN\nrulers\tNNS\nrules\tNNS\nRules\tNNS\nRULES\tNNS\nrules\tVBZ\nrule\tVB\nrule\tVBP\nrulings\tNNS\nruling\tVBG\nrumbled\tVBD\nrumbles\tVBZ\nrumble\tVBP\nrumblings\tNNS\nrumbling\tVBG\nruminated\tVBD\nruminated\tVBN\nruminations\tNNS\nrumored\tVBD\nrumored\tVBN\nrumors\tNNS\nRumors\tNNS\nrumors\tVBZ\nrung\tVBN\nrun-ins\tNNS\nrunners\tNNS\nrunners-up\tNNS\nrunning\tVBG\nRunning\tVBG\nruns\tNNS\nruns\tVBZ\nrunups\tNNS\nrun\tVB\nrun\tVBD\nrun\tVBN\nrun\tVBP\nRunways\tNNS\nruptured\tVBD\nruptured\tVBN\nrupturing\tVBG\nrushed\tVBD\nrushed\tVBN\nrushes\tVBZ\nrushing\tVBG\nrush\tVB\nrush\tVBP\nrusted\tVBN\nrusticated\tVBN\nrustlers\tNNS\nrustlings\tNNS\nRVs\tNNS\nSaatchi\tNNS\nsabers\tNNS\nsabotage\tVB\nsacked\tVBD\nsackings\tNNS\nsacking\tVBG\nsacks\tNNS\nsacrifices\tNNS\nsacrifices\tVBZ\nsacrifice\tVB\nsacrificing\tVBG\nsaddled\tVBD\nsaddled\tVBN\nsafeguarded\tVBN\nsafeguarding\tVBG\nsafeguards\tNNS\nsafeguard\tVB\nsafety\tVB\nsages\tNNS\nsagged\tVBD\nsagged\tVBN\nsagging\tVBG\nsagging\tVBG|JJ\nSagos\tNNS\nsaid\tVBD\nSaid\tVBD\nSAID\tVBD\nsaid\tVBN\nSaid\tVBN\nsailed\tVBD\nsailing\tVBG\nsailors\tNNS\nsails\tNNS\nsaints\tNNS\nsalaries\tNNS\nSALARIES\tNNS\nsalarymen\tNNS\nsalesmen\tNNS\nsales\tNNS\nSales\tNNS\nSALES\tNNS\nsalespeople\tNNS\nSalespeople\tNNS\nsalicylates\tNNS\nsalted\tVBD\nsaluted\tVBD\nsaluting\tVBG\nsalvaged\tVBN\nsalvage\tVB\nsalve\tVB\nSamaritans\tNNS\nsamovars\tNNS\nsampled\tVBN\nsamples\tNNS\nsanctioned\tVBN\nsanctioning\tVBG\nsanctions\tNNS\nsanctions\tVBZ\nSandinistas\tNNS\nsands\tNNS\nsandwiched\tVBD\nsandwiched\tVBN\nSandwiched\tVBN\nsandwiches\tNNS\nsang\tVBD\nsanitationists\tNNS\nsanitized\tVBN\nsanitize\tVBP\nsank\tVBD\nSan\tNNS\nsapped\tVBN\nsapping\tVBG\nsap\tVB\nsatellites\tNNS\nsatisfied\tVBD\nsatisfied\tVBN\nsatisfied\tVBN|JJ\nsatisfies\tNNS\nsatisfies\tVBZ\nsatisfying\tVBG\nSatisfying\tVBG\nsatisfy\tVB\nsaturated\tVBD\nsaturated\tVBN\nsaturate\tVB\nsat\tVBD\nsat\tVBN\nsaucers\tNNS\nsauces\tNNS\nsaunas\tNNS\nsaved\tVBD\nsaved\tVBN\nsavers\\/investors\tNNS\nsavers\tNNS\nsaves\tVBZ\nsave\tVB\nSave\tVB\nsave\tVBP\nsavings\tNNS\nSavings\tNNS\nsaving\tVBG\nsavored\tVBD\nsavoring\tVBG\nsavors\tNNS\nsavor\tVB\nsaw\tVBD\nsaying\tVBG\nSaying\tVBG\nsays\tVBZ\nSays\tVBZ\nsay\tVB\nSay\tVB\nsay\tVBP\nSay\tVBP\nscabs\tNNS\nscalawags\tNNS\nscaled\tVBD\nscaled\tVBN\nscales\tNNS\nscale\tVB\nscaling\tVBG\nscalps\tNNS\nscammed\tVBD\nscammers\tNNS\nscamper\tVBP\nscams\tNNS\nscandalized\tVBD\nscandals\tNNS\nscanners\tNNS\nscanning\tVBG\nscans\tNNS\nscans\tVBZ\nscan\tVB\nscape\tVB\nscared\tVBD\nscared\tVBN\nscares\tNNS\nscares\tVBZ\nscare\tVB\nscare\tVBP\nscarfing\tVBG\nscaring\tVBG\nScaring\tVBG\nscarred\tVBN\nscars\tNNS\nscattered\tVBN\nscavengers\tNNS\nscenarios\tNNS\nScenarios\tNNS\nscenes\tNNS\nscents\tNNS\nscheduled\tVBD\nscheduled\tVBN\nschedules\tNNS\nschedule\tVB\nscheduling\tVBG\nschemers\tNNS\nschemes\tNNS\nscheming\tVBG\nschmumpered\tVBD\nscholarships\tNNS\nscholars\tNNS\nschoolboys\tNNS\nschoolchildren\tNNS\nschoolmates\tNNS\nschools\tNNS\nSchools\tNNS\nschoolteachers\tNNS\nschool\tVB\nsciences\tNNS\nscientists\tNNS\nScientists\tNNS\nSciMed\tVBD\nSciMed\tVBN\nscoffed\tVBD\nscoffs\tVBZ\nscoff\tVBP\nscolded\tVBN\nscold\tVB\nscooped\tVBD\nscooping\tVBG\nscoops\tNNS\nscoops\tVBZ\nscoop\tVB\nscooted\tVBD\nscored\tVBD\nscored\tVBN\nscores\tNNS\nScores\tNNS\nscore\tVBP\nscoring\tVBG\nscorn\tVB\nscorn\tVBP\nscotched\tVBD\nscotches\tNNS\nscourges\tNNS\nscouring\tVBG\nscour\tVB\nscouting\tVBG\nscout\tVB\nscowls\tVBZ\nscowl\tVBP\nscrambled\tVBD\nscrambled\tVBN\nscrambles\tNNS\nscramble\tVBP\nscrambling\tVBG\nscrape\tVB\nscrapped\tVBD\nscrapped\tVBN\nscrapping\tVBG\nscraps\tNNS\nScraps\tNNS\nscraps\tVBZ\nscrap\tVB\nSCRAP\tVBP\nscratched\tVBN\nscratching\tVBG\nscratch\tVB\nscratch\tVBP\nscreamed\tVBD\nscreaming\tVBG\nscreams\tVBZ\nscream\tVB\nscream\tVBP\nscreeched\tVBD\nscreeching\tVBG\nscreened\tVBD\nscreenings\tNNS\nscreening\tVBG\nscreens\tNNS\nscreens\tVBZ\nscreen\tVB\nscreenwriters\tNNS\nscrewed\tVBD\nscrewed\tVBN\nscrews\tNNS\nscribbled\tVBD\nscribblers\tNNS\nscribbling\tVBG\nscribes\tNNS\nscrimped\tVBD\nscrimping\tVBG\nscripts\tNNS\nscriptwriters\tNNS\nscrounged\tVBD\nscrounge\tVBP\nscrubbers\tNNS\nscrutinized\tVBD\nscrutinized\tVBN\nscrutinize\tVB\nscrutinizing\tVBG\nsculptors\tNNS\nsculptures\tNNS\nscurries\tNNS\nscurrying\tVBG\nscurry\tVB\nscuttled\tVBD\nscuttled\tVBN\nscuttle\tVB\nsealants\tNNS\nsealed\tVBD\nsealed\tVBN\nsealing\tVBG\nseals\tNNS\nseal\tVB\nseamen\tNNS\nsearched\tVBD\nsearches\tNNS\nsearches\tVBZ\nsearching\tVBG\nsearch\tVB\nsearch\tVBP\nsearing\tVBG\nSears\tNNS\nseas\tNNS\nseasonings\tNNS\nseasons\tNNS\nseated\tVBN\nseating\tVBG\nseats\tNNS\nSeats\tNNS\nseats\tVBZ\nsecede\tVB\nsecond-guessed\tVBN\nsecond-guess\tVB\nseconds\tNNS\nSeconds\tNNS\nsecretaries\tNNS\nsecrets\tNNS\nsections\tNNS\nsectors\tNNS\nsecured\tVBD\nsecured\tVBN\nsecure\tVB\nsecure\tVBP\nsecuring\tVBG\nsecurites\tNNS\nsecurities\tNNS\nSecurities\tNNS\nsedans\tNNS\nseduce\tVB\nseducing\tVBG\nseeded\tVBN\nseeds\tNNS\nseeing\tVBG\nSeeing\tVBG\nseekers\tNNS\nseeking\tVBG\nSeeking\tVBG\nSEEKING\tVBG\nseeks\tVBZ\nseek\tVB\nseek\tVBP\nSeek\tVBP\nseemed\tVBD\nseemed\tVBN\nseems\tVBZ\nSeems\tVBZ\nseem\tVB\nseem\tVBP\nseen\tVBN\nSeen\tVBN\nseeped\tVBD\nseesawing\tVBG\nsees\tVBZ\nSees\tVBZ\nseething\tVBG\nsee\tVB\nSee\tVB\nsee\tVBP\nSEE\tVBP\nsegmenting\tVBG\nsegments\tNNS\nsegregated\tVBD\nsegregate\tVB\nseized\tVBD\nseized\tVBN\nSeizes\tVBZ\nseize\tVB\nseize\tVBP\nseizing\tVBG\nseizures\tNNS\nselected\tVBD\nselected\tVBN\nselecting\tVBG\nselections\tNNS\nSelections\tNNS\nselects\tVBZ\nselect\tVB\nselect\tVBP\nself-destructed\tVBD\nself-insure\tVBP\nself-reinsure\tVB\nself-starters\tNNS\nsellers\tNNS\nselling\tVBG\nSelling\tVBG\nsell-off\tNNS\nsell-offs\tNNS\nselloffs\tNNS\nsells\tVBZ\nsell\tVB\nSell\tVB\nSELL\tVB\nsell\tVBP\nsemantics\tNNS\nsemesters\tNNS\nsemi-celebrities\tNNS\nsemiconductors\tNNS\nsemifinished\tVBN\nSemifinished\tVBN\nseminars\tNNS\nsenators\tNNS\nSenators\tNNS\nsenders\tNNS\nsending\tVBG\nsends\tVBZ\nsend\tVB\nSend\tVB\nsend\tVBP\nseniors\tNNS\nSeniors\tNNS\nsensed\tVBD\nsenses\tNNS\nsense\tVB\nsense\tVBP\nsensibilities\tNNS\nsensing\tVBG\nsensitives\tNNS\nsensitivities\tNNS\nsensitize\tVB\nSens.\tNNS\nsensors\tNNS\nsentenced\tVBD\nsentenced\tVBN\nsentences\tNNS\nSentences\tNNS\nsentencings\tNNS\nsentencing\tVBG\nSentencing\tVBG\nsentiments\tNNS\nsent\tVBD\nsent\tVBN\nseparated\tVBN\nSEPARATED\tVBN\nseparate\tVB\nseparate\tVBP\nseparating\tVBG\nsequels\tNNS\nsequestered\tVBN\nsequester\tVB\nsequester\tVBP\nsequins\tNNS\nsergeants\tNNS\nserials\tNNS\nseries\tNNS\nSeries\tNNS\nservants\tNNS\nserved\tVBD\nserved\tVBN\nServed\tVBN\nservers\tNNS\nserves\tVBZ\nserve\tVB\nserve\tVBP\nserviced\tVBN\nservices\tNNS\nSERVICES\tNNS\nservices\tVBZ\nservice\tVB\nservicing\tVBG\nserving\tVBG\nsessions\tNNS\nsetbacks\tNNS\nsets\tNNS\nsets\tVBZ\nsetters\tNNS\nsettings\tNNS\nsetting\tVBG\nSetting\tVBG\nsettled\tVBD\nsettled\tVBN\nsettlements\tNNS\nSettlements\tNNS\nsettlers\tNNS\nsettles\tVBZ\nsettle\tVB\nSettle\tVB\nsettle\tVBP\nsettling\tVBG\nset\tVB\nset\tVBD\nset\tVBN\nset\tVBP\nsevered\tVBD\nsevered\tVBN\nsevering\tVBG\nsever\tVB\nsevices\tNNS\nsewers\tNNS\nsew\tVB\nsexes\tNNS\nshacks\tNNS\nshades\tNNS\nshadowing\tVBG\nshadows\tNNS\nshadow\tVB\nshags\tVBZ\nshaken\tVBN\nShaken\tVBN\nshakes\tVBZ\nshake\tVB\nSHAKE\tVB\nshake\tVBP\nshaking\tVBG\nshaped\tVBN\nshapes\tNNS\nshapes\tVBZ\nshape\tVB\nshape\tVBP\nshaping\tVBG\nshards\tNNS\nsharecroppers\tNNS\nshared\tVBD\nshared\tVBN\nshareholders\tNNS\nShareholders\tNNS\nshareholdings\tNNS\nshares\tNNS\nShares\tNNS\nshares\tVBZ\nshare\tVB\nshare\tVBP\nsharing\tVBG\nSharing\tVBG\nsharks\tNNS\nsharpening\tVBG\nsharpens\tVBZ\nsharpen\tVB\nshattered\tVBD\nshattered\tVBN\nshattering\tVBG\nshatters\tNNS\nshaved\tVBD\nshaved\tVBN\nshaves\tVBZ\nshave\tVB\nShaving\tVBG\nsheared\tVBN\nsheaths\tNNS\nshedding\tVBG\nSHEDDING\tVBG\nshed\tVB\nshed\tVBD\nshed\tVBN\nshed\tVBP\nsheets\tNNS\nsheiks\tNNS\nshelled\tVBD\nshelling\tVBG\nshells\tNNS\nshell\tVB\nsheltered\tVBN\nsheltering\tVBG\nshelters\tNNS\nSHELTERS\tNNS\nshelved\tVBD\nshelves\tNNS\nshepherded\tVBD\nsheriffs\tNNS\nshied\tVBD\nshied\tVBN\nshielded\tVBD\nshielded\tVBN\nshielding\tVBG\nshields\tNNS\nshield\tVB\nshifted\tVBD\nshifted\tVBN\nshifting\tVBG\nshifts\tNNS\nshifts\tVBZ\nshift\tVB\nshift\tVBP\nshillings\tNNS\nshills\tNNS\nshimmered\tVBD\nshine\tVB\nshine\tVBP\nshining\tVBG\nShining\tVBG\nshins\tNNS\nshipbuilders\tNNS\nshipments\tNNS\nShipments\tNNS\nshipped\tVBD\nshipped\tVBN\nshippers\tNNS\nShippers\tNNS\nshipping\tVBG\nshipsets\tNNS\nships\tNNS\nship\tVB\nship\tVBP\nshipyards\tNNS\nshirking\tVBG\nshirk\tVB\nshirts\tNNS\nshivering\tVBG\nshivers\tNNS\nshocked\tVBD\nshocked\tVBN\nShocked\tVBN\nshocks\tNNS\nshock\tVB\nshoehorned\tVBN\nshoe-horn\tVB\nshoelaces\tNNS\nshoemaking\tVBG\nshoes\tNNS\nshook\tVBD\nShook\tVBD\nshootings\tNNS\nshooting\tVBG\nshoots\tNNS\nshoots\tVBZ\nshoot\tVB\nshoot\tVBP\nshopkeepers\tNNS\nshopped\tVBD\nshopped\tVBN\nshoppers\tNNS\nShoppers\tNNS\nSHOPPERS\tNNS\nshopping\tVBG\nshops\tNNS\nShops\tNNS\nSHOPS\tNNS\nshop\tVB\nshores\tNNS\nshore\tVB\nshoring\tVBG\nshorn\tVB\nShorn\tVBN\nshortages\tNNS\nshortchanged\tVBN\nshort-circuited\tVBN\nshortcomings\tNNS\nshorted\tVBN\nshortened\tVBN\nshortening\tVBG\nshorten\tVB\nshorting\tVBG\nShorting\tVBG\nshort-sellers\tNNS\nshorts\tNNS\nshort\tVB\nshots\tNNS\nshot\tVBD\nshot\tVBN\nshouldering\tVBG\nshoulders\tNNS\nshoulder\tVB\nshoulder\tVBP\nshouted\tVBD\nshouting\tVBG\nshouts\tVBZ\nshout\tVB\nshout\tVBP\nshoved\tVBD\nshovels\tNNS\nshoves\tVBZ\nshove\tVB\nshoving\tVBG\nshowcases\tNNS\nshowed\tVBD\nshowed\tVBN\nshowers\tNNS\nshower\tVB\nshowgirls\tNNS\nshowings\tNNS\nshowing\tVBG\nShowing\tVBG\nshown\tVBN\nshowrooms\tNNS\nShowrooms\tNNS\nshows\tNNS\nshow-stoppers\tNNS\nshows\tVBZ\nShows\tVBZ\nshow\tVB\nShow\tVB\nshow\tVBP\nshrank\tVBD\nshredded\tVBD\nshrieked\tVBD\nshrines\tNNS\nshrinking\tVBG\nShrinking\tVBG\nshrinks\tVBZ\nshrink\tVB\nshriveled\tVBD\nshrouded\tVBD\nshrubs\tNNS\nShrubs\tNNS\nshrugged\tVBD\nshrugs\tNNS\nshrugs\tVBZ\nshrug\tVB\nshrunk\tVBN\nshudders\tNNS\nshuffled\tVBD\nshuffling\tVBG\nshunned\tVBD\nshunning\tVBG\nshun\tVB\nshun\tVBP\nSHUN\tVBP\nshutdowns\tNNS\nshuts\tNNS\nshuts\tVBZ\nshuttered\tVBD\nshuttering\tVBG\nshutting\tVBG\nshuttled\tVBD\nshuttled\tVBN\nshuttles\tNNS\nshuttling\tVBG\nshut\tVB\nshut\tVBD\nshut\tVBN\nshying\tVBG\nshy\tVB\nshy\tVBP\nsiblings\tNNS\nsided\tVBD\nsided\tVBN\nsidelined\tVBN\nsidelines\tNNS\nsidelining\tVBG\nsides\tNNS\nSIDES\tNNS\nsidestepped\tVBD\nsidestepping\tVBG\nsidesteps\tVBZ\nsidestep\tVB\nsidestep\tVBP\nsidetrack\tVB\nside\tVB\nside\tVBP\nsidewalks\tNNS\nsifted\tVBN\nsift\tVB\nsift\tVBP\nsighed\tVBD\nsighing\tVBG\nsighs\tNNS\nsighs\tVBZ\nsightings\tNNS\nsights\tNNS\nsignaled\tVBD\nsignaled\tVBN\nSIGNALED\tVBN\nsignaling\tVBG\nsignalling\tVBG\nsignals\tNNS\nsignals\tVBZ\nsignal\tVB\nSIGNAL\tVB\nsignal\tVBP\nsignatories\tNNS\nsignatures\tNNS\nsignboards\tNNS\nsigned\tVBD\nsigned\tVBN\nSigned\tVBN\nsignified\tVBD\nsignifying\tVBG\nsignify\tVB\nsigning\tVBG\nsigns\tNNS\nSigns\tNNS\nsigns\tVBZ\nsign\tVB\nsilenced\tVBN\nsilences\tNNS\nSilences\tNNS\nsilence\tVB\nsilted\tVBN\nsimilarities\tNNS\nsimmering\tVBG\nsimmer\tVB\nsimplicities\tNNS\nsimplified\tVBN\nsimplifying\tVBG\nSIMPLIFYING\tVBG\nsimplify\tVB\nsimplify\tVBP\nsimulates\tVBZ\nsimulate\tVBP\nsimulations\tNNS\nsimulators\tNNS\nsingers\tNNS\nsinging\tVBG\nSingin\tVBG\nsingled\tVBD\nsingled\tVBN\nsingles\tNNS\nsingle\tVB\nsingle\tVBP\nsingling\tVBG\nsings\tVBZ\nsing\tVB\nsing\tVBP\nsinking\tVBG\nsink\tVB\nsink\tVBP\nsins\tNNS\nsiphoned\tVBD\nsiphoned\tVBN\nsiphoning\tVBG\nsiphon\tVB\nsipped\tVBD\nsipping\tVBG\nsirens\tNNS\nsisters\tNNS\nsitcoms\tNNS\nsites\tNNS\nsits\tVBZ\nsitting\tVBG\nSitting\tVBG\nsitting\tVBG|JJ\nsituated\tVBN\nsituations\tNNS\nsit\tVB\nSit\tVB\nsit\tVBP\nsix-packs\tNNS\nsixties\tNNS\nSixties\tNNS\nsized\tVBD\nsizes\tNNS\nsize\tVBP\nsizzle\tVB\nsizzling\tVBG\nskateboards\tNNS\nskeptics\tNNS\nSkeptics\tNNS\nsketches\tNNS\nsketches\tVBZ\nsketching\tVBG\nskewed\tVBN\nskidded\tVBD\nSKIDDED\tVBD\nskidded\tVBN\nskidding\tVBG\nskids\tNNS\nskid\tVB\nskid\tVBD\nskiers\tNNS\nskies\tNNS\nskiing\tVBG\nskills\tNNS\nSkills\tNNS\nskimmers\tNNS\nskimp\tVB\nskim\tVB\nskins\tNNS\nskipped\tVBD\nskipper\tVB\nskipping\tVBG\nSkipping\tVBG\nskip\tVB\nskip\tVBP\nskirmished\tVBN\nskirmishes\tNNS\nskirted\tVBN\nskirting\tVBG\nskirts\tVBZ\nskis\tNNS\nski\tVB\nSKr1.5\tNNS\nSKr205\tNNS\nSKr20\tNNS\nSKr225\tNNS\nSKr29\tNNS\nSkulls\tNNS\nskyrocketed\tVBD\nskyrocketed\tVBN\nskyrocketing\tVBG\nslabs\tNNS\nslackened\tVBD\nslackened\tVBN\nslackening\tVBG\nslacks\tNNS\nslack\tVB\nslain\tVBN\nslam-dunk\tVB\nslammed\tVBD\nslam\tVBP\nslapped\tVBD\nslapped\tVBN\nslaps\tNNS\nslaps\tVBZ\nslap\tVB\nslashed\tVBD\nslashed\tVBN\nslashes\tNNS\nslashing\tVBG\nslash\tVB\nslash\tVBP\nslated\tVBD\nslated\tVBN\nSlated\tVBN\nslats\tNNS\nslaughtered\tVBN\nslayings\tNNS\nSlay\tVBP\nsleeping\tVBG\nSleeping\tVBG\nsleeps\tVBZ\nsleep\tVB\nsleep\tVBP\nSleep\tVBP\nsleeves\tNNS\nslept\tVBD\nslice\tVB\nslice\tVBP\nslicing\tVBG\nslide-packs\tNNS\nslides\tNNS\nSlides\tNNS\nslides\tVBZ\nslide\tVB\nsliding\tVBG\nslid\tVBD\nslid\tVBN\nslimmed\tVBN\nslimming\tVBG\nslingers\tNNS\nslings\tNNS\nslipped\tVBD\nslipped\tVBN\nslipping\tVBG\nslips\tVBZ\nslip\tVB\nslip\tVBP\nslithered\tVBD\nslithering\tVBG\nslits\tNNS\nslivered\tVBN\nslogans\tNNS\nslogs\tVBZ\nslog\tVB\nslopes\tNNS\nslopes\tVBZ\nsloshing\tVBG\nslots\tNNS\nslough\tVB\nslowdowns\tNNS\nslowed\tVBD\nslowed\tVBN\nslowing\tVBG\nSlowing\tVBG\nslows\tVBZ\nslow\tVB\nS&Ls\tNNS\nslumped\tVBD\nslumped\tVBN\nslumping\tVBG\nslumps\tNNS\nslump\tVB\nslump\tVBP\nslums\tNNS\nslurs\tNNS\nsmacks\tVBZ\nsmack\tVBP\nsmarting\tVBG\nsmashed\tVBD\nsmashed\tVBN\nsmashing\tVBG\nsmash\tVB\nsmelled\tVBD\nsmelling\tVBG\nsmells\tVBZ\nsmell\tVBP\nsmelt\tVBN\nsmidgins\tNNS\nsmiled\tVBD\nsmiles\tNNS\nsmiles\tVBZ\nsmile\tVBP\nsmiling\tVBG\nSmiling\tVBG\nsmoked\tVBD\nsmokers\tNNS\nSmokers\tNNS\nsmokescreens\tNNS\nsmokes\tVBZ\nsmoke\tVB\nsmoke\tVBP\nsmoking\tVBG\nsmoldering\tVBG\nsmolder\tVBP\nsmoothed\tVBN\nsmooth\tVB\nsmothered\tVBN\nsmothering\tVBG\nsmother\tVB\nsmuggle\tVB\nsnafus\tNNS\nsnagged\tVBN\nsnags\tNNS\nsnakes\tNNS\nsnakes\tVBZ\nsnaking\tVBG\nsnapped\tVBD\nsnapped\tVBN\nsnapping\tVBG\nsnapshots\tNNS\nsnaps\tVBZ\nsnap\tVBP\nsnare\tVB\nsnarls\tNNS\nsnatched\tVBD\nsnatched\tVBN\nsnatch\tVB\nsneaked\tVBD\nsneakers\tNNS\nsneaking\tVBG\nsneak\tVBP\nsniffed\tVBD\nsniffing\tVBG\nsniffs\tVBZ\nsniff\tVB\nsniped\tVBD\nsnippets\tNNS\nsniveling\tVBG\n's\tNNS\nsnooping\tVBG\nSnoozing\tVBG\nsnoring\tVBG\nsnorts\tVBZ\nsnowballed\tVBD\nsnowball\tVB\nsnowbirds\tNNS\nsnubbed\tVBD\nsnubbing\tVBG\nsoaking\tVBG\nsoak\tVB\nsoaps\tNNS\nsoapsuds\tNNS\nsoared\tVBD\nsoared\tVBN\nsoaring\tVBG\nSoaring\tVBG\nsoars\tNNS\nsoar\tVB\nsoar\tVBP\nsobered\tVBD\nsob\tVB\nsocalled\tVBN\nsocial-affairs\tNNS\nsocialists\tNNS\nSocialists\tNNS\nsocialize\tVB\nsocializing\tVBG\nsocieties\tNNS\nsociologists\tNNS\nsocks\tNNS\nsock\tVB\nSock\tVB\nsodas\tNNS\nsofas\tNNS\nsoft-drinks\tNNS\nsoftened\tVBD\nsoftened\tVBN\nsoftening\tVBG\nsoftens\tVBZ\nsoften\tVB\nsofties\tNNS\nsoiled\tVBD\nsoil-nutrients\tNNS\nsoils\tNNS\nsoirees\tNNS\nsoldiers\tNNS\nSoldiers\tNNS\nsold\tVBD\nsold\tVBN\nsolicitations\tNNS\nsolicited\tVBD\nsolicited\tVBN\nsoliciting\tVBG\nsolicitors\tNNS\nsolicits\tVBZ\nsolicit\tVB\nsolidified\tVBD\nsolidify\tVB\nsolidify\tVBP\nsolutions\tNNS\nsolved\tVBD\nSolved\tVBD\nsolved\tVBN\nsolvents\tNNS\nsolves\tVBZ\nsolve\tVB\nsolving\tVBG\nSolving\tVBG\nsongs\tNNS\nSONGsters\tNNS\nsongwriters\tNNS\nsons\tNNS\nsoothe\tVB\nsoothing\tVBG\nsophisticates\tNNS\nsops\tNNS\nsoreheads\tNNS\nsorted\tVBN\nsorting\tVBG\nSorting\tVBG\nsorts\tNNS\nsort\tVB\nsought\tVBD\nsought\tVBN\nsoulmates\tNNS\nsouls\tNNS\nsounded\tVBD\nsounded\tVBN\nsoundings\tNNS\nsounding\tVBG\nsounds\tNNS\nsounds\tVBZ\nSounds\tVBZ\nsound\tVB\nSound\tVB\nsound\tVBP\nsoups\tNNS\nsources\tNNS\nSources\tNNS\nsourcing\tVBG\nsoured\tVBD\nsoured\tVBN\nsour\tVB\nsoviets\tNNS\nSoviets\tNNS\nsowed\tVBD\nSows\tNNS\nsows\tVBZ\nsow\tVB\nsow\tVBP\nsoybeans\tNNS\nSoybeans\tNNS\nSOYBEANS\tNNS\nspaceships\tNNS\nspaces\tNNS\nspackle\tVB\nspaghetti\tNNS\nspanned\tVBD\nspanning\tVBG\nspans\tNNS\nspan\tVBP\nspared\tVBD\nspared\tVBN\nspares\tNNS\nspares\tVBZ\nspare\tVB\nspare\tVBP\nsparing\tVBG\nsparked\tVBD\nsparked\tVBN\nsparking\tVBG\nsparkplugs\tNNS\nsparks\tNNS\nsparks\tVBZ\nSparks\tVBZ\nspark\tVB\nsparred\tVBD\nsparring\tVBG\nspasms\tNNS\nspas\tNNS\nspawned\tVBD\nspawned\tVBN\nspawns\tVBZ\nspawn\tVB\nspeakers\tNNS\nSpeakers\tNNS\nspeaking\tVBG\nSpeaking\tVBG\nspeaks\tVBZ\nspeak\tVB\nspeak\tVBP\nspearheaded\tVBD\nspearheaded\tVBN\nspearheading\tVBG\nspecialists\tNNS\nSpecialists\tNNS\nspecialized\tVBN\nspecializes\tVBZ\nspecialize\tVB\nspecialize\tVBP\nspecializing\tVBG\nspecials\tNNS\nspecialties\tNNS\nspecialty-chemicals\tNNS\nspecialty-metals\tNNS\nspecies\tNNS\nspecifications\tNNS\nSpecifications\tNNS\nspecifics\tNNS\nspecified\tVBD\nspecified\tVBN\nspecifies\tVBZ\nspecifying\tVBG\nspecify\tVB\nspecimens\tNNS\nspecs\tNNS\nspectators\tNNS\nspeculated\tVBD\nspeculated\tVBN\nspeculate\tVB\nspeculate\tVBP\nspeculating\tVBG\nspeculations\tNNS\nspeculators\tNNS\nSpeculators\tNNS\nsped\tVBD\nsped\tVBN\nspeeches\tNNS\nspeeded\tVBD\nspeeding\tVBG\nspeeds\tNNS\nspeeds\tVBZ\nspeed\tVB\nSpeed\tVB\nspelled\tVBN\nspelling\tVBG\nspells\tNNS\nspells\tVBZ\nspell\tVB\nspenders\tNNS\nspending\tVBG\nSpending\tVBG\nspending\tVBG|NN\nspends\tVBZ\nspendthrifts\tNNS\nspend\tVB\nSpend\tVB\nspend\tVBP\nspent\tVBD\nspent\tVBN\nspewed\tVBD\nspewing\tVBG\nspices\tNNS\nspiders\tNNS\nspied\tVBD\nspies\tNNS\nspies\tVBZ\nspigots\tNNS\nspiked\tVBN\nspilled\tVBD\nspilled\tVBN\nspills\tNNS\nspills\tVBZ\nspill\tVB\nspinning\tVBG\nspinoffs\tNNS\nspins\tVBZ\nspin\tVB\nspiraled\tVBD\nspiraling\tVBG\nspiralled\tVBD\nspirited\tVBN\nspirits\tNNS\nSpirits\tNNS\nsplashed\tVBD\nsplints\tNNS\nsplits\tNNS\nSplits\tNNS\nsplits\tVBZ\nsplit\tVB\nsplit\tVBD\nsplit\tVBN\nspoiled\tVBD\nspoil\tVBP\nspoken\tVBN\nspokesmen\tNNS\nSpokesmen\tNNS\nspokes\tNNS\nSpokespersons\tNNS\nspoke\tVBD\nsponsored\tVBD\nsponsored\tVBN\nsponsoring\tVBG\nsponsors\tNNS\nSponsors\tNNS\nsponsors\tVBZ\nsponsor\tVB\nsponsor\tVBP\nspooked\tVBD\nspooked\tVBN\nSpooked\tVBN\nspooks\tNNS\nspook\tVBP\nspoonbills\tNNS\nspoonfuls\tNNS\nspores\tNNS\nsported\tVBD\nsporting-goods\tNNS\nsporting\tVBG\nsportsmen\tNNS\nsports\tNNS\nSports\tNNS\nsports\tVBZ\nspotlight\tVB\nspots\tNNS\nSpots\tNNS\nspots\tVBZ\nspotted\tVBD\nspotted\tVBN\nSpotted\tVBN\nspotting\tVBG\nspotting\tVBG|NN\nspot\tVB\nspouses\tNNS\nspout\tVBP\nsprang\tVBD\nsprawling\tVBG\nspraying\tVBG\nsprays\tNNS\nspreading\tVBG\nspreadsheets\tNNS\nspreads\tNNS\nSpreads\tNNS\nspreads\tVBZ\nspread\tVB\nspread\tVBD\nspread\tVBN\nspread\tVBP\nspringing\tVBG\nsprings\tVBZ\nspring\tVB\nsprinkled\tVBD\nsprinklers\tNNS\nsprinkles\tVBZ\nsprinkle\tVBP\nspritzers\tNNS\nsprouting\tVBG\nsprout\tVB\nsprout\tVBP\nspruce\tVB\nSPRUCING\tVBG\nsprung\tVBN\nspuds\tNNS\nspun\tVBD\nspun\tVBN\nspurned\tVBD\nspurned\tVBN\nspurning\tVBG\nspurns\tVBZ\nspurn\tVB\nspurn\tVBP\nspurred\tVBD\nspurred\tVBN\nSpurred\tVBN\nspurring\tVBG\nspurs\tVBZ\nspurted\tVBD\nspurted\tVBN\nspurts\tNNS\nspurts\tVBZ\nspurt\tVBP\nspur\tVB\nsputtered\tVBD\nsputter\tVBP\nspying\tVBG\nspy\tVB\nsquabbles\tNNS\nsquads\tNNS\nsqualls\tNNS\nsquandered\tVBN\nsquandering\tVBG\nsquared\tVBD\nsquared\tVBN\nsquares\tNNS\nsquare\tVB\nsquaring\tVBG\nsquatted\tVBN\nsqueaking\tVBG\nsqueegee\tVBP\nsqueezed\tVBD\nsqueezed\tVBN\nsqueezes\tVBZ\nsqueeze\tVB\nsqueeze\tVBP\nsqueezing\tVBG\nsquelched\tVBN\nsquelch\tVBP\nsquinted\tVBD\nsquinting\tVBG\nsquirming\tVBG\nstabbed\tVBD\nstabbed\tVBN\nstabbing\tVBG\nstabilized\tVBD\nstabilized\tVBN\nstabilizes\tVBZ\nstabilize\tVB\nstabilize\tVBP\nstabilizing\tVBG\nStabilizing\tVBG\nstacked\tVBN\nstacking\tVBG\nstacks\tNNS\nstack\tVB\nstadiums\tNNS\nStadiums\tNNS\nstaffed\tVBN\nstaffers\tNNS\nStaffers\tNNS\nstaffing\tVBG\nstaffs\tNNS\nstaff\tVB\nstaged\tVBD\nSTAGED\tVBD\nstaged\tVBN\nstages\tNNS\nstages\tVBZ\nstage\tVB\nstage\tVBP\nstagewhispers\tVBZ\nstaggered\tVBD\nstaggering\tVBG\nstaging\tVBG\nstagnated\tVBD\nstagnating\tVBG\nstaid\tVBN\nstains\tNNS\nstain\tVB\nstaircases\tNNS\nstairs\tNNS\nstaked\tVBN\nstakes\tNNS\nstake\tVB\nstalked\tVBN\nstalking\tVBG\nstalled\tVBD\nstalled\tVBN\nStalled\tVBN\nstalling\tVBG\nstalls\tNNS\nstalls\tVBZ\nstall\tVB\nstalwarts\tNNS\nstampeded\tVBN\nstampede\tVB\nstampings\tNNS\nstamping\tVBG\nstamps\tNNS\nstamp\tVB\nstanch\tVB\nstandardize\tVB\nstandards\tNNS\nstandbys\tNNS\nstanding\tVBG\nStanding\tVBG\nstands\tNNS\nstands\tVBZ\nstand\tVB\nStand\tVB\nstand\tVBP\nStand\tVBP\nstapling\tVBG\nstared\tVBD\nstares\tNNS\nstare\tVBP\nstaring\tVBG\nstarring\tVBG\nstars\tNNS\nstars\tVBZ\nstarted\tVBD\nstarted\tVBN\nStarted\tVBN\nstarters\tNNS\nstarting\tVBG\nStarting\tVBG\nstartled\tVBD\nstarts\tNNS\nstarts\tVBP\nstarts\tVBZ\nstart-up\tNNS\nstart-ups\tNNS\nstart\tVB\nstart\tVBP\nstar\tVB\nstarved\tVBN\nstarve\tVB\nStarve\tVB\nstarving\tVBG\nstashed\tVBD\nstash\tVB\nstated\tVBD\nstated\tVBN\nstatements\tNNS\nStatements\tNNS\nstatesmen\tNNS\nstates\tNNS\nStates\tNNS\nSTATES\tNNS\nstates\tVBZ\nstate\tVB\nstate\tVBP\nstating\tVBG\nstationed\tVBN\nstations\tNNS\nstatisticians\tNNS\nstatistics\tNNS\nStatistics\tNNS\nstatues\tNNS\nstatutes\tNNS\nstaunch\tVB\nstave\tVB\nstayed\tVBD\nstayed\tVBN\nstaying\tVBG\nstays\tNNS\nstays\tVBZ\nstay\tVB\nstay\tVBP\nsteadied\tVBD\nsteadied\tVBN\nsteaks\tNNS\nstealing\tVBG\nsteals\tVBZ\nsteal\tVB\nsteal\tVBP\nsteamed\tVBD\nsteamed\tVBN\nsteaming\tVBG\nsteelmakers\tNNS\nSteelmakers\tNNS\nsteels\tNNS\nsteelworkers\tNNS\nsteered\tVBD\nsteered\tVBN\nsteering\tVBG\nsteers\tVBZ\nsteer\tVB\nstemmed\tVBD\nstemmed\tVBN\nstemming\tVBG\nstems\tVBZ\nstem\tVB\nstem\tVBP\nstepped\tVBD\nstepped\tVBN\nstepping\tVBG\nsteps\tNNS\nSteps\tNNS\nsteps\tVBZ\nSteps\tVBZ\nstep\tVB\nStep\tVB\nstep\tVBP\nstereos\tNNS\nstereotypes\tNNS\nsteriles\tNNS\nsterilized\tVBN\nsterilize\tVB\nsterilizing\tVBG\nsteroids\tNNS\nstewards\tNNS\nstewed\tVBD\nstickers\tNNS\nsticking\tVBG\nsticks\tNNS\nsticks\tVBZ\nstick\tVB\nStick\tVB\nstick\tVBP\nstifles\tVBZ\nstifle\tVB\nstifling\tVBG\nstill-raging\tVBG\nstilts\tNNS\nstimulated\tVBN\nstimulate\tVB\nstimulating\tVBG\nstimulators\tNNS\nstimuli\tNNS\nstinging\tVBG\nstingrays\tNNS\nstink\tVBP\nstints\tNNS\nstipends\tNNS\nstipulated\tVBD\nstipulates\tVBZ\nstipulate\tVB\nstirred\tVBD\nstirrings\tNNS\nstirring\tVBG\nstirrups\tNNS\nstirs\tVBZ\nStirs\tVBZ\nstir\tVB\nstir\tVBP\nstitched\tVBN\nstitches\tNNS\nstockbrokers\tNNS\nStockbrokers\tNNS\nstockbuilding\tVBG\nstocked\tVBN\nstockholders\tNNS\nStockholders\tNNS\nstockholdings\tNNS\nstock-index-futures\tNNS\nstock-index\tNNS\nstocking\tVBG\nstock-options\tNNS\nstockpiles\tNNS\nstockpile\tVB\nstocks\tNNS\nStocks\tNNS\nSTOCKS\tNNS\nstocks\tVBZ\nstock\tVB\nstockyards\tNNS\nstoked\tVBN\nstoke\tVB\nstoking\tVBG\nstolen\tVBN\nstole\tVBD\nstomachs\tNNS\nstomach\tVB\nstomped\tVBD\nstomping\tVBG\nstoned\tVBN\nstones\tNNS\nstonewalled\tVBD\nstood\tVBD\nstood\tVBN\nstooges\tNNS\nstools\tNNS\nstoppages\tNNS\nstopped\tVBD\nstopped\tVBN\nstopping\tVBG\nstops\tNNS\nstops\tVBZ\nstop\tVB\nStop\tVB\nstop\tVBP\nstored\tVBD\nstored\tVBN\nstorefronts\tNNS\nstores\tNNS\nStores\tNNS\nstores\tVBZ\nstore\tVB\nstore\tVBP\nstories\tNNS\nstoring\tVBG\nstormed\tVBD\nstorming\tVBG\nstorms\tNNS\nstorytellers\tNNS\nstowed\tVBD\nstraddling\tVBG\nstrafe\tVB\nstraightening\tVBG\nstraighten\tVB\nstrained\tVBN\nstrainers\tNNS\nstraining\tVBG\nstrains\tNNS\nstrain\tVB\nstrain\tVBP\nstraits\tNNS\nstranded\tVBN\nstranding\tVBG\nstrands\tNNS\nstrangled\tVBN\nstrangles\tVBZ\nstrapped\tVBD\nstrapped\tVBN\nstratagems\tNNS\nstrategies\tNNS\nStrategies\tNNS\nstrategists\tNNS\nStrategists\tNNS\nstrawberries\tNNS\nstraying\tVBG\nstray\tVB\nstray\tVBP\nstreaked\tVBD\nstreamed\tVBD\nstreaming\tVBG\nstreamlined\tVBD\nstreamline\tVB\nstreamlining\tVBG\nstreams\tNNS\nstreets\tNNS\nStreetspeak\tVB\nstrengthened\tVBD\nstrengthened\tVBN\nstrengthening\tVBG\nstrengthens\tVBZ\nstrengthen\tVB\nstrengthen\tVBP\nstrengths\tNNS\nstressed\tVBD\nstressed\tVBN\nstresses\tNNS\nstresses\tVBZ\nstressing\tVBG\nstressors\tNNS\nstress\tVB\nstress\tVBP\nstretched\tVBD\nstretched\tVBN\nstretches\tNNS\nstretches\tVBZ\nstretching\tVBG\nStretching\tVBG\nstretch\tVB\nstretch\tVBP\nstrewn\tVBN\nstricken\tVBN\nstrides\tNNS\nstrikers\tNNS\nstrikes\tNNS\nStrikes\tNNS\nstrikes\tVBZ\nstrike\tVB\nstrike\tVBP\nstriking\tVBG\nStriking\tVBG\nstrings\tNNS\nstring\tVB\nstripes\tNNS\nstripped\tVBD\nstripped\tVBN\nstripping\tVBG\nstrips\tNNS\nStrips\tNNS\nstrips\tVBZ\nstrip\tVB\nstriven\tVBN\nstrives\tVBZ\nstrive\tVBP\nstriving\tVBG\nstrode\tVBD\nstrokes\tNNS\nStrokes\tNNS\nstrokes\tVBZ\nStroking\tVBG\nstrolling\tVBG\nstroll\tVB\nstroll\tVBP\nstrongholds\tNNS\nstrove\tVBD\nstruck\tVBD\nSTRUCK\tVBD\nstruck\tVBN\nstructured\tVBD\nstructured\tVBN\nstructures\tNNS\nstructure\tVB\nstructuring\tVBG\nstruggled\tVBD\nSTRUGGLED\tVBD\nstruggled\tVBN\nstruggles\tNNS\nstruggles\tVBZ\nstruggle\tVB\nstruggling\tVBG\nSTUBBED\tVBN\nstuck\tVBD\nstuck\tVBN\nstudded\tVBN\nstudent-athletes\tNNS\nstudentled\tVBN\nstudents\tNNS\nStudents\tNNS\nSTUDENTS\tNNS\nstudied\tVBD\nstudied\tVBN\nstudies\tNNS\nStudies\tNNS\nstudies\tVBZ\nstudios\tNNS\nStudios\tNNS\nstudying\tVBG\nstudy\tVB\nstudy\tVBP\nstuffed\tVBD\nstuffed\tVBN\nStuffing\tVBG\nstuff\tVBP\nstumbled\tVBD\nstumbled\tVBN\nstumble\tVB\nstumble\tVBP\nstumbling\tVBG\nstung\tVBN\nStung\tVBN\nstunned\tVBD\nstunned\tVBN\nStunned\tVBN\nstunted\tVBN\nstyled\tVBN\nstyles\tNNS\nstyling\tVBG\nstymied\tVBN\nsubcommittees\tNNS\nsubcompacts\tNNS\nsubconferences\tNNS\nsubcontractors\tNNS\nSubcontractors\tNNS\nsubcontract\tVB\nsubdued\tVBN\nsubgroups\tNNS\nsubjected\tVBN\nsubjecting\tVBG\nsubjects\tNNS\nsubjects\tVBZ\nsubject\tVB\nsublet\tVB\nsubmarines\tNNS\nsub-markets\tNNS\nsubmits\tVBZ\nsubmitted\tVBD\nsubmitted\tVBN\nsubmitting\tVBG\nsubmit\tVB\nsubmit\tVBP\nsubordinated\tVBD\nsubordinated\tVBN\nsubordinates\tNNS\nsubordinates\tVBZ\nsubordinate\tVB\nsubpoenaed\tVBN\nsubpoenas\tNNS\nsubpoena\tVB\nsubscribed\tVBN\nsubscribers\tNNS\nSubscribers\tNNS\nsubscribes\tVBZ\nsubscribe\tVB\nsubscribe\tVBP\nsubscribing\tVBG\nSubscribing\tVBG\nsubscriptions\tNNS\nsub-segments\tNNS\nsubsided\tVBD\nsubsided\tVBN\nsubsides\tNNS\nsubsidiaries\tNNS\nsubsidies\tNNS\nSubsidies\tNNS\nsubsidized\tVBN\nsubsidizes\tVBZ\nsubsidize\tVB\nsubsidizing\tVBG\nSubsidizing\tVBG\nsubskills\tNNS\nsubstances\tNNS\nsubstantiate\tVB\nsubstations\tNNS\nsubstituted\tVBD\nsubstituted\tVBN\nsubstitutes\tNNS\nsubstitute\tVB\nsubstituting\tVBG\nSubstituting\tVBG\nsubstracting\tVBG\nsubsumed\tVBN\nsubtilis\tNNS\nsubtitled\tVBN\nsubtracted\tVBD\nsubtracted\tVBN\nsubtracting\tVBG\nsubtract\tVB\nsub-underwriters\tNNS\nsub-underwriting\tVBG\nsuburbs\tNNS\nsubversives\tNNS\nsubverted\tVBN\nsubverts\tVBZ\nsubvert\tVB\nsubways\tNNS\nsucceeded\tVBD\nsucceeded\tVBN\nsucceeding\tVBG\nSucceeding\tVBG\nsucceeds\tVBZ\nsucceed\tVB\nsucceed\tVBP\nsuccesses\tNNS\nsuccessors\tNNS\nsuccumbed\tVBD\nsuccumbed\tVBN\nsuccumbing\tVBG\nsuckers\tNNS\nsucks\tVBZ\nsued\tVBD\nsued\tVBN\nSUES\tVBZ\nsue\tVB\nsuffered\tVBD\nsuffered\tVBN\nsuffering\tVBG\nSuffering\tVBG\nsuffers\tVBZ\nsuffer\tVB\nsuffer\tVBP\nsufficed\tVBD\nsuffice\tVB\nSuffice\tVB\nsuggested\tVBD\nsuggested\tVBN\nsuggesting\tVBG\nsuggestions\tNNS\nsuggests\tVBZ\nsuggest\tVB\nsuggest\tVBP\nsuing\tVBG\nsuited\tVBN\nsuites\tNNS\nsuitors\tNNS\nSuitors\tNNS\nsuits\tNNS\nSuits\tNNS\nsuits\tVBZ\nsuit\tVB\nsummaries\tNNS\nsummarized\tVBD\nsummarize\tVB\nsummarizing\tVBG\nSummarizing\tVBG\nsummed\tVBD\nsummers\tNNS\nsummoned\tVBD\nsummoned\tVBN\nsummoning\tVBG\nsummon\tVB\nsums\tNNS\nsums\tVBZ\nsum\tVB\nSundays\tNNS\nsunflowers\tNNS\nsunglasses\tNNS\nsung\tVBN\nsunk\tVBD\nsunk\tVBN\nsunsets\tNNS\nsuperceded\tVBD\nsupercede\tVBP\nsupercomputers\tNNS\nsuperconcentrates\tNNS\nSuperconcentrates\tNNS\nsuperconductors\tNNS\nSuperconductors\tNNS\nsuperimposed\tVBN\nsuperintendents\tNNS\nsuperiors\tNNS\nsupermarkets\tNNS\nSupermarkets\tNNS\nsuperpowers\tNNS\nsuperpremiums\tNNS\nsuperseded\tVBD\nsuperseded\tVBN\nsupersede\tVB\nsuperstars\tNNS\nsupervised\tVBD\nsupervised\tVBN\nsupervises\tVBZ\nsupervise\tVB\nsupervise\tVBP\nsupervising\tVBG\nsupervisors\tNNS\nsupplements\tNNS\nsupplement\tVB\nsupplied\tVBD\nsupplied\tVBN\nsuppliers\tNNS\nSuppliers\tNNS\nsupplies\tNNS\nsupplies\tVBZ\nsupplying\tVBG\nsupply\tVB\nsupply\tVBP\nsupported\tVBD\nsupported\tVBN\nsupporters\tNNS\nSupporters\tNNS\nsupporting\tVBG\nSupporting\tVBG\nsupports\tNNS\nsupports\tVBZ\nsupport\tVB\nsupport\tVBP\nsupposed\tVBD\nsupposed\tVBN\nsuppose\tVB\nSuppose\tVB\nsuppose\tVBP\nsuppressants\tNNS\nsuppressed\tVBD\nsuppressed\tVBN\nsuppress\tVB\nsurfaced\tVBD\nsurfaced\tVBN\nsurfaces\tVBZ\nsurface\tVB\nsurface\tVBP\nsurfacing\tVBG\nsurfers\tNNS\nsurged\tVBD\nSURGED\tVBD\nsurged\tVBN\nsurges\tNNS\nsurges\tVBZ\nsurge\tVB\nsurging\tVBG\nsurmounting\tVBG\nsurpassed\tVBD\nsurpassed\tVBN\nsurpassing\tVBG\nsurpass\tVB\nsurpass\tVBP\nsurpluses\tNNS\nsurprised\tVBD\nsurprised\tVBN\nsurprises\tNNS\nSurprises\tNNS\nsurprise\tVB\nsurprising\tVBG\nsurrendered\tVBD\nsurrendered\tVBN\nsurrendering\tVBG\nsurrender\tVB\nSurrender\tVB\nsurrounded\tVBN\nSurrounded\tVBN\nsurrounded\tVBN|JJ\nsurrounding\tVBG\nsurround\tVBP\nsurtaxes\tNNS\nsurveyed\tVBD\nsurveyed\tVBN\nSurveying\tVBG\nsurveys\tNNS\nSurveys\tNNS\nsurveys\tVBZ\nsurvey\tVB\nsurvey\tVBP\nsurvived\tVBD\nSurvived\tVBD\nsurvived\tVBN\nsurvives\tVBZ\nsurvive\tVB\nSurvive\tVB\nsurvive\tVBP\nsurviving\tVBG\nSurviving\tVBG\nsurvivors\tNNS\nsuspected\tVBD\nsuspected\tVBN\nsuspecting\tVBG\nsuspects\tNNS\nsuspects\tVBZ\nsuspect\tVB\nsuspect\tVBP\nsuspended\tVBD\nsuspended\tVBN\nsuspending\tVBG\nsuspend\tVB\nsuspensions\tNNS\nsuspicions\tNNS\nsustained\tVBD\nsustained\tVBN\nsustaining\tVBG\nsustains\tVBZ\nsustain\tVB\nsutures\tNNS\nsuvivors\tNNS\n's\tVBZ\ns\tVBZ\n'S\tVBZ\nswallowed\tVBN\nswallowing\tVBG\nswallow\tVB\nswamped\tVBD\nswamped\tVBN\nswamp\tVB\nswans\tNNS\nswapped\tVBN\nswapping\tVBG\nswaps\tNNS\nswap\tVB\nswap\tVBP\nswarms\tNNS\nswathed\tVBN\nswayed\tVBN\nswaying\tVBG\nsway\tVB\nsway\tVBP\nswears\tVBZ\nswear\tVB\nswear\tVBP\nsweated\tVBN\nsweaters\tNNS\nSweating\tVBG\nsweatshirts\tNNS\nsweat\tVB\nsweat\tVBD\nsweat\tVBP\nSwedes\tNNS\nsweepers\tNNS\nsweeping\tVBG\nsweeps\tNNS\nsweepstakes\tNNS\nsweeps\tVBZ\nsweep\tVB\nsweetened\tVBD\nsweetened\tVBN\nsweeteners\tNNS\nsweeten\tVB\nsweets\tNNS\nswelled\tVBD\nswelled\tVBN\nswelling\tVBG\nswells\tNNS\nswells\tVBZ\nswell\tVB\nswell\tVBP\nswept\tVBD\nswept\tVBN\nswerve\tVBP\nswimming\tVBG\nswim\tVBP\nswindled\tVBN\nswine\tNNS\nswinging\tVBG\nswings\tNNS\nswings\tVBZ\nswing\tVB\nswing\tVBP\nswitched\tVBD\nswitched\tVBN\nswitchers\tNNS\nswitches\tNNS\nswitches\tVBZ\nswitching\tVBG\nSWITCHING\tVBG\nswitch\tVB\nswitch\tVBP\nswiveling\tVBG\nSwiveling\tVBG\nswore\tVBD\nsworn\tVBN\nswung\tVBD\nSWUNG\tVBD\nswung\tVBN\nsycophants\tNNS\nsymbolized\tVBN\nsymbolizes\tVBZ\nsymbols\tNNS\nsympathies\tNNS\nsympathizers\tNNS\nsympathize\tVBP\nsymposiums\tNNS\nsymptoms\tNNS\nSyms\tNNS\nsynchronized\tVBN\nSynchronized\tVBN\nsynchronize\tVBP\nsyndciated\tVBN\nsyndicated\tVBN\nsyndicates\tNNS\nSyndicates\tNNS\nsyndicate\tVB\nsyndicating\tVBG\nsyndications\tNNS\nsynergies\tNNS\nsyngeries\tNNS\nsynonyms\tNNS\nsynthesizers\tNNS\nsynthesize\tVB\nsynthetics\tNNS\nsystems\tNNS\nSystems\tNNS\ntabacs\tNNS\ntables\tNNS\ntablespoons\tNNS\ntablets\tNNS\ntabloids\tNNS\ntaboo\tVB\ntabs\tNNS\ntacked\tVBD\ntacking\tVBG\ntackled\tVBN\ntackles\tVBZ\ntackle\tVB\ntackle\tVBP\ntackling\tVBG\ntacos\tNNS\ntactics\tNNS\ntagged\tVBN\ntags\tNNS\ntag\tVB\ntailing\tVBG\ntailored\tVBN\ntailoring\tVBG\ntails\tNNS\ntainted\tVBN\ntaken\tVBN\nTaken\tVBN\ntakeovers\tNNS\ntakers\tNNS\ntakes\tVBZ\nTakes\tVBZ\ntaketh\tVB\ntake\tVB\nTake\tVB\ntake\tVBP\nTake\tVBP\ntakings\tNNS\ntaking\tVBG\nTaking\tVBG\ntalents\tNNS\ntales\tNNS\nTales\tNNS\ntalked\tVBD\ntalked\tVBN\ntalking\tVBG\nTalking\tVBG\ntalks\tNNS\nTalks\tNNS\ntalks\tVBZ\nTALKS\tVBZ\ntalk\tVB\ntalk\tVBP\ntallied\tVBD\ntallies\tNNS\ntallying\tVBG\ntaming\tVBG\nTaming\tVBG\ntampering\tVBG\ntampers\tNNS\ntamper\tVB\ntampons\tNNS\nTandy\tVB\ntangled\tVBN\ntangle\tVBP\ntangoed\tVBD\ntanked\tVBN\ntankers\tNNS\nTankers\tNNS\ntanks\tNNS\nTanks\tNNS\ntank\tVB\ntanned\tVBN\ntans\tNNS\ntantalizing\tVBG\ntaped\tVBD\ntaped\tVBN\nTaped\tVBN\ntapering\tVBG\ntapers\tNNS\ntapers\tVBZ\ntaper\tVB\ntapes\tNNS\ntapestries\tNNS\ntape\tVB\ntape\tVBP\ntapings\tNNS\ntaping\tVBG\ntapped\tVBD\ntapped\tVBN\ntapping\tVBG\ntaps\tNNS\nTaps\tVBZ\ntap\tVB\nTap\tVB\ntargeted\tVBD\ntargeted\tVBN\ntargeting\tVBG\ntargeting\tVBG|NN\ntargets\tNNS\ntargets\tVBZ\ntarget\tVB\ntarget\tVBP\ntariffs\tNNS\ntarnished\tVBD\ntarnished\tVBN\ntarnish\tVB\ntarred\tVBD\ntartans\tNNS\ntasks\tNNS\ntassels\tNNS\ntastes\tNNS\nTastes\tNNS\ntastes\tVBZ\ntaste\tVB\ntaste\tVBP\ntaught\tVBD\ntaught\tVBN\ntaunted\tVBD\ntax-deductions\tNNS\ntaxed\tVBD\ntaxed\tVBN\ntaxes\tNNS\nTaxes\tNNS\ntaxes\tVBZ\ntax-exempts\tNNS\nTax-exempts\tNNS\ntaxpayers\tNNS\nTaxpayers\tNNS\nTAXPAYERS\tNNS\ntax-reducing\tVBG\ntax\tVB\ntax-writers\tNNS\nT-bills\tNNS\nteachers\tNNS\nteaches\tVBZ\nteaching\tVBG\nteach\tVB\nTEACH\tVB\nteach\tVBP\nteamed\tVBD\nteaming\tVBG\nteammates\tNNS\nteams\tNNS\nTeams\tNNS\nteam\tVB\ntearing\tVBG\ntears\tNNS\ntear\tVB\nteased\tVBN\nteaspoons\tNNS\ntechnical-services\tNNS\ntechnicians\tNNS\ntechniques\tNNS\ntechnocrats\tNNS\ntechnologies\tNNS\nteemed\tVBD\nteeming\tVBG\nteen-agers\tNNS\nteenagers\tNNS\nteens\tNNS\nTeens\tNNS\nteetering\tVBG\nteeth\tNNS\ntelecines\tNNS\ntelecommunications\tNNS\ntelecommunications\tNNS|NN\nTelecussed\tVBD\ntelegraphed\tVBD\ntelegraph\tVBP\ntelemarketers\tNNS\ntelephoned\tVBD\nTelephone-operations\tNNS\ntelephones\tNNS\ntelephone\tVB\ntelephoning\tVBG\ntelesystems\tNNS\nTelesystems\tNNS\ntelevised\tVBN\ntelevisions\tNNS\ntelexes\tNNS\ntellers\tNNS\ntelling\tVBG\ntells\tVBZ\ntell\tVB\nTell\tVB\ntell\tVBP\ntemblors\tNNS\ntemperatures\tNNS\ntempered\tVBN\ntemps\tNNS\ntempted\tVBD\ntempted\tVBN\ntempts\tVBZ\ntempt\tVB\ntempt\tVBP\ntenants\tNNS\ntended\tVBD\ntended\tVBN\ntendencies\tNNS\ntendered\tVBD\ntendered\tVBN\ntendering\tVBG\ntenders\tNNS\nTenders\tNNS\ntender\tVB\ntender\tVBP\ntending\tVBG\ntends\tVBZ\ntend\tVB\ntend\tVBP\ntenets\tNNS\ntensions\tNNS\ntens\tNNS\ntenths\tNNS\ntermed\tVBD\ntermed\tVBN\nterminals\tNNS\nTerminals\tNNS\nterminated\tVBD\nterminated\tVBN\nterminate\tVB\nterminating\tVBG\nterminations\tNNS\nterms\tNNS\nTerms\tNNS\nterm\tVB\nterrified\tVBN\nterrify\tVB\nterritories\tNNS\nterrorists\tNNS\ntestaments\tNNS\ntest-drive\tVB\ntest-drive\tVBP\ntested\tVBD\ntested\tVBN\ntesters\tNNS\ntest-fired\tVBN\ntestified\tVBD\ntestifies\tVBZ\ntestifying\tVBG\nTestifying\tVBG\ntestify\tVB\ntestify\tVBP\ntesting\tVBG\ntesting\tVBG|NN\ntests\tNNS\nTESTS\tNNS\ntests\tVBZ\ntest\tVB\ntest\tVBP\ntethered\tVBN\ntextbooks\tNNS\ntextiles\tNNS\ntexts\tNNS\nthanked\tVBD\nthanking\tVBG\nthanks\tNNS\nThanks\tNNS\nTHANKS\tNNS\nthank\tVB\nThank\tVB\nthank\tVBP\nthat\tVBP\ntheaters\tNNS\nthefts\tNNS\nThefts\tNNS\nthemed\tVBN\nthemes\tNNS\ntheologians\tNNS\ntheories\tNNS\ntheorists\tNNS\nTheorists\tNNS\ntheorized\tVBD\ntheory-teaching\tVBG\ntherapies\tNNS\ntherapists\tNNS\nthermometers\tNNS\nThe\tVB\nthe\tVBP\nthieves\tNNS\nThieves\tNNS\nthings\tNNS\nThings\tNNS\nthinking\tVBG\nthinks\tVBZ\nthink\tVB\nThink\tVB\nthink\tVBP\nThink\tVBP\nthinned\tVBN\nthinning\tVBG\nthirds\tNNS\nthirties\tNNS\nThirties\tNNS\nthistles\tNNS\nthoroughbreds\tNNS\nthoughts\tNNS\nthought\tVBD\nthought\tVBN\nthousands\tNNS\nThousands\tNNS\nthrashed\tVBD\nthrashing\tVBG\nthrash\tVB\nthreads\tNNS\nthreatened\tVBD\nthreatened\tVBN\nthreatening\tVBG\nthreatens\tVBZ\nthreaten\tVB\nthreaten\tVBP\nthreats\tNNS\nthree-fourths\tNNS\nthree-quarters\tNNS\nthree-sevenths\tNNS\nthrew\tVBD\nthrifts\tNNS\nThrifts\tNNS\nthrilled\tVBN\nthrills\tVBZ\nthrill\tVB\nthrived\tVBD\nthrives\tVBZ\nthrive\tVB\nthrive\tVBP\nThrive\tVBP\nthriving\tVBG\nthroats\tNNS\nthrowers\tNNS\nthrowing\tVBG\nThrowing\tVBG\nthrown\tVBN\nthrows\tVBZ\nthrow\tVB\nThrow\tVB\nthrow\tVBP\nthrusting\tVBG\nthrusts\tNNS\nthrust\tVBD\nthrust\tVBN\nthugs\tNNS\nthumbing\tVBG\nthumbs\tNNS\nthumbs\tVBZ\nthumb\tVB\nthundered\tVBD\nthwarted\tVBD\nthwarted\tVBN\nthwarting\tVBG\nthwart\tVB\nticked\tVBN\nticketed\tVBN\nticketing\tVBG\ntickets\tNNS\nticking\tVBG\nticks\tVBZ\ntick\tVB\ntidbits\tNNS\ntides\tNNS\ntied\tVBD\ntied\tVBN\ntie-ins\tNNS\nties\tNNS\nties\tVBZ\ntie-ups\tNNS\ntie\tVB\ntigers\tNNS\ntightened\tVBD\ntightened\tVBN\ntightening\tVBG\ntighten\tVB\ntighten\tVBP\ntiles\tNNS\ntilted\tVBN\ntilts\tVBZ\ntilt\tVB\ntimberlands\tNNS\ntimbers\tNNS\ntimed\tVBD\ntimed\tVBN\ntime-hotels\tNNS\ntime-shares\tNNS\ntimes\tNNS\nTimes\tNNS\ntime\tVB\ntiming\tVBG\ntimpani\tNNS\ntinged\tVBN\ntinges\tNNS\ntinkered\tVBN\ntinkering\tVBG\ntinker\tVB\ntins\tNNS\ntipped\tVBD\ntips\tNNS\ntipsters\tNNS\ntiptoed\tVBD\ntiptoed\tVBN\ntiptoe\tVB\ntip\tVB\ntired\tVBN\ntire-kickers\tNNS\ntires\tNNS\ntires\tVBZ\ntissues\tNNS\ntitans\tNNS\ntitled\tVBD\ntitled\tVBN\nTitled\tVBN\ntitles\tNNS\ntoasted\tVBD\ntoddlers\tNNS\ntoeholds\tNNS\ntoes\tNNS\ntoiled\tVBD\ntoiletries\tNNS\ntoiling\tVBG\ntoil\tVB\ntoil\tVBP\ntold\tVBD\ntold\tVBN\ntolerate\tVB\ntolls\tNNS\nTolls\tNNS\ntomatoes\tNNS\ntoned\tVBN\ntones\tNNS\ntongues\tNNS\ntonnages\tNNS\ntons\tNNS\nTons\tNNS\ntook\tVBD\ntools\tNNS\ntooted\tVBD\ntopics\tNNS\ntopped\tVBD\ntopped\tVBN\ntopping\tVBG\nTopping\tVBG\ntoppled\tVBD\ntoppled\tVBN\ntopple\tVB\ntops\tNNS\ntops\tVBZ\ntop\tVB\ntop\tVBP\ntorched\tVBD\ntorched\tVBN\ntore\tVBD\nTories\tNNS\ntormentors\tNNS\ntorments\tVBZ\ntorment\tVB\ntornadoes\tNNS\ntorn\tVBN\ntorpedoed\tVBN\ntorpedo\tVB\ntorts\tNNS\ntortured\tVBN\ntorture\tVB\ntorture\tVBP\ntossed\tVBD\ntossed\tVBN\ntossers\tNNS\ntossing\tVBG\ntoss\tVB\ntotaled\tVBD\ntotaled\tVBN\ntotaling\tVBG\ntotalling\tVBG\ntotals\tNNS\nTotals\tNNS\ntotals\tVBZ\ntotal\tVB\ntotal\tVBP\ntoted\tVBN\ntote\tVB\ntoting\tVBG\ntottering\tVBG\ntotter\tVB\ntouched\tVBD\ntouched\tVBN\ntouches\tNNS\nTouches\tVBZ\ntouching\tVBG\ntouch\tVB\ntouch\tVBP\ntoughened\tVBD\ntoughen\tVB\ntough\tVB\ntoured\tVBD\ntouring\tVBG\ntourists\tNNS\ntournaments\tNNS\ntours\tNNS\ntours\tVBZ\ntour\tVB\ntouted\tVBD\ntouted\tVBN\ntouting\tVBG\ntouts\tVBZ\ntout\tVB\ntowels\tNNS\nTowering\tVBG\ntowers\tNNS\ntower\tVB\ntownhouses\tNNS\ntownships\tNNS\ntowns\tNNS\ntows\tNNS\ntoying\tVBG\ntoys\tNNS\ntraced\tVBD\ntraced\tVBN\ntraces\tNNS\ntraces\tVBZ\ntrace\tVB\ntracing\tVBG\ntracked\tVBD\ntracked\tVBN\ntracking\tVBG\ntracks\tNNS\ntracks\tVBZ\ntrack\tVB\ntrack\tVBP\ntractors\tNNS\ntracts\tNNS\ntraded\tVBD\ntraded\tVBN\ntraded\tVBN|VBD\ntrademarks\tNNS\ntrade-offs\tNNS\ntradeoffs\tNNS\ntraders\tNNS\nTraders\tNNS\ntrades\tNNS\ntrades\tVBZ\ntrade\tVB\ntrade\tVBP\ntrading\tVBG\nTrading\tVBG\ntrading\tVBG|NN\ntraditionalists\tNNS\ntraditions\tNNS\ntraduced\tVBN\ntraduce\tVB\ntraffickers\tNNS\ntragedies\tNNS\ntrailed\tVBD\ntrailed\tVBN\ntrailers\tNNS\ntrailing\tVBG\ntrails\tNNS\ntrails\tVBZ\ntrail\tVB\ntrail\tVBP\ntrained\tVBN\nTrained\tVBN\ntrainers\tNNS\ntraining\tVBG\ntrains\tNNS\nTrains\tNNS\ntrains\tVBZ\ntrain\tVB\ntrain\tVBP\ntraipse\tVB\ntraits\tNNS\ntramping\tVBG\ntrampled\tVBN\ntrampling\tVBG\ntransacted\tVBN\ntransacting\tVBG\ntransactions\tNNS\nTransactions\tNNS\ntransact\tVB\ntranscribe\tVBP\ntranscripts\tNNS\ntransferred\tVBD\ntransferred\tVBN\ntransferring\tVBG\ntransfers\tNNS\ntransfers\tVBZ\ntransfer\tVB\ntransformed\tVBD\ntransformed\tVBN\ntransforming\tVBG\ntransforms\tVBZ\ntransform\tVB\ntransfusions\tNNS\ntransistors\tNNS\ntranslated\tVBN\nTranslated\tVBN\ntranslate\tVB\ntranslate\tVBP\ntranslating\tVBG\ntranslations\tNNS\ntransmissions\tNNS\ntransmitted\tVBD\ntransmitted\tVBN\ntransmitting\tVBG\ntransmit\tVB\ntransmogrified\tVBD\ntransmogrified\tVBN\ntransplanted\tVBN\ntransplanting\tVBG\ntransplants\tNNS\ntransplant\tVB\ntransported\tVBD\ntransported\tVBN\ntransporting\tVBG\ntransports\tNNS\nTransports\tNNS\ntransports\tVBZ\ntransport\tVB\nTransport\tVB\ntransvestites\tNNS\ntrapped\tVBN\ntrappings\tNNS\ntrapping\tVBG\ntraps\tNNS\ntrashing\tVBG\ntrash\tVB\ntraumas\tNNS\ntraumatized\tVBD\ntraumatized\tVBN\ntravails\tNNS\ntraveled\tVBD\ntraveled\tVBN\ntravelers\tNNS\nTravelers\tNNS\ntraveling\tVBG\nTraveling\tVBG\ntravelogues\tNNS\ntravels\tNNS\ntravels\tVBZ\nTRAVELS\tVBZ\ntravel\tVB\ntravel\tVBP\ntrays\tNNS\ntreadmills\tNNS\ntreads\tVBZ\ntread\tVB\ntreasurers\tNNS\ntreasures\tNNS\nTreasures\tNNS\ntreasuries\tNNS\nTreasurys\tNNS\ntreated\tVBD\ntreated\tVBN\ntreating\tVBG\nTREATING\tVBG\ntreatises\tNNS\ntreatments\tNNS\ntreats\tNNS\ntreats\tVBZ\ntreat\tVB\ntreat\tVBP\ntrebled\tVBD\ntrees\tNNS\ntrekked\tVBD\ntrembling\tVBG\ntremors\tNNS\ntrenches\tNNS\ntrending\tVBG\ntrend-setters\tNNS\ntrends\tNNS\ntrend\tVB\ntrespasses\tNNS\ntrespass\tVBP\ntrials\tNNS\ntriangles\tNNS\ntribes\tNNS\ntributes\tNNS\ntrickle\tVB\ntrickle\tVBP\ntrickling\tVBG\ntricks\tNNS\ntrick\tVB\ntried\tVBD\ntried\tVBN\ntries\tNNS\ntries\tVBZ\ntriggered\tVBD\ntriggered\tVBN\ntriggering\tVBG\nTriggering\tVBG\ntriggers\tVBZ\ntrigger\tVB\ntrigger\tVBP\ntrillions\tNNS\ntrills\tNNS\ntrimesters\tNNS\ntrimmed\tVBD\ntrimmed\tVBN\ntrimming\tVBG\nTRIMMING\tVBG\ntrim\tVB\nTrim\tVBP\ntripled\tVBD\ntripled\tVBN\ntriples\tNNS\ntriple\tVB\ntripling\tVBG\ntripped\tVBN\ntrips\tNNS\ntriumphed\tVBD\ntrivialize\tVB\ntrivia\tNNS\ntroops\tNNS\nTroops\tNNS\nTROs\tNNS\ntrotted\tVBD\ntrotted\tVBN\nTrotting\tVBG\ntroubled\tVBD\ntroubled\tVBN\nTroubled\tVBN\ntroublemakers\tNNS\ntroubles\tNNS\nTROUBLES\tNNS\ntroubles\tVBZ\ntroubling\tVBG\ntroughed\tVBD\ntroughs\tNNS\ntroupes\tNNS\ntrousers\tNNS\ntrout\tNNS\nTruckers\tNNS\ntruck-parts\tNNS\ntrucks\tNNS\ntrucks\tVBZ\ntruck\tVBP\ntrudging\tVBG\nTrumped\tVBN\ntrumpeting\tVBG\ntrumpets\tVBZ\ntrumpet\tVBP\ntrundles\tVBZ\ntrunks\tNNS\ntrusted\tVBD\ntrusted\tVBN\ntrustees\tNNS\nTrustees\tNNS\ntrusting\tVBG\ntrusts\tNNS\nTRUSTS\tNNS\ntrusts\tVBZ\ntrust\tVB\ntrust\tVBP\ntruths\tNNS\ntrying\tVBG\nTrying\tVBG\ntryouts\tNNS\ntry\tVB\nTry\tVB\ntry\tVBP\nTry\tVBP\nT-shirt\tNNS\nT-shirts\tNNS\ntubes\tNNS\ntubs\tNNS\ntucked\tVBD\ntucked\tVBN\ntucking\tVBG\ntuck\tVB\ntugged\tVBD\ntugging\tVBG\ntuitions\tNNS\nTuitions\tNNS\ntumbled\tVBD\ntumbled\tVBN\ntumbles\tNNS\ntumbles\tVBZ\ntumble\tVB\ntumbling\tVBG\ntumors\tNNS\ntumor-suppressors\tNNS\ntuned\tVBN\ntunes\tNNS\ntune\tVB\ntune\tVBP\ntuning\tVBG\ntunnels\tNNS\nturbans\tNNS\nturbines\tNNS\nturboprops\tNNS\nturmoils\tNNS\nturnarounds\tNNS\nturned\tVBD\nturned\tVBN\nTurned\tVBN\nturning\tVBG\nturn-ons\tNNS\nturns\tNNS\nturns\tVBZ\nTurns\tVBZ\nTURNS\tVBZ\nturn\tVB\nturn\tVBP\nturtles\tNNS\ntusks\tNNS\ntutored\tVBN\ntutorials\tNNS\ntutoring\tVBG\ntuxedos\tNNS\nTVs\tNNS\ntweezers\tNNS\ntwenties\tNNS\ntwiddling\tVBG\ntwinned\tVBN\ntwisted\tVBN\ntwisting\tVBG\ntwists\tNNS\ntwists\tVBZ\ntwist\tVBP\ntwitch\tVB\ntwo-hundredths\tNNS\ntwo-sevenths\tNNS\ntwo-thirds\tNNS\ntwothirds\tNNS\nTwo-thirds\tNNS\ntycoons\tNNS\ntying\tVBG\ntyped\tVBN\ntypefaces\tNNS\ntypes\tNNS\ntypewriters\tNNS\ntyphoons\tNNS\ntypified\tVBD\ntypifies\tVBZ\nUFOs\tNNS\nulcers\tNNS\nultimatums\tNNS\numbrellas\tNNS\nun-advertisers\tNNS\nunbanning\tVBG\nunblock\tVB\nuncertainties\tNNS\nunchlorinated\tVBN\nuncles\tNNS\nunconsolidated\tVBD\nunconsolidated\tVBN\nuncovered\tVBD\nuncovered\tVBN\nuncovering\tVBG\nuncover\tVB\nUnderclass\tNNS\nundercutting\tVBG\nundercut\tVB\nundercut\tVBD\nundercut\tVBN\nundercut\tVBP\nunderestimated\tVBD\nunderestimated\tVBN\nunderfunded\tVBN\nundergirded\tVBD\nundergoing\tVBG\nundergone\tVBN\nundergo\tVB\nundergo\tVBP\nunderlie\tVB\nunderlined\tVBD\nunderline\tVB\nunderlying\tVBG\nUnderlying\tVBG\nundermined\tVBD\nundermined\tVBN\nundermine\tVB\nundermine\tVBP\nundermining\tVBG\nunderperformed\tVBN\nunderperformers\tNNS\nunderperforming\tVBG\nunderperforms\tVBZ\nunderperform\tVB\nunderpinned\tVBN\nunderpinning\tVBG\nunderpin\tVB\nunderpriced\tVBN\nunderreacting\tVBG\nunderscored\tVBD\nunderscored\tVBN\nunderscores\tVBZ\nunderscore\tVB\nunderscore\tVBP\nunderscoring\tVBG\nUnderscoring\tVBG\nunderselling\tVBG\nunderstaffs\tVBZ\nunderstand\\/adopt\tVB\nunderstanding\tVBG\nunderstands\tVBZ\nunderstand\tVB\nunderstand\tVBP\nunderstated\tVBD\nunderstated\tVBN\nunderstate\tVBP\nunderstating\tVBG\nunderstood\tVBD\nunderstood\tVBN\nundertaken\tVBN\nundertake\tVB\nundertakings\tNNS\nundertaking\tVBG\nundertones\tNNS\nundertook\tVBD\nunderutilized\tVBN\nundervalued\tVBD\nundervalued\tVBN\nunderweighted\tVBN\nunderwent\tVBD\nunderwhelmed\tVBN\nunderwiters\tNNS\nunderwriters\tNNS\nUnderwriters\tNNS\nunderwrites\tVBZ\nunderwrite\tVB\nunderwrite\tVBP\nunderwritings\tNNS\nunderwriting\tVBG\nUnderwriting\tVBG\nunderwritten\tVBN\nunderwrote\tVBD\nundid\tVBD\nundone\tVBN\nundo\tVB\nundulate\tVB\nundulate\tVBP\nunearthed\tVBD\nunexecuted\tVBN\nunfazed\tVBN\nunfocussed\tVBN\nunfolded\tVBD\nunfolding\tVBG\nunfolds\tVBZ\nunfold\tVB\nunhinged\tVBN\nunhocked\tVBN\nunhusked\tVBN\nunified\tVBN\nuniforms\tNNS\nunifying\tVBG\nunify\tVB\nUnify\tVB\nunionists\tNNS\nunionized\tVBD\nunions\tNNS\nUnions\tNNS\nunited\tVBN\nUnited\tVBN\nunites\tVBZ\nunite\tVB\nunitholders\tNNS\nUnitholders\tNNS\nunits\tNNS\nUnits\tNNS\nuniversities\tNNS\nUniversities\tNNS\nunknowns\tNNS\nunleashed\tVBD\nunleashed\tVBN\nunleashes\tVBZ\nunleashing\tVBG\nunleash\tVB\nunleash\tVBP\nunloaded\tVBN\nunloading\tVBG\nUnloading\tVBG\nunload\tVB\nunload\tVBP\nunlocked\tVBD\nunlocks\tVBZ\nunlock\tVBP\nunmasks\tVBZ\nunmask\tVB\nunmaterialized\tVBN\nunnerved\tVBD\nunnerving\tVBG\nunplug\tVB\nunpolarizing\tVBG\nunraveled\tVBD\nunraveled\tVBN\nunraveling\tVBG\nunravel\tVB\nunrolls\tVBZ\nunroll\tVBP\nunseated\tVBD\nunseating\tVBG\nunsettled\tVBD\nunsettled\tVBN\nunveiled\tVBD\nunveiled\tVBN\nUnveiled\tVBN\nunveiling\tVBG\nunveils\tVBZ\nunveil\tVB\nunwavering\tVBG\nunwinding\tVBG\nunwind\tVB\nupdated\tVBN\nupdates\tNNS\nupdate\tVB\nupdating\tVBG\nupgraded\tVBD\nupgraded\tVBN\nupgrades\tNNS\nUpgrades\tNNS\nupgrade\tVB\nupgrading\tVBG\nupheavals\tNNS\nupheld\tVBD\nupheld\tVBN\nUPHELD\tVBN\nupholding\tVBG\nupholds\tVBZ\nuphold\tVB\nupped\tVBD\nUpping\tVBG\nuprooted\tVBD\nups-and-downs\tNNS\nupsets\tNNS\nupsetting\tVBG\nupset\tVB\nupset\tVBD\nupset\tVBN\nUpset\tVBN\nups\tNNS\nuptick\tVB\nUrals\tNNS\nurged\tVBD\nurged\tVBN\nURGED\tVBN\nurges\tVBZ\nurge\tVB\nurge\tVBP\nurgings\tNNS\nurging\tVBG\nused\tVBD\nused\tVBN\nUsed\tVBN\nusers\tNNS\nUsers\tNNS\nuses\tNNS\nuses\tVBZ\nuse\tVB\nUse\tVB\nuse\tVBP\nushered\tVBD\nushered\tVBN\nushering\tVBG\nushers\tNNS\nushers\tVBZ\nusing\tVBG\nUsing\tVBG\nusurp\tVB\nU.S.\tVBP\nutilities\tNNS\nUtilities\tNNS\nutilize\tVB\nutmosts\tNNS\nutopians\tNNS\nutterances\tNNS\nuttered\tVBD\nuttering\tVBG\nUVB\tNN\nvacancies\tNNS\nVacancies\tNNS\nvacated\tVBD\nvacated\tVBN\nvacate\tVB\nvacating\tVBG\nvacationers\tNNS\nvacationing\tVBG\nvacations\tNNS\nvaccines\tNNS\nvacillate\tVB\nvacuum\tVB\nvagabonds\tNNS\nvagaries\tNNS\nvalidating\tVBG\nvaluations\tNNS\nvalued\tVBD\nvalued\tVBN\nValued\tVBN\nvalues\tNNS\nvalues\tVBZ\nvalue\tVB\nvalue\tVBP\nvaluing\tVBG\nvalves\tNNS\nvandalized\tVBD\nvanished\tVBD\nvanished\tVBN\nvanishes\tVBZ\nvanishing\tVBG\nvanish\tVB\nvanish\tVBP\nVanities\tNNS\nvans\tNNS\nvapors\tNNS\nvariables\tNNS\nvariations\tNNS\nvaried\tVBD\nvaried\tVBN\nvaries\tVBZ\nvarieties\tNNS\nvarying\tVBG\nvary\tVB\nvary\tVBP\nVary\tVBP\nvases\tNNS\nvassals\tNNS\nvaults\tNNS\nvault\tVB\nVCRs\tNNS\nveering\tVBG\nveer\tVB\nvegetables\tNNS\nVegetables\tNNS\nvegetarians\tNNS\nvehicles\tNNS\nveiled\tVBN\nVeiling\tVBG\nvending\tVBG\nvendors\tNNS\nvented\tVBD\nventilated\tVBD\nventilated\tVBN\nventures\tNNS\nventure\tVB\nventuring\tVBG\nvent\tVB\nvenues\tNNS\nverdicts\tNNS\nverged\tVBD\nverified\tVBN\nverify\tVB\nversions\tNNS\nvessels\tNNS\nvested\tVBN\nvestments\tNNS\nvests\tNNS\nveterans\tNNS\nveterinarians\tNNS\nvetoed\tVBD\nvetoed\tVBN\nvetoes\tNNS\nvetoing\tVBG\nveto\tVB\n've\tVB\n've\tVBP\nvexed\tVBN\nviaducts\tNNS\nvibrating\tVBG\nvicars\tNNS\nvices\tNNS\nvicissitudes\tNNS\nvicitims\tNNS\nvictimized\tVBN\nvictims\tNNS\nVICTIMS\tNNS\nvictories\tNNS\nVICTORIES\tNNS\nvideocassettes\tNNS\nvideodisks\tNNS\nvideos\tNNS\nvideotapes\tNNS\nvideotape\tVB\nVietnamese\tNNS\nviewed\tVBD\nviewed\tVBN\nviewers\tNNS\nViewers\tNNS\nviewings\tNNS\nviewing\tVBG\nviewpoints\tNNS\nviews\tNNS\nViews\tNNS\nviews\tVBZ\nview\tVB\nview\tVBP\nvignettes\tNNS\nvillagers\tNNS\nvillages\tNNS\nVillages\tNNS\nvillains\tNNS\nvindicated\tVBN\nvineyards\tNNS\nvintages\tNNS\nvinyl-products\tNNS\nviolated\tVBD\nviolated\tVBN\nviolates\tVBZ\nviolate\tVB\nviolate\tVBP\nviolating\tVBG\nviolations\tNNS\nvirgins\tNNS\nvirtues\tNNS\nvirtuosos\tNNS\nvisages\tNNS\nvisionaries\tNNS\nvisions\tNNS\nvisited\tVBD\nvisited\tVBN\nvisiting\tVBG\nVisiting\tVBG\nvisitors\tNNS\nVisitors\tNNS\nvisits\tNNS\nvisits\tVBZ\nvisit\tVB\nvisit\tVBP\nvisualize\tVB\nVISUALIZING\tVBG\nvisuals\tNNS\nvitiate\tVB\nvoiced\tVBD\nvoiced\tVBN\nvoices\tNNS\nVoices\tNNS\nvoices\tVBZ\nvoice\tVB\nvoice\tVBP\nvoicing\tVBG\nvoided\tVBN\nvoid\tVB\nvolumes\tNNS\nvolunteered\tVBD\nvolunteered\tVBN\nvolunteers\tNNS\nvolunteer\tVB\nvomiting\tVBG\nvoted\tVBD\nvoted\tVBN\nvote-getters\tNNS\nvoters\tNNS\nVoters\tNNS\nvotes\tNNS\nvotes\tVBZ\nvote\tVB\nvote\tVBP\nvoting\tVBG\nvouchers\tNNS\nvowed\tVBD\nvowed\tVBN\nvowels\tNNS\nVowels\tNNS\nvowing\tVBG\nVowing\tVBG\nvows\tVBZ\nvying\tVBG\nwaddles\tVBZ\nwade\tVB\nwad\tVB\nwafers\tNNS\nwaffled\tVBD\nwaffled\tVBN\nwafting\tVBG\nwaged\tVBD\nwages\tNNS\nwage\tVB\nwaging\tVBG\nwagons\tNNS\nwags\tNNS\nwailing\tVBG\nwail\tVB\nwaited\tVBD\nwaited\tVBN\nwaiters\tNNS\nwaiting\tVBG\nwaits\tVBZ\nwait\tVB\nWait\tVB\nwait\tVBP\nwaived\tVBD\nwaived\tVBN\nwaivered\tVBN\nwaivers\tNNS\nwaives\tVBZ\nwaive\tVB\nwaiving\tVBG\nwake\tVBP\nwaking\tVBG\nwalked\tVBD\nwalkie-talkies\tNNS\nwalking\tVBG\nWalking\tVBG\nwalking\tVBG|NN\nwalkouts\tNNS\nwalks\tNNS\nwalks\tVBZ\nwalk\tVB\nWalk\tVB\nwalk\tVBP\nWalk\tVBP\nwallcoverings\tNNS\nwallets\tNNS\nwallops\tVBZ\nwallowing\tVBG\nwalls\tNNS\nwall\tVB\nwandering\tVBG\nwanders\tVBZ\nwander\tVB\nwander\tVBP\nwaned\tVBD\nwaned\tVBN\nwanes\tVBZ\nWANES\tVBZ\nwane\tVB\nwane\tVBP\nwaning\tVBG\nwanted\tVBD\nwanted\tVBN\nWanted\tVBN\nwanting\tVBG\nwants\tVBZ\nwant\tVB\nwant\tVBP\nwarded\tVBN\nwardens\tNNS\nward\tVB\nwarehouses\tNNS\nwares\tNNS\nwarheads\tNNS\nwarmed\tVBD\nwarmed\tVBN\nwarming\tVBG\nwarming\tVBG|NN\nwarm\tVB\nwarned\tVBD\nWarned\tVBD\nWARNED\tVBD\nwarned\tVBN\nWarners\tNNS\nwarnings\tNNS\nwarning\tVBG\nwarns\tVBZ\nwarn\tVB\nwarn\tVBP\nwarranted\tVBN\nwarranties\tNNS\nwarrants\tNNS\nwarrants\tVBZ\nwarrant\tVB\nwarrant\tVBP\nWarrens\tNNS\nwarring\tVBG\nwarriors\tNNS\nwars\tNNS\nWars\tNNS\nWARS\tNNS\nwarts\tNNS\nwar\tVB\nwashed\tVBD\nwashed\tVBN\nwashing\tVBG\nwash\tVB\nwash\tVBP\nwasted\tVBD\nwasted\tVBN\nwastes\tNNS\nwastes\tVBZ\nwaste\tVB\nwasting\tVBG\nwas\tVBD\nWas\tVBD\nWAS\tVBD\nwatchdogs\tNNS\nwatched\tVBD\nwatched\tVBN\nwatchers\tNNS\nwatches\tNNS\nwatches\tVBZ\nwatching\tVBG\nWatching\tVBG\nwatch\tVB\nWATCH\tVB\nwatch\tVBP\nwatered\tVBD\nwaterfalls\tNNS\nwatering\tVBG\nwaters\tNNS\nwater\tVB\nwaterworks\tNN|NNS\nwatts\tNNS\nwaved\tVBD\nwavelengths\tNNS\nwavered\tVBD\nwavering\tVBG\nwaves\tNNS\nwaving\tVBG\nwaxed\tVBD\nwaxed\tVBN\nways\tNNS\nWays\tNNS\nweakened\tVBD\nweakened\tVBN\nweakening\tVBG\nWeakening\tVBG\nweakens\tVBZ\nWeakens\tVBZ\nweaken\tVB\nweaken\tVBP\nweaknesses\tNNS\nwean\tVB\nweapons\tNNS\nwearing\tVBG\nWearing\tVBG\nwears\tVBZ\nwear\tVB\nWear\tVB\nwear\tVBP\nweasling\tVBG\nweather\tVB\nweather\tVBP\nweaves\tVBZ\nweaving\tVBG\nwebs\tNNS\nwedded\tVBN\nweddings\tNNS\nwedged\tVBD\nwedged\tVBN\nwedge\tVB\nWednesdays\tNNS\nweds\tVBZ\nwed\tVBN\nWeeds\tNNS\nweekdays\tNNS\nweekends\tNNS\nweeklies\tNNS\nweeknights\tNNS\nweeks\tNNS\nweepers\tNNS\nweeping\tVBG\nweighed\tVBD\nweighed\tVBN\nweighing\tVBG\nWeighing\tVBG\nweighs\tVBZ\nweighted\tVBN\nweightings\tNNS\nweights\tNNS\nweight\tVB\nweigh\tVB\nweigh\tVBP\nwelcomed\tVBD\nwelcomed\tVBN\nwelcomes\tVBZ\nwelcome\tVB\nWelcome\tVB\nwelcome\tVBP\nwelcoming\tVBG\nwelded\tVBN\nwells\tNNS\nwell-stated\tVBN\nwell-wishers\tNNS\nwent\tVBD\nWENT\tVBD\nwept\tVBD\nwere\tVB\nWere\tVB\nwere\tVBD\nWere\tVBD\nwe\tVBP\nwhacked\tVBD\nwhacked\tVBN\nwhack\tVB\nwhales\tNNS\nwheelbases\tNNS\nwheeled\tVBN\nwheels\tNNS\nwheezing\tVBG\nwhen-issued\tVBN\nwhile\tVB\nwhimpers\tNNS\nwhims\tNNS\nwhipped\tVBN\nwhippings\tNNS\nwhipping\tVBG\nwhipsawed\tVBN\nwhipsaw\tVB\nwhips\tVBZ\nwhip\tVB\nwhirlwinds\tNNS\nwhirring\tVBG\nwhisked\tVBN\nwhispered\tVBN\nwhispering\tVBG\nwhispers\tNNS\nwhistled\tVBN\nwhistles\tNNS\nwhistle\tVBP\nwhistling\tVBG\nwhites\tNNS\nwhitewashed\tVBN\nwhittled\tVBN\nwhizzes\tNNS\nwholesalers\tNNS\nWholesalers\tNNS\nwholesaling\tVBG\nwhooping\tVBG\nwhoosh\tVBP\nwidened\tVBD\nWidened\tVBD\nwidened\tVBN\nwidening\tVBG\nwidens\tVBZ\nwiden\tVB\nwiden\tVBP\nwidgets\tNNS\nwidowed\tVBN\nwidows\tNNS\nwielded\tVBN\nwielding\tVBG\nwields\tVBZ\nwield\tVB\nwiggled\tVBD\nwiggle\tVBP\nwiggling\tVBG\nwigs\tNNS\nWilfred\tVBD\nwillies\tNNS\nwilling\tVBG\nwills\tNNS\nwill\tVB\nwimping\tVBG\nwimp\tVB\nwindfalls\tNNS\nwinding\tVBG\nwindows\tNNS\nwindshields\tNNS\nwinds\tNNS\nwind\tVB\nwind\tVBP\nwineries\tNNS\nwines\tNNS\nwings\tNNS\nwinking\tVBG\nwinners\tNNS\nWinners\tNNS\nwinning\tVBG\nWinning\tVBG\nwins\tNNS\nwins\tVBZ\nwinters\tNNS\nwin\tVB\nwin\tVBP\nwiped\tVBD\nwiped\tVBN\nwipe\tVB\nwipe\tVBP\nwiping\tVBG\nwired\tVBN\nwires\tNNS\nwiretaps\tNNS\nwire\tVB\nwisecracks\tNNS\nwished\tVBD\nwished\tVBN\nwishes\tNNS\nWishes\tNNS\nwishes\tVBZ\nwishing\tVBG\nwish-lists\tNNS\nwish\tVB\nwish\tVBP\nwitches\tNNS\nwithdrawals\tNNS\nwithdrawing\tVBG\nwithdrawn\tVBN\nwithdraw\tVB\nwithdrew\tVBD\nwithering\tVBG\nwither\tVBP\nwithheld\tVBD\nwithheld\tVBN\nwithholdings\tNNS\nwithholding\tVBG\nwithhold\tVB\nWithhold\tVB\nwithhold\tVBP\nwithstanding\tVBG\nwithstand\tVB\nwithstood\tVBD\nwithstood\tVBN\nwitnessed\tVBN\nwitnesses\tNNS\nWitnesses\tNNS\nwitnessing\tVBG\nWitness\tVB\nwives\tNNS\nWives\tNNS\nwizards\tNNS\nwoes\tNNS\nwoken\tVBN\nwoke\tVBD\nwolves\tNNS\nwomanizing\tVBG\nwomen\tNNS\nWomen\tNNS\nwonderbars\tNNS\nwondered\tVBD\nwondering\tVBG\nwonders\tNNS\nwonders\tVBZ\nwonder\tVB\nwonder\tVBP\nwon\tNNS\nwon\tVBD\nwon\tVBN\nWON\tVBP\nwoodchucks\tNNS\nwood-products\tNNS\nwoods\tNNS\nwooed\tVBN\nwooing\tVBG\nwoo\tVB\nworded\tVBN\nword-processing\tNNS\nwords\tNNS\nwore\tVBD\nworkbooks\tNNS\nworkdays\tNNS\nworked\tVBD\nworked\tVBN\nworkers\tNNS\nWorkers\tNNS\nWORKERS\tNNS\nworkings\tNNS\nworking\tVBG\nWorking\tVBG\nworking\tVBG|NN\nworkmen\tNNS\nworksheets\tNNS\nWorksheets\tNNS\nworks\tNNS\nworkstations\tNNS\nworks\tVBZ\nwork\tVB\nWork\tVB\nwork\tVBP\nWORK\tVBP\nworld-affairs\tNNS\nworlds\tNNS\nworms\tNNS\nworn\tVBN\nWorn\tVBN\nworried\tVBD\nworried\tVBN\nWorried\tVBN\nworriers\tNNS\nworries\tNNS\nWorries\tNNS\nworries\tVBZ\nworrying\tVBG\nworry\tVB\nworry\tVBP\nworsened\tVBD\nworsened\tVBN\nworsening\tVBG\nworsen\tVB\nwounded\tVBN\nwounds\tNNS\nwound\tVB\nwound\tVBD\nwound\tVBN\nwoven\tVBN\nwows\tVBZ\nwracked\tVBD\nwracked\tVBN\nwrack\tVBP\nwrangling\tVBG\nwrapped\tVBD\nwrapped\tVBN\nwrappers\tNNS\nwrapping\tVBG\nwraps\tVBZ\nwrap\tVB\nwreaked\tVBD\nwreaking\tVBG\nwreak\tVB\nwrecked\tVBD\nwrecking\tVBG\nwreck\tVB\nwrenched\tVBN\nwrenching\tVBG\nwrested\tVBN\nwrestlers\tNNS\nwrestles\tVBZ\nwrestle\tVB\nwrestling\tVBG\nwrest\tVB\nwriggling\tVBG\nwring\tVB\nwrists\tNNS\nwrite-downs\tNNS\nwritedowns\tNNS\nwrite-offs\tNNS\nwriteoffs\tNNS\nwriter\\/producers\tNNS\nwriters\tNNS\nwrites\tVBZ\nWrites\tVBZ\nwrite\tVB\nwrite\tVBP\nwrithing\tVBG\nwritings\tNNS\nwriting\tVBG\nWriting\tVBG\nwritten\tVBN\nWritten\tVBN\nwronged\tVBN\nwrote\tVBD\nWrote\tVBD\nwrought\tVBD\nwrung\tVB\nX-rays\tNNS\nyachts\tNNS\nyanked\tVBD\nyanked\tVBN\nyanking\tVBG\nyank\tVB\nyards\tNNS\nyearbooks\tNNS\nyearlings\tNNS\nyears\tNNS\nYears\tNNS\nYEARS\tNNS\nyelled\tVBD\nyelling\tVBG\nyellows\tNNS\nyells\tVBZ\nyell\tVB\nyelped\tVBD\nyen\tNNS\nyielded\tVBD\nyielded\tVBN\nyielding\tVBG\nyields\tNNS\nYields\tNNS\nyields\tVBZ\nyield\tVB\nyield\tVBP\nyoungsters\tNNS\nYounkers\tNNS\nyouths\tNNS\nyuppies\tNNS\nzappers\tNNS\nzapping\tVBG\nzeroing\tVBG\nzeros\tNNS\nzero\tVB\nzero\tVBP\nzigzags\tNNS\nzig-zag\tVBP\nzip\tVB\nzip\tVBP\nzlotys\tNNS\nzombies\tNNS\nzones\tNNS\nZones\tNNS\nzoning\tVBG\nzoomed\tVBD\nzoomed\tVBN\nzoom\tVB\n" }, { "path": "ccgbank/data/wsj-nns-vb-stems", "content": "12-inch\n15\n16-year-old\n1850\n1890\n18-year-old\n1900\n1920\n1930\n1940\n1950\n1960\n1970\n1980\n1990\n19-year-old\n204\n20\n20-stock\n3090\n'30\n30\n323\n'40\n'50\n'60\n'70\n747-400\n757-200\n75-cents-an-hour\n'80\n80\n8300\n'90\na330-300\nabacus\nabandon\nabandon\nabandon\nabandon\nabandon\nabandon\nabandon\nabate\nabate\nabate\nabate\nabate\nabdicate\nabet\nabet\nabide\nability\nabolish\nabolish\nabolish\nabolish\nabort\nabortion-rights\nabortion-rights\nabortion\nabound\nabound\nabound\nabound\nabrasive\nabridge\nabsence\nabsent\nabsent\nabsolve\nabsolve\nabsorb\nabsorb\nabsorb\nabsorber\nabsorb\nabsorb\nabsorb\nabsorb\nabstain\nabstention\nabstract\nabuse\nabuse\nabuse\nabuse\nabuse\nacademic\naccede\naccede\naccelerate\naccelerate\naccelerate\naccelerate\naccelerate\naccelerate\naccent\nacceptance\nacceptance\naccept\naccept\naccept\naccept\naccept\naccept\naccessory\naccessory\naccess\naccident\nacclaim\naccolade\naccommodate\naccommodate\naccommodate\naccommodate\naccommodation\naccomodate\naccompany\naccompany\naccompany\naccompany\naccompany\naccomplish\naccomplish\naccomplish\naccomplishment\naccomplish\naccord\naccord\naccord\naccord\naccord\naccord\naccountant\naccount\naccount\naccount\naccount\naccount\naccount\naccount\naccount\naccrete\naccrual\naccrue\naccrue\naccrue\naccrue\naccrue\naccumulate\naccumulate\naccumulate\naccusation\naccuse\naccuse\naccuser\naccuse\naccuse\naccuse\naccuse\naccustom\nace\nachieve\nachieve\nachievement\nachieve\nachieve\nachieve\nachieve\nache\nacidify\nacid\nacknowledge\nacknowledge\nacknowledge\nacknowledge\nacknowledge\nacknowledge\nacorn\nacquaint\nacquiesce\nacquiesce\nacquire\nacquire\nacquire\nacquirer\nacquirer\nacquire\nacquire\nacquire\nacquisition\nacquisition\nacquit\nacquit\nacre\nacross-the-board-cut\nact\nact\nact\nact\naction\nactivate\nactivate\nactive\nactivist\nactivist\nactivity\nactor\nact\nact\nactuary\nact\nact\nadapt\nadapt\nadapt\nadapt\nadapt\nadapt\nadd\nadd\nadd\nadd\naddict\naddict\nadd\nadd\naddition\nadditive\naddress\naddress\naddress\naddress\naddress\naddress\naddress\naddress\nadd\nadd\nadd\nadd\nadd\nadept\nadhere\nadhere\nadhesive\nadjourn\nadjudicator\nadjust\nadjust\nadjuster\nadjuster\nadjust\nadjustment\nadjust\nadjust\nadjust\nadjust\nadman\nadminister\nadminister\nadminister\nadministration\nadministrator\nadmire\nadmire\nadmirer\nadmire\nadmission\nadmit\nadmit\nadmit\nadmit\nadmit\nadmit\nadmit\nadmit\nadmonish\nadolescent\nadopt\nadopt\nadopt\nadopt\nadopt\nadopt\nadopt\nadopt\nadorn\nadorn\nadorn\nadr\nad\nad\nad\nadult\nadult\nadvanced-ceramic\nadvance\nadvance\nadvancement\nadvancer\nadvancer\nadvance\nadvance\nadvance\nadvance\nadvance\nadvance\nadvantage\nadvantage\nadventure\nadversary\nadvertise\nadvertise\nadvertisement\nadvertiser\nadvertiser\nadvertise\nadvertise\nadvertise\nadvertise\nadvertise\nadvertise\nadvise\nadvise\nadviser\nadvise\nadvise\nadvise\nadvise\nadvise\nadvisory\nadvocate\nadvocate\nadvocate\nadvocate\nadvocate\nadvocate\nadvocate\nadvocate\naerobic\naffair\naffected\naffect\naffect\naffect\naffection\naffect\naffect\naffect\naffidavit\naffiliate\naffiliate\naffiliate\naffiliate\naffinity\naffirm\naffirm\nafflict\nafflict\nafflict\nafford\nafford\nafford\naftereffect\naftereffect\nafter-hour\nafternoon\naftershock\naftershock\naftershock\nage\nagency\nagenda\nagent\nagent\nage\nage\naggravate\naggravate\naggravate\naggravate\naggravate\naggravate\naggregate\nage\nagitate\nagonize\nagree\nagree\nagree\nagree\nagreement\nagree\nagree\nagree\nagree\nagriproduct\naichus\naid\naid\naid\naide\naide\naid\naid\naid\naid\nail\nailment\nail\naim\naim\naim\naim\naim\naim\naim\naim\naim\naim\naircraft\nair\nair\nairfield\nair\nairlift\nairlift\nairliner\nairline\nairline\nairman\nairplane\nairport\nair\nair\nair\nair\nairwave\naisle\nbe\nbe\nalarm\nalarm\nalarm\nalarm\nalarm\nalbanian\nalbum\nalchemist\nalcoholic\nalert\nalert\nalert\nalert\nalienate\nalienate\nalienate\nalienate\nalien\nalign\nalign\nallay\nallay\nallegation\nallege\nallege\nallege\nallege\nallege\nallege\nallege\nallergy\nalleviate\nalleviate\nalley\nalliance\nally\nally\nally\nalligator\nallocate\nallocate\nallocate\nallocate\nallocate\nallocation\nallotment\nallowance\nallow\nallow\nallow\nallow\nallow\nallow\nallow\nallow\nallow\nalloy\nallrightnik\nallude\nallusion\nally\nally\nalter\nalter\nalter\nalternate\nalternate\nalternate\nalternative\nalter\nalter\nalter\naluminum-maker\nalumnus\namah\namalgamate\namalgamation\namass\namass\namass\namass\namass\namateur\namaze\namaze\nambassador\nambiguity\nambition\namble\nambush\namend\namend\namend\namend\namendment\namend\namenity\namenity\na-man\namericana\namericanize\namerican\namerican\namortize\namount\namount\namount\namount\namount\namount\namount\namphobile\namplify\namplifier\namplify\namplify\namp\namuse\namusement\namuse\nbe\nbe\nbe\nanachronism\nanalysis\nanalysis\nanalyst\nanalyst\nanalyze\nanalyze\nanalyze\nanalyze\nanalyze\nanchor\nanchor\nanchor\nanchor\nand\nanemia\nanemic\nangel\nanger\nanger\nanger\nangle\nanimal\nanimal\nanimate\nanimosity\nankle\nanne\nanniversary\nannounce\nannounce\nannounce\nannouncement\nannounce\nannounce\nannounce\nannounce\nannoy\nannoy\nannualize\nannualize\nannuity\nannuity\nannuity\nanoint\nanomaly\nanswer\nanswer\nanswer\nanswer\nanswer\nanswer\nanswer\nanswer\nantagonist\nantagonize\nanteater\nantecedent\nantelope\nanther\nanti-abortionist\nantibody\nanticipate\nanticipate\nanticipate\nanticipate\nanticipate\nanticipate\nanticipate\nantic\nanti-hero\nanti-infective\nantipathy\nanti-programmer\nantiquity\nant\nanxiety\napartment\napology\napologist\napologize\napologize\napologize\napologize\nappal\napparition\nappeal\nappeal\nappeal\nappeal\nappeal\nappeal\nappeal\nappearance\nappear\nappear\nappear\nappear\nappear\nappear\nappear\nappear\nappease\nappease\nappendage\nappend\nappend\napplaud\napplaud\napplaud\napplaud\napplaud\napple\nappliance\napplicant\napplication\napplication\napply\napply\napply\napply\napply\napply\napply\napply\nappoint\nappoint\nappoint\nappointee\nappointment\nappoint\nappraisal\nappraise\nappraiser\nappraise\nappreciate\nappreciate\nappreciate\nappreciate\nappreciate\nappreciate\napprehension\napprise\napproach\napproach\napproach\napproach\napproach\napproach\napproach\nappropriate\nappropriate\nappropriation\nappropriation\nappropriator\napproval\napprove\napprove\napprove\napprove\napprove\napprove\napprove\napproximate\narab\narbitrager\narbitrager\narbitrageur\narbitrage\narbitrage\narbitrate\narbitrate\narborist\narb\narb\narcade\narch\narchitect\narchitect\narch\narea\narea\narena\nbe\nbe\nbe\nargue\nargue\nargue\nargue\nargue\nargue\nargue\nargument\narise\narise\narise\narise\narise\narise\narkansa\narmadillo\narm\narm\narmy\narmpit\narm\narm\narm\naroma\narise\narouse\narouse\narouse\narouse\narouse\narpeggio\narraignment\narrange\narrange\narrangement\narrangement\narrange\narrange\narrange\narrears\narrest\narrest\narrest\narrest\narrest\narrival\narrive\narrive\narrive\narrive\narrive\narrive\narrive\narrow\narsenal\nartery\narticle\narticulate\nartifact\nartillerist\nartist\nartist\nart\nart\nartwork\nasahus\nascend\nascertain\nascribe\nascribe\nash\nashland\nashtray\nask\nask\nask\nask\nask\nask\nask\nask\nask\naspect\naspen\naspersion\naspiration\naspire\naspire\naspire\naspire\nassail\nassail\nassassinate\nassassinate\nassassinate\nassassinate\nassassination\nassault\nassault\nassemblage\nassemble\nassemble\nassemble\nassemble\nassembly\nassemble\nassert\nassert\nassert\nassertion\nassert\nassert\nassert\nass\nassess\nassess\nassess\nassessment\nassess\nassess\nasset\nasset\nasset\nassign\nassignment\nassign\nassign\nassign\nassimilate\nassistant\nassist\nassist\nassist\nassist\nassist\nassociate\nassociate\nassociate\nassociate\nassociate\nassociation\nassuage\nassume\nassume\nassume\nassume\nassume\nassume\nassume\nassume\nassumption\nassurance\nassure\nassure\nassure\nassure\nassure\nasteroid\nastound\nastronaut\neat\neat\nathlete\nathletics\natlanti\natoll\natom\natone\natrocity\nattach\nattach\nattach\nattach\nattach\nattach\nattack\nattack\nattacker\nattack\nattack\nattack\nattack\nattack\nattack\nattain\nattain\nattarck\nattempt\nattempt\nattempt\nattempt\nattempt\nattempt\nattempt\nattempt\nattendant\nattendant\nattend\nattend\nattendee\nattend\nattend\nattend\nattend\nattention\nattest\nattest\nattic\nattitude\nattitude\nattorney\nattorney\nattract\nattract\nattract\nattract\nattraction\nattract\nattract\nattract\nattribute\nattribute\nattribute\nattribute\nattribute\nattribute\nattribute\nauction\nauction\nauction\nauction\nauction\naudience\naudiocassette\naudiophile\naudit\naudit\naudit\naudition\nauditor\nauditor\naudit\naudit\naudit\naugment\nauspices\nauthor\nauthority\nauthority\nauthorization\nauthorize\nauthorize\nauthorize\nauthorize\nauthorize\nauthor\naution\nauto-emission\nautograph\nautograph\nautograph\nauto\\/homeowners\nautomaker\nautomate\nautomate\nautomate\nautomate\nautomobile\nautomobile\nauto-sale\nauto\nautumn\navenge\navenue\naverage\naverage\naverage\naverage\naverage\naverage\naverage\naver\navert\navert\navert\navert\naviator\navoid\navoid\navoid\navoid\navoid\navoid\navoid\navoid\nawait\nawait\nawait\nawait\nawait\nawaken\naward\naward\naward\naward\naward\naward\nawake\naxiom\naxle\nb-2\nbabelist\nbaby\nbaby\nbackdate\nback-date\nback\nback\nback\nbacker\nbackfire\nbackfire\nbackfire\nbackfire\nbackfire\nbackflip\nbackground\nback\nbacklog\nbacklot\nbackpacker\nbackpack\nbackpedal\nbackslap\nback\nback\nback\nback-up\nback\nback\nbacterium\nbadge\nbag\nbail\nbailiff\nbail\nbailout\nbail\nbakery\nbaker\nbalance\nbalance-of-payment\nbalance\nbalance\nbalance\nbalance\nbalance\nbalcony\nbale\nbalk\nbalk\nbalk\nbalk\nbalk\nballerina\nballet\nballoon\nballoon\nballoon\nballoonist\nballoon\nballoon\nballoon\nballot\nballot\nballpark\nballplayer\nball\nballyhoo\nbamboozle\nbanana\nbandage\nband\nbandy\nband\nband\nbang\nbang\nbanish\nbanish\nbanker\nbanker\nbanker\nbank\nbankroll\nbankroll\nbankroll\nbankruptcy\nbankrupt\nbankrupt\nbank\nbank\nbank\nbank\nbank\nban\nban\nbanner\nban\nbanshee\nban\nban\nban\nban\nbarb\nbarber\nbarb\nbargain\nbargain\nbargain-hunter\nbargain-hunt\nbargain\nbargain\nbargain\nbarge\nbarge\nbare\nbark\nbarnacle\nbarn\nbaron\nbar\nbar\nbar\nbarrel\nbarrel\nbarrel\nbarricade\nbarrier\nbar\nbar\nbar\nbar\nbartender\nbarter\nbar\nbar\nbaseball\nbase\nbase\nbase\nbasement\nbasis\nbase\nbase\nbase\nbash\nbash\nbash\nbasics\nbase\nbasket\nbasket\nbastion\nbatch\nbath\nbathroom\nbath\nbat\nbattalion\nbat\nbat\nbatter\nbatter\nbattery\nbatter\nbatter\nbat\nbattle\nbattle\nbattle\nbattlegroup\nbattlements\nbattle\nbattle\nbattle\nbattle\nbattle\nbay\nbeach\nbead\nbeam\nbeam\nbeam\nbeanball\nbean\nbearing\nbear\nbear\nbear\nbear\nbear\nbear\nbeastie\nbeast\nbeat\nbeat\nbeat\nbeat\nbeat\nbeat\nbeat\nbecome\nbecome\nbecome\nbecome\nbecome\nbecome\nbecome\nbecome\nbecome\nbedevil\nbedevil\nbedfellow\nbedfellow\nbedpan\nbedroom\nbed\nbeef\nbeef\nbeef\nbeef\nbe\nbe\nbe\nbeep\nbeep\nbeer\nbee\nbefall\nbefall\nbefall\nbefriend\nbefuddle\nbegin\nbeggar\nbeg\nbeg\nbeginning\nbegin\nbegin\nbegin\nbegin\nbegin\nbegin\nbeget\nbeg\nbegin\nbeg\nbehave\nbehave\nbehave\nbehave\nbehave\nbehavior\nbehead\nbehemoth\nbeing\nbe\nbe\nbe\nbe\nbeleaguer\nbelfry\nbelie\nbelief\nbelie\nbelie\nbelieve\nbelieve\nbelieve\nbelieve\nbelieve\nbelieve\nbelittle\nbelly\nbellow\nbellringer\nbell\nbellwether\nbelly-flop\nbelong\nbelongings\nbelong\nbelong\nbelong\nbelong\nbelt\nbelt\nbemoan\nbench\nbenchmark\nbend\nbend\nbenefactor\nbeneficiary\nbeneficiary\nbenefit\nbenefit\nbenefit\nbenefit\nbenefit\nbenefit\nbenefit\nbenefit\nbenefit\nbequeath\nbequest\nberate\nberry\nbeset\nbesiege\nbest\nbest\nbestir\nbestow\nbeta\nbetray\nbet\nbet\nbetters\nbetter\nbet\nbet\nbet\nbet\nbet\nbe\nbe\nbe\nbe\nbeverage\nbeware\nbeware\nbeware\nbewilder\nbewilder\nbewitch\nbias\nbias\nbible\nbicker\nbicker\nbicycle\nbidder\nbid\nbid\nbid\nbid\nbid\nbid\nbid\nbid\nbifurcate\nbiker\nbiker\nbike\nbike\nbilge\nbilk\nbill\nbill\nbilling\nbilling\nbill\nbillion\nbillion\nbillow\nbill\nbill\nbill\nbill\nbill\nbill\nbinder\nbind\nbinge\nbinoculars\nbin\nbioengineer\nbiographer\nbiologist\nbiomedical-product\nbiopsy\nbird\nbird\nbirthday\nbirth\nbiscuit\nbishop\nbite\nbite\nbite\nbite\nbit\nbite\nbite\nblab\nblack\nblacken\nblacklist\nblackmail\nblackmailer\nblackmail\nblackmail\nblackout\nblack\nblack\nblackstone\nblade\nblame\nblame\nblame\nblame\nblame\nblame\nblame\nblame\nblanding\nblanket\nblanket\nblanket\nblare\nblast\nblast\nblast\nblast\nblaze\nblaze\nbleacher\nbleed\nbleed\nbleed\nblemish\nblend\nblend\nblend\nbless\nblessing\nbless\nblow\nblindfold\nblind\nblini\nblink\nblinkers\nblink\nblink\nblip\nblip\nblip\nblitz\nblock\nblock\nblock\nblock\nblock\nblock\nblock\nblock\nblonde\nbloody\nbloodlet\nblood\nbloom\nblossom\nblossom\nblossom\nblot\nblow\nblow\nblow\nblow\nbludgeon\nbludgeon\nblueblood\nblue-chip\nblues\nblues\nblunder\nblunder\nblunder\nblunt\nblunt\nblunt\nblur\nblur\nblurt\nblurt\nblur\nbmw\nboard\nboardroom\nboardroom\nboard\nboard\nboast\nboast\nboast\nboast\nboast\nboater\nboat\nboat\nbodega\nbode\nbode\nbody\nbodyworker\nboe\nbog\nbog\nbog\nboil\nboiler\nboil\nboil\nboil\nbolster\nbolster\nbolster\nbolster\nbolster\nbolster\nbolster\nbolt\nbolt\nbombard\nbombard\nbombard\nbomb\nbomb\nbomber\nbombing\nbomblet\nbomb\nbomb\nbond\nbond-holder\nbondholder\nbondholding\nbond\nbond\nbond\nbone\nbone\nbonnet\nbonus\nbonus\nboo\nbook\nbook\nbooker\nbooking\nbooking\nbook\nbooklet\nbook\nbook\nbookstore\nbook\nboom\nboom\nboomer\nboom\nboom\nboom\nboost\nboost\nbooster\nboost\nboost\nboost\nboost\nboost\nboost\nboost\nbooth\nbooth\nbootleg\nboot\nbooze\nborder\nborder\nborder\nbore\nbear\nbore\nbear\nbear\nbear\nborrow\nborrow\nborrow\nborrower\nborrower\nborrowing\nborrow\nborrow\nborrow\nborrow\nborrow\nboss\nboss\nbotch\nbother\nbother\nbother\nbother\nbother\nbottle\nbottleneck\nbottler\nbottle\nbottle\nbottle\nbottom\nbottom\nbottom\nbottom\nbottom\nbuy\nbuy\nbounce\nbounce\nbounce\nbounce\nbounce\nbounce\nboundary\nbound\nbound\nbound\nbind\nbourbon\nbourse\nboutique\nbout\nbouygue\nbow\nbow\nbow\nbowl\nbowl\nbowl\nbow\nbow\nbox\nboycott\nboycott\nboyfriend\nboy\nboy\nbrace\nbrace\nbrace\nbrace\nbrag\nbrag\nbraid\nbrain\nbrake\nbrake\nbrake\nbranch\nbranch\nbranch\nbranch\nbranch\nbrandish\nbrandish\nbrand\nbrand\nbrand\nbrassiere\nbrat\nbrave\nbrave\nbrave\nbrawl\nbrazen\nbrazilian\nbreach\nbreach\nbreach\nbreach\nbread\nbreakdown\nbreaker\nbreak\nbreak\nbreak\nbreak\nbreakthrough\nbreak\nbreak\nbreast\nbreathe\nbreathe\nbreathe\nbreath\nbreed\nbreed\nbreeder\nbreeder\nbreed\nbreed\nbreed\nbreeze\nbrethren\nbrew\nbrewery\nbrewery\nbrewer\nbrew\nbrew\nbribe\nbribe\nbribe\nbribe\nbribe\nbrick\nbridge\nbridge\nbridge\nbriefcase\nbrief\nbrief\nbriefing\nbriefs\nbriefs\nbrief\nbrigade\nbrighten\nbrighten\nbrighten\nbrim\nbring\nbring\nbring\nbring\nbristle\nbristle\nbristle\nbritish\nbroadcaster\nbroadcaster\nbroadcast\nbroadcast\nbroadcast\nbroadcast\nbroadcast\nbroadcast\nbroadcast\nbroadcast\nbroaden\nbroaden\nbroaden\nbroaden\nbroaden\nbrochure\nbreak\nbrokerage\nbroker-dealer\nbroker\nbroker\nbroker\nbroker\nbreak\nbronco\nbronc\nbrother\nbring\nbring\nbrowbeat\nbrowse\nbrowse\nbruise\nbruise\nbruise\nbrushback\nbrush\nbrush\nbrush\nbrush\nbrush\nbrussels\nbubble\nbubble\nbuccaneer\nbuck\nbuck\nbucket\nbuck\nbuck\nbuckle\nbuckle\nbuckle\nbuck\nbuck\nbuck\nbuck\nbuddy\nbud\nbudge\nbudge\nbudget\nbudgeteer\nbudget\nbudge\nbud\nbuffer\nbuffet\nbuffet\nbuffet\nbuffet\nbuff\nbuff\nbug\nbug\nbug\nbuilder\nbuilder\nbuilding-material\nbuilding-product\nbuilding\nbuilding\nbuild\nbuild\nbuild\nbuild\nbuild\nbuild\nbuild\nbuild\nbuild\nbulb\nbulge\nbulkhead\nbulldoze\nbulldozer\nbulletin\nbullet\nbullhorn\nbully\nbully\nbull\nbull\nbully\nbumble\nbumble\nbumble\nbump\nbump\nbump\nbump\nbum\nbunch\nbundle\nbundle\nbundle\nbungle\nbun\nbuoy\nbuoy\nbuoy\nbuoy\nbuoy\nbuoy\nburble\nburden\nburden\nburden\nburden\nburden\nbureaucracy\nbureaucrat\nbureaucrat\nbureau\nburgeon\nburger\nburglary\nburglarize\nburg\nburial\nbury\nbury\nbury\nburn\nburn\nburn\nburn\nburnish\nburnout\nburn\nburn\nburn\nburn\nburn\nburst\nburst\nburst\nburst\nburst\nbury\nbury\nbury\nbus\nbushel\nbush\nbusy\nbusiness-communication\nbusiness\nbusiness\nbusiness-machine\nbusinessman\nbusinessman\nbusiness-partner\nbusload\nbust\nbust\nbust\nbust\nbutler\nbutterfly\nbutton\nbuttress\nbuttress\nbuttress\nbutt\nbuy-back\nbuy-back\nbuyer\nbuyer\nbuyer\nbuying\nbuy\nbuy\nbuy-out\nbuyout\nbuy\nbuy\nbuy\nbuy\nbuy\nbuy\nbuzz\nbuzz\nbuzz\nbuzzword\nbylaw\nbyline\nbypass\nbystander\ncabin-crew\ncabinet\ncable\ncab\ncache\ncadet\ncadge\ncafe\ncafeteria\ncake\ncalamity\ncalavera\ncalculate\ncalculate\ncalculate\ncalculate\ncalculate\ncalculate\ncalculate\ncalculate\ncalculation\ncalculator\ncalibrate\ncalifornian\ncall\ncall\ncall\ncall\ncall\ncall\ncaller\ncaller\ncall\ncall\ncall\ncall\ncall\ncall\ncall\ncall\ncall\ncalm\ncalm\ncalm\ncalorie\ncaltran\ncalf\ncamera\ncome\ncamouflage\ncampaign\ncampaign\ncampaign\ncampaign\ncamp\ncamper\ncamp\ncampus\ncampus\ncanal\ncancel\ncancel\ncancel\ncancel\ncancellation\ncancel\ncancel\ncancer\ncancer-suppressor\ncandidate\ncandy\ncandle\ncan\ncan\ncan\ncanvas\ncanvass\ncanyon\ncapability\ncapacity\ncapacitor\ncapital-asset\ncapital-drain\ncapital-gain\ncapitalgain\ncapital-goods\ncapitalist\ncapitalize\ncapitalize\ncapitalize\ncapitalize\ncapitalize\ncapital\ncapital-to-asset\ncapita\ncapitulate\ncap\ncap\ncap\ncap\ncapsule\ncaptain\ncaption\ncaptivate\ncaptive\ncapture\ncapture\ncapture\ncapture\ncapture\ncap\ncap\ncarat\ncarbide-product\ncar-dealer\ncardholder\ncardinal\ncard\ncare\ncare\ncareen\ncareen\ncareen\ncareer\ncareer\ncare\ncare\ncare\ncaricature\ncaricature\ncarillon\ncare\ncarnivore\ncarpenter\ncarpetbagger\ncarpet\ncarpet\ncarp\ncarry\ncarry\ncarry\ncarrier\ncarrier\ncarry\ncarry-forward\ncarryforward\ncarry\ncarry\ncarry\ncarry\ncar\ncar\ncart\ncart\ncarton\ncartoonist\ncartoon\ncartridge\ncart\ncart\ncarve\ncarve\ncarver\ncarve\ncarve\ncarve\ncarve\ncascade\ncascade\ncaseload\ncase\ncase\ncase\ncash\ncash\ncash\ncash\ncasing\ncasino\ncasino\ncasket\ncassette\ncastigate\ncastigate\ncastigate\ncasting\ncast\ncastle\ncast\ncast\ncast\ncast\ncast\ncasualty\ncataclysm\ncatalog\ncatalog\ncatapult\ncatapult\ncataract\ncatastrophe\ncatcher\ncatch\ncatch\ncatch\ncatch\ncatch\ncategory\ncategorize\ncater\ncater\ncater\ncater\ncater\ncatfish\ncatheter\ncathode\ncat\ncattle\ncattle\ncatch\ncatch\ncause\ncause\ncause\ncause\ncause\ncause\ncause\ncaution\ncaution\ncaution\ncaution\ncaution\ncaveat\ncave\ncave\nc.d.s\ncd\ncease\ncease\ncease\ncease\ncease\ncede\ncede\ncede\ncede\nceiling\ncelebrate\ncelebrate\ncelebrate\ncelebrate\ncelebrate\ncelebration\ncelebrity\ncellar\ncellist\ncell\ncelluloid\ncement\ncement\ncement-maker\ncement\ncensor\ncensor\ncensure\ncentenarian\ncenter\ncenter\ncenter\ncenter\ncenter\ncenter\ncenter\ncentimeter\ncentralize\ncentralize\ncent\ncent\ncentury\ncenturion\nceo\nceramic\ncereal\nceremony\ncertificate\ncertificate\ncertificate\ncertify\ncertify\ncfc\nchafe\nchafe\nchain\nchain\nchain\nchair\nchair\nchairman\nchair\nchair\nchalk\nchalk\nchallenge\nchallenge\nchallenger\nchallenge\nchallenge\nchallenge\nchallenge\nchallenge\nchallenge\nchamber\nchamber\nchampagne\nchampion\nchampion\nchampion\nchampionship\nchampion\nchampion\nchamp\nchance\nchance\nchandelier\nchange\nchange\nchange\nchange\nchange\nchange\nchange\nchange\nchange\nchange\nchange\nchange\nchannel\nchannel\nchannel\nchannel\nchan\nchant\nchant\nchant\nchant\nchap\nchapter\ncharacteristic\ncharacterize\ncharacterize\ncharacterize\ncharacterize\ncharacterize\ncharacterize\ncharacter\ncharacter\ncharge\ncharge\ncharge-off\ncharge\ncharge\ncharge\ncharge\ncharge\ncharge\ncharge\ncharity\ncharity\ncharlatan\ncharlestonian\nchar\ncharter\ncharter\nchart\nchart\nchart\nchart\nchase\nchaser\nchase\nchase\nchassis\nchasten\nchasten\nchastise\nchastise\nchastise\nchat\nchat\nchauffeur\nchauffeur\ncheapen\ncheat\ncheater\ncheat\ncheat\ncheat\ncheckbook\ncheck\ncheck\ncheck\ncheckpoint\ncheck\ncheck\ncheck\ncheck\ncheck\ncheer\ncheer\ncheer\ncheerleader\ncheer\ncheer\ncheese\nchef\nchelicerate\nchemical\nchemical\nchemical-weapon\nchemist\ncherish\ncherish\ncherry\ncherub\nchew\nchew\nchew\nchew\nchew\nchicken\nchicken\nchide\nchide\nchide\nchief\nchild\nchild\nchild\nchill\nchill\nchill\nchime\nchimney\nchimpanzee\nchinese\nchip\nchip\nchip\nchisel\nchlorofluorocarbon\nchoice\nchoice\nchoke\nchoke\nchoke\nchoke\nchoose\nchoose\nchoose\nchoose\nchoose\nchop\nchop\nchop\nchopstick\nchop\nchord\nchoreographer\nchore\nchore\nchortle\nchorus\nchoose\nchoose\nchristian\nchromosome\nchronicle\nchronicle\nchuck\nchuckle\nchuckle\nchuckle\nchug\nchum\nchunk\nchurch\nchurch-goer\nchurn\nchurn\ncial\ncigarette\ncigar\ncinch\ncincinnatus\ncircle\ncircle\ncircle\ncircuit\ncircular\ncirculate\ncirculate\ncirculate\ncirculate\ncirculate\ncirculation\ncircumstance\ncircumvent\ncircumvent\ncircumvent\ncitation\ncite\ncite\nCite\ncite\ncite\ncite\nciticorp\ncity\ncity\ncite\nCite\ncitizen\ncitizen\ncitizen\ncivic\ncivilian\ncivil-rights\nclothe\nclothe\nclaimant\nclaimant\nclaim\nclaim\nclaim\nclaim\nclaim\nclaim\nclaim\nclaim\nclaim\nclaim\nclamber\nclamor\nclampdown\nclamp\nclamp\nclamp\nclang\nclank\nclarification\nclarify\nclarify\nclarify\nclarify\nclash\nclash\nclash\nclash\nclass\nclass\nclass\nclassic\nclassic\nclassification\nclassify\nclassify\nclassify\nclassify\nclassmate\nclassroom\nclause\nclaw\nclean\nclean\ncleaner\nclean-fuel\nclean\nclean\ncleanse\ncleanse\ncleanser\ncleanse\ncleanse\nclean\nclean\nclearance\nclear\nclear\nclear\nclear\nclear\nclear\nclear\nclergy\ncleric\nclerk\nclick\nclientele\nclient\nclient\nclimb\nclimb\nclimber\nclimb\nclimb\nclimb\nclinch\nclinch\ncling\ncling\nclinical-product\nclinic\nclinker\nclip\nclip\nclipping\nclip\nclip\nclique\ncloak\nclobber\nclobber\nclobber\nclock\nclock\nclock\nclog\nclog\nclone\nclone\nclose\nclose\nclose\nclose\nclose\nclose\nclose\nclose\nclose\nclose\nclosing\nclose\nclose\nclosure\nclothe\nclothes\nclothier\ncloud\ncloud\ncloud\ncloud\ncloud\nclown\nclub\nclub\nclub\nclub\nclub\nclue\nclump\ncluster\ncluster\nclutch\nclutch\nclutter\nclutter\nc'mon\ncoach\ncoach\nco-anchor\ncoast\ncoaster\ncoast\ncoat\ncoating\ncoat\ncoat\ncoattail\nco-author\nco-author\ncoax\ncoax\ncobble\ncobble\ncobb\nco-chairman\ncockatoo\ncockroach\ncocktail\ncoconut\ncoddle\ncode\nco-defendant\ncode-name\ncode\ncode\nco-developer\ncodify\ncodify\nco-edit\ncoerce\nco-exist\ncoextrude\ncoffer\nco-found\nco-founder\ncognoscenti\ncohere\ncohort\nco-host\ncoincide\ncoincide\ncoincide\ncoin\ncoin\ncola\ncold-cut\ncoli\ncollaborate\ncollaborate\ncollaborate\ncollaborate\ncollaborator\ncollage\ncollapse\ncollapse\ncollapse\ncollapse\ncollapse\ncollapse\ncollar\ncollateralize\ncolleague\ncolleague\ncollect\ncollect\ncollectible\ncollectible\ncollect\ncollection\ncollective\ncollectivizer\ncollector\ncollector\ncollect\ncollect\ncollect\ncollege\ncollege\ncollege-sport\ncolloquy\ncolonialist\ncolony\ncolonist\ncolor-code\ncolor-code\ncolor\ncolor\ncolor\ncolumnist\ncolumn\nco-manage\nco-manage\ncombatant\ncombate\ncombat\ncombat\ncomb\ncombination\ncombine\ncombine\ncombine\ncombine\ncombine\ncombine\ncombine\ncomb\ncombine\ncomedy\ncomestible\ncome\ncome\ncomet\ncome\ncome\ncome\ncome\ncome\ncome\ncomfort\ncomfort\ncomfort\ncome\ncome\ncommand\ncommander\ncommand\ncommando\ncommand\ncommand\ncommemorate\ncommemorate\ncommemorate\ncommence\ncommence\ncommend\ncommend\ncommentary\ncommentator\ncommentator\ncomment\ncomment\ncomment\ncomment\ncomment\ncomment\ncomment\ncomment\ncommercialize\ncommercialize\ncommercial\ncommercial\ncommission\ncommission\ncommissioner\ncommission\ncommission\ncommission\ncommission\ncommission\ncommitment\ncommit\ncommit\ncommit\ncommittee\ncommittee\ncommittee\ncommitte\ncommit\ncommit\ncommit\ncommodity\ncommodity\ncommoditize\ncommunicate\ncommunicate\ncommunicate\ncommunicate\ncommunication\ncommunique\ncommunist\ncommunist\ncommunity\ncommuter\ncommuter\ncommute\ncommute\ncompact\ncompany\ncompany\ncompany\ncompanion\ncompaq\ncompare\ncompare\ncompare\ncompare\ncompare\ncompare\ncompare\ncompare\ncompare\ncompare\ncomparison\ncompatriot\ncompel\ncompel\ncompel\ncompel\ncompensate\ncompensate\ncompensate\ncompensation\ncompete\ncompete\ncompete\nCompete\ncompete\ncompete\ncompete\nCompete\ncompete\ncompetition\ncompetitor\ncompetitor\ncompile\ncompile\ncompile\ncompile\ncompile\ncomplain\ncomplain\ncomplain\ncomplain\ncomplaint\ncomplaint\ncomplain\ncomplain\ncomplement\ncomplement\ncomplete\ncomplete\ncomplete\ncomplete\ncomplete\ncompletion\ncomplex\ncomplexity\ncomplicate\ncomplicate\ncomplicate\ncomplicate\ncomplication\ncomply\ncomply\ncompliment\ncompliment\ncomply\ncomply\ncomply\ncomponent\ncompose\ncomposer\ncomposite\ncomposition\ncompound\ncompound\ncompound\ncompound\ncompound\ncompound\ncompress\ncompressor\ncomprise\ncomprise\ncomprise\ncomprise\ncomprise\ncompromise\ncompromise\ncompromise\ncompromise\ncompromise\ncompromise\ncompulsion\ncomputation\ncomputerize\ncomputerize\ncomputerize\ncomputer-product\ncomputer-service\ncomputer\ncomputer\ncomputer\ncomputer-system\ncompute\ncompute\ncompute\ncomrade\nconceal\nconceal\nconceal\nconcede\nconcede\nconcede\nconcede\nconcede\nconcede\nconceive\nconceive\nconceive\nconceive\nconcentrate\nconcentrate\nconcentrate\nconcentrate\nconcentrate\nconcentrate\nconcentration\nconception\nconcept\nconcern\nconcern\nconcern\nconcern\nconcern\nconcern\nconcern\nconcern\nconcern\nconcerto\nconcert\nconcession\nconclude\nconclude\nconclude\nconclude\nconclude\nconclude\nconclusion\nconcoct\nconcoction\nconcoct\nconcur\nconcur\ncondemn\ncondemn\ncondemn\ncondemn\ncondemn\ncondenser\ncondition\nconditioner\ncondition\ncondominium\ncondom\ncondone\ncondone\ncondo\nconduct\nconduct\nconduct\nconduct\nconduct\nconduct\nconduit\nconduit\ncone\nconferee\nconferee\nconference\nconfer\nconfer\nconfer\nconfer\nconfer\nconfess\nconfess\nconfess\nconfess\nconfession\nconfidant\nconfide\nconfide\nconfide\nconfiguration\nconfine\nconfines\nconfirm\nconfirm\nconfirm\nconfirm\nconfirm\nconfirm\nconfirm\nconfiscate\nconfiscate\nconfiscate\nconfiscate\nconflict\nconflict\nconflict\nconflict\nconform\nconform\nconform\nconform\nconfrontation\nconfrontation\nconfront\nconfront\nconfront\nconfront\nconfront\nconfront\nconfuse\nconfuse\nconfuse\nconfuse\nconfuse\nconfusion\nconglomerate\ncongratulate\ncongratulate\ncongratulate\ncongressman\ncongressman\nconjure\nconjure\nconnect\nconnect\nconnection\nconnect\nconnect\nconnect\nconnotation\nconnote\nconquer\nconscript\nconsent\nconsent\nconsent\nconsent\nconsent\nconsequence\nconservationist\nconservationist\nconservative\nconservative\nconservative\nconserve\nconsideration\nconsider\nconsider\nconsider\nconsider\nconsider\nconsider\nconsider\nconsider\nconsider\nconsign\nconsign\nconsist\nconsist\nconsist\nconsist\nconsist\nconsist\nconsist\ncon\nconsole\nconsole\nconsolidate\nconsolidate\nconsolidate\nconsolidate\nconsolidate\nconsolidation\nconsonant\nconsort\nconspiracy\nconspirator\nconspire\nconspire\nconspire\nconstituency\nconstituent\nconstitute\nconstitute\nconstitute\nconstitute\nconstrain\nconstrain\nconstraint\nconstrain\nconstrict\nconstrictor\nconstruct\nconstruct\nconstruction\nconstruct\nconstrue\nconstrue\nconstrue\nconstrue\nconsultant\nconsultant\nconsultation\nconsult\nconsult\nconsult\nconsume\nconsume\nconsumer-electronics\nconsumer-goods\nconsumer-product\nconsumer\nconsumer\nconsume\nconsume\nconsummate\ncontact\ncontact\ncontact\ncontact\ncontact\ncontain\ncontain\ncontainer\ncontain\ncontain\ncontain\ncontain\ncontaminate\ncont'd.\ncontemplate\ncontemplate\ncontemplate\ncontemplate\ncontemplate\ncontemplate\ncontemporary\ncontemporize\ncontend\ncontend\ncontender\ncontend\ncontend\ncontend\ncontend\ncontention\ncontents\ncontent\ncontestant\ncontest\ncontest\ncontest\ncontest\ncontest\ncontinental\ncontingency\ncontinue\ncontinue\ncontinue\ncontinue\ncontinue\ncontinue\ncontinue\ncontinue\ncontraceptive\ncontract-drill\ncontract\ncontract\ncontract\ncontract\ncontraction\ncontractor\ncontractor\ncontract\ncontract\ncontract\ncontract\ncontract\ncontradict\ncontradiction\ncontradict\ncontradict\ncontra\ncontrast\ncontrast\ncontrast\ncontrast\ncontrast\ncontrast\ncontrast\ncontravene\ncontribue\ncontribute\ncontribute\ncontribute\ncontribute\ncontribute\ncontribute\ncontribute\ncontribution\ncontributor\ncontrol\ncontrol\ncontroller\ncontrol\ncontrol\ncontrol\ncontrol\ncontrol\ncontroversy\ncon\nconvenant\nconvene\nconvene\nconvene\nconvene\nconvene\nconvene\nconventional-arm\nconventioner\nconvention-goer\nconvention\nconverge\nconversation\nconversion\nconvert\nconvert\nconverter\nconvertible\nconvert\nconvert\nconvert\nconvert\nconvert\nconvey\nconvey\nconvey\nconvey\nconvict\nconvict\nconviction\nconvict\nconvict\nconvince\nconvince\nconvince\nconvince\nconvince\nconvince\nconvince\nconvolute\nconvolution\nconvoy\nconvulsion\ncookbook\ncook\ncookie\ncook\ncook\ncook\ncoolant\ncool\ncool\ncool\ncool\ncool\ncool\ncool\ncooperate\ncooperate\ncooperate\ncooperate\ncooperate\ncooperative\ncoordinate\ncoordinate\ncoordinate\ncoordinate\ncoor\nco-payment\ncope\ncopy\ncopy\ncopier\ncopy\nco-pilot\ncope\nco-president\nco-produce\ncopycat\ncopy\ncopyright\ncopyright\ncopy\ncopy\ncopy\ncore\ncorkscrew\ncorner\ncorner\ncornerstone\ncorner\ncornice\ncorporate-earnings\ncorporate\ncorporation\ncorporation\ncorral\ncorrect\ncorrect\ncorrect\ncorrection\ncorrect\ncorrect\ncorrect\ncorrelate\ncorrespond\ncorrespondent\ncorrespond\ncorrespond\ncorridor\ncorroborate\ncorvette\ncosmetic\ncosmetic\ncosmology\nco-sponsor\nco-sponsor\nco-sponsor\ncosta\ncost\ncost\ncost\ncost\ncostume\ncostume\ncost\ncost\ncost\ncost\ncot\ncottage\ncouch\ncouch\ncough\ncough\ncough\ncough\ncough\ncouncilor\ncouncil\ncounsel\ncounsel\ncounselor\ncounsel\ncounsel\ncount\ncount\ncountenance\ncounteract\ncounteract\ncounterattack\ncounterbalance\ncounterbidder\ncounterbid\ncounter-claim\ncounterclaim\ncounter\ncounter\ncounter\ncounter\ncountermeasure\ncounterpart\ncounterprogram\ncounter\ncountersue\ncountersue\ncountersue\ncountervail\ncounter\ncounter\ncounty\ncount\ncount\ncountry\ncountry\ncountryman\ncount\ncount\ncount\ncount\ncount\ncouple\ncouple\ncouple\ncouplet\ncoup-maker\ncoupon\ncoupon\ncoup\ncourier\ncourse\ncourse\ncourt\ncourt\ncourtesy\ncourthouse\ncourt\ncourtroom\ncourt\ncourt\ncourt\ncourt\ncourt\ncousin\ncovenant\ncoverage\ncover\ncover\ncovering\ncover\ncover\ncover\ncovert\ncover\ncover\ncovet\ncovet\ncoward\ncowboy\ncowboy\ncower\nco-worker\ncow\ncpa\ncrab\ncrack\ncrack\ncrack\ncrack\ncrackle\ncrack\ncrack\ncrack\ncraft\ncraft\ncraft\ncraftsman\ncraft\ncram\ncram\ncramp\ncram\ncrane\ncrane\ncrank\ncrank\ncrank\ncrank\ncrash\ncrash\ncrash\ncrash\ncrash\ncrash\ncrash\ncrater\ncrate\ncrave\ncrawl\ncrawl\ncrawl\ncrawl\ncreak\ncream\ncream\ncreate\ncreate\nCreate\ncreate\ncreate\ncreate\ncreate\ncreate\ncreate\ncreation\ncreator\ncreator\ncreature\ncredentials\ncredit-datum\ncredit\ncredit\ncredit\ncreditor\ncreditor\ncredit-rating\ncredit\ncredit\ncredit\ncredit\ncreep\ncrematorium\ncreep\ncreep\ncrest\ncrevasse\ncrevice\ncrew\ncry\ncrime\ncriminalize\ncriminal\ncrimp\ncrimp\ncringe\ncripple\ncripple\ncripple\ncrisis\ncrisis\ncrisscross\ncriss-cross\ncriterion\ncriticism\ncriticize\ncriticize\ncriticize\ncriticize\ncriticize\ncriticize\ncritic\ncritic\ncroak\ncroissant\ncrony\ncrook\ncroon\ncroon\ncrop\ncrop\ncrop\ncrop\ncrop\ncross-breed\ncross\ncross\ncross\ncross\ncross-pollinate\ncrossroads\ncross-shareholding\ncross\ncross\ncrouch\ncrouch\ncrowd\ncrowd\ncrowd\ncrowd\ncrowd\ncrow\ncrown\ncrow\ncr\ncrude\ncruiser\ncruise\ncruise\ncrumble\ncrumble\ncrumble\ncrumble\ncrumble\ncrumple\ncrumple\ncrunch\ncrunch\ncrush\ncrush\ncrush\ncrush\ncry\ncryptographer\ncrystal\ncry\ncry\ncub\ncuckoo\ncue\ncuff\ncull\nculminate\nculminate\nculminate\nculminate\nculprit\ncultivate\ncultivate\ncultivate\nculture\ncup\ncurator\ncurb\ncurb\ncurb\ncurb\ncurb\ncurb\ncurb\ncure\ncure\ncure\ncure\ncurl\ncurl\ncurrency\ncurrent\ncurry\ncurse\ncurse\ncurtail\ncurtail\ncurtail\ncurtail\ncurtail\ncurtain\ncushion\ncushion\ncushion\ncustomer\ncustomer\ncustomize\ncustom\ncustom\ncutback\ncutout\ncut\ncut\ncutter\ncutting-tool\ncut\ncut\ncut\ncut\ncut\ncut\ncut\ncut\ncuvee\ncvb\ncycad\ncycad\ncycle\ncyclical\ncycle\ncyclist\nc-yield\nczar\ndabble\ndabble\ndabble\ndab\ndaily\ndalliance\ndamage\ndamage\ndamages\ndamage\ndamage\ndamage\ndamage\ndamp\ndamp\ndampen\ndampen\ndamp\ndamp\ndamp\ndam\ndancer\ndance\ndance\ndance\ndanger\ndanger\ndangle\ndangle\ndare\ndare\ndare\ndarling\ndash\ndash\ndash\ndash\ndatabase\ndatum\ndatum\ndatum\ndatum\ndate\ndate\ndate\ndate\ndate\ndate\ndaughter\ndaunt\ndaunt\ndawdle\ndawn\ndawn\nday.\nday\nday\ndazzle\ndeactivate\ndeadbeat\ndeadline\ndeadlock\ndead\ndeafen\ndealership\ndealer\ndealer\ndealings\ndeal\ndeal\ndealmaker\ndeal\ndeal\ndeal\ndeal\ndeal\ndeal\ndeath\ndebacle\ndebate\ndebate\ndebate\ndebate\ndebate\ndebenture\ndebtholder\ndebtor\ndebt\ndebunk\ndebut\ndebut\ndecade\ndecade\ndecapitalize\ndecease\ndeceive\ndeceive\ndeceive\ndecelerate\ndecelerate\ndecentralize\ndecentralize\ndecertify\ndecide\ndecide\ndecide\ndecide\ndecide\ndecide\ndecide\ndecide\ndecimate\ndecision-maker\ndecision\ndecision\ndeck\ndeckhand\ndeck\ndeck\ndeclaration\ndeclare\ndeclare\ndeclare\ndeclare\ndeclare\ndeclare\ndeclare\ndeclassify\ndecline\ndecline\ndecliner\ndecliner\ndecline\ndecline\ndecline\ndecline\ndecline\ndecline\ndecline\ndecontaminate\ndecorate\ndecorate\ndecorator\ndecoy\ndecrease\ndecrease\ndecrease\ndecrease\ndecrease\ndecrease\ndecrease\ndecree\ndecribe\ndecry\ndecry\ndecry\ndec.\ndedicate\ndedicate\ndeduce\ndeduct\ndeductible\ndeduct\ndeduction\ndeduction\ndeduct\ndeed\ndeem\ndeem\nde-emphasize\nde-emphasize\ndeem\ndeem\ndeepen\ndeepen\ndeer\ndeer\ndefault\ndefault\ndefaulter\ndefault\ndefault\ndefault\ndefault\ndefeat\ndefeat\ndefeat\ndefeat\ndefeat\ndefeat\ndefect\ndefect\ndefection\ndefection\ndefect\ndefect\ndefendant\ndefendant\ndefend\ndefend\ndefender\ndefender\ndefend\ndefend\ndefend\ndefend\ndefend\ndefense-electronics\ndefens\ndefer\ndefer\ndefer\ndeficiency\ndeficit\ndefy\ndefy\ndefine\ndefine\ndefine\ndefine\ndefine\ndefine\ndefinition\ndeflate\ndeflate\ndeflate\ndeflator\ndeflect\ndeflect\ndeflect\ndeform\ndefraud\ndefraud\ndefraud\ndefraud\ndefunct\ndefuse\ndefuse\ndefy\ndefy\ndegenerate\ndegenerate\ndegenerate\ndegree\ndelay\ndelay\ndelaying\ndelay\ndelay\ndelay\ndelay\ndelay\ndelay\ndelegate\ndelegate\ndelegate\ndelegate\ndelete\ndelete\ndelete\ndeletion\ndeliberate\ndeliberate\ndeliberation\ndelight\ndelight\ndelight\ndelight\ndelight\ndeli\ndelinquency\ndelinquent\ndelist\ndeliver\ndeliver\ndelivery\ndelivery\ndeliver\ndeliver\ndeliver\ndeliver\ndeliver\ndelouse\ndeluge\ndeluge\ndelve\ndelve\ndemagogue\ndemand\ndemand\ndemand\ndemand\ndemand\ndemand\ndemand\ndemean\ndemeanor\ndemilitarize\ndemobilize\ndemobilize\ndemocracy\ndemocratize\ndemocratize\ndemocrat\ndemographic\ndemographic\ndemolish\ndemolish\ndemolish\ndemolish\ndemonize\ndemon\ndemonstrate\ndemonstrate\ndemonstrate\ndemonstrate\ndemonstrate\ndemonstrate\ndemonstration\ndemonstration\ndemonstrator\ndemote\ndemur\ndenationalize\ndenial\ndeny\ndeny\ndeny\ndeny\ndenims\ndenizen\ndenominate\ndenomination\ndenounce\ndenounce\ndenounce\ndenounce\ndenounce\ndentist\ndent\ndenuclearize\ndenude\ndeny\ndeny\ndeny\ndepart\ndepart\ndepart\ndepartment\ndeparture\ndepart\ndepart\ndepend\ndepend\ndependent\ndepend\ndepend\ndepend\ndepend\ndepend\ndepict\ndepict\ndepict\ndepict\ndepict\ndepict\ndeplete\ndeplete\ndeplete\ndeplete\ndeplore\ndeplore\ndeploy\ndeploy\ndeport\ndepose\ndeposit\ndeposit\ndeposit\ndeposition\ndepositor\ndeposit\ndeposit\ndeposit\ndeposit\ndepot\ndepreciate\ndepredation\ndepress\ndepress\ndepress\ndepress\ndepression\ndepress\ndepress\ndeprivation\ndeprive\ndeprive\ndeprive\ndeprive\ndeprive\ndeprive\ndeprogramming\ndepths\ndeputy\ndeputy\nderail\nderail\nderail\nderegulate\nderegulate\nderide\nderide\nderivative\nderive\nderive\nderive\nderive\nderive\ndescendant\ndescend\ndescend\ndescend\ndescent\ndescribe\ndescribe\ndescribe\ndescribe\ndescribe\ndescribe\ndescribe\ndescription\ndesert\ndesert\ndesert\ndeserve\ndeserve\ndeserve\ndeserve\ndesignate\ndesignate\ndesignate\ndesignate\ndesignation\ndesign\ndesign\ndesignee\ndesigner\ndesign\ndesign\ndesign\ndesign\ndesign\ndesire\ndesire\ndesire\ndesire\ndesist\ndesk\ndespair\ndespise\ndespise\ndespot\ndestabilize\ndestabilize\ndestination\ndestination\ndestine\ndestroy\ndestroy\ndestroy\ndestroy\ndestroy\ndestroy\ndetach\ndetach\ndetail\ndetail\ndetail\ndetail\ndetain\ndetain\ndetain\ndetect\ndetect\ndetect\ndetective\ndetector\ndetect\ndetect\ndetergent\ndeteriorate\ndeteriorate\ndeteriorate\ndeteriorate\ndeteriorate\ndeteriorate\ndetermine\ndetermine\ndetermine\ndetermine\ndetermine\ndetermine\ndetermine\ndeter\ndeterrent\ndeter\ndeter\ndeter\ndeter\ndetest\ndethrone\ndetractor\ndetract\ndetract\ndevaluation\ndevalue\ndevalue\ndevastate\ndevastate\ndevastate\ndevelop\ndevelop\ndevelop\ndeveloper\ndeveloper\ndevelop\ndevelop\ndevelopment\ndevelopment\ndevelop\ndevelop\ndevelop\ndevelop\ndeviate\ndeviation\ndevice\ndevil\ndevise\ndevise\ndevise\ndevise\ndevise\ndevote\ndevote\ndevotee\ndevotee\ndevote\ndevote\ndevote\ndevour\ndevour\ndewater\ndiabetic\ndiagnose\ndiagnose\ndiagnose\ndiagnostic\ndiagram\ndialect\ndial\ndial\ndial\ndial\ndiamond\ndiaper\ndiary\ndibenzofuran\ndice\ndicker\ndictate\ndictate\ndictate\ndictate\ndictate\ndictate\ndictatorship\ndictator\ndo\ndo\ndie\ndie\ndie\ndie\ndie\ndie-hard\ndiehard\ndiesel\ndie\ndie\ndie\ndie\ndie\ndie\ndiffer\ndifference\ndifference\ndifferential\ndifferentiate\ndifferentiate\ndiffer\ndiffer\ndiffer\ndiffer\ndifficulty\ndigest\ndigest\ndigest\ndig\ndigit\ndignify\ndignitary\ndig\ndig\ndig\ndig\ndilute\ndilute\ndilute\ndilute\ndilute\ndimension\ndime\ndiminish\ndiminish\ndiminish\ndiminish\ndiminish\ndiminish\ndim\ndine\ndine\ndiner\ndine\ndinner\ndinosaur\ndioxin\ndiplomat\ndip\ndip\ndip\ndip\ndip\ndip\ndirect\ndirect\ndirect\ndirection\ndirective\ndirectory\ndirector\ndirector\ndirect\ndirect\ndirk\ndisability\ndisable\ndisabled-worker\ndisable\ndisadvantage\ndisagree\ndisagree\ndisagreement\ndisagree\ndisagree\ndisagree\ndisallow\ndisappear\ndisappear\ndisappear\ndisappear\ndisappear\ndisappoint\ndisappoint\ndisappointment\ndisappointment\ndisappointment\ndisappoint\ndisapprove\ndisapprove\ndisapprove\ndisapprove\ndisarm\ndisarm\ndisassemble\ndisassociate\ndisaster\ndisaster\ndisavow\ndisband\ndisband\ndisband\ndisburse\ndisbursement\ndiscard\ndiscard\ndiscard\ndiscern\ndiscern\ndischarge\ndischarge\ndischarge\ndisciple\ndiscipline\ndiscipline\ndiscipline\ndisclaim\ndisclose\ndisclose\ndisclose\ndisclose\ndisclose\ndisclosure\ndisclosure\ndiscolor\ndiscomfit\ndisconnect\ndisconnect\ndiscontinue\ndiscontinue\ndiscontinue\ndisco\ndisco\ndiscount\ndiscount\ndiscount\ndiscount\ndiscount\ndiscount\ndiscount\ndiscourage\ndiscourage\ndiscourage\ndiscourage\ndiscourage\ndiscourage\ndiscover\ndiscover\ndiscovery\ndiscover\ndiscover\ndiscover\ndiscover\ndiscredit\ndiscredit\ndiscrepancy\ndiscrepancy\ndiscrepency\ndiscriminate\ndisc\ndiscuss\ndiscuss\ndiscuss\ndiscuss\ndiscussion\ndiscuss\ndiscuss\ndisdain\ndisdain\ndisease\ndisembark\ndisenchant\ndisengage\ndisgorge\ndisgruntle\ndisguise\ndisguise\ndisguise\ndisgust\ndisgust\ndish\ndish\ndishwasher\ndisincline\ndisinfectant\ndisintegrate\ndisintegrate\ndisk\ndislike\ndislike\ndislike\ndislocation\ndismantle\ndismantle\ndismantle\ndismantle\ndismay\ndismember\ndismiss\ndismiss\ndismiss\ndismiss\ndismiss\ndismiss\ndismiss\ndisobey\ndisorder\ndisparage\ndisparage\ndisparage\ndisparity\ndispatch\ndispatch\ndispatcher\ndispatch\ndispatch\ndispatch\ndispel\ndispel\ndispense\ndispense\ndispense\ndispense\ndispersant\ndisperse\ndisperse\ndisplace\ndisplace\ndisplay\ndisplay\ndisplay\ndisplay\ndisplay\ndisplay\ndisplay\ndisplease\ndisposable\ndisposal\ndispose\ndispose\ndispose\ndispose\ndispose\ndisposition\ndisprove\ndispute\ndispute\ndispute\ndispute\ndispute\ndispute\ndisqualify\ndisqualify\ndisregard\ndisregard\ndisrupt\ndisrupt\ndisrupt\ndisruption\ndisrupt\ndissatisfy\ndissect\ndissect\ndisseminate\ndisseminate\ndisseminate\ndissent\ndissent\ndissenter\ndissent\ndissident\ndissident\ndissipate\ndissipate\ndissipate\ndissociate\ndissociate\ndissolve\ndissolve\ndissolve\ndissolve\ndissuade\ndistance\ndistance\ndistance\ndistil\ndistiller\ndistil\ndistinction\ndistinguish\ndistinguish\ndistinguish\ndistort\ndistortion\ndistort\ndistort\ndistort\ndistract\ndistract\ndistract\ndistraction\ndistract\ndistribute\ndistribute\ndistribute\ndistribute\ndistribute\ndistribute\ndistribute\ndistribute\ndistribution\ndistributor\ndistrict\ndistricts\\/states\ndisturbance\ndisturb\ndisturb\ndisturb\ndisturb\ndither\ndive\ndiverge\ndiverge\ndiversife\ndiversification\ndiversify\ndiversify\ndiversify\ndiversify\ndiversion\ndiver\ndivert\ndivert\ndivert\ndivert\ndivest\ndivest\ndivest\ndivestiture\ndivest\ndivest\ndive\ndive\ndive\ndivide\ndivide\ndividend\ndivide\ndivide\ndivide\ndivide\ndivide\ndivide\ndive\ndivision\ndivorce\ndivorce\ndivulge\ndivvy\ndivvy\ndizzy\ndocket\ndock-sider\ndoctor\ndoctor\ndoctor\ndoctor\ndoctrine\ndocudrama\ndocumentary\ndocument\ndocument\ndocument\ndocument\ndocument\ndocument\ndodge\ndodge\ndo\ndo\ndog\ndog\ndog\ndo-gooder\ndog\ndog\ndog\ndog\ndo\ndo\ndoldrums\ndole\ndole\ndole\ndole\ndollar\ndoll\ndolphin\ndome\ndominate\ndominate\ndominate\ndominate\ndominate\ndominate\ndonate\ndonate\ndonate\ndonate\ndonate\ndonate\ndonation\ndonation\ndo\ndon\ndon\ndonor\ndon\ndon\ndon't\ndon\ndoom\ndoom\ndoom\ndoomsayer\ndoom\ndoorman\ndoor\ndose\ndossier\ndote\ndoth\ndot\ndot\ndouble-cross\ndouble\ndouble\ndouble\ndouble\ndouble\ndouble\ndouble\ndoubt\ndoubter\ndoubt\ndoubt\ndoubt\ndoubt\ndoubt\ndo\ndo\ndo\ndo\ndovetail\ndive\ndown\ndowngrade\ndowngrade\ndowngrade\ndowngrade\ndowngrade\ndownpayment\ndownplay\ndownsize\ndownsize\ndownsize\ndowns\ndownturn\ndown\ndozen\ndozen\ndraft\ndraft\ndraft\ndraftsman\ndraft\ndrag\ndrag\ndrag\ndrag\ndrag\ndrag\ndrag\ndrain\ndrain\ndrain\ndrain\ndramatization\ndram\ndrape\ndrape\ndrawback\ndrawing\ndraw\ndraw\ndraw\ndraw\ndraw\ndraw\ndraw\ndread\ndream\ndream\ndream\ndream\ndream\ndream\ndream\ndream\ndream\ndream\ndress\ndress\ndress\ndress\ndress\ndress\ndress\ndraw\ndry\ndry\ndrift\ndrift\ndrift\ndrift\ndrill\ndrill\ndriller\ndrill\ndrill\ndrill\ndrink\ndrink\ndrink\ndrink\ndrip\ndrive\ndriver\ndrive\ndrive\ndrive\ndrive\ndrive\ndrive\ndrool\ndrool\ndrop\ndroplet\ndropout\ndrop\ndrop\ndropper\ndrop\ndrop\ndrop\ndrop\ndrop\ndrought\ndrove\ndrive\ndrown\ndrown\ndrown\ndrug-sale\ndrug\ndrug\ndrugstore\ndrum\ndrum\ndrum\ndry\ndry\nd\ndub\ndub\ndub\ndub\nduck\nduckling\nduck\nduck\nduck\nduck\nduct\ndud\nduel\nduel\ndue\ndue\nduffer\ndig\ndull\ndummy\ndump\ndump\ndump\ndump\ndump\ndump\ndump\ndune\ndupe\nduplicate\nduplicate\nduplicate\nduplication\ndurable-goods\ndurable\ndust\ndust\nduty\n'd\ndwarf\ndwarf\ndwarf\ndwarf\ndweller\ndwelling\ndwindle\ndwindle\ndye\ndye\ndie\ndynamics\ndynamo\nearmark\nearmark\nearmark\nearmark\nearn\nearn\nearner\nearnig\nearnings\nearnings\nearnings\nearn\nearn\nearn\nearn\nearn\nearring\near\nearthling\nearthquake\nearthworm\nease\nease\nease\nease\nease\nease\neasterner\neast\neast\neat\neater\neat\neat\neat\neat\neavesdrop\nebb\nebb\neccentric\nechelon\necho\necho\necho\necho\necho\necho\necho\neclair\neclipse\neclipse\neconomics\neconomy\neconomist\neconomist\neconomize\nedge\nedge\nedge\nedge\nedge\nedition\neditorial\neditorial\neditor\nedit\neducate\neducate\neducate\neducation\neducator\neducator\neffect\neffect\neffect\nefficiency\neffort\neffort\negg\nego\neject\neke\neke\nelaborate\nelaborate\nelaborate\nelapse\nelbow\nelder\nelect\nelect\nelect\nelection\nelection\nelection\nelectrical-product\nelectrify\nelectrochemical\nelectrode\nelectrogalvanize\nelectrogalvanize\nelectromagnet\nelectronic-datum\nelectronics\nelectronic-system\nelectro-optics\nelect\nelect\nelement\nelement\nelephant\nelevate\nelevate\nelevation\nelevator\neliminate\neliminate\neliminate\neliminate\neliminate\neliminate\neliminate\neliminate\nelite\nelite\nelitist\nelongate\nelude\nelude\nemasculate\nembargo\nembargo\nembargo\nembark\nembark\nembark\nembark\nembarrass\nembarrass\nembarrass\nembarrass\nembassy\nembattle\nembed\nembed\nembellish\nembezzle\nembezzle\nemblem\nembody\nembody\nembody\nembolden\nembolden\nembrace\nembrace\nembrace\nembrace\nembrace\nembroil\nemerge\nemerge\nemergency\nemerge\nemerge\nemerge\nemerge\nemerge\nemigrate\nemigrate\nemigre\nemissary\nemission\nemit\nemote\nemote\nemotion\nempathize\nemphasize\nemphasize\nemphasize\nemphasize\nemphasize\nemphasize\nemphaticize\nempire\nemploy\nemployee\nemployee\nemployee\nemployee\nemployer\nemployer\nemploy\nemploy\nemploy\nemploy\nempower\nempower\nempower\nempower\nempty\nempty\nempty\nempty\nemulate\nemulate\nemulate\nenable\nenable\nenable\nenable\nenable\nenable\nenact\nenact\nenact\nenact\nencapsulate\nencase\nencircle\nenclose\nenclose\nencompass\nencompass\nencompass\nencounter\nencounter\nencounter\nencounter\nencounter\nencourage\nencourage\nencourage\nencourage\nencourage\nencourage\nencourage\nencourage\nencourage\nencroach\nencrust\nencrypt\nencumber\nendangered-species\nendanger\nendanger\nendeavor\nendeavor\nendeavor\nend\nend\nend\nending\nend\nend\nendorse\nendorse\nendorsement\nendorser\nendorse\nendorse\nendorse\nendorse\nendow\nendow\nendow\nend\nend\nend\nend\nend-tail\nendure\nendure\nendure\nendure\nendure\nendure\nend\nend\nend\nenemy\nenergy\nenergize\nenergy-service\nenforce\nenforcer\nenforcer\nenforce\nenforce\nenforce\nenforce\nengage\nengage\nengagement\nengage\nengage\nengage\nengage\nengineer\nengineer\nengineer\nengineer\nengineer\nengine\nenglish\nengrave\nengulf\nengulf\nenhance\nenhance\nenhancement\nenhance\nenhance\nenhance\nenjoin\nenjoin\nenjoin\nenjoy\nenjoy\nenjoy\nenjoy\nenjoy\nenjoy\nenjoy\nenlarge\nenlarger\nenlarge\nenlarge\nenlighten\nenlighten\nenlighten\nenlist\nenlist\nenlist\nenliven\nennumerate\nenrage\nenrich\nenrich\nenrol\nenrol\nenrollee\nenrollment\nenroll\nenroll\nensconce\nensemble\nensnarl\nensue\nensue\nensue\nensure\nensure\nensure\nensure\nentail\nentail\nentail\nentangle\nenter\nenter\nenter\nenterprise\nenter\nentertain\nentertain\nentertainer\nentertain\nentertain\nenter\nenter\nenter\nenthusiasm\nenthusiast\nenthusiast\nentice\nentice\nentice\nentice\nentity\nentitle\nentitle\nentitlement\nentitle\nentitle\nentitle\nentitle\nentomb\nentrance\nentrant\nentrench\nentrench\nentrepreneur\nentrepreneur\nentry\nentrust\nentrust\nentrust\nentwine\nenvelope\nenvironmentalist\nenvironmentalist\nenvironment\nenvisage\nenvisage\nenvision\nenvision\nenvision\nenvision\nenvy\neon\nepileptic\nepisode\nepitomize\nequal\nequal\nequal\nequal\nequal\nequal\nequate\nequate\nequate\nequestrian\nequip\nequip\nEquip\nequip\nequip\nequip\nequity\nequity\nequivalent\nequivalent\neradicate\nerase\nerase\nerase\nerase\nerase\nerase\nerasure\nerect\nerect\nerect\nerode\nerode\nerode\nerode\nerode\nerode\nerode\nerratum\nerr\nerr\nerror\nerr\nerr\nerr\nerr\nerupt\nerupt\nerupt\nerupt\nescalate\nescalate\nescalate\nescalate\nescalator\nescape\nescape\nescape\nescape\neschew\nescort\nescort\nescrow\nespouse\nesp\nessay\nessential\nestablish\nestablish\nestablish\nestablish\nestablish\nestablish\nestablishment\nestablish\nestablish\nestablsh\nestimate\nestimate\nestimate\nestimate\nestimate\nestimate\nestimate\nestimate\nestimate\nestimate\nestimator\nestrange\nethic\neuphemism\neurobond\neurodebenture\neurodollar\neurodollar\neuroissue\neuronote\neuropean\nevacuate\nevacuate\nevade\nevader\nevade\nevade\nevaluate\nevaluate\nevaluate\nevaluate\nevaluate\nevaluate\nevaluate\nevaluation\nevaluation\nevangelist\nevaporate\nevaporate\nevaporate\neven\nevening\neven\nevent\nevent\neven\neverglade\nevidence\nevil-doer\nevince\nevince\neviscerate\nevoke\nevoke\nevoke\nevolve\nevolve\nevolve\nevolve\nexacerbate\nexacerbate\nexacerbate\nexacerbate\nexacerbate\nexaggerate\nexaggerate\nexamination\nexamine\nexamine\nexaminer\nexaminer\nexamine\nexamine\nexamine\nexamine\nexamine\nexample\nexample\nexam\nexcavate\nexcavate\nexcavator\nexceed\nexceed\nexceed\nexceed\nexceed\nexceed\nexcel\nexcel\nexception\nexcept\nexcerpt\nexcerpt\nexcess\nexchange\nexchange\nexchange\nexchange\nexchange\nexchange\nexcise\nexcise\nexcise\nexcite\nexcite\nexclaim\nexclude\nexclude\nexclude\nexclude\nexclude\nexclude\nexclude\nexclude\nexclude\nexclusion\nexcorciate\nexcoriate\nexcursion\nexcuse\nexcuse\nexcuse\nexcutive\nexec\nexecute\nexecute\nexecute\nexecute\nexecute\nexecute\nexecute\nexecution\nexecution\nexecutive\nexecutive\nexecutive\nexecutor\nexemplify\nexemplify\nex-employee\nexempt\nexempt\nexemption\nexempt\nexercise\nexercise\nexercise\nexercise\nexercise\nexercise\nexercise\nexerpt\nexert\nexert\nexert\nexhale\nexhaust\nexhaust\nexhaust\nexhaust\nexhaust\nexhibit\nexhibit\nexhibition\nexhibitor\nexhibit\nexhibit\nexhibit\nexhort\nexile\nexile\nexist\nexist\nexist\nexist\nexist\nexist\nexist\nexit\nexit\nexit\nexit\nexonerate\nexonerate\nexonerate\nexorcise\nexorcism\nexpand\nexpand\nexpand\nexpand\nexpand\nexpand\nexpand\nexpansionist\nexpansion\nexpectation\nexpectation\nexpect\nexpect\nexpect\nexpect\nexpect\nexpect\nexpect\nexpect\nexpect\nexpect\nexpedient\nexpedite\nexpedite\nexpedite\nexpel\nexpel\nexpel\nexpend\nexpenditure\nexpenditure\nexpense\nexpense\nexperience\nexperience\nexperience\nexperience\nexperience\nexperience\nexperience\nexperience\nexperiment\nexperiment\nexperiment\nexperiment\nexperiment\nexpert\nexpert\nexpiration\nexpire\nexpire\nexpire\nexpire\nexpire\nexpire\nexplain\nexplain\nexplain\nexplain\nexplain\nexplain\nexplain\nexplain\nexplanation\nexplode\nexplode\nexplode\nexplode\nexplode\nexploit\nexploiter\nexploit\nexploit\nexploit\nexploit\nexploit\nexplore\nexplore\nexplore\nexplore\nexplosion\nexplosion\nexplosive\nexport\nexport\nexporter\nexport\nexport\nexport\nexport\nexport\nexport\nexpose\nexpose\nexpose\nexpose\nexpose\nexposure\nexpound\nexpress\nexpress\nexpress\nexpress\nexpression\nexpress\nexpress\nexpunge\nexpunge\nextend\nextend\nextend\nextend\nextend\nextend\nextension\nextension\nexterior\nextinguish\nextort\nextort\nextort\nextract\nextract\nextract\nextract\nextract\nextradite\nextradition\nextrapolate\nextra\nextreme\nextremist\nextricate\nextrusion\nexude\nexude\neyeball\neyeball\neyebrow\neye\neyeglass\neye\neye\neye\neyewitness\neye\nf16\nf-18\nf18\nfabricate\nfabricate\nfabrication\nfabricator\nfabric\nfacade\nface\nface\nface\nfacelift\nface\nface\nface\nface\nfacilitate\nfacilitate\nfacilitate\nfacility\nfacing\nface\nface\nfacsimile\nfaction\nfactory\nfactory\nfactor\nfactor\nfactor\nfactor\nfactory-job\nfact\nfade\nfade\nfade\nfade\nfade\nfad\nfail\nfail\nfail\nfailing\nfail\nfail\nfail\nfailure\nfailure\nfail\nfail\nfaint\nfaint\nfair\nfajita\nfake\nfake\nfake\nfall\nfall\nfall\nfall\nfall\nfall\nfalsify\nfalsify\nfalsify\nfalter\nfalter\nfalter\nfalter\nfalter\nfame\nfamiliarize\nfamily\nfamily\nfamine\nfanatic\nfancy\nfang\nfan\nfan\nfan\nfan\nfantasy\nfantasize\nfantasize\nfan\nfare\nfare\nfare\nfare\nfare\nfare\nfare\nfare\nfarmer\nfarmer\nfarmer\nfarm\nfarm\nfarmstead\nfarm\nfarm\nfarmwife\nfascinate\nfascist\nfashion\nfashion\nfashion\nfastball\nfasten\nfastener\nfatality\nfatality\nfather\nfather\nfat\nfatten\nfatten\nfatten\nfault\nfaultline\nfault\nfault\nfauna\nfavor\nfavor\nfavor\nfavorite\nfavor\nfavor\nfavor\nfavor\nfawn\nfax\nfax\nfear\nfear\nfear\nfear\nfear\nfear\nfear\nfear\nfeast\nfeast\nfeather\nfeat\nfeature\nfeature\nfeature\nfeature\nfeature\nfeature\nfeature\nfeed\nfeed\nfeed\nfeedlot\nfeedlot\nfeed\nfeed\nfeed\nfeeler\nfeeling\nfeeling\nfeel\nfeel\nfeel\nfeel\nfeel\nfeel\nfee\nfee\nfoot\nfella\nfell\nfell\nfellow\nfall\nfall\nfelony\nfelon\nfeel\nfeel\nfemale\nfeminist\nfence\nfend\nfend\nfend\nfend\nferret\nferret\nferry\nferry\nferry\nfertilize\nfertilizer\nfertilize\nfester\nfestival\nfestivity\nfestoon\nfestoon\nfetch\nfetch\nfetch\nfetch\nfetch\nfetus\nfeud\nfeud\nfiberglass\nfiber\nficial\nfidget\nfiefdom\nfield\nfield\nfield\nfield\nfield\nfield\nfighter-bomber\nfighter\nfight\nfight\nfight\nfight\nfight\nfight\nfigure\nfigure\nfigure\nfigure\nfigure\nfigure\nfigure\nfigure\nfigure\nfilbert\nfilch\nfile\nfile\nfiler\nfile\nfile\nfile\nfile\nfilings\nfile\nfilipino\nfill\nfill\nfill\nfill\nfill\nfill\nfill\nfilm\nfilm\nfilm\nfilm\nfilm-maker\nfilm\nfilter\nfilter\nfilter\nfilter\nfilter\nfinagle\nfinalist\nfinalize\nfinalize\nfinalize\nfinal\nfinance\nfinance\nfinances\nfinances\nfinance\nfinance\nfinancial-crime\nfinancial-service\nfinancier\nfinancing\nfinance\nfinding\nfind\nfind\nfind\nfind\nfind\nfind\nfind\nfine-art\nfine\nfine\nfine\nfinesse\nfine-tune\nfine\nfinger\nfinger\nfingerling\nfingerprint\nfinger\nfinger\nfine\nfinish\nfinish\nfinish\nfinish\nfinish\nfinish\nfinish\nfinish\nfireball\nfire\nfire\nfirefighter\nfirehoop\nfireman\nfireplace\nfireproof\nfire\nfire\nfire\nfire\nfirework\nfiring\nfire\nfirm\nfirm\nfirm\nfirm\nfirm\nfirm\nfirm\nfishery\nfisherman\nfish\nfissure\nfist\nfit\nfit\nfit\nfit\nfit\nfix\nfix\nfix\nfix\nfixture\nfix\nfizz\nfizzle\nfizzle\nflabbergast\nflag\nflag\nflame\nflame\nflank\nflap\nflap\nflare\nflare\nflare\nflashback\nflash\nflash\nflash\nflash\nflash\nflashlight\nflash\nflash\nflat\nflatten\nflatten\nflatten\nflatten\nflaunt\nflaunt\nflavor\nflaw\nflaw\nflaw\nflay\nflea\nfledge\nflee\nflee\nfleece\nflee\nfleet\nfleet\nflee\nfleshpot\nfly\nflex\nflicker\nflick\nfly\nflier\nfly\nfly\nflight-attendant\nflight\nflight\nflinch\nfling\nfling\nflip\nflip\nflip\nflip\nflirt\nflirt\nflirt\nfloat\nfloat\nfloat\nfloat\nfloat\nfloat\nfloat\nflock\nflock\nflock\nflog\nflood\nflood\nflood\nflood\nfloor\nflop\nflop\nfloppy\nflora\nflotation\nflounder\nflounder\nflounder\nflourish\nflourish\nflourish\nflourish\nflout\nflow\nflower\nflower\nflow\nfly\nflow\nflow\nflow\nflow\nfluctuate\nfluctuate\nfluctuate\nfluctuate\nfluctuate\nfluctuate\nfluctuation\nfluid\nflunk\nflunk\nfluoropolymer\nflush\nflush\nfly\nfly\nfly\nfly\nfoam\nfocus\nfocus\nfocus\nfocus\nfocus\nfocus\nfocus\nfoe\nfoe\nfog\nfoil\nfoil\nfoil\nfoil\nfoil\nfold\nfold\nfolder\nfold\nfold\nfold\nfolk\nfolk\nfolly\nfollow\nfollow\nfollower\nfollow\nfollow\nfollow\nfollow\nfollow\nfollow\nfoment\nfood-fish\nfood-service\nfood\nfoodstuff\nfool\nfool\nfool\nfool\nfoothill\nfoothill\nfoot\nfootnote\nfootnote\nfootstep\nfoot\nfoot\nforay\nforbid\nforbid\nforbid\nforbid\nforce\nforce\nforce\nforce\nforce\nforce\nforce\nforce\nforce\nforecaster\nforecast\nforecast\nforecast\nforecast\nforecast\nforecast\nforecast\nforecast\nforeclose\nforeclose\nforeclose\nforeclose\nforeclosure\nforeclosure\nforefather\nforeigner\nforeigner\nforensic\nforerunner\nforesee\nforesee\nforesee\nforesee\nforesee\nforeshadow\nforest-product\nforest-product\nforest\nforfeiture\nforfeit\nforge\nforgery\nforget\nforget\nforget\nforget\nforget\nforge\nforging\nforge\nforgive\nforgive\nforgive\nforget\nforget\nforget\nforgo\nfork\nfork\nforklift\nfork\nformality\nformalize\nformat\nform\nform\nform\nform\nform\nform\nformula\nformulate\nformulate\nformulate\nformulate\nformulation\nform\nform\nforsake\nfortify\nfortune\nforum\nforward\nfossil\nfoster\nfoster\nfoster\nfoster\nfight\nfight\nfoul\nfoundation\nfound\nfound\nfound\nfounder\nfounder\nfounder\nfounder\nfound\nfound\nfind\nfind\nfountain\nfour-fifth\nfox\nfraction\nfraction\nfracture\nfracture\nfragment\nfragment\nfragment\nfrailty\nframe\nframer\nframe\nframe\nframe\nfranchise\nfranchisee\nfranchisee\nfranchiser\nfranchise\nfranchise\nfranchise\nfranchise\nfranciscan\nfranc\nfraternity\nfraud\nfray\nfray\nfreak\nfreak\nfreedom\nfree\nfree\nfree\nfree\nfreeholder\nfree\nfree\nfree\nfree\nfreeway\nfreezer\nfreeze\nfreeze\nfreeze\nfreighter\nfreight\nfrench\nfrequency\nfrequent\nfreshman\nfret\nfret\nfret\nfret\nfret\nfriction\nfriday\nfry\nfriendship\nfriend\nfriend\nfrier\nfrieze\nfrigate\nfrighten\nfrighten\nfrighten\nfringe\nfrippery\nfritter\nfritter\nfrock\nfrogman\nfrog\nfrolic\nfrolic\nfrond\nfront\nfroth\nfreeze\nfreeze\nfruit\nfrustrate\nfrustrate\nfrustrate\nfrustrate\nfrustration\nfry\nfudge\nfudge\nfuel\nfuel\nfuel\nfuel\nfuel-service\nfuel\nfuel\nfugitive\nfujus\nfulfil\nfulfil\nfulfil\nfulfil\nfulfill\nfulfill\nfulmination\nfume\nfume\nfume\nfumper\nfunctionary\nfunction\nfunction\nfunction\nfunction\nfunction\nfundamentalist\nfundamental\nfund\nfund\nfund\nfund\nfund-raiser\nfundraiser\nfund-raiser\nfundraise\nfund\nfund\nfund\nfund\nfund\nfund\nfungus\nfunnel\nfunnel\nfunnel\nfunnel\nfurlough\nfurlough\nfurnace\nfurnish\nfurnishings\nfurnish\nfurnish\nfurrier\nfurrow\nfur\nfurther\nfurther\nfurther\nfuss\nfuture\nfuture\nfuture\ngadget\ngag\ngain\ngain\ngain\ngainer\ngainer\ngain\ngain\ngain\ngain\ngain\ngain\ngalaxy\ngallery\ngallon\ngallop\ngallstone\ngalvanize\ngalvanize\ngalvanize\ngambler\ngamble\ngamble\ngame\ngangbuster\ngang\ngangster\ngape\ngarage\ngardener\ngardenette\ngarden\ngarden\ngarden\ngarment\ngarner\ngarner\ngarner\ngarner\ngas\ngas\ngas-gather\ngasoline\ngasp\ngasp\ngate\ngather\ngather\ngather\ngathering\ngather\ngather\ngather\ngather\ngauge\ngauge\ngauge\ngive\ngive\ngaze\ngear\ngear\ngear\ngear\ngear\ngeek\ngeek\ngemsbok\ngem\ngender\ngeneralist\ngeneralization\ngeneralization\ngeneralize\ngenerate\ngenerate\ngenerate\ngenerate\ngenerate\ngenerate\ngeneration\ngenerator\ngene\ngenre\ngentlelady\ngentleman\ngeoscience\ngerman\ngerman\ngerm\ngesture\ngesture\nget\nget\nget\nget\nget-together\nget\nget\nget\nget\nget\ngeyser\nghetto\nghostbuster\nghostbuster\nghost\ngiant\ngiant\ngift\ngilt\ngilt\ngimmick\ngird\ngird\ngirl\ngiveaway\ngiveaway\ngiveback\ngive\ngive\ngive\ngive\ngiveth\ngive\ngive\ngive\ngive\ngive\ngive\ngizmo\nglamorize\nglamorize\nglance\nglare\nglass\nglaze\nglaze\ngleam\nglean\nglean\nglide\nglide\nglimpse\nglitch\nglitterati\ngloat\ngloater\ngloat\ngloat\ngloat\nglobalist\ngloss\ngloss\nglove\nglow\nglow\nglue\nglue\nglut\nglut\nglut\nglycol\ngnaw\ngoal\ngoat\ngobble\ngobble\ngoblin\ngod\ngo\ngo\ngoings-on\ngo\ngo\ngoldband\ngold\ngolfer\ngolf\ngoliath\ngo\ngo\ngo\ngon\ngon\ngoody\ngoods\ngoodyear\ngore\ngore\ngorilla\ngossip\nget\nget\nget\nget\nget\nget\ngo\ngo\ngo\ngovern\ngovern\ngovern\ngovernmental-affair\ngovernment-relation\ngovernment-security\ngovernment-set\ngovernmentset\ngovernment\ngovernment\ngovernor\ngovernor\ngovern\ngovern\ngovern\ngrab\ngrab\ngrab\ngrab\ngrab\ngrab\ngrace\ngrader\ngrade\ngrad\ngraduate\ngraduate\ngraduate\ngraduate\ngraduate\ngraft\ngrain\ngrain\ngrain\ngrammy\ngram\ngrandchild\ngrandee\ngrandkid\ngrandmaster\ngrandmother\ngrandparent\ngrange\ngrant\ngrant\ngrant\ngrant\ngrant\ngrant\ngrant\ngrape\ngrape\ngraphic\ngraphic\ngraph\ngrapple\ngrapple\ngrapple\ngrasp\ngrasp\ngrasp\ngrassroot\ngratuity\ngray\ngraze\ngrazer\ngraze\ngreat-grandchild\ngreat\ngreenhouse\ngreen\ngreen\ngreet\ngreet\ngreeting\ngreet\ngreet\ngreet\ngrenade\ngrow\ngrow\ngrow\ngridlock\ngrievance\ngrill\ngrimace\ngrimace\ngrimace\ngrinder\ngrind\ngrind\ngrind\ngringo\ngrin\ngrin\ngrin\ngripe\ngripe\ngrip\ngrip\ngrip\ngroan\ngrocery\ngrope\ngross\ngrotto\ngroundbreaker\nground\nground-handling\nground\ngrounds\ngrind\ngrind\ngrind\ngroup\ngroup\ngroup\ngroup\ngrouse\nGrouse\ngrouse\ngrovel\ngrower\ngrow\ngrow\ngrow\ngrowl\ngrow\ngrow\ngrow\ngrowth\ngrowth\ngrow\ngrow\ngruel\ngrumble\ngrumble\nguarantee\nguarantee\nguarantee\nguarantee\nguarantee\nguarantee\nguarantee\nguarantee\nguard\nguard\nguard\nguard\nguard\nguard\nguard\nguber-peter\nguerrilla\nguess\nguess\nguess\nguess\nguest\nguest\nguide\nguide\nguideline\nguideline\nguidepost\nguide\nguide\nguide\nguilder\ngunboat\ngunman\ngunman\ngun\ngunner\ngunsling\ngun\ngun\nguru\nguru\ngush\ngush\ngut\ngut\nguy\nguy\nguzzle\ngymnastics\ngyrate\ngyrate\ngyrate\ngyration\nhabeas\nhabitat\nhabit\nhacker\nhackles\nhack\nhack\nhave\nhave\nhave\nhaggle\nhaggling\nhaggle\nhail\nhail\nhail\nhail\nhalf-state\nhall\nhallway\nhalogenate\nhalt\nhalt\nhalt\nhalt\nhalt\nhalt\nhalt\nhalve\nhalve\nhalf\nhalve\nhalve\nhamburger\nhammer\nhammer\nhammer\nham\nhamper\nhamper\nhamper\nhamper\nhamper\nhamstring\nhamstring\nhamstring\nhandbill\nhandbook\nhand-carry\nhandcuff\nhandcuffs\nhand\nhand\nhandicapped\nhandicap\nhandicap\nhandle\nhandle\nhandler\nhandle\nhandle\nhandle\nhandle\nhandpick\nhand\nhandstand\nhand\nhand\nhang\nhang\nhang\nhang\nhang\nhappen\nhappen\nhappen\nhappen\nhappening\nhappen\nhappen\nhappen\nhappen\nharangue\nharangue\nharass\nharass\nharass\nharbor\nharbor\nharbor\nharbor\nharden\nhardship\nharm\nharm\nharm\nharm\nharm\nharm\nharness\nharp\nharp\nharp\nharry\nharvest\nharvest\nharvest\nhash\nhassle\nhasten\nhasten\nhave\nhave\nhave\nhave\nhave\nhatch\nhatch\nhate\nhate\nhate\nhate\nhat\nhaul\nhauler\nhaul\nhaul\nhaunt\nhaunt\nhaunt\nhaunt\nhaunt\nhaunt\nhaven\nhave\nhave\nhave\nhave\nhave\nhave\nhave\nhave\nhawaius\nhawker\nhawk\nhawk\nhawk\nhazard\nhcfc\nhdtv\nheadache\nhead\nhead\nhead\nhead\nhead\nheadlight\nheadline\nheadline\nheadline\nheadphone\nheadquarters\nheadrest\nheadset\nhead\nhead\nhead\nhead\nheal\nheal\nhealth-product\nheal\nheap\nheap\nheap\nhear\nhear\nhear\nhearing\nhearing\nhear\nhear\nhear\nhear\nhearten\nheart\nhear\nhear\nheat\nheater\nheat\nheat\nheat\nheat\nheave\nheave\nheaven\nheave\nheavyweight\nheckle\nhedger\nhedge\nhedge\nhedge\nhedge\nhedge\nheebie-jeeby\nheed\nheed\nheed\nheel\nheighborhood\nheighten\nheighten\nheighten\nheight\nheir\nhold\nhold\nhelicopter\nhelp\nhelp\nhelp\nhelp\nhelp\nhelp\nhelp\nhelp\nhelp\nhelp\nhelp\nhemorrhage\nhemorrhage\nhemorrhoid\nhen\nhen\nherald\nherald\nherald\nherbicide\nhercule\nherd\nherd\nherniate\nhero\nheron\nhesitate\nhesitate\nhesitate\nhew\nhew\nhew\nhide\nhideout\nhider\nhide\nhide\nhide\nhide\nhighland\nhighlight\nhighlight\nhighlight\nhighlight\nhighlight\nhighlight\nhighlight\nhigh-rise\nhigh\nhightail\nhightop\nhighway\nhighway\nhike\nhiker\nhike\nhill\nhinder\nhinder\nhinder\nhinder\nhinder\nhinge\nhint\nhint\nhinterland\nhint\nhint\nhint\nhint\nhint\nhip\nhire\nhire\nhire\nhire\nhire\nhire\nhire\nhire\nhispanic\nhiss\nhistorian\nhistorical-claim\nhistoricize\nhitch\nhitch\nhit\nhit\nhitter\nhit\nhit\nhit\nhit\nhit\nhit\nhit\nhoard\nhoard\nhoard\nhobby\nhobble\nhobble\nhobble\nhobbyist\nhobo\nhoe\nhog\nhog\nhog\nhoist\nholder\nholder\nholding\nholding\nhold\nhold\nholdout\nholdover\nhold\nholdup\nhold\nhold\nhold\nhold\nhold\nhold\nhole\nhole\nhole\nholiday\nholler\nhomefe\nhomeowner\nhomer\nhomer\nhomer\nhome\nhomicide\nhomosexual\nhomosexual\nhonduran\nhondura\nhone\nhone\nhonorarium\nhonor\nhonor\nhonor\nhonor\nhonor\nhood\nhoodwink\nhook\nhook\nhook\nhook\nhookup\nhook\nhoof\nhope\nhope\nhope\nhope\nhope\nhope\nhope\nhope\nhope\nHope\nhop\nhopscotch\nhop\nhorizon\nhormone\nhorn\nhoroscope\nhorrible\nhorror\nhorse\nhorse\nhorticultural-product\nhosanna\nhose\nhose\nhose\nhospitalization\nhospitalize\nhospital\nhospital\nhostage\nhost\nhostility\nhost\nhost\nhost\nhost\nhot-cereal\nhotel-casino\nhotelier\nhotel\nhotline\nhound\nhound\nhour\nhour\nhouse\nhousehold\nhouse\nhouse\nhouse\nhouse\nhouseware\nhousewife\nhousing\nhover\nhover\nhover\nhowl\nhuckster\nhuddle\nhug\nhug\nhug\nhulk\nhumanities\nhumanize\nhuman-resource\nhuman-rights\nhuman\nhumble\nhum\nhunch\nhundred\nhundred\nhang\nhang\nhunker\nhunt\nhunter-gatherer\nhunter\nhunt\nhunt\nhunt\nhunt\nhurdle\nhurl\nhurl\nhurl\nhurricane\nhurry\nhurry\nhurry\nhurt\nhurtle\nhurt\nhurt\nhurt\nhurt\nhurt\nhurt\nhusband\nhusband\nhustler\nhustle\nhybrid\nhyena\nhype\nhypermarket\nhype\nhypnotize\nhypocrite\nhypothesize\nice-bagger\nideal\nidea\nidea\nidentify\nidentify\nidentify\nidentify\nidentify\nidentify\nidentity\nideology\nideologue\nideologue\nidiot\nidle\nidle\nidle\nidle\nignite\nignite\nignite\nignore\nignore\nignore\nignore\nignore\nignore\nignore\nIgnore\nillegality\nillness\nill\nilluminate\nilluminate\nillusion\nillustrate\nillustrate\nillustrate\nillustrate\nillustrate\nillustration\nimage\nimagine\nimagine\nimagine\nimagine\nimagine\nimagine\nimagine\nimbalance\nimitate\nimitate\nimitate\nimmerse\nimmigrant\nimmigrate\nimmunity\nimpact\nimpact\nimpact\nimpair\nimpair\nimpart\nimpeach\nimpede\nimpede\nimpede\nimpediment\nimpede\nimpel\nimpend\nimperative\nimperfection\nimperialist\nimperil\nimpersonation\nimplant\nimplant\nimplant\nimplant\nimplement\nimplement\nimplement\nimplement\nimplement\nimplicate\nimplicate\nimplication\nimply\nimply\nimply\nimplore\nimplore\nimply\nimply\nimply\nimport\nimport\nimport\nimporter\nimport\nimport\nimport\nimport\nimport\nimport\nimpose\nimpose\nimpose\nimpose\nimpose\nimpose\nimpound\nimpound\nimpoverish\nimpress\nimpress\nimpress\nimpressionist\nimpress\nimprison\nimprison\nimprison\nimprison\nimpropriety\nimprove\nimprove\nimprovement\nimprove\nimprove\nimprove\nimprove\nimprove\nimprovise\nimpugn\nimpulse\ninaugurate\nincarcerate\nincense\nincentive\ninch\ninch\ninch\ninch\ninch\nincident\nincision\nincite\nincline\ninclude\ninclude\ninclude\ninclude\ninclude\ninclude\ninclude\ninclude\ninclude\nincome\nincongruity\ninconsistency\nincorporate\nincorporate\nincorporate\nincorporate\nincorporate\nincrease\nincrease\nincrease\nincrease\nincrease\nincrease\nincrease\nincrease\nincrease\nincrease\nincrease\nincrease\nincrease\nincrement\nincriminate\nincumbent\nincur\nincur\nincur\nincur\nincur\nindemnify\nindependent\nindex\nindexer\nindex\nindex\nindex-future\nindex\nindex-option\nindicate\nindicate\nindicate\nindicate\nindicate\nindicate\nindication\nindicator\nindex\nindict\nindict\nindictment\nindict\nindividual\nindividual\nindoctrinate\ninduce\ninduce\ninducement\ninduce\ninduce\ninduce\nindulgence\nindulge\nindulge\nindulge\nindustrialist\nindustrialize\nindustrialize\nindustrial\nindustrial\nindustry\nindustry\ninefficiency\ninequality\ninequity\ninfant\ninfant\ninfect\ninfect\ninfect\ninfection\ninference\ninfer\ninfiltrate\ninfiltrate\ninfiltrate\ninflame\ninflate\ninflate\ninflate\ninflate\ninflate\ninflation-adjust\ninflict\ninflict\ninflict\ninflow\ninfluence\ninfluence\ninfluence\ninfluence\ninfluence\ninfluence\ninfluence\ninformation-service\ninformation-system\ninform\ninform\ninform\ninform\ninform\ninfraction\ninfringe\ninfringe\ninfringe\ninfringe\ninfringe\ninfuriate\ninfuriate\ninfuse\ninfuse\ningest\ningot\ningrate\ningratiate\ningredient\ninhabit\ninhabit\ninhabit\ninherit\ninherit\ninherit\ninherit\ninhibit\ninhibit\ninhibit\ninitial\ninitial\ninitial\ninitiate\ninitiate\ninitiate\ninitiate\ninitiate\ninitiatior\ninitiative\ninitiative\ninject\ninject\ninject\ninjection\ninject\ninject\ninject\ninjunction\ninjure\ninjure\ninjure\ninjury\ninjure\ninjustice\nink\ninmate\ninning\ninnocent\ninnoculate\ninnovate\ninnovate\ninnovation\ninnovator\ni\ninn\ninnuendo\ninput\ninquire\ninquiry\ninquire\ninroad\ninsect\ninsert\ninsert\ninsert\ninsert\ninsert\ninsider\ninsider\ninsight\ninsinuate\ninsist\ninsist\ninsist\ninsist\ninsist\ninsist\ninsist\ninspect\ninspect\ninspect\ninspection\ninspector\ninspector\ninspect\ninspect\ninspect\ninspiration\ninspire\ninspire\ninspire\ninspire\ninspire\ninstallation\ninstall\ninstall\ninstall\ninstallment\ninstall\ninstall\ninstance\ninstigate\ninstill\ninstinct\ninstitute\ninstitute\ninstitute\ninstitute\ninstitution\ninstitution\ninstruct\ninstruct\ninstruct\ninstruction\ninstructor\ninstruct\ninstruct\ninstrumentalist\ninstrument\ninstrument\ninsulate\ninsulate\ninsulate\ninsulate\ninsulin\ninsult\ninsult\ninsurance-claim\ninsure\ninsure\ninsurere\ninsurer\ninsurer\ninsurer\ninsure\ninsure\ninsure\ninsurgent\ninsure\nintegrated-technology\nintegrate\nintegrate\nintegrate\nintellectual\nintend\nintend\nintend\nintend\nintend\nintensify\nintensify\nintensify\nintensify\nintention\nintent\ninteraction\nintercede\nintercept\nintercept\ninterconnect\ninterconnect\ninterest\ninterest\ninterest\ninterest\ninterfere\ninterfere\ninterfere\ninterfere\ninterfere\ninterior-furnishings\ninterior\ninterject\ninterlope\nintermediary\nintermix\ninternationalist\ninternational-operation\ninterpretation\ninterpret\ninterpret\ninterpret\ninterpret\ninterpret\ninterpret\ninterrogate\ninterrogator\ninterrupt\ninterrupt\ninterruption\ninterrupt\nintersection\nintersperse\nintersperse\ninterstate\nintertitle\nintertwine\nintertwine\ninterval\nintervene\nintervene\nintervene\nintervene\ninterventionist\nintervention\ninterview\ninterview\ninterview\ninterview\ninterview\ninterview\nintimate\nintimidate\nintimidate\nintimidate\nintimidation\nintone\nintrigue\nintrigue\nintroduce\nintroduce\nintroduce\nintroduce\nintroduce\nintroduce\nintroduce\nintroduction\nintrude\nintrusion\ninundate\ninvade\ninvader\ninvade\ninvade\ninvade\ninvade\ninvalidate\ninvent\ninvent\ninvent\ninvention\ninventory\ninventory\ninventor\ninvent\ninvent\ninvert\ninvest\ninvest\ninvestigate\ninvestigate\ninvestigate\ninvestigate\ninvestigate\ninvestigate\ninvestigation\ninvestigator\ninvestigator\ninvest\ninvest\ninvest\ninvestment\ninvestor-relation\ninvestor\ninvestor\ninvest\ninvest\ninvest\ninvitation\ninvite\ninvite\ninvite\ninvite\ninvite\ninvoice\ninvoke\ninvoke\ninvoke\ninvoke\ninvoke\ninvolve\ninvolve\ninvolve\ninvolve\ninvolve\ninvolve\ninvolve\ninvolve\nious\nipo\nira\niris\nirk\nirk\nirk\nirk\nirony\niron\niron\nirradiate\nirregularity\nirritate\nirritate\nisland\nisland\nis\nisolate\nisolate\nisolate\nisolate\nisolate\nissue\nissue\nissuer\nissue\nissue\nissue\nissue\nissue\nissue\nissue\nbe\nbe\nbe\nitemize\nitem\nitem\njab\njackal\njack\njack\njacket\njacket\njackhammer\njack\njack\njail\njail\njail\njail\njam\njam\njam\njam\njan.\njapanese-american\njapanese\njapanese\njar\njar\njaunt\njaunt\njeans\njeep\njell\njeopardize\njeopardize\njeopardize\njeopardize\njerk\njetliner\njet\njet\njettison\njet\njeweler\njewel\njew\njiggle\njillion\njingle\njinks\njinx\njitters\njob\njob\njockey\njock\njog\njoin\njoin\njoin\njoin\njoin\njoin\njoint-implant\njoint\njoin\njoin\njoin\njoke\njoke\njoke\njoke\njolt\njolt\njolt\njolt\njolt\njostle\njournalist\njournalist\njournal\njoust\njoy\njudge\njudgeship\njudge\njudge\njudge\njudge\njudge\njudge\njudge\njudgment\njuggler\njuggle\njuggle\njug\njuice\njumbo\njump\njump\njump\njump\njump\njump\njump\njump\njuncture\njunior\njunk-bond\njunket\njunk-holder\njunkholder\njunk-holder\njunkie\njunk\njury\njurisdiction\njurist\njuror\njury-rig\njustice\njustify\njustify\njustify\njustify\njut\njut\njuxtapose\nkayo\nkc-135\nkeen\nkeeper\nkeep\nkeep\nkeep\nkeep\nkeep\nkeep\nkeep\nkeep\nketchup\nkeyboard\nkey\nkey\nkey\nkickback\nkick\nkick\nkicker\nkick\nkick\nkick\nkick\nkick\nkiddie\nkid\nkidnap\nkidnap\nkidnapper\nkidnap\nkid\nkid\nkill\nkill\nkiller\nkilling\nkill\nkill\nkill\nkill\nkill\nkilobyte\nkilogram\nkilometer\nkindle\nkind\nkinfolk\nkingpin\nking\nkiss\nkiss\nkit\nknead\nknead\nknee\nknow\nknight\nknit\nknit\nknit\nknock\nknock\nknock\nknock\nknock\nknot\nknow\nknow\nknown\nknow\nknow\nknow\nknow\nknow\nknow\nknuckle\nkowtow\nkrona\nkudos\nlabel\nlabel\nlabel\nLabel\nlabel\nlabel\nlabel\nlabel\nlaboratory\nlaborer\nlabor\nlabor\nlabor\nlab\nlace\nlack\nlack\nlackey\nlack\nlack\nlack\nlady\nlaggard\nlag\nlag\nlag\nlagoon\nlag\nlag\nlag\nlag\nlag\nlay\nlay\nlake\nlambast\nlambaste\nlament\nlament\nlaminate\nlamppost\nlamp\nland\nland\nlandfill\nlandholding\nlanding\nland\nlandlord\nlandowner\nlandowner\nlandscaper\nlandscape\nlandslide\nland\nland\nland\nlane\nlanguage\nlanguish\nlanguish\nlanguish\nlanguish\nlanguish\nlapse\nlapse\nlapse\nlap\nlaptop\nlaptop\nlap\nlaser\nlash\nlash\nlash\nlash\nlast\nlast\nlast\nlast\nlast\nlast\nlatch\nlatch\nlatch\nlatch\nlathe\nlaud\nlaud\nlaugh\nlaugh\nlaugh\nlaugh\nlaugh\nlaugh\nlaunch\nlaunch\nlaunch\nlaunch\nlaunch\nlaunch\nlaunder\nlaunder\nlaunder\nlaunderer\nlaunder\nlaunder\nlaurel\nlavish\nlavish\nlawbreaker\nlawmaker\nlawmaker\nlawmaker\nlaw-making\nlawn\nlaw\nlawsuit\nlawsuit\nlawyer\nlawyer\nlawyer\nlaxative\nlayer\nlay\nlayoff\nlayoff\nlay\nlay\nlay\nlay\nlbo\nleader\nleader\nlead\nlead\nlead\nlead\nlead\nlead\nlead\nlead\nleaf\nleaflet\nleaguer\nleague\nleak\nleak\nleaker\nleak\nleak\nlean\nlean\nlean\nlean\nlean\nlean\nlean\nleap\nleap\nleapfrog\nleap\nleap\nleap\nleap\nleap\nleap\nlearn\nlearn\nlearn\nlearn\nlearn\nlearn\nlearn\nlease\nlease\nlease\nlease\nlease\nlease\nlease\nlease\nleather\nleaf\nleave\nleave\nleave\nleave\nleave\nleave\nlecture\nlecture\nlecture\nlecture\nledger\nlead\nlead\nlead\nleech\nleftist\nleftovers\nleave\nleave\nleave\nleave\nlegalize\nlegal-service\nlegion\nlegislate\nlegislate\nlegislator\nlegislator\nlegislature\nlegitimize\nlegitimize\nleg\nlemming\nlemon\nlender\nlender\nlend\nlend\nlend\nlend\nlend\nlengthen\nlengthen\nlengthen\nlengthen\nlengthen\nlength\nlens\nlens\nlend\nlend\nleotard\nlesbian\nlesion\nlessen\nlessen\nlesser\nlesson\nlet\nletter\nletter\nlet\nlet\nlet\nlet\nlet\nlet\nlet\nlet\nlevel\nlevel\nlevel\nlevel\nlevel\nlevel\nleverage\nleverage\nleverage\nleverage\nleverage\nlevy\nlevy\nliability\nliaison\nliaison\nliar\nlibel\nliberalization\nliberalize\nliberalize\nliberalize\nliberalize\nliberalize\nliberal\nliberal\nliberate\nlibertarian\nliberty\nlibrarian\nlibrary\nlouse\nlicense\nlicense\nlicens\nlicense\nlicense\nlicense\nlicense\nlick\nlid\nlie\nlie\nlie\nlie\nlieutenant\nlie\nlie\nlifeguard\nlife\nlift\nlift\nlift\nlift\nlift\nlift\nlift\nlift\nlight\nlighten\nlighten\nlighten\nlight\nlight\nlight\nlight\nlike\nlike\nliken\nliken\nliken\nlikes\nlike\nlike\nlike\nlimb\nlimitation\nlimit\nlimit\nlimit\nlimit\nlimit\nlimit\nlimit\nlimit\nlimousine\nlimp\nlinebacker\nline\nline\nliner\nline\nline\nlineup\nline\nline\nlinger\nlinger\nlinger\nlinger\nline\nlinkage\nlink\nlink\nlink\nlink\nlink\nlink\nlink\nlink\nlink\nlion\nlipoprotein\nlip\nlipstick\nliquefy\nliquefy\nliquefy\nliquefy\nliquidate\nliquidate\nliquidate\nliquidate\nliquidate\nliquid\nlira\nlist\nlist\nlisten\nlisten\nlistener\nlistener\nlisten\nlisten\nlisten\nlisten\nlisten\nlisting\nlist\nlist\nlist\nlist\nlist\nlithograph\nlitigant\nlitigator\nlitter\nlitter\nlight\nlight\nlive\nlive\nlive-haul\nlive-hauler\nlife\nlife\nlive\nlive\nlive\nlive\nlive\nlive\nlive\nlive\nload\nload\nloading\nload\nload\nload\nloafer\nloan\nloan\nloan\nloan\nloan\nloan\nloathe\nloathe\nloathe\nloaf\nlobby\nlobby\nlobby\nlobby\nlobbyist\nlobby\nlocale\nlocality\nlocalize\nlocal\nlocate\nlocate\nlocate\nlocate\nlocate\nlocation\nlock\nlock\nlock\nlock\nlock\nlock\nlock\nlocution\nlodge\nlodge\nlodge\nlodgings\nlog\nlog\nlogger\nlog\nlogistics\nlogo\nlog-roll\nlog\nlog\nlog\nlog\nlong-term\nlong\nlook\nlook\nlookee-loo\nlook\nlook\nlook\nlook\nlook\nlook\nlook\nloom\nloom\nloom\nloom\nloom\nloom\nloony\nloophole\nloop\nloosen\nloosen\nloosen\nloose\nloot\nloot\nlop\nlord\nlord\nloser\nlose\nlose\nlose\nlose\nlose\nloss\nloss\nloss\nlose\nlose\nlose\nlotion\nlot\nlot\nlottery\nloudspeaker\nloui\nlounge\nlovebird\nlove\nlove\nlover\nlove\nlove\nlove\nlove\nlower\nlower\nlower\nlower\nlower\nlower\nlow-life\nlow\nloyalty\nlubricant\nluck\nlug\nlug\nlug\nlull\nlumber\nluminary\nlump\nlump\nlump\nlump\nlunch\nlunge\nlunge\nlung\nlurch\nlurch\nlurch\nlure\nlure\nlure\nlure\nlure\nlure\nlure\nlurk\nluxury\nlie\nlyric\nlyric\nmachete\nmachiguenga\nmachine\nmachine\nmachinist\nmachinist\nmachinist\nmake\nmake\nmake\nmafia\nmafioso\nmagazine\nmagazine\nmaggot\nmagician\nmagistrate\nmagnetize\nmagnet\nmagnify\nmagnify\nmagnify\nmagnolia\nmaharaja\nmail\nmail\nmailer\nmailing\nmail\nmailman\nmail-sort\nmail\nmail\nmainframe\nmain\nmaintain\nmaintain\nmaintain\nmaintain\nmaintain\nmaintain\nmaintain\nmajor\nmajor\nmaker\nmaker\nmake\nmake\nmake\nmake\nmake\nmake\nmake\nmake\nmake\nmake\nmakin\nmalefactor\nmale\nmalfunction\nmalign\nmall\nmammoth\nmanacle\nmanage\nmanage\nmanage\nmanagement\nmanager\nmanager\nmanage\nmanage\nmanage\nmanage\nmanage\nmandate\nmandate\nmandate\nmandate\nmandate\nmandate\nmaneuver\nmaneuver\nmaneuvering\nmaneuver\nmaneuver\nmaneuver\nmaneuver\nmaneuver\nmanhandle\nmanifestation\nmanifesto\nmanifest\nmaninstay\nmanipulate\nmanipulate\nmanipulate\nmanipulate\nmanipulate\nmanipulation\nmanipulator\nman\nmannerism\nmanners\nman\nmansion\nmanual\nmanual\nmanuever\nmanufacture\nmanufacture\nmanufacturer\nmanufacturer\nmanufacture\nmanufacture\nmanufacture\nmanufacture\nman\nmany\nmany\nmap\nmap\nmap\nmap\nmap\nmarathon\nmarble\nmarch\nmarch\nmarcher\nmarch\nmarch\nmarch\nmarch\nmargeote\nmarginalia\nmarginalize\nmargin-call\nmargin\nmargin\nmargin\nmargin\nmarkdown\nmark\nmark\nmarket\nmarket\nmarketeer\nmarketer\nmarketer\nmarketing-communication\nmarket\nmarket-maker\nmarketplace\nmarket\nmarket\nmarket\nmarket\nmarket\nmark\nmarkka\nmark\nmark\nmark\nmark\nmark\nmarquee\nmar\nmarriage\nmarry\nmarry\nmarry\nmarry\nmarsh\nmarvel\nmarvel\nmarvel\nmask\nmask\nmask\nmask\nmason\nmasquerade\nmassacre\nmassage\nmassage\nmassage\nmass\nmass\nmasseur\nmasseuse\nmass-media\nmass-produce\nmaster\nmasterpiece\nmaster\nmaster\nmatch\nmatch\nmatch\nmatch\nmatch\nmatch\nmatch\nmate\nmaterialize\nmaterialize\nmaterialize\nmaterialize\nmaterialize\nmaterial\nmaterial\nmate\nmathematics\nmate\nmatter\nmatter\nmatter\nmatter\nmatter\nmatter\nmatter\nmature\nmature\nmature\nmature\nmature\nmaturity\nmaturity\nmaul\nmaven\nmaximize\nmaximize\nmaxim\nmayor\nmaze\nmcdonald\nmeadow\nmeal\nmeander\nmeander\nmeaning\nmean\nmeans\nmeans\nmean\nmean\nmean\nmean\nmean\nmean\nmean\nmeasure\nmeasure\nmeasure\nmeasurement\nmeasure\nmeasure\nmeasure\nmeasure\nmeasure\nmeat\nmeat\nmechanic\nmechanism\nmedallion\nmeddle\nmeddle\nmedia\nmediate\nmediator\nmedical-product\nmedicine\nmeeting\nmeeting\nmeet\nmeet\nmeet\nmeet\nmeet\nmeet\nmeet\nmegabyte\nmega-crash\nmega-hit\nmega-issue\nmega-merger\nmega-problem\nmega-project\nmegaquestion\nmega-resort\nmegawatt\nmeld\nmeld\nmeld\nmellow\nmelody\nmelt\nmelt\nmelt\nmembership\nmember\nmember\nmemento\nmemoirs\nmemorabilia\nmemorandum\nmemorandum\nmemorialize\nmemory\nmemorize\nmemo\nmemo\nmend\nman\nman\nmention\nmention\nmention\nmention\nmention\nmention\nmentor\nmenu\nmercantilist\nmercedes-benz\nmercedes\nmerchandise\nmerchandiser\nmerchandise\nmerchant\nmerchant\nmerge\nmerge\nmerger\nmerge\nmerge\nmerge\nmeringue\nmerit\nmerit\nmerit\nmesh\nmessage\nmessenger\nmess\nmessr.\nmess\nmetabolize\nmetal\nmetal\nmetal\nmetal-worker\nmetalworker\nmetaphor\nmete\nmeter\nmethodology\nmethod\nmete\nmetric\nmeet\nmeet\nmouse\nmouse\nmicoprocessor\nmicrobe\nmicrochip\nmicrocomputer\nmicroeconomic\nmicroelectronics\nmicrophone\nmicroprocessor\nmicrowave\nmid-1940\nmid-1960\nmid-1970\nmid-1980\nmid-1990\nmiddleman\nmidsize\nmiff\nmiff\nmig-29\nmighta\nmigrate\nmigration\nmile\nmilestone\nmilitary-electronics\nmilitate\nmilitia\nmilk\nmilk\nmilk\nmill\nmillionaire\nmillion-plus\nmillion\nmillion\nmill\nmimic\nmimic\nmimic\nmimic\nmince\nmind\nmind\nmind\nmind\nminefield\nmineral\nminer\nmine\nmine\nmingle\nminiaturize\nminicar\nminicomputer\nminimill\nminimize\nminimize\nminimize\nminimum\nmine\nminiseries\nminister\nministry\nminisupercomputer\nminivan\nminneapoli\nminority\nminor\nmint\nmint\nmint\nminus\nminus\nminute\nminutiae\nmips\nmips\nmire\nmirror\nmirror\nmirror\nmirror\nmisadventure\nmiscalculate\nmiscalculate\nmiscarriage\nmisclassify\nmiscreant\nmisdeed\nmisdemeanor\nmisfortune\nmisguide\nmishandle\nmishandle\nmisinterpret\nmisinterpret\nmisjudge\nmisjudgment\nmislay\nmislead\nmislead\nmislead\nmismeasurement\nmisperception\nmisplace\nmisquote\nmisrepresentation\nmisrepresent\nmisrepresent\nmisrepresent\nmisrepresent\nmisroute\nmiss\nmiss\nmiss\nmissile\nmiss\nmissionary\nmission\nmisspend\nmisstate\nmisstate\nmisstatement\nmisstate\nmiss\nmiss\nmistake\nmistake\nmistake\nmistreat\nmistress\nmistrial\nmist\nmisunderstanding\nmisunderstand\nmisunderstand\nmisuse\nmite\nmitigate\nmitigate\nmitsubishus\nmitsuus\nmix\nmix\nmix\nmixer\nmix\nmix\nmixture\nmix\nmoan\nmoan\nmoan\nmobilize\nmobilize\nmobilize\nmobilize\nmock\nmock\nmodel\nmodel\nmodel\nmodel\nmodel\nmodem\nmoderate\nmoderate\nmoderate\nmoderate\nmoderate\nmodernize\nmodernize\nmodernize\nmodernize\nmode\nmodification\nmodification\nmodify\nmodify\nmodify\nmodulate\nmoisturizer\nmold\nmold\nmolecule\nmollify\nmollify\nmoment\nmonday\nmonetarist\nmoneymaker\nmonitor\nmonitor\nmonitor\nmonitor\nmonitor\nmonitor\nmonitor\nmonitor\nmonkey\nmonolith\nmonologue\nmonopoly\nmonopolize\nmonopolize\nmonopolize\nmonopolize\nmonth\nmonth\nmoonie\nmoonlight\nmoon\nmop\nmop\nmoral\nmores\nmorning\nmorsel\nmortgage\nmortgage\nmortgage\nmoslem\nmotel\nmother\nmother\nmotif\nmotion\nmotion\nmotivate\nmotivate\nmotivate\nmotivate\nmotive\nmotorcycle\nmotorist\nmotorize\nmotor\nmound\nmountain\nmount\nmount\nmount\nmount\nmount\nmount\nmourn\nmourn\nmousetrap\nmouth\nmouth\nmove\nmove\nmove\nmovement\nmove\nmove\nmove\nmove\nmove\nmove\nmove\nmove\nmovie\nmove\nmove\nmow\nmuck\nmuddy\nmuddle\nmuffle\nmuff\nmull\nmull\nmull\nmultimedia\nmultinational\nmultiple\nmultiple\nmultiply\nmultiply\nmultiply\nmultiply\nmumble\nmummy\nmunch\nmunicipality\nmunicipal\nmunicipal\nmunicipal\nmuni\nmuni\nmural\nmurder\nmurderer\nmurder\nmurder\nmurmur\nmuscle\nmuscle\nmuscle\nmuscovite\nmuse\nmuse\nmuseum\nmuseum\nmuse\nmushroom\nmushroom\nmushroom\nmusician\nmuster\nmutate\nmutate\nmutation\nmute\nmute\nmutilate\nmutiny\nmutter\nmutt\nmuzzle\nbe\nmystery\nmyth\nnab\nnagging\nnag\nnag\nnail\nnail\nnail\nname-dropper\nname-drop\nname\nname\nname\nnameplate\nname\nname\nname\nname\nname\nname\nnarcotic\nnarrative\nnarrow\nnarrow\nnarrow\nnarrow\nnarrow\nnarrow\nnationalist\nnationalize\nnationalize\nnational\nnation\nnative\nnatural-food\nnatural-resource\nnavy\nnavigate\nnaysayer\nnaysay\nnazi\nneanderthal\nnear\nnear\nnear-monopoly\nnear\nnecessitate\nnecessitate\nnecessity\nneck\nnecktie\nneed\nneed\nneed\nneed\nneed\nneed\nneed\nneed\nneed\nnegate\nnegative\nneglect\nneglect\nneglect\nnegotiate\nnegotiate\nnegotiate\nnegotiate\nnegotiate\nnegotiate\nnegotiation\nnegotiation\nnegotiator\nnegotiator\nneighbhorhood\nneighborhood\nneighbor\nneighbor\nneighbour\nneophyte\nnerd\nnerd\nnerve\nnestle\nnest\nnet\nnet\nnet\nnet\nnet\nnet\nnetwork\nnetwork\nnetwork\nnetwork\nneurologist\nneutralize\nneutralize\nneutron\nnewborn\nnewcast\nnewcomer\nnewscast\nnewsy\nnewsletter\nnews\nnewspaper\nnewspaper\nnewsprint\nnewsstand\nnewsstand\nnews-weekly\nnibble\nniche\nnichola\nnick\nnickname\nnickname\nnightclub\nnightmare\nnight\nnine-month\nnine-tenth\nnix\nnobleman\nnod\nno-load\nnominate\nnominate\nnominate\nnomination\nnominee\nnon-client\nnon-communist\nnonconformist\nnon-economist\nnon-lawyer\nnon-lawyer\nnonoperate\nnonperformer\nnonperform\nnonperform\nnonrecurr\nnon-seaman\nnonstop\nnoodle\nnormalize\nnorm\nnorm\nnorthrop\nnose-dive\nnosedive\nnose-dive\nnosedive\nnose\nno.\nnotch\nnotch\nnotebook\nnote\nnote\nnoteholder\nnote\nnote\nnote\nnote\nnote\nnote\nnote\nnote\nnotice\nnotice\nnotice\nnotice\nnotice\nnotice\nnotification\nnotify\nnotify\nnotify\nnotify\nnotify\nnotify\nnote\nnote\nnotion\nno.\nnovel\nnovelty\nnovitiate\nnov.\nnozzle\nnuance\nnuclear-arm\nnudge\nnullify\nnullify\nnullify\nnumber-cruncher\nnumber\nnumber\nnumber\nnumber\nnurse\nnursery\nnurse\nnurse\nnurture\nnurture\nnurture\nnurture\nnut\noak\noasis\noats\nobey\nobfuscate\nobject\nobject\nobject\nobjection\nobjection\nobjective\nobject\nobject\nobject\nobligate\nobligate\nobligation\nobligation\noblige\noblige\noblige\nobliterate\nobscure\nobscure\nobscure\nobservation\nobserve\nobserve\nobserver\nobserver\nobserve\nobserve\nobserve\nobserve\nobserve\nobsess\nobsolete\nobsolete\nobstacle\nobstruct\nobstruct\nobstruct\nobtain\nobtain\nobtain\nobtain\nobtain\nobviate\noccasion\noccasion\noccupation\noccupy\noccupy\noccupy\noccupy\noccupy\noccur\noccur\noccurrence\noccur\noccur\noccur\noccur\nocean\noctave\noctogenarian\noddball\noddity\noddity\nodds\nodds\noffender\noffend\noffend\noffend\noffensive\noffer\noffer\noffer\noffer\noffering\noffer\noffer\noffer\noffer\noffer\noffer\noffer\noff-hour\noffical\nofficer\nofficer\noffice\noffice\noffice-supplies\nofficial\nofficial\nofficial\noffus\noffset\noffset\noffset\noffset\noffset\noffset\noffset\noffshoot\noffspring\nogle\nogle\noils\noils\nold\nold-timer\nolefin\nolympics\nomen\nomission\nomit\nomit\nomit\nomit\nomit\noncogene\noncogene\none\nonlooker\nonlooker\non-ramp\nooze\nooze\nopen\nopen\nopenend\nopener\nopening\nopen\nopen\nopen\nopen\nopen\nopera\noperate\noperate\noperate\noperate\noperate\noperate\noperate\noperate\noperation\noperative\noperator\nopine\nopinion-maker\nopinion\nopinion\nopponent\nopponent\nopportunist\nopportunity\noppose\noppose\noppose\noppose\noppose\noppose\noppose\nopt\nopt\noptical-product\noptimist\nopt\noption\noption\noption\noption\nopt\nopt\norange\norchardist\norchard\norchestra\norchestrate\norchestrate\norchestrate\norchid\nordain\norder\norder\norder\norder\norder\norder\norder\norder\nordinance\norganism\norganization\norganization\norganize\norganize\norganize\norganizer\norganize\norganize\norganize\norgan\norient\noriginate\noriginate\noriginate\noriginate\noriginate\norigination\noriginator\norigin\norkem\nornament\norphan\norphan\nother\nother\nounce\noust\noust\noust\noust\noust\noutage\noutbid\noutbid\noutbid\noutbreak\noutcome\noutdate\noutdo\noutdistance\noutdo\noutfielder\noutfit\noutfit\noutflank\noutflow\noutflow\noutfly\noutgain\noutgrow\noutgrow\nouting\noutlander\noutlast\noutlast\noutlaw\noutlaw\noutlaw\noutlay\noutlay\noutleap\noutlet\noutline\noutline\noutline\noutline\noutline\noutlook\noutmode\noutnumber\nout-of-stater\noutpace\noutpace\noutpace\noutpace\noutpace\noutpace\noutperform\noutperform\noutperform\noutperform\noutperform\noutperform\noutpost\noutrage\noutrank\noutsell\noutsell\noutsell\noutshine\noutshine\noutsider\noutskirts\nout-smart\noutsell\noutstrip\noutstrip\noutstrip\noutstrip\noutstrip\nout-trade\noutweigh\noutweigh\noutweigh\noutweigh\noven\nover-allotment\noveralls\noverarch\noverbid\noverbid\noverbought\noverburden\novercome\novercharge\novercollateralize\novercome\novercome\novercome\novercome\novercommit\noverdo\noverdo\noverdose\noverdose\noveremphasize\noverflow\noverhang\noverhaul\noverhaul\noverhaul\noverheate\noverheate\noverlay\noverlap\noverlap\noverlap\noverlay\noverlook\noverlook\noverlook\noverlook\noverlook\noverlook\nover-magazine\noverpay\noverpay\noverpay\noverpay\noverplant\noverpower\noverprice\noverpurchase\noverreact\noverreact\noverreact\noverreact\noverreact\noverride\noverride\noverride\noverrule\noverrule\noverrule\noverrule\noverrun\noversee\noversee\noversee\noversee\noverseer\noversee\noversee\novershadow\novershadow\novershadow\noversimplify\noversell\noversell\noverstate\noverstate\noverstate\noverstate\noverstrain\noversubscribe\novertake\novertax\noverthrow\noverthrow\noverthrow\noverture\noverturn\noverturn\noverturn\noverturn\noveruse\novervalue\novervalue\noverweight\noverwhelm\noverwhelm\noverwhelm\noverwhelm\noverwhelm\noverwork\nowe\nowe\nowe\nowe\nowe\nowe\nown\nown\nowner\nowner\nown\nown\nown\nown\nown\nown\npace\npacemaker\npace\npacify\npackaged-goods\npackaged-goods\npackage\npackage\npackage\npackage\npackage\npackage\npack\npack\npacket\npack\npack\npack\npack\npack\npac\npac\npact\npaddle\npaean\npage\npay\npay\npay\npay\npainewebber\npain\npaint\npaint\npainter\npainting\npainting\npaint\npaint\npaint\npaint\npair\npair\npair\npalazzo\npale\npale\npale\npale\npalm\npalmtop\npalmtop\npalm\npal\npamper\npamphlet\npancake\npander\npanelist\npanel\npang\npanic\npanic\npanic\npanic\npanjandrum\npan\npan\npan\npantage\npanties\npants\npan\npaper-goods\npaper\npaper-product\npapers\npapers\nparachute\nparachute\nparade\nparakeet\nparallel\nparallel\nparallel\nparalyze\nparalyze\nparamedic\nparameter\nparaphernalia\nparaphrase\nparasite\nparastatal\nparcel\nparcel\nparcel\nparch\npardon\npare\npare\nparent\nparent\npare\npariba\nparimutuel\npare\nparish\nparishioner\nparity\npark\npark\npark\npark\npark\npark\nparlor\nparry\nparry\npartake\npart\nparticipant\nparticipant\nparticipate\nparticipate\nparticipate\nparticipate\nparticipate\nparticipate\nparticipation\nparticulars\nparty\nparty\nparty\npart\npartisan\npartisan\npartnership\npartnership\npartner\npartner\npartner\npart\npart\npart\npasha\npassage\npass\npass\npass\npassenger-kilometer\npassenger\npasser-by\npass\npass\npass\npassion\npassport\npass\npass\npassword\npaste\npastel\npasteurize\npastime\npastor\npatch\npatch\npatent\npatent\npatent\npath\npatient\npatient\npatrol\npatrol\npatronize\npatronize\npatronize\npatron\npattern\npattern\npattern\npause\npause\npause\npause\npause\npause\npave\npave\npave\npave\npaw\npawn\npawn\npayable\npaycheck\npayer\npayer\npay\npay\npayment\npayment\npayment\npayoff\npayout\npayout\npayroll\npay\npay\npay\npay\npay\npay\npcb\npc\npeacemaker\npeach\npeak\npeak\npeak\npeak\npeak\npeal\npeal\npeanut\npearl\npear\npeasant\npeasant\npea\npeccadillo\npeck\npeck\npeculiarity\npedal\npedal\npedal\npeddle\npeddle\npeddle\npeddle\npeddle\npedestrian\npedigree\npeek\npeel\npeer\npeer\npeer\npeer\npeer\npeg\npeg\npeg\npeg\npeg\npeg\npellet\npenalize\npenalize\npenalize\npenalty\npenny\npencil\npencil\npencil\npencil\npend\npend\npend\npenetrate\npenetrate\npenetrate\npenis\npen\npenny\npension\npen\npeople\npeople\npeople\npeople\npepper\npepper\npepper\npepsi\nperceive\nperceive\nperceive\nperceive\npercentage\nperception\nperch\nperfect\nperformance\nperform\nperform\nperformer\nperforming-art\nperform\nperform\nperform\nperform\nperform\nperil\nperiodical\nperiod\nperiod\nperipheral\nperishable\nperish\nperk\nperk\nperk\npermeate\npermeate\npermit\npermit\npermit\npermit\npermit\npermit\npermit\nperpetrate\nperpetuate\nperpetuate\nperpetuate\nperpetuate\npersecute\npersecute\npersist\npersist\npersist\npersist\npersist\npersonality\npersonalize\npersonalize\npersonnel\npersonnel\nperson\npersuade\npersuade\npersuade\npersuade\npersuade\npersuade\npertain\nperturb\nperuse\nperuse\npervade\npervade\nperversity\npesata\npeseta\npeso\npessimist\npester\npesticide\npetition\npetition\npetition\npetrochemical\npet\npet\npharaoh\npharmaceutical\npharmaceutical\npharmacy\npharmacist\nphase\nphase\nphase\nphase\nphase\nphenomenon\nphillip\nphilosopher\nphilosophy\nphobia\nphone\nphone\nphone\nphone\nphone\nphone\nphotocopier\nphotocopy\nphotocopy\nphotofinisher\nphotograph\nphotographer\nphotograph\nphotograph\nphotograph\nphoto\nphrase\nphysician\nphysics\npiano\npiaster\npick\npick\npicker\npick\npickin\npickle\npick\npick\npickup\npick\npick\npick\npicture\npicture\npicture\npicture\npicture\npicture\npiece\npiece\npiece\npiece\npierce\npier\npie\npiggyback\npiggyback\npiglet\npigment\npig\npile\npile\npile\npile\npile\npiling\npile\npile\npillar\npillory\npillory\npillowcase\npillow\npill\npilot\npilot\npimp\npinch\npinch\npinch\npinch\npine\nping\nping\npin\npin\npin-point\npinpoint\npinpoint\npin\npin\npint\npin\npioneer\npioneer\npioneer\npipe\npipe\npipeline\npipe\npirate\npirate\npiroghi\npistil\npistol\npiston\npitch\npitch\npitcher\npitch\npitch\npitch\npitchman\npitch\npitch\npitfall\npitfall\npit\npit\npit\npit\npit\npit\npivot\npizzas-with-everything\npizzeria\nplacate\nplacate\nplace\nplace\nplacement\nplace\nplace\nplace\nplace\nplace\nplace\nplague\nplague\nplague\nplague\nplainclothes\nplain\nplain\nplaintiff\nplaintiff\nplane\nplanet\nplank\nplan\nplan\nplanner\nplanner\nplan\nplan\nplan\nplan\nplan\nplan\nplan\nplantation\nplant\nplant\nplant\nplant\nplant\nplant\nplan\nplan\nplastic\nplastic\nplate\nplatform\nplatitude\nplaudit\nplay\nplay\nplayer\nplayer\nplayground\nplay\nplay\nplayoff\nplay\nplay\nplay\nplay\nplay\nplead\nplead\npleading\nplead\nplead\npleasantry\nplease\nplease\nplease\nplease\nplease\nplease\nplease\nplea\npleasure\nplea\npledge\npledge\npledge\npledge\npledge\npliers\nply\nplight\nplod\nplot\nplot\nplotter\nplot\nplot\nplow\nplow\nplow\nplow\nploy\nploy\npluck\npluck\nplug\nplug\nplug\nplug\nplug\nplummet\nplummet\nplummet\nplummet\nplummet\nplunge\nplunge\nplunge\nplunge\nplunge\nplunge\nplunge\nplunk\nplus\nplus\nply\npoacher\npoach\npocket\npocket\npocket\npocket\npocket\npockmark\npoint\npoint\npointer\npoint\npoint\npoint\npoint\npoint\npoise\npoison\npoison\npoke\npoke\npoke\npolarize\npole\npole\npolice\npolice\npolice\npolice\npolicy\npolicy\npolice\npolicyholder\npolicy-maker\npolicy-make\npolish\npolish\npolish\npolish\npolitician\npolitician\npoliticize\npolitico-plaintiff\npolitics\npolitics\npolitics\npoll\npoll\npollen-produce\npollinate\npollinate\npollinate\npollinate\npoll\npoll\npollster\npoll\npollutant\npolluter\npollute\npollute\npoll\npol\npoltergeist\npolymer\npolyol\npolyp\npolyrhythm\nponder\nponder\nponder\npond\npony\npony\npontificate\npony\npool\npool\npool\npool\npop\npop\npopularize\npopularize\npopularize\npopulate\npopulate\npopulate\npopulation\npop\npop\nporcelain\nporch\npore\npore\npore\npork-barreler\nportable\nportend\nportend\nportfolio\nportfolio\nport\nportion\nportrait\nportrayal\nportray\nportray\nportray\nportray\nportray\nportray\nport\nport\npose\npose\npose\npose\npose\npose\nposition\nposition\nposition\nposition\nposition\npossess\npossess\npossess\npossess\npossession\npossess\npossess\npossibility\npostcard\npost\npost\npost\npost\nposter\nposting\npost\npostmark\npostmark\npostpone\npostpone\npostpone\npostpone\npostpone\npost\npost\npost\nposture\npost\npotable\npotato\npotentate\npotentiality\npothole\npot\npot\npouch\npounce\npound\npound\npound\npound\npour\npour\npour\npour\npour\npour\npowder\npower\npowerhouse\npower\npower\npractice\npractice\npractice\npractice\npractice\npractitioner\npragmatist\nprairie\npraise\npraise\npraise\npraise\npraise\npraise\npraise\nprance\nprayer\npray\npreach\npreach\npreach\npre-approve\npreapprove\nprearrange\nprecaution\nprecede\nprecedent\nprecede\nprecede\nprecede\nprecinct\nprecious-metal\nprecious-metal\nprecipice\nprecipitate\nprecipitate\npreclude\npreclude\npreclude\npredate\npredator\npredecessor\npredetermine\npredicate\npredict\\/advocate\npredict\npredict\npredict\npredict\nprediction\nprediction\npredict\npredict\npredict\npredispose\npre-empt\npre-empt\npreference\nprefer\nprefer\nprefer\nprefer\nprefer\nprefer\nprefer\nprejudice\nprejudice\npremiere\npremiere\npremiere\npremiere\npremise\npremium\npreoccupy\nprepay\nprepay\npreparation\npreparative\nprepare\nprepare\npreparer\nprepare\nprepare\nprepare\nprepare\nprepay\nprepayment\nprepayment\nprepay\nprep\npre-register\npre-register\nprerogative\npresage\npresage\npreschooler\nprescribe\nprescribe\nprescribe\nprescribe\nprescription\npresentation\npresent\npresent\npresenter\npresent\npresent\npresent\npresent\npresent\npresent\npreserve\npreserve\npreserve\npreserve\npreserve\npreside\npreside\npresident\npresident\npreside\npreside\npre-sign\npress\npress\npress\npress\npress\npressure\npressure\npressure\npressure\npressure\npressure\npressure\npress\npress\npress\npresume\npresume\npresume\npretend\npretend\npretend\npretension\npre-test\npre-try\nprevail\nprevail\nprevail\nprevail\nprevail\nprevail\nprevent\nprevent\nprevent\nprevent\nprevent\nprevent\nprevent\nprevent\npreview\npreview\nprey\nprice\nprice\nprice-earnings\nprice\\/earnings\nprice\nprice\nprice\nprice\nprice\nprice\npricing\nprice\nprick\npriest\nprime\nprimitive\nprince\nprincipal\nprinciple\nprint\nprint\nprinter\nprint\nprint\nprintout\nprint\nprint\nprint\nprint\npriority\nprisoner\nprison\nprivatize\nprivatize\nprivatize\nprivilege\nprize\nprize\nprobability\nprobe\nprobe\nprobe\nprobe\nprobe\nproblematic\nproblem\nprocedure\nproceed\nproceed\nproceedings\nproceedings\nproceed\nproceeds\nproceeds\nproceed\nproceed\nproceed\nprocess\nprocess\nprocess\nprocess\nprocess\nprocess\nprocessor\nprocess\nprocess\nproclaim\nproclaim\nproclaim\nproclaim\nproclaim\nproclamation\nprocure\nprod\nprod\nprod\nproduce\nproduce\nproducer\nproducer\nproduce\nproduce\nproduce\nproduce\nproduce\nproduction\nproduct\nproduct\nprod\nprofess\nprofess\nprofess\nprofessional\nprofessional\nprofession\nprofessor\nprofess\nproffer\nproffer\nprofferr\nprofile\nprofile\nprofile\nprofit\nprofit\nprofiteer\nprofiteer\nprofit\nprofit-sharing\nprofit\nprofit\nprofit\nprofit\nprofit-taking\nprofitt\nprofit\nprofit\nprogenitor\nprognosticator\nprogram\nprogrammer\nprogram\nprogram\nprogram\nprogram\nprogram\nprogress\nprogress\nprogress\nprogress\nprogression\nprogressive\nprogress\nprohibit\nprohibit\nprohibit\nprohibition\nprohibit\nprohibit\nprohibit\nproject\nproject\nproject\nproject\nproject\nprojection\nprojector\nproject\nproject\nproject\nproject\nproject\nproliferate\nproliferate\nproliferate\nprolong\nprolong\npromise\npromise\npromise\npromise\npromise\npromise\npromise\npromise\npromote\npromote\npromoter\npromote\npromote\npromote\npromote\npromotion\nprompt\nprompt\nprompt\nprompt\nprompt\nprompt\npromulgate\nprong\npronounce\npronounce\npronouncement\npronounce\nproof-of-purchase\nproofread\npropagandist\npropagandize\npropagandize\npropel\npropel\npropel\npropel\nproperty\nproponent\nproponent\nproportion\nproposal\nproposal\nproposal\npropose\npropose\npropose\npropose\npropose\npropose\nproposition\nprop\nprop\nprop\nproprietorship\nproprietor\npropse\nprop\nprop\nproscribe\nproscribe\nprosecute\nprosecute\nprosecute\nprosecute\nprosecution\nprosecutor\nprosecutor\nprosecutor\npro\npro\nprospects\nprospects\nprospects\nprospectus\nprosper\nprosper\nprostitute\nprotect\nprotect\nprotect\nprotection\nprotector\nprotect\nprotect\nprotect\nprotege\nprotein\nprotein\nprotest\nprotest\nprotester\nprotest\nprotestor\nprotest\nprotest\nprotest\nprotocol\nprototype\nprove\nprove\nprove\nproverb\nprove\nprove\nprove\nprove\nprovide\nprovide\nprovide\nprovider\nprovide\nprovide\nprovide\nprovide\nprovide\nprovide\nprovince\nprove\nprovision\nprovision\nprovoke\nprovoke\nprovoke\nprovoke\nproxy\nprune\nprune\npseudo-lobbyist\npsychic\npsychologist\npsychologist\npublication\npublicize\npublicize\npublic-relation\npublic-works\npublic-works\npublish\npublish\npublish\npublisher\npublisher\npublish\npublish\npublish\npub\npuffer\npull-back\npullback\npull\npull\npull\npullout\npull\npull\npull\npull\npull\npulverize\npummel\npummel\npummel\npump\npump\npump\npump\npump\npump\npunch\npunch\npuncher\npunch\npunch\npunch\npunch\npundit\npunish\npunish\npunish\npun\npunt\npupil\npuppet\npuppy\npurchase\npurchase\npurchaser\npurchase\npurchase\npurchase\npurchase\npurchase\npurchase\npurge\npurge\npurge\npurge\npurge\npurist\npur-poise\npurport\npurport\npurpose\npurr\npurse-snatching\npurse\npursue\npursue\npursuer\npursue\npursue\npursue\npursue\npursuit\npush\npush\npusher\npush\npush\npush\npush\npush\npush\nput\nput\nput\nput\nput\nput\nput\nput\nput\nput\nput\nput\nput\npuzzle\npuzzle\npuzzle\npuzzle\npyramid\npyramid\nquack\nquack\nquadruped\nquadruple\nquadruple\nquadruple\nquadruple\nquake\nqualification\nqualify\nqualify\nqualify\nqualify\nqualify\nqualify\nquality\nqualm\nquantify\nquantify\nquantity\nquarrel\nquarrel\nquarterback\nquarter\nquartet\nquash\nquash\nquash\nqueer\nquell\nquery\nquery\nquery\nquestion\nquestion\nquestion\nquestion\nquestion\nquestion\nquestion\nquestion\nquestion\nqueue\nqueue\nquibble\nquicken\nquiet\nquiet\nquiet\nquiet\nquip\nquip\nquip\nquip\nquirk\nquit\nquit\nquit\nquit\nquit\nquit\nquiver\nquiz\nquota-cheater\nquota\nquotation\nquote\nquote\nquote\nquote\nquote\nquote\nquote\nquote\nrace\nracehorse\nrace\nracetrack\nrace\nrace\nrack\nrack\nracketeer\nracket\nrack\nrack\nradar-elude\nradical\nradio\nradio\nrafter\nrage\nrage\nrage\nrage\nrage\nraid\nraid\nraider\nraid\nraid\nraid\nrailbike\nrailcar\nrailing\nrailroad\nrail\nrail\nrailway\nrain\nrain\nrain\nraise\nraise\nraiser\nraise\nraise\nraise\nraise\nraise\nraise\nraise\nrake\nrake\nrally\nrally\nrally\nrally\nrally\nrally\nrally\nrally\nrally\nramble\nramification\nram\nrampart\nramp\nramp\nrancher\nranch\nrand\nrange\nrange\nranger\nrange\nrange\nrange\nrange\nrange\nring\nrank\nrank\nranking\nrank\nrankle\nrank\nrank\nrank\nrank\nrun\nrape\nrape\nrapeseed\nrape\nrapist\nraptor\nrarefy\nratchet\nrate\nrate\nrate\nratepayer\nrate\nrate\nrate\nrate\nrate\nrate\nratify\nratify\nratify\nratify\nrating\nrating\nrate\nrationalization\nrationalize\nrationalize\nration\nratio\nratio\nrat\nrattle\nrattle\nrattle\nrattle\nrattle\nrat\nravage\nravages\nravages\nrave\nrave\nravine\nraw-material\nray\nraze\nraze\nreach\nreach\nreach\nreach\nreach\nreach\nreach\nreach\nreach\nreact\nreact\nreact\nreaction\nreactivate\nreactivate\nreactor\nreact\nreact\nreact\nreader\nreader\nready\nreading\nread\nread\nreadmit\nread\nread\nread\nread\nread\nread\nread\nread\nreaffirm\nreaffirm\nreaffirm\nreaffirm\nreaganaut\nrealestate\nrealign\nrealign\nrealignment\nrealign\nrealist\nreality\nrealize\nrealize\nrealize\nrealize\nrealize\nrealize\nreallocate\nreallocate\nrealm\nream\nreap\nreap\nreap\nreappoint\nreapportion\nreappraise\nreappraise\nreappraise\nreap\nreap\nreard\nrear\nrear\nrearm\nrearrange\nrearrange\nreason\nreason\nreason\nreason\nreason\nreason\nreason\nreassert\nreassert\nreassert\nreassess\nreassess\nreassign\nreassign\nreassignment\nreassume\nreassurance\nreassure\nreassure\nreassure\nreassure\nreauthorize\nreawaken\nrebate\nrebel\nrebel\nrebound\nrebound\nrebound\nrebound\nrebound\nrebound\nrebound\nrebind\nrebuff\nrebuff\nrebuild\nrebuild\nrebuild\nrebuild\nrebuke\nrebut\nrebut\nrebut\nrecalculate\nrecalculate\nrecalculation\nrecall\nrecall\nrecall\nrecall\nrecall\nrecall\nrecall\nrecall\nrecall\nrecant\nrecant\nrecapitalization\nrecapitalize\nrecapture\nrecede\nrecede\nreceipt\nreceivable\nreceive\nreceive\nreceiver\nreceive\nreceive\nreceive\nreceive\nrecentralize\nreceptionist\nreceptor\nrecess\nrecession\nrecharge\nrecipe\nrecipient\nrecite\nrecite\nreckon\nreckon\nreckon\nreckon\nreckon\nreclaim\nreclaim\nreclaim\nreclaim\nreclaim\nreclassify\nrecline\nrecognize\nrecognize\nrecognize\nrecognize\nrecognize\nrecognize\nrecommendation\nrecommendaton\nrecommend\nrecommend\nrecommend\nrecommend\nrecommend\nrecommend\nreconcile\nreconcile\nreconcile\nreconnect\nreconsider\nreconsider\nreconstruct\nreconstruct\nreconstruct\nreconstruct\nrecord\nrecord\nrecorder\nrecording\nrecord\nrecord\nrecord\nrecord\nrecord\nrecord\nrecount\nrecount\nrecount\nrecount\nrecoup\nrecoup\nrecoup\nrecover\nrecover\nrecovery\nrecover\nrecover\nrecover\nrecover\nrecover\nrecraft\nre-creaction\nrecreate\nre-create\nre-creation\nrecruit\nrecruit\nrecruit\nrecruit\nrecruit\nrecruit\nrecruit\nrectangle\nrectify\nrectify\nrecuperate\nrecur\nrecuse\nrecycle\nrecycle\nrecycle\nrecycle\nredden\nredeem\nredeem\nredeem\nredefine\nredefine\nredefine\nredemption\nredeploy\nredesign\nredesign\nredesign\nredesign\nredevelop\nred-flag\nredial\nredirect\nrediscover\nredistribute\nredistribute\nredline\nredo\nredouble\nredound\nredo\nredraw\nredress\nred\nred\nreduce\nreduce\nreduce\nreduce\nreduce\nreduce\nreduce\nreduction\nreef\nre-elect\nre-elect\nreel\nreel\nreel\nre-emerge\nre-emphasize\nre-enact\nre-enactment\nre-enactment\nre-enter\nre-enter\nre-enter\nre-establish\nre-establish\nreestablish\nre-evaluate\nre-evaluate\nre-evaluate\nre-examine\nreexamine\nre-export\nrefashion\nreferee\nreference\nreferral\nrefer\nrefer\nrefer\nrefer\nrefer\nrefer\nrefer\nrefile\nrefinance\nrefinance\nrefinance\nrefinance\nrefine\nrefinery\nrefiner\nrefiner\nrefine\nrefine\nrefit\nreflect\nreflect\nreflect\nreflect\nreflect\nreflect\nreflect\nrefocus\nrefocus\nrefocus\nrefocus\nrefocus\nreform\nreformer\nreform\nreform\nreformulate\nreform\nrefrain\nrefrain\nrefresh\nrefrigerator\nrefuel\nrefugee\nrefund\nrefund\nrefund\nrefund\nrefund\nrefurbish\nrefurbish\nrefurbish\nrefuse\nrefuse\nrefuser\nrefuse\nrefuse\nrefuse\nrefuse\nrefute\nrefute\nregain\nregain\nregain\nregain\nregain\nregain\nregain\nregard\nregard\nregard\nregard\nregard\nregard\nregard\nregard\nregenerate\nregion\nregister\nregister\nregister\nregister\nregister\nregister\nregistrant\nregistration\nregret\nregret\nregret\nregret\nregret\nregret\nregroup\nregroup\nregulate\nregulate\nregulate\nregulate\nregulation\nregulation\nregulator\nregulator\nregulator\nregumm\nrehabilitate\nrehabilitate\nrehash\nreign\nreign\nreignite\nreignite\nreignite\nreignite\nreimburse\nreimburse\nreimbursement\nreimburse\nreimburse\nreimburse\nreimpose\nreincorporate\nreincorporate\nreindict\nreinforce\nreinforce\nreinforcement\nreinforce\nreinforce\nreinforce\nreinforce\nreinforce\nrein\nrein\nreinstall\nreinstate\nreinstate\nreinstate\nreinstate\nreinstitute\nreinsurer\nreintegrate\nreintroduce\nrein\nreinvent\nreinvent\nreinvest\nreinvest\nreinvest\nreinvest\nreinvest\nreinvigorate\nreinvigorate\nreinvigorate\nreiterate\nreiterate\nreiterate\nreit\nreject\nreject\nreject\nrejection\nreject\nreject\nreject\nrejoice\nrejoin\nrejoin\nrejoin\nrejoin\nrejuvenate\nrekindle\nrekindle\nrekindle\nrekindle\nrelabel\nrelate\nrelate\nrelate\nrelate\nrelate\nrelate\nrelate\nrelationship\nrelation\nrelation\nrelative\nrelaunch\nrelaunch\nrelax\nrelax\nrelax\nrelax\nrelax\nrelay\nrelease\nrelease\nrelease\nrelease\nrelease\nrelease\nrelease\nrelease\nrelegate\nrelent\nrelent\nrelent\nrelic\nrely\nrely\nrely\nrely\nrelieve\nrelieve\nrelieve\nreligion\nrelinquish\nrelinquish\nrelinquish\nrelinquish\nrelish\nrelish\nrelish\nrelive\nrelocate\nrelocate\nrelocate\nrelocate\nrelocation\nrely\nrely\nrely\nrely\nremake\nremain\nremain\nremain\nremain\nremains\nremains\nremain\nremain\nremain\nremake\nremand\nremark\nremark\nremarketer\nremarketing\nremark\nremark\nremark\nremedy\nremedy\nremedy\nremember\nremember\nremember\nremember\nremember\nremember\nremember\nremic\nremic\nremind\nremind\nreminder\nremind\nremind\nremind\nremind\nremittance\nremnant\nremodel\nremodel\nremora\nremove\nremove\nremove\nremove\nremove\nremove\nremunerate\nrename\nrename\nrename\nrender\nrender\nrendering\nrender\nrender\nrendezvous\nrenege\nrenege\nrenegotiate\nrenegotiate\nrenegotiate\nrenewal\nrenew\nrenew\nrenew\nrenew\nrenew\nrenew\nrenounce\nrenounce\nrenounce\nrenovate\nrenovate\nrenovate\nrenown\nrental\nrent\nrenter\nrent\nrent\nrent\nrent\nrent\nreoffer\nreopen\nreopen\nreopen\nreopen\nreopen\nreorganize\nreorganize\nreorganize\nreorient\nrepackage\nrepackage\nrepackage\nrepay\nrepay\nrepaint\nrepair\nrepair\nrepair\nrepair\nrepair\nrepair\nrepair\nreparation\nrepass\nrepatriate\nrepay\nrepayment\nrepay\nrepeal\nrepeal\nrepeal\nrepeat\nrepeat\nrepeater\nrepeat\nrepeat\nrepeat\nrepeat\nrepel\nrepel\nrepercussion\nreplace\nreplace\nreplacement\nreplace\nreplace\nreplace\nreplace\nreplace\nreplaster\nreplay\nreplenish\nreplenish\nreplicate\nreplicate\nreplicate\nreply\nreply\nreply\nreply\nreply\nreply\nrepond\nreport\nreport\nreporter\nreport\nreport\nreport\nreport\nreport\nreport\nreport\nreport\nreport\nrepose\nreposition\nrepository\nrepossess\nrepresentation\nrepresentative\nrepresentative\nrepresent\nrepresent\nrepresent\nrepresentive\nrepresent\nrepresent\nrepresent\nrepress\nrepress\nreprice\nreprint\nreprint\nreprint\nreprint\nreprisal\nreproduce\nreproduce\nreproduce\nreprove\nrep\nrepublican\nrepublic\nrepudiate\nrepurchase\nrepurchase\nrepurchase\nrepurchase\nrepurchase\nrepurchase\nrepurchase\nreputation\nrepute\nrequest\nrequest\nrequest\nrequest\nrequest\nrequest\nrequire\nrequire\nrequirement\nrequire\nrequire\nrequire\nrequire\nrequire\nrequisition\nreroute\nreroute\nrerun\nresale\nreschedule\nreschedule\nreschedule\nrescind\nrescind\nrescind\nrescission\nrescue\nrescuer\nrescue\nrescue\nresearch\nresearcher\nresearcher\nresearcher\nresearch\nresearch\nresearch\nreseller\nresell\nresell\nresell\nresell\nresemblance\nresemble\nresemble\nresemble\nresemble\nresent\nresent\nresent\nreservation\nreserve\nreserve\nreserve\nreserve\nreserve\nreserve\nreserve\nreserve\nreserve\nreservoir\nreset\nreshape\nreshape\nreshape\nreshape\nreshuffle\nreshuffle\nreshuffling\nreshuffle\nreside\nresidence\nresident\nresident\nreside\nreside\nreside\nresidue\nresignation\nresignation\nresign\nresign\nresign\nresign\nresign\nresin\nresist\nresist\nresist\nresist\nresist\nresist\nresist\nresell\nresell\nresell\nresolution\nresolve\nresolve\nresolve\nresolve\nresonate\nresonate\nresonate\nresort\nresort\nresort\nresort\nresource\nrespect\nrespect\nrespects\nrespect\nrespect\nrespond\nrespond\nrespondent\nrespond\nrespond\nrespond\nrespond\nrespond\nresponse\nresponse\nresponsibility\nrestart\nrestart\nrestarter\nrestart\nrestart\nrestate\nrestate\nrestate\nrestate\nrestaurant\nrestaurant\nrest\nrestore\nrestore\nrestore\nrestore\nrestrain\nrestrain\nrestrain\nrestraint\nrestrain\nrestrict\nrestrict\nrestrict\nrestriction\nrestrict\nrestrict\nrestrict\nrestrict\nrestructure\nrestructure\nrestructure\nrestructure\nrestructure\nrestructuring\nrestructure\nrest\nrest\nrest\nrest\nrest\nrestyle\nresubmit\nresult\nresult\nresult\nresult\nresult\nresult\nresult\nresult\nresult\nresume\nresume\nresume\nresume\nresume\nresume\nresume\nre-supply\nresurface\nresurface\nresurge\nresurrect\nresurrect\nresurrect\nresurrect\nresurrect\nresuscitate\nresuscitate\nretailer\nretailer\nretail\nretail-sale\nretail\nretail\nretail\nretain\nretain\nretain\nretain\nretain\nretain\nretake\nretaliate\nretardant\nretard\nrethink\nrethink\nretire\nretire\nretire\nretiree\nretirement\nretire\nretire\nretire\nretire\nretool\nretool\nretort\nretort\nretrace\nretract\nretrain\nretreat\nretreat\nretreat\nretreat\nretreat\nretrench\nretrieve\nretrieve\nretrieve\nretrieve\nretrofit\nretry\nreturn\nreturn\nreturn\nreturn\nreturn\nreturn\nreturn\nreturn\nreunion\nreunite\nreunite\nreuse\nrevalue\nrevamp\nrevamp\nrevamp\nrevamp\nbe\nreveal\nreveal\nreveal\nreveal\nreveal\nreveal\nreveal\nrevelation\nreveler\nrevel\nrevel\nrevel\nrevenue\nrevenue\nreverberate\nreverberate\nreverberate\nreverberation\nreversal\nreverse\nreverse\nreverse\nreverse\nreverse\nreverse\nreverse\nrevert\nrevert\nreview\nreview\nreview\nreview\nreview\nreview\nreview\nreview\nrevise\nrevise\nrevise\nrevise\nrevise\nrevise\nrevise\nrevisionist\nrevision\nrevisit\nrevisit\nrevisit\nrevitalize\nrevitalize\nrevival\nrevival\nrevive\nrevive\nrevive\nrevive\nrevive\nrevoke\nrevoke\nrevoke\nrevolutionary\nrevolutionize\nrevolutionize\nrevolve\nrevolve\nrevolve\nrev\nreward\nreward\nreward\nreward\nreward\nreward\nreward\nrework\nrewrite\nrewrite\nrewrite\nrhyme\nrhyme\nrhyme\nribby\nribbon\nrib\nrican\nriches\nrider\nride\nride\nride\nridge\nridicule\nridicule\nride\nrid\nrid\nrid\nrid\nrid\nriff\nrifle\nrig\nrig\nright\nrights\nrights\nrights\nright-to-lifer\nright-winger\nrigor\nrig\nrile\nrile\nrim\nringer\nringer\nring\nring\nring\nring\nring\nriot\nripen\nripen\nripoff\nrip\nrip\nripple\nripple\nripple\nrise\nrise\nrise\nrise\nrise\nrise\nrise\nrisk\nrisk\nrisk\nrisk\nrisk\nrisk\nrisk\nrite\nritual\nrival\nrivalry\nrival\nrival\nrival\nrival\nriver\nrivet\nrivet\nrivet\nrivet\nroadblock\nroad\nroad\nroadway\nroam\nroam\nroar\nroar\nrob\nrobbery\nrobber\nrobber\nrob\nrobe\nrobot\nrock\nrock\nrocker\nrocket\nrocket\nrocket\nrock\nrock\nrock\nrodent\nride\nride\nrod\nrogue\nroil\nroil\nrole\nrollback\nroll\nroll\nroll\nroller\nroll\nrollover\nroll\nroll\nroll\nroll\nromance\nromanticize\nromp\nromp\nroofer\nroof\nrooftop\nroommate\nroom\nroost\nroot\nrooter\nroot\nroot\nroot\nroot\nrope\nrosarian\nrose\nrise\nrise\nrotate\nrotate\nrotate\nrot\nrot\nrough\nroughneck\nround\nround\nround\nround\nround\nroustabout\nroute\nroute\nroute\nroutine\nroute\nrow\nrow\nrow\nroyalty\nrub\nrubber-neck\nrubber-stamp\nrubdown\nrubfest\nrubin\nruble\nrub\nrub\nrub\nrub\nruffle\nruffle\nrug\nruin\nruin\nruin\nruin\nruin\nruin\nrule\nrule\nruler\nrule\nrule\nrule\nrule\nrule\nrule\nruling\nrule\nrumble\nrumble\nrumble\nrumbling\nrumble\nruminate\nruminate\nrumination\nrumor\nrumor\nrumor\nrumor\nrumor\nring\nrun-in\nrunner\nrunner-up\nrun\nrun\nrun\nrun\nrunup\nrun\nrun\nrun\nrun\nrunway\nrupture\nrupture\nrupture\nrush\nrush\nrush\nrush\nrush\nrush\nrust\nrusticate\nrustler\nrustling\nrv\nsaatchus\nsaber\nsabotage\nsack\nsacking\nsack\nsack\nsacrifice\nsacrifice\nsacrifice\nsacrifice\nsaddle\nsaddle\nsafeguard\nsafeguard\nsafeguard\nsafeguard\nsafety\nsage\nsag\nsag\nsag\nsag\nsago\nsay\nsay\nsay\nsay\nsay\nsail\nsail\nsailor\nsail\nsaint\nsalary\nsalary\nsalaryman\nsalesman\nsale\nsale\nsale\nsalespeople\nsalespeople\nsalicylate\nsalt\nsalute\nsalute\nsalvage\nsalvage\nsalve\nsamaritan\nsamovar\nsample\nsample\nsanction\nsanction\nsanction\nsanction\nsandinista\nsand\nsandwich\nsandwich\nsandwich\nsandwich\nsing\nsanitationist\nsanitize\nsanitize\nsink\nsan\nsap\nsap\nsap\nsatellite\nsatisfy\nsatisfy\nsatisfy\nsatisfy\nsatisfy\nsatisfy\nsatisfy\nsatisfy\nsaturate\nsaturate\nsaturate\nsit\nsit\nsaucer\nsauce\nsauna\nsave\nsave\nsavers\\/investors\nsaver\nsave\nsave\nsave\nsave\nsavings\nsavings\nsave\nsavor\nsavor\nsavor\nsavor\nsee\nsay\nsay\nsay\nsay\nsay\nsay\nsay\nsay\nscab\nscalawag\nscale\nscale\nscale\nscale\nscale\nscalp\nscam\nscammer\nscamper\nscam\nscandalize\nscandal\nscanner\nscan\nscan\nscan\nscan\nscape\nscare\nscare\nscare\nscare\nscare\nscare\nscarf\nscare\nscare\nscar\nscar\nscatter\nscavenger\nscenario\nscenario\nscene\nscent\nschedule\nschedule\nschedule\nschedule\nschedule\nschemer\nscheme\nscheme\nschmumper\nscholarship\nscholar\nschoolboy\nschoolchildren\nschoolmate\nschool\nschool\nschoolteacher\nschool\nscience\nscientist\nscientist\nscime\nscime\nscoff\nscoff\nscoff\nscold\nscold\nscoop\nscoop\nscoop\nscoop\nscoop\nscoot\nscore\nscore\nscore\nscore\nscore\nscore\nscorn\nscorn\nscotch\nscotch\nscourge\nscour\nscour\nscout\nscout\nscowl\nscowl\nscramble\nscramble\nscramble\nscramble\nscramble\nscrape\nscrap\nscrap\nscrap\nscrap\nscrap\nscrap\nscrap\nscrap\nscratch\nscratch\nscratch\nscratch\nscream\nscream\nscream\nscream\nscream\nscreech\nscreech\nscreen\nscreening\nscreen\nscreen\nscreen\nscreen\nscreenwriter\nscrew\nscrew\nscrew\nscribble\nscribbler\nscribble\nscribe\nscrimp\nscrimp\nscript\nscriptwriter\nscrounge\nscrounge\nscrubber\nscrutinize\nscrutinize\nscrutinize\nscrutinize\nsculptor\nsculpture\nscurry\nscurry\nscurry\nscuttle\nscuttle\nscuttle\nsealant\nseal\nseal\nseal\nseal\nseal\nseaman\nsearch\nsearch\nsearch\nsearch\nsearch\nsearch\nsear\nsear\nsea\nseasoning\nseason\nseat\nseat\nseat\nseat\nseat\nsecede\nsecond-guess\nsecond-guess\nseconds\nseconds\nsecretary\nsecret\nsection\nsector\nsecure\nsecure\nsecure\nsecure\nsecure\nsecurite\nsecurity\nsecurity\nsedan\nseduce\nseduce\nseed\nseed\nsee\nsee\nseeker\nseek\nseek\nseek\nseek\nseek\nseek\nseek\nseem\nseem\nseem\nseem\nseem\nseem\nsee\nsee\nseep\nseesaw\nsee\nsee\nseethe\nsee\nsee\nsee\nsee\nsegment\nsegment\nsegregate\nsegregate\nseize\nseize\nseize\nseize\nseize\nseize\nseizure\nselect\nselect\nselect\nselection\nselection\nselect\nselect\nselect\nself-destruct\nself-insure\nself-reinsure\nself-starter\nseller\nsell\nsell\nsell-off\nsell-off\nselloff\nsell\nsell\nsell\nsell\nsell\nsemantics\nsemester\nsemi-celebrity\nsemiconductor\nsemifinish\nsemifinish\nseminar\nsenator\nsenator\nsender\nsend\nsend\nsend\nsend\nsend\nsenior\nsenior\nsense\nsens\nsense\nsense\nsensibility\nsense\nsensitive\nsensitivity\nsensitize\nsen.\nsensor\nsentence\nsentence\nsentence\nsentence\nsentencing\nsentence\nsentence\nsentiment\nsend\nsend\nseparate\nseparate\nseparate\nseparate\nseparate\nsequel\nsequester\nsequester\nsequester\nsequin\nsergeant\nserial\nseries\nseries\nservant\nserve\nserve\nserve\nserver\nserve\nserve\nserve\nservice\nservice\nservice\nservice\nservice\nservice\nserve\nsession\nsetback\nset\nset\nsetter\nsetting\nset\nset\nsettle\nsettle\nsettlement\nsettlement\nsettler\nsettle\nsettle\nsettle\nsettle\nsettle\nset\nset\nset\nset\nsever\nsever\nsever\nsever\nsevice\nsewer\nsew\nsex\nshack\nshades\nshadow\nshadow\nshadow\nshag\nshake\nshake\nshake\nshake\nshake\nshake\nshake\nshape\nshape\nshape\nshape\nshape\nshape\nshard\nsharecropper\nshare\nshare\nshareholder\nshareholder\nshareholding\nshare\nshare\nshare\nshare\nshare\nshare\nshare\nshark\nsharpen\nsharpen\nsharpen\nshatter\nshatter\nshatter\nshatter\nshave\nshave\nshave\nshave\nshave\nshear\nsheath\nshed\nshed\nshed\nshed\nshed\nshed\nsheet\nsheik\nshell\nshell\nshell\nshell\nshelter\nshelter\nshelter\nshelter\nshelve\nshelf\nshepherd\nsheriff\nshy\nshy\nshield\nshield\nshield\nshield\nshield\nshift\nshift\nshift\nshift\nshift\nshift\nshift\nshilling\nshill\nshimmer\nshine\nshine\nshine\nshine\nshin\nshipbuilder\nshipment\nshipment\nship\nship\nshipper\nshipper\nship\nshipset\nship\nship\nship\nshipyard\nshirk\nshirk\nshirt\nshiver\nshiver\nshock\nshock\nshock\nshock\nshock\nshoehorn\nshoe-horn\nshoelace\nshoemake\nshoe\nshake\nshake\nshooting\nshoot\nshoot\nshoot\nshoot\nshoot\nshopkeeper\nshop\nshop\nshopper\nshopper\nshopper\nshop\nshop\nshop\nshop\nshop\nshore\nshore\nshore\nshear\nshear\nshortage\nshortchange\nshort-circuit\nshortcoming\nshort\nshorten\nshorten\nshorten\nshort\nshort\nshort-seller\nshorts\nshort\nshot\nshoot\nshoot\nshoulder\nshoulder\nshoulder\nshoulder\nshout\nshout\nshout\nshout\nshout\nshove\nshovel\nshove\nshove\nshove\nshowcase\nshow\nshow\nshower\nshower\nshowgirl\nshowing\nshow\nshow\nshow\nshowroom\nshowroom\nshow\nshow-stopper\nshow\nshow\nshow\nshow\nshow\nshrink\nshred\nshriek\nshrine\nshrink\nshrink\nshrink\nshrink\nshrivel\nshroud\nshrub\nshrub\nshrug\nshrug\nshrug\nshrug\nshrink\nshudder\nshuffle\nshuffle\nshun\nshun\nshun\nshun\nshun\nshutdown\nshut\nshut\nshutter\nshutter\nshut\nshuttle\nshuttle\nshuttle\nshuttle\nshut\nshut\nshut\nshy\nshy\nshy\nsibling\nside\nside\nsideline\nsideline\nsideline\nside\nside\nsidestep\nsidestep\nsidestep\nsidestep\nsidestep\nsidetrack\nside\nside\nsidewalk\nsift\nsift\nsift\nsigh\nsigh\nsigh\nsigh\nsighting\nsight\nsignal\nsignal\nSIGNAL\nsignal\nsignal\nsignal\nsignal\nsignal\nsignal\nsignal\nsignatory\nsignature\nsignboard\nsign\nsign\nsign\nsignify\nsignify\nsignify\nsign\nsign\nsign\nsign\nsign\nsilence\nsilence\nsilence\nsilence\nsilt\nsimilarity\nsimmer\nsimmer\nsimplicity\nsimplify\nsimplify\nsimplify\nsimplify\nsimplify\nsimulate\nsimulate\nsimulation\nsimulator\nsinger\nsing\nsingin\nsingle\nsingle\nsingles\nsingle\nsingle\nsingle\nsing\nsing\nsing\nsink\nsink\nsink\nsin\nsiphon\nsiphon\nsiphon\nsiphon\nsip\nsip\nsiren\nsister\nsitcom\nsite\nsit\nsit\nsit\nsit\nsituate\nsituation\nsit\nsit\nsit\nsix-pack\nsixty\nsixty\nsize\nsize\nsize\nsizzle\nsizzle\nskateboard\nskeptic\nskeptic\nsketch\nsketch\nsketch\nskew\nskid\nskid\nskid\nskid\nskid\nskid\nskid\nskier\nsky\nski\nskill\nskill\nskimmer\nskimp\nskim\nskin\nskip\nskipper\nskip\nskip\nskip\nskip\nskirmish\nskirmish\nskirt\nskirt\nskirt\nski\nski\nskr1.5\nskr205\nskr20\nskr225\nskr29\nskull\nskyrocket\nskyrocket\nskyrocket\nslab\nslacken\nslacken\nslacken\nslacks\nslack\nslay\nslam-dunk\nslam\nslam\nslap\nslap\nslap\nslap\nslap\nslash\nslash\nslash\nslash\nslash\nslash\nslate\nslate\nslate\nslat\nslaughter\nslaying\nslay\nsleep\nsleep\nsleep\nsleep\nsleep\nsleep\nsleeve\nsleep\nslice\nslice\nslice\nslide-pack\nslide\nslide\nslide\nslide\nslide\nslide\nslide\nslim\nslim\nslinger\nsling\nslip\nslip\nslip\nslip\nslip\nslip\nslither\nslither\nslit\nsliver\nslogan\nslog\nslog\nslope\nslope\nslosh\nslot\nslough\nslowdown\nslow\nslow\nslow\nslow\nslow\nslow\ns&ls\nslump\nslump\nslump\nslump\nslump\nslump\nslum\nslur\nsmack\nsmack\nsmart\nsmash\nsmash\nsmash\nsmash\nsmell\nsmell\nsmell\nsmell\nsmell\nsmidgin\nsmile\nsmile\nsmile\nsmile\nsmile\nsmile\nsmoke\nsmoker\nsmoker\nsmokescreen\nsmoke\nsmoke\nsmoke\nsmoke\nsmolder\nsmolder\nsmooth\nsmooth\nsmother\nsmother\nsmother\nsmuggle\nsnafus\nsnag\nsnag\nsnake\nsnake\nsnake\nsnap\nsnap\nsnap\nsnapshot\nsnap\nsnap\nsnare\nsnarl\nsnatch\nsnatch\nsnatch\nsneak\nsneaker\nsneak\nsneak\nsniff\nsniff\nsniff\nsniff\nsnipe\nsnippet\nsnivel\n'\nsnoop\nsnooze\nsnore\nsnort\nsnowball\nsnowball\nsnowbird\nsnub\nsnub\nsoak\nsoak\nsoap\nsoapsud\nsoar\nsoar\nsoar\nsoar\nsoar\nsoar\nsoar\nsober\nsob\nsocall\nsocial-affair\nsocialist\nsocialist\nsocialize\nsocialize\nsociety\nsociologist\nsock\nsock\nsock\nsoda\nsofa\nsoft-drink\nsoften\nsoften\nsoften\nsoften\nsoften\nsoftie\nsoil\nsoil-nutrient\nsoil\nsoiree\nsoldier\nsoldier\nsell\nsell\nsolicitation\nsolicit\nsolicit\nsolicit\nsolicitor\nsolicit\nsolicit\nsolidify\nsolidify\nsolidify\nsolution\nsolve\nsolve\nsolve\nsolvent\nsolve\nsolve\nsolve\nsolve\nsong\nsongster\nsongwriter\nson\nsoothe\nsoothe\nsophisticate\nsop\nsorehead\nsort\nsort\nsort\nsort\nsort\nseek\nseek\nsoulmate\nsoul\nsound\nsound\nsounding\nsound\nsound\nsound\nsound\nsound\nsound\nsound\nsoup\nsource\nsource\nsource\nsour\nsour\nsour\nsoviet\nsoviet\nsow\nsow\nsow\nsow\nsow\nsoybean\nsoybean\nsoybean\nspaceship\nspace\nspackle\nspaghetti\nspan\nspan\nspan\nspan\nspare\nspare\nspare\nspare\nspare\nspare\nspare\nspark\nspark\nspark\nsparkplug\nspark\nspark\nspark\nspark\nspar\nspar\nspasm\nspa\nspawn\nspawn\nspawn\nspawn\nspeaker\nspeaker\nspeak\nspeak\nspeak\nspeak\nspeak\nspearhead\nspearhead\nspearhead\nspecialist\nspecialist\nspecialize\nspecialize\nspecialize\nspecialize\nspecialize\nspecial\nspecialty\nspecialty-chemical\nspecialty-metal\nspecies\nspecification\nspecification\nspecifics\nspecify\nspecify\nspecify\nspecify\nspecify\nspecimen\nspec\nspectator\nspeculate\nspeculate\nspeculate\nspeculate\nspeculate\nspeculation\nspeculator\nspeculator\nspeed\nspeed\nspeech\nspeed\nspeed\nspeed\nspeed\nspeed\nspeed\nspell\nspell\nspell\nspell\nspell\nspender\nspend\nspend\nspend\nspend\nspendthrift\nspend\nspend\nspend\nspend\nspend\nspew\nspew\nspice\nspider\nspy\nspy\nspy\nspigot\nspike\nspill\nspill\nspill\nspill\nspill\nspin\nspinoff\nspin\nspin\nspiral\nspiral\nspiral\nspirit\nspirit\nspirit\nsplash\nsplint\nsplits\nsplits\nsplit\nsplit\nsplit\nsplit\nspoil\nspoil\nspeak\nspokesman\nspokesman\nspoke\nspokesperson\nspeak\nsponsor\nsponsor\nsponsor\nsponsor\nsponsor\nsponsor\nsponsor\nsponsor\nspook\nspook\nspook\nspook\nspook\nspoonbill\nspoonful\nspore\nsport\nsporting-goods\nsport\nsportsman\nsport\nsport\nsport\nspotlight\nspot\nspot\nspot\nspot\nspot\nspot\nspot\nspot\nspot\nspouse\nspout\nspring\nsprawl\nspray\nspray\nspread\nspreadsheet\nspread\nspread\nspread\nspread\nspread\nspread\nspread\nspring\nspring\nspring\nsprinkle\nsprinkler\nsprinkle\nsprinkle\nspritzer\nsprout\nsprout\nsprout\nspruce\nspruce\nspring\nspud\nspin\nspin\nspurn\nspurn\nspurn\nspurn\nspurn\nspurn\nspur\nspur\nspur\nspur\nspur\nspurt\nspurt\nspurt\nspurt\nspurt\nspur\nsputter\nsputter\nspy\nspy\nsquabble\nsquad\nsquall\nsquander\nsquander\nsquare\nsquare\nsquare\nsquare\nsquare\nsquat\nsqueak\nsqueegee\nsqueeze\nsqueeze\nsqueeze\nsqueeze\nsqueeze\nsqueeze\nsquelch\nsquelch\nsquint\nsquint\nsquirm\nstab\nstab\nstab\nstabilize\nstabilize\nstabilize\nstabilize\nstabilize\nstabilize\nstabilize\nstack\nstack\nstack\nstack\nstadium\nstadium\nstaff\nstaffer\nstaffer\nstaff\nstaff\nstaff\nstage\nstage\nstage\nstage\nstage\nstage\nstage\nstagewhisper\nstagger\nstagger\nstage\nstagnate\nstagnate\nstaid\nstain\nstain\nstaircase\nstair\nstake\nstake\nstake\nstalk\nstalk\nstall\nstall\nstall\nstall\nstall\nstall\nstall\nstalwart\nstampede\nstampede\nstamping\nstamp\nstamp\nstamp\nstanch\nstandardize\nstandard\nstandby\nstand\nstand\nstand\nstand\nstand\nstand\nstand\nstand\nstaple\nstare\nstare\nstare\nstare\nstar\nstar\nstar\nstart\nstart\nstart\nstarter\nstart\nstart\nstartle\nstart\nstart\nstart\nstart-up\nstart-up\nstart\nstart\nstar\nstarve\nstarve\nstarve\nstarve\nstash\nstash\nstate\nstate\nstatement\nstatement\nstatesman\nstate\nstate\nstate\nstate\nstate\nstate\nstate\nstation\nstation\nstatistician\nstatistics\nstatistics\nstatue\nstatute\nstaunch\nstave\nstay\nstay\nstay\nstay\nstay\nstay\nstay\nsteady\nsteady\nsteak\nsteal\nsteal\nsteal\nsteal\nsteam\nsteam\nsteam\nsteelmaker\nsteelmaker\nsteel\nsteelworker\nsteer\nsteer\nsteer\nsteer\nsteer\nstem\nstem\nstem\nstem\nstem\nstem\nstep\nstep\nstep\nstep\nstep\nstep\nstep\nstep\nstep\nstep\nstereo\nstereotype\nsterile\nsterilize\nsterilize\nsterilize\nsteroid\nsteward\nstew\nsticker\nstick\nstick\nstick\nstick\nstick\nstick\nstifle\nstifle\nstifle\nstill-rage\nstilt\nstimulate\nstimulate\nstimulate\nstimulator\nstimulus\nsting\nstingray\nstink\nstint\nstipend\nstipulate\nstipulate\nstipulate\nstir\nstirring\nstir\nstirrup\nstir\nstir\nstir\nstir\nstitch\nstitch\nstockbroker\nstockbroker\nstockbuild\nstock\nstockholder\nstockholder\nstockholding\nstock-index-future\nstock-index\nstock\nstock-option\nstockpile\nstockpile\nstock\nstock\nstock\nstock\nstock\nstockyard\nstoke\nstoke\nstoke\nsteal\nsteal\nstomach\nstomach\nstomp\nstomp\nstone\nstone\nstonewall\nstand\nstand\nstooge\nstool\nstoppage\nstop\nstop\nstop\nstop\nstop\nstop\nstop\nstop\nstore\nstore\nstorefront\nstore\nstore\nstore\nstore\nstore\nstory\nstore\nstorm\nstorm\nstorm\nstoryteller\nstow\nstraddle\nstrafe\nstraighten\nstraighten\nstrain\nstrainer\nstrain\nstrain\nstrain\nstrain\nstrait\nstrand\nstrand\nstrand\nstrangle\nstrangle\nstrap\nstrap\nstratagem\nstrategy\nstrategy\nstrategist\nstrategist\nstrawberry\nstray\nstray\nstray\nstreak\nstream\nstream\nstreamline\nstreamline\nstreamline\nstream\nstreet\nstreetspeak\nstrengthen\nstrengthen\nstrengthen\nstrengthen\nstrengthen\nstrengthen\nstrength\nstress\nstress\nstress\nstress\nstress\nstressor\nstress\nstress\nstretch\nstretch\nstretch\nstretch\nstretch\nstretch\nstretch\nstretch\nstrew\nstricken\nstride\nstriker\nstrike\nstrike\nstrike\nstrike\nstrike\nstrike\nstrike\nstring\nstring\nstripe\nstrip\nstrip\nstrip\nstrip\nstrip\nstrip\nstrip\nstrive\nstrive\nstrive\nstrive\nstride\nstroke\nstroke\nstroke\nstroke\nstroll\nstroll\nstroll\nstronghold\nstrive\nstrike\nstrike\nstrike\nstructure\nstructure\nstructure\nstructure\nstructure\nstruggle\nstruggle\nstruggle\nstruggle\nstruggle\nstruggle\nstruggle\nstub\nstick\nstick\nstud\nstudent-athlete\nstudentle\nstudent\nstudent\nstudent\nstudy\nstudy\nstudy\nstudy\nstudy\nstudio\nstudio\nstudy\nstudy\nstudy\nstuff\nstuff\nstuff\nstuff\nstumble\nstumble\nstumble\nstumble\nstumble\nsting\nsting\nstun\nstun\nstun\nstunt\nstyle\nstyle\nstyle\nstymie\nsubcommittee\nsubcompact\nsubconference\nsubcontractor\nsubcontractor\nsubcontract\nsubdue\nsubgroup\nsubject\nsubject\nsubject\nsubject\nsubject\nsublet\nsubmarine\nsub-market\nsubmit\nsubmit\nsubmit\nsubmit\nsubmit\nsubmit\nsubordinate\nsubordinate\nsubordinate\nsubordinate\nsubordinate\nsubpoena\nsubpoena\nsubpoena\nsubscribe\nsubscriber\nsubscriber\nsubscribe\nsubscribe\nsubscribe\nsubscribe\nsubscribe\nsubscription\nsub-segment\nsubside\nsubside\nsubside\nsubsidiary\nsubsidy\nsubsidy\nsubsidize\nsubsidize\nsubsidize\nsubsidize\nsubsidize\nsubskill\nsubstance\nsubstantiate\nsubstation\nsubstitute\nsubstitute\nsubstitute\nsubstitute\nsubstitute\nsubstitute\nsubstract\nsubsume\nsubtilis\nsubtitle\nsubtract\nsubtract\nsubtract\nsubtract\nsub-underwriter\nsub-underwrite\nsuburb\nsubversive\nsubvert\nsubvert\nsubvert\nsubway\nsucceed\nsucceed\nsucceed\nsucceed\nsucceed\nsucceed\nsucceed\nsuccess\nsuccessor\nsuccumb\nsuccumb\nsuccumb\nsucker\nsuck\nsue\nsue\nsue\nsue\nsuffer\nsuffer\nsuffer\nsuffer\nsuffer\nsuffer\nsuffer\nsuffice\nsuffice\nsuffice\nsuggest\nsuggest\nsuggest\nsuggestion\nsuggest\nsuggest\nsuggest\nsue\nsuit\nsuite\nsuitor\nsuitor\nsuit\nsuit\nsuit\nsuit\nsummary\nsummarize\nsummarize\nsummarize\nsummarize\nsum\nsummer\nsummon\nsummon\nsummon\nsummon\nsum\nsum\nsum\nsunday\nsunflower\nsunglass\nsing\nsink\nsink\nsunset\nsupercede\nsupercede\nsupercomputer\nsuperconcentrate\nsuperconcentrate\nsuperconductor\nsuperconductor\nsuperimpose\nsuperintendent\nsuperior\nsupermarket\nsupermarket\nsuperpower\nsuperpremium\nsupersede\nsupersede\nsupersede\nsuperstar\nsupervise\nsupervise\nsupervise\nsupervise\nsupervise\nsupervise\nsupervisor\nsupplement\nsupplement\nsupply\nsupply\nsupplier\nsupplier\nsupplies\nsupply\nsupply\nsupply\nsupply\nsupport\nsupport\nsupporter\nsupporter\nsupport\nsupport\nsupport\nsupport\nsupport\nsupport\nsuppose\nsuppose\nsuppose\nsuppose\nsuppose\nsuppressant\nsuppress\nsuppress\nsuppress\nsurface\nsurface\nsurface\nsurface\nsurface\nsurface\nsurfer\nsurge\nsurge\nsurge\nsurge\nsurge\nsurge\nsurge\nsurmount\nsurpass\nsurpass\nsurpass\nsurpass\nsurpass\nsurplus\nsurprise\nsurprise\nsurprise\nsurprise\nsurprise\nsurprise\nsurrender\nsurrender\nsurrender\nsurrender\nsurrender\nsurround\nsurround\nsurround\nsurround\nsurround\nsurtax\nsurvey\nsurvey\nsurvey\nsurvey\nsurvey\nsurvey\nsurvey\nsurvey\nsurvive\nsurvive\nsurvive\nsurvive\nsurvive\nsurvive\nsurvive\nsurvive\nsurvive\nsurvivor\nsuspect\nsuspect\nsuspect\nsuspect\nsuspect\nsuspect\nsuspect\nsuspend\nsuspend\nsuspend\nsuspend\nsuspension\nsuspicion\nsustain\nsustain\nsustain\nsustain\nsustain\nsuture\nsuvivor\nbe\ns\nbe\nswallow\nswallow\nswallow\nswamp\nswamp\nswamp\nswan\nswap\nswap\nswap\nswap\nswap\nswarm\nswathe\nsway\nsway\nsway\nsway\nswear\nswear\nswear\nsweat\nsweater\nsweat\nsweatshirt\nsweat\nsweat\nsweat\nswede\nsweeper\nsweep\nsweep\nsweepstakes\nsweep\nsweep\nsweeten\nsweeten\nsweetener\nsweeten\nsweet\nswell\nswell\nswell\nswell\nswell\nswell\nswell\nsweep\nsweep\nswerve\nswim\nswim\nswindle\nswine\nswing\nswing\nswing\nswing\nswing\nswitch\nswitch\nswitcher\nswitch\nswitch\nswitch\nswitch\nswitch\nswitch\nswivel\nSwivel\nswear\nswear\nswing\nswing\nswing\nsycophant\nsymbolize\nsymbolize\nsymbol\nsympathy\nsympathizer\nsympathize\nsymposium\nsymptom\nsym\nsynchronize\nsynchronize\nsynchronize\nsyndciate\nsyndicate\nsyndicate\nsyndicate\nsyndicate\nsyndicate\nsyndication\nsynergy\nsyngery\nsynonym\nsynthesizer\nsynthesize\nsynthetic\nsystem\nsystem\ntabac\ntable\ntablespoon\ntablet\ntabloid\ntaboo\ntab\ntack\ntack\ntackle\ntackle\ntackle\ntackle\ntackle\ntaco\ntactic\ntag\ntag\ntag\ntail\ntailor\ntailor\ntail\ntaint\ntake\ntake\ntakeover\ntaker\ntake\ntake\ntaketh\ntake\ntake\ntake\ntake\ntakings\ntake\ntake\ntalent\ntale\ntale\ntalk\ntalk\ntalk\ntalk\ntalk\ntalk\ntalk\ntalk\ntalk\ntalk\ntally\ntally\ntally\ntame\ntame\ntamper\ntamper\ntamper\ntampon\ntandy\ntangle\ntangle\ntango\ntank\ntanker\ntanker\ntank\ntank\ntank\ntan\ntan\ntantalize\ntape\ntape\nTape\ntaper\ntaper\ntaper\ntaper\ntape\ntapestry\ntape\ntape\ntaping\ntape\ntap\ntap\ntap\ntap\ntap\ntap\ntap\ntarget\ntarget\ntarget\ntarget\ntarget\ntarget\ntarget\ntarget\ntariff\ntarnish\ntarnish\ntarnish\ntar\ntartan\ntask\ntassel\ntaste\ntaste\ntaste\ntaste\ntaste\nteach\nteach\ntaunt\ntax-deduction\ntax\ntax\ntax\ntax\ntax\ntax-exempt\ntax-exempt\ntaxpayer\ntaxpayer\ntaxpayer\ntax-reduce\ntax\ntax-writer\nt-bill\nteacher\nteach\nteach\nteach\nteach\nteach\nteam\nteam\nteammate\nteam\nteam\nteam\ntear\ntear\ntear\ntease\nteaspoon\ntechnical-service\ntechnician\ntechnique\ntechnocrat\ntechnology\nteem\nteem\nteen-ager\nteenager\nteens\nteens\nteeter\ntooth\ntelecine\ntelecommunications\ntelecommunications\ntelecuss\ntelegraph\ntelegraph\ntelemarketer\ntelephone\ntelephone-operation\ntelephone\ntelephone\ntelephone\ntelesystem\ntelesystem\ntelevise\ntelevision\ntelex\nteller\ntell\ntell\ntell\ntell\ntell\ntemblor\ntemperature\ntemper\ntemp\ntempt\ntempt\ntempt\ntempt\ntempt\ntenant\ntend\ntend\ntendency\ntender\ntender\ntender\ntender\ntender\ntender\ntender\ntend\ntend\ntend\ntend\ntenet\ntension\nten\ntenth\nterm\nterm\nterminal\nterminal\nterminate\nterminate\nterminate\nterminate\ntermination\nterm\nterm\nterm\nterrify\nterrify\nterritory\nterrorist\ntestament\ntest-drive\ntest-drive\ntest\ntest\ntester\ntest-fire\ntestify\ntestify\ntestify\ntestify\ntestify\ntestify\ntest\ntest\ntest\ntest\ntest\ntest\ntest\ntether\ntextbook\ntextile\ntext\nthank\nthank\nthanks\nthanks\nthanks\nthank\nthank\nthank\nthat\ntheater\ntheft\ntheft\ntheme\ntheme\ntheologian\ntheory\ntheorist\ntheorist\ntheorize\ntheory-teach\ntherapy\ntherapist\nthermometer\nthe\nthe\nthief\nthief\nthing\nthing\nthink\nthink\nthink\nthink\nthink\nthink\nthin\nthin\nthird\nthirty\nthirty\nthistle\nthoroughbred\nthought\nthink\nthink\nthousand\nthousand\nthrash\nthrash\nthrash\nthread\nthreaten\nthreaten\nthreaten\nthreaten\nthreaten\nthreaten\nthreat\nthree-fourth\nthree-quarter\nthree-seventh\nthrow\nthrift\nthrift\nthrill\nthrill\nthrill\nthrive\nthrive\nthrive\nthrive\nthrive\nthrive\nthroat\nthrower\nthrow\nthrow\nthrow\nthrow\nthrow\nthrow\nthrow\nthrust\nthrust\nthrust\nthrust\nthug\nthumb\nthumb\nthumb\nthumb\nthunder\nthwart\nthwart\nthwart\nthwart\ntick\nticket\nticket\nticket\ntick\ntick\ntick\ntidbit\ntide\ntie\ntie\ntie-in\ntie\ntie\ntie-up\ntie\ntiger\ntighten\ntighten\ntighten\ntighten\ntighten\ntile\ntilt\ntilt\ntilt\ntimberland\ntimber\ntime\ntime\ntime-hotel\ntime-share\ntime\ntime\ntime\ntime\ntimpani\ntinge\ntinge\ntinker\ntinker\ntinker\ntin\ntip\ntip\ntipster\ntiptoe\ntiptoe\ntiptoe\ntip\ntire\ntire-kicker\ntire\ntire\ntissue\ntitan\ntitle\ntitle\ntitle\ntitle\ntoast\ntoddler\ntoehold\ntoe\ntoil\ntoiletries\ntoil\ntoil\ntoil\ntell\ntell\ntolerate\ntoll\ntoll\ntomato\ntone\ntone\ntongue\ntonnage\nton\nton\ntake\ntool\ntoot\ntopic\ntop\ntop\ntop\ntop\ntopple\ntopple\ntopple\ntop\ntop\ntop\ntop\ntorch\ntorch\ntear\ntory\ntormentor\ntorment\ntorment\ntornado\ntear\ntorpedo\ntorpedo\ntort\ntorture\ntorture\ntorture\ntoss\ntoss\ntosser\ntoss\ntoss\ntotal\ntotal\ntotal\ntotal\ntotal\ntotal\ntotal\ntotal\ntotal\ntote\ntote\ntote\ntotter\ntotter\ntouch\ntouch\ntouch\ntouch\ntouch\ntouch\ntouch\ntoughen\ntoughen\ntough\ntour\ntour\ntourist\ntournament\ntour\ntour\ntour\ntout\ntout\ntout\ntout\ntout\ntowel\ntower\ntower\ntower\ntownhouse\ntownship\ntown\ntow\ntoy\ntoy\ntrace\ntrace\ntrace\ntrace\ntrace\ntrace\ntrack\ntrack\ntrack\ntrack\ntrack\ntrack\ntrack\ntractor\ntract\ntrade\ntrade\ntrade\ntrademark\ntrade-off\ntradeoff\ntrader\ntrader\ntrade\ntrade\ntrade\ntrade\ntrade\ntrade\ntrade\ntraditionalist\ntradition\ntraduce\ntraduce\ntrafficker\ntragedy\ntrail\ntrail\ntrailer\ntrail\ntrail\ntrail\ntrail\ntrail\ntrain\ntrain\ntrainer\ntrain\ntrain\ntrain\ntrain\ntrain\ntrain\ntraipse\ntrait\ntramp\ntrample\ntrample\ntransact\ntransact\ntransaction\ntransaction\ntransact\ntranscribe\ntranscript\ntransfer\ntransfer\ntransfer\ntransfer\ntransfer\ntransfer\ntransform\ntransform\ntransform\ntransform\ntransform\ntransfusion\ntransistor\ntranslate\ntranslate\ntranslate\ntranslate\ntranslate\ntranslation\ntransmission\ntransmit\ntransmit\ntransmit\ntransmit\ntransmogrify\ntransmogrify\ntransplant\ntransplant\ntransplant\ntransplant\ntransport\ntransport\ntransport\ntransport\ntransport\ntransport\ntransport\ntransport\ntransvestite\ntrap\ntrappings\ntrap\ntrap\ntrash\ntrash\ntrauma\ntraumatize\ntraumatize\ntravail\ntravel\ntravel\ntraveler\ntraveler\ntravel\nTravel\ntravelogue\ntravels\ntravel\ntravel\ntravel\ntravel\ntray\ntreadmill\ntread\ntread\ntreasurer\ntreasure\ntreasure\ntreasury\ntreasury\ntreat\ntreat\ntreat\ntreat\ntreatise\ntreatment\ntreat\ntreat\ntreat\ntreat\ntreble\ntree\ntrek\ntremble\ntremor\ntrench\ntrend\ntrend-setter\ntrend\ntrend\ntrespass\ntrespass\ntrial\ntriangle\ntribe\ntribute\ntrickle\ntrickle\ntrickle\ntrick\ntrick\ntry\ntry\ntry\ntry\ntrigger\ntrigger\ntrigger\ntrigger\ntrigger\ntrigger\ntrigger\ntrillion\ntrill\ntrimester\ntrim\ntrim\ntrim\ntrim\ntrim\ntrim\ntriple\ntriple\ntriple\ntriple\ntriple\ntrip\ntrip\ntriumph\ntrivialize\ntrivia\ntroops\ntroops\ntro\ntrot\ntrot\ntrot\ntrouble\ntrouble\ntrouble\ntroublemaker\ntrouble\ntrouble\ntrouble\ntrouble\ntrough\ntrough\ntroupe\ntrousers\ntrout\ntrucker\ntruck-part\ntruck\ntruck\ntruck\ntrudge\ntrump\ntrumpet\ntrumpet\ntrumpet\ntrundle\ntrunk\ntrust\ntrust\ntrustee\ntrustee\ntrust\ntrust\ntrust\ntrust\ntrust\ntrust\ntruth\ntry\ntry\ntryout\ntry\ntry\ntry\ntry\nt-shirt\nt-shirt\ntube\ntub\ntuck\ntuck\ntuck\ntuck\ntug\ntug\ntuition\ntuition\ntumble\ntumble\ntumble\ntumble\ntumble\ntumble\ntumor\ntumor-suppressor\ntune\ntune\ntune\ntune\ntune\ntunnel\nturban\nturbine\nturboprop\nturmoil\nturnaround\nturn\nturn\nturn\nturn\nturn-on\nturn\nturn\nturn\nturn\nturn\nturn\nturtle\ntusk\ntutor\ntutorial\ntutor\ntuxedo\ntv\ntweezers\ntwenty\ntwiddle\ntwin\ntwist\ntwist\ntwist\ntwist\ntwist\ntwitch\ntwo-hundredth\ntwo-seventh\ntwo-third\ntwothird\ntwo-third\ntycoon\ntie\ntype\ntypeface\ntype\ntypewriter\ntyphoon\ntypify\ntypify\nufo\nulcer\nultimatum\numbrella\nun-advertiser\nunban\nunblock\nuncertainty\nunchlorinate\nuncle\nunconsolidate\nunconsolidate\nuncover\nuncover\nuncover\nuncover\nunderclass\nundercut\nundercut\nundercut\nundercut\nundercut\nunderestimate\nunderestimate\nunderfund\nundergird\nundergo\nundergo\nundergo\nundergo\nunderlie\nunderline\nunderline\nunderlie\nunderlie\nundermine\nundermine\nundermine\nundermine\nundermine\nunderperform\nunderperformer\nunderperform\nunderperform\nunderperform\nunderpin\nunderpin\nunderpin\nunderprice\nunderreact\nunderscore\nunderscore\nunderscore\nunderscore\nunderscore\nunderscore\nunderscore\nundersell\nunderstaff\nunderstand\\/adopt\nunderstand\nunderstand\nunderstand\nunderstand\nunderstate\nunderstate\nunderstate\nunderstate\nunderstand\nunderstand\nundertake\nundertake\nundertaking\nundertake\nundertone\nundertake\nunderutilize\nundervalue\nundervalue\nunderweight\nundergo\nunderwhelm\nunderwiter\nunderwriter\nunderwriter\nunderwrite\nunderwrite\nunderwrite\nunderwriting\nunderwrite\nunderwrite\nunderwrite\nunderwrite\nundo\nundo\nundo\nundulate\nundulate\nunearth\nunexecute\nunfaze\nunfocus\nunfold\nunfold\nunfold\nunfold\nunhinge\nunhock\nunhusk\nunify\nuniform\nunify\nunify\nunify\nunionist\nunionize\nunion\nunion\nunite\nUnite\nunite\nunite\nunitholder\nunitholder\nunit\nunit\nuniversity\nuniversity\nunknown\nunleash\nunleash\nunleash\nunleash\nunleash\nunleash\nunload\nunload\nunload\nunload\nunload\nunlock\nunlock\nunlock\nunmask\nunmask\nunmaterialize\nunnerve\nunnerve\nunplug\nunpolarize\nunravel\nunravel\nunravel\nunravel\nunroll\nunroll\nunseat\nunseat\nunsettle\nunsettle\nunveil\nunveil\nunveil\nunveil\nunveil\nunveil\nunwaver\nunwind\nunwind\nupdate\nupdate\nupdate\nupdate\nupgrade\nupgrade\nupgrade\nupgrade\nupgrade\nupgrade\nupheaval\nuphold\nuphold\nuphold\nuphold\nuphold\nuphold\nup\nup\nuproot\nups-and-downs\nupset\nupset\nupset\nupset\nupset\nupset\nup\nuptick\nural\nurge\nurge\nurge\nurge\nurge\nurge\nurging\nurge\nuse\nuse\nUse\nuser\nuser\nuse\nuse\nuse\nuse\nuse\nusher\nusher\nusher\nusher\nusher\nuse\nUse\nusurp\nu.s.\nutility\nutility\nutilize\nutmost\nutopian\nutterance\nutter\nutter\nuvb\nvacancy\nvacancy\nvacate\nvacate\nvacate\nvacate\nvacationer\nvacation\nvacation\nvaccine\nvacillate\nvacuum\nvagabond\nvagary\nvalidate\nvaluation\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalve\nvandalize\nvanish\nvanish\nvanish\nvanish\nvanish\nvanish\nvanity\nvan\nvapor\nvariable\nvariation\nvary\nvary\nvary\nvariety\nvary\nvary\nvary\nvary\nvase\nvassal\nvault\nvault\nvcr\nveer\nveer\nvegetable\nvegetable\nvegetarian\nvehicle\nveil\nveil\nvend\nvendor\nvent\nventilate\nventilate\nventure\nventure\nventure\nvent\nvenue\nverdict\nverge\nverify\nverify\nversion\nvessel\nvest\nvestment\nvest\nveteran\nveterinarian\nveto\nveto\nveto\nveto\nveto\nhave\nhave\nvex\nviaduct\nvibrate\nvicar\nvice\nvicissitude\nvicitim\nvictimize\nvictim\nvictim\nvictory\nvictory\nvideocassette\nvideodisk\nvideo\nvideotape\nvideotape\nvietnamese\nview\nview\nviewer\nviewer\nviewing\nview\nviewpoint\nview\nview\nview\nview\nview\nvignette\nvillager\nvillage\nvillage\nvillain\nvindicate\nvineyard\nvintage\nvinyl-product\nviolate\nviolate\nviolate\nviolate\nviolate\nviolate\nviolation\nvirgin\nvirtue\nvirtuoso\nvisage\nvisionary\nvision\nvisit\nvisit\nvisit\nvisit\nvisitor\nvisitor\nvisit\nvisit\nvisit\nvisit\nvisualize\nvisualize\nvisual\nvitiate\nvoice\nvoice\nvoice\nvoice\nvoice\nvoice\nvoice\nvoice\nvoid\nvoid\nvolume\nvolunteer\nvolunteer\nvolunteer\nvolunteer\nvomit\nvote\nvote\nvote-getter\nvoter\nvoter\nvote\nvote\nvote\nvote\nvote\nvoucher\nvow\nvow\nvowel\nvowel\nvow\nvow\nvow\nvie\nwaddle\nwade\nwad\nwafer\nwaffle\nwaffle\nwaft\nwage\nwages\nwage\nwage\nwagon\nwag\nwail\nwail\nwait\nwait\nwaiter\nwait\nwait\nwait\nwait\nwait\nwaive\nwaive\nwaiver\nwaiver\nwaive\nwaive\nwaive\nwake\nwake\nwalk\nwalkie-talkie\nwalk\nwalk\nwalk\nwalkout\nwalk\nwalk\nwalk\nwalk\nwalk\nwalk\nwallcovering\nwallet\nwallop\nwallow\nwall\nwall\nwander\nwander\nwander\nwander\nwane\nwane\nwane\nwane\nwane\nwane\nwane\nwant\nwant\nwant\nwant\nwant\nwant\nwant\nward\nwarden\nward\nwarehouse\nwares\nwarhead\nwarm\nwarm\nwarm\nwarm\nwarm\nwarn\nwarn\nwarn\nwarn\nwarner\nwarning\nwarn\nwarn\nwarn\nwarn\nwarrant\nwarranty\nwarrant\nwarrant\nwarrant\nwarrant\nwarren\nwar\nwarrior\nwar\nwar\nwar\nwart\nwar\nwash\nwash\nwash\nwash\nwash\nwaste\nwaste\nwaste\nwaste\nwaste\nwaste\nbe\nbe\nbe\nwatchdog\nwatch\nwatch\nwatcher\nwatch\nwatch\nwatch\nwatch\nwatch\nwatch\nwatch\nwater\nwaterfall\nwater\nwater\nwater\nwaterworks\nwatt\nwave\nwavelength\nwaver\nwaver\nwave\nwave\nwax\nwax\nway\nway\nweaken\nweaken\nweaken\nweaken\nweaken\nweaken\nweaken\nweaken\nweakness\nwean\nweapon\nwear\nwear\nwear\nwear\nwear\nwear\nweasle\nweather\nweather\nweave\nweave\nweb\nwed\nwedding\nwedge\nwedge\nwedge\nwednesday\nwed\nwed\nweed\nweekday\nweekend\nweekly\nweeknight\nweek\nweeper\nweep\nweigh\nweigh\nweigh\nweigh\nweigh\nweight\nweighting\nweight\nweight\nweigh\nweigh\nwelcome\nwelcome\nwelcome\nwelcome\nwelcome\nwelcome\nwelcome\nweld\nwell\nwell-state\nwell-wisher\ngo\ngo\nweep\nbe\nbe\nbe\nbe\nwe\nwhack\nwhack\nwhack\nwhale\nwheelbase\nwheel\nwheel\nwheeze\nwhen-issue\nwhile\nwhimper\nwhim\nwhip\nwhipping\nwhip\nwhipsaw\nwhipsaw\nwhip\nwhip\nwhirlwind\nwhir\nwhisk\nwhisper\nwhisper\nwhisper\nwhistle\nwhistle\nwhistle\nwhistle\nwhites\nwhitewash\nwhittle\nwhizz\nwholesaler\nwholesaler\nwholesale\nwhoop\nwhoosh\nwiden\nwiden\nwiden\nwiden\nwiden\nwiden\nwiden\nwidget\nwidow\nwidow\nwield\nwield\nwield\nwield\nwiggle\nwiggle\nwiggle\nwig\nwilfr\nwillie\nwill\nwill\nwill\nwimp\nwimp\nwindfall\nwind\nwindow\nwindshield\nwind\nwind\nwind\nwinery\nwine\nwing\nwink\nwinner\nwinner\nwin\nwin\nwin\nwin\nwinter\nwin\nwin\nwipe\nwipe\nwipe\nwipe\nwipe\nwire\nwire\nwiretap\nwire\nwisecrack\nwish\nwish\nwish\nwish\nwish\nwish\nwish-list\nwish\nwish\nwitch\nwithdrawal\nwithdraw\nwithdraw\nwithdraw\nwithdraw\nwither\nwither\nwithhold\nwithhold\nwithholding\nwithhold\nwithhold\nwithhold\nwithhold\nwithstand\nwithstand\nwithstand\nwithstand\nwitness\nwitness\nwitness\nwitness\nwitness\nwife\nwife\nwizard\nwoe\nwake\nwake\nwolf\nwomanize\nwoman\nwoman\nwonderbar\nwonder\nwonder\nwonder\nwonder\nwonder\nwonder\nwon\nwin\nwin\nwin\nwoodchuck\nwood-product\nwood\nwoo\nwoo\nwoo\nword\nword-processing\nword\nwear\nworkbook\nworkday\nwork\nwork\nworker\nworker\nworker\nworkings\nwork\nwork\nwork\nworkman\nworksheet\nworksheet\nwork\nworkstation\nwork\nwork\nwork\nwork\nwork\nworld-affair\nworld\nworm\nwear\nwear\nworry\nworry\nworry\nworrier\nworry\nworry\nworry\nworry\nworry\nworry\nworsen\nworsen\nworsen\nworsen\nwound\nwound\nwind\nwind\nwind\nweave\nwow\nwrack\nwrack\nwrack\nwrangle\nwrap\nwrap\nwrapper\nwrap\nwrap\nwrap\nwreak\nwreak\nwreak\nwreck\nwreck\nwreck\nwrench\nwrench\nwrest\nwrestler\nwrestle\nwrestle\nwrestle\nwrest\nwriggle\nwring\nwrist\nwrite-downs\nwritedown\nwrite-off\nwriteoff\nwriter\\/producers\nwriter\nwrite\nwrite\nwrite\nwrite\nwrithe\nwritings\nwrite\nwrite\nwrite\nwrite\nwrong\nwrite\nwrite\nwork\nwring\nx-ray\nyacht\nyank\nyank\nyank\nyank\nyard\nyearbook\nyearling\nyear\nyear\nyear\nyell\nyell\nyellow\nyell\nyell\nyelp\nyen\nyield\nyield\nyield\nyield\nyield\nyield\nyield\nyield\nyoungster\nyounker\nyouth\nyuppie\nzapper\nzap\nzero\nzero\nzero\nzero\nzigzag\nzig-zag\nzip\nzip\nzloty\nzombie\nzone\nzone\nzone\nzoom\nzoom\nzoom\n" }, { "path": "ccgbank/data/wsj_0595Corrected.auto", "content": "ID=wsj_0595.1 PARSER=GOLD NUMPARSE=1\n( ( () ) ( ( () () ) ( () ( () ( ( () ) ( () ( () ) ) ) ) ) ) ) \nID=wsj_0595.2 PARSER=GOLD NUMPARSE=1\n( ( ( () ( ( () ) ( ( () () ) ( ( () () ) ) ) ) ) ( () ( () ( () ( () ( () ( () ( ( () ) ( ( () ( ( () ( () () ) ) ( ( () () ) ( () ( () () ) ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.3 PARSER=GOLD NUMPARSE=1\n( ( ( ( () ( () ( () ( () ( () ( () ( () () ) ) ) ) ) ) ) ) ( () ( ( () ( () () ) ) ( () ( ( ( () () ) ) ( () ( ( ( ( ( ( () () ) () ) () ) ) ( () ( () ) ) ) ( () ( ( () () ) ( () ( ( () ( () () ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.4 PARSER=GOLD NUMPARSE=1\n( ( ( () () ) ( () ( ( ( () ( () () ) ) ( () ( () ( ( () ( () () ) ) ) ) ) ) ( () ( ( () ( () () ) ) ( ( ( () ) () ) ) ) ) ) ) ) () ) \nID=wsj_0595.5 PARSER=GOLD NUMPARSE=1\n( ( ( () () ) ( () ( ( () ) ( () ( ( () ( () ( () () ) ) ) ( () ( () ( ( ( ( () () ) () ) () ) ( () ( ( ( ( ( () () ) () ) ) ( () ( () ( () ( () ( () ( () () ) ) ) ) ) ) ) ( () ( () ( ( ( () () ) ) ( () ( ( () ) ( () ( ( ( () ( () ) ) ) ( () ( ( ( () () ) ) ( () ( ( () ( () () ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.6 PARSER=GOLD NUMPARSE=1\n( ( () ( () ( ( () ) ( () ( ( () () ) ) ) ) ) ) ( ( ( () ( () ) ) ( () ( ( () ) ( () ( () ) ) ) ) ) () ) ) \nID=wsj_0595.7 PARSER=GOLD NUMPARSE=1\n( ( () ( ( () () ) ( () ( () ( ( () ( ( ( ( () ) () ) ( () () ) ) ( ( () ( ( ( () () ) () ) () ) ) () ) ) ) ( () ( ( ( ( () () ) () ) ( () () ) ) ( () ( ( () ( ( () ( ( ( () () ) () ) () ) ) ( () ( () ( () ( () () ) ) ) ) ) ) ( () ( () ( ( () ( ( ( () () ) () ) () ) ) ( () () ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.8 PARSER=GOLD NUMPARSE=1\n( ( ( () () ) ( ( () ( () () ) ) ( () ( ( ( () ( () () ) ) () ) ) ) ) ) () ) \nID=wsj_0595.9 PARSER=GOLD NUMPARSE=1\n( ( ( () ( () () ) ) ( () ( ( () ) ( ( () () ) ( ( ( ( ( () () ) () ) () ) ) ( () ( () ( ( () ) ( () ( ( () ( () ( () () ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.10 PARSER=GOLD NUMPARSE=1\n( ( () ( ( ( () () ) ( () ( ( ( () () ) () ) ) ) ) ( () ( ( ( ( ( () () ) () ) () ) ) ( () ( () ( () ( ( () () ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.11 PARSER=GOLD NUMPARSE=1\n( ( () ( ( ( () () ) ( () ( ( ( ( ( () () ) ) ) () ) () ) ) ) ( () ( ( ( () ) ( () ( ( () () ) ( () ( ( () () ) ) ) ) ) ) ( () ( () ( () ( () ( () ( () ( () ( () () ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.13 PARSER=GOLD NUMPARSE=1\n( ( ( () ( () () ) ) ) ( () ( () ( () ( ( () () ) ) ) ) ) ) \nID=wsj_0595.14 PARSER=GOLD NUMPARSE=1\n( ( ( ( ( ( () () ) ( () ( () ( () () ) ) ) ) ) ( ( () () ) ( () ( ( () () ) () ) ) ) ) ( () ( ( ( () ( () () ) ) ) () ) ) ) () ) \nID=wsj_0595.16 PARSER=GOLD NUMPARSE=1\n( ( ( ( () () ) ) ( ( () () ) ( () ( () ( ( ( ( () ( () ( () ( () () ) ) ) ) ) ( () ( () ( () ( () ( () () ) ) ) ) ) ) ( () ( () ( () ( ( ( () () ) ( () ( () ) ) ) ( () ( () ( () ( ( () () ) ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.17 PARSER=GOLD NUMPARSE=1\n( ( () ( ( ( ( () () ) ) ( () ( ( ( ( () ( () ( () ( () () ) ) ) ) ) ( () ( () ( () () ) ) ) ) ( () ( ( ( () () ) ) ( () ( () ( () () ) ) ) ) ) ) ) ) ( () ( () ( ( ( ( () ( ( () () ) () ) ) ( ( () ( ( ( ( () () ) () ) () ) ) ) () ) ) ( () ( () ) ) ) ( ( ( () () ) ( () ( ( () () ) ) ) ) ( () ( ( ( ( () () ) ( () ( ( () () ) ) ) ) ( () ( () ( () ( () ( () () ) ) ) ) ) ) ( () ( () ( () ( () ( () () ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.18 PARSER=GOLD NUMPARSE=1\n( () ( ( () () ) ( ( ( ( ( ( () () ) ( () ( () ) ) ) ( () ( () ( () ( () ( () () ) ) ) ) ) ) ( () ( ( () ( ( () () ) () ) ) ( () ( ( () () ) () ) ) ) ) ) () ) () ) ) ) \nID=wsj_0595.19 PARSER=GOLD NUMPARSE=1\n( ( () ( ( ( () ( ( () () ) () ) ) ( ( () ( ( () () ) ) ) () ) ) ( ( () ( ( () () ) ) ) ( () ( ( () () ) ( () ( () ( () ( ( () () ) () ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.20 PARSER=GOLD NUMPARSE=1\n( ( ( ( () () ) ) ( ( ( () ( () ( () ( () ( () () ) ) ) ) ) () ) ( () ( ( () () ) ( () ( ( () () ) ( () ( ( () ) ( () ( ( ( () () ) ) ( () ( () ( () () ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.21 PARSER=GOLD NUMPARSE=1\n( ( ( () () ) ( () ( () ( () ( ( ( ( () () ) () ) ) ( ( () ( () ( () ( () ( () ( () () ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.22 PARSER=GOLD NUMPARSE=1\n( ( ( () ( () ( () ( () ( () () ) ) ) ) ) ( ( ( ( () () ) () ) () ) ( () ( ( ( ( () ( () ( () () ) ) ) ) ( () ( () ( ( ( () () ) () ) ) ) ) ) ( () ( () ( ( ( () ( () ( () () ) ) ) ) ( ( () ( () () ) ) ( () ( ( () ( ( () () ) () ) ) ( () ( ( () () ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.23 PARSER=GOLD NUMPARSE=1\n( ( () ( ( () ( ( ( () ) () ) () ) ) ( ( () ( () ( () ( () ( () ( () ( () () ) ) ) ) ) ) ) ( () ( ( () () ) ( () ( () ( ( () ( () () ) ) ( () ( ( ( () () ) ( ( () () ) ) ) ( () ( ( () () ) ( () ( ( () ) ( () ( ( () () ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.24 PARSER=GOLD NUMPARSE=1\n( ( ( () ( () () ) ) ) ( () ( ( () () ) ) ) ) \nID=wsj_0595.25 PARSER=GOLD NUMPARSE=1\n( ( ( ( () ( () ( () ( () ( () ( () () ) ) ) ) ) ) ( ( ( () ( ( () () ) ) ) ( () ( ( ( ( () () ) ) () ) () ) ) ) ) ) ( () ( ( ( () ( () ( () ( () ( ( () ( () () ) ) ) ) ) ) ) () ) ( () ( () ( ( () ) ( () ( ( () ( () ( () ( () () ) ) ) ) ( () ( ( () () ) ( () ( () ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.26 PARSER=GOLD NUMPARSE=1\n( ( ( ( () ( () () ) ) ) ( () ( () ( () ( ( () ( () () ) ) ( () ( () ( () ( ( () () ) ( () () ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.27 PARSER=GOLD NUMPARSE=1\n( ( ( ( ( () ) () ) ( ( () () ) ( () () ) ) ) ( ( ( () ( () ( () () ) ) ) () ) ( ( ( () () ) ( () ( () () ) ) ) ( () ( ( () () ) ) ) ) ) ) () ) \nID=wsj_0595.28 PARSER=GOLD NUMPARSE=1\n( ( () ( () ( ( () ( ( () ( () ( () () ) ) ) ) ) ( () ( () ( ( () () ) ( () ( ( () ( ( ( () ) ( () ( () ) ) ) ( () () ) ) ) ( () ( () ( ( ( () () ) ) ( () ( ( ( () ( () ( () () ) ) ) () ) ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.29 PARSER=GOLD NUMPARSE=1\n( ( ( ( () ) ( () ( ( () ( () () ) ) ( () ( () ( () ( () ( () ( () ( () ( () ( () ( () () ) ) ) ) ) ) ) ) ) ) ) ) ) ( () ( () ( ( ( ( () ( () () ) ) ) ( () ( () ( ( () ( () () ) ) ( () () ) ) ) ) ) ( () ( ( ( ( () () ) () ) ( () ( ( () ( () () ) ) ) ) ) ( () ( () ( ( ( () ( ( () ( () () ) ) ( () () ) ) ) ( () ( ( ( () ) () ) () ) ) ) ( () ( () ( ( () () ) ( () ( () () ) ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.30 PARSER=GOLD NUMPARSE=1\n( ( () ( ( () () ) ( ( ( () () ) ( () ( () () ) ) ) ( () ( ( () ( () ( () () ) ) ) ( () ( () ( () ( () ( () () ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.31 PARSER=GOLD NUMPARSE=1\n( ( ( () ( () ) ) ( () ( ( ( ( () () ) ) ( () ( ( () ( ( () ( () () ) ) ( () () ) ) ) ( () ( () ( () ( () () ) ) ) ) ) ) ) ( () ( ( () ( () ( () () ) ) ) ( ( ( () ( () ) ) ( () ( ( () ( () ( ( () () ) ) ) ) ( () ( () ( () ( ( () () ) ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.32 PARSER=GOLD NUMPARSE=1\n( ( () ( () ( () ( ( ( ( ( () () ) ) ( () ( () ( () ( () () ) ) ) ) ) ( () ( () ( () ( () ( () () ) ) ) ) ) ) ( () ( ( ( () () ) ) ( () ( () ( () ( () ( () () ) ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.33 PARSER=GOLD NUMPARSE=1\n( ( ( ( () ( () ( () ( ( () ( () () ) ) ( () ( () ( () ( () ( ( () ) ( ( () () ) ( () ( ( () ( () ( () ) ) ) ( () ( () ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ( () ( ( ( ( () () ) ) ( () ( () ( () () ) ) ) ) ( ( () () ) ( () ( () ( ( ( ( ( () () ) () ) ) ( () ( ( () ( () ( () () ) ) ) ) ) ) ( () ( () ( () () ) ) ) ) ) ) ) ) ) ) () ) \nID=wsj_0595.34 PARSER=GOLD NUMPARSE=1\n( ( () ( () () ) ) ) \nID=wsj_0595.35 PARSER=GOLD NUMPARSE=1\n( ( ( ( ( ( ( () ( () () ) ) ( () ( () () ) ) ) ) ( () ( ( ( () () ) ( () ( () ( () ( () ) ) ) ) ) ( () ( ( () () ) ) ) ) ) ) ( () ( ( ( () () ) ( () ( ( () () ) ) ) ) () ) ) ) () ) () ) \nID=wsj_0595.36 PARSER=GOLD NUMPARSE=1\n( ( ( ( ( () () ) ) ( () ( ( () ( () ( () () ) ) ) ( () ( () ( ( () () ) ( () ( ( () () ) ) ) ) ) ) ) ) ) ( () ( ( () ( () ( () () ) ) ) () ) ) ) () ) \nID=wsj_0595.37 PARSER=GOLD NUMPARSE=1\n( ( () ( () ( ( ( () () ) ( () ( ( ( () () ) ) ( () ( ( () () ) ( () ( () ( () () ) ) ) ) ) ) ) ) ( () ( ( ( ( () ( () ( ( () () ) ) ) ) ( () ( () ( () ( () () ) ) ) ) ) () ) ( ( () ( () ( () () ) ) ) ( () ( () ( ( () ( ( ( ( () () ) () ) ) ( () ( () ( () () ) ) ) ) ) ( () ( ( ( () () ) ) ( () ( () ) ) ) ) ) ) ) ) ) ) ) ) ) () ) \n" }, { "path": "ccgbank/extract/add-chunks.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n\r\n\r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "ccgbank/extract/convert-to-graph.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "ccgbank/extract/convert-to-hlds.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n Error: node with id = should be a reference (with idref)\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "ccgbank/extract/grammar.xml", "content": "\n\n\n \n \n \n\n \n\n \n \n \n \n\n \n \n \n \n\n\n" }, { "path": "ccgbank/extract/raise-nodes.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n Next\r\n \r\n \r\n\r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "ccgbank/models/hypertagger/ht-prior.flm", "content": "\n## A prior probability model that estimates p(supertag | word, pos)\n## with smoothed back-off (a \"soft tagging dictionary\" if you will).\n\n1\n \n## lexical category (\"supertag\") (T) given POS tag (P) and word (W).\n\nT : 2 P(0) W(0) t_p0w0.count t_p0w0.lm 3\n P0,W0 W0 wbdiscount gtmin 15 \n P0 P0 wbdiscount\n 0 0 wbdiscount\n" }, { "path": "ccgbank/models/hypertagger/ht.config", "content": "# location of FLM file and vocab file for prior model\n#priorModel=ht-prior.flm\n#priorModelVocab=vocab.st\n\n# maxent model (trained using ZL's toolkit) for hypertagging\nmaxentModel=ht.mod\n\n# beam width for beta-best search through the tags returned by the maxent model\nbetas = 0.075 0.03 0.01 0.003 0.001 0.0003 0.0001 0.00001\n\n# if using tagging dictionaries (instead of prior models) specify the dictionary threshold\n# (\"K\" value from Clark 2002)\ndictk=10\n\n# specify tagging dictionaries. these are only used if prior models are NOT specified!\n#wDict=word.dict.min10\n#posDict=pos.dict.min10\n\n# filename for configuration of POS tagger\nposConfig=pos.config\n\n# arg names and short names\nargnames = Arg0:A0 Arg1:A1 Arg1a:A1a Arg1b:A1b Arg2:A2 Arg2a:A2a Arg2b:A2b Arg3:A3 Arg4:A4 Arg5:A5 \n" }, { "path": "ccgbank/models/hypertagger/ht2.config", "content": "# location of FLM file and vocab file for prior model\n#priorModel=stprior.flm\n#priorModelVocab=vocab.st\n\n# maxent model (trained using ZL's toolkit) for hypertagging\nmaxentModel=ht.mod\n# second-pass model\nmaxentModel2=ht2.mod\n\n# beam width for beta-best search through the tags returned by the maxent model\n# these will be applied to both hypertagging models\nbetas = 0.075 0.03 0.01 0.003 0.001 0.0003 0.0001 0.00001\n\n# if using tagging dictionaries (instead of prior models) specify the dictionary threshold\n# (\"K\" value from Clark 2002)\n#dictk=10\n\n# specify tagging dictionaries. these are only used if prior models are NOT specified!\n#wDict=word.dict.min10\n#posDict=pos.dict.min10\n\n# filename for configuration of POS tagger\nposConfig=pos.config\n\n# arg names and short names\nargnames = Arg0:A0 Arg1:A1 Arg1a:A1a Arg1b:A1b Arg2:A2 Arg2a:A2a Arg2b:A2b Arg3:A3 Arg4:A4 Arg5:A5\n" }, { "path": "ccgbank/models/hypertagger/ht2.train.config", "content": "# config file for generating events during realizer training\n#\n# a single beta level (not the most strict one) is used, rather \n# than adaptive backoff (todo: try reverse adaptation)\n\n# maxent model (trained using ZL's toolkit) for hypertagging\nmaxentModel=ht.mod\n# second-pass model\nmaxentModel2=ht2.mod\n\n# beam width for beta-best search through the tags returned by the maxent model\n# these will be applied to both hypertagging models\nbetas = 0.001\n\n# filename for configuration of POS tagger\nposConfig=pos.config\n" }, { "path": "ccgbank/models/hypertagger/pos.config", "content": "# this model gives priors on POS tags.\n#priorModel=posprior.flm\n#priorModelVocab=vocab.pos\n\n# this is a Zhang Le-style MEM.\nmaxentModel=pos.mod\n\n# you can also configure the tagger's beta-value here using the key \"beta\"\n" }, { "path": "ccgbank/models/hypertagger/posprior.flm", "content": "\n## A prior probability model that estimates p(pos | word)\n## with smoothed back-off (a \"soft tagging dictionary\" if you will).\n\n1\n \n## POS tag (P) given word (W) with a back-off to the prior on the POS itself.\n\nP : 1 W(0) p_w0.count p_w0.lm 2\n W0 W0 wbdiscount gtmin 1 \n 0 0 wbdiscount gtmin 1 \n" }, { "path": "ccgbank/models/hypertagger/vocab.flm", "content": "\n## flm file for determining the vocab\n\n3\n \n## word (W) unigram\n\nW : 0 w.count w.lm 1\n 0 0\n\n## supertag (T) unigram\n\nT : 0 t.count t.lm 1\n 0 0\n\n## pos tag (P) unigram \n\nP : 0 p.count p.lm 1\n 0 0\n\n\n" }, { "path": "ccgbank/models/parser/binary.flm", "content": "\n## binary step of Hockenmaier's HWDep generative syntactic model\n## with added pos tags\n\n6\n \n## expansion (E) given parent cat (P), lexcat parent (CP), head postag parent (T) and headword parent (W)\n\nE : 4 P(0) CP(0) T(0) W(0) e_p0cp0t0w0.count e_p0cp0t0w0.lm 5\n P0,CP0,T0,W0 W0 wbdiscount gtmin 10\n P0,CP0,T0 T0 wbdiscount\n P0,CP0 CP0 wbdiscount\n P0 P0 wbdiscount\n 0 0 wbdiscount\n\n## head (H) given expansion (E), parent cat (P) and lexcat parent (CP)\n\nH : 3 E(0) P(0) CP(0) h_e0p0cp0.count h_e0p0cp0.lm 4\n E0,P0,CP0 CP0 wbdiscount\n E0,P0 P0 wbdiscount\n E0 E0 wbdiscount\n 0 0 wbdiscount\n\n## sibling (S) given expansion (E), parent cat (P), head cat (H), lexcat parent (CP), head postag parent and headword parent (W)\n\nS : 6 E(0) P(0) H(0) CP(0) T(0) W(0) s_e0p0h0cp0t0w0.count s_e0p0h0cp0t0w0.lm 7\n E0,P0,H0,CP0,T0,W0 W0 wbdiscount gtmin 10\n E0,P0,H0,CP0,T0 T0 wbdiscount\n E0,P0,H0,CP0 CP0 wbdiscount\n E0,P0,H0 H0 wbdiscount\n E0,P0 P0 wbdiscount\n E0 E0 wbdiscount\n 0 0 wbdiscount\n\n## lexcat sibling (CS) given expansion (E), sibling (S), parent cat (P) and head cat (H)\n\nCS : 4 E(0) S(0) P(0) H(0) cs_e0s0p0h0.count cs_e0s0p0h0.lm 5\n E0,S0,P0,H0 H0 wbdiscount\n E0,S0,P0 P0 wbdiscount\n E0,S0 S0 wbdiscount\n E0 E0 wbdiscount\n 0 0 wbdiscount\n\n## head postag sibling (TS) given lexcat sibling (CS), sibling (S), parent cat (P), head cat (H), head postag parent (T) and headword parent (W)\n\nTS : 6 CS(0) S(0) P(0) H(0) T(0) W(0) ts_cs0s0p0h0t0w0.count ts_cs0s0p0h0t0w0.lm 7\n CS0,S0,P0,H0,T0,W0 W0 wbdiscount gtmin 10\n CS0,S0,P0,H0,T0 T0 wbdiscount\n CS0,S0,P0,H0 H0 wbdiscount\n CS0,S0,P0 P0 wbdiscount\n CS0,S0 S0 wbdiscount\n CS0 CS0 wbdiscount\n 0 0 wbdiscount\n\n## headword sibling (WS) given lexcat sibling (CS), sibling (S), parent cat (P), head cat (H), head postag parent (T), headword parent (W) and head postag sibling (TS) \n\nWS : 7 CS(0) S(0) P(0) H(0) T(0) W(0) TS(0) ws_cs0s0p0h0t0w0ts0.count ws_cs0s0p0h0t0w0ts0.lm 8\n CS0,S0,P0,H0,T0,W0,TS0 TS0 wbdiscount gtmin 5\n CS0,S0,P0,H0,T0,W0 W0 wbdiscount gtmin 10\n CS0,S0,P0,H0,T0 T0 wbdiscount gtmin 5\n CS0,S0,P0,H0 H0 wbdiscount\n CS0,S0,P0 P0 wbdiscount\n CS0,S0 S0 wbdiscount\n CS0 CS0 wbdiscount\n 0 0 wbdiscount\n\n" }, { "path": "ccgbank/models/parser/gen-events.prefs", "content": "\n\n" }, { "path": "ccgbank/models/parser/leaf.flm", "content": "\n## leaf step of Hockenmaier's HWDep generative syntactic model\n## with added pos tags\n\n1\n \n## expansion (E) given parent cat (P), lexcat parent (CP), head postag parent (T) and headword parent (W)\n\nE : 4 P(0) CP(0) T(0) W(0) e_p0cp0t0w0.count e_p0cp0t0w0.lm 5\n P0,CP0,T0,W0 W0 wbdiscount gtmin 10\n P0,CP0,T0 T0 wbdiscount\n P0,CP0 CP0 wbdiscount\n P0 P0 wbdiscount\n 0 0 wbdiscount\n\n\n" }, { "path": "ccgbank/models/parser/model.init", "content": "1\ngenlogprob 1\n" }, { "path": "ccgbank/models/parser/parse.prefs", "content": "\n\n" }, { "path": "ccgbank/models/parser/top.flm", "content": "\n## top step of Hockenmaier's HWDep generative syntactic model\n## with added pos tags\n\n4\n \n## head (H) given expansion (E), parent cat (P) and lexcat parent (CP)\n## nb: E, P and CP should all equal when invoked\n\nH : 3 E(0) P(0) CP(0) h_e0p0cp0.count h_e0p0cp0.lm 4\n E0,P0,CP0 CP0 wbdiscount\n E0,P0 P0 wbdiscount\n E0 E0 wbdiscount\n 0 0 wbdiscount\n\n## lexcat top (CT) given parent cat (P)\n\nCT : 1 P(0) ct_p0.count ct_p0.lm 2\n P0 P0 wbdiscount\n 0 0 wbdiscount\n\n## head postag top (TT) given lexcat parent (CP)\n\nTT : 1 CP(0) tt_cp0.count tt_cp0.lm 2\n CP0 CP0 wbdiscount\n 0 0 wbdiscount\n\n## headword top (WT) given lexcat parent (CP), head postag top (TT)\n\nWT : 2 CP(0) TT(0) wt_cp0tt0.count wt_cp0tt0.lm 3\n CP0,TT0 TT0 wbdiscount gtmin 5\n CP0 CP0 wbdiscount\n 0 0 wbdiscount\n\n" }, { "path": "ccgbank/models/parser/unary.flm", "content": "\n## unary step of Hockenmaier's HWDep generative syntactic model\n## with added pos tags\n\n2\n \n## expansion (E) given parent cat (P), lexcat parent (CP), head postag parent (T) and headword parent (W)\n\nE : 4 P(0) CP(0) T(0) W(0) e_p0cp0t0w0.count e_p0cp0t0w0.lm 5\n P0,CP0,T0,W0 W0 wbdiscount gtmin 10\n P0,CP0,T0 T0 wbdiscount\n P0,CP0 CP0 wbdiscount\n P0 P0 wbdiscount\n 0 0 wbdiscount\n\n## head (H) given expansion (E), parent cat (P) and lexcat parent (CP)\n\nH : 3 E(0) P(0) CP(0) h_e0p0cp0.count h_e0p0cp0.lm 4\n E0,P0,CP0 CP0 wbdiscount\n E0,P0 P0 wbdiscount\n E0 E0 wbdiscount\n 0 0 wbdiscount\n\n" }, { "path": "ccgbank/models/parser/vocab.flm", "content": "\n## flm file for determining the vocab\n\n13\n \n## headword (W) unigram\n\nW : 0 w.count w.lm 1\n 0 0\n\n## headword top (WT) \n\nWT : 0 wt.count wt.lm 1\n 0 0\n\n## headword sibling (WS)\n\nWS : 0 ws.count ws.lm 1\n 0 0\n\n## expansion (E)\n\nE : 0 e.count e.lm 1\n 0 0\n\n## head (H) cat\n\nH : 0 h.count h.lm 1\n 0 0\n\n## parent (P) cat\n\nP : 0 p.count p.lm 1\n 0 0\n\n## lexcat parent (CP)\n\nCP : 0 cp.count cp.lm 1\n 0 0\n\n## lexcat top (CT)\n\nCT : 0 ct.count ct.lm 1\n 0 0\n\n## head postag top (TT)\n\nTT : 0 tt.count tt.lm 1\n 0 0\n\n## head postag parent (T)\n\nT : 0 t.count t.lm 1\n 0 0\n\n## sibling (S) cat\n\nS : 0 s.count s.lm 1\n 0 0\n\n## lexcat sibling (CS)\n\nCS : 0 cs.count cs.lm 1\n 0 0\n\n## head postag sibling (TS)\n\nTS : 0 ts.count ts.lm 1\n 0 0\n\n\n" }, { "path": "ccgbank/models/realizer/alph.init", "content": "6\ngenlogprob 1\n$ngram0 1\n$ngram1 1\n$ngram2 1\n$ngram3 1\n$deplen 1\n" }, { "path": "ccgbank/models/realizer/gen-events.prefs", "content": "\n\n" }, { "path": "ccgbank/models/realizer/model.init", "content": "2\ngenlogprob 1\n$ngram0 1\n" }, { "path": "ccgbank/models/realizer/rz-test.prefs", "content": "\n\n" }, { "path": "ccgbank/models/realizer/stp3.flm", "content": "\n## Supertags FLM\n\n## Supertag (T) based on POS tags (P), plus POS trigram\n\n2\n\n## POS trigram\nP : 2 P(-1) P(-2) p_p1p2.count p_p1p2.lm 3\n P1,P2 P2 kndiscount\n P1 P1 kndiscount\n 0 0 ndiscount\n\n## 3gram with prev two POSs\nT : 3 P(0) P(-1) P(-2) t_p0p1p2.count t_p0p1p2.lm 4\n P0,P1,P2 P2 kndiscount\n P0,P1 P1 kndiscount\n P0 P0 kndiscount\n 0 0 kndiscount\n\n" }, { "path": "ccgbank/models/supertagger/pos.config", "content": "# an example POS tagger config file (D.N. Mehay)\n# change to suit your needs (e.g., replace the following paths\n# with paths that point to the relevant files).\n\n# \"basic\" = Ratnaparkhi-style features, inter alia\n# there is nothing else in OpenCCG, at the moment.\ntaggerType=basic\n\n# this model gives priors on POS tags.\npriorModel=posprior.flm\npriorModelVocab=vocab.pos\n\n# this is a Zhang Le-style MEM.\nmaxentModel=pos.mod\n\n# this last must be an ARPA-formatted n-gram model over POS tags (with and )\n# (7- to 9-grams work well, without much memory usage).\nsequenceModel=pos.lm\n\n# tagging beam width (beta=1 means \"single-best\", i.e., unitagging)\nbeta=0.044\n#beta=0.1\n#beta=1.0\n\n# tagging algorithm. (choose from 'forward' and 'forward-backward')\n# the former is faster, the latter is better.\ntaggingAlgorithm=forward-backward\n\n# self-explanatory (will potentially print certain errors and warnings, if set to 'true')\nverbose=true" }, { "path": "ccgbank/models/supertagger/posprior.flm", "content": "\n## A prior probability model that estimates p(pos | word)\n## with smoothed back-off (a \"soft tagging dictionary\" if you will).\n\n1\n \n## POS tag (P) given word (W) with a back-off to the prior on the POS itself.\n\nP : 1 W(0) p_w0.count p_w0.lm 2\n W0 W0 wbdiscount gtmin 1 \n 0 0 wbdiscount gtmin 1 \n" }, { "path": "ccgbank/models/supertagger/st.config", "content": "# an example supertagger config file (D.N. Mehay)\n# change to suit your needs (e.g., replace the following paths\n# with paths that point to the relevant files).\n\n# this is a comment (\"basic\" means C,C & Vadas (2006)-style features).\n# there is nothing else in OpenCCG, at the moment.\ntaggerType=basic\n\n# this model gives priors on supertags.\n#priorModel=stprior.flm\n#priorModelVocab=vocab.st\n\n# this is a Zhang Le-style MEM.\nmaxentModel=st.mod\n\n# this last must be an ARPA-formatted n-gram model over supertags (with and )\n# (7- to 9-grams work well, without much memory usage).\nsequenceModel=st.lm\n\n# tagging beam widths (first try beta1, then beta2 if that doesn't give a parse, etc.).\n# give as many as you want, but keep in mind that your parser will try them all out\n# (which may make it try to parse in vain -- i.e., when it just doesn't have the\n# categories to do it).\n#betas = 0.024 0.003875 0.001225 0.0005377 0.000275 0.0000925 0.00004\n#betas = 0.075 0.03 0.01 0.005 0.001 0.0005\nbetas = 0.075 0.03 0.01 0.003 0.001 0.0003 0.0001 0.00001\n\n# if not using the prior model (above), you must give 'K' values (see Clark and Curran (2007)).\n# the first one is for all beta values but the last. the second one is for\n# the last.\nfirstK=20\nlastK=100\n\n# also, if you're not using the prior models, you need to specify a word-keyed tagging\n# dictionary (this interacts with the 'K' values) and a POS-keyed tagging dictionary.\nwDict=word.dict.min10\nposDict=pos.dict.min10\n\n# use automatic POS features? (even if you are automatically POS tagging,\n# but only using single-best, set this to false).\nautoPOS=true\n\n# if autoPOS=true, you need to specify a POS config file.\nposConfig=pos.config\n\n# tagging algorithm. (choose from 'forward' and 'forward-backward')\n# the former is faster, the latter is better.\ntaggingAlgorithm=forward-backward\n\n# self-explanatory (will print certain errors and warnings, if set to 'true')\nverbose=true\n\n# option to include gold tags\n#includeGold=true\n\n# POS-specific relaxation\n#betaMultipliers=, 0.01 : 0.01 . 0.1 CC 0.1\nbetaMultipliers=, 0.01 : 0.01 . 0.1\n" }, { "path": "ccgbank/models/supertagger/st.config.train", "content": "# an example supertagger config file (D.N. Mehay)\n# change to suit your needs (e.g., replace the following paths\n# with paths that point to the relevant files).\n\n# this is a comment (\"basic\" means C,C & Vadas (2006)-style features).\n# there is nothing else in OpenCCG, at the moment.\ntaggerType=basic\n\n# this model gives priors on supertags.\n#priorModel=stprior.flm\n#priorModelVocab=vocab.st\n\n# this is a Zhang Le-style MEM.\nmaxentModel=st.mod\n\n# this last must be an ARPA-formatted n-gram model over supertags (with and )\n# (7- to 9-grams work well, without much memory usage).\nsequenceModel=st.lm\n\n# tagging beam widths (first try beta1, then beta2 if that doesn't give a parse, etc.).\n# give as many as you want, but keep in mind that your parser will try them all out\n# (which may make it try to parse in vain -- i.e., when it just doesn't have the\n# categories to do it).\n#betas = 0.024 0.003875 0.001225 0.0005377 0.000275 0.0000925 0.00004\n#betas = 0.075 0.03 0.01 0.005 0.001 0.0005\nbetas = 0.075 0.03 0.01\n\n# if not using the prior model (above), you must give 'K' values (see Clark and Curran (2007)).\n# the first one is for all beta values but the last. the second one is for\n# the last.\nfirstK=20\nlastK=100\n\n# also, if you're not using the prior models, you need to specify a word-keyed tagging\n# dictionary (this interacts with the 'K' values) and a POS-keyed tagging dictionary.\nwDict=word.dict.min10\nposDict=pos.dict.min10\n\n# use automatic POS features? (even if you are automatically POS tagging,\n# but only using single-best, set this to false).\nautoPOS=false\n\n# if autoPOS=true, you need to specify a POS config file.\nposConfig=pos.config\n\n# tagging algorithm. (choose from 'forward' and 'forward-backward')\n# the former is faster, the latter is better.\ntaggingAlgorithm=forward-backward\n\n# self-explanatory (will print certain errors and warnings, if set to 'true')\nverbose=true\n\n# option to include gold tags\nincludeGold=true\n\n# POS-specific relaxation\nbetaMultipliers=, 0.01 : 0.01 . 0.1 CC 0.1\n" }, { "path": "ccgbank/models/supertagger/st.noprior.config", "content": "# an example supertagger config file (D.N. Mehay)\n# change to suit your needs (e.g., replace the following paths\n# with paths that point to the relevant files).\n\n# this is a comment (\"basic\" means C,C & Vadas (2006)-style features).\n# there is nothing else in OpenCCG, at the moment.\ntaggerType=basic\n\n# this model gives priors on supertags. use it.\n# nb: testing out no prior model here ...\n#priorModel=stprior.flm\n#priorModelVocab=vocab.st\n\n# this is a Zhang Le-style MEM.\n#maxentModel=st.mod\nmaxentModel=st.noprior.mod\n\n# this last must be an ARPA-formatted n-gram model over supertags (with and )\n# (7- to 9-grams work well, without much memory usage).\nsequenceModel=st.lm\n\n# tagging beam widths (first try beta1, then beta2 if that doesn't give a parse, etc.).\n# give as many as you want, but keep in mind that your parser will try them all out\n# (which may make it try to parse in vain -- i.e., when it just doesn't have the\n# categories to do it).\n#betas = 0.024 0.003875 0.001225 0.0005377 0.000275 0.0000925 0.00004\n#betas = 0.075 0.03 0.01 0.005 0.001 0.0005\nbetas = 0.075 0.03 0.01 0.003 0.001 0.0003 0.0001 0.00001\n\n# if not using the prior model (above), you must give 'K' values (see Clark and Curran (2007)).\n# the first one is for all beta values but the last. the second one is for\n# the last.\nfirstK=20\nlastK=100\n\n# also, if you're not using the prior models, you need to specify a word-keyed tagging\n# dictionary (this interacts with the 'K' values) and a POS-keyed tagging dictionary.\nwDict=word.dict.min10\nposDict=pos.dict.min10\n\n# use automatic POS features? (even if you are automatically POS tagging,\n# but only using single-best, set this to false).\nautoPOS=true\n\n# if autoPOS=true, you need to specify a POS config file.\nposConfig=pos.config\n\n# tagging algorithm. (choose from 'forward' and 'forward-backward')\n# the former is faster, the latter is better.\ntaggingAlgorithm=forward-backward\n\n# self-explanatory (will print certain errors and warnings, if set to 'true')\nverbose=true\n\n# option to include gold tags\n#includeGold=true\n\n# POS-specific relaxation\n#betaMultipliers=, 0.01 : 0.01 . 0.1 CC 0.1\nbetaMultipliers=, 0.01 : 0.01 . 0.1\n" }, { "path": "ccgbank/models/supertagger/stprior.flm", "content": "\n## A prior probability model that estimates p(supertag | word, pos)\n## with smoothed back-off (a \"soft tagging dictionary\" if you will).\n\n1\n \n## lexical category (\"supertag\") (T) given POS tag (P) and word (W).\n\nT : 2 P(0) W(0) t_p0w0.count t_p0w0.lm 3\n P0,W0 W0 wbdiscount gtmin 15 \n P0 P0 wbdiscount\n 0 0 wbdiscount\n" }, { "path": "ccgbank/models/supertagger/vocab.flm", "content": "\n## flm file for determining the vocab\n\n3\n \n## word (W) unigram\n\nW : 0 w.count w.lm 1\n 0 0\n\n## supertag (T) unigram\n\nT : 0 t.count t.lm 1\n 0 0\n\n## pos tag (P) unigram \n\nP : 0 p.count p.lm 1\n 0 0\n\n\n" }, { "path": "ccgbank/original/models/postagger/pos.config", "content": "# an example POS tagger config file (D.N. Mehay)\n# change to suit your needs (e.g., replace the following paths\n# with paths that point to the relevant files).\n\n# \"basic\" = Ratnaparkhi-style features, inter alia\n# there is nothing else in OpenCCG, at the moment.\ntaggerType=basic\n\n# this model gives priors on POS tags.\npriorModel=posprior.flm\npriorModelVocab=vocab.pos\n\n# this is a Zhang Le-style MEM.\nmaxentModel=pos.mod\n\n# this last must be an ARPA-formatted n-gram model over POS tags (with and )\n# (7- to 9-grams work well, without much memory usage).\nsequenceModel=pos.lm\n\n# tagging beam width (beta=1 means \"single-best\", i.e., unitagging)\nbeta=0.044\n#beta=0.1\n#beta=1.0\n\n# tagging algorithm. (choose from 'forward' and 'forward-backward')\n# the former is faster, the latter is better.\ntaggingAlgorithm=forward-backward\n\n# self-explanatory (will potentially print certain errors and warnings, if set to 'true')\nverbose=true" }, { "path": "ccgbank/original/models/postagger/posprior.flm", "content": "\n## A prior probability model that estimates p(pos | word)\n## with smoothed back-off (a \"soft tagging dictionary\" if you will).\n\n1\n \n## POS tag (P) given word (W) with a back-off to the prior on the POS itself.\n\nP : 1 W(0) p_w0.count p_w0.lm 2\n W0 W0 wbdiscount gtmin 1 \n 0 0 wbdiscount gtmin 1 \n" }, { "path": "ccgbank/original/models/supertagger/st.config", "content": "# an example supertagger config file (D.N. Mehay)\n# change to suit your needs (e.g., replace the following paths\n# with paths that point to the relevant files).\n\n# this is a comment (\"basic\" means C,C & Vadas (2006)-style features).\n# there is nothing else in OpenCCG, at the moment.\ntaggerType=basic\n\n# this model gives priors on supertags.\n#priorModel=stprior.flm\n#priorModelVocab=vocab.st\n\n# this is a Zhang Le-style MEM.\nmaxentModel=st.mod\n\n# this last must be an ARPA-formatted n-gram model over supertags (with and )\n# (7- to 9-grams work well, without much memory usage).\nsequenceModel=st.lm\n\n# tagging beam widths (first try beta1, then beta2 if that doesn't give a parse, etc.).\n# give as many as you want, but keep in mind that your parser will try them all out\n# (which may make it try to parse in vain -- i.e., when it just doesn't have the\n# categories to do it).\nbetas = 0.024 0.003875 0.001225 0.0005377 0.000275 0.0000925 0.00004\n#betas = 0.075 0.03 0.01 0.005 0.001\n\n# if not using the prior model (above), you must give 'K' values (see Clark and Curran (2007)).\n# the first one is for all beta values but the last. the second one is for\n# the last.\nfirstK=20\nlastK=100\n\n# also, if you're not using the prior models, you need to specify a word-keyed tagging\n# dictionary (this interacts with the 'K' values) and a POS-keyed tagging dictionary.\nwDict=word.dict.min10\nposDict=pos.dict.min10\n\n# use automatic POS features? (even if you are automatically POS tagging,\n# but only using single-best, set this to false).\nautoPOS=true\n\n# if autoPOS=true, you need to specify a POS config file.\nposConfig=../postagger/pos.config\n\n# tagging algorithm. (choose from 'forward' and 'forward-backward')\n# the former is faster, the latter is better.\ntaggingAlgorithm=forward-backward\n\n# self-explanatory (will print certain errors and warnings, if set to 'true')\nverbose=true" }, { "path": "ccgbank/original/models/supertagger/stprior.flm", "content": "\n## A prior probability model that estimates p(supertag | word, pos)\n## with smoothed back-off (a \"soft tagging dictionary\" if you will).\n\n1\n \n## lexical category (\"supertag\") (T) given POS tag (P) and word (W).\n\nT : 2 P(0) W(0) t_p0w0.count t_p0w0.lm 3\n P0,W0 W0 wbdiscount gtmin 15 \n P0 P0 wbdiscount\n 0 0 wbdiscount\n" }, { "path": "ccgbank/original/models/supertagger/vocab.flm", "content": "\n## flm file for determining the vocab\n\n3\n \n## word (W) unigram\n\nW : 0 w.count w.lm 1\n 0 0\n\n## supertag (T) unigram\n\nT : 0 t.count t.lm 1\n 0 0\n\n## pos tag (P) unigram \n\nP : 0 p.count p.lm 1\n 0 0\n\n\n" }, { "path": "ccgbank/plugins/MyGenSynScorer.java", "content": "\npackage plugins;\n\nimport opennlp.ccg.synsem.*;\nimport java.io.*;\n\npublic class MyGenSynScorer extends GenerativeSyntacticModel\n{\n static String modeldir = getModelDir();\n static String getModelDir() {\n\tString retval = System.getProperty(\"gensyn.model.dir\", \"models/parser\");\n\tif (!retval.endsWith(\"/\")) retval += \"/\";\n\treturn retval;\n }\n\n public MyGenSynScorer() throws IOException {\n super(modeldir+\"top.flm\", modeldir+\"leaf.flm\", modeldir+\"unary.flm\", modeldir+\"binary.flm\");\n }\n}" }, { "path": "ccgbank/plugins/MyNgramCombo.java", "content": "package plugins;\r\n\r\nimport opennlp.ccg.ngrams.*;\r\nimport java.io.*;\r\nimport java.util.*;\r\nimport java.lang.Thread;\r\n\r\npublic class MyNgramCombo extends LinearNgramScorerCombo\r\n{\r\n static String bigWordsLM() { \r\n\tString retval = System.getProperty(\"big.words.lm\", \"models/realizer/gigaword4.5g.kenlm.bin\"); \r\n\tif (new File(retval).exists()) return retval;\r\n\tSystem.out.println(\"Reusing trigram model as a stand-in for the big LM\");\r\n\treturn null;\r\n }\r\n static String wordsLM() { return System.getProperty(\"words.lm\", \"models/realizer/train.3bo\"); }\r\n static String wordsSCLM() { return System.getProperty(\"words.sc.lm\", \"models/realizer/train-sc.3bo\"); }\r\n static String stposFLM() { return System.getProperty(\"stpos.flm\", \"models/realizer/stp3.flm\"); }\r\n\r\n // map to keep track of trigram model for reuse\r\n static Map lmMap = new IdentityHashMap(5);\r\n\r\n // return big lm, while setting trigram model if using it as a stand-in\r\n static NgramScorer getBigLM() throws IOException {\r\n\tString biglm = bigWordsLM();\r\n\tif (biglm != null) return new KenNgramModel(5, biglm, false, true, true, '_', false);\r\n\tNgramScorer retval = new StandardNgramModel(3, wordsLM());\r\n\tlmMap.put(Thread.currentThread(), retval);\r\n\treturn retval;\r\n }\r\n\r\n // return trigram lm, reusing existing one if present\r\n static NgramScorer getWordsLM() throws IOException {\r\n\tNgramScorer retval = lmMap.get(Thread.currentThread());\r\n\tif (retval != null) {\r\n\t lmMap.remove(Thread.currentThread());\r\n\t return retval;\r\n\t}\r\n\treturn new StandardNgramModel(3, wordsLM());\r\n }\r\n\r\n public MyNgramCombo() throws IOException {\r\n\tsuper(new NgramScorer[] { \r\n\t\tgetBigLM(), \r\n\t\tgetWordsLM(),\r\n\t\tnew StandardNgramModel(3, wordsSCLM(), true), \r\n\t\tnew FactoredNgramModelFamily(stposFLM()) \r\n\t });\r\n }\r\n}\r\n" }, { "path": "ccgbank/plugins/MyNgramGenSynProduct.java", "content": "package plugins;\r\n\r\nimport opennlp.ccg.ngrams.*;\r\nimport opennlp.ccg.synsem.*;\r\nimport java.io.*;\r\n\r\npublic class MyNgramGenSynProduct extends SignScorerProduct\r\n{\r\n public MyNgramGenSynProduct() throws IOException {\r\n\tsuper(new SignScorer[] { new MyGenSynScorer(), new MyNgramCombo() }); \r\n }\r\n}\r\n" }, { "path": "ccgbank/plugins/MyNgramPrecisionBaselineGenInterp.java", "content": "\npackage plugins;\n\nimport opennlp.ccg.ngrams.*;\nimport opennlp.ccg.synsem.*;\n\nimport java.io.*;\n\npublic class MyNgramPrecisionBaselineGenInterp extends SignScorerInterpolation implements SelfParaphraseBiaser\n{\n static String[] targets = { \"e plurubus unum\" };\n\n NgramPrecisionModel selfBiaser;\n\n public MyNgramPrecisionBaselineGenInterp() throws IOException {\n super(\n\t new SignScorer[] { new NgramPrecisionModel(targets), new MyNgramGenSynProduct() },\n\t new double[] { 100.0, 1.0 }\n\t);\n\tselfBiaser = (NgramPrecisionModel) models[0];\n }\n\n public void setTargets(String[] targets) { selfBiaser.setTargets(targets); }\n}" }, { "path": "ccgbank/plugins/MyNgramPrecisionPerceptronInterp.java", "content": "\npackage plugins;\n\nimport opennlp.ccg.ngrams.*;\nimport opennlp.ccg.synsem.*;\n\nimport java.io.*;\n\npublic class MyNgramPrecisionPerceptronInterp extends SignScorerInterpolation implements SelfParaphraseBiaser\n{\n static String[] targets = { \n\t\"lee said brianna had dragged food , toys and other things into the bedroom .\", \n\t\"lee , 33 , said the girl had dragged the food , toys and other things into her mother 's bedroom .\" \n\t//\"charles o. prince , 53 , was named as mr. weill 's successor .\",\n\t//\"mr. weill 's longtime confidant , charles o. prince , 53 , was named as his successor .\"\n };\n\n NgramPrecisionModel selfBiaser;\n\n public MyNgramPrecisionPerceptronInterp() throws IOException {\n super(\n\t new SignScorer[] { new NgramPrecisionModel(targets), new MyRealizerPerceptronScorer() },\n\t //new double[] { 100.0, 1.0 }\n\t new double[] { 10000.0, 1.0 }\n\t);\n\tselfBiaser = (NgramPrecisionModel) models[0];\n }\n\n public void setTargets(String[] targets) { selfBiaser.setTargets(targets); }\n}" }, { "path": "ccgbank/plugins/MyParserPerceptronScorer.java", "content": "\npackage plugins;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.perceptron.*;\nimport opennlp.ccg.hylo.*;\n\nimport java.io.*;\n\npublic class MyParserPerceptronScorer extends ReRankingPerceptronScorer\n{\n static String modeldir = getModelDir();\n static String getModelDir() {\n\tString retval = System.getProperty(\"parser.models.dir\", \"models/parser\");\n\tif (!retval.endsWith(\"/\")) retval += \"/\";\n\treturn retval;\n }\n\n static String modelname = getModelName();\n static String getModelName() { return System.getProperty(\"parser.model.name\", \"model.gz\"); }\n\n public MyParserPerceptronScorer() throws IOException {\n super(\n\t //new ComposedFeatureExtractor(new MyGenSynScorer(), new SyntacticFeatureExtractor()), \n\t new ComposedFeatureExtractor(new MyGenSynScorer(), new MySynSemFeatureExtractor()), \n\t new Model(modeldir + modelname)\n\t);\n }\n\n protected SignScorer getBaseScorer(FeatureExtractor featureExtractor) {\n\treturn (SignScorer) ((ComposedFeatureExtractor)featureExtractor).featureExtractors[0];\n }\n}" }, { "path": "ccgbank/plugins/MyRealizerPerceptronScorer.java", "content": "\npackage plugins;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.perceptron.*;\n\nimport java.io.*;\n\npublic class MyRealizerPerceptronScorer extends PerceptronScorer\n{\n static String getModelDir() {\n\tString retval = System.getProperty(\"realizer.models.dir\", \"models/realizer\");\n\tif (!retval.endsWith(\"/\")) retval += \"/\";\n\treturn retval;\n }\n\n static String getModelName() { return System.getProperty(\"realizer.model.name\", \"model.gz\"); }\n\n static String getAgrMultiplier() { return System.getProperty(\"realizer.agr.mult\", \"10\"); }\n\n static double calcAgrMultiplier() {\n\ttry {\n\t double mult = Double.parseDouble(getAgrMultiplier());\n\t return mult;\n\t}\n\tcatch (NumberFormatException exc) {\n\t System.out.println(\"Ignoring unparseable negative agreement feature weight multiplier: \" + getAgrMultiplier());\n\t return 1.0;\n\t}\n }\n\n static double agrMult = calcAgrMultiplier();\n\n public MyRealizerPerceptronScorer() throws IOException {\n super(\n\t //new ComposedFeatureExtractor(new MyNgramGenSynProduct(), new MySynAgrFeatureExtractor()), \n\t new ComposedFeatureExtractor(new MyNgramGenSynProduct(), new MySynSemAgrFeatureExtractor()), \n\t new Model(getModelDir() + getModelName(), agreementFilter)\n\t);\n\tSystem.out.println(\"Loading perceptron model from: \" + getModelDir() + getModelName());\n\tSystem.out.println(\"Boosting negative agreement and punctuation balancing feature weights by a factor of \" + agrMult);\n }\n\n // feature filter for boosting negative agreement feature weights\n private static Model.FeatureFilter agreementFilter = new Model.FeatureFilter() {\n\t /** Returns the modified feature weight for the given feature. */\n\t public double adjustedWeight(String name, double weight) {\n\t\tif (weight >= 0) return weight;\n\t\tif (name.equals(\"$punct\") || name.startsWith(\"AGR\") || isAdjacentPunctFeat(name)) \n\t\t return weight * agrMult;\n\t\treturn weight;\n\t }\n\t};\n\n // returns whether a feature is an adjacent punctuation n-gram feature\n private static boolean isAdjacentPunctFeat(String name) {\n\tString[] keys = name.split(\":\");\n\tif (keys.length >= 2 && isPunct(keys[0]) && isPunct(keys[1]))\n\t return true;\n\tif (keys.length >= 4 && keys[0].equals(\"P\") && keys[2].equals(\"P\") && isPunct(keys[1]) && isPunct(keys[3]))\n\t return true;\n\treturn false;\n }\n\n // sentence-boundary markers treated like punctuation\n private static boolean isPunct(String token) {\n\treturn token.equals(\"-\") || token.equals(\"--\") \n\t || token.equals(\",\") || token.equals(\";\") \n\t || token.equals(\":\") || token.equals(\"!\")\n\t || token.equals(\"?\") || token.equals(\".\")\n\t || token.equals(\"...\") || token.equals(\"``\")\n\t || token.equals(\"'\") || token.equals(\"''\")\n\t || token.equals(\"LRB\") || token.equals(\"RRB\");\n }\n}" }, { "path": "ccgbank/plugins/MySynAgrFeatureExtractor.java", "content": "\npackage plugins;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.perceptron.*;\nimport opennlp.ccg.hylo.*;\n\npublic class MySynAgrFeatureExtractor extends ComposedFeatureExtractor\n{\n public MySynAgrFeatureExtractor() {\n super(new SyntacticFeatureExtractor(), new EnglishAgreementExtractor());\n }\n}" }, { "path": "ccgbank/plugins/MySynSemAgrFeatureExtractor.java", "content": "\npackage plugins;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.perceptron.*;\nimport opennlp.ccg.hylo.*;\n\npublic class MySynSemAgrFeatureExtractor extends ComposedFeatureExtractor\n{\n public MySynSemAgrFeatureExtractor() {\n super(new MySynSemFeatureExtractor(), new EnglishAgreementExtractor());\n }\n}" }, { "path": "ccgbank/plugins/MySynSemFeatureExtractor.java", "content": "\npackage plugins;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.perceptron.*;\nimport opennlp.ccg.hylo.*;\n\npublic class MySynSemFeatureExtractor extends ComposedFeatureExtractor\n{\n public MySynSemFeatureExtractor() {\n super(new SyntacticFeatureExtractor(), new LexDepFeatureExtractor());\n }\n}" }, { "path": "ccgbank/stanford-nlp/classifiers/stanfordner-README", "content": "Place your Stanford NE tagging models here." }, { "path": "ccgbank/stanford-nlp/stanfordnlp-README", "content": "Place your Stanford core NLP jar file here (preferably renaming it to 'stanford-core-nlp.jar') and\nthen also place, e.g., NE tagging models under './classifiers'." }, { "path": "ccgbank/templates/addFilterLexFeats.xsl", "content": "\n\n\n\n \n \n \n\n \n\n\t\n\n\t\n\t \n\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\n\t\t\t\n\n\t\n\t \n\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\n\t\t\t\n\n\t\n\t \n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\n\t\t\t\n\n\t \n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n\n \n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/addStems.xsl", "content": "\n\n\n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n PERF\n \n \n \n PROG\n \n \n \n PASS\n \n \n \n \n \n\n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "ccgbank/templates/adjustAppos.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n 1 and not(@pos1=',')]\"/>\n\n\n\n 1 and not(@pos1=',')]\"/>\n\n\n\n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/adjustCandCcats1.xsl", "content": "\n\n\n\n\n \n \n \n \n\n \n\n\t\n \n \n \n \n\n \n \n \n \n \n \n \n\n\t\n\n\t\t\n \n \n \n \n\n\t\n\n\t\t\n \n \n\n \n\n\t \n \n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/adjustCats.xsl", "content": "\n\n\n\n\n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/adjustParenthetical.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\t\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/adjustReportedSpeech.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\t\n\n\n\n\n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/adjustRoles.xsl", "content": "\n\n\n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "ccgbank/templates/adv-placement.xsl", "content": "\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\t\n\t\t\n\t\n\t\n\n\n\n\n\t\n\t\n\t\t\n\t\n\n\n\n\n\t\n\t\n\t\n\n\t\n\n\t\n\t\n\t\t\n\t\n\n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/agr-macroInsert.xsl", "content": "\n\n\n\n \n \n \n \n \n \n \n\n \n\n \n\n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n\t\t\t\n \n \n \n \n \n \n \n \n \n\t\t\t\n \n \n \n\n \n \n \n \n\t\t\t\n \n \n \n \n \n\n\t\t\t\n\n\t\t\t\n \n \n\t\t\t\t \n\t\t\t \n \t \n\t\t\t\n \n \n \n \n \n\n\t\t\t\n\n\t\t\t\n \n \n\t\t\t\t \n\t\t\t\n\n \n\t\t\t\n \n \n\n\t\n \n \n \n \n\n\t\n \n \n \n \n\n\n\t\n\t\n\t\t \n\t\t\t\n\t\t\n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "ccgbank/templates/allotIdLeaf.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n \n \n \n \n\n\n''\n\n\n\n\t\n \n \n\t\t\n \n \n \n\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/allotIdTree.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n \n \n \n \n\n\n\n\n\n\t\n\t\n\t\n\n\t\n\n\t\t\n\t \t0 and (java:checkTreeInfo($obj,$header,$nt-id,number(count(*[1]/descendant::atomcat))))\">\n \n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n \n\t\t\n\t\t\t\t\n\t\t\n\t\t\n \n\n\n\n\n\n\n \n \n \n \n \n\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/allotIndexRel.xsl", "content": "\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n \n \n \n \n \n\n\t\t\t\t\n\n\t\t\t\t\n\t\t\t\t\n First\n \n\t\t\t\t\n\t\t\t\t\n GenRel\n \n\n\t\t\t\t\n\t\t\t\t\tGenRel\n \n\n\t\t\t\t\n\t\t\t\t\tGenRel\n \n\n\t\t\t\t\n\t\t\t\t\tmoodColon\n \n\n\t\t\t\t\n\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n whApposRel\n \n\n\t\t\t\t\n\t\t\t\t\n emph-intro\n \n\t\t\t\t\n EmphIntro\n \n\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tElabRel\n \n\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tmodFeat\n \n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\temph-final\n \n\t\t\t\t\n\t\t\t\t\tEmphFinal\n \n\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tinterrupt\n \n\t\t\t\t\n\t\t\t\t\tInterruptRel\n \t\n\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tParenRel\n\t\t\t\t\n\n\t\t\t\t\n\t\t\t\t\n colonExp\n \n\n\t\t\t\t\n\t\t\t\t\n DashInterp\n \n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n EllipsisRel\n \n\t\t\t\n\t\t\t\n\t\t\n\t\t\t\n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/anim-macroInsert.xsl", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n\t\n \n \n \n\t\t\t\n\t\t \n\t\t\t\n \n \n \n \n \n \n \n \n \n \n\t\t\t\n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n\t\t\t\n \n \n\n \n \n\n\t\t\t\n\n\t\t\t\n\t\t\t\t\n\t \t\t\t\n \t\t\t\n\t\t\t\n\t\t\t\t\t\t\n\n \n \n \n\t\n \n\t\t\t\t\t\t\n \n \n\t\n \n\t\t\t\t\t\t\n \n \n\n\t\n\t\n\t\t \n\t\t\t\n\t\t\n \n\n\n \n \n \n \n \n \n \n\n\n\n" }, { "path": "ccgbank/templates/annotateAppos-Dash.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n \t\n\n\n\n\n\t\t\n\n\n\n\n\t\n\t\t\n\t\t\n \t\t\n \t\t\n \t\n \t\n \t\n \t\n \t\t\n\t\t\t\t\t\n \t\n\t\t\n\t\t\n\t\t\n\t\n\n\n\n\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n \t\n \t\n \t\n\t\t\t\t\t\t\n \t\n \t\n \t\n \t\n\t\t\t\t\t\t\n \t\n\t\t\t\t\t\n \t\n \t\n \t\n \t\n \t\n \t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n \t\t\n \t\t\n \t\t\n \t\n \t\n \t\n\t\t\n\t\t\n\t\t\t\t\n\t\t\n\t\n\n\n\n\t\n\t\n \t\n \t\n \t\n\t\t\t\t\n \n \n \n \t\n\t\t\t\t\n \n\t\t\t\n \t\n\t \t\n \t \t\t\n \t\t\t\t\n\t\t\t\t\t\n \t\n\t\t\t\n\t\t\t\n\t\t\t\n \t\t\n\t\t\t\t\t\n \t\n \t\n \t\n \t\t\n\t\t\t\t\t\n \t\n\t\t\t\n\t\t\n\t\n\t\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/annotateAppos1.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\t\n\n\n\n\n\n\t\n\t\n \t\n\t\n\n\n\n\t\n\n\n\n\t\n\t\n\n\t\n \n\t\n\t\n\n\t \n\n\t\t\n\n\t\t\n\t\t\n\t\t\n\t\t\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\n\n\t\n\t\n\t\n\t\n\t\t\n\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n \t\t\t\t\n\t\t\t\t\t\t\n \t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n \t\t\t\t\n\t\t\t\t\t\t\n \t\t\t\n\t\t\t\t\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n \t\t\t\t\t\n\t\t\t\t\t\t\t\n \t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n \t\t\t\t\t\n\t\t\t\t\t\t\t\n \t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n \t\t\t\t\t\n\t\t\t\t\t\t\t\n \t\t\t\t\n\t\t\t\t\t\n\t\t \t\t\n \t\t\t2]\"/>\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n \t\t\n \t\n\t\t\t\n\t\n\t\n\n\n\n\t\n\t\n\t\n\t\n\n\n\n\n\n\n\t\n \n\n\n\n\t\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/annotateAppos2.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\t\n\t\n \t\n\t\n\n\n\n\n\n\n\n\t\n\t\n\t\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\n\t\n\t\n\n\t\n\t\n\t\n\t\n\t\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\t\n \t\t\n\t\t\t\t\n \t\n\t\t\t\n\t\t\t\n \t\t\n\t\t\t\t\n \t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t\t\t\n\t\t\t\t\t\n \t\t\n\t\t\t\t\n\t\t\t\t\n \t\t\t\n\t\t\t\t\t\n \t\t\n\t\t\t\t\n\t\t\t\t\n \t\t\t\n\t\t\t\t\t\n \t\t\n\t\t\t\n\t\t\n \t2]\"/>\n \n\t\t\n\t\n\t\t\n \t\n \n\t\n\t\t\n\t\n\n\n\n\t\n\t\n\t\n\t\n\n\n\n\n\n\n\t\n\t\n\t\n\t\n\n\n\n\n\n\n\n\t\n \n\n\n\t\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/annotateAppos3.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n \t\n\n\n\n\n\n\n\n\t\n\n\n\n\n\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n \t\t\n \t\t\n \t\t\n\t\t\t\t\t\t\n \t\n \t\n \t\n \t\t\n\t\t\t\t\t\n \t\n\t\t\t\t\n\t\t\t\t\n \t\t\n \t\t\n \t\t\n \t\n \t\n \t\n \t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n \t\n \t \n\t\t\t\n\t\t\n\t\t\n\t\t\n\n\n\n\n\n\n\n\t\n\n\n\n\n\t\n\t\t\n\t\t\n\t\t\t\t\n \t\n \t\n \t\t\n \t\n \n \t\n\t\t\n\t\t\t\n\t\t\t\n \t\t\n\t\t\t\t\n\t\t\t\t\n \t \t\n \t \t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\n\t\t\t\n\t\t\t\t\n \t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n \t \t\t\n \t \t\t\t\n\t\t\t\t\t\t\n \t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\n\n\n\n\n\n\n\t\n\n\n\n\n\t\n\t\t\n\n\t\t\n\t\t\n\t\t\t\n \t\t\n \t\t\n \t\t\n\t\t\t\t\t\n \t\n \t\n \t\n \t\t\n\t\t\t\t\t\n \t\n\t\t\t\t\n\t\t\t\t\n \t\t\t\n \t\t\t\n\t\t\t\t\t\t\n \t\t\n \t\t\n \t\t\n \t\t\t\n\t\t\t\t\t\t\n \t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/annotateBrackets.xsl", "content": "\n\n\n\n\n \n \n\n\n\n\n\n\n \n\n\n\n\n\n\t\n\t\n\t\n\t\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n \t\t\n \t\t\n \t\t\n \t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n \n \t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n \t\n \t \n \t\n \t\n\t\t\t\t\t\n\t\t\t\t\t\n \n \t\t\t\t\n \t\n\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\t\n \t\n \t\n \t\n \n \n \n\t\n\n\t\n\n\n\t\n\t\n\t\n\t\n\t\t\n\t\t\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n \t\t\n \t\t\n \t\t\n \t\n\t\t\t\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n \tres\n \n\t\t\t\t\t\n\t\t\t\t\t\n \t\n \t \n \t\n \t\n\t\t\t\t\t\n\t\t\t\t\t\n \targ\n \n \t\n\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n \t\n \t\n \t\n \n \n \n\t\n\n\n\n\t\n\t\t\t\n\t\t\t\t\n \t\t\n\t\t\t\t\t\n \t\n \t\n \t\n \t\t\n\t\t\t\t\t\n \t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n \t\t\t\n\t\t\t\t\t\t\n \t\t\n \t\t\n \t\t\n \t\t\t\n \t\t\n\t\t\t\t\n\t\t\t\n\n\n\n\n\t\t\t\n\n\n\t\t\t\n\n\n\n\t\t\t\n\n\n\n\n\t\n \t\t\n\t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/annotateColons.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\t\n\n\n\n\n\n\t\n\t\t\n\t\t\n\t\t\n \n \n \n \n \n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n \t\t\n\t\t \n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \n \n\t\n\n\n\n\n\n\n\n\t\n\t\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n \t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n \t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n \t\n \t\t\n\t\t\t\t\t\n\t\t\t\t\n \t\t\n\t\t\n\t\t\n\n\n\n\n\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\t\n \t \t\n\t\t\t\t\t\n \t\n \t\n \t\n \t\n \t\t\n\t\t\t\t\t\n \t\n \t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t \t\t\n\t\t\t\t\t\t\n \t\t\n \t\t\n \t\t\n \t\t\n \t\t\t\n\t\t\t\t\t\t\t\t\n \t\t\n \t\t\t\t\t\n\t\t\t\n\t\t\n\t\n\t\n\n\n\n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/annotateDots.xsl", "content": "\n\n\n\n\n \n \n\n\n\n\n\n\n \n\n\n\n\n\t\n\t\n\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\n\n\t\n\n\t\n\t\t\t\n\t\t\t\t\n\n\t\t\n\t\t\n\t\t\t\n\t\t\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\n\t\t\t\n \t\n \t\n\t\t\t\t\t\t\n \n \n \n \n\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\n \n\t\t\t\t\n \t\n \t\t\n\t\t\t\t\t\t\t\n \t\n \t\n \t\n \t\n\t\t\t\t\t\t\n\t\t\t\t\t \n \n \n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\n\t\n\n\n\n\n\n\n\t\n \t\t\n\t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/annotateExtraposedAppos.xsl", "content": "\n\n\n\n\n \n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n \n \n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t \n\t\t\t\t\n\t\t\t\t\n\t\t\t\t \n\t\t\t\t\n\t\t\t\n\t\t\t\n \t\n \n\t\t\n \n\n\n\n\n\t\t\t\n\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\n\t\t\n\t\n\t\n\n\n\t\n\t\t\n\t\n\n\n\n \n \n \n \n \n\n\n\n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/annotateNom-AdjConj.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \t\n\n\n\n\n\t\n\t\t\n\t\t\n\t\t\t\n \t\n \n\t\t\t\n\t\t\t\n \t\t\n \t\t\n \t\n \t\n \t\n \t\t\n \t\n\t\t\t\n\t\t\n\n\t\t\n\t\t\t\n\n\t\t\t\n\t\t\t\t\n \t\t\n \t\n\t\t\t\t\n\t\t\t\t\n \t\t\t\n \t\t\t\n \t\t\n \t\t\n \t\t\n \t\t\t\n \t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n \t\t\n \t\n \t\n\t\t\n\t\t\n\t\n\n\n\n\n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/annotateParentheticals1.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\t\n\t\n\t\t\n\t\n\n\n\n\n\n\n\t\n\t\n\t\n\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n \t\n \t\t\n \n \t\n\t\t\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n \t\t\n \t\t\t\n \t\n \t\n\t\t\t\t\t\n\t\t\t\n\t\t\t\tADV-PAREN\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n \t\t\t\n \t\t\t\t\n \t\t\n \t\t\n\t\t\t\t\t\n\t\t\t\t\t\n \t\t\t\n \t\t\t\t\n \t\t\n \t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n \t\t\n \t\t\t\n \t\n \t\t\n\t\t\t\t\t\n\t\t\n\t\n\n\n\n\n\n\t\n\t\n\t\n\t\n\n\t\n\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\n\t\n\t\t\t\n\t\t\t\n\n\t\n\t\t\n\t\t\n\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n \t\t\n \t \t\n \t\n \t\n\t\t\t\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n \t\t\t\n \t \t\t\n \t\t\n \t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\tPP-PAREN\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n \t\n \t\t\n \n \t\n\t\t\t\n\t\n\n\n\n\n\n\t\t\t\n\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\n\t\t\n\t\n\t\n\n\n\t\n\t\t\n\t\n\n\n\n\n\t\t\t\n\n\t\n\t\t\n\t\t\t\t\t\n\t\t\t\n\t\t\n\t\t\t\n\t\t\n\t\n\n\t\n\t\t\t\t\n\t\t\n\t\n\t\n\n\n\t\n\t\t\n\t\n\n\n\n \n \n \n \n \n\n\n \n \n \n \n \n\n\n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/annotateParentheticals2.xsl", "content": "\n\n\n\n\n \n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\t\n\t\t\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\n\t\t\t\n \t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \t\t\n\t\t\t\t\t\t\n\t\t1]\"/>\n\t\t\t\t\n\n\n\n\n\t\n\t\t\n \n\t\t\t\n\t\t\t\n \t\n \n \n \n \n \t\n \n \n \n \t\n \t\n \n \n \n \n \t\n \n \n\t\t\t\n\t\t\t\n \t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n \t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n \t\t\t\n\t\t\t\n\t\t\t\n \t \n \n \n\t\n\n\n\t\n\t\t\n\t\n\t\n\n\n\t\n\t\t\n\t\n\t\n\n\n\n\t\n\t\t\n \n\t\t\t\n\t\t\t\n \t\n \n \n \n \n \t\n \n \n \n \t\n \t\n \n \n \n \n \t\n \n \n\t\t\t\n\t\t\t\n \t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \t\t\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n \t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n \t\t\t\n\t\t\t\t\t\t\t\n \n \n\t\n\n\n\n\t\n\t\n\t\t\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\n\t\t\t\n \t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \t\t\n\t\t\t\n\t\t\t\n \t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t1]\"/>\n\t\t\t\t\n\t\n\n\n\n\t\n\t\n\n\t\n\t\n\t\t\n\t\t\n\n\t\n\t\t\n\t\t\t\t\t\n\t\t\t\n\t\t\t\n \t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t\n \n \n \n \n \t\n \n \n \n \n \t\n \t\n \n \n \n \n \t\n \t\n \n \n\t\t\t\t\n\t\t\t\t\n \t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n \t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\n\t\n \t\n \t\t\n\t\t\t\t\n\t\t\t\n \t\n \t\n\t\n\t\n\n\n\n\t\n\t \t\n \n\n\n\n\t\n\t \t\n \n\n\n\n\n \n \n \n\n\n\n\n \n \n \n\n\n\n\n\t\n \t\t\n\t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/annotatePlace.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n \t\n\n\n\n\n\t1 and not(@pos1='PUNCT_APPOS_PLACE2') and not(@pos1='PUNCT_APPOS_PLACE_BAL')]\"/>\n\n\n\n\n\n\n\t\n\t\t\n\t\t\n \n \t\n\t\t\t\t\n \n \n \n \t\n\t\t\t\t\n \t\n\t\t\n\t\t\t\n\t\n\n\n\n\n\n\t\n\t\t\n \n \t\n\t\t\t\t\n \n \n \n \t\n\t\t\t\t\n \t\n\t\t\n\t\t\n\t\n\n\n\n\n\t\n\t\t\n\t\t\t\n \t\n \t\n \t\n\t\t\t\t\t\t\n \t\n \t\n \t\n \t\n\t\t\t\t\t\t\n \t\n\t\t\t\t\t\n \t\n \t\n \t\n \t\n \t\n \t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n \t\t\n \t\t\n \t\t\n \t\n \t\n \t\n\t\t\n\t\t\n\t\t\t\t\n\t\t\n\t\n\n\n\n\n\t\n\t\t\n\t\t\t\n \t\t\n\t\t\t\t\n \t\n \t\n \t\n \t\n\t \n \n\t\t\t\n\t\t\t\n\t\t\t\t\n \t\n \t\n \t\n \n\t\t\t\t\n\t\t\t\n\t\t\t\n\t \t\t\n \t\n \t\t\t\t\n\t\t\n\t\t\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\n\n\n\n\n\t\n\t\t\n \n \n\n\n\n\n\t\n\t\n\t\t\n\t\t\n \t\n \t\n\t\t\t\t\n \n \n \n \n\t\t\t\t\n \n\t\t\t\n\n\t\t\n\t\t\t\n \t\n \t\n \t\n\t\t\t\t\t\n \t\n \t\n \t\n \t\n\t\t\t\t\t\n \t\n\t\t\t\t\n \t\n \t\n \t\n \t\n \t\n \t\n\t\t\t\n\t\t\t\t\n \t\t\n \t\t\t\n\t\t\t\t\t\t\n \t\t\n \t\t\n \t\t\n \t\t\t\n\t\t\t\t\t\t\n \t\t\n\t\t\t\t\t\n\t \t\t\n \t \t\t\t\n \t\t\t\t\t\n\t\t\t\t\t\t\n \t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n \t\t\t\t\n\t\t\t\t\t\t\t\n \t\t\t\n \t\t\t\n \t\t\t\n \t\t\t\t\t\n\t\t\t\t\t\t\t\n \t\t\t\n\t\t\t\t\t\t\n \t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n \t\n \t\n \t\t\n \t\n \t\n \n\t\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/annotatePrtConjs.xsl", "content": "\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\t\n\n\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t1]\"/>\n\t\t\n\t\n\n\n\n\n\n\t\t\t\t\n\n\t\n\t\t\t\t\t\n\t\n\n\t\n\t\t\n\t\t\t\n\t\t\n\t\n\n\n\n\t\n\n\t\n\t\t\t\t\n\t\t\t\t\n\t\n\t\t0\">\n\n\t\t\t\t\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t0\">\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n \t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n \t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\n\t\t\t\t\t\n\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\n\t\t\t\t\t\n\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\n\t\t\t\t\t\n\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t \t \n\t\t\t\n\t\t\n\t\n\n\n\n\t\n\n\t\t\t\n\n\t\n \t\n\t\t\t\t\n\t\t\n \n\n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/annotateQuotes.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n'' \n`` \n\n\n\n\n\n\t\n\n\t\n\t\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n \n \n\t\t\t\n \n \n\n\n\n\n\t\n\n\t\n\t\n\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\n\n\n\n\n\t\n\t\n\t\n\t\n\n\t\n \n \n \n \n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n \n \n \n \n \n\n\n\n\t\n\t\t\n\t\n\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/annotateReportedSpeech.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\t\n\t\n\t\t\n\t\n\n\n\n\n\n\n\n\t\n\t\t\n\t\t\n\t\t\n \t\t\n \n\t\t\t\n\t\t\t\t\n \n \n \t\n \t\t\n \n \n \n \n \t\n \t\n \n \n \n\t\t\n\t\t\n\t\t\t\n \t\n \t\n\t\t\t\t\t\n \t\n \n \n\t\n\n\n\n\n\n\t\n\t\n\t\n\t\n\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n \t\t\t\t\n \t \t\t\t\n \t\t\t\n \t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n \t \t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n \t\t\t\n \t\t\t\t\n \t\t\n \t\t\t\n\t\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n \t \t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\t\n\n\n\n\t\n\t\n\n\t\n\t\n\n\t\n\t\t\n\t\t\n\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t \t\n\t\t\t\n\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t\t\n \t \t\n \t\n \t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\t\n\t\t\n\t\n\t\n\t\t\n\t\t\t\n\n\n\n\n\n\t\n\t\n\t\t\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\n\n\n\n\t\t\t\n\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\n\t\t\n\t\n\n\n\t\n\t\t\n\t\n\n\n\n\n\t\t\t\n\t\n\t\t\n\t\t\n\t\n\t\n\n\n\n \n \n \n \n \n\n\n \n \n \n \n \n\n\n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/annotateStrayAppos.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \t\n\n\n\n\n\n\n\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n \t\n \t\n \n \n\t\t\t\n\n\t\t\t\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n \t\t\t\n\t\t\t\t\t\t\n \t\t\n\t\t\t\t\t\n \t\t\t\n \t\t\t\n\t\t\t\t\t\t\n \t\t\n\t\t\t\t\t\n\t\t\t\t\t\n \t\t\n \t\t\n \t\n \t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n \t\t\t\t\t\t\n \t\t\t\t\t\n\t\t\t\t\t\t\t\t\n \t\t\t\t\t\t\n \t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n \t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t \n \t\n \t\n \n \n \t\n \n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n \t\n\t\t\t\t\t\t\n\t\t\t\t\t\n \t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n \t\n \t\n\t\t\t\t\t\n \t\n \n \n\t\n\n\n\n\t\n\n\n\n\n \n \n\t\t\n\t\t\n\t\t\n \n\n\n\n\n \n \n \n\n\n\n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/annotateVPCommas.xsl", "content": "\n\n\n\n\n \n \n\n\n\n\n\n\n\n \n\n\n\n\t\n\t\n\t\t\t\t\n\t\t\n\t\t\t\\\n\t\t\t\n\t\t\n\t\t\n\n\t\n\n\t\t\t\t\n\t\t\n\t\t\t/\n\t\t\t\n\t\t\n\t\t\n\n\n\n\n\t\n\t\n\n\t\n\t\t\t\n\t\t\n \t\n \t\n \t\n \n \n \n \t\n \t\n \t\n \n \n \n \t\n\t \t\n \t \t\t\n \t\t\n \t\n \t\n \t\n \t\t\n\t\t\t\t\t\n \t\n \t\n \n\t\n\n\n\n\n\n\n\t\n\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\n\n\n\n\n\n\t\t\t\n\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\n\t\t\n\t\n\n\n\t\n\t\t\n\t\n\n\n\n\n\n\t\n\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\n\n\n\n\n\n\t\t\t\n\n\t\n\t\t\n\t\t\t\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\n\t\n\n\t\n\t\t\t\t\n\t\t\n\t\n\t\n\n\n\t\n\t\t\n\t\n\n\n\n\n\n\t\n\n\n\n\n \n \n \n \n \n\n\n\n \n \n \n\n\n\n\n\t\n \t\t\n\t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/balanceAppos.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\t\n\t\t\n\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n \t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n \t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n \t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n \t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \t \t\t\n\t\t\t\n\t\t\n\t\t\n \t\t\n \t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \t\t\n \t\t\n\t\n\n\n\n\n\n\t\n\t\t\n\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n \t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n \t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n \t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n \t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \t \t\t\n\t\t\t\n\t\t\n\t\t\n \t\t\n \t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \t\t\n \t\t\n\t\n\n\n\n\t\n\t\t\n\t\t\n\t\n\n\n\n\n \n \n \n \n \n\n\n\n \n \n \n \n \n\n\n\n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/balanceDash-Paren.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\t\n\n\n\n\n\t\n\n\n\n\t\n\t\n\t\t\n \n\t\t\t\n \n \n \n\t\t\t\n \n \n\t\n\t\n\n\t\n\t\t\t\n\t\t\n\t\t\t\t\n \n\t\t\t\t\t\n\t\t\t\t\n \n \t\n \t\n \t\n \n \n\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\n\t\t\t\t\t\n\t\t\t\t\t\n \t \t\t\t\n \t\t\t\t\t\n\t\t\t\t\t\t\n \t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n \t\n \t\t\n \t\t\n \t\n \t\n \n\t\t\n\t\t\t\n\t\t\n\n\n\n\t\n \n \n \n\t\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/catCheck.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \t\n\n\n\n\t\n\t\n\n\n\n\n\t\n\t\t\t\n\t\n\t\n\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/ccgRules.xsl", "content": "\n\n\n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n\n \n \n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n\t \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n Purpose\n GenRel\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n\n \n \n \n\t\n \n \n \n \n \n \n \n \n\t\n \n \n\t\n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n\t\t\n \n \n \n\t\n \n \n \n\t\n \n \n \n\t\n \n \n \n \n \n \n \n \n\t\n \n \n\t\n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n\t\t\n \n \n \n\t\n \n \n \n\t\n \n \n \n\t\n \n \n \n \n \n \n \n \n\t\n \n \n\t\n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n \n\t\t\n \n \n \n\t\t\n \n \n \n \n \n\t\t\n \n \n \n\t\t\n \n \n \n \n \n \n \n\t\t\n \n \n \n\t\t\n \n \n \n \n \n\t\t\n \n \n \n\t\t\n \n \n \n \n \n\t\t\n \n \n \n \n \n\t\t\n \n \n \n\t\t\n \n \n \n \n \n\t\t\n \n \n \n \n \n \n\t\t\n \n \n \n\t\n \n \n \n\n \n \n \n\t\n \t \n \n \n \n \n \n \n \n \t\n \n \n\t\n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n \n\t\t\n \n \n \n\t\t\n \n \n \n \n \n\t\t\n \n \n \n\t\t\n \n \n \n \n \n \n\t\t\n \n \n \n\t\n \n \n \n\n\n \n \n \n \n \n \n \n \n\t\t\n\t\t \n\t\t\n \n\t \n \n \n \n \n \n\t \n \n \n \n\t \n \n \n \n \n \n\t\n \n\t \n \n \n \n\t \n \n \n \n \n\t\t\n \n \n \n\t\n \n \n \n\n \n \n \n \n\t\n \n \n \n\t\t\n \n \n\t \n \n\t\n \n \n\t\n \n\t \n \n \n \n\t \n \n \n \n \n\t\t\n \n \n \n\t\n \n \n \n\n \n \n \n \n \n \n \n \n\t\t\n\t\t \n\t\t\n \n\t \n \n \n \n \n \n\t \n \n \n \n\t \n \n \n \n \n \n\t \n \n \n \n\t \n \n \n \n \n \n\t\n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n\t\t\n \n \n \n\t\n \n \n \n\n \n \n \n \n \n \n\t \n\t \n \n\t \n \n \n \n \n \n\t\n \n \n\t \n\t\t\n \n\t\t\n\t \n \n \n \n\t \n \n \n \t\n \t \n \n \t \n \t\n \n \n\t \n\t \n \n \t\n \t \n \n \t \n \t\n \n \n\t \n \n \n \n\t\t\n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n\t \n\t \n \n\t \n \n \n \n \n \n\t\n \n \n\t \n\t\t\n \n\t\t\n\t \n \n \n \n\t \n \n \n \t\n \t \n \n \t \n \t\n \n \n\t \n\t \n \n \t\n \t \n \n \t \n \t\n \n \n\t \n\t\n \n \n \n \n \n \n \n \n\t\n\t \n\t \n\t \n\t\t\n\t \n\t \n\t \n\t\n \n \n\t\n\t \n\t \n\t \n\t\t\n\t\t \n\t\t\n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t\t\n\t\t \n\t\t\n\t \n\t \n\t \n\t \n\t \n\t \n\t\t\n\t \n\t \n\t \n\t\n \n \n \n\n\n \n \n \n \n \n\t\n \n \n \n\t\t\n \n \n \n\t\n \n \n\t\n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n \n\t\t\n \n \n\t\t\n \n \n \n\t\n \n \n \n \n\t\n \n \n \n\t\t\n \n \n \n\t\n \n \n\t\n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n \n\t\t\n \n\t\t\n \n \n \n \n \n \n \n\t\t\n \n \n\t\t\n \n \n \n\t\n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/changePunct.xsl", "content": "\n\n\n\n\n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\t\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/closedCatInsert.xsl", "content": "\n\n\n\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n\t\t\t\t\n \n *NoSem*\n \n \n \n *NoSem*\n \n\t\t\t\t\n\t\t\t\t\n *NoSem*\n \n\n \n \n *NoSem*\n \n \n \n *NoSem*\n \n \n\t\n\t\n \n \n GenOwn\n \n \n \n \n \n \n\t\t\t\t\tGenRel\n \n \n \n \n \n \n \n\t\t\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n int\n excl\n \n imp\n dcl\n \n \n \n \n \n \n\n\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/collapseMWUFull.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\t\n\n\t\n\t\t\n\t\n\n\t\n\n\t\n\t\t\n\t\n\n\t\n\t\n\n\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\n\n\n\n\n\t\n\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\n\t\n\n\n\n\t\n\t\n\n\n\n \n \n \n \n \n\n\n\n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/collapseMWUPart.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n \n\n\n\n\n\t\n\n\t\n\t\t\n\t\n\n\t\n\t\n\t\n\n\t\n\t\n\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\n\t\t\n\t\t\t\n\t\t\t\t\n \t\t\t\n \t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\n\n\n\n\n\t\n\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\n\t\n\n\n\n\t\n\t\t\n\n\n\n \n \n \n \n \n\n\n\n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/collapseMWUSharedTask.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n \n\n\n\n2 and descendant::Leafnode[last()]/child::atomcat]\">\n\n\t\n\n\t\n\t\t\n\t\n\n\t\n\n\t\n\t\t\n\t\n\n\t\n\t\n\n\t\n \n\t\t\n\t\t\t\n\t\t\n\t\t\n \t\n\t\t\n\t\n\t\n\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/computeCats.xsl", "content": "\n\n\n\n \n \n \n\n\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n (\n \n )\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n $\n \n _\n \n \n \n \n \n \n [\n \n ]\n \n \n \n _\n \n \n \n \n \n ~\n \n \n \n \n\n \n \n\n\n \n \n \n \n \n \n \n (\n \n )\n \n \n \n \n \n \n \n \n \n \n \n $\n \n\n \n \n [\n \n ]\n \n \n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/convTags.xsl", "content": "\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\t\n\t\t\n\t\n\n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/correctMistakes1.xsl", "content": "\n\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\n \n \n\t\n\n \n \n \n \n \n \n \n \t\n \n \n\t\n\t \n \t\n\t\n\t \n\t\t\n\t\n\t \n\t \n\t \n\t\n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/correctPPHeads.xsl", "content": "\n\n\n\n\n\n\t\n \n\n\t\n\t\n \n\n\n\n \n \n \n\n\n" }, { "path": "ccgbank/templates/exportToAuto.xsl", "content": "\n\n\n\n \n\n \n \n \n \n\n \n \n \n \n PARSER=GOLD NUMPARSE=1\n \n \n \n\t\n \n \n \n \n \n \n\n \n \n Missing head index or length in \n\t\n \n \n (<T \n \n \n \n > \n \n ) \n \n\n \n (<L \n \n \n \n \n \n >) \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n (\n \n )\n \n \n \n \n \n \n \n \n \n \n (\n \n \n \n \n\t)\n \n \n \n \n \n \n \n \n \n \n \n \n \n $\n \n _\n \n \n \n \n \n \n [\n \n ]\n \n \n _\n \n \n \n \n \n ~\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/filterLex.xsl", "content": "\n\n\n\n\n \n \n \n \n \n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\t \n\t\n\t\n\n\t\n\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n\n \n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/find-s-back-n.xsl", "content": "\r\n\r\n\r\n\r\n\r\n \r\n\r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n" }, { "path": "ccgbank/templates/genchal11-out.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "ccgbank/templates/inferConjRules.xsl", "content": "\n\n\n\n\n \n \n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\t\n\n\t\n\n\t\n\t\n\n\t\n\t\n\t\n\t\n\n\t\n\t\t\n \t\t\n\t\t\n\t\t\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t \t\t\n \t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t1]\"/>\n\t\n\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\n\t\n\t\n\t\n\n\t\n\t\n\n\t\t\t\n\n\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t1]\"/> \t \n\t\n\n \n\n\n \n\n\t\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\n\t\n\t\n\t\n\t\n\n\t\n\t\n\t\t\t\n\n\t\n\t\t\n \t\t\n\t\t\n\t\t\t\n\t\n\n\t\n\t\n\t\t\n\n\t\t\n\n\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\n\t\t\t\n\t\t\t\n\n\t\t\t\t\n\t\n\t\t\t\t\t\n\n\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t \t\t\t\n \t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t \t\t\t\t\t\n \t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t \t\t\t\t\t\n \t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t \t \n\t\n \n\n\n\n\t\n \n\n\n\t\n\t\n\t\n\n\t\n\t\t\n\t\n\t\n\t\t\t\n\t\n\t\n\t\t\t\n\t\n\t\t\n\n\n\n\n\t\n\t1]\" mode=\"args\"/>\n \n\n\n\n\t\n\t\n\n\t\t\n \t\t\n\t\t\t\n \t\t\n \t\n \n\n\n\n\n\n\n\n\t\n\t\t\n \t\t\n \t\t \n \t\t\n \t\n\n \n\n\n\n\n\t\n\t\n\t\n \t\t\n\t\t\t\n \t\t\n\t\n\t\n \t\t\n\t\t\t\n \t\t\n\t\n\n \n\n\n\n\t\n\t \n \n\n\n\n\n \n \t\n\t\n\n\n\n\n\t\n \t\t\n\t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/insertLF.xsl", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\t\t\t\t \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \t \t \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \t \t \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \t \t \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \t \t \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \t \t \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \t \t \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n\t\t\t\t\t\n\t\t\t\t\t\t\n \t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n\n\t\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/insertOrigPunctsLF.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n \t\n\n\n\n\n\t\n\t\t*NoSem*\n\t\t\n\t\n\n\n\n\n\n\n\n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/insertPTBInfo.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\t\n\t\n\t\n\t\n \n\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\n \n\t\t\n \n\n\n\n\n\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\n\n\n\n\n \n \n \n \n \n\n\n\n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/insertPunctLF-PosMod.xsl", "content": "\n\n\n\n \n \n\n\t\n \t\n\t\n\n\t\n\n\t\n\t\n\n\t\n\t\n \n \n \n \n \n\n\t\t\t\t\n\n\t\t\t\t\n\t\t\t\t\n First\n \n\t\t\t\t\n\t\t\t\t\n GenRel\n \n\n\t\t\t\t\n\t\t\t\t\tGenRel\n \n\n\t\t\t\t\n\t\t\t\t\tGenRel\n \n\n\t\t\t\t\n\t\t\t\t\tmoodColon\n \n\n\t\t\t\t\n\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n whApposRel\n \n\n\t\t\t\t\n\t\t\t\t\n emph-intro\n \n\t\t\t\t\n EmphIntro\n \n\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tElabRel\n \n\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tmodFeat\n \n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\temph-final\n \n\t\t\t\t\n\t\t\t\t\tEmphFinal\n \n\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tinterrupt\n \n\t\t\t\t\n\t\t\t\t\tInterruptRel\n \t\n\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tParenRel\n\t\t\t\t\n\n\t\t\t\t\n\t\t\t\t\n colonExp\n \n\n\t\t\t\t\n\t\t\t\t\n DashInterp\n \n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n EllipsisRel\n \n\n\t\t\t\t\n\t\t\t\t\n quote-rel\n \n\n\t\t\t\n\t\t\t\n\t\t\n\t\t\t\n\n\t\n\n\t\n\n\t\n\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n \n\t\t\t\t\t\n \t\n\t \n \n \t\n \n\t\t\t\t\n\t\n\n\n\t\n\n\n\t\n\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n \n\t\t\t\t\t\n \t\n\t \n \n \t\n \n\t\t\t\t\n\t\n\n\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t\t\t\n \t\t\n \t\n \t\t\t\n\t\t\t\n\t\t\t\t\n\t\n\n\n\t\n\t\t\t\n\t\t\t\n\t\t\t\n \n \t\n\t \n \n \t\n \n\t\t\t\t\n\t\n\t\n\t\n\n\t \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n \t\n \t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n \t\n \n \n\t\t\t\t\n\t\n\n\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n \t\n \t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\n \t\n \n \n\t\t\t\t\n\t\n\n\n\n\t\t\n\t\t\n\t\t\n \n\t \t\n\t\t\t\t\t\n \n \t\n \t\n\t\t\t\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \t\n \t\n \t\n \n\t\t\t\t\n\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \n \n \n \t\t\n\t\t\t\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \n \n \n \n\t\t\t\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \n \t\n \t\n \n \t\n \n\t\t\t\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \t\n \n \n \n\t\t\t\t\t\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t \t\n\t\t\t\t\t\n \n\t\t\t\n\n \n \t\n\t\t\t\n\n\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \n \n \n \n\t\t\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \t \t\n \t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n \n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \n \n\t\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \t\n \t\n \n \n \t\n \n \n \n\t\t\t\n\n\n\n\t\t\t\n\t\t\n\t\t\n \t\n \t\n \t\n \n \n \n\t\t\t\t\t\n\n\n\n\n\n\t\n\t\n\n\t\n\n\t\n\t\n\n\t\t\n\t\t\n\t\t\n \t\n\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\n \t\n \t\n\t\t\t\t\t\n \n \n \n\t\t\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n\t \t\n \n \n \n\t\t\n\n\n\n\n\n\t\n\t\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \t\n \n \n \n\t\t\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t \t\n\t\t\t\t\t\n \n\t\t\t\n* \n\t\t\t\t\t\n\n\t\t\t\n\n\n\n\n\n\t\n \t\n \n\n\n\n\n\n\t\n\t\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \n \n \n \n\t\t\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \n \n \n \t\n \n \t\n\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \t\n \n \n\t \n\t\t\t\n\t\t\n\t\t\n\n\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \t\n\t\t\t\t\t\n \n\t \n\t\t\n\t\t\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \t\n \t\n \t\n \n \n\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \t\n\t\t\t\t\t\n \n\t \n\t\t\n\t\t\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \t\n \t\n \t\n \n \n\t\t\n\n\n\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \n \n\t\t\n\n\n\n\n\n\t\t\n\t\n\t \n \t\n \t\n\t\n\n\n" }, { "path": "ccgbank/templates/insertPunctLF.xsl", "content": "\n\n\n\n \n \n\n\t\n \t\n\t\n\n\t\n\n\t\n\t\n\n\t\n\t\n \n \n \n \n \n\n\t\t\t\t\n\n\t\t\t\t\n\t\t\t\t\n First\n \n\n\t\t\t\t\n GenRel\n \n\n\t\t\t\t\n\t\t\t\t\tGenRel\n \n\n\t\t\t\t\n\t\t\t\t\tGenRel\n \n\n\t\t\t\t\n\t\t\t\t\tmoodColon\n \n\n\n\t\t\t\t\n\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n\t\t\t\t\n ApposRel\n \n\n\t\t\t\t\n whApposRel\n \n\n\t\t\t\t\n\t\t\t\t\n emph-intro\n \n\t\t\t\t\n EmphIntro\n \n\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tElabRel\n \n\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tmodFeat\n \n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\temph-final\n \n\t\t\t\t\n\t\t\t\t\tEmphFinal\n \n\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tinterrupt\n \n\t\t\t\t\n\t\t\t\t\tInterruptRel\n \t\n\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tParenRel\n\t\t\t\t\n\n\t\t\t\t\n\t\t\t\t\n colonExp\n \n\n\t\t\t\t\n\t\t\t\t\n DashInterp\n \n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n EllipsisRel\n \n\t\t\t\n\t\t\t\n\t\t\n\t\t\t\n\n\t\n\n\t\n\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n \n\t\t\t\t\t\n \t\n\t \n \n \t\n \n\t\t\t\t\n\t\n\n\n\t\n\n\n\t\n\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n \n\t\t\t\t\t\n \t\n\t \n \n \t\n \n\t\t\t\t\n\t\n\n\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t\t\t\n \t\t\n \t\n \t\t\t\n\t\t\t\n\t\t\t\t\n\t\n\n\n\t\n\t\t\t\n\t\t\t\n\t\t\t\n \n \t\n\t \n \n \t\n \n\t\t\t\t\n\t\n\t\n\t\n\n\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n \t\n \t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n \t\n \n \n\t\t\t\t\n\t\n\n\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n \t\n \t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\n \t\n \n \n\t\t\t\t\n\t\n\n\n\n\t\t\n\t\t\n\t\t\n \n\t \t\n\t\t\t\t\t\n \n \t\n \t\n\t\t\t\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \t\n \t\n \t\n \n\t\t\t\t\n\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \n \n \n \t\t\n\t\t\t\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \n \n \n \n\t\t\t\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \n \t\n \t\n \n \t\n \n\t\t\t\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \t\n \n \n \n\t\t\t\t\t\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t \t\n\t\t\t\t\t\n \n\t\t\t\n\n \n \t\n\t\t\t\n\n\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \n \n \n \n\t\t\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \t \t\n \t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n \n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n \n \n\t\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \t\n \t\n \n \n \t\n \n \n \n\t\t\t\n\n\n\n\t\t\t\n\t\t\n\t\t\n \t\n \t\n \t\n \n \n \n\t\t\t\t\t\n\t\n\n\n\n\n\t\n\t\n\n\t\n\n\t\n\t\n\n\t\t\n\t\t\n\t\t\n \t\n\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\n \t\n \t\n\t\t\t\t\t\n \n \n \n\t\t\n\n\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n\t \t\n \n \n \n\t\t\n\n\n\n\n\n\t\n\t\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \t\n \n \n \n\t\t\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t \t\n\t\t\t\t\t\n \n\t\t\t\n* \n\t\t\t\t\t\n\n\t\t\t\n\n\n\n\n\n\t\n \t\n \n\n\n\n\n\n\t\n\t\n\n\t\t\n\t\t\n\t\t\n \t\n \t\n \n \n \n \n\t\t\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \n \n \n \t\n \n \t\n\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \t\n \n \n\t \n\t\t\t\n\t\t\n\t\t\n\n\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \t\n\t\t\t\t\t\n \n\t \n\t\t\n\t\t\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \t\n \t\n \t\n \n \n\t\t\n\n\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \t\n\t\t\t\t\t\n \n\t \n\t\t\n\t\t\n\n\n\t\t\n\t\t\n\t\t\n\t\t\t\n \t\n \t\n \t\n \n \n\t\t\n\n\n\n\n\n\t\t\n\t\n\t \n \t\n \t\n\t\n\n\n" }, { "path": "ccgbank/templates/insertQuoteSemClassInfo.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\t\n\n\t\n\n\t\n\t\n\t\t\n\t\n\t\n\n\t\n\t\n\t \n\t\t\n\t \n\n\t\n\t\n\n\t\n\t\t \n\t\t\n \n\t\t\t\n\t\t\t 0\">\n\t\t\t\t \n \n\t\t\t \n \n\t\t\t\t \n\t\t\t\t \n\t\t\t\t \n\t\t\t\t\t \n\t\t\t\t \n\t\t\t\t \n 0\">\n\t\t\t\t\t \n\t\t\t\t \n \n \n\n\t\t0 and not(@Header)\">\n\t\t\t\n \n\t\t\n\t\n\n\n\n\n\n \n\n\t\t \n\t\t\t\n\t\t \n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t0 and not(contains(@pos1,'PUNCT'))\">\t\t\t\n\t\t\t\n\t\t\tNAME\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t0\">\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t0\"> \n \n \n \n \n\n\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/insertSemFeats.xsl", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/introduceMMExtns.xsl", "content": "\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\t\n\n\n\t\n\n\n\n\n\t\n\n\n\n\n\t\n\n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/labelAppos.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \t\n\n\n\n\n\n\t\n\t\t\n\t\t\t\t\n\t\t\n\n\t\t\n\t\t\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\n\t\n\t\n\t\t\t\n\t\n\t\t\n \n\t\n\n\n\n\n\n\t\n\t\t\t\t\n\t\n\n\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/labelConj1.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\t\n\t\t\n \t\n \t\n\t\t\t\t\n\t\t\t\n \n \n\n\n\n\n\n\t\n\n\t \n \n \n\n\t\n\t\n\t\n\n\t\n\t\t\n\t\t1]\">\n \t\n\t\t\t\t\n\t\t\t\n \n\n\t\n\n\t\n\n\n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/labelConj2.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n \n \n \n \n\n\n\n\n\t\n\t\t\n \t\n \t\n\t\t\t\t\n\t\t\t\n \n \n\n\n\n\n\t\n\t\t\n \t\n \t\n\t\t\t\t\n\t\t\t\n \n \n\n\n\n\n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/labelConj3.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\t\n\n\n\n\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\n\t\n\n\n\n\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\n\t\n\n\n\n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/labelPlace1.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \t\n\n\n\n\n\n\t\n\t\t\n\t\t\n \t\n\t\t\t\n \n\t\n\n\n\n\n\n\n\t\n\t\t\n\t \t\n \t\n \n \n\n\n\n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/labelPlace2.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\t\n\t\t\n \t\n \t\n \n \n\n\n\n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/labelPuncts.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \t\n\n\n\n\n\n\t\n\t\t\n \t\n \t\n \n \n\t\n\n\n\n\n\t\n\t\t\n \t\n \t\n \n \n\n\n\n\n\n\n\t\n\n\t\t\n\t\t\t\n\t\t\n\t\n\t\t\n\t\t\t\n\t\t\n\n\t\n\t\n\t\n\t\n\t\t\n \n \t\n \t\n\t\n\t\n\n\n\n\t\n\n\n\n\n\n\n\n\n\n\t\n\n\t\n\n\t\t\n\t\t\t\n \n\n\t\t\n\t\t\t\n \n\n\t\t\n\t\t\t\n\t\t\n\n\t\n\n\t\n\t\n\t\t\n \n \t\n \t\n\t\t\n\n\n\n\t\n\t\t\n \n \t\n \t\n\t\t\n\n\n\n\n \n\n\t\n\n\t\n\t\t\n\t\t\n\t\t\t\n \n\n \n\t\t\t\n \n\t\t\n\t\t\n\t\t\t\n \n\t\t\n\t\t\t\n\t\t\n\t\n\n\t\n\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n \t\t\n\t\t\t\t\t\n \t\t\t\t\n \t\t\n \t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\n\n\n\n\n\t\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n \t\t\n\t\t\t\t\t\n \t\t\t\t\n \t\t\n \t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\n\t\n\n\n\n\n\t\n\t\t\n \n \t\n \t\n\t\n\n\n\n\n\n\t\n\t\t\n \n\t\n\n\n\n\n\n\n\t\n\t\t\n\t\n\n\n\n\n\n\t\n \n\t\n\n\n\n\n\n\t\n \n\t\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/lexExtr.xsl", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \t\n \n \n \n \n \n \n \n \n \n \n \n\n \n\t\t\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\t\t\t\t\n\t\t\t\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/macroInsert.xsl", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n\n \n\n \n \n \n \n \n \n \n\n \n \n\n \n \n \n\n \n \n \n \n\t\t\t\n \n \n \n \n \n \n\t\t\t \n \n \n \n \n \n \n\t\t\t\n \n \n\n\t\n\t\n\t\t \n\t\t\t\n\t\t\n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "ccgbank/templates/macroLexDef.xsl", "content": "\n\n\n\n \n \n \n \n \n \n \n \n\n\t\n\t\t\n\t\t\n\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\n\n \n\t\n\n\t\n\n\n \n \n \n \n \n \n \n\n\n\n" }, { "path": "ccgbank/templates/markMistakes.xsl", "content": "\n\n\n\n\n \n \n \n \n\n \n \n\n\t\n\n\t\n\n \n \n \n \n \n\t\n\t\n\n\t\t\n \n \n\n\t\n\n \n \n \n \n \n \n\n" }, { "path": "ccgbank/templates/markUnmatched.xsl", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n \n" }, { "path": "ccgbank/templates/mergeMorph.xsl", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n morph.xml\n \n \n \n \n \n \n \n \n \n\t\n\t\n\t\n\t \n\t\n \n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "ccgbank/templates/morphExtr.xsl", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "ccgbank/templates/normPTBTags.xsl", "content": "\n\n\n\n\n \n \n\n\n\n\n \n\n\n\n \n \n \n\n\n\n \n\t\tX\n \n \n\n\n\n \n\t\t \n\t \n \n\n\n\n\n \n \n \n\n\n\n \n\t\t \n\t \n \n\n\n\n \n\t\tX\n \n \n\n\n\n\n\n \n \n \n\n\n\n" }, { "path": "ccgbank/templates/normPunctPos.xsl", "content": "\n\n\n\n\n \n \n\n\n\n\n \n\n\n\n\n\n\n\n\t\n\t\t\n\t\n \n\n\n\n\t\n\t\t\n\t\n \n\n\n\n\n\t\n\t\t\n\t\n\n\n\n\n\t\n\t\t\n\t\n\n\n\n\n\n\t\n\t\t\n \t\n \t\n \n \n\n\n\n\n\n\n\t\n\t\t\t\n\t\t\tPUNCT_APPOS\n\t\t\n\t\n\n\n\n\n\n\n\n\t\n\t\n\t\t\n\n\t\tnp_~1\\np_1/s[dcl]_2\\(np_3\\np_4/np_1)\n\t\t\n\t\n\n\n\t\n\n\n\n\n\t\n\t\t\n\t\t\n\t\n\n\n\t\n\t\t\n\t\t\n\n\n\n\n\n\n\t\n\t\n\t0]\"/>\n\t\n\n\n\n\n\n\n\n\n\n\t\n\t\t\n\n\t\t \n\n\t\t \n\t\t\t\n\t\t \n\t\t\n \n\n\n\n\n\n\t\n\t\t\n\n\t\t pp[]_~2/np_2\n\t\t \n \n \n \n \n \n\t\t\t\t\t\t\n \n \n \n \n \n \n \n \n\n\n\n\n\t\n\t\t\n\t\n\n\n\n\n\t1]\"/>\n\n\n\n\t1]\"/>\n\n\n\n\n\t\n\t\t\t\n\t\t\n \n \n\t\t1]\"/>\n\t\n\n\n\n\n\t\n \t\t\n\t\n\n\n\n\n\t\n \t\t\n\t\n\n\n\n\n\t\n \t\t\n\t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/origPunctRules.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n,\n\n\n\n \t\n\n\n\n\n\t\n\t\n\t\t\n\t\t\n\t\n\n\n\n\t\n\t\n\t\t\n\t\t\n\t\n\n\n\n\n\t\n\t\n\t\n\t\n\t\n\n\t\t\n\t\n\t\t\n\t\t\t\n\t\t\t0\">\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t0\">\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\n\t\t\n\t\n\n\n\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\n\n\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\n\n\n\n\n\n \n \n \n \n\n\n \n \n \n \n\n\n \n \n \n \n \n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/overtWHLexRels.xsl", "content": "\n\n\n\n \n \n\n\t\n \t\n\t\n\n\t \n \n\n\n\t\n \n\t\t \n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t \n \n\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\n\n\t\n\t\n\t \n \t\n \t\n\t\n\n\t\t\n\t\n\t \n \t\n \t\n\t\n\n\n" }, { "path": "ccgbank/templates/overtWHPronouns.xsl", "content": "\n\n\n\n \n \n\n\t\n \t\n\t\n\n\t\n \n \n \n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n \n \n \n\t\t\t\t\t\n\t\t\t\t\n \n \n\t\t\n \n\n\t\n\t\n\t\t\t\n\t\t\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n \n \n \n\t\t\t\t\t\n\t\t\t\t\n \n \n\t\t\t\t\t\n\t\n\n\t\t\n\t\n\t \n \t\n \t\n\t\n\n\n" }, { "path": "ccgbank/templates/phraseExtractor.xsl", "content": "\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\t\n\t\t\n\t\n\t\n\n\n\n\n\t\n\t\n\t\t\n\t\n\n\n\n\n\n\n\n\t\n\t\t\n\t\n\n\t\n\n\t\n\t\n\t\t\n\t\n\n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/preSentAdj.xsl", "content": "\n\n\n\n\n \n \n\n \n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\t\n\t\t\n\t\t\t\t\n\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\n\t\n\n\n\n\n\t\n\t\t\n\t\n\t\n\n\n\n\t\n\t\t\n\t\n\n\n\n\n\n\n\n\n\t\n\t\t\n\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\n\t\n\n\n\n\n\n\n\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t\n \n \n \n \n \t\n \n \n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\n\n\n\n\n\t\t\t\n\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\n\t\t\n\t\n\n\n\t\n\t\t\n\t\n\n\n\n\n\t2]\"/>\n\n\n\n\n \n \n \n\n\n\n \n \n \n \n \n\n\n\n\t\n \t\t\n\t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/punctLexConjRules.xsl", "content": "\n\n\n\n\n \n \n\n\n\n\n\n\n \n\n\n\n\n\n\n\n\t\n\t\n\t\n\n\t\n\t\n\t\n\t\n\t\t\n\t\t \n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n \t \t\t\n\t\t\t\t\targs\n\t\t\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n \t \t\t\t\n\t\t\t\t\t\tres\n\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t \n\t\t1]\"/>\n\t\n\n\n\n\n\n\n\n\n\n\n\t\n\t\n\t\n\t\n\n\t\n\t\n\t\n\n\t\n\t\t\t\n\n\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t \t\n\t\t\t\tres\n\t\t\t\t\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t \t\n\t\t\t\targs\n\t\t\t\t\t\t\t\t\n\t\t\t\n\t\t1]\"/>\n\t\n\n\n\n\n\n\n\n\n\n\t\n\n\t\n\t\n\n\t\n\t\n\t\n\n\t\n\t\n\t\n\n\t\n\t\n\t\n\n\t\n\t\t\t\n\n\t\n\t\t\t\t\n\t\t\t\n\t\n\t\t\t\n\t\t\t\t\n \t \t\n\t\t\t\tres\n\t\t\t\t\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t \t\n\t\t\t\targs\n\t\t\t\t\t\t\t\t\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t \t\n\t\t\t\targs\n\t\t\t\t\t\t\t\t\n\n\t\t\n\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\t\n\n\t\t\t\n \t\n\t\t\t\t\t\n\t\t\t\t\t\t\n \t \t\t\t\n\t\t\t\t\t\tres\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n \t \t\t\n\t\t\t\t\targs\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n \t\n \t\n \t\n \t\n \t\n\t\t\t\t\t\n \t\n \t\n \t\n \t\n \t\n \t\n\t\t\t\t1]\"/>\n\t\t\t\t\n\t\t\t\n \t\t\n \t\t\n \t\t\n \t\n \t\n \t\n\t\t\t\n\t\t\t\n\t\t\t\t1]\"/>\n\t\t\t\n\t\t\n\t\n\n \n\n\n\t\n \n\n\n\n\n\n\n\t\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\n\t\n\t\n\t\n\n\t\n\t\n\t\n\t\n\n\t\n\t\n\n\t\t\n\t\n\t\n\t\t\t\n\n\t\t\n\t\t\n\n\t\t\t\n\t\t\t\t\n \t \t\n\t\t\t\tres\n\t\t\t\t\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t \t\n\t\t\t\targs\n\t\t\t\t\t\t\t\t\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t \t\n\t\t\t\targs\n\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n \t \t\t\n\t\t\t\t\targs\n\t\t\t\t\t\t\t\t\t\n\t\t\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n \t \t\n\t\t\t\targs\n\t\t\t\t\t\t\t\t\n\n\t\t \t \n\t\n\n \n\n\n\n\t\n \n\n\n\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\t\t\n\t \t\n \t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t \t\t\n \t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\n \n\n\n\n\n\t\n\t1]\" mode=\"args\"/>\n \n\n\n\n\t\n\t\n\t\t\n \t\t\n\t\t\t\n \t\t\n \t\n\t\n \n\n\n\n\n\n\n\n\t\n\t\t\n \t\t\n \t\t \n \t\t\n \t\n\n \n\n\n\n\t\n\t \n \n\n\n\n\n \n \t\n\t\n\n\n\n\n\t\n \t\t\n\t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/reinsertPTBInfo.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\t\n\t\t\n\t\n\n\t\n\t\t\n\t\n\n\n\t\n\n \n \n \n\n\t\n\n \n\n\t\t\n\t\t\t\n\t\t\t\n\n\t\t\n\n \n \n\n\t\n\t\t\n\t\n\n\n\n\n\n \n\n \n\t\t\n \t\n\t\t\t\t\t\n\t\t\t\n\t\t\n \n \n\n\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/repairUnmatched.xsl", "content": "\n\n\n\n \n \n \n \n\n\t\n \n \n\n\t\n\t\t\n\t\t\t\n\t\t\n\t\n\n\t\n\n \n\n\t\n\t\t\n\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\n\t\t\n\t\t\n\n\t\t\n\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n \n\n\t\n\t\t\n\n\t\t\n\n\t\t\n\t\t\t\n\t\t\t\t\n \t\t\n\t\t\n\t\t\t\n \n\t\n\t\n \n \n \n \n\n \n \n \n \n \n \n \n" }, { "path": "ccgbank/templates/replaceColons.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\t\n\t \n\t\t\n\t\t\t\n\t\t\t\n\n\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\n\n\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\n \n\t\t\n\t\n\n\t\n\t\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\n\t\n\n\n\n\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\n\t\n\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/rulesExtr.xsl", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \t\n \n \n \n \n \n \n \n \t\t \n \n\n \n \t\t \n \n\n \t\n \t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t \n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n \t\t\t \n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t \n \n \n \t \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "ccgbank/templates/sentFinalPuncts.xsl", "content": "\n\n\n\n\n \n \n\n\n\n\n\n\n \n\n\n\n\n\t1]\"/>\t\t\n\n\n\n\n\t\n\n\n\n\n\n\n\t\n\t\t\n\t\t\n \t\n \t\n \n \n\t\t\t \n\t\t\t\t\t\n \t\t\t\n\t\t\t\t\t\n\t\t\t \n \n\t\n\t\n\n\n\n\n\n\t\n \t\t\n\t\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/trueCaser.xsl", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n \n \n \n \n \n\n\n\n\n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "ccgbank/templates/uncurryBareParse.xsl", "content": "\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n\n\t\n\t\t\n\t\n\n\n\n\n\t\n \n\n\n\n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "devel/BEN.TODO", "content": "for tuesday 1/9/07:\r\n\r\n-- test new code in ccg.ply for --prefix and such. first make sure it\r\n still works with no --prefix option, producing files with prefixes from\r\n the filename and that foo-grammar.xml has the right names in it that it\r\n points to. (and no interference from the graphics code.) then test with\r\n --prefix and make sure it does what it should (again check grammar.xml),\r\n even if --prefix is specified to be blank.\r\n-- make sure that the existing test grammars still work. (might need to be\r\n done after resolving the issue of caps vs. lowercase, since the current\r\n grammars already use switch feature vars to lowercase, which will be\r\n wrongly interpreted as semantic vars.) check that the testbed still does\r\n what it should and gives results it should.\r\n-- take a look at the rough grammar; make sure it gets compiled OK by TeX\r\n and the results come out. maybe add some more text.\r\n-- think about that mod to family, so that a simple family with one entry\r\n can appear without extra syntax:\r\n\r\nfamily foo: entry of some sort: hybrid logic spec;\r\n\r\nwhich is the same as\r\n\r\n\r\nfamily foo {\r\n entry: entry of some sort: hybrid logic spec;\r\n}\r\n" }, { "path": "devel/schedule.txt", "content": "CCG editor schedule\r\n\r\nStep 1\r\n\r\n- get a simple editor working; has buttons and/or menu across the top with\r\n new/edit/save buttons; for \"formatted text\" just display text as-is\r\n -- add some small buttons at top of editing area for insertion of common\r\n CCG structures\r\n -- add popup help on these buttons (right click, select \"help\", or something)\r\n that shows in more detail the expected format\r\n -- add extra buttons (top or bottom) for particular CCG symbols (e.g. the\r\n symbols used to indicate different slash modes); pressing on them\r\n inserts the appropriate symbol char(s) and they have help text shown\r\n when moving over them\r\n\r\nMore strictly:\r\n\r\n-- use the PyEdit code from Python Cookbook (not sure about legality; if\r\n not, we can just rewrite it, but i don't think it'll be a problem).\r\n-- run the editor, examine it in action, figure out how it works by examining\r\n the code; in particular, figure out how keystrokes are handled, since i\r\n don't see any keystroke code.\r\n-- create two separate modules. one just compiles the code, while the other\r\n is the GUI.\r\n-- one idea for layout:\r\n(1) for the GUI, originally follow Wikipedia's model in which the layout\r\n view is the \"normal\" view and editing is performed as a temporary\r\n operation with a \"preview\" that brings up the layout view in a pane\r\n above the edit window, so that further editing can be performed,\r\n in addition to an \"accept\" that goes back to the regular layout view only.\r\n\r\n(2) this one is editor centric. View pops up a \"view\" [FIX THIS] add a\r\n toolbar under the window with textual buttons for Load, New, View;\r\n originally display an empty edit screen. New, Load create new\r\n top-level windows; make sure that multiple windows work. Load loads an\r\n existing file into a new window. View runs the compiler (see below),\r\n and displays the error results into a new pane below the edit window.\r\n There should be next-error and previous-error commands. This pane can\r\n be hidden, but it reappears on the next next/previous-error commands.\r\n\r\n-- copy all related files from quiche/mangue, get full environment set up\r\n locally; also create subversion repository\r\n\r\n-- add a \"compile\" button; this runs the CCG2XML compiler and displays any\r\n error messages in a separate pane of the window; pane can be viewed or\r\n hidden using menu item or button\r\n -- then, parse error messages to get the line number; clicking on an error\r\n message shows the appropriate line in the text\r\n-- improve on display\r\n -- run compiler before displaying; using result of parsing, display separate\r\n sections corresponding to major sections of text (e.g. feature {}\r\n declaration, family {} declarations)\r\n -- initially, just show actual text corresponding to section\r\n -- get working the ability to hide/display a particular section and to\r\n edit just that section\r\n -- gradually, format into something more intelligent; do this section\r\n by section\r\n -- we already have some code for formatting CCG categories\r\n -- add controls for display of CCG categories\r\n-- figure out what to do about macros\r\n -- at first, just do whatever works in order to get the editor up and running\r\n -- macro definitions themselves should be shown with their raw text; it's\r\n not clear we can do much more\r\n -- macros can cause arbitrary text to be expanded at a particular position;\r\n perhaps in display we show macro-expanded sections in a separate color,\r\n and allow the sections to be collapsed back to the original text that\r\n led to the expansions, on a macro-by-macro basis (some sort of options\r\n menu showing all macros and check marks by them for expansion or not)\r\n\r\n\r\n\r\n\r\nStep 1:\r\n\r\ntoolbar across top, with buttons \"Display\", \"Edit\", \"Test\"\r\neach of these changes the widgets below it, and potentially the menubar above\r\n it; each has a second toolbar above a large text widget showing something\r\ntitle bar should show name of file\r\nstatus bar at the bottom, ala emacs\r\n shows the current mode (display/edit/test) plus other mode-specific info\r\n (e.g. for edit, success or failure of recent compilation)\r\nDisplay:\r\n buttons for controlling the display (e.g. presence or absence of features)\r\n widget showing formatted display\r\nEdit:\r\n buttons for editing common CCG structures\r\n has a preview button, which compiles the text and then displays a second\r\n pane below, showing either the errors from compilation or \r\n has a save button, which saves \r\n text editor widget\r\nTest:\r\n FIXME: fill in; should allow for running the CCG interpreter and/or web\r\n front end (or something that behaves similarly)\r\n\r\nimplementation:\r\n\r\n- should allow for multiple top-level windows editing different files;\r\n hence do not use global variables for status information, but store\r\n inside of a \"file\" object\r\n\r\nDisplay, Edit, Test are subclasses of Frame\r\nclicking on Display/Edit/Test toolbar buttons:\r\n -- hides the existing frame for this mode and displays the new frame\r\n -- should not cause window resize!\r\n\r\n-- should start in Edit mode\r\n-- when switching from edit to display, if text has been modified, we either\r\nneed to bring up a dialog box asking whether to discard the text, or\r\nremember the modified text and display in the status bar that the text has\r\nbeen modified\r\n-- in Edit mode, you have Save and Display buttons; Display asks to save Save saves text\r\n\r\n another row of toolbar buttons below, for inserting common brings up a \r\n- get a simple editor working; has buttons and/or menu across the top with\r\n new/edit/save buttons; for \"formatted text\" just display text as-is\r\n -- add some small buttons at top of editing area for insertion of common\r\n CCG structures\r\n -- add popup help on these buttons (right click, select \"help\", or something)\r\n that shows in more detail the expected format\r\n -- add extra buttons (top or bottom) for particular CCG symbols (e.g. the\r\n symbols used to indicate different slash modes); pressing on them\r\n inserts the appropriate symbol char(s) and they have help text shown\r\n when moving over them\r\n" }, { "path": "docs/build.xml", "content": "\n\n\n\n \n \t\n \t\n \n\n \n \n \n \n \n" }, { "path": "docs/ccgbank-README", "content": "\nIntroduction\n============\n\nThis README describes how to use the pre-built English models trained\nusing the CCGbank, as well as how to train these models yourself\nstarting with the CCGbank. On the realization side, the models make\nuse of all published work on realization ranking (as of March, 2013)\n-- including discriminatively trained syntactic models, various\nn-grams, features for syntactic agreement and balanced punctuation,\nand features for dependency ordering and dependency length\nminimization -- as well as unpublished improvements to the hypertagger\nthat make use of a two-stage, 'stacked' approach to supertag\nprediction. For linux, this release also includes support for using a\nvery large 5-gram memory-mapped language model with KenLM. On the\nparsing side, a reimplementation of Hockenmaier's generative parse\nmodel is used, along with a reimplementation of Curran, Clark &\nVadas's supertagger in Java. Note that the supertagger can also be\nused as a stand-alone tool; see taggers-README for details. The\ngrammars take advantage of refinements to the CCGbank that make use of\nPropbank analyses as well as more precise analyses of punctuation.\n(See references at bottom.)\n\nSince the pre-built English models and CCGbank data for training\nrepresent much larger downloads than the OpenCCG core files, they are\navailable as separate downloads (where YYYY-MM-DD represents the date\nof creation):\n\nenglish-models.YYYY-MM-DD.tgz\nccgbank-data.YYYY-MM-DD.tgz\n\nFor linux, the very large KenLM language model, based on 5-grams in\nthe Gigaword-4 corpus, is available as follows:\n \ngigaword4.5g.kenlm.bin\n\n\nUsing the pre-built English models\n==================================\n\nThe pre-built statisical models for English allow you to parse novel\ntext in English and to generate English sentences from the (quasi-)\nlogical forms of the resulting parses, thereby producing a variety of\ngrammatical paraphrases. Future releases will contain tools for\ngenerating a broader range of paraphrases using disjunctive logical\nforms, which can handle logical forms with similar, but not identical,\nstructures.\n\nIt is also possible to use the pre-built models to realize sentences\nfrom logical forms constructed programmatically. Note, however, that\nin comparison to realization with small, hand-crafted grammars,\nrealization with the broad coverage grammar derived from the CCGbank\nis much slower (with realization typically taking a few seconds per\nsentence). For NLG applications, an interesting task for future work\nwould be to automatically shrink the grammar's coverage to what is\nneeded for a specific domain, which should yield considerable\nimprovements in efficiency.\n\nThe pre-built statistical models do not require you to have a copy of\nthe CCGbank, and should run across different Java platforms. They do,\nhowever, assume the use of the Stanford Core NLP tools for\ntokenization, named entity tagging and morphological analysis. The\nStanford NLP tools are licensed under the full GPL (rather than the\nLGPL, as with OpenCCG), and thus these tools are only loosely\nintegrated into a chain of command-line tool invocations. If the GPL\nis not adequate for your purposes, you'll need to find your own\nsubstitute tools.\n\nThe first step is to make sure you have configured your environment\nvariables and increased your Java memory limit as described in the\nmain README ($OPENCCG_HOME/README). A limit of 2g may be ok, though\n4g may work even better; when using the very large KenLM language\nmodel, you should use a limit of at least 8g.\n\nThe next step is to download the current version of\nenglish-models.YYYY-MM-DD.tgz, and move it into the ccgbank directory,\ni.e. $OPENCCG_HOME/ccgbank/, with the undated name english-models.tgz.\nFrom this directory, you can unpack the English models archive as\nshown below:\n\n$ mv english-models.YYYY-MM-DD.tgz $OPENCCG_HOME/ccgbank/english-models.tgz\n$ cd $OPENCCG_HOME/ccgbank\n$ ccg-build -f build-release.xml extract-models\n\nThis command uses ant to extract the models in a cross-platform way.\nAs noted in the main README, ccg-build is a simple front end for ant\nthat configures the classpath (and a couple of properties) before\ninvoking ant. The option '-f build-release.xml' simply says to use\nthe build-release.xml build file instead of the default build file.\nThe 'extract-models' target unpacks the archive in a way that makes\nsure the archive is in the right place before unpacking it. If\nextracting the models this way yields an error, however, you should\nuse tar (or some other archive extraction tool), eg as follows:\n\n$ mv english-models.YYYY-MM-DD.tgz $OPENCCG_HOME/ccgbank/english-models.tgz\n$ cd $OPENCCG_HOME/ccgbank\n$ tar xzf english-models.tgz\n\nOn linux, after downloading the very large language model file, you\ncan install it for use as follows:\n\n$ mv gigaword4.5g.kenlm.bin $OPENCCG_HOME/ccgbank/models/realizer/.\n\nAs noted in the main README, to use the very large LM, you'll also\nneed to set the library load path:\n\n$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$OPENCCG_HOME/lib\n\nIf the file gigaword4.5g.kenlm.bin is not found, the Treebank-trained\ntrigram model is reused as a stand-in, which will negatively impact\nrealization quality to some extent. In principle it should also be\npossible to build your own large binary 5-gram model. At present,\nhowever, there is no working JNI interface to KenLM for OS X or\nWindows included. See http://kheafield.com/code/kenlm/ for further\ninformation on building large language models and getting KenLM to\ncompile on different platforms, if you would like to try getting the\nJNI interface working beyond linux.\n\nAfter that, the next step is installing the Stanford Core NLP tools,\nas described in the section with this name below.\n\nAt this point you should be ready to try out the models. You can try\nparsing and realization on a file with a couple of novel sentences\ncontaining some named entities that did not exist when the Penn\nTreebank was created, as well as an adverb that is too infrequent to\nappear in the training set lexicon, as follows:\n\n$ ccg-build -f build-ps.xml test-novel &> logs/log.ps.test.novel &\n\nYou can follow the progress of the parsing tool chain by looking in\nthe log file. The input file is data/novel/two-sents, and the output\nfiles are stored in a newly created directory\ndata/novel/two-sents.dir/. The tool chain does PTB tokenization,\ntruecasing, named entity tagging, POS tagging and stemming in order to\ncreate a truecased version of each sentence as well as a morph file\nthat includes all the words in the file. The sentences are then\nparsed, and the resulting logical forms appear in a testbed file\ndata/novel/two-sents.dir/tb.xml. The tool chain takes a while to run\n(a couple of minutes, perhaps) as it must load several large data\nmodels; thus, for more efficient processing, it will make sense to run\nmany more than two sentences in a batch.\n\nOnce the parser has run, you can test the realizer on the resulting\ntestbed file as follows:\n\n$ ccg-build -f build-rz.xml test-novel &> logs/log.rz.test.novel &\n\nAgain, you can follow the progress by consulting the log file. The\nrealizations should appear in data/novel/two-sents.dir/realize.nbest.\n\nThe input file for this test is given by the novel.file property (in\nbuild-models.properties). You can override this property on the\ncommand line to re-use these build files on your own text, where\n is the name of your file:\n\n$ ccg-build -Dnovel.file= -f build-ps.xml test-novel &> logs/log.ps.test.novel &\n$ ccg-build -Dnovel.file= -f build-rz.xml test-novel &> logs/log.rz.test.novel &\n\nAfter running these commands, the parses and realizations will be in a\nnew directory .dir/.\n\n\nTokenization and Normalization of Novel Input Texts\n===================================================\n\nNote that when parsing text, we assume Penn Treebank III (PTB3) \ntokenization and character escaping conventions (see the PTB3 \ndocumentation).***\n\nWe assume the following:\n\n-Text is one sentence (or coherent utterance) per line with no \n intervening blank lines. (This you must do yourself.)\n\n-PTB3 parser-friendly tokenization, e.g.:\n\n do n't, wo n't, it 's, John 's, etc.\n\n and not: \n\n don 't, won 't, it ' s, John ' s, etc.\n (as some tokenization scripts are fond of doing)\n\n Of course, punctuation should be split off from words, so, e.g.:\n\n John said ``Hello'' to Mary, who replied ``Hi, John''.\n\n should be tokenized as:\n\n John said `` Hello '' to Mary , who replied `` Hi , John '' .\n\n-PTB3 escapes for characters that are meta-symbols in the PTB3, e.g.:\n (, ), { and } \n\n become: \n\n -LRB-, -RRB-, -LCB- and -RCB-, respectively.\n\n-\"LaTeX\"-style double quotation marks:\n\n `` Hello '' , said John to Mary .\n\n and not:\n\n \" Hello \" , said John to Mary .\n\t\n-Attributive quotations should be formatted in the \"logical\"\n or \"British\" style and not the \"American\" style, so, e.g.:\n\n `` Hello '' , said John to Mary .\n\n and NOT:\n\n `` Hello , '' said John to Mary.\n\n (N.B. this is the only other thing related to sentence segmentation\n and tokenization that the Stanford PTBTokenizer.java code does not \n accomplish.)\n\n-Also, Unicode punctuation symbols outside the ASCII range\n should be converted to their ASCII equivalents, e.g.:\n\n … should be re-written as ... \n\n-When a sentence-final abbreviation ending with a period/full-stop,\n ends a declarative sentence (i.e., there is therefore no sentence\n final punctuation mark), the final period/full-stop should not be\n split off so that we have, e.g.:\n\n Many products sold at Smithy's Department Store are not produced\n in the U.S.\n\n tokenized as:\n\n Many products sold at Smithy 's Department Store are not produced\n in the U.S. \n\n and not:\n\n Many products sold at Smithy 's Department Store are not produced\n in the U.S . \n\nThese text transformations will bring novel texts more in line with the \ntexts the parser and realizer were trained on and will improve parsing\nand realization performance. \n\nFinally, all text passed to the parser is assumed to be encoded as \nUTF-8 or ASCII (the latter being a subset of UTF-8). Other encodings\nmay cause the parser or realizer to crash in unforseen ways. \n\n***The Stanford Core NLP tools (which we use, see below) already do most\nof what this section covers, so if you plan not to use the Stanford\ntools you will need to find a replacement. The Stanford PTBTokenizer.java\ncode does all that we mentioned above, except transforming \"American\"-\nstyle quotations to \"British/logical\"-style quotations, and ensuring \nthat texts are encoded in UTF-8. American-to-logical quotation \ntransformation is performed by the code in:\n\n$OPENCCG_HOME/ccgbank/bin/american-to-logical-quotes.py \n\nThis is invoked by Ant through build-ps.xml.\n\nUnicode X-to-UTF-8 conversion can be accomplished with the script: \n\n$OPENCCG_HOME/ccgbank/bin/toUTF-8.py \n(assuming that Python's 'chardet' package has been installed). \n\nThe first takes text from and pipe text to ; the second\nhas various option flags (type 'python toUTF-8.py -h' for more \ninformation. Both have only been tested with Python 2.6x.\n\nAs for formatting your texts to be one sentence per line, this you\nmust do yourself, as we cannot anticipate what forms of marked up\ntexts will be passed in for parsing.\n\n\nBuilding the English models from the CCGbank\n============================================\n\nYou can train your own English models if you have a licensed copy of\nthe CCGbank. To build the models, you'll need to download and patch\nZhang Le's maxent toolkit and install the SRILM language modeling\ntoolkit, assuming their licenses are compatible with your usage. In\ntheory it may be possible to use these tools on different platforms,\nbut in practice it will be much easier to use a linux platform,\npreferably one with multiple processors and lots of memory. For\ninstalling Zhang Le's maxent toolkit, see the section with this title\nbelow. For the SRILM toolkit, follow the installation instructions on\nthe SRILM website, and make sure the SRILM executables are available\non your PATH environment variable.\n\nAfter installing the required toolkits, the next step is to download\nthe current version of ccgbank-data.YYYY-MM-DD.tgz, and move it into\nthe ccgbank directory, i.e. $OPENCCG_HOME/ccgbank/, with the undated\nname ccgbank-data.tgz. You'll also need to create a symbolic link to\nyour original CCGbank directory from $OPENCCG_HOME/ccgbank/.\n(Alternatively, you can edit the original.ccgbank.dir property in the\nbuild.properties file in the ccgbank directory.) From this directory,\nyou can unpack the data archive as shown below (where\n is the path to your original CCGbank directory):\n\n$ mv ccgbank-data.YYYY-MM-DD.tgz $OPENCCG_HOME/ccgbank/ccgbank-data.tgz\n$ cd $OPENCCG_HOME/ccgbank\n$ ln -s /ccgbank1.1\n$ ccg-build -f build-release.xml extract-data\n\nThe ccgbank-data tarball contains a patch file for converting the\noriginal CCGbank to the Propbank-enhanced version described in Boxwell\nand White (2008). The 'extract-data' target in build-release.xml does\nthe patching after unpacking the archive and doing a space-to-newline\nconversion that enables the patch file to only contain the real\ndifferences between these CCGbank versions, given that diff works\nline-by-line and CCGbank derivations are given one per line. (Note\nthat this conversion also ensures that the CCGbank cannot be recovered\nfrom the patch file, thereby avoiding a copyright violation for\ndistributing the CCGbank.)\n\nThe ccgbank-data tarball also contains various auxiliary files that\nmake it possible to use the BBN named entity annotations on the Penn\nTreebank as well as to insert the quotes that were unfortunately\nremoved from the original CCGbank.\n\nOnce the Propbank-enhanced version of the CCGbank has been created and\nthe aux files unpacked, the next step is to convert this version of\nthe CCGbank to the one used by OpenCCG, which has a refined treatment\nof punctuation, refined categories for various function words,\ncollapsed named entities and truecased text. The conversion is done\nby a series of XSLT transformations, which have the advantage of the\ndeclarative use of XPATH matching but unfortunately end up making the\nconversion quite slow. As such, the best way to do the conversion is\nto convert the sections in parallel. The bin/convert_all script\nconverts all sections in parallel, as follows:\n\n$ cd $OPENCCG_HOME/ccgbank\n$ bin/convert_all\n\nAs each section requires up to 1GB of memory to convert, it really\nonly makes sense to convert all sections in parallel on a machine with\nat least 25GB of memory and multiple processors. The bin/convert_all\nscript is very simple and can easily be edited to run fewer sections\nin parallel at a time. Running all sections in parallel should take\nless than an hour.\n\nOnce the corpus conversion is complete, the next step is to extract\ngrammars for the training, development and test sections of the\nCCGbank. (For testing, normally only the morph file is used from the\ndev and test sets, together with the training set grammar.) The\ngrammar extraction process also creates testbed files for these\nsections, which contain logical forms derived by following the\ngold-standard derivations. Grammar extraction is done using the\n'extract-various' target in the main build file (build.xml, the\ndefault):\n\n$ cd $OPENCCG_HOME/ccgbank\n$ ccg-build extract-various &> logs/log.extract.various &\n\nExtracting the grammars and creating the logical forms may take up to\nan hour and a half or so. Following corpus conversion and grammar\nextraction, the next step is to train the models. Most of the models\ncan be trained using the 'all' target in build-models.xml:\n\n$ cd $OPENCCG_HOME/ccgbank\n$ ccg-build -f build-models.xml all &> logs/log.models.all &\n\nThis target trains the supertagger and hypertagger as well as the\ngenerative parsing and realization models. As these models require\nseveral maxent training runs, this step will take a while, for example\nup to 24 hours depending on the speed of the machine. (In principle\nthese steps could be partly parallelized, but doing so would be\nnontrivial given the existing dependencies between steps.)\n\nThe final step is to train the realizer's averaged perceptron model.\n(It is also possible to train an averaged perceptron model for the\nparser, but it has not been found to yield significant gains over the\ngenerative model, most likely due to the size of the discrimination\nspace.) Note that you should first install the very large language\nmodel as described in the section on using the pre-built models, if\npossible.\n\nTraining the perceptron model requires generating training events for\neach training section, which is quite time consuming. Event\ngeneration is easily done in parallel, so the perceptron training\nsequence has been broken up to allow this step to be done separately:\n\n$ cd $OPENCCG_HOME/ccgbank\n$ ccg-build -f build-rz.xml event-gen-prep &> logs/log.rz.event.gen.prep &\n$ bin/gen_realizer_events_a &\n$ bin/gen_realizer_events_b &\n$ bin/gen_realizer_events_c &\n$ bin/gen_realizer_events_d &\n$ bin/gen_realizer_events_e &\n\nPreparing for event generation is fairly quick; once that's done, the\nbin/gen_realizer_events* scripts can all be run at the same time. These\nscripts are set up to run five sections at a time; they can be easily\nedited to run more or fewer in parallel. Event generation may take up\nto 12 hours, even with running 5 sections in parallel.\n\nOnce event generation is complete, the actual perceptron training can\nbe run:\n\n$ ccg-build -f build-rz.xml train-perceptron &> logs/log.rz.train.perceptron &\n\nPerceptron training is apt to take 8 hours. If your machine has 16g\nof memory available, training can be run in less than half the time by\ncommenting in the \"-in_mem\" options in build-rz.xml, and editing\nbin/ccg-env to use a memory limit of 16g.\n\nWith the models all built, the parser and realizer can be tested on\nthe CCGbank development section:\n\n$ cd $OPENCCG_HOME/ccgbank\n$ ccg-build -f build-ps.xml test &> logs/log.ps.test &\n$ ccg-build -f build-rz.xml test-perceptron &> logs/log.rz.test.perceptron &\n\nIf the models have been built correctly, the realization exact matches\nshould be over 46% (using the very large language model), and the\nparsing unlabeled dependencies f-score should be about 0.89. (Note\nthat with named entities collapsed and some function words not\nrepresented in the logical forms, these f-scores are not comparable to\nthe dependency f-scores reported for other CCG parsers on the\nCCGbank.)\n\nNaturally, the models can also be run on novel text, as described in\nthe section on using pre-built models.\n\n\nViewing CCGbank derivations\n===========================\n\nDerivations in the original or converted CCGbank can be viewed as\ntrees using ccg-draw-tree, a front-end to the tree-drawing routine in\nNLTK's Tree class. This tool reads in .auto files, so to view\nconverted CCGbank derivations, you must first export them to .auto\nformat, as shown below:\n\n$ cd $OPENCCG_HOME/ccgbank\n$ ccg-build export-to-auto\n$ ccg-draw-tree convert/00/wsj_0001.auto wsj_0001.1\n\nThe export-to-auto target creates auto files for the current 'sect'\nand 'file' properties, which can be set in build.properties or\nprovided on the command line (e.g. by -Dfile=*). This example assumes\nthat the first file in Section 00 is included in the files to export\nin .auto format. Displaying the derivation for the first sentence in\nthe converted CCGbank shows how balanced appositive commas are handled\nwith \"Pierre_Vinken , 61 years old ,\" (as well as a collapsed named\nentity), and also shows how \"as a nonexecutive director\" has been\nconverted into an adjunct rather than an argument of \"join\".\n\n\nInstalling the Stanford Core NLP tools\n======================================\n\nFor novel text, it is useful to perform named entity (NE) \nrecognition and labelling, as well as morphological processing \n(lemmatization). For this we use the Stanford Core NLP tools, \navailable at:\n\nhttp://www-nlp.stanford.edu/software/corenlp.shtml\n\nTo install them, simply download the archive from the\nabove link, unpack it, and place the Core NLP JAR file\nin the 'ccgbank/stanford-nlp' directory, re-naming it\n'stanford-core-nlp.jar'. \n\nIf you wish to use another location or naming convention, \nyou will need to update the properties file \n(ccgbank/build-ps.properties).\n\nIf you're using a bash shell, you might do the following:\n\n$ mkdir $OPENCCG_HOME/ccgbank/tmp\n$ cd $OPENCCG_HOME/ccgbank/tmp\n$ wget http://www-nlp.stanford.edu/software/stanford-corenlp-20xx-xx-xx.tgz\n$ tar xvfz stanford-corenlp-20xx-xx-xx.tgz\n$ cd stanford-corenlp-20xx-xx-xx\n$ cp stanford-corenlp-20xx-xx-xx.jar $OPENCCG_HOME/ccgbank/stanford-nlp/stanford-core-nlp.jar\n\nNote you will need to fill in the date, 20xx-xx-xx, with whatever \nthe date of the most recent release is (see the particular file\nyou get from the Stanford website).\n\nTo get the NE tagging models out of this download (the Stanford NE \ntagger combines preditions from multiple models), you will need to \nlocate the 'stanford-corenlp-20xx-xx-xx-models.jar' file (in the \nsame directory as the other JAR files) and un-jar it, like so \n[assuming you're still in the same directory as above]:\n\n$ jar xf stanford-corenlp-20xx-xx-xx-models.jar\n$ cp edu/stanford/nlp/models/ner/* $OPENCCG_HOME/ccgbank/stanford-nlp/classifiers/.\n$ rm -rf edu\n\nYou should check that the filenames for the NE tagging models copied\nto $OPENCCG_HOME/ccgbank/stanford-nlp/classifiers match those listed\nin $OPENCCG_HOME/ccgbank/build-ps.properties as ner.model1, ner.model2\nand ner.model3, updating this properties file if necessary.\n\nThis will set up the only external dependency we have for\nparsing novel text using 'ccgbank/build-ps.xml'.\n\nIf you wish to recompile (perhaps after modifying) the \napplication that interfaces with the Stanford NE tagger's API, \nsee the ant build file $OPENCCG_HOME/ccgbank/bin/ner/build-ner-api.xml\nand its corresponding properties file (in the same location).\n\n\nInstalling Zhang Le's maxent toolkit\n====================================\n\nTo train the models, you'll need to have Zhang Le's maxent toolkit\nworking with a small patch. Do the following to install the patched\nversion of Zhang Le's toolkit in a directory of your choice:\n\n$ cd\n$ wget http://homepages.inf.ed.ac.uk/lzhang10/software/maxent/maxent-20061005.tar.bz2\n\nUnpack and patch the maxent.cpp file (it doesn't\ncover the case where ':' can be part of the feature\nsymbol itself, and not just a delimiter that separates\nstring representations of features (contextual predicates, \nactually) from their real-valued activations).\n\n$ bunzip2 maxent-20061005.tar.bz2\n$ tar xf maxent-20061005.tar\n$ cd maxent-20061005/src\n$ patch maxent.cpp $OPENCCG_HOME/docs/maxent.cpp.patched\n\nNow compile the maxent code.\n\n$ cd \n$ cd maxent-20061005\n$ make clean all unittest\n\nTest to make sure it (more or less) works (7 out of the 8 tests only\nseem to pass, but the training seems to work):\n\n$ cd test\n$ ./runall.py\n\nFinally, add the 'maxent' binary (under 'maxent-20061005/src/opt')\nto your PATH environment variable.\n\n\nReferences\n==========\n\nMichael White and Rajakrishnan Rajkumar. 2012. Minimal Dependency\nLength in Realization Ranking. In Proc. EMNLP-12.\nhttp://aclweb.org/anthology-new/D/D12/D12-1023.bib\n\nRajakrishnan Rajkumar and Michael White. 2010. Designing Agreement\nFeatures for Realization Ranking. In Proc. of COLING-10.\nhttp://aclweb.org/anthology-new/C/C10/C10-2119.bib\n\nMichael White and Rajakrishnan Rajkumar. 2009. Perceptron Reranking\nfor CCG Realization. In Proc. of the Conference on Empirical Methods\nin Natural Language Processing (EMNLP 2009).\nhttp://aclweb.org/anthology-new/D/D09/D09-1043.bib\n\nRajakrishnan Rajkumar, Michael White and Dominic\nEspinosa. 2009. Exploiting Named Entity Classes in CCG Surface\nRealization. In Proc. of Human Language Technologies: The 2009 Annual\nConference of the North American Chapter of the Association for\nComputational Linguistics (NAACL HLT 2009).\nhttp://aclweb.org/anthology-new/N/N09/N09-2041.bib\n\nMichael White and Rajakrishnan Rajkumar. 2008. A More Precise Analysis\nof Punctuation for Broad-Coverage Surface Realization with CCG. In\nProc. of the Workshop on Grammar Engineering Across Frameworks (GEAF08).\nhttp://aclweb.org/anthology-new/W/W08/W08-1704.bib\n\nDominic Espinosa, Michael White and Dennis Mehay. 2008. Hypertagging:\nSupertagging for Surface Realization with CCG. In Proceedings of the\n46th Annual Meeting of the Association for Computational Linguistics:\nHuman Language Technologies (ACL-08: HLT).\nhttp://aclweb.org/anthology-new/P/P08/P08-1022.bib\n\nStephen A. Boxwell and Michael White. 2008. Projecting Propbank Roles\nonto the CCGbank. In Proceedings of the Sixth International Conference\non Language Resources and Evaluation (LREC-08).\nhttp://www.lrec-conf.org/proceedings/lrec2008/pdf/789_paper.pdf\n\nJames R. Curran, Stephen Clark and David Vadas. 2006. Multi-Tagging\nfor Lexicalized-Grammar Parsing. In Proc. ACL-06.\nhttp://aclweb.org/anthology-new/P/P06/P06-1088.bib\n\nJulia Hockenmaier and Mark Steedman. 2002. Generative Models for\nStatistical Parsing with Combinatory Categorial Grammar. In Proc. ACL-02.\nhttp://aclweb.org/anthology-new/P/P02/P02-1043.bib\n\n" }, { "path": "docs/guide/build.xml", "content": "\n\n\n\n\t\n\t\t\n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\n\t \t\n\t \n\n\t \n\t \n\t\t \n\t \n\t \n\t\t \n\t \n\t \n\t \n\t \n\t \n\n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t\t \n\t \n\t \n\t \n\t\t\n\t \n\t \n\t \n\t \n\t \n\n" }, { "path": "docs/guide/cgloss4e.sty", "content": "% -*- LaTeX -*-\r\n% Following borrows from Covington's style files inspired by Midnight by M.\r\n% de Groot, adapted to be used with gb4e.sty: examples beginning with \\ex can\r\n% contain glosses directly. Default is\r\n% Linguistic Inquiry style with all lines in \\rm; to change a line (eg. to\r\n% \\it for a particular journal, change the appropriate line: e.g.,\r\n% \\let\\eachwordone=\\rm in a copy of this file. Note that it will NOT work\r\n% to put \\it before the line as the words are parsed separately.\r\n\r\n% Use \\singlegloss to force single-spaced glosses even in double-space\r\n% environments. Works also in footnotes (^M as delimiter replaced by\r\n% \\\\)---hpk\r\n%\r\n%%%\r\n%%% Sentences with word-by-word glosses\r\n%%%\r\n\r\n% See covingtn.tex for full documentation. Some examples:\r\n%\r\n% Displayed sentence with gloss and translation:\r\n%\r\n% \\gll Dit is een Nederlands voorbeeld.\\\\\r\n% This is a Dutch example.\\\\\r\n% \\glt `This is an example in Dutch.'\r\n%\r\n% Same, using bracketing where words do not correspond one-to-one:\r\n%\r\n% \\gll Dit is een voorbeeldje in het Nederlands.\\\\\r\n% This is a {little example} in {} Dutch.\\\\\r\n% \\glt `This is a little example in Dutch.'\r\n%\r\n% If you want to align 3 lines rather than two, use \\glll instead of \\gll.\r\n%\r\n% Layout is critical between \\gll (or \\glll) and \\glt (or \\gln).\r\n%\r\n% Thanks to Marcel R. van der Goot for permission to reproduce code.\r\n\\let\\@gsingle=1\r\n\\def\\singlegloss{\\let\\@gsingle=1}\r\n\\def\\nosinglegloss{\\let\\@gsingle=0}\r\n\\@ifundefined{new@fontshape}%\r\n {\\def\\@selfnt{\\ifx\\@currsize\\normalsize\\@normalsize\\else\\@currsize\\fi}}\r\n {\\def\\@selfnt{\\selectfont}}\r\n\r\n\\def\\gll% % Introduces 2-line text-and-gloss.\r\n {\\begin{flushleft}\r\n \\ifx\\@gsingle1% conditionally force single spacing (hpk/MC)\r\n \\vskip\\baselineskip\\def\\baselinestretch{1}%\r\n \\@selfnt\\vskip-\\baselineskip\\fi%\r\n \\bgroup\r\n \\twosent\r\n }\r\n\r\n\\def\\glll% % Introduces 3-line text-and-gloss.\r\n {\\begin{flushleft}\r\n \\ifx\\@gsingle1% conditionally force single spacing (hpk/MC)\r\n \\vskip\\baselineskip\\def\\baselinestretch{1}%\r\n \\@selfnt\\vskip-\\baselineskip\\fi%\r\n \\bgroup\r\n \\threesent\r\n }\r\n\r\n\\def\\glt{\\vskip.17\\baselineskip}\r\n% Introduces a translation\r\n\\let\\trans\\glt\r\n\r\n\\def\\glend{} % obsolete\r\n % Ends the gloss environment.\r\n\r\n% The following TeX code is adapted, with permission, from:\r\n% gloss.tex: Macros for vertically aligning words in consecutive sentences.\r\n% Version: 1.0 release: 26 November 1990\r\n% Copyright (c) 1991 Marcel R. van der Goot (marcel@cs.caltech.edu).\r\n% Original Midnight/gloss.tex and Midnight/gloss.doc are available from\r\n% csvax.cs.caltech.edu [131.215.131.131] in pub/tex\r\n% and many other anonymous ftp archives.\r\n\r\n\\newbox\\lineone% boxes with words from first line\r\n\\newbox\\linetwo%\r\n\\newbox\\linethree%\r\n\\newbox\\wordone% a word from the first line (hbox)\r\n\\newbox\\wordtwo%\r\n\\newbox\\wordthree%\r\n\\newbox\\gline% the constructed double line (hbox)\r\n\\newskip\\glossglue% extra glue between glossed pairs or triples\r\n\\glossglue = 0pt plus 2pt minus 1pt % allow stretch/shrink between words\r\n%\\glossglue = 5pt plus 2pt minus 1pt % allow stretch/shrink between words\r\n\\newif\\ifnotdone\r\n\r\n\\@ifundefined{eachwordone}{\\let\\eachwordone=\\rm}{\\relax}\r\n\\@ifundefined{eachwordtwo}{\\let\\eachwordtwo=\\rm}{\\relax}\r\n\\@ifundefined{eachwordthree}{\\let\\eachwordthree=\\rm}{\\relax}\r\n\r\n\\def\\lastword#1#2#3% #1 = \\each, #2 = line box, #3 = word box\r\n {\\setbox#2=\\vbox{\\unvbox#2%\r\n \\global\\setbox#3=\\lastbox%\r\n }%\r\n \\ifvoid#3\\global\\setbox#3=\\hbox{#1\\strut{} }\\fi\r\n % extra space following \\strut in case #1 needs a space\r\n }\r\n\r\n\\def\\testdone\r\n {\\ifdim\\ht\\lineone=0pt\r\n \\ifdim\\ht\\linetwo=0pt \\notdonefalse % tricky space after pt\r\n \\else\\notdonetrue\r\n \\fi\r\n \\else\\notdonetrue\r\n \\fi\r\n }\r\n\r\n\\gdef\\getwords(#1,#2)#3 #4\\\\% #1=linebox, #2=\\each, #3=1st word, #4=remainder\r\n {\\setbox#1=\\vbox{\\hbox{#2\\strut#3 }% adds space\r\n \\unvbox#1%\r\n }%\r\n \\def\\more{#4}%\r\n \\ifx\\more\\empty\\let\\more=\\donewords\r\n \\else\\let\\more=\\getwords\r\n \\fi\r\n \\more(#1,#2)#4\\\\%\r\n }\r\n\r\n\\gdef\\donewords(#1,#2)\\\\{}%\r\n\r\n\\gdef\\twosent#1\\\\ #2\\\\{% #1 = first line, #2 = second line\r\n \\getwords(\\lineone,\\eachwordone)#1 \\\\%\r\n \\getwords(\\linetwo,\\eachwordtwo)#2 \\\\%\r\n \\loop\\lastword{\\eachwordone}{\\lineone}{\\wordone}%\r\n \\lastword{\\eachwordtwo}{\\linetwo}{\\wordtwo}%\r\n \\global\\setbox\\gline=\\hbox{\\unhbox\\gline\r\n \\hskip\\glossglue\r\n \\vtop{\\box\\wordone % vtop was vbox\r\n \\nointerlineskip\r\n \\box\\wordtwo\r\n }%\r\n }%\r\n \\testdone\r\n \\ifnotdone\r\n \\repeat\r\n \\egroup % matches \\bgroup in \\gloss\r\n \\gl@stop}\r\n\r\n\\gdef\\threesent#1\\\\ #2\\\\ #3\\\\{% #1 = first line, #2 = second line, #3 = third\r\n \\getwords(\\lineone,\\eachwordone)#1 \\\\%\r\n \\getwords(\\linetwo,\\eachwordtwo)#2 \\\\%\r\n \\getwords(\\linethree,\\eachwordthree)#3 \\\\%\r\n \\loop\\lastword{\\eachwordone}{\\lineone}{\\wordone}%\r\n \\lastword{\\eachwordtwo}{\\linetwo}{\\wordtwo}%\r\n \\lastword{\\eachwordthree}{\\linethree}{\\wordthree}%\r\n \\global\\setbox\\gline=\\hbox{\\unhbox\\gline\r\n \\hskip\\glossglue\r\n \\vtop{\\box\\wordone % vtop was vbox\r\n \\nointerlineskip\r\n \\box\\wordtwo\r\n \\nointerlineskip\r\n \\box\\wordthree\r\n }%\r\n }%\r\n \\testdone\r\n \\ifnotdone\r\n \\repeat\r\n \\egroup % matches \\bgroup in \\gloss\r\n \\gl@stop}\r\n\r\n\\def\\gl@stop{{\\hskip -\\glossglue}\\unhbox\\gline\\end{flushleft}}\r\n\r\n\\endinput\r\n" }, { "path": "docs/guide/gb4e.sty", "content": "% -*- LaTeX -*-\r\n\\def\\gbVersion{4e}\r\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\r\n% Version 4export (= v. 4 minus the compatibility code)\r\n% Based on hpk's gb.sty, revised for GM syllabus by ct,\r\n% and incorporating macros adapted from J.Frampton, M. de Groot en M.\r\n% Covington. Full documentation soon to come in gb4doc.tex. Bug-reports\r\n% and suggestions for improvements, other used features, please!\r\n%\r\n% Notes:\r\n% \r\n% Various styles for X-bar levels; can be changed, but note that {picture}\r\n% environements (e.g. trees) will then come out wrong and have to be\r\n% fixed\r\n%\r\n% This file allows _ and ^ to be used in ordinary text, hence must be\r\n% loaded AFTER any file that uses them in their TeX meaning. Hence\r\n% cgloss(n).sty is loaded early in this file.\r\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\r\n\r\n%%%%%%%%%%%%%%%%%%%%%%%%\r\n% Format of examples: %\r\n%%%%%%%%%%%%%%%%%%%%%%%%\r\n% \\begin{exe} or \\exbegin\r\n% (arab.)\r\n% \\begin{xlist} or \\xlist\r\n% (1st embedding, alph.)\r\n% \\begin{xlisti} or \\xlisti\r\n% (2st embedding, rom.)\r\n% \\end{xlisti} or \\endxlisti\r\n% \r\n% \\end{xlist} or \\endxlist\r\n% \r\n% \\end{exe} or \\exend\r\n%\r\n% Other sublist-styles: xlistA (Alph.), xlistI (Rom.), xlistn (arab)\r\n%\r\n% \\ex (produces Number)\r\n% \\ex (numbered example)\r\n% \\ex[jdgmt]{sentence} (numbered example with judgement)\r\n%\r\n% \\exi{ident} (produces identifier)\r\n% \\exi{ident} (example numbered with identifier)\r\n% \\exi{ident}[jdgmt]{sentence} (dito with judgement)\r\n% (\\exr, \\exp and \\sn are defined in terms of \\exi)\r\n%\r\n% \\exr{label} (produces cross-referenced Num.)\r\n% \\exr{label} (cross-referenced example)\r\n% \\exr{label}[jdgmt]{sentence} (cross-referenced example with judgement)\r\n%\r\n% \\exp{label} (same as\r\n% \\exp{label} \\exr but\r\n% \\exp{label}[jdgmt]{sentence} with prime)\r\n%\r\n% \\sn (unnumbered example)\r\n% \\sn[jdgmt]{sentence} (unnumbered example with judgement)\r\n%\r\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\r\n\r\n\\@ifundefined{new@fontshape}{\\def\\reset@font{}\\let\\mathrm\\rm}{}\r\n\r\n\\let\\prmbrs=0\r\n\\def\\primebars{\\let\\prmbrs=1}\r\n\\def\\obar#1{\\ifmmode#1^{0}\\else#1$^{0}$\\fi} %% FIX\r\n\\def\\mbar#1{\\ifmmode#1^{\\mathrm{max}}\\else#1$^{\\mathrm{max}}$\\fi}\r\n\\def\\ibar#1{\\ifx\\prmbrs0%\r\n \\ifmmode\\overline{\\mathrm{#1}}\\else$\\overline{\\mbox{#1}}$\\fi%\r\n \\else\\ifmmode#1^{'}\\else#1$^{'}$\\fi\\fi}\r\n\\def\\iibar#1{\\ifx\\prmbrs0%\r\n \\ifmmode\\overline{\\overline{\\mathrm{#1}}}%\r\n \\else$\\overline{\\overline{\\mbox{#1}}}$\\fi%\r\n \\else #1P\\fi}\r\n\\def\\th{\\ifmmode\\theta\\else$\\theta$\\fi}\r\n\\def\\al{\\ifmmode\\alpha\\else$\\alpha$\\fi}\r\n\\def\\be{\\ifmmode\\beta\\else$\\beta$\\fi}\r\n\\def\\ga{\\ifmmode\\gamma\\else$\\gamma$\\fi}\r\n\\def\\de{\\ifmmode\\delta\\else$\\delta$\\fi}\r\n\\def\\spec#1{[Spec,#1]} %Def. of \"Specifier of #1\"\r\n\\def\\ct#1{{\\em #1\\/}} %Citation of linguistic material with alternative style:\r\n%\\def\\ct#1{`#1'}\r\n\\def\\tx{\\bf} %Introduction of technical terms with alternative style:\r\n%\\def\\tx{\\em}\r\n\r\n\\input{cgloss\\gbVersion.sty}\r\n\r\n%%% NEWSTUFF:\r\n%\\newcommand{\\indexgroupmark}[1]{\\item{\\bf #1}} % ?? -CT\r\n\r\n% this allows _ to be used in horizontal mode (from J.Frampton):\r\n% \\catcode`_=\\active\r\n% \\def_#1{\\ifmmode\\mit{\\sb{#1}}\\else${}\\sb{#1}$\\fi}\r\n% \\catcode`^=\\active\r\n% \\def^#1{\\ifmmode\\mit{\\sp{#1}}\\else${}\\sp{#1}$\\fi}\r\n\r\n% \\def\\lb#1{\\@ifnextchar [{\\@glarph{#1}}{\\@bl{#1}}}\r\n%\\def\\@glarph#1[#2]{\\ifmmode{[}\\sb{{\\mathrm{#1}}\\sb{#2}}\\else%\r\n% ${[}\\sb{{\\mathrm{#1}}\\sb{#2}}$\\fi}\r\n% \\def\\@bl#1{\\ifmmode{[}\\sb{\\mathrm{#1}}\\;\\else${[}\\sb{\\mathrm{#1}}\\;$\\fi}\r\n\r\n% \\def\\rb#1{\\@ifnextchar [{\\@grarph{#1}}{\\@br{#1}}}\r\n%\\def\\@grarph#1[#2]{\\ifmmode{]}\\sb{{\\mathrm{#1}}\\sb{#2}}\\else%\r\n% ${]}\\sb{{\\mathrm{#1}}\\sb{#2}}$\\fi}\r\n% \\def\\@br#1{\\ifmmode{]}\\sb{\\mathrm{#1}}\\;\\else${]}\\sb{\\mathrm{#1}}\\;$\\fi}\r\n\r\n%%% END_NEWSTUFF.\r\n\r\n\r\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\r\n% Macros for examples, roughly following Linguistic Inquiry style. %\r\n% From here on best not to tamper, else all the examples and cross- %\r\n% references will come out scrambled! (see also note below) - CT %\r\n% Completely rewritten for more robustness and flexibility. (hpk) %\r\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\r\n\r\n\\def\\qlist{\\begin{list}{\\Alph{xnum}.}{\\usecounter{xnum}%\r\n\\setlength{\\rightmargin}{\\leftmargin}}}\r\n\\def\\endqlist{\\end{list}}\r\n\r\n\\newif\\if@noftnote\\@noftnotetrue\r\n\\newif\\if@xrec\\@xrecfalse\r\n\\@definecounter{fnx}\r\n\r\n%%%% adapted from latex.tex to get examples in footnotes right\r\n\\long\\def\\@footnotetext#1{%\r\n \\@noftnotefalse\\setcounter{fnx}{0}%\r\n \\insert\\footins{\\reset@font\\footnotesize\r\n \\interlinepenalty\\interfootnotelinepenalty\r\n \\splittopskip\\footnotesep\r\n \\splitmaxdepth \\dp\\strutbox \\floatingpenalty \\@MM\r\n \\hsize\\columnwidth \\@parboxrestore\r\n \\edef\\@currentlabel{\\csname p@footnote\\endcsname\\@thefnmark}\\@makefntext\r\n {\\rule{\\z@}{\\footnotesep}\\ignorespaces\r\n #1\\strut}}\\@noftnotetrue}\r\n\r\n\\newcount\\@xnumdepth \\@xnumdepth = 0\r\n\r\n\\@definecounter{xnumi}\r\n\\@definecounter{xnumii}\r\n\\@definecounter{xnumiii}\r\n\\@definecounter{xnumiv}\r\n\\@definecounter{exx}\r\n\\setcounter{exx}{0}\r\n\\def\\thexnumi{\\@xsi{xnumi}}\r\n\\def\\thexnumii{\\@xsii{xnumii}}\r\n\\def\\thexnumiii{\\@xsiii{xnumiii}}\r\n\\def\\thexnumiv{\\@xsiv{xnumiv}}\r\n\\def\\p@xnumii{\\thexnumi}\r\n\\def\\p@xnumiii{\\thexnumi\\thexnumii-}\r\n\\def\\p@xnumiv{\\thexnumi\\thexnumii-\\thexnumiii-}\r\n\r\n\\def\\xs@default#1{\\csname @@xs#1\\endcsname}\r\n\\def\\@@xsi{\\let\\@xsi\\arabic}\r\n\\def\\@@xsii{\\let\\@xsii\\alph}\r\n\\def\\@@xsiii{\\let\\@xsiii\\roman}\r\n\\def\\@@xsiv{\\let\\@xsi\\arabic}\r\n\r\n\\@definecounter{rxnumi}\r\n\\@definecounter{rxnumii}\r\n\\@definecounter{rxnumiii}\r\n\\@definecounter{rxnumiv}\r\n\r\n\\def\\save@counters{%\r\n\\setcounter{rxnumi}{\\value{xnumi}}%\r\n\\setcounter{rxnumii}{\\value{xnumii}}%\r\n\\setcounter{rxnumiii}{\\value{xnumiii}}%\r\n\\setcounter{rxnumiv}{\\value{xnumiv}}}%\r\n\r\n\r\n\\def\\reset@counters{%\r\n\\setcounter{xnumi}{\\value{rxnumi}}%\r\n\\setcounter{xnumii}{\\value{rxnumii}}%\r\n\\setcounter{xnumiii}{\\value{rxnumiii}}%\r\n\\setcounter{xnumiv}{\\value{rxnumiv}}}%\r\n\r\n\r\n\\def\\exewidth#1{\\def\\@exwidth{#1}} \\exewidth{(234)}\r\n\\def\\exe{\\@ifnextchar [{\\@exe}{\\@exe[\\@exwidth]}}\r\n\r\n\\def\\@exe[#1]{\\ifnum \\@xnumdepth >0%\r\n \\if@xrec\\@exrecwarn\\fi%\r\n \\if@noftnote\\@exrecwarn\\fi%\r\n \\@xnumdepth0\\@listdepth0\\@xrectrue%\r\n \\save@counters%\r\n \\fi%\r\n \\advance\\@xnumdepth \\@ne \\@@xsi%\r\n \\begin{list}{(\\thexnumi)}%\r\n {\\usecounter{xnumi}\\@subex{#1}{1em}%\r\n \\if@noftnote%\r\n \\setcounter{xnumi}{\\value{exx}}%\r\n \\else%\r\n \\setcounter{xnumi}{\\value{fnx}}%\r\n \\fi}}\r\n\r\n\\def\\endexe{\\if@noftnote\\setcounter{exx}{\\value{xnumi}}%\r\n \\else\\setcounter{fnx}{\\value{xnumi}}%\r\n \\reset@counters\\@xrecfalse\\fi\\end{list}}\r\n\r\n\\def\\@exrecwarn{\\typeout{*** Recursion on \"exe\"---your\r\n example numbering will probably be screwed up!}}\r\n\r\n\\def\\xlist{\\@ifnextchar [{\\@xlist{}}{\\@xlist{}[iv.]}}\r\n\\def\\xlista{\\@ifnextchar [{\\@xlist{\\alph}}{\\@xlist{\\alph}[m.]}}\r\n\\def\\xlisti{\\@ifnextchar [{\\@xlist{\\roman}}{\\@xlist{\\roman}[iv.]}}\r\n\\def\\xlistn{\\@ifnextchar [{\\@xlist{\\arabic}}{\\@xlist{\\arabic}[9.]}}\r\n\\def\\xlistA{\\@ifnextchar [{\\@xlist{\\Alph}}{\\@xlist{\\Alph}[M.]}}\r\n\\def\\xlistI{\\@ifnextchar [{\\@xlist{\\Roman}}{\\@xlist{\\Roman}[IV.]}}\r\n\r\n\\def\\endxlist{\\end{list}}\r\n\\def\\endxlista{\\end{list}}\r\n\\def\\endxlistn{\\end{list}}\r\n\\def\\endxlistA{\\end{list}}\r\n\\def\\endxlistI{\\end{list}}\r\n\\def\\endxlisti{\\end{list}}\r\n\r\n\r\n%%% a generic sublist-styler\r\n\\def\\@xlist#1[#2]{\\ifnum \\@xnumdepth >3 \\@toodeep\\else%\r\n \\advance\\@xnumdepth \\@ne%\r\n \\edef\\@xnumctr{xnum\\romannumeral\\the\\@xnumdepth}%\r\n \\def\\@bla{#1}\r\n \\ifx\\@bla\\empty\\xs@default{\\romannumeral\\the\\@xnumdepth}\\else%\r\n \\expandafter\\let\\csname @xs\\romannumeral\\the\\@xnumdepth\\endcsname#1\\fi\r\n \\begin{list}{\\csname the\\@xnumctr\\endcsname.}%\r\n {\\usecounter{\\@xnumctr}\\@subex{#2}{1.5ex}}\\fi}\r\n\r\n\\def\\@subex#1#2{\\settowidth{\\labelwidth}{#1}\\itemindent\\z@\\labelsep#2%\r\n \\ifnum\\the\\@xnumdepth=1\\topsep 7\\p@ plus2\\p@ minus3\\p@\\else%\r\n \\topsep 2\\p@ plus2\\p@\\fi\\parsep 2\\p@ plus\\p@ minus\\p@%\r\n \\itemsep \\parsep\\leftmargin\\labelwidth\\advance\\leftmargin#2\\relax}\r\n\r\n%%% the example-items\r\n\\def\\ex{\\@ifnextchar [{\\@ex}{\\item}}\r\n\\def\\@ex[#1]#2{\\item\\@exj[#1]{#2}}\r\n\\def\\@exj[#1]#2{\\@exjbg{#1} #2 \\end{list}}\r\n\\def\\exi#1{\\item[#1]\\@ifnextchar [{\\@exj}{}}\r\n\\def\\judgewidth#1{\\def\\@jwidth{#1}}\r\n\\judgewidth{??}\r\n\\def\\@exjbg#1{\\begin{list}{#1}{\\@subex{\\@jwidth}{.5ex}}\\item}\r\n\\def\\exr#1{\\exi{{(\\ref{#1})}}}\r\n\\def\\exp#1{\\exi{{(\\ref{#1}$'$)}}}\r\n\\def\\sn{\\exi{}}\r\n\\def\\bu{\\item[$\\bullet$]}\r\n\r\n\r\n%%%%%%% \\lcomment for breaks in (example-)lists (leaves all counters\r\n%%%%%%% as they are) (hpk) \r\n\\newlength{\\lcommentsep}\r\n\\lcommentsep = 1ex\r\n\r\n\\long\\def\\lcomment#1%\r\n {\\vspace{\\lcommentsep}\r\n \\item[]\\hspace*{-\\leftmargin}%\r\n \\@tempskipa=\\linewidth%\r\n \\addtolength{\\@tempskipa}{\\rightmargin}%\r\n \\addtolength{\\@tempskipa}{\\leftmargin}%\r\n \\parbox{\\@tempskipa}{#1}%\r\n \\vspace{\\lcommentsep}%\r\n }\r\n\r\n%%%%%% control the alignment of exampleno. and (picture-)example\r\n%%%%%% (by Lex Holt ).\r\n\\def\\attop#1{\\leavevmode\\vtop{\\strut\\vskip-\\baselineskip\\vbox{#1}}}\r\n\\def\\atcenter#1{$\\vcenter{#1}$}\r\n%%%%%%\r\n\r\n%-------------------Move Arrows (from J.Frampton):\r\n\r\n\\def\\leaderfill{\\leaders\\hrule\\hfil}\r\n\\def\\pointerup{\\hbox to 0pt{\\hss\r\n \\vbox{\\offinterlineskip\\vskip-1pt\\hbox{\\elevenex\\char'170}\\null}\\hss}}\r\n\\def\\pointerdown{\\hbox to 0pt{\\hss\r\n \\vtop{\\offinterlineskip\\null\\hbox{\\elevenex\\char'171}\\vskip-1pt}\\hss}}\r\n\r\n\\let\\pu=\\pointerup\r\n\\let\\pd=\\pointerdown\r\n\\let\\lf=\\leaderfill\r\n\r\n\\def\\spacer{\\hskip4.5pt}\r\n\\def\\fillright#1{\\hfil#1\\leaderfill}\r\n\\def\\fillleft#1{\\leaderfill#1\\hfil}\r\n% Changed spelling to \\centr, else conflicts with LaTeX \\center{} -CT\r\n\\def\\centr#1{\\leaderfill#1\\leaderfill}\r\n\\def\\link#1{\\multispan#1\\leaderfill}\r\n\r\n\\def\\arrowalign#1{\\vtop{\\baselineskip=0pt \\lineskiplimit=0pt \\lineskip=2pt\r\n \\halign{&##\\cr#1}}}\r\n\r\n%\\font\\elevenex=cmex10 scaled\\magstephalf % just for the arrow!\r\n %PS: this may not work on some installations, not sure why. CT\r\n %%PPS: (e.g., PCTeX, but it works find works fine with EmTeX)\r\n%----------------END Move Arrows\r\n\r\n\\def\\pijl{$\\rightarrow$\\ }\r\n\r\n% Special accents for Vata & Gbadi; Navajo coming soon, I hope...:\r\n%\\def\\bb#1{$\\mathrm{\\overline{#1}}$} Following looks better:\r\n\\def\\bb#1{\\ifmmode\\overline{\\mathrm{#1}}\\else$\\bar{\\mathrm{#1}}$\\fi}\r\n\r\n\\def\\boven#1#2{\\raisebox{-0.2pt}{$\\stackrel{#1}{\\mathrm{#2}}$}}\r\n\\def\\bovenop#1#2{\\raisebox{-0.06ex}[0ex][0ex]{$\\stackrel{#1}{\\mathrm{#2}}$}}\r\n\\def\\vl{\\rule{0.05em}{0.30em}}\r\n\\def\\|#1{\\ifmmode\\vert#1\\else\\bovenop{\\vl}{#1}\\fi}\r\n" }, { "path": "docs/guide/guide.tex", "content": "\n%%\n%% nb: use pdflatex to create pdf file with hyperlinks\n%%\n\n%% =====================================================================\n%% DOCUMENT DATA\n%% =====================================================================\n\n\\documentclass[11pt]{article}\n\n\\title{Specifying Grammars for OpenCCG: \\\\ A Rough Guide}\n\\author{Cem Boz\\c{s}ahin \\and Geert-Jan M. Kruijff \\and Michael White}\n\n\n%% =====================================================================\n%% PACKAGES\n%% =====================================================================\n\n\\usepackage{openccg} % for hlds/ccg\n\\usepackage{graphicx} % for figs\n\\usepackage{gb4e} % for examples\n\n\\usepackage[\n colorlinks=true, linkcolor=blue, citecolor=blue, urlcolor=blue,\n pdfstartview=FitH,\n pdftitle={Specifying Grammars for OpenCCG: A Rough Guide},\n pdfauthor={Cem Bozsahin, Geert-Jan M. Kruijff and Michael White}\n]{hyperref}\n\n\n%% =====================================================================\n%% NEW COMMANDS\n%% =====================================================================\n\n%\\newcommand{\\occg}{\\textsf{OpenCCG}}\n\\newcommand{\\occg}{OpenCCG}\n\\newcommand{\\tccg}{\\textsf{tccg}}\n\n \n%% =====================================================================\n%% DOCUMENT BODY\n%% =====================================================================\n\n\\begin{document}\n\n\\thispagestyle{empty}\n\\maketitle\n\\tableofcontents\n\\listoftables\n\\listoffigures\n\\newpage\n\n\\section{OpenCCG} \n\n\\occg\\ is an open source natural language processing library written in\nJava, which provides parsing and realization services based on Mark\nSteedman's Combinatory Categorial Grammar (CCG) formalism\n\\cite{Steedman:SynProc}. The library makes use of the multi-modal\nextensions to CCG devised by Jason Baldridge in his dissertation\n\\cite{Baldridge:2002} and in a joint EACL-03 paper with Geert-Jan\nKruijff \\cite{Baldridge/Kruijff:2003}. For a concise introduction to CCG\nwith these extensions, see \\cite{Steedman/Baldridge:2003}.\n\n\\occg\\ grew out of the Grok system developed by Gann Bierner and Jason\nBaldridge, and has been refined and extended by Michael White, with\nfurther contributions from Cem Boz\\c{s}ahin, G\\\"une\\c{s} Erkan,\nGeert-Jan Kruijff, David Reitter and Alexandros Triantafyllidis. Recent\ndevelopment efforts, managed by Michael White, have focused on making\nthe realizer\n\\cite{White/Baldridge:2003,White-RLAC:2004,White-INLG:2004,White-ACLSoft:2005}\npractical to use in dialogue systems, and improving (somewhat) the grammar\ndevelopment process.\n\nYou can download and install \\occg\\ from its website, located at\n\\url{http://openccg.sourceforge.net}. Once you've unpacked the archive,\nhave a look at the \\texttt{README} file for installation\ninstructions.\n\n\n\\section{About this (rough) guide}\n\nThis guide is intended to provide a brief introduction to writing grammars for\n\\occg. The system is implemented in Java, but you do not need to know Java.\n\\occg\\ provides its own formats for describing grammars, including the\ncombinatory rules, the lexicon (i.e.\\ the lexicalized grammar), feature\nstructures, LF, morphology etc. Two formats are available; one is based on XML\nand one is a higher-level format that looks similar to C or Java. The syntax of\nthe XML-based format is very simple, but at the same time it can be verbose and\nhard-to-read. The other format, the so-called ``CCG format'' (\\texttt{.ccg}),\nwas specifically designed to be written by hand, and has a richer and more\nconcise syntax. It is a ``front-end'' format in that it is converted internally\nto XML before it is actually used by \\occg, using the \\texttt{ccg2xml} tool. As\na result, the two formats share many conceptual similarities.\n\\textbf{NB:} Note that the XML format is more stable than the \\texttt{.ccg} format, and in particular, the way in which unification constraints are \nspecified in the \\texttt{.ccg} format is apt to change.\n\nThis manual was originally created before the CCG format existed. As a result,\nit is primarily geared towards writing grammars directly in XML. Over time,\nhowever, it will be updated to cover the use of the CCG format as well. For the\ntime being, see \\texttt{src/ccg2xml/README} for documentation of the\n\\texttt{.ccg} format.\n\n\n\\section{Using the XML-based format}\n\nIn order to write \\occg grammars directly in the XML-based format, you\nshould be familiar with XML. Actually, all you need to know is that tags\ncan be hierarchically and linearly organized, and that they must be\n``closed'' (by\n\\texttt{} or\n\\texttt{/>}) with proper nesting, e.g.\n\n\\begin{verbatim}\n \n Bond James \n Jimbo Double-Oh (Seven) \n \n\\end{verbatim}\n\n\\begin{verbatim} \n \n\\end{verbatim}\n\nAll the \\occg-defined elements and attributes are listed in the XML\nschema validation files. For reference documentation, you can have a\nlook at these files, which are located in the\n\\texttt{\\$OPENCCG\\_HOME/grammars/} directory of your installation. For\nexample, \\texttt{categories.xsd} describes the tags that go into \\occg\\\ncategories.\n\nFor more advanced use of \\occg, it helps to know\n\\href{http://www.w3.org/Style/XSL/}{XSLT}.\n\n\\subsection{XML-based grammar architecture in \\occg}\n\nA run-time grammar for \\occg\\ typically consists of \nfive primary files, with the following canonical names:\n\n\\begin{description}\n\n\\item[\\texttt{grammar.xml}] Specifies the name of the grammar, and lists\nthe names of the other files. This file may also specify XSLT\ntransformations to use in converting LFs to/from XML, and/or properties\nof a custom tokenizer (see \\texttt{grammar.xsd} for details).\n \n\\item[\\texttt{lexicon.xml}] Specifies \\emph{lexical families}. A lexical\nfamily specifies one or more related categories, with their associated\nfeature structures and logical forms. Lexical families are loosely based\non the notion of \\emph{tree families} in\n\\href{http://www.cis.upenn.edu/~xtag/}{XTAG}.\n \n\\item[\\texttt{morph.xml}] Specifies the \\emph{words} of the grammar.\nEach word is related to a lexical family through the part-of-speech tag\nof the word. If a family is a closed class, we specify explicitly with a\nfamily what words are its members.\n\n\\item[\\texttt{rules.xml}] Specifies which combinatory rules are\navailable to the grammar. For the purpose of this document, we assume\nthat application, type-raising, and composition (harmonic as well as\ncrossed) are available. Unary type changing rules are also placed into\n\\texttt{rules.xml}.\n\n\\item[\\texttt{types.xml} (optional)] Specifies the syntactic and semantic\ntype/sort hierarchies. Unlike HPSG, only atomic types are\nsupported in \\occg. Multiple-inheritance is allowed \\cite{erkanms03}.\n \n\\end{description}\n\nStandard practice is to store these files in a directory under the\n\\texttt{grammars} directory of the \\occg\\ distribution. Besides the\nabove files, it is also a good idea to have a \\texttt{testbed.xml} file.\nA testbed is a list of test expressions, where we specify for each\nexpression the number of parses ($\\geq 0$) the grammar should yield, and\noptionally the intended LF.\n\n\n\\section{Words and categories}\n\\label{sec:cats}\n\n\\subsection{Lexical families}\n\nTraditionally, the lexicon for a categorial grammar specifies for each\nword its own category. In \\occg, categories are instead organized into\nlexical \\textsl{families}, which are related to whole sets of words. (As\nmentioned earlier, the idea of families we employ here is loosely based\non the notion of \\emph{tree families} in XTAG.) This makes it possible\nto avoid giving the same specification over and over again in a lexicon.\n\nThe simplest way in which words can be related to families is through\ntheir parts of speech: for a word we have to specify its part of speech,\nand for a family we have to specify the part of speech a word has to\nhave for the family to be applicable. To control the applicability of a\nfamily, we can also declare it to be \\textsl{closed}. A closed \nfamily is not applicable to \\emph{every} word that has the appropriate\npart of speech, but only to those words (stems) that are listed with the\nfamily as its members. Note that a closed family does not exactly\ncorrespond to the notion of a closed class word, as open class words\n(especially verbs) are often listed as members of closed families, in\norder to assign them appropriate subcategorization frames.\n\nTo illustrate, let's look at some examples from the \\texttt{tiny} sample\ngrammar. A family is defined within the following element:\n\n\\begin{verbatim}\n \n \n :\n \n \n\n \n \n :\n \n \n \n \n \n \n \n\\end{verbatim}\n\n\\noindent These two families are for nouns and pronominal NPs, as their\n\\textsl{name} attributes indicate; they have parts of speech \\texttt{N}\nand \\texttt{Pro}, respectively, given by the \\textsl{pos} attribute.\nThe pronominal NP family has \\texttt{closed=\"true\"}, indicating that\nit's a closed family. The members are \\texttt{pro1} \\ldots\n\\texttt{pro3n}, where \\texttt{pro1} is an abstract stem for the first\nperson pronouns \\gf{I, we, me, us}, and so on.\n\nIn each family, we define one or more entries, using an \\textsl{entry}\nelement. Each entry defines a category with accompanying feature\nstructure and logical form. Each entry is given a name; we usually give \nthe main entry \\texttt{name=\"Primary\"}. An example of a family with\nmultiple (ok, two) entries appears below. The first entry is named\n\\texttt{DTV}, for ditransitive verb, as it specifies a category with two\nNP complements; the second entry is named \\texttt{NP-PPfor}, as it\nspecifies a category with an NP complement followed by a PP complement\nheaded by \\gf{for}. In both cases, the extra complement plays the role\nof Beneficiary in the semantics, motivating the grouping of these two\nentries into a single family.\n\n\\begin{small}\n\\begin{verbatim}\n \n \n :\n \n \n :\n \n \n \n \n\\end{verbatim}\n\\end{small}\n\n\\subsection{Categories}\n\nWithin an entry we define a category. A category can either be atomic \nor complex (i.e.\\ a function). The example below illustrates how we \nspecify an atomic category using the \\textsl{atomcat} element, \ngiving its label as a value of the attribute \\textsl{type}. \n\n\\begin{verbatim}\n \n \n \n :\n \n \n \n\\end{verbatim}\n\nWe can assign a feature structure to an atomic category using the\n\\textsl{fs} element. The \\textsl{fs} element has an \\textsl{id}\nattribute so that we can explicitly reference the feature structure,\nwhen needed.\n\n\\begin{verbatim}\n \n .. \n :\n \n\\end{verbatim}\n\nWe can add individual features using \\textsl{feat} elements. In their\nsimplest form, a feature has an \\textsl{attr} specifying the\nattribute and a \\textsl{val} giving the value of the attribute.\n\n\\begin{verbatim}\n \n \n \n\\end{verbatim}\n\nNow, since we don't want all nouns to be singular, we can instead\ndeclare the value of the \\texttt{num} feature to be a variable, as\nfollows:\n\n\\begin{verbatim}\n \n \n \n :\n \n :\n \n\\end{verbatim}\n\n\\noindent Here the \\textsl{featvar} element introduces a variable with\n\\texttt{name=\"NUM\"} as the value of the feature. Note that this feature\nspecification serves as an implicit declaration that nouns have a\n\\texttt{num} feature. As such, it interacts with the\n\\textsl{inheritsFrom} mechanism for default unification, as will be\nexplained below. With basic categories such as this one, it is a good\nidea to specify all relevant features.\n\nAn entry can also specify a \\emph{complex} category, i.e.\\ a function.\nFor that, we use the \\textsl{complexcat} element. This element is\nessentially a list, enumerating the result category and its arguments in\nthe order as given by a Steedman-style category. Argument categories may\nbe atomic or complex (i.e., creating a higher-order function); the\nresult category must be atomic (see \\cite{Baldridge:2002} for\ndiscussion).\n\n\\begin{table}\n\\begin{center}\n\\begin{tabular}{rcc}\nRules & \\occg & MMCCG \\\\ \\hline\napplication only & \\texttt{*} & $\\star$ \\\\\nassociative & \\verb+^+ & $\\diamond$ \\\\\npermutative & \\texttt{x} & $\\times$ \\\\\npermutative right & \\texttt{x>} & $\\times\\triangleright$\\\\\npermutative left & \\texttt{} & $\\triangleright$\\\\\nassociative permutative left & \\texttt{<} & $\\triangleleft$\\\\\nall rules & \\texttt{.} & $\\bullet$ \\\\ \\hline %[.2em]\n\\end{tabular}\n\\end{center}\n\\caption{Slash modes}\n\\label{slash-modes}\n\\end{table}\n\nFor each argument we give the slash using a \\textsl{slash} element that\nhas attributes \\textsl{mode} and \\textsl{dir} to specify what kind of\nslash we are dealing with. The available slash modes are given in\nTable~\\ref{slash-modes}. (Note that in XML, the angle brackets\n\\texttt{<} and \\texttt{>} must be escaped as \\texttt{\\<} and\n\\texttt{\\>}, respectively.) See \\cite{Baldridge:2002}[p.\\ 100] and\n\\cite{Baldridge/Kruijff:2003} for discusion of the slash modes in\nmultimodal CCG. Slashes may also have variables over modes, and may be\ninert, as discussed in \\cite{Baldridge:2002}[Ch.\\ 8].\n\n\\begin{figure}\n\\begin{quote}\n\\begin{verbatim}\n\n \n .. \n \n \n \n .. \n \n \n \n .. \n \n :\n\n\\end{verbatim}\n\\end{quote}\n\\[\n\\cf{s\\fsb{1}{}} \\bs \\cf{np\\fsb{2}{nom}} / \\cf{np\\fsb{3}{acc}} \n\\]\n\\caption{Transitive verb category}\n\\label{tv-cat}\n\\end{figure}\n\nFigure~\\ref{tv-cat} shows how the category for a transitive verb can be\ndefined; at the bottom of the figure is a more human-friendly notation\nfor the category. The result category is \\cf{s}. There are two argument\ncategories, an \\cf{np} with accusative case to the right, and an \\cf{np}\nwith nominative case to the left. In the human notation, the feature\nstructure id's are shown subscripted in angle brackets, followed by the\nfeatures themselves. Note that when the intended feature is evident from\nthe feature value, the feature name is left off; also, when the slash\nmode is consistent with the slash direction (e.g.\\ $\\triangleright$ and\n/), the mode is not shown, as in \\cite{Baldridge:2002}.\n\n\\subsection{Words}\n\\label{words}\n\nSince pronouns retain case marking in English, the case requirements on\nthe arguments of a transitive verb have the effect of determining which\npronouns can appear in which positions. For example, the first person\npronoun \\gf{I} is allowed in subject position, while \\gf{me} is\nallowed in object position, but not vice-versa. \n\nThis naturally leads us to how we specify properties of words in the\n\\texttt{morph.xml} file. For each word, we have to give its wordform \nand part of speech, as follows:\n\n\\begin{verbatim}\n \n\\end{verbatim}\n\n\\noindent If the word's stem differs from its form, the stem must be\nlisted too:\n\n\\begin{verbatim}\n \n\\end{verbatim}\n\n\\begin{figure}\n%\\begin{quote}\n\\begin{small}\n\\begin{verbatim}\n\n\n\n\n:\n \n \n\n\n \n\n\\end{verbatim}\n\\end{small}\n%\\end{quote}\n\\[\n\\begin{array}{rcl}\n\\gf{I} & \\vdash & \\cf{np\\fsb{2}{1st,sg,nom}} \\\\ \n\\gf{me} & \\vdash & \\cf{np\\fsb{2}{1st,sg,acc}} \\\\\n\\gf{we} & \\vdash & \\cf{np\\fsb{2}{1st,pl,nom}} \\\\ \n\\gf{us} & \\vdash & \\cf{np\\fsb{2}{1st,pl,acc}} \\\\\n\\end{array}\n\\]\n\\caption{Case macros}\n\\label{case-macros}\n\\end{figure}\n\nTo add further information, such as case, we use \\textsl{macros}, as\nillustrated in Figure~\\ref{case-macros}. In the figure, the entries for\nthe first person pronouns are given, along with their syntactic macros,\nspecified by the \\textsl{macros} attribute. The case macros, named\n\\texttt{@nom} and \\texttt{@acc}, appear next in the figure, defined by\nthe \\textsl{macro} elements. These macros set the case feature on the\ncategory associated with the word (via its part of speech), by accessing\nthe feature structure with id 2 and setting the value of the\n\\texttt{case} feature to \\texttt{nom} and \\texttt{acc}, respectively.\n(The number macros \\texttt{@sg} and \\texttt{@pl} are analogous.) \nThe effects of the macros are shown at the bottom of the figure, where\nthe word forms for the first person pronouns are paired with their\nassociated categories, which differ in their number and case values.\n\n\\begin{figure}\n%\\begin{quote}\n\\begin{small}\n\\begin{verbatim}\n\n\n\n\n:\n \n \n \n \n \n\n\\end{verbatim}\n\\end{small}\n%\\end{quote}\n\\[\n\\begin{array}{rcl}\n\\gf{buy} & \\vdash & \n\\cf{s\\fsb{1}{}} \\bs \\cf{np\\fsb{2}{non\\mbox{-}3rd,sg,nom}} / \\cf{np\\fsb{3}{acc}} \\\\ \n\\gf{buys} & \\vdash & \n\\cf{s\\fsb{1}{}} \\bs \\cf{np\\fsb{2}{3rd,sg,nom}} / \\cf{np\\fsb{3}{acc}} \\\\\n\\gf{buy} & \\vdash & \n\\cf{s\\fsb{1}{}} \\bs \\cf{np\\fsb{2}{pl,nom}} / \\cf{np\\fsb{3}{acc}} \\\\ \n\\gf{bought} & \\vdash & \n\\cf{s\\fsb{1}{}} \\bs \\cf{np\\fsb{2}{nom}} / \\cf{np\\fsb{3}{acc}} \\\\ \n\\end{array} \n\\]\n\\caption{Person macros}\n\\label{pers-macros}\n\\end{figure}\n\nAs another example, Figure~\\ref{pers-macros} shows how the person macros\nare used (together with the number macros) in setting up person and\nnumber agreement constraints with various forms of the verb \\gf{buy}.\nNote that the tense macros \\texttt{@pres} and \\texttt{@past} do not\ncontribute syntactic features; instead they contribute semantic features\nto the logical form (cf.\\ Section~\\ref{lfs}). Additionally, note that\nthe macro \\texttt{@non-3rd} supplies a syntactic person value that is\ncompatible with both \\texttt{1st} and \\texttt{2nd}, as specified in\n\\texttt{types.xml} (cf.\\ Section~\\ref{types}).\n\nIt is important to note that macro instantiation does not involve\nunification: macros set feature values regardless of any value that\nmight already be present for the feature in the feature structure.\nConceivably, it would be convenient on occasion (though computationally \nmore expensive) to use unification, rather than overwriting, during\nmacro instantiation, but there is no support for doing so at present.\n\n\\subsection{Unification}\n\n\\begin{figure}\n\\begin{quote}\n%\\begin{small}\n\\begin{verbatim}\n\n \n .. \n \n \n \n .. \n \n\n\\end{verbatim}\n%\\end{small}\n\\end{quote}\n\\[\n\\gf{the} ~ \\vdash ~ \\cf{np\\fsb{2}{3rd}}/_{\\!\\!\\diamond}\\cf{n\\fsb{2}{}}\n\\]\n\\caption{The definite article}\n\\label{def-art}\n\\end{figure}\n\n\\begin{figure}\n\\begin{center}\n\\deriv{3}{\n\\gf{the} & \\gf{teacher} & \\gf{buys} \\\\\n\\uline{1} & \\uline{1} & \\uline{1} \\\\\n\\cf{np\\fsb{2}{3rd}}/_{\\!\\!\\diamond}\\cf{n\\fsb{2}{}} &\n\\cf{n\\fb{sg}} & \n\\cf{s} \\bs \\cf{np\\fb{3rd,sg,nom}} / \\cf{np\\fb{acc}} \\\\\n\\fapply{2} \\\\\n\\cmc{2}{\\cf{np\\fsb{2}{3rd,sg}}} \\\\\n\\ftype{2} \\\\\n\\cmc{2}{\\cf{s\\fsb{1}{}} / \\cf{s\\fsb{1}{}} \\bs \\cf{np\\fsb{2}{3rd,sg}}} \\\\\n\\fcomp{3} \\\\\n\\cmc{3}{\\cf{s\\fsb{1}{}} / \\cf{np\\fb{acc}}}\n}\n\n\\vspace{1cm}\n\\deriv{3}{\n\\gf{the} & \\gf{teachers} & \\gf{*buys} \\\\\n\\uline{1} & \\uline{1} & \\uline{1} \\\\\n\\cf{np\\fsb{2}{3rd}}/_{\\!\\!\\diamond}\\cf{n\\fsb{2}{}} &\n\\cf{n\\fb{pl}} & \n\\cf{s} \\bs \\cf{np\\fb{3rd,sg,nom}} / \\cf{np\\fb{acc}} \\\\\n\\fapply{2} \\\\\n\\cmc{2}{\\cf{np\\fsb{2}{3rd,pl}}} \\\\\n\\ftype{2} \\\\\n\\cmc{2}{\\cf{s\\fsb{1}{}} / \\cf{s\\fsb{1}{}} \\bs \\cf{np\\fsb{2}{3rd,pl}}} \\\\\n\\badcomb{3}{{>}\\mathbf{B}} \\\\\n\\cmc{3}{\\cf{s\\fsb{1}{}} / \\cf{np\\fb{acc}}}\n}\n\\end{center}\n\\caption{Unification and subject-verb agreement}\n\\label{subj-v-agr}\n\\end{figure}\n\nSpeaking of unification, let's examine the role it plays in enforcing\nsubject-verb agreement. The category for the definite article is given\nin Figure~\\ref{def-art}. The definite article is compatible with both\nsingular and plural nouns, but it must retain this number information\nfor subject-verb agreement to work. Propagating number information is\naccomplished here by setting the feature structure id to be same on both\nthe \\cf{np} result category and on the argument category \\cf{n} (i.e.,\nwe have \\texttt{id=\"2\"} in both cases). Figure~\\ref{subj-v-agr} provides\nan illustration, showing how \\gf{the teacher} ends up as a singular\n\\cf{np}, while \\gf{the teachers} ends up as a plural one.\\footnote{Note\nthat feature structure id's are only shown when relevant to unification.\nAlso, in \\occg\\ derivations, the id's are actually mapped to ``fresh''\nones after lexical lookup, to avoid any accidental coindexations across\ndifferent lexical items.} Type raising the \\cf{np} and forward composing\nit with \\gf{buys} requires it to unify with the backwards \\cf{np}\nargument of the verb, and in particular, requires the number feature to\nbe singular. Since this is only the case with \\gf{the teacher}, the last\nstep in the derivation of \\gf{the teachers *buys} will be blocked by a\nunification failure.\n\n\\begin{figure}\n%\\begin{quote}\n\\begin{small}\n\\begin{verbatim}\n\n \n \n \n \n \n \n \n\n\\end{verbatim}\n\\end{small}\n%\\end{quote}\n\\[\n\\gf{for} ~ \\vdash ~ \\cf{pp\\fb{for,acc,X}} \\, /_{\\!\\!\\triangleleft} \\, \\cf{n\\fb{acc,X}}\n\\]\n\\caption{Default unification with case marking prepositions}\n\\label{prep-nom}\n\\end{figure}\n\nCoindexing two feature structures ensures that all their features will\ntake on the same values. There are times, however, when we want two\nfeature structures to just mostly take on the same values, except for\none or two particular features. To support such cases, \\occg\\ includes a\nlimited form of default unification, specified by the\n\\textsl{inheritsFrom} attribute of a feature structure element.\n\nThe \\textsl{inheritsFrom} mechanism is implemented by compiling out the\ndefault unification of two feature structures into individual feature\nequations at the time of lexical lookup. This works as follows. First,\nany features appearing on the target category, but not the result\ncategory, are copied over. Then, for every feature that has been\nobserved in the grammar for the result category---except for any\nfeatures that already appear there---a feature equation is added, i.e.\\\nthe feature is set to the same variable on both the target and result\ncategories.\n\nAs an example, Figure~\\ref{prep-nom} shows the category for ``case\nmarking'' prepositions, i.e.\\ those prepositions which are assumed to\nplay a purely syntactic role. The \\cf{pp} result category has a\n\\texttt{lex} feature which is instantiated by the stem of the actual\nlexical item, as specified by the keyword \\texttt{\"[*DEFAULT*]\"}. Its\nremaining features are ``inherited from'' the feature structure with id\n3, i.e.\\ the one for the argument \\cf{np}, as specified by\n\\texttt{inheritsFrom=\"3\"}. When the category for a case-marking\npreposition (such as \\gf{for} here) is instantiated, a feature equation\nis established between the \\cf{pp} and \\cf{np} categories for the\n\\texttt{index} feature (whose purpose will be discussed in the next\nsection); the value of the \\texttt{case} feature is also copied over.\nThus, in the figure, both the \\cf{pp} result category and the \\cf{np}\nargument category have an index variable \\texttt{X} (and accusative\ncase), while only the result PP has a \\texttt{lex} feature, with\n\\texttt{for} as its value.\\footnote{Since the case feature is arguably\nsuperfluous on the result PP, one could avoid it by just including an\nexplicit feature equation for the \\texttt{index} variable. Generally\nthough, it's simpler and less error-prone to use the\n\\textsl{inheritsFrom} mechanism than to manually include all the\nrelevant feature equations.}\n\n\\subsection{Set args}\n\n\\begin{figure}\n\\begin{quote}\n\\begin{verbatim}\n\n .. \n \n \n \n .. \n \n \n \n .. \n \n \n :\n\n\\end{verbatim}\n\\end{quote}\n\\[\n\\gf{nanghuhuli (``catches'')} ~ \\vdash ~ \n\\cf{s} \\{ / \\cf{np\\fb{nom}}, /\\!_{\\times} \\cf{np\\fb{gen}} \\} \n\\]\n\\caption{Tagalog transitive verb with set args}\n\\label{tagalog-set-args}\n\\end{figure}\n\nTo conclude our discussion of specifying syntactic categories, we should\nmention the availability of set args and dollar variables in \\occg. Set\nargs enable us to define categories that allow arguments to appear in\nany order, as illustrated in Figure~\\ref{tagalog-set-args} for the\nTagalog verb \\gf{nanghuhuli (``catches'')}. In the figure, both the\nnominative and genitive arguments must appear to the right of the verb,\nbut their relative order is unconstrained. Note that the nominative\nargument is given the more powerful associative and permutative slash,\nwhile the genitive argument is given the permutative-only slash; see\n\\cite{Baldridge:2002}[Ch.\\ 7] for discussion, and\n\\cite{bozsahinsteedman03} for further examples.\n\n\\subsection{Dollar variables}\n\n\\begin{figure}\n\\begin{quote}\n\\begin{verbatim}\n\n .. \n \n \n \n \n .. \n \n \n \n \n \n .. \n \n \n \n\n\\end{verbatim}\n\\end{quote}\n\\[\n\\gf{and} ~ \\vdash ~ \\cf{s}\\$_{1} \\bs_{\\star} \\cf{s}\\$_{1} /\\!_{\\star} \\cf{s}\\$_{1}\n\\]\n\\caption{Category with dollar variables for sentential coordination}\n\\label{sent-coord-dollar}\n\\end{figure}\n\nDollar variables range over a stack of arguments, and can be useful in\ndefining categories for conjunctions, type-raised categories for\nquantifiers, and categories for unary rules (cf.\\\n\\cite{Baldridge:2002}[Ch.\\ 8]). Figure~\\ref{sent-coord-dollar} shows the\ncategory for \\gf{and} that allows a range of clausal categories to be\ncoordinated---e.g., transitive verbs, verb phrases, or subject-verb\nconstituents, in right-node raising---as discussed in\n\\cite{White-RLAC:2004}.\n\n\n\\section{Logical forms}\n\\label{lfs}\n\n\\subsection{Hybrid logic dependency semantics}\n\nTo associate meanings with categories, we need to take care of two\nthings: the structure of the meaning (logical form) itself, and the\nrelation between the category and that meaning. Usually the latter comes\ndown to specifying how the meanings of arguments are to be fit into the\nlogical form.\n\nAs logical forms we use \\emph{hybrid logical terms} that specify\nsemantic dependency structures; for details, see\n\\cite{Kruijff:2001,Baldridge/Kruijff:2002,White/Baldridge:2003,White-RLAC:2004}.\n(If you're wondering where the $\\lambda$'s have gone, see\nSection~\\ref{tko-lambdas}.)\n\nWe give the logical form of a category using the \\textsl{lf} element:\n\n\\begin{verbatim}\n \n :\n .. \n \n\\end{verbatim}\n\n\\noindent The \\textsl{lf} element must always appear at the end of a\ncategory specification (whether atomic or complex).\n\nThe simplest logical form is of the form $@_{X} \\phi$, with $\\phi$ a\nproposition. We interpret $X$ as the \\emph{discourse referent} of the\nproposition. For the proposition itself, we can follow linguistic\ntradition and use a word's stem to represent its meaning, except that\nwe'll use boldface rather than prime notation (i.e., we'll represent the\nmeaning of \\gf{word} as \\C{word} rather than $\\mathit{word}'$). We\nachieve exactly this effect using the keyword \\texttt{\"[*DEFAULT*]\"}, as\nshown below:\n\n\\begin{verbatim}\n \n \n \n \n \n\\end{verbatim}\n\n\\noindent The \\textsl{satop} element introduces a satisfaction operator\n@, along with a nominal variable, or \\textsl{nomvar}, $X$. (In the\ngrammar, we use logic variables rather than concrete instantiations for\nnominals; during parsing or realization, \\occg\\ instantiates these\nvariables dynamically.) Nominals can have types (or sorts) associated\nwith them, as will be explained further in Section~\\ref{types}; here,\n$X$ is allowed to be any subtype of semantic object. The \\textsl{prop}\nelement specifies the proposition.\n\n\\begin{figure}\n\\begin{small}\n\\begin{verbatim}\n[lexicon.xml]\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n[morph.xml]\n \n :\n \n \n \n \n \n \n \n\\end{verbatim}\n\\end{small}\n\\[\n\\gf{flower} ~ \\vdash ~ \\cf{n\\fsb{2}{sg,X\\!:\\con{thing}}} ~ : ~ \n@_{X\\!:\\con{thing}}(\\C{flower} \\wedge \\modp{num}\\con{sg})\n\\]\n\\caption{Noun with logical form}\n\\label{noun-lf}\n\\end{figure}\n\n\\subsection{The syntax-semantics interface}\n\nTo establish the interface between syntactic structure, as defined by\nthe category, and the logical form, we co-index categories with nominals\nin the logical form, by adding an attribute \\texttt{index} to the\nfeature structure of each category and giving that attribute the\ncorresponding nominal as value. To illustrate, Figure~\\ref{noun-lf}\n(top) shows the complete category definition for nouns. Note how the\nfeature structure includes a feature named \\texttt{index}, whose value\nis a logical form just consisting of the nominal variable $X$---that is,\na variable with the same name as the one introduced in the satisfaction\noperator of the LF further down.\\footnote{It suffices to put all the\nsemantic type restrictions on the nominals in the LF; i.e., it's not\nnecessary to put them on the \\texttt{index} variables as well.}\n\nIn the middle of Figure~\\ref{noun-lf}, the entry for the noun\n\\gf{flower} in \\texttt{morph.xml} is shown. This entry includes\n\\texttt{thing} as the semantic class (i.e.\\ its semantic type), which is\nunified with the type of the nominal head $X$ during lexical\ninstantiation. The entry also includes the macros \\texttt{@sg} and\n\\texttt{@sg-X}, where the former adds the syntactic feature of singular\nnumber, while the latter adds the semantic feature of singular number on\nthe nominal $X$ (as shown below the entry). At the bottom of the figure\nis the complete category that results from lexical lookup and\ninstantiation of \\gf{flower}.\n\nNote that it is possible to fill in the semantic head's proposition with\nsomething other than the stem. To do so, we can specify a \\textsl{pred}\nto use in place of the stem, when listing a stem as a member of a\nfamily:\n\n\\begin{verbatim}\n \n \n :\n \n :\n \n :\n \n\\end{verbatim}\n\n\\noindent With this specification, the proposition \\C{catch} will\nappear in the logical form for the Tagalog verb \\gf{nanghuhuli}:\n\n\\begin{exe}\n\\ex \\label{tg-catch}\n\\(\n\\begin{array}{rcl}\n\\gf{nanghuhuli} & \\vdash & \n\\cf{s\\fb{E}} \\{ / \\cf{np\\fb{nom,X}}, /\\!_{\\times} \\cf{np\\fb{gen,Y}} \\} \n~ : ~ \\\\\n&& @_{E}(\\C{catch} \\wedge \\modp{Actor}X \\wedge \\modp{Patient}Y) \\\\\n\\end{array} \n\\)\n\\end{exe}\n\n\\begin{figure}\n\\begin{small}\n\\begin{verbatim}\n \n \n \n \n \n \n \n \n ..\n \n \n \n \n \n ..\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\\end{verbatim}\n\\end{small}\n\\[\n\\begin{array}{rcl}\n\\gf{bought} & \\vdash & \n\\cf{s\\fsb{1}{E:\\con{action}}} \n \\bs \\cf{np\\fsb{2}{nom,X\\!:\\con{animate\\mbox{-}being}}} \n / \\cf{np\\fsb{3}{acc,Y\\!:\\con{sem\\mbox{-}obj}}} ~ : \\\\\n&& @_{E:\\con{action}}(\\C{buy} \\wedge \\modp{tense}\\con{past}) \\wedge \\\\ \n&& @_{E:\\con{action}}(\\modp{Actor}X\\!\\!:\\!\\con{animate\\mbox{-}being}) \\wedge \\\\\n&& @_{E:\\con{action}}(\\modp{Patient}Y\\!\\!:\\!\\con{sem\\mbox{-}obj}) \\\\\n\\end{array}\n\\]\n\\caption{Transitive verb with logical form}\n\\label{tv-lf}\n\\end{figure}\n\n\\subsection{Dependency relations}\n\nTo introduce dependency relations---like those we just saw for the\nTagalog verb \\gf{nanghuhuli}---we use the \\textsl{diamond} element. For\nexample, to state that an English transitive verb has a logical form\nwith an \\modp{Actor} and a \\modp{Patient}, we can specify the category\nshown in Figure~\\ref{tv-lf}. At the bottom of the figure, this category\nappears instantiated for the verb \\gf{bought}. Note that for ease of\ndisplay, the logical form has been partially flattened, with each\ndependency relation appearing on a separate line. (In \\occg, logical\nforms are automatically flattened prior to parsing or realization.)\n\n\\begin{figure}\n\\begin{small}\n\\begin{verbatim}\n \n \n \n \n ..\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\\end{verbatim}\n\\end{small}\n\\[\n\\gf{the} ~ \\vdash ~ \n\\cf{np\\fsb{2}{3rd,X\\!:\\con{sem\\mbox{-}obj}}}/_{\\!\\!\\diamond}\\cf{n\\fsb{2}{X\\!:\\con{sem\\mbox{-}obj}}} ~ : ~\n@_{X\\!:\\con{sem\\mbox{-}obj}}(\\modp{det}\\con{the})\n\\]\n\\caption{Determiner with logical form}\n\\label{det-lf}\n\\end{figure}\n\n\\subsection{Function words}\n\nFinally, Figure~\\ref{det-lf} illustrates how we can specify the meaning\nof function words, i.e.\\ words that have no independent meaning. The\nexample gives a specification for determiners, which add a semantic\nfeature $\\modp{det}$ to the meaning of the nominal head they modify. The\nnoun itself provides the meaning through co-indexation with $X$, the\nroot nominal of the logical form for the determiner. At the bottom of\nthe figure is the category instantiated for the definite article\n\\gf{the}.\n\nTo support the realization of function words, the semantic relation or\nfeature introduced by the word must be declared using the\n\\textsl{indexRel} attribute on the \\textsl{family} element.\\footnote{In\nprinciple, it would be possible for \\occg\\ to figure out when a semantic\nrelation or feature should be used for indexing purposes, but the\npossibility of adding further semantic content via macros makes it\nnon-trivial to do so.} For example, in Figure~\\ref{det-lf} we have\n\\texttt{indexRel=\"det\"}, which indicates that the $\\modp{det}$ feature\nshould be used to trigger the lookup of the appropriate determiner.\n(Normally, semantic relations or features are introduced as part of the\nmeaning of content words.) When function words are semantically\nnull---e.g., with case-marking prepositions in the \\texttt{tiny}\ngrammar---the keyword \\texttt{*NoSem*} should be given as the value of\nthe \\textsl{indexRel} attribute.\n\n\\subsection{Relation sorting}\n\nIt is possible to specify the order in which to display relations\nappearing at the same level in the logical forms. By default, relations\nare sorted alphabetically, with a few exceptions, e.g.\\ that\n$\\modp{Restr}$ should appear before $\\modp{Body}$. You can customize the\norder in which relations appear using the optional\n\\textsl{relation-sorting} element in \\texttt{lexicon.xml}. See the\n\\texttt{lexicon.xsd} schema for details.\n \n\\subsection{From CCG to \\occg: Taking care of lambdas}\n\\label{tko-lambdas}\n\nIn CCG, logical forms are normally given using terms from the lambda\ncalculus, e.g.\n\n\\begin{exe}\n\\ex \\label{buy-lambdas}\n\\(\n\\gf{buy} ~ \\vdash ~ \n(\\cf{s} \\bs \\cf{np\\fb{nom}}) / \\cf{np\\fb{acc}} ~ : ~\n\\lambda x_2 x_1 . \\mathrm{buy}^\\prime x_2 x_1\n\\)\n\\end{exe}\n\n\\begin{exe}\n\\ex \\label{tg-catch-lambdas}\n\\(\n\\gf{nanghuhuli} ~ \\vdash ~ \n\\cf{s} \\{ / \\cf{np\\fb{nom}}, /\\!_{\\times} \\cf{np\\fb{gen}} \\} ~ : ~ \n\\lambda \\{ x_1, x_2 \\} . \\mathrm{catch}^\\prime x_2 x_1\n\\)\n\\end{exe}\n\n\\noindent In (\\ref{buy-lambdas}), $x_2$ corresponds to the outermost\nargument, $/ \\cf{np\\fb{acc}}$, and $x_1$ to the innermost one, $\\bs\n\\cf{np\\fb{nom}}$. Example (\\ref{tg-catch-lambdas}) uses set-lambda\nnotation with the following convention (cf. \\cite{bozsahinsteedman03}):\nthe lambda operator binds a set of variables which are paired with the\nset of arguments in left-to-right order. Thus, in the second example\nabove, $x_1$ corresponds to $/ \\cf{np\\fb{nom}}$, and $x_2$ to\n$/\\!_{\\times} \\cf{np\\fb{gen}}$.\n\nThe interpretation of the CCG $\\lambda$-terms above is as in\n(\\ref{pas}), where the argument $x_i$ c-commands $x_{i+j}$ for\n$j=1,2,\\cdots n-i$, at the level of predicate-argument structure\n(c-command is called LF-command in CCG for that reason). This is how\n(and where) CCG defines binding constraints.\n\n\\begin{exe}\n\\ex \\label{pas}\n\\begin{minipage}{0.7\\textwidth}\n\\includegraphics{pas.pdf}\n\\end{minipage}\n\\end{exe}\n\nAs we have seen earlier in this section, \\occg\\ uses hybrid logic\ndependency semantics (HLDS) terms, rather than $\\lambda$-terms, in its\nlogical forms. For example, the HLDS terms for (\\ref{buy-lambdas}) and\n(\\ref{tg-catch-lambdas}) appeared in Figure~\\ref{tv-lf} and in\n(\\ref{tg-catch}). These terms differ from their $\\lambda$-counterparts\nin a couple of ways. First, in semantic construction, argument binding\nis accomplished through unification, rather than via function\napplication. And second, predicates are typically connected to their\narguments via semantic roles, such as $\\modp{Agent}$ and\n$\\modp{Patient}$---though nothing prevents relations such as\n$\\modp{Arg1}$ and $\\modp{Arg2}$ from being used instead. Using semantic\nroles can be more convenient for applications, and makes it possible to\ncapture semantic similarities across argument structure alternations.\nThe downside is that it makes it impossible to enforce binding\nconstraints. In principle, relations encoding semantic roles (e.g.\\\n$\\modp{Agent}$ and $\\modp{Patient}$) could be combined with ones for\nargument structure (e.g.\\ $\\modp{Arg1} \\ldots \\modp{ArgN}$) in the same\nHLDS logical form, though this has not yet been tried.\n\n\n\\section{Types}\n\\label{types}\n\nTypes (aka sorts) allow for some abstraction, generalization and\nspecialization in an \\occg\\ grammar. Unlike HPSG, \\occg\\ only employs\natomic types. These types may be used as restrictions on syntactic or\nsemantic feature variables, or given as values of syntactic or semantic\nfeatures. Multiple-inheritance is allowed (see \\cite{erkanms03} for\nfurther information).\n\nTypes are kept in the types file, usually named \\texttt{types.xml}. This\nfile is optional, which means that features can be untyped (actually,\nall features will be considered to be of type \\texttt{top} in this case,\nwhich is the only predefined type). \n\n\\begin{figure}\n\\begin{verbatim}\n \n \n \n \n \n \n\\end{verbatim}\n\\caption{Hierarchy of syntactic person values}\n\\label{pers-vals}\n\\end{figure}\n\nIn Section~\\ref{words}, Figure~\\ref{pers-macros}, we saw how the value\nof the person feature \\texttt{non-3rd} could be used to define a\ncategory compatible with both \\texttt{1st} and \\texttt{2nd} person\nsingular subjects. This definition relies on the following specification\nof person values in the \\texttt{tiny} grammar's \\texttt{types.xml} file,\nlisted in Figure~\\ref{pers-vals}. As this example shows, types are\ndefined using a \\textsl{type} element and are required to have a\n\\textsl{name}. They may also have a space-separated list of one or more\n\\textsl{parent} types. Indenting may be used to show the primary\ntype-subtype hierarchy:\\footnote{Since multiple parents are allowed,\nnesting of elements is not used to define type-subtype relationships.}\nhere, \\texttt{1st} and \\texttt{2nd} are subtypes of \\texttt{non-3rd},\nwhile \\texttt{non-3rd} and \\texttt{3rd} are subtypes of\n\\texttt{pers-vals}. If no parent types are listed---as with\n\\texttt{pers-vals}---the type is implicitly a subtype of \\texttt{top}.\n\n\\begin{figure}\n\\begin{verbatim}\n \n \n \n \n \n \n \n \n \n \n\\end{verbatim}\n\\caption{Hierarchy of semantic types/sorts}\n\\label{ont-sorts}\n\\end{figure}\n\nThe \\occg\\ type system does not distinguish between syntactic and\nsemantic types; it is up to the grammar designer to ensure their\nsystematic use. For example, if there is to be a syntactic hierarchy and\na semantic hierarchy of types, it is a good idea to define a `top\nobject' for each (or at least for one of them), e.g.\\ \\texttt{sem-obj}\nas the root of the semantic type hierarchy, as illustrated in\nFigure~\\ref{ont-sorts}. The types in this figure are the ones assumed by\nthe transitive verb category given in Figure~\\ref{tv-lf}. With these\ntypes, \\occg\\ can parse and realize \\gf{he bought a flower}, but not\n\\gf{*a flower bought he}, since \\gf{flower} has type \\texttt{thing}, and\n\\texttt{thing} is not compatible with the type \\texttt{animate-being},\nas is required for the \\modp{Actor} of a \\C{buy} action.\n\n\n\\section{Rules}\n\\label{rules}\n\nThe rules file, typically named \\texttt{rules.xml}, specifies the\ncombinatory rules for a grammar. The rule specifications for the\n\\texttt{tiny} grammar appear below:\n\n\\begin{verbatim}\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n \n \n \n\\end{verbatim}\n\n\\noindent In addition to the rules shown here, it is also possible to\nhave substitution rules, as well as additional type raising rules. By\ndefault, the argument and result categories for a type raising rule are\n\\cf{np} and \\cf{s}, respectively. To create a type raising rule using\ndifferent categories, you can use an \\textsl{arg} and/or \\textsl{result}\nelement to specify the desired atomic category. For example, a backward\ntype raising rule for prepositional phrases is included as the last rule\nabove.\n\nTheoretically speaking, CCG combinatory rules are universal; \\emph{any}\nlexicalized grammar has access to them if it uses in the lexical\ncategories the modalities licensed by the rules (see\n\\cite{Steedman/Baldridge:2003} for further information). The rules file\ncan also incorporate rules that are language specific. To illustrate,\nlet's consider the case of pro drop in Turkish. Turkish is a pro-drop\nlanguage, which means that subjects of finite clauses can be dropped\nbecause morphology of the finite verb already indicates the subject:\n\n\\begin{itemize}\n\\item[(a)]\n\\begin{tabular}{ll}\nBen & uyu-du-m/*-n \\\\\nI & sleep-PAST-1SG/*-2SG \\\\\n\\multicolumn{2}{l}{`I slept.'} \\\\\n\\end{tabular}\n\n\\item[(b)]\n\\begin{tabular}{l}\nUyu-du-m \\\\\nsleep-PAST-1SG \\\\\n`(I) slept.' \\\\\n\\end{tabular}\n\\end{itemize}\n\nPro drop can be modelled in different ways. For example, one can write a\nlexical rule to generate the derived lexical entries of finite verbs\n\\emph{in} the lexicon, so that every finite verb has two lexical\nentries, one derived from the other---e.g.\\ (\\ref{ex:prodrop}) below,\nwhich asserts $\\cf{s\\fb{fin}}\\bs\\cf{np\\fb{acc}}$ and $\\cf{s\\fb{fin}}$\nentries in the lexicon from\n$\\cf{s\\fb{fin}}\\bs\\cf{np\\fb{nom}}\\bs\\cf{np\\fb{acc}}$ and\n$\\cf{s\\fb{fin}}\\bs\\cf{np\\fb{nom}}$, etc.\n\n\\begin{equation}\n\\label{ex:prodrop}\n\\cf{s\\fb{fin}}\\bs\\cf{np\\fb{nom}}\\$_1\n\\Rightarrow \n\\cf{s\\fb{fin}}\\$_1\n\\end{equation}\n\nThis strategy has theoretical and practical implications. Theoretically,\nit assumes that \\emph{all} morphology is confined to the lexicon,\nincluding inflectional morphology, which is usually regarded as part of\nsyntax (see e.g.\\ \\cite{bozsahin02cl} for its implications for the\ntransparency of syntax-semantics correspondence). \\occg\\ does not assume\nthat there are only words in the lexicon; anything that can bear a\ncategory (words, affixes, clitics) can be a lexical\nitem.\\footnote{Currently, \\occg\\ does not have any mechanism to enforce\nthe \\emph{Lexical Integrity Principle} of \\cite{bresnanmchombo95}, which\nbasically states that words are islands as far as syntax is concerned,\ne.g.\\ it is not possible to extract out of a word. \\cite{bozsahin02cl}\nproposes that different attachment characteristics of words and bound\nmorphemes can be factored into CCG's lexical entries and combinatory\nrules (the latter simply projects them onto surface grammar), in effect\nrendering LIP as a phonological principle, but this is an open problem\nfor now.}\n\nOn the practical side, it assumes that all inflected forms of the verb\nare listed in the lexicon. For morphologically rich languages such as\nTurkish, this amounts to around 2$^{8}$ entries per verb because Turkish\nhas 8 inflections in the verb paradigm, all of which are optional.\n\n\\begin{figure}\n\\begin{verbatim}\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\\end{verbatim}\n\\caption{Unary rule for pro drop}\n\\label{pro-drop-rule}\n\\end{figure}\n\nAn alternative is to add a unary rule for pro drop to\n\\texttt{rules.xml}. This rule will apply ``on the fly'', that is, it can\napply to lexical or combinatorially-derived inflected verb forms. The\nrule, which implements (\\ref{ex:prodrop}), may be specified as shown in\nFigure~\\ref{pro-drop-rule} (we omit the \\$ variable for simplicity).\nUnary rules are defined using a \\textsl{typechanging} element, since\nsuch rules must change the type of the argument category---otherwise,\nnothing would prevent the rule from applying again and again to its own\noutput.\n\n\n\\section{Trying it out}\n\nOnce you've configured and built \\occg\\, per the \\texttt{README} file,\nyou're ready to try out the grammar testing tools. You can experiment\nwith the grammars described in \\texttt{SAMPLE\\_GRAMMARS} or make one of\nyour own. In the latter case, it will be easier if you create your\ngrammar in its own subdirectory of the \\texttt{grammars} directory.\n\nThere are (at present) three command line tools for trying grammars out:\n\\texttt{tccg}, \\texttt{ccg-test} and \\texttt{ccg-realize}.\n\n\\subsection{\\texttt{tccg}}\n\nThe \\texttt{tccg} tool (for ``text CCG'') is for interactively testing a\ngrammar. Its (primary) usage is\n\n\\begin{verbatim}\n tccg ()\n\\end{verbatim}\n\n\\noindent The default grammar file name is \\texttt{grammar.xml}. You can\ntry it out by going to the \\texttt{grammars/tiny} directory and running\n\\texttt{tccg}, like so:\\footnote{Examples like this one may have \nan occasional extra line break to improve readability.}\n\n\\begin{small}\n\\begin{verbatim}\nD:\\Mike\\dev\\openccg\\grammars\\tiny>tccg\nLoading grammar from URL: file:/D:/Mike/dev/openccg/grammars\n/tiny/grammar.xml\nGrammar 'tiny' loaded.\n\nEnter strings to parse.\nType ':r' to realize selected reading of previous parse.\nType ':h' for help on display options and ':q' to quit.\nYou can use the tab key for command completion,\nCtrl-P (prev) and Ctrl-N (next) to access the command history,\nand emacs-style control keys to edit the line.\n\ntccg>\n\\end{verbatim}\n\\end{small}\n\nTyping in \\texttt{:h} shows all the available commands. For example,\n\\texttt{:derivs} turns on the display of derivations when you parse an\nexpression:\n\n\\begin{small}\n\\begin{verbatim}\ntccg> :derivs\ntccg> the teacher buys\n3 parses found.\n\nParse 1: s/np\n------------------------------\n(lex) the :- np/^n\n(lex) teacher :- n\n(>) the teacher :- np\n(>T) the teacher :- s/@i(s\\@inp)\n(lex) buys :- s\\np/np\n(>B) the teacher buys :- s/np\n\ntccg>\n\\end{verbatim}\n\\end{small}\n\n\\noindent Here we see a (simplified) vertical display of the derivation\nseen earlier in Figure~\\ref{subj-v-agr}. (If you have \\LaTeX\\ installed,\nit's also possible to see derivations like those in\nFigure~\\ref{subj-v-agr} using the \\texttt{:vison} command, but note that\nits current behavior is a bit flaky.) Only the first parse is shown; the\nother two parses, for the ditransitive and \\cf{np} \\cf{pp\\fb{for}}\ncategories of the verb, can be seen by turning on all derivations with\nthe \\texttt{:all} command. To see the features on the categories, you\ncan use the \\texttt{:feats} command, optionally with a subset of\nfeatures to show.\n\nLogical forms can be shown with the \\texttt{:sem} command:\n\n\\begin{small}\n\\begin{verbatim}\ntccg> :noderivs\ntccg> :sem\ntccg> she bought the policeman a flower\n1 parse found.\n\nParse: s :\n @b1:action(buy ^\n past ^\n (p1:animate-being ^ pro3f ^\n sg) ^\n (p2:person ^ policeman ^\n the ^\n sg) ^\n (f1:thing ^ flower ^\n a ^\n sg))\ntccg>\n\\end{verbatim}\n\\end{small}\n\nTo see the realizations for this logical form (i.e., the one from the \nprevious parse), use the \\texttt{:r} command:\n\n\\begin{small}\n\\begin{verbatim}\ntccg> :nosem\ntccg> :r\n[1.000] she bought the policeman a flower :- s\n[0.167] she bought a flower for the policeman :- s\ntccg>\n\\end{verbatim}\n\\end{small}\n\n\\noindent Realizations are ordered by their n-gram similarity to the\npreviously entered expression. You can have a look in the\n\\texttt{morph.xml} file for more words to form expressions with.\n\nNote that the settings of the various options available in\n\\texttt{tccg}; use \\texttt{:reset} to undo all these settings and return\nto the default ones.\n\n\\subsection{\\texttt{ccg-test}}\n\nThe \\texttt{ccg-test} tool is for regression testing, and also provides\noptions for timing the realizer. Its (primary) usage is\n\n\\begin{verbatim}\n ccg-test (-noparsing|-norealization) \n (-g ) \n ()\n\\end{verbatim}\n\nBy default, \\texttt{ccg-test} will use the grammar in the current\ndirectory and the default regression file, \\texttt{testbed.xml}.\n\nNote that you can set realizer options, such as its time limit, in\n\\texttt{tccg}, e.g.\\ by issuing the command \\texttt{:tl 1000} (for time\nlimit 1000 ms.), and this value will persist and be used by\n\\texttt{ccg-test}.\n\n\\subsection{\\texttt{ccg-realize}}\n\nThe \\texttt{ccg-realize} tool provides a sample interface to the\nrealizer (see the underlying \\texttt{opennlp/ccg/Realize.java} file),\nand can be an aid in debugging realization. It loads a grammar, runs the\nrealizer on an input XML file, and logs its processing to an output text\nfile (or to system out). Its usage is\n\n\\begin{verbatim}\n ccg-realize (-g ) ()\n\\end{verbatim}\n \nYou can create input files for \\texttt{ccg-realize} using the\n\\texttt{:2xml} option in \\texttt{tccg}.\n\n\\section{Building grammars}\n\n\\occg\\ comes with various utilities to help you build the files used by\nthe runtime system---and to validate their contents---rather than\nwriting them entirely by hand. The utilities take advantage of the\n\\texttt{ccg-build} front end to the Apache Ant\n(\\url{http://ant.apache.org}) build tool. In principle,\n\\texttt{ccg-build} allows you to organize your files in any way you like\nto produce the runtime grammar files.\n\n\\subsection{Validating the grammar files}\n\nYou can use \\texttt{ccg-build} to validate the grammar files against\ntheir XML schemas. To do so, you need to have a \\texttt{build.xml} file\nin your grammar directory, which contains build tasks for the Apache Ant\ntool to carry out. The \\texttt{tiny} grammar directory contains a build\nfile which just validates the runtime files, as shown below:\n\n\\begin{small}\n\\begin{verbatim}\nD:\\Mike\\dev\\openccg\\grammars\\tiny>ccg-build\nBuildfile: build.xml\n\ninit:\n [echo] ----------- OpenCCG ------------\n\ngrammar:\n [echo] Validating grammar.xml, lexicon.xml, morph.xml, \n rules.xml and types.xml\n\nBUILD SUCCESSFUL\nTotal time: 4 seconds\n\\end{verbatim}\n\\end{small}\n\n\\noindent If there are any errors, validation with \\texttt{ccg-build}\ngives relatively informative error messages. Loading a grammar into\n\\texttt{tccg} will perform some further checks, but note that loading a\ngrammar with errors usually means \\texttt{tccg} croaks---outputting\nonly (less informative) stack traces---so it's good practice to validate\nany changes you make to your grammars prior to running \\texttt{tccg}.\n\n\\subsection{Using a \\texttt{dict.xml} file}\n\nRather than creating a \\texttt{morph.xml} file directly, you can employ\na \\texttt{dict.xml} file, which groups word forms by their stems and\nparts of speech, and lists the closed families for a given stem. A\n\\texttt{dict.xml} files usually works together with a file called\n\\texttt{lexicon-base.xml}, which does not contain \\textsl{member}\nentries for families. From the \\texttt{dict.xml} file---which also\ncontains macro definitions---the \\texttt{morph.xml} file can be\ngenerated automatically, with proper hooks to a derived\n\\texttt{lexicon.xml} file, using \\texttt{ccg-build}. (See the\n\\texttt{dict.xsd} schema in the \\texttt{grammars} directory for a\ncomplete description.)\n\nIn short, the simplest way to use a \\texttt{dict.xml} file with\n\\texttt{ccg-build} is to prepare a family of categories in the file\nnamed \\texttt{lexicon-base.xml} without \\textsl{member} entries, and to\ngroup stems and their word forms in a file called \\texttt{dict.xml},\nalong with macro definitions.\n\nA sample entry from the \\texttt{cem-english} \\texttt{dict.xml} \nfile appears below:\n\n\\begin{verbatim}\n \n \n \n \n \n \n \n \n\\end{verbatim}\n\n\\noindent Note that the stem \\gf{eat} is declared intransitive and\ntransitive without duplication in the morph or lexicon files. If you run\n\\texttt{ccg-build} as follows,\n\n\\begin{verbatim}\ncem-english> ccg-build grammar\nBuildfile: build.xml\n\ninit:\n [echo] ----------- OpenCCG ------------\n\ngrammar:\n [echo] Adding family members from dict.xml to \n lexicon-base.xml, yielding lexicon.xml\n\n [echo] Extracting morph items from dict.xml to morph.xml\n\n [echo] Validating grammar.xml, lexicon.xml, morph.xml, \n rules.xml and types.xml\n\nBUILD SUCCESSFUL\nTotal time: 5 seconds\n\\end{verbatim}\n\n\\noindent there will be two lexical assignments for every word form of\n\\gf{eat}, one for its intransitive use, and one for transitive.\n\n\\subsection{Reducing redundancy with XSLT}\n\n\\href{http://www.w3.org/Style/XSL/}{XSLT} is a language for transforming\nXML documents. Two XSLT transformations, \\texttt{add-family-members.xsl}\nand \\texttt{extract-morph.xsl}, are used by \\texttt{ccg-build} to handle\n\\texttt{dict.xml} files. You can also use XSLT transformations to reduce\nredundancy in the lexico-grammar specifications.\n\nFor example, the \\texttt{worldcup} grammar illustrates a couple of ways\nof using XSLT to improve the specification of lexical families. With\nthis grammar, the \\texttt{lexicon-base.xml} file is generated from an\nXSLT transformation, called \\texttt{lexicon-base.xsl}. This file begins\nwith the definition of variables for the various atomic categories used\nin later complex category and family definitions. The variable names\nfollow the format\n\n\\begin{verbatim} \n (.)?(.from-)?(.?)(.)*\n\\end{verbatim}\n\n\\noindent i.e., the category label, followed optionally by the\n\\texttt{id}, the \\texttt{inheritsFrom} id, the \\texttt{index} variable,\nand any further feature descriptions. This convention allows one to see\nwhat atomic categories are already in use, and to determine the contents\nof an atomic category at a glance. For example, \\texttt{np.3.Y.acc} is\nthe name of the category with label \\texttt{np}, id \\texttt{3}, index\n\\texttt{Y}, and the case value \\texttt{acc}; \n\\texttt{np.2.X.default} is similar, but has default variables for\nall features other than the \\texttt{index}.\n\nOnce variables have been declared, they can be referenced further on\nusing \\texttt{xsl:copy-of} statements. For example, the variable\n\\texttt{np.2.X.default} is referenced seven times in\n\\texttt{lexicon-base.xsl}. In this way, if a change to\n\\texttt{np.2.X.default} is desired, it can be made in one place in the\nfile, rather than seven.\n\nAnother XSLT mechanism employed in \\texttt{lexicon-base.xsl} is a named\ntemplated called \\texttt{extend} which serves to append one element as\nthe last child of another. This mechanism is used to associate logical\nforms with syntactic categories, as well as to create new categories\nfrom existing ones. For example, in\n\n\\begin{small}\n\\begin{verbatim} \n \n \n \n \n \n \n\\end{verbatim}\n\\end{small}\n\n\\noindent the category for a subject type-raised \\emph{wh}-determiner,\nlike \\gf{which}, is created by extending the category of a subject\ntype-raised \\emph{wh}--noun phrase (e.g.\\ \\gf{who}) with an extra\nnominal argument, \\texttt{fslash-n.2.X} (declared earlier as a forward\nslash plus a category with label \\texttt{n}, id \\texttt{2} and index\n\\texttt{X}).\\footnote{The \\texttt{xalan:nodeset} function makes it\npossible to use a variable as a parameter to a named template; its use\nwon't be necessary in future versions of XSLT.} In this way, any changes\nmade to the category for \\emph{wh}-determiners will carry over to the\ncategory for \\emph{wh}--noun phrases.\n\nAs XSLT is a powerful and extensible XML transformation language, there\nare many further possibilities for using it in grammar\ndevelopment---limited only by your imagination (and hacking ability). \n\n%% NB: Could eventually add lexical rule example.\n\n%% NB: Should add discussion of chunking rules.\n%% NB: Should add discussion of \\textsl{licensing-features}\n\n\n%% =====================================================================\n%% BIBLIOGRAPHY\n%% =====================================================================\n\n\\addcontentsline{toc}{section}{References}\n\\bibliographystyle{alpha}\n\\bibliography{openccg}\n\n\\end{document}\n\n" }, { "path": "docs/guide/openccg.bib", "content": "@string{nllt=\"Natural Language and Linguistic Theory\"}\n\n@phdthesis{Baldridge:2002,\n author = {Baldridge, Jason},\n title = {Lexically Specified Derivational Control in {C}ombinatory {C}ategorial {G}rammar},\n school = {School of Informatics, University of Edinburgh},\n year = {2002},\n note={Available from \\url{http://homepages.inf.ed.ac.uk/jbaldrid/dissertation/}}\n}\n% address = {Edinburgh, Scotland},\n\n@inproceedings{Baldridge/Kruijff:2002,\n author={Baldridge, Jason and Kruijff, Geert-Jan M.},\n title={Coupling {CCG} and Hybrid Logic Dependency Semantics},\n booktitle={Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002)},\n year = {2002},\n address = {Philadelphia, Pennsylvania}\n}\n\n@inproceedings{Baldridge/Kruijff:2003,\n author={Baldridge, Jason and Kruijff, Geert-Jan M.},\n title={Multi-Modal Combinatory Categorial Grammar},\n booktitle={Proceedings of the 10th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2003)},\n year = {2003},\n address = {Budapest, Hungary}}\n}\n\n@phdthesis{Kruijff:2001,\n author={Kruijff, Geert-Jan M.},\n title={A Categorial-Modal Logical Architecture of Informativity: Dependency Grammar Logic \\& Information Structure},\n school={Faculty of Mathematics and Physics, Charles University},\n address={Prague, Czech Republic},\n year={2001}}\n\n@book{Steedman:SynProc,\n author = {Mark Steedman},\n title = {The Syntactic Process},\n publisher = {The MIT Press},\n address=\"Cambridge Mass.\",\n year = {2000},\n}\n\n@inproceedings{White/Baldridge:2003,\n author = {White, Michael and Baldridge, Jason},\n title = {Adapting Chart Realization to {CCG}},\n booktitle = {Proceedings of the Ninth European Workshop on Natural Language Generation},\n year = {2003},\n address = {Budapest, Hungary}}\n}\n\n@Article{White-RLAC:2004,\nauthor = {Michael White}, \nyear = 2004, \ntitle = \"{Efficient Realization of Coordinate Structures in Combinatory Categorial Grammar}\",\njournal = {Research on Language and Computation}, \nnote = {To appear}\n}\n\n@InProceedings{White-INLG:2004,\nauthor = {Michael White}, \nyear = 2004, \ntitle = \"{Reining in CCG Chart Realization}\",\nbooktitle = {Proceedings of the Third International Conference on Natural Language Generation, INLG-04}\n}\n\n@InProceedings{White-ACLSoft:2005,\nauthor = {Michael White}, \nyear = 2005, \ntitle = \"Designing an Extensible {API} for Integrating Language Modeling and Realization\",\nbooktitle = {Proc.\\ ACL-05 Workshop on Software}\n}\n\n\n@mastersthesis{erkanms03, \nauthor = {G{\\\"u}ne{\\c{s}} Erkan}, \ntitle = {A Type System for {CCG}}, \nschool = {Middle East Technical University, Ankara},\nnote={Available from \\url{http://www.LcsL.metu.edu.tr/ftp/theses/erkan-ms-03.pdf.gz}},\nyear = {2003}, \n} \n\n@unpublished{bozsahinsteedman03,\nauthor=\"Cem Bozsahin and Mark Steedman\",\ntitle=\"Lexicalized Asymmetry and Syntactic Projection\",\nyear=\"2005\",\nnote=\"{Manuscript}, University of {E}dinburgh\"\n}\n\n@unpublished{Steedman/Baldridge:2003,\n author = \t {Mark Steedman and Jason Baldridge},\n title = \t {Combinatory {C}ategorial {G}rammar},\n note = {Tutorial paper, available from \\url{http://homepages.inf.ed.ac.uk/jbaldrid/ccg.pdf}},\n year = {2003},\n}\n\n@article{bozsahin02cl,\nauthor=\"Cem Bozsahin\",\ntitle= \"The Combinatory Morphemic Lexicon\",\njournal= \"Computational Linguistics\",\nvolume=28, \nnumber=2,\npages=\"145--176\",\nyear=\"2002\"\n}\n\n@article{bresnanmchombo95,\nauthor=\"Joan Bresnan and Sam A. Mchombo\",\ntitle= \"The {L}exical {I}ntegrity {P}rinciple: Evidence from {B}antu\",\njournal= nllt,\nvolume=\"13\",\npages=\"181--254\",\nyear=\"1995\"\n}\n\n" }, { "path": "docs/guide/openccg.sty", "content": "\r\n%%\r\n%% This style file contains a minimal set of commands for \r\n%% OpenCCG categories and logical forms, plus derivations. \r\n%% It borrows from earlier style files by Gann, Jason and Geert-Jan, \r\n%% and from Beryl and Mark for the derivations.\r\n%%\r\n\r\n\\newcommand{\\bs}{\\backslash} % backslash, to save typing\r\n\r\n\\newcommand{\\gf}[1]{\\textsf{\\textsl{#1}}} % gloss font, for words\r\n\\newcommand{\\cf}[1]{\\ensuremath{\\mathsf{#1}}} % category font\r\n\r\n\\newcommand{\\fb}[1]{\\ensuremath{_{\\mathit{#1}}}} % features, subscripted\r\n\\newcommand{\\fsb}[2]{\\ensuremath{_{\\langle#1\\rangle\\mathit{#2}}}} % fs index and features, subscripted\r\n\r\n\\newcommand{\\C}[1]{\\textbf{#1}} % concept font\r\n\\newcommand{\\con}[1]{\\ensuremath{\\mathrm{#1}}} % constant font, for sem feature values or sem sorts\r\n\r\n\\newcommand{\\modp}[1]{\\ensuremath{\\langle}\\textsc{#1}\\ensuremath{\\rangle}} % sem relation/feature\r\n\r\n\r\n\r\n\r\n% CCG derivations\r\n% arguments: #1 = no. of words, #2 = body\r\n% Carsten suggests removing @{}, in order to eliminate a gap on the \r\n% left end of lines in the derivation \r\n\\newcommand{\\deriv}[2]\r\n{ \r\n \\renewcommand{\\arraystretch}{.5}\r\n $\\begin{array}[t]{*{#1}{c}}\r\n #2\r\n \\end{array}$ \r\n}\r\n\r\n% centered multicolumn (NB: changed \\mc to \\cmc to avoid conflict with kluwer.cls) \r\n\\newcommand{\\cmc}[2]{\\multicolumn{#1}{c}{#2}}\r\n\r\n% Rules, argument #1 gives the number of columns to cover.\r\n\\newcommand{\\uline}[1]\r\n{\\cmc{#1}{\\hrulefill} }\r\n\\newcommand{\\fapply}[1]\r\n{ \\cmc{#1}{\\hrulefill_{>}} }\r\n\\newcommand{\\bapply}[1]\r\n{ \\cmc{#1}{\\hrulefill_{<}} }\r\n\\newcommand{\\fcomp}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}}}}\r\n\\newcommand{\\fxcomp}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}_{\\times}}}}\r\n\\newcommand{\\fxcompN}[2]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}^{#2}_{\\times}}}}\r\n\\newcommand{\\fcomptwo}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}^2}}}\r\n\\newcommand{\\fxcomptwo}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}_{\\times}^2}}}\r\n\\newcommand{\\fcompthree}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}^3}}}\r\n\\newcommand{\\fxcompthree}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}_{\\times}^3}}}\r\n\\newcommand{\\bcomp}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}}}}\r\n\\newcommand{\\bxcomp}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}_{\\times}}}}\r\n\\newcommand{\\bxcompN}[2]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}^{#2}_{\\times}}}}\r\n\\newcommand{\\bcomptwo}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}^2}}}\r\n\\newcommand{\\bxcomptwo}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}_{\\times}^2}}}\r\n\\newcommand{\\bcompthree}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}^3}}}\r\n\\newcommand{\\bxcompthree}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}_{\\times}^3}}}\r\n\\newcommand{\\fsubst}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{S}}}}\r\n\\newcommand{\\bsubst}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{S}}}}\r\n\\newcommand{\\fxsubst}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{S}_{\\times}}}}\r\n\\newcommand{\\bxsubst}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{S}_{\\times}}}}\r\n\\newcommand{\\ftype}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{T}}}}\r\n\\newcommand{\\btype}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{T}}}}\r\n\\newcommand{\\conj}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\Phi{>}}}}\r\n\\newcommand{\\boundary}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{\\%}}}}\r\n\\newcommand{\\asterisk}[1]\r\n{ \\cmc{#1}{\\hrulefill_{\\mathbf{*}}}}\r\n\\newcommand{\\comb}[2] % an arbitrary combinator\r\n{ \\cmc{#1}{\\hrulefill_{#2}}}\r\n\\newcommand{\\badcomb}[2] % an inappropriate use of a combinator\r\n{ \\cmc{#1}{\\hrulefill_{\\mbox{ *** }}\\hrulefill_{#2}}}\r\n\\newcommand{\\dcomp}[2] % an arbitrary dcomposition\r\n{ \\cmc{#1}{\\dotfill_{#2}}}\r\n\\newcommand{\\unfreeze}[1]\r\n{\\cmc{#1}{\\hrulefill_{\\mathbf{@}}} }\r\n\r\n\\newcommand{\\ul}{\\uline{1}}\r\n\r\n\\newcounter{CCG@counter}\r\n\\newcommand{\\CCG@amp}{&}\r\n\r\n\\newcommand{\\ulines}[1]{\r\n \\uline{1}\r\n \\setcounter{CCG@counter}{1} \r\n \\whiledo{\\value{CCG@counter} < #1} { \r\n \\CCG@amp \\uline{1}\r\n \\stepcounter{CCG@counter} \r\n }\r\n}\r\n\r\n" }, { "path": "docs/index.html", "content": "\n\n\n\n\n\n\n\n\n\n\nThe OpenCCG Homepage\n\n\n\n\n\n\n \n \n \n \n\n

OpenCCG: The OpenNLP CCG Library

Home

 
\n\n\n \n\n \n\n\n\n
 \n\n
\n\n

\nOpenCCG, the OpenNLP CCG Library, is an \nopen source natural language processing library written in\nJava, which provides parsing and realization services based on \nMark Steedman's \nCombinatory\nCategorial Grammar (CCG) formalism. \n

\n\n

\nThe library makes use of multi-modal extensions to CCG developed by \nJason Baldridge \nas part of the Grok system\n(the precursor to OpenCCG).\nThese extensions are described in Jason's \ndissertation \nand in a joint EACL-03 paper with \nGeert-Jan Kruijff. \n

\n\n

\nSubsequent development efforts, led by Michael White,\nhave focused on making the realizer practical to use in dialogue\nsystems, and more recently, on realization with broad coverage\ngrammars. See the papers on Mike's web page for details. Since\nversion 0.9.4, OpenCCG has included broad coverage English parsing and\nrealization support that together make it possible to experiment with\nopen domain grammatical paraphrasing. Version 0.9.5 adds features for\ndependency ordering and dependency length minimization in realization,\nas in White and\nRajkumar (2012), along with support for using 5-gram gigaword\nlanguage models with KenLM, and creating\ndisjunctive logical forms based on the differences between aligned\nsemantic graphs, as in Martin and\nWhite (2011). It also includes ccg2jsgf, an extension\ndeveloped for Knexus Research Corporation for compiling an OpenCCG\ngrammar into a context-free grammar in the Java Speech Grammar Format\nused by the Sphinx speech recognizer, now released open\nsource. (new!)\n

\n\n

\nAlso, Jason Baldridge and students at UT Austin have developed DotCCG,\na new format for specifying OpenCCG grammars, and VisCCG, an editor\nand visualizer for grammars written in DotCCG format. These\ndevelopments are described in Baldridge,\nChatterjee, Palmer and Wing (2007). See the UT Austin\ncomputational linguistics lab's OpenCCG\nwiki, which has a number of tutorials and example grammars for\nDotCCG and VisCCG.\n

\n\n

\nOpenCCG has been and is being used for a number of dialog systems: see\nthe\nlist of projects using OpenCCG. Please get in touch with Jason if you\nwould like add yours.\n

\n\n

\nFor the latest news about OpenCCG, check out the \nSourceForge\nproject page.\n

\n\n

\nFurther development of OpenCCG has moved to\nGitHub, \nwhere you can get the very latest code on branch\nmaster; releases and binaries will continue to be posted\non SourceForge.\n

\n\n
\n[Home]\n[Wiki]\n[GitHub]\n[SourceForge]\n[Download]\n\n[Forums]\n
\n\n
\n

\nEmail: mwhite at (no spam please) ling dot osu dot edu
\n2015 March 16
\n
\n \"SourceForge.net \n
\n

\n\n		 
\n" }, { "path": "docs/maxent.cpp.patch", "content": "--- /home/dnm/maxent/maxent-20061005/src/maxent.cpp\t2009-05-08 23:57:52.000000000 -0400\n+++ /home/dnm/maxentfixes/maxentpatched.cpp\t2009-05-08 23:57:47.000000000 -0400\n@@ -154,6 +154,9 @@\n const char* q = it->second;\n while (p < q && *p != ':')\n ++p;\n+\t // fixes the case where ':' is in the feature string (D.N. Mehay)\n+\t if((p+1) < q && *++p != ':')\n+\t --p;\n if (p == q)\n return false;\n context.push_back(make_pair(string(it->first, p - it->first), \n" }, { "path": "docs/realizer/build.xml", "content": "\n\n\n\n\t\n\t\n \n \n \n \n \n \n \n \n\n \t\n \n\n \n \n\t \n \n \n \t\n \t\n \n \t\n \t\n \n \t\n \t\n \n \t\n \t\n \n \t\n \t\n \n \t\n \t\n \n\t \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t \n \n \n \n\n \n \n \n \n \n\n" }, { "path": "docs/realizer/cgloss4e.sty", "content": "% -*- LaTeX -*-\r\n% Following borrows from Covington's style files inspired by Midnight by M.\r\n% de Groot, adapted to be used with gb4e.sty: examples beginning with \\ex can\r\n% contain glosses directly. Default is\r\n% Linguistic Inquiry style with all lines in \\rm; to change a line (eg. to\r\n% \\it for a particular journal, change the appropriate line: e.g.,\r\n% \\let\\eachwordone=\\rm in a copy of this file. Note that it will NOT work\r\n% to put \\it before the line as the words are parsed separately.\r\n\r\n% Use \\singlegloss to force single-spaced glosses even in double-space\r\n% environments. Works also in footnotes (^M as delimiter replaced by\r\n% \\\\)---hpk\r\n%\r\n%%%\r\n%%% Sentences with word-by-word glosses\r\n%%%\r\n\r\n% See covingtn.tex for full documentation. Some examples:\r\n%\r\n% Displayed sentence with gloss and translation:\r\n%\r\n% \\gll Dit is een Nederlands voorbeeld.\\\\\r\n% This is a Dutch example.\\\\\r\n% \\glt `This is an example in Dutch.'\r\n%\r\n% Same, using bracketing where words do not correspond one-to-one:\r\n%\r\n% \\gll Dit is een voorbeeldje in het Nederlands.\\\\\r\n% This is a {little example} in {} Dutch.\\\\\r\n% \\glt `This is a little example in Dutch.'\r\n%\r\n% If you want to align 3 lines rather than two, use \\glll instead of \\gll.\r\n%\r\n% Layout is critical between \\gll (or \\glll) and \\glt (or \\gln).\r\n%\r\n% Thanks to Marcel R. van der Goot for permission to reproduce code.\r\n\\let\\@gsingle=1\r\n\\def\\singlegloss{\\let\\@gsingle=1}\r\n\\def\\nosinglegloss{\\let\\@gsingle=0}\r\n\\@ifundefined{new@fontshape}%\r\n {\\def\\@selfnt{\\ifx\\@currsize\\normalsize\\@normalsize\\else\\@currsize\\fi}}\r\n {\\def\\@selfnt{\\selectfont}}\r\n\r\n\\def\\gll% % Introduces 2-line text-and-gloss.\r\n {\\begin{flushleft}\r\n \\ifx\\@gsingle1% conditionally force single spacing (hpk/MC)\r\n \\vskip\\baselineskip\\def\\baselinestretch{1}%\r\n \\@selfnt\\vskip-\\baselineskip\\fi%\r\n \\bgroup\r\n \\twosent\r\n }\r\n\r\n\\def\\glll% % Introduces 3-line text-and-gloss.\r\n {\\begin{flushleft}\r\n \\ifx\\@gsingle1% conditionally force single spacing (hpk/MC)\r\n \\vskip\\baselineskip\\def\\baselinestretch{1}%\r\n \\@selfnt\\vskip-\\baselineskip\\fi%\r\n \\bgroup\r\n \\threesent\r\n }\r\n\r\n\\def\\glt{\\vskip.17\\baselineskip}\r\n% Introduces a translation\r\n\\let\\trans\\glt\r\n\r\n\\def\\glend{} % obsolete\r\n % Ends the gloss environment.\r\n\r\n% The following TeX code is adapted, with permission, from:\r\n% gloss.tex: Macros for vertically aligning words in consecutive sentences.\r\n% Version: 1.0 release: 26 November 1990\r\n% Copyright (c) 1991 Marcel R. van der Goot (marcel@cs.caltech.edu).\r\n% Original Midnight/gloss.tex and Midnight/gloss.doc are available from\r\n% csvax.cs.caltech.edu [131.215.131.131] in pub/tex\r\n% and many other anonymous ftp archives.\r\n\r\n\\newbox\\lineone% boxes with words from first line\r\n\\newbox\\linetwo%\r\n\\newbox\\linethree%\r\n\\newbox\\wordone% a word from the first line (hbox)\r\n\\newbox\\wordtwo%\r\n\\newbox\\wordthree%\r\n\\newbox\\gline% the constructed double line (hbox)\r\n\\newskip\\glossglue% extra glue between glossed pairs or triples\r\n\\glossglue = 0pt plus 2pt minus 1pt % allow stretch/shrink between words\r\n%\\glossglue = 5pt plus 2pt minus 1pt % allow stretch/shrink between words\r\n\\newif\\ifnotdone\r\n\r\n\\@ifundefined{eachwordone}{\\let\\eachwordone=\\rm}{\\relax}\r\n\\@ifundefined{eachwordtwo}{\\let\\eachwordtwo=\\rm}{\\relax}\r\n\\@ifundefined{eachwordthree}{\\let\\eachwordthree=\\rm}{\\relax}\r\n\r\n\\def\\lastword#1#2#3% #1 = \\each, #2 = line box, #3 = word box\r\n {\\setbox#2=\\vbox{\\unvbox#2%\r\n \\global\\setbox#3=\\lastbox%\r\n }%\r\n \\ifvoid#3\\global\\setbox#3=\\hbox{#1\\strut{} }\\fi\r\n % extra space following \\strut in case #1 needs a space\r\n }\r\n\r\n\\def\\testdone\r\n {\\ifdim\\ht\\lineone=0pt\r\n \\ifdim\\ht\\linetwo=0pt \\notdonefalse % tricky space after pt\r\n \\else\\notdonetrue\r\n \\fi\r\n \\else\\notdonetrue\r\n \\fi\r\n }\r\n\r\n\\gdef\\getwords(#1,#2)#3 #4\\\\% #1=linebox, #2=\\each, #3=1st word, #4=remainder\r\n {\\setbox#1=\\vbox{\\hbox{#2\\strut#3 }% adds space\r\n \\unvbox#1%\r\n }%\r\n \\def\\more{#4}%\r\n \\ifx\\more\\empty\\let\\more=\\donewords\r\n \\else\\let\\more=\\getwords\r\n \\fi\r\n \\more(#1,#2)#4\\\\%\r\n }\r\n\r\n\\gdef\\donewords(#1,#2)\\\\{}%\r\n\r\n\\gdef\\twosent#1\\\\ #2\\\\{% #1 = first line, #2 = second line\r\n \\getwords(\\lineone,\\eachwordone)#1 \\\\%\r\n \\getwords(\\linetwo,\\eachwordtwo)#2 \\\\%\r\n \\loop\\lastword{\\eachwordone}{\\lineone}{\\wordone}%\r\n \\lastword{\\eachwordtwo}{\\linetwo}{\\wordtwo}%\r\n \\global\\setbox\\gline=\\hbox{\\unhbox\\gline\r\n \\hskip\\glossglue\r\n \\vtop{\\box\\wordone % vtop was vbox\r\n \\nointerlineskip\r\n \\box\\wordtwo\r\n }%\r\n }%\r\n \\testdone\r\n \\ifnotdone\r\n \\repeat\r\n \\egroup % matches \\bgroup in \\gloss\r\n \\gl@stop}\r\n\r\n\\gdef\\threesent#1\\\\ #2\\\\ #3\\\\{% #1 = first line, #2 = second line, #3 = third\r\n \\getwords(\\lineone,\\eachwordone)#1 \\\\%\r\n \\getwords(\\linetwo,\\eachwordtwo)#2 \\\\%\r\n \\getwords(\\linethree,\\eachwordthree)#3 \\\\%\r\n \\loop\\lastword{\\eachwordone}{\\lineone}{\\wordone}%\r\n \\lastword{\\eachwordtwo}{\\linetwo}{\\wordtwo}%\r\n \\lastword{\\eachwordthree}{\\linethree}{\\wordthree}%\r\n \\global\\setbox\\gline=\\hbox{\\unhbox\\gline\r\n \\hskip\\glossglue\r\n \\vtop{\\box\\wordone % vtop was vbox\r\n \\nointerlineskip\r\n \\box\\wordtwo\r\n \\nointerlineskip\r\n \\box\\wordthree\r\n }%\r\n }%\r\n \\testdone\r\n \\ifnotdone\r\n \\repeat\r\n \\egroup % matches \\bgroup in \\gloss\r\n \\gl@stop}\r\n\r\n\\def\\gl@stop{{\\hskip -\\glossglue}\\unhbox\\gline\\end{flushleft}}\r\n\r\n\\endinput\r\n" }, { "path": "docs/realizer/gb4e.sty", "content": "% -*- LaTeX -*-\r\n\\def\\gbVersion{4e}\r\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\r\n% Version 4export (= v. 4 minus the compatibility code)\r\n% Based on hpk's gb.sty, revised for GM syllabus by ct,\r\n% and incorporating macros adapted from J.Frampton, M. de Groot en M.\r\n% Covington. Full documentation soon to come in gb4doc.tex. Bug-reports\r\n% and suggestions for improvements, other used features, please!\r\n%\r\n% Notes:\r\n% \r\n% Various styles for X-bar levels; can be changed, but note that {picture}\r\n% environements (e.g. trees) will then come out wrong and have to be\r\n% fixed\r\n%\r\n% This file allows _ and ^ to be used in ordinary text, hence must be\r\n% loaded AFTER any file that uses them in their TeX meaning. Hence\r\n% cgloss(n).sty is loaded early in this file.\r\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\r\n\r\n%%%%%%%%%%%%%%%%%%%%%%%%\r\n% Format of examples: %\r\n%%%%%%%%%%%%%%%%%%%%%%%%\r\n% \\begin{exe} or \\exbegin\r\n% (arab.)\r\n% \\begin{xlist} or \\xlist\r\n% (1st embedding, alph.)\r\n% \\begin{xlisti} or \\xlisti\r\n% (2st embedding, rom.)\r\n% \\end{xlisti} or \\endxlisti\r\n% \r\n% \\end{xlist} or \\endxlist\r\n% \r\n% \\end{exe} or \\exend\r\n%\r\n% Other sublist-styles: xlistA (Alph.), xlistI (Rom.), xlistn (arab)\r\n%\r\n% \\ex (produces Number)\r\n% \\ex (numbered example)\r\n% \\ex[jdgmt]{sentence} (numbered example with judgement)\r\n%\r\n% \\exi{ident} (produces identifier)\r\n% \\exi{ident} (example numbered with identifier)\r\n% \\exi{ident}[jdgmt]{sentence} (dito with judgement)\r\n% (\\exr, \\exp and \\sn are defined in terms of \\exi)\r\n%\r\n% \\exr{label} (produces cross-referenced Num.)\r\n% \\exr{label} (cross-referenced example)\r\n% \\exr{label}[jdgmt]{sentence} (cross-referenced example with judgement)\r\n%\r\n% \\exp{label} (same as\r\n% \\exp{label} \\exr but\r\n% \\exp{label}[jdgmt]{sentence} with prime)\r\n%\r\n% \\sn (unnumbered example)\r\n% \\sn[jdgmt]{sentence} (unnumbered example with judgement)\r\n%\r\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\r\n\r\n\\@ifundefined{new@fontshape}{\\def\\reset@font{}\\let\\mathrm\\rm}{}\r\n\r\n\\let\\prmbrs=0\r\n\\def\\primebars{\\let\\prmbrs=1}\r\n\\def\\obar#1{\\ifmmode#1^{0}\\else#1$^{0}$\\fi} %% FIX\r\n\\def\\mbar#1{\\ifmmode#1^{\\mathrm{max}}\\else#1$^{\\mathrm{max}}$\\fi}\r\n\\def\\ibar#1{\\ifx\\prmbrs0%\r\n \\ifmmode\\overline{\\mathrm{#1}}\\else$\\overline{\\mbox{#1}}$\\fi%\r\n \\else\\ifmmode#1^{'}\\else#1$^{'}$\\fi\\fi}\r\n\\def\\iibar#1{\\ifx\\prmbrs0%\r\n \\ifmmode\\overline{\\overline{\\mathrm{#1}}}%\r\n \\else$\\overline{\\overline{\\mbox{#1}}}$\\fi%\r\n \\else #1P\\fi}\r\n\\def\\th{\\ifmmode\\theta\\else$\\theta$\\fi}\r\n\\def\\al{\\ifmmode\\alpha\\else$\\alpha$\\fi}\r\n\\def\\be{\\ifmmode\\beta\\else$\\beta$\\fi}\r\n\\def\\ga{\\ifmmode\\gamma\\else$\\gamma$\\fi}\r\n\\def\\de{\\ifmmode\\delta\\else$\\delta$\\fi}\r\n\\def\\spec#1{[Spec,#1]} %Def. of \"Specifier of #1\"\r\n\\def\\ct#1{{\\em #1\\/}} %Citation of linguistic material with alternative style:\r\n%\\def\\ct#1{`#1'}\r\n\\def\\tx{\\bf} %Introduction of technical terms with alternative style:\r\n%\\def\\tx{\\em}\r\n\r\n\\input{cgloss\\gbVersion.sty}\r\n\r\n%%% NEWSTUFF:\r\n%\\newcommand{\\indexgroupmark}[1]{\\item{\\bf #1}} % ?? -CT\r\n\r\n% this allows _ to be used in horizontal mode (from J.Frampton):\r\n% \\catcode`_=\\active\r\n% \\def_#1{\\ifmmode\\mit{\\sb{#1}}\\else${}\\sb{#1}$\\fi}\r\n% \\catcode`^=\\active\r\n% \\def^#1{\\ifmmode\\mit{\\sp{#1}}\\else${}\\sp{#1}$\\fi}\r\n\r\n% \\def\\lb#1{\\@ifnextchar [{\\@glarph{#1}}{\\@bl{#1}}}\r\n%\\def\\@glarph#1[#2]{\\ifmmode{[}\\sb{{\\mathrm{#1}}\\sb{#2}}\\else%\r\n% ${[}\\sb{{\\mathrm{#1}}\\sb{#2}}$\\fi}\r\n% \\def\\@bl#1{\\ifmmode{[}\\sb{\\mathrm{#1}}\\;\\else${[}\\sb{\\mathrm{#1}}\\;$\\fi}\r\n\r\n% \\def\\rb#1{\\@ifnextchar [{\\@grarph{#1}}{\\@br{#1}}}\r\n%\\def\\@grarph#1[#2]{\\ifmmode{]}\\sb{{\\mathrm{#1}}\\sb{#2}}\\else%\r\n% ${]}\\sb{{\\mathrm{#1}}\\sb{#2}}$\\fi}\r\n% \\def\\@br#1{\\ifmmode{]}\\sb{\\mathrm{#1}}\\;\\else${]}\\sb{\\mathrm{#1}}\\;$\\fi}\r\n\r\n%%% END_NEWSTUFF.\r\n\r\n\r\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\r\n% Macros for examples, roughly following Linguistic Inquiry style. %\r\n% From here on best not to tamper, else all the examples and cross- %\r\n% references will come out scrambled! (see also note below) - CT %\r\n% Completely rewritten for more robustness and flexibility. (hpk) %\r\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\r\n\r\n\\def\\qlist{\\begin{list}{\\Alph{xnum}.}{\\usecounter{xnum}%\r\n\\setlength{\\rightmargin}{\\leftmargin}}}\r\n\\def\\endqlist{\\end{list}}\r\n\r\n\\newif\\if@noftnote\\@noftnotetrue\r\n\\newif\\if@xrec\\@xrecfalse\r\n\\@definecounter{fnx}\r\n\r\n%%%% adapted from latex.tex to get examples in footnotes right\r\n\\long\\def\\@footnotetext#1{%\r\n \\@noftnotefalse\\setcounter{fnx}{0}%\r\n \\insert\\footins{\\reset@font\\footnotesize\r\n \\interlinepenalty\\interfootnotelinepenalty\r\n \\splittopskip\\footnotesep\r\n \\splitmaxdepth \\dp\\strutbox \\floatingpenalty \\@MM\r\n \\hsize\\columnwidth \\@parboxrestore\r\n \\edef\\@currentlabel{\\csname p@footnote\\endcsname\\@thefnmark}\\@makefntext\r\n {\\rule{\\z@}{\\footnotesep}\\ignorespaces\r\n #1\\strut}}\\@noftnotetrue}\r\n\r\n\\newcount\\@xnumdepth \\@xnumdepth = 0\r\n\r\n\\@definecounter{xnumi}\r\n\\@definecounter{xnumii}\r\n\\@definecounter{xnumiii}\r\n\\@definecounter{xnumiv}\r\n\\@definecounter{exx}\r\n\\setcounter{exx}{0}\r\n\\def\\thexnumi{\\@xsi{xnumi}}\r\n\\def\\thexnumii{\\@xsii{xnumii}}\r\n\\def\\thexnumiii{\\@xsiii{xnumiii}}\r\n\\def\\thexnumiv{\\@xsiv{xnumiv}}\r\n\\def\\p@xnumii{\\thexnumi}\r\n\\def\\p@xnumiii{\\thexnumi\\thexnumii-}\r\n\\def\\p@xnumiv{\\thexnumi\\thexnumii-\\thexnumiii-}\r\n\r\n\\def\\xs@default#1{\\csname @@xs#1\\endcsname}\r\n\\def\\@@xsi{\\let\\@xsi\\arabic}\r\n\\def\\@@xsii{\\let\\@xsii\\alph}\r\n\\def\\@@xsiii{\\let\\@xsiii\\roman}\r\n\\def\\@@xsiv{\\let\\@xsi\\arabic}\r\n\r\n\\@definecounter{rxnumi}\r\n\\@definecounter{rxnumii}\r\n\\@definecounter{rxnumiii}\r\n\\@definecounter{rxnumiv}\r\n\r\n\\def\\save@counters{%\r\n\\setcounter{rxnumi}{\\value{xnumi}}%\r\n\\setcounter{rxnumii}{\\value{xnumii}}%\r\n\\setcounter{rxnumiii}{\\value{xnumiii}}%\r\n\\setcounter{rxnumiv}{\\value{xnumiv}}}%\r\n\r\n\r\n\\def\\reset@counters{%\r\n\\setcounter{xnumi}{\\value{rxnumi}}%\r\n\\setcounter{xnumii}{\\value{rxnumii}}%\r\n\\setcounter{xnumiii}{\\value{rxnumiii}}%\r\n\\setcounter{xnumiv}{\\value{rxnumiv}}}%\r\n\r\n\r\n\\def\\exewidth#1{\\def\\@exwidth{#1}} \\exewidth{(234)}\r\n\\def\\exe{\\@ifnextchar [{\\@exe}{\\@exe[\\@exwidth]}}\r\n\r\n\\def\\@exe[#1]{\\ifnum \\@xnumdepth >0%\r\n \\if@xrec\\@exrecwarn\\fi%\r\n \\if@noftnote\\@exrecwarn\\fi%\r\n \\@xnumdepth0\\@listdepth0\\@xrectrue%\r\n \\save@counters%\r\n \\fi%\r\n \\advance\\@xnumdepth \\@ne \\@@xsi%\r\n \\begin{list}{(\\thexnumi)}%\r\n {\\usecounter{xnumi}\\@subex{#1}{1em}%\r\n \\if@noftnote%\r\n \\setcounter{xnumi}{\\value{exx}}%\r\n \\else%\r\n \\setcounter{xnumi}{\\value{fnx}}%\r\n \\fi}}\r\n\r\n\\def\\endexe{\\if@noftnote\\setcounter{exx}{\\value{xnumi}}%\r\n \\else\\setcounter{fnx}{\\value{xnumi}}%\r\n \\reset@counters\\@xrecfalse\\fi\\end{list}}\r\n\r\n\\def\\@exrecwarn{\\typeout{*** Recursion on \"exe\"---your\r\n example numbering will probably be screwed up!}}\r\n\r\n\\def\\xlist{\\@ifnextchar [{\\@xlist{}}{\\@xlist{}[iv.]}}\r\n\\def\\xlista{\\@ifnextchar [{\\@xlist{\\alph}}{\\@xlist{\\alph}[m.]}}\r\n\\def\\xlisti{\\@ifnextchar [{\\@xlist{\\roman}}{\\@xlist{\\roman}[iv.]}}\r\n\\def\\xlistn{\\@ifnextchar [{\\@xlist{\\arabic}}{\\@xlist{\\arabic}[9.]}}\r\n\\def\\xlistA{\\@ifnextchar [{\\@xlist{\\Alph}}{\\@xlist{\\Alph}[M.]}}\r\n\\def\\xlistI{\\@ifnextchar [{\\@xlist{\\Roman}}{\\@xlist{\\Roman}[IV.]}}\r\n\r\n\\def\\endxlist{\\end{list}}\r\n\\def\\endxlista{\\end{list}}\r\n\\def\\endxlistn{\\end{list}}\r\n\\def\\endxlistA{\\end{list}}\r\n\\def\\endxlistI{\\end{list}}\r\n\\def\\endxlisti{\\end{list}}\r\n\r\n\r\n%%% a generic sublist-styler\r\n\\def\\@xlist#1[#2]{\\ifnum \\@xnumdepth >3 \\@toodeep\\else%\r\n \\advance\\@xnumdepth \\@ne%\r\n \\edef\\@xnumctr{xnum\\romannumeral\\the\\@xnumdepth}%\r\n \\def\\@bla{#1}\r\n \\ifx\\@bla\\empty\\xs@default{\\romannumeral\\the\\@xnumdepth}\\else%\r\n \\expandafter\\let\\csname @xs\\romannumeral\\the\\@xnumdepth\\endcsname#1\\fi\r\n \\begin{list}{\\csname the\\@xnumctr\\endcsname.}%\r\n {\\usecounter{\\@xnumctr}\\@subex{#2}{1.5ex}}\\fi}\r\n\r\n\\def\\@subex#1#2{\\settowidth{\\labelwidth}{#1}\\itemindent\\z@\\labelsep#2%\r\n \\ifnum\\the\\@xnumdepth=1\\topsep 7\\p@ plus2\\p@ minus3\\p@\\else%\r\n \\topsep 2\\p@ plus2\\p@\\fi\\parsep 2\\p@ plus\\p@ minus\\p@%\r\n \\itemsep \\parsep\\leftmargin\\labelwidth\\advance\\leftmargin#2\\relax}\r\n\r\n%%% the example-items\r\n\\def\\ex{\\@ifnextchar [{\\@ex}{\\item}}\r\n\\def\\@ex[#1]#2{\\item\\@exj[#1]{#2}}\r\n\\def\\@exj[#1]#2{\\@exjbg{#1} #2 \\end{list}}\r\n\\def\\exi#1{\\item[#1]\\@ifnextchar [{\\@exj}{}}\r\n\\def\\judgewidth#1{\\def\\@jwidth{#1}}\r\n\\judgewidth{??}\r\n\\def\\@exjbg#1{\\begin{list}{#1}{\\@subex{\\@jwidth}{.5ex}}\\item}\r\n\\def\\exr#1{\\exi{{(\\ref{#1})}}}\r\n\\def\\exp#1{\\exi{{(\\ref{#1}$'$)}}}\r\n\\def\\sn{\\exi{}}\r\n\\def\\bu{\\item[$\\bullet$]}\r\n\r\n\r\n%%%%%%% \\lcomment for breaks in (example-)lists (leaves all counters\r\n%%%%%%% as they are) (hpk) \r\n\\newlength{\\lcommentsep}\r\n\\lcommentsep = 1ex\r\n\r\n\\long\\def\\lcomment#1%\r\n {\\vspace{\\lcommentsep}\r\n \\item[]\\hspace*{-\\leftmargin}%\r\n \\@tempskipa=\\linewidth%\r\n \\addtolength{\\@tempskipa}{\\rightmargin}%\r\n \\addtolength{\\@tempskipa}{\\leftmargin}%\r\n \\parbox{\\@tempskipa}{#1}%\r\n \\vspace{\\lcommentsep}%\r\n }\r\n\r\n%%%%%% control the alignment of exampleno. and (picture-)example\r\n%%%%%% (by Lex Holt ).\r\n\\def\\attop#1{\\leavevmode\\vtop{\\strut\\vskip-\\baselineskip\\vbox{#1}}}\r\n\\def\\atcenter#1{$\\vcenter{#1}$}\r\n%%%%%%\r\n\r\n%-------------------Move Arrows (from J.Frampton):\r\n\r\n\\def\\leaderfill{\\leaders\\hrule\\hfil}\r\n\\def\\pointerup{\\hbox to 0pt{\\hss\r\n \\vbox{\\offinterlineskip\\vskip-1pt\\hbox{\\elevenex\\char'170}\\null}\\hss}}\r\n\\def\\pointerdown{\\hbox to 0pt{\\hss\r\n \\vtop{\\offinterlineskip\\null\\hbox{\\elevenex\\char'171}\\vskip-1pt}\\hss}}\r\n\r\n\\let\\pu=\\pointerup\r\n\\let\\pd=\\pointerdown\r\n\\let\\lf=\\leaderfill\r\n\r\n\\def\\spacer{\\hskip4.5pt}\r\n\\def\\fillright#1{\\hfil#1\\leaderfill}\r\n\\def\\fillleft#1{\\leaderfill#1\\hfil}\r\n% Changed spelling to \\centr, else conflicts with LaTeX \\center{} -CT\r\n\\def\\centr#1{\\leaderfill#1\\leaderfill}\r\n\\def\\link#1{\\multispan#1\\leaderfill}\r\n\r\n\\def\\arrowalign#1{\\vtop{\\baselineskip=0pt \\lineskiplimit=0pt \\lineskip=2pt\r\n \\halign{&##\\cr#1}}}\r\n\r\n%\\font\\elevenex=cmex10 scaled\\magstephalf % just for the arrow!\r\n %PS: this may not work on some installations, not sure why. CT\r\n %%PPS: (e.g., PCTeX, but it works find works fine with EmTeX)\r\n%----------------END Move Arrows\r\n\r\n\\def\\pijl{$\\rightarrow$\\ }\r\n\r\n% Special accents for Vata & Gbadi; Navajo coming soon, I hope...:\r\n%\\def\\bb#1{$\\mathrm{\\overline{#1}}$} Following looks better:\r\n\\def\\bb#1{\\ifmmode\\overline{\\mathrm{#1}}\\else$\\bar{\\mathrm{#1}}$\\fi}\r\n\r\n\\def\\boven#1#2{\\raisebox{-0.2pt}{$\\stackrel{#1}{\\mathrm{#2}}$}}\r\n\\def\\bovenop#1#2{\\raisebox{-0.06ex}[0ex][0ex]{$\\stackrel{#1}{\\mathrm{#2}}$}}\r\n\\def\\vl{\\rule{0.05em}{0.30em}}\r\n\\def\\|#1{\\ifmmode\\vert#1\\else\\bovenop{\\vl}{#1}\\fi}\r\n" }, { "path": "docs/realizer/manual.tex", "content": "\n%%\n%% nb: use pdflatex to create pdf file with hyperlinks\n%%\n\n%% =====================================================================\n%% DOCUMENT DATA\n%% =====================================================================\n\n\\documentclass[11pt]{article}\n\n\\title{OpenCCG Realizer Manual}\n\\author{Michael White}\n\n\n%% =====================================================================\n%% PACKAGES\n%% =====================================================================\n\n\\usepackage{openccg} % for hlds/ccg\n\\usepackage{graphicx} % for figs\n\\usepackage{gb4e} % for examples\n%\\usepackage{cgmacros,hylo,ccg} % for hlds/ccg\n\n\\usepackage[\n colorlinks=true, linkcolor=blue, citecolor=blue, urlcolor=blue,\n pdfstartview=FitH,\n pdftitle={OpenCCG Realizer Manual},\n pdfauthor={Michael White}\n]{hyperref}\n\n%\\usepackage{mathptmx}\n\n%% listing settings \n%% nb: not crazy about font, esp that bold not working with keywords\n\\usepackage{listings,color}\n\\lstset{language=Java,basicstyle=\\ttfamily\\footnotesize,keywordstyle=\\underline,commentstyle=\\itshape\\color{blue}}\n%basicstyle=\\ttfamily\\small\n\n%% =====================================================================\n%% NEW COMMANDS\n%% =====================================================================\n\n%\\newcommand{\\occg}{\\textsf{OpenCCG}}\n\\newcommand{\\occg}{OpenCCG}\n\\newcommand{\\tccg}{\\textsf{tccg}}\n\\newcommand{\\ccgrz}{\\textsf{ccg-realize}}\n\\newcommand{\\ccgtest}{\\textsf{ccg-test}}\n\n\\newcommand{\\code}[1]{\\texttt{#1}} %\\small \n\\newcommand{\\eref}[2][]{(\\ref{ex:#2}#1)} % ref to examples\n\\newcommand{\\secref}[1]{Section~\\ref{sec:#1}} % ref to sections\n\\newcommand{\\figref}[1]{Figure~\\ref{fig:#1}} % ref to figures\n\\newlength{\\mytablen} % for indenting in terms\n\\newcommand{\\mytab}[1]{\n \\settowidth{\\mytablen}{\\ensuremath{#1}}\n \\mbox{\\hspace{\\mytablen}}\n}\n\\newcommand{\\xor}{~\\underline{\\vee}~}\n\\newcommand{\\shared}[1]{\\fbox{\\ensuremath{#1}}}\n\\newcommand{\\alt}[1]{\\mathsf{alt}_{#1}}\n\\newcommand{\\opt}[1]{\\mathsf{opt}_{#1}}\n \n%% =====================================================================\n%% DOCUMENT BODY\n%% =====================================================================\n\n\\begin{document}\n\n\\thispagestyle{empty}\n\\maketitle\n\\tableofcontents\n%\\listoftables\n\\listoffigures\n\\newpage\n\n\n%% to do:\n%% making an n-gram model\n%% DLFs\n\n\\section{About this manual}\n\nThis manual is a programmer's guide to using the \\occg\\ surface realizer\nin Java applications. You can download and install \\occg\\ from its\nwebsite, \\url{http://openccg.sourceforge.net}. Once you've unpacked the\narchive, have a look at the \\texttt{README} file for installation\ninstructions. For a brief introduction to writing grammars for \\occg,\nsee the ``rough guide'' in \\texttt{docs/grammars-rough-guide.pdf}.\n\n\n\\section{About the OpenCCG realizer}\n\\label{overview}\n\nThe OpenCCG realizer\n\\cite{White/Baldridge:2003,White-RLAC:2004,White-INLG:2004,White-ACLSoft:2005}\nis an open source surface realizer for Steedman's\n\\cite{Steedman-LI:2000,Steedman:SynProc} Combinatory Categorial Grammar\n(CCG), including the multi-modal extensions to CCG devised by Baldridge\nand Kruijff \\cite{Baldridge:PhD,Baldridge/Kruijff:2003}.\n\nLike other chart realizers\n\\cite{Kay:1996,Shemtov:PhD,Carroll-and-co:1999,Bob-Moore:2002}, the\nOpenCCG realizer takes as input a logical form specifying the\npropositional meaning of a sentence, and returns one or more surface\nstrings that express this meaning according to the lexicon and grammar.\nA distinguishing feature of OpenCCG is that it implements a hybrid\nsymbolic-statistical chart realization algorithm that combines (1) a\ntheoretically grounded approach to syntax and semantic composition, with\n(2) the use of integrated language models for making choices among the\noptions left open by the grammar, thereby reducing the need for\nhand-crafted rules.\n\nTo allow language models to be combined in flexible ways---as well as to\nenable research on how to best combine language modeling and\nrealization---OpenCCG's design includes an extensible API (application\nprogramming interface) that allows user-defined functions to be used for\nscoring partial realizations and for pruning low-scoring ones during the\nsearch. The design also includes classes for supporting a range of\nlanguage models and typical ways of combining them.\n\n\\begin{figure*}%[t]%[t]%[!h]\n\\begin{center}\n\\mbox{}\n\\includegraphics[width=\\textwidth]{realizer-class.pdf} \n\\caption{High-level architecture of the OpenCCG realizer}\n\\label{realizer-class}\n\\end{center}\n\\end{figure*}\n\nThe UML class diagram in Figure~\\ref{realizer-class} shows the\nhigh-level architecture of the OpenCCG realizer. A realizer\ninstance is constructed with a reference to a CCG grammar (which\nsupports both parsing and realization). The grammar's lexicon has\nmethods for looking up lexical items via their surface forms (for\nparsing), or via the principal predicates or relations in their\nsemantics (for realization). A grammar also has a set of hierarchically\norganized atomic types, which can serve as the values of features in the\nsyntactic categories, or as ontological sorts for the discourse\nreferents in the logical forms (LFs).\n\nLexical lookup yields lexical signs. A sign pairs a list of words with a\ncategory, which itself pairs a syntactic category with a logical form.\nLexical signs are combined into derived signs using the rules in the\ngrammar's rule group. Derived signs maintain a derivation history, and\ntheir word lists share structure with the word lists of their input\nsigns.\n\nFor generality, the realizer makes use of a configurable sign scorer and\npruning strategy. A sign scorer implements a function that returns a\nnumber between 0 and 1 for an input sign. For example, a standard\ntrigram language model can be used to implement a sign scorer, by\nreturning the probability of a sign's words as its score. A pruning\nstrategy implements a method for determining which edges to prune during\nthe realizer's search. The input to the method is a ranked list of edges\nfor signs that have equivalent categories (but different words);\ngrouping edges in this way ensures that pruning cannot ``break'' the\nrealizer, i.e.\\ prevent it from finding some grammatical derivation when\none exists. By default, an N-best pruning strategy is employed, which\nkeeps the N highest scoring input edges, pruning the rest (where N is\ndetermined by the current preference settings).\n\n\n\\begin{figure*}%[p]%[t]%[!h]\n\\begin{center}\n% \\mbox{}\n% \\includegraphics{code/realize.pdf} \n\\begin{lstlisting}\n// load grammar, instantiate realizer\nURL grammarURL = ...;\nGrammar grammar = new Grammar(grammarURL);\nRealizer realizer = new Realizer(grammar);\n\n// configure realizer with trigram backoff model \n// and 10-best pruning strategy\nrealizer.signScorer = new StandardNgramModel(3, \"lm.3bo\");\nrealizer.pruningStrategy = new NBestPruningStrategy(10);\n\n// ... then, for each request:\n\n// get LF from input XML\nDocument inputDoc = ...;\nLF lf = realizer.getLfFromDoc(inputDoc);\n\n// realize LF and get output words in XML\nEdge bestEdge = realizer.realize(lf);\nDocument outputDoc = bestEdge.sign.getWordsInXml();\n\n// return output\n... outputDoc ...;\n\\end{lstlisting}\n\\caption{Example realizer usage}\n\\label{realizer-usage}\n\\end{center}\n\\end{figure*}\n\n\\section{Using the realizer}\n\nSample Java code for using the realizer appears in\nFigure~\\ref{realizer-usage}. The input is an XML document that contains\nan \\code{lf} element either as the root or as a child of the root. To\ncreate a sample XML document with an acceptable format, you can use the\n\\tccg\\ tool's \\code{:2xml } command. Note that the\ninput XML document can be created in any way that is allowed by the JDOM\nAPI. For example, if the logical form is created by a Java XSLT-based\nsentence planner in the same process, the XSLT output can be captured in\na JDOM document, and then simply passed by reference to the realizer.\n\nThe output of the realizer is typically an XML document, as shown in the\nfigure. In such documents, each word in the output sequence appears in\nits own element; additionally, any pitch accents and boundary tones\nappear in separate elements, and any expanded multi-words are indicated.\nOutput documents of this kind can be easily processed into other formats\nusing XSLT. If a simple string output suffices, the\n\\code{Sign.getOrthography()} method can be used instead.\n\nThe realization algorithm is implemented by the \\code{realize(LF)} method.\nAs in the chart realizers cited earlier, the algorithm makes use of a\nchart and an agenda to perform a bottom-up dynamic programming search\nfor signs whose LFs completely cover the elementary predications in the\ninput logical form. The algorithm's details and a worked example appear\nin \\cite{White-RLAC:2004,White-INLG:2004}. To see a full realization\ntrace, you can use \\ccgrz\\ to realize an LF stored in an XML file (e.g.\\\none created using \\tccg). As shown in Figure~\\ref{realizer-usage}, the\n\\code{realize(LF)} method returns the edge for the best realization of the\ninput LF, as determined by the sign scorer. After a realization request,\nthe N-best complete edges---or more generally, all the edges for\ncomplete realizations that survived pruning---are also available from\nthe chart. To access these edges, you can invoke\n\\code{realizer.getChart().bestEdges()}.\n\nThe search for complete realizations proceeds in one of two modes,\nanytime and two-stage (packing/unpacking). In the anytime mode, a\nbest-first search is performed with a configurable time limit (which may\nbe a limit on how long to look for a better realization, after the first\ncomplete one is found). With this mode, the scores assigned by the sign\nscorer determine the order of the edges on the agenda, and thus have an\nimpact on realization speed. In the two-stage mode, a packed forest of\nall possible realizations is created in the first stage; then in the\nsecond stage, the packed representation is unpacked in bottom-up\nfashion, with scores assigned to the edge for each sign as it is\nunpacked, much as in \\cite{Langkilde:2000}. In both modes, the pruning\nstrategy is invoked to determine whether to keep or prune newly\nconstructed edges. For single-best output, the anytime mode can provide\nsignficant time savings by cutting off the search early; see\n\\cite{White-INLG:2004} for discussion. For N-best output---especially\nwhen a complete search (up to the edges that survive the pruning\nstrategy) is desirable---the two-stage mode can be more efficient.\n\n\n\\section{Scoring signs}\n\nThe classes for implementing sign scorers appear in\nFigure~\\ref{scorer-class}. In the diagram, classes for n-gram scoring\nappear towards the bottom, while classes for combining scorers appear on\nthe left, and the class for avoiding repetition appears on the right.\n\n\\begin{figure*}%[p]%[t]%[!h]\n\\begin{center}\n\\mbox{}\n\\includegraphics[width=\\textwidth]{scorer-class.pdf} \n\\caption{Classes for scoring signs}\n\\label{scorer-class}\n\\end{center}\n\\end{figure*}\n\n\\subsection{Standard n-gram models}\n\\label{standard-ngrams}\n\nThe \\code{Standard\\-Ngram\\-Model} class can load standard n-gram backoff\nmodels for scoring, as shown earlier in Figure~\\ref{realizer-usage}.\nSuch models can be constructed with the SRILM toolkit\n\\cite{SRILM-ICSLP:2002}, as described in Section~\\ref{using-srilm}; in\nprinciple, other toolkits could be used instead, as long as their output\ncould be converted into the same file formats. Since the SRILM toolkit\nhas more restrictive licensing conditions than those of OpenCCG's LGPL\nlicense, OpenCCG includes its own classes for scoring with n-gram\nmodels, in order to avoid any necessary runtime dependencies on the\nSRILM toolkit.\n\nThe n-gram tables are efficiently stored in a trie data structure (as in\nthe SRILM toolkit), thereby avoiding any arbitrary limit on the n-gram\norder. To save memory and speed up equality tests, each string is\ninterned (replaced with a canonical instance) at load time, which\naccomplishes the same purpose as replacing the strings with integers,\nbut without the need to maintain a separate mapping from integers back\nto strings. For better generalization, certain words may be dynamically\nreplaced with the names of their semantic classes when looking up n-gram\nprobabilities. Words are assigned to semantic classes in the lexicon,\nand the semantic classes to use in this way may be configured at the\ngrammar level. Note that \\cite{Oh/Rudnicky:2002} and \\cite{Adwait:2002}\nmake similar use of semantic classes in n-gram scoring, by deferring the\ninstantiation of classes (such as \\textit{departure city}) until the end\nof the generation process; our approach accomplishes the same goal in a\nslightly more flexible way, in that it also allows the specific word to\nbe examined by other scoring models, if desired.\n\nAs discussed in \\cite{White-INLG:2004}, with dialogue systems like COMIC\nn-gram models can do an excellent job of placing underconstrained\nadjectival and adverbial modifiers---as well as boundary tones---without\nresorting to the more complex methods investigated for adjective\nordering in \\cite{Shaw/Hatzi:1999,Malouf:2000}. For instance, in\nexamples like those in \\eref{adv-placement}, they correctly select the\npreferred positions for \\textit{here} and \\textit{also} (as well as for\nthe boundary tones), with respect to the verbal head and sister\ndependents:\n \n\\begin{exe}\n %\\small\n \\ex \\label{ex:adv-placement}\n \\begin{xlist}\n \\ex \n Here$_{L+H*}$ LH\\% we have a design in the classic$_{H*}$ style LL\\% .\n \\ex \n This$_{L+H*}$ design LH\\% here$_{L+H*}$ LH\\% is also$_{H*}$ classic LL\\% .\n \\end{xlist}\n\\end{exe}\n\nWe have also found that it can be useful to use reverse (or\n``right-to-left'') models, as they can help to place adverbs like\n\\textit{though}, as in \\eref{though}:\n\n\\begin{exe}\n %\\small\n \\ex \\label{ex:though}\n The tiles are also$_{H*}$ from the Jazz$_{H*}$ series though LL\\% .\n\\end{exe}\n\n\\noindent In principle, the forward and reverse probabilities should be\nthe same---as they are both derived via the chain rule from the same\njoint probability of the words in the sequence---but we have found that\nwith sparse data the estimates can differ substantially. In particular,\nsince \\textit{though} typically appears at the end of a variety of\nclauses, its right context is much more predictable than its left\ncontext, and thus reverse models yield more accurate estimates of its\nlikelihood of appearing clause-finally. \n\n\\subsection{N-gram scorers}\n\nThe \\code{Standard\\-Ngram\\-Model} class is implemented as a subclass of\nthe base class \\code{Ngram\\-Scorer}. All \\code{Ngram\\-Scorer} instances\nmay have any number of \\code{Ngram\\-Filter} instances, whose\n\\code{filter\\-Out} methods are invoked prior to n-gram scoring; if any\nof these methods return true, a score of zero is immediately returned.\nThe \\code{AAn\\-Filter} provides one concrete implementation of the\n\\code{Ngram\\-Filter} interface, and returns true if it finds a bigram\nconsisting of \\textit{a} followed by a vowel-inital word, or \\textit{an}\nfollowed by a consonant-initial word, subject to a configurable set of\nexceptions that can be culled from bigram counts. We have found that\nsuch n-gram filters can be more efficient, and more reliable, than\nrelying on n-gram scores alone; in particular, with \\textit{a/an}, since\nthe unigram probability for \\textit{a} tends to be much higher than that\nof \\textit{an}, with unseen words beginning with a vowel, there may not\nbe a clear preference for the bigram beginning with \\textit{an}.\n\nThe base class \\code{Ngram\\-Scorer} implements the bulk of the\n\\code{score} method, using an abstract \\code{log\\-Prob\\-From\\-Ngram} method\nfor subclass-specific calculation of the log probabilities (with\nbackoff) for individual n-grams. The \\code{score} method also invokes\nthe \\code{prepare\\-To\\-Score\\-Words} method, in order to allow for\nsubclass-specific pre-processing of the words in the given sign. With\n\\code{Standard\\-Ngram\\-Model}, this method is used to extract the word forms\nor semantic classes into a list of strings to score. It also appends any\npitch accents to the word forms or semantic classes, effectively\ntreating them as integral parts of the words.\n\nSince the realizer builds up partial realizations bottom-up rather than\nleft-to-right, it only adds start of sentence (and end of sentence) tags\nwith complete realizations. As a consequence, the words with less than a\nfull $n-1$ words of history are scored with appropriate sub-models. For\nexample, the first word of a phrase is scored with a unigram sub-model,\nwithout imposing backoff penalties.\n\nAnother consequence of bottom-up realization is that both the left- and\nright-contexts may change when forming new signs from a given input\nsign. Consequently, it is often not possible (even in principle) to use\nthe score of an input sign directly in computing the score of a new\nresult sign. If one could make assumptions about how the score of an\ninput sign has been computed---e.g., by a bigram model---one could\ndetermine the score of the result sign from the scores of the input\nsigns together with an adjustment for the word(s) whose context has\nchanged. However, our general approach to sign scoring precludes making\nsuch assumptions. Nevertheless, it is still possible to improve the\nefficiency of n-gram scoring by caching the log probability of a sign's\nwords, and then looking up that log probability when the sign is used as\nthe first input sign in creating a new combined sign---thus retaining\nthe same left context---and only recomputing the log probabilities for\nthe words of any input signs past the first one. (With reverse models,\nthe sign must be the last sign in the combination.) In principle, the\nderivation history could be consulted further to narrow down the words\nwhose n-gram probabilities must be recomputed to the minimum possible,\nthough \\code{Ngram\\-Scorer} only implements a single-step lookup at\npresent.\\footnote{Informal experiments indicate that caching log\nprobabilities in this way can yield an overall reduction in best-first\nrealization times of 2-3\\% on average.} Finally, note that a Java\n\\code{Weak\\-Hash\\-Map} is used to implement the cache, in order to avoid\nan undesirable buildup of entries across realization requests.\n\n\\subsection{Interpolation}\n\\label{interpolation}\n\nScoring models may be linearly interpolated in two ways. Sign scorers\nof any variety may be combined using the \\code{Sign\\-Scorer\\-Interpolation} \nclass. For example, Figure~\\ref{forward-reverse-interpolation} shows \nhow forward and reverse n-gram models may be interpolated.\n\n\\begin{figure*}%[p]%[t]%[!h]\n\\begin{center}\n\\begin{lstlisting}\n// configure realizer with 4-gram forward and reverse backoff \n// models, interpolated with equal weight\nNgramScorer forwardModel = new StandardNgramModel(4, \"lm.4bo\"); \nNgramScorer reverseModel = new StandardNgramModel(4, \"lm-r.4bo\"); \nreverseModel.setReverse(true);\nrealizer.signScorer = new SignScorerInterpolation(\n new SignScorer[] { forwardModel, reverseModel }\n);\n\\end{lstlisting}\n\\caption{Example interpolated n-gram model}\n\\label{forward-reverse-interpolation}\n\\end{center}\n\\end{figure*}\n\nWith n-gram models of the same direction, it is also possible to\nlinearly interpolate models at the word level, using the\n\\code{Linear\\-Ngram\\-Scorer\\-Combo} class. Word-level interpolation\nmakes it easier to use cache models created with maximum likelihood\nestimation, as word-level interpolation with a base model avoids\nproblems with zero probabilities in the cache model. As discussed in\n\\cite{Carsten-Alignment:2005}, cache models can be used to promote\nalignment with a conversational partner, by constructing a cache model\nfrom the bigrams in the partner's previous turn, and interpolating it\nwith a base model.\\footnote{At present, such cache models must be\nconstructed with a call to the SRILM toolkit; it would not be difficult\nto add OpenCCG support for constructing them though, since these models\ndo not require smoothing.} Figure~\\ref{base-cache-interpolation} shows\none way to create such an interpolated model.\n\n\\begin{figure*}%[p]%[t]%[!h]\n\\begin{center}\n\\begin{lstlisting}\n// configure realizer with 4-gram backoff base model, \n// interpolated at the word level with a bigram maximum-likelihood \n// cache model, with more weight given to the base model\nNgramScorer baseModel = new StandardNgramModel(4, \"lm.4bo\"); \nNgramScorer cacheModel = new StandardNgramModel(2, \"lm-cache.mle\"); \nrealizer.signScorer = new LinearNgramScorerCombo(\n new SignScorer[] { baseModel, cacheModel }, \n new double[] { 0.6, 0.4 }\n);\n\\end{lstlisting}\n\\caption{Example word-level interpolation of a cache model}\n\\label{base-cache-interpolation}\n\\end{center}\n\\end{figure*}\n\n\\subsection{N-gram precision models}\n\\label{ngram-precision}\n\nThe \\code{NgramPrecisionModel} subclass of \\code{Ngram\\-Scorer} computes a\nmodified version of the Bleu score used in MT evaluation\n\\cite{Bleu:2001}. Its constructor takes as input an array of target\nstrings---from which it extracts the n-gram sequences to use in\ncomputing the n-gram precision score---and the desired order. Unlike\nwith the Bleu score, rank order centroid weights (rather than the\ngeometric mean) are used to combine scores of different orders, which\navoids problems with scoring partial realizations which have no n-gram\nmatches of the target order. For simplicity, the score also does not\ninclude the Bleu score's bells and whistles to make cheating on length\ndifficult.\n \nWe have found n-gram precision models to be very useful for regression\ntesting the grammar, as an n-gram precision model created just from the\ntarget string nearly always leads the realizer to choose that exact\nstring as its preferred realization. These models can also be useful for\nevaluating the success of different scoring models in a cross-validation\nsetup, though with high quality output, manual inspection is usually\nnecessary to determine the importance of any differences between the\npreferred realization and the target string. Finally, note that n-gram\nprecision models can be used as a quick-and-dirty substitute for\nstandard n-gram models, if one does not have time to install and use the\nSRILM toolkit.\n\n\\subsection{Factored language models}\n\nA factored language model \\cite{Bilmes-Kirchoff:2003} is a new kind of\nlanguage model that treats words as bundles of factors. To support\nscoring with such models, OpenCCG represents words as objects with a\nsurface form, pitch accent, stem, part of speech, supertag, and semantic\nclass. Words may also have any number of further attributes, such as\nassociated gesture classes, in order to handle in a general way elements\nlike pitch accents that are ``coarticulated'' with words. \n\nTo represent words efficiently, and to speed up equality tests, all\nattribute values are interned, and the \\code{Word} objects themselves\nare interned via a factory method. Note that in Java, it is\nstraightforward to intern objects other than strings by employing a\n\\code{Weak\\-Hash\\-Map} to map from an object key to a weak reference to\nitself as the canonical instance. (Using a weak reference avoids\naccumulating interned objects that would otherwise be garbage\ncollected.)\n\nWith the SRILM toolkit, factored language models can be constructed that\nsupport \\textit{generalized parallel backoff}: that is, backoff order is\nnot restricted to just dropping the most temporally distant word first,\nbut rather may be specified as a path through the set of contextual\nparent variables; additionally, parallel backoff paths may be specified,\nwith the possibility of combining these paths dynamically in various\nways. In OpenCCG, the \\code{Factored\\-Ngram\\-Model} class supports\nscoring with factored language models that employ generalized backoff,\nthough parallel backoff is not yet supported, as it remains somewhat\nunclear whether the added complexity of parallel backoff is worth the\nimplementation effort. Typically, several related factored language\nmodels are specified in a single file and loaded by a\n\\code{Factored\\-Ngram\\-Model\\-Family}, which can multiplicatively score\nmodels for different child variables, and include different sub-models\nfor the same child variable.\n\nTo illustrate, let us consider a simplified version of the factored\nlanguage model family used in the COMIC realizer. This model computes\nthe probability of the current word given the preceding ones according\nto the formula shown in \\eref{comic-flm}, where a word consists of the\nfactors word (W), pitch accent (A), gesture class (GC), and gesture\ninstance (GI), plus the other standard factors which the model ignores:\n\n\\begin{exe}\n\\ex \\label{ex:comic-flm}\n\\begin{small}\n\\(\n\\begin{array}{l}\nP(\\langle W,A,GC,GI \\rangle \\, | \\, \\langle W,A,GC,GI \\rangle_{-1} \\, \\ldots) \\approx \\\\\n\\; \\; \\; P(W \\, | \\, W_{-1} W_{-2} A_{-1} A_{-2}) \\; \\times \\\\\n\\; \\; \\; P(GC \\, | \\, W) \\; \\times \\\\\n\\; \\; \\; P(GI \\, | \\, GC) \\\\\n\\end{array}\n\\)\n\\end{small}\n\\end{exe}\n\n\\noindent In \\eref{comic-flm}, the probability of the current word is\napproximated by the probability of the current word form given the\npreceding two word forms and preceding two pitch accents, multiplied by\nthe probability of the current gesture class given the current word\nform, and by the probability of the current gesture instance given the\ncurrent gesture class. Note that in the COMIC grammar, the choice of\npitch accent is entirely rule governed, so the current pitch accent is\nnot scored separately in the model. However, the preceding pitch accents\nare taken into account in predicting the current word form, as\nperplexity experiments have suggested that they do provide additional\ninformation beyond that provided by the previous word forms.\n\nThe specification file for this model appears in Figure~\\ref{flm-spec}.\nThe format of the file is a restricted form of the files used by the\nSRILM toolkit to build factored language models. The file specifies four\nmodels, where the first, third and fourth models correspond to those in\n\\eref{comic-flm}. With the first model, since the previous words are\ntypically more informative than the previous pitch accents, the backoff\norder specifies that the most distant accent, \\code{A(-2)}, should be\ndropped first, followed by the previous accent, \\code{A(-1)}, then the\nmost distant word, \\code{W(-2)}, and finally the previous word,\n\\code{W(-1)}. The second model is considered a sub-model of the\nfirst---since it likewise predicts the current word---to be used when\nthere is only one word of context available (i.e.\\ with bigrams). Note\nthat when scoring a bigram, the second model will take the previous\npitch accent into account, whereas the first model would not. For\ndocumentation of the file format as it is used in the SRILM toolkit, see\n\\cite{FLM-JHSW:2002}.\n\n\\begin{figure*}%[p]%[t]%[!h]\n\\begin{footnotesize}\n\\begin{verbatim}\n## Simplified COMIC realizer FLM spec file\n\n## Trigram Word model based on previous words and accents, dropping accents first, \n## with bigram sub-model;\n## Unigram Gesture Class model based on current word; and \n## Unigram Gesture Instance model based on current gesture class\n\n4\n\n## 3gram with A\nW : 4 W(-1) W(-2) A(-1) A(-2) w_w1w2a1a2.count w_w1w2a1a2.lm 5\n W1,W2,A1,A2 A2 ndiscount gtmin 1 \n W1,W2,A1 A1 ndiscount gtmin 1 \n W1,W2 W2 ndiscount gtmin 1 \n W1 W1 ndiscount gtmin 1 \n 0 0 ndiscount gtmin 1\n\n## bigram with A\nW : 2 W(-1) A(-1) w_w1a1.count w_w1a1.lm 3\n W1,A1 A1 ndiscount gtmin 1 \n W1 W1 ndiscount gtmin 1 \n 0 0 ndiscount gtmin 1\n\n## Gesture class depends on current word\nGC : 1 W(0) gc_w0.count gc_w0.lm 2\n W0 W0 ndiscount gtmin 1 \n 0 0 ndiscount gtmin 1\n\n## Gesture instance depends only on class\nGI : 1 GC(0) gi_gc0.count gi_gc0.lm 2\n GC0 GC0 ndiscount gtmin 1\n 0 0\n\\end{verbatim}\n\\end{footnotesize}\n\\caption{Example factored language model family specification}\n\\label{flm-spec}\n\\end{figure*}\n\nLike \\code{Standard\\-Ngram\\-Model}, the \\code{Factored\\-Ngram\\-Model} class\nstores its n-gram tables in a trie data structure, except that it stores\nan interned factor key (i.e.\\ a factor name and value pair, or just a\nstring, in the case of the word form) at each node, rather than a simple\nstring. During scoring, the \\code{log\\-Prob\\-From\\-Ngram} method determines\nthe log probability (with backoff) of a given n-gram by extracting the\nappropriate sequence of factor keys, and using them to compute the log\nprobability as with standard n-gram models. The\n\\code{Factored\\-Ngram\\-Model\\-Family} class computes log probabilities by\ndelegating to its component factored n-gram models (choosing appropriate\nsub-models, when appropriate) and summing the results.\n \n\\subsection{Avoiding repetition}\n\nWhile cache models appear to be a promising avenue to promote lexical\nand syntactic alignment with a conversational partner, a different\nmechanism appears to be called for to avoid ``self-alignment''---that\nis, to avoid the repetitive use of words and phrases. As a means to\nexperiment with avoiding repetition, OpenCCG includes the\n\\code{Repetition\\-Scorer} class. This class makes use of a configurable\npenalty plus a set of methods for dynamically managing the context. It\nreturns a score of \\( 10^{- c_r \\times p} \\), where $c_r$ is the count\nof repeated items, and $p$ is the penalty. Note that this formula\nreturns 1 if there are no repeated items, and returns a score that is\nlinear in log space with the number of repeated items otherwise.\n\nA repetition scorer can be combined multiplicatively with an n-gram\nmodel, in order to discount realizations that repeat items from the\nrecent context. Figure~\\ref{rep-scorer} shows such a combination,\ntogether with the operations for updating the context. By default, open\nclass stems are the considered the relevant items over which to count\nrepetitions, though this behavior can be specialized by subclassing\n\\code{Repetition\\-Scorer} and overriding the \\code{updateItems} method.\nNote that in counting repetitions, full counts are given to items in the\nprevious words or recent context, while fractional counts are given to\nolder items; the exact details may likewise be changed in a subclass, by\noverriding the \\code{repeatedItems} method.\n\n\\begin{figure*}%[p]%[t]%[!h]\n\\begin{center}\n\\begin{lstlisting}\n// set up n-gram scorer and repetition scorer\nString lmfile = \"ngrams/combined.flm\";\nNgramScorer ngramScorer = new FactoredNgramModelFamily(lmfile, true);\nngramScorer.addFilter(new AAnFilter());\nRepetitionScorer repetitionScorer = new RepetitionScorer();\n\n// combine n-gram scorer with repetition scorer\nrealizer.signScorer = new SignScorerProduct(\n new SignScorer[] { ngramScorer, repetitionScorer }\n);\n\n// ... then, after each realization request, \nEdge bestEdge = realizer.realize(lf);\n\n// ... update repetition context for next realization:\nrepetitionScorer.ageContext();\nrepetitionScorer.updateContext(bestEdge.getSign());\n\\end{lstlisting}\n\\caption{Example combination of an n-gram scorer and a repetition scorer}\n\\label{rep-scorer}\n\\end{center}\n\\end{figure*}\n\n\\subsection{Building language models with the SRILM toolkit}\n\\label{using-srilm}\n\nYou can use \\occg's regression testing tool, \\ccgtest, to help build and\ntest language models built with the SRILM toolkit. By default, running\n\\ccgtest\\ will use the grammar in the current directory to parse and\nrealize the default regression file, \\code{testbed.xml}, using an n-gram\nprecision model constructed for each test item. Using the appropriate\ncommand-line options, it is also possible to export the text of the test\nitems in order to construct an n-gram model with the SRILM toolkit, and\nthen use the resulting model in testing the realizer.\n\nTo display the syntax of \\ccgtest's command-line options, you can invoke\nit with the the \\code{-h} option, as shown in \\eref{ccg-test-help}. To\nexport the text of the test items to a text file, you use the\n\\code{-text} option, as in \\eref{export-text}. The next step is to use\nSRILM's \\code{ngram-count} tool to build an n-gram language model. In\n\\eref{make-lm},\\footnote{This command, and the ensuing ones, should be\nentered on one line.} \\code{ngram-count} is used to build a 4-gram\nbackoff model, \\code{n.4bo}, from the text file \\code{tb.txt}, using\nRistad's ``natural'' discounting method \\cite{Ristad:1995}. For small\ntest sets, we have found that Ristad's method works better than the\ndefault one (Good-Turing). Note that in \\eref{make-lm}, the \\code{-unk}\noption is used to reserve some probability for unknown words; the\n\\code{-gtmin 1} options (for N=1 to 4) specify to keep all 1-counts;\nand the \\code{-ndiscount} options (for N=1 to 4) specify the use of\nnatural discounting for unigrams through 4-grams. Finally, to test the\nresulting language model, you use \\ccgtest's \\code{-ngramorder} and\n\\code{-lm} options, as shown in \\eref{test-lm}.\n\n\\begin{exe}\n %\\small\n \\ex %\\label{ex:make-test-lm}\n \\begin{xlist}\n \\ex \\label{ex:ccg-test-help}\n \\code{ccg-test -h}\n \\ex \\label{ex:export-text} \n \\code{ccg-test -text tb.txt}\n \\ex \\label{ex:make-lm} \n \\code{ngram-count -order 4 -unk -text tb.txt -lm n.4bo \\\\\n -gt1min 1 -gt2min 1 -gt3min 1 -gt4min 1 \\\\\n -ndiscount1 -ndiscount2 -ndiscount3 -ndiscount4}\n \\ex \\label{ex:test-lm}\n \\code{ccg-test -noparsing -ngramorder 4 -lm n.4bo}\n \\end{xlist}\n\\end{exe}\n\nTo perform a simple 2-fold cross-validation, \\ccgtest\\ includes options\nfor exporting or testing just the even or odd test items. The command in\n\\eref{export-even} shows how you can export just the text of the\neven-numbered test items. Note that the \\code{-textsc} option specifies\nthat the text be exported using semantic class replacement, i.e.\\ with\ncertain words replaced with their semantic classes; the classes to use\nfor this purpose are specified using the \\code{replacement-sem-classes}\nattribute of the \\code{tokenizer} element in the \\code{grammar.xml}\nfile. The next step is to build a language model just as before; the\nabbreviated command appears in \\eref{make-even-lmsc}. Finally, you can\ntest the language model on just the odd-numbered items, as in\n\\eref{test-odd}, where the \\code{-lmsc} option specifies that semantic\nclass replacement should be employed when scoring realizations with the\nmodel. Naturally, you can switch the \\code{-even} and \\code{-odd}\nflags, and adjust the text and language model names, to test realization\non the even-numbered items, using a language model trained from the\nodd-numbered ones.\n\n\\begin{exe}\n %\\small\n \\ex %\\label{ex:even-odd}\n \\begin{xlist}\n \\ex \\label{ex:export-even} \n \\code{ccg-test -even -textsc tb-sc.even.txt}\n \\ex \\label{ex:make-even-lmsc} \n \\code{ngram-count -order 4 -unk -text tb-sc.even.txt \\\\\n -lm n-sc.even.4bo ...}\n \\ex \\label{ex:test-odd}\n \\code{ccg-test -noparsing -odd -ngramorder 4 -lmsc n-sc.even.4bo}\n \\end{xlist}\n\\end{exe}\n\nAn example of building a factored language model appears next. In\n\\eref{export-fsc}, the text of the test items is exported, where each\nword appears with all its factors, and word forms are replaced with\nsemantic classes when appropriate. In \\eref{make-flmsc}, SRILM's\n\\code{fngram-count} is used to create a factored language model from the\nspec file named \\code{spec.flm}. (Note that the various individual\nlanguage model files are listed in the spec file.) Finally,\n\\eref{test-flm} shows how the factored language model can be tested in\n\\ccgtest.\n\n\\begin{exe}\n %\\small\n \\ex \n \\begin{xlist}\n \\ex \\label{ex:export-fsc} \n \\code{ccg-test -textfsc tb-fsc.txt}\n \\ex \\label{ex:make-flmsc} \n \\code{fngram-count -factor-file spec.flm -text tb-fsc.txt -lm -unk}\n \\ex \\label{ex:test-flm}\n \\code{ccg-test -noparsing -flmsc spec.flm}\n \\end{xlist}\n\\end{exe}\n\n\n\\section{Pruning Strategies}\n\\label{pruning}\n\nThe classes for defining edge pruning strategies appear in\nFigure~\\ref{pruner-class}. As mentioned in Section~\\ref{overview}, an N-best\npruning strategy is employed by default, where N is determined by the\ncurrent preference settings. It is also possible to define custom\nstrategies. To support the definition of a certain kind of custom\nstrategy, the abstract class \\code{Diversity\\-Pruning\\-Strategy}\nprovides an N-best pruning strategy that promotes diversity in the edges\nthat are kept, according to the equivalence relation established by the\nabstract \\code{not\\-Compellingly\\-Different} method. In particular, in\norder to determine which edges to keep, a diversity pruning strategy\nclusters the edges into a ranked list of equivalence classes, which are\nsequentially sampled until the limit N is reached. If the\n\\code{single\\-Best\\-Per\\-Group} flag is set, then a maximum of one edge\nper equivalence class is retained.\n\n\\begin{figure*}%[p]%[t]%[!h]\n\\begin{center}\n\\mbox{}\n%scale=1.25\n\\includegraphics[width=\\textwidth]{pruner-class.pdf} \n\\caption{Classes for defining pruning strategies}\n\\label{pruner-class}\n\\end{center}\n\\end{figure*}\n\nAs an example, the COMIC realizer's diversity pruning strategy appears\nin Figure~\\ref{gest-diversity-strategy}. The idea behind this strategy\nis to avoid having the N-best lists become full of signs whose words\ndiffer only in the exact gesture instance associated with one or more of\nthe words. With this strategy, if two signs differ in just this way, the\nedge for the lower-scoring sign will be considered ``not compellingly\ndifferent'' and pruned from the N-best list, making way for other edges\nwhose signs exhibit more interesting differences.\n\n\\begin{figure*}%[p]%[t]%[!h]\n\\begin{center}\n\\begin{lstlisting}\n// configure realizer with gesture diversity pruner\nrealizer.pruningStrategy = new DiversityPruningStrategy() {\n /** \n * Returns true iff the given signs are not compellingly different; \n * in particular, returns true iff the words differ only in their \n * gesture instances. */\n public boolean notCompellinglyDifferent(Sign sign1, Sign sign2) {\n List words1 = sign1.getWords(); List words2 = sign2.getWords();\n if (words1.size() != words2.size()) return false;\n for (int i = 0; i < words1.size(); i++) {\n Word w1 = (Word) words1.get(i); Word w2 = (Word) words2.get(i);\n if (w1 == w2) continue;\n if (w1.getForm() != w2.getForm()) return false;\n if (w1.getPitchAccent() != w2.getPitchAccent()) return false;\n if (w1.getVal(\"GC\") != w2.getVal(\"GC\")) return false;\n // nb: assuming that they differ in the val of GI at this point \n }\n return true;\n }\n};\n\\end{lstlisting}\n\\caption{Example diversity pruning strategy}\n\\label{gest-diversity-strategy}\n\\end{center}\n\\end{figure*}\n\nOpenCCG also provides a concrete subclass of\n\\code{Diversity\\-Pruning\\-Strategy} named\n\\code{Ngram\\-Diversity\\-Pruning\\-Strategy}, which generalizes the\napproach to pruning described in \\cite{Langkilde:2000}. With this class,\ntwo signs are considered not compellingly different if they share the\nsame $n\\!-\\!1$ initial and final words, where $n$ is the n-gram order.\nWhen one is interested in single-best output, an n-gram diversity\npruning strategy can increase efficiency while guaranteeing no loss in\nquality---as long as the reduction in the search space outweighs the\nextra time necessary to check for the same initial and final\nwords---since any words in between an input sign's $n\\!-\\!1$ initial and\nfinal ones cannot affect the n-gram score of a new sign formed from the\ninput sign. However, when N-best outputs are desired, or when repetition\nscoring is employed, it is less clear whether it makes sense to use an\nn-gram diversity pruning strategy; for this reason, a simple N-best\nstrategy remains the default option.\n\n\n\\section{Disjunctive logical forms}\n\\label{sec:disj-lf}\n\nIn applications, to specify the desired space of possible paraphrases,\none may either provide an input logical form that underspecifies certain\nrealization choices, or include explicit disjunctions in the input LF\n(or both). In our experience, we have found disjunctive LFs---inspired\nby those found in \\cite{Shemtov:PhD}---to be an important capability,\nespecially as one seeks to make grammars reusable across applications.\n\n\\begin{figure}%[t]%[!h]\n\\begin{small}\n\\begin{center}\n\\includegraphics[width=0.52\\textwidth]{ex1a} \n\\end{center}\n\\begin{itemize}\n\\item[(a)] Semantic dependency graph for \\textit{The design (is$\\mid$'s) \nbased on the Funny Day collection by Villeroy and Boch.}\n\\end{itemize}\n\n\\vspace{3mm}\n\\begin{center}\n\\includegraphics[width=0.52\\textwidth]{ex1b} \n\\end{center}\n\\begin{itemize}\n\\item[(b)] Semantic dependency graph for \\textit{The design (is$\\mid$'s) \nbased on Villeroy and Boch's Funny Day series.}\n\\end{itemize}\n\n\\vspace{3mm}\n\\begin{center}\n\\includegraphics[width=0.52\\textwidth]{ex1c} \n\\end{center}\n\\begin{itemize}\n\\item[(c)] Disjunctive semantic dependency graph covering (a)-(b), i.e.\\\n\\textit{The design (is$\\mid$'s) based on \n(the Funny Day (collection$\\mid$series) by Villeroy and Boch $\\mid$\nVilleroy and Boch's Funny Day (collection$\\mid$series)).}\n\\end{itemize}\n\\end{small}\n\n\\caption{Example semantic dependency graphs.}\n\\label{fig:ex1}\n\\end{figure}\n\n\\begin{figure}%[t]%[!h]\n\\begin{small}\n\\ensuremath{\n@_{e}(\\C{be} \\wedge \\modp{tense}\\con{pres} \\wedge \\modp{mood}\\con{dcl} \\wedge \\\\\n \\mytab{@_{e}(}\n \\modp{Arg}(d \\wedge \\C{design} \\wedge \\modp{det}\\con{the} \\wedge \\modp{num}\\con{sg}) \\wedge \\\\\n \\mytab{@_{e}(}\n \\modp{Prop}(p \\wedge \\C{based-on} \\wedge \\\\\n \\mytab{@_{e}(\\modp{Prop}(}\n \\modp{Artifact}d \\wedge \\\\\n \\mytab{@_{e}(\\modp{Prop}(}\n \\modp{Source}(c \\wedge \\C{collection} \\wedge \\modp{det}\\con{the} \\wedge \\modp{num}\\con{sg} \\wedge \\\\\n \\mytab{@_{e}(\\modp{Prop}(\\modp{Source}(}\n \\modp{HasProp}(f \\wedge \\C{Funny\\_Day}) \\wedge \\\\\n \\mytab{@_{e}(\\modp{Prop}(\\modp{Source}(}\n \\modp{Creator}(v \\wedge \\C{V\\&B}))))\n}\n\n\\begin{center}\n(a)\n\n\\vspace{2mm}\n\\mbox{}\\vdots\n\\vspace{2mm}\n\\end{center}\n\n\\ensuremath{\n@_{e}(\\C{be} \\wedge \\modp{tense}\\con{pres} \\wedge \\modp{mood}\\con{dcl} \\wedge \\\\\n \\mytab{@_{e}(}\n \\modp{Arg}(d \\wedge \\C{design} \\wedge \\modp{det}\\con{the} \\wedge \\modp{num}\\con{sg}) \\wedge \\\\\n \\mytab{@_{e}(}\n \\modp{Prop}(p \\wedge \\C{based-on} \\wedge \\\\\n \\mytab{@_{e}(\\modp{Prop}(}\n \\modp{Artifact}d \\wedge \\\\\n \\mytab{@_{e}(\\modp{Prop}(}\n \\modp{Source}(c \\wedge \\modp{num}\\con{sg} \\wedge (\\modp{det}\\con{the})? \\wedge \\\\\n \\mytab{@_{e}(\\modp{Prop}(\\modp{Source}(}\n (\\C{collection} \\xor \\C{series}) \\wedge \\\\\n \\mytab{@_{e}(\\modp{Prop}(\\modp{Source}(}\n \\modp{HasProp}(f \\wedge \\C{Funny\\_Day}) \\wedge \\\\\n \\mytab{@_{e}(\\modp{Prop}(\\modp{Source}(}\n (\\modp{Creator}\\shared{v} \\xor \\modp{GenOwner}\\shared{v})))) \\\\\n\\wedge @_{v}(\\C{Villeroy\\_and\\_Boch})\n}\n\n\\begin{center}\n(c)\n\\end{center}\n\\end{small}\n\n\\caption{HLDS for examples in \\figref{ex1}.}\n\\label{fig:ex1-hlds}\n\\end{figure}\n\nAs an illustration of disjunctive logical forms, consider the semantic\ndependency graphs in \\figref{ex1}, which are taken from the\nCOMIC\\footnote{\\texttt{http://www.hcrc.ed.ac.uk/comic/}} multimodal\ndialogue system.\\footnote{To simplify the exposition, the features\n specifying information structure and deictic gestures have been\n omitted, as have the semantic sorts of the discourse referents.}\nGiven the lexical categories in the COMIC grammar, the graphs in\n\\figref{ex1}(a) and (b) fully specify their respective realizations,\nwith the exception of the choice of the full or contracted form of the\ncopula.\\footnote{Note that to be consistent with the distributed\n grammar, the predicate $\\C{based\\_on}$ should actually be\n $\\C{based-on}$; this discrepancy has been corrected in the\n subsequent figures.} To generalize over these alternatives, the\ndisjunctive graph in (c) may be employed. This graph allows a free\nchoice between the domain synonyms \\textit{collection} and\n\\textit{series}, as indicated by the vertical bar between their\nrespective predications. The graph also allows a free choice between\nthe \\modp{Creator} and \\modp{GenOwner} relations---lexicalized via\n\\textit{by} and the possessive, respectively---connecting the head $c$\n(\\textit{collection} or \\textit{series}) with the dependent $v$ (for\n\\textit{Villeroy and Boch}); this choice is indicated by an arc\nbetween the two dependency relations.\\footnote{Note that the arc and\n vertical bar are just presentation devices; there is no difference\n in the underlying implementation.} Finally, the determiner feature\n(\\modp{det}\\con{the}) on $c$ is indicated as optional, via the\nquestion mark.\\footnote{Another option would be to include the\n determiner feature in the alternative with the \\modp{Creator}\n relation, but that would make the graph harder to draw and would not\n illustrate optionality.}\n\nIt is worth pausing at this point to observe that in designing the\nCOMIC grammar, the differences between (a) and (b) could perhaps have\nbeen collapsed. However, such a move would make it more difficult to\nreuse the grammar in other applications---and indeed, the core of the\ngrammar is shared with the FLIGHTS system\n\\cite{FLIGHTS-FLAIRS:2004}---as it would presuppose that these\nparaphrases should always available in the same contexts. An example\nwhere the disjunctively specified paraphrases have applicable contexts\nthat are more clearly limited appears in \\eref{ex2}:\n\n\\begin{exe}\n\\ex \\label{ex:ex2}\n(This design $\\mid$ This one $\\mid$ This) (is$\\mid$'s) (classic $\\mid$\nin the classic style) $\\mid$ Here we have a (classic design $\\mid$\ndesign in the classic style).\n\\end{exe}\n\n\\noindent This example shows some of the phrasings that may be used in\nCOMIC to describe the style of a design that has not been discussed\npreviously. The example includes a top-level disjunction between the\nuse of a deictic NP \\textit{this design $\\mid$ this one $\\mid$ this}\n(with an accompanying pointing gesture) followed by the copula, or the\nuse of the phrase \\textit{here we have} to introduce the design.\nWhile these alternatives can function as paraphrases in this context,\nit is difficult to see how one might specify them in a single\nunderspecified (and application-neutral) logical form.\n\nGraphs such as those in \\figref{ex1} are represented internally using\nHybrid Logic Dependency Semantics (HLDS), as in \\figref{ex1-hlds}. In\nHLDS, as can be seen in \\figref{ex1-hlds}(a), each semantic head is\nassociated with a nominal that identifies its discourse referent, and\nheads are connected to their dependents via dependency relations, which\nare modeled as modal relations; modal relations are also used to\nrepresent semantic features (in which case the relation is to an\nanonymous node). In (c), two new operators are introduced to represent\nperiphrastic alternatives and optional parts of the meaning, namely\n$\\xor$ and $(\\cdot)?$, for exclusive-or and optionality, respectively.\nTo indicate that a nominal represents a reference to a node that is\nconsidered a shared part of multiple alternatives, the nominal is\nannotated with a box, as exemplified by \\shared{v}. This notion of\nshared references is needed during the logical form flattening stage of\nthe realization algorithm in order to determine which elementary\npredications are part of each alternative.\n\n\\begin{figure}%[t]%[!h]\n\\begin{footnotesize}\n\\begin{verbatim}\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\\end{verbatim}\n\\end{footnotesize}\n\n\\caption{XML for example (a) \\figref{ex1}.}\n\\label{fig:ex1-xml-a}\n\\end{figure}\n\n\\begin{figure}%[t]%[!h]\n\\begin{footnotesize}\n\\begin{verbatim}\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\\end{verbatim}\n\\end{footnotesize}\n\n\\caption{XML for example (c) \\figref{ex1}.}\n\\label{fig:ex1-xml-c}\n\\end{figure}\n\nTo specify inputs to the realizer, an XML representation of HLDS terms\nmay be employed; alternatively, the more intuitive XML graph\nrepresentation illustrated in \\figref{ex1-xml-a} and \\figref{ex1-xml-c}\nmay be used, with an automatic translation converting such\nrepresentations to HLDS. As can be seen in \\figref{ex1-xml-a}, the nodes\nand dependency relations in the graph are represented by \\texttt{node}\nand \\texttt{rel} elements. Note that \\texttt{node} elements that\nrepresent subordinated, reentrant references to a node use an\n\\texttt{idref} attribute, as exemplified by the \\texttt{Artifact}\nrelation to the \\texttt{node} element with \\texttt{idref=\"d\"}.\n\\figref{ex1-xml-c} shows how periphrastic alternatives and optional\nparts of the meaning are specified using the \\texttt{one-of} and\n\\texttt{opt} elements, respectively. Where the alternatives involve\nattributes of a node, an \\texttt{atts} element is used to provide the\nlexical predications or semantic features in question. Finally, note\nthat \\texttt{node} elements that represent references to a node that is\nconsidered a shared part of multiple alternatives is marked with the\n\\texttt{shared=\"true\"} attribute, as is the case here with the\nreferences to the dependent node $v$ (for \\textit{Villeroy and Boch}).\n\n\n\n%% =====================================================================\n%% BIBLIOGRAPHY\n%% =====================================================================\n\n\\addcontentsline{toc}{section}{References}\n\\bibliographystyle{alpha}\n\\bibliography{refs}\n\n\\end{document}\n\n" }, { "path": "docs/realizer/openccg.sty", "content": "\r\n%%\r\n%% This style file contains a minimal set of commands for \r\n%% OpenCCG categories and logical forms, plus derivations. \r\n%% It borrows from earlier style files by Gann, Jason and Geert-Jan, \r\n%% and from Beryl and Mark for the derivations.\r\n%%\r\n\r\n\\newcommand{\\bs}{\\backslash} % backslash, to save typing\r\n\r\n\\newcommand{\\gf}[1]{\\textsf{\\textsl{#1}}} % gloss font, for words\r\n\\newcommand{\\cf}[1]{\\ensuremath{\\mathsf{#1}}} % category font\r\n\r\n\\newcommand{\\fb}[1]{\\ensuremath{_{\\mathit{#1}}}} % features, subscripted\r\n\\newcommand{\\fsb}[2]{\\ensuremath{_{\\langle#1\\rangle\\mathit{#2}}}} % fs index and features, subscripted\r\n\r\n\\newcommand{\\C}[1]{\\textbf{#1}} % concept font\r\n\\newcommand{\\con}[1]{\\ensuremath{\\mathrm{#1}}} % constant font, for sem feature values or sem sorts\r\n\r\n\\newcommand{\\modp}[1]{\\ensuremath{\\langle}\\textsc{#1}\\ensuremath{\\rangle}} % sem relation/feature\r\n\r\n\r\n\r\n\r\n% CCG derivations\r\n% arguments: #1 = no. of words, #2 = body\r\n% Carsten suggests removing @{}, in order to eliminate a gap on the \r\n% left end of lines in the derivation \r\n\\newcommand{\\deriv}[2]\r\n{ \r\n \\renewcommand{\\arraystretch}{.5}\r\n $\\begin{array}[t]{*{#1}{c}}\r\n #2\r\n \\end{array}$ \r\n}\r\n\r\n% centered multicolumn (NB: changed \\mc to \\cmc to avoid conflict with kluwer.cls) \r\n\\newcommand{\\cmc}[2]{\\multicolumn{#1}{c}{#2}}\r\n\r\n% Rules, argument #1 gives the number of columns to cover.\r\n\\newcommand{\\uline}[1]\r\n{\\cmc{#1}{\\hrulefill} }\r\n\\newcommand{\\fapply}[1]\r\n{ \\cmc{#1}{\\hrulefill_{>}} }\r\n\\newcommand{\\bapply}[1]\r\n{ \\cmc{#1}{\\hrulefill_{<}} }\r\n\\newcommand{\\fcomp}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}}}}\r\n\\newcommand{\\fxcomp}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}_{\\times}}}}\r\n\\newcommand{\\fxcompN}[2]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}^{#2}_{\\times}}}}\r\n\\newcommand{\\fcomptwo}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}^2}}}\r\n\\newcommand{\\fxcomptwo}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}_{\\times}^2}}}\r\n\\newcommand{\\fcompthree}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}^3}}}\r\n\\newcommand{\\fxcompthree}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{B}_{\\times}^3}}}\r\n\\newcommand{\\bcomp}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}}}}\r\n\\newcommand{\\bxcomp}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}_{\\times}}}}\r\n\\newcommand{\\bxcompN}[2]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}^{#2}_{\\times}}}}\r\n\\newcommand{\\bcomptwo}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}^2}}}\r\n\\newcommand{\\bxcomptwo}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}_{\\times}^2}}}\r\n\\newcommand{\\bcompthree}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}^3}}}\r\n\\newcommand{\\bxcompthree}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{B}_{\\times}^3}}}\r\n\\newcommand{\\fsubst}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{S}}}}\r\n\\newcommand{\\bsubst}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{S}}}}\r\n\\newcommand{\\fxsubst}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{S}_{\\times}}}}\r\n\\newcommand{\\bxsubst}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{S}_{\\times}}}}\r\n\\newcommand{\\ftype}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{>}\\mathbf{T}}}}\r\n\\newcommand{\\btype}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{T}}}}\r\n\\newcommand{\\conj}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\Phi{>}}}}\r\n\\newcommand{\\boundary}[1]\r\n{ \\cmc{#1}{\\hrulefill_{{<}\\mathbf{\\%}}}}\r\n\\newcommand{\\asterisk}[1]\r\n{ \\cmc{#1}{\\hrulefill_{\\mathbf{*}}}}\r\n\\newcommand{\\comb}[2] % an arbitrary combinator\r\n{ \\cmc{#1}{\\hrulefill_{#2}}}\r\n\\newcommand{\\badcomb}[2] % an inappropriate use of a combinator\r\n{ \\cmc{#1}{\\hrulefill_{\\mbox{ *** }}\\hrulefill_{#2}}}\r\n\\newcommand{\\dcomp}[2] % an arbitrary dcomposition\r\n{ \\cmc{#1}{\\dotfill_{#2}}}\r\n\\newcommand{\\unfreeze}[1]\r\n{\\cmc{#1}{\\hrulefill_{\\mathbf{@}}} }\r\n\r\n\\newcommand{\\ul}{\\uline{1}}\r\n\r\n\\newcounter{CCG@counter}\r\n\\newcommand{\\CCG@amp}{&}\r\n\r\n\\newcommand{\\ulines}[1]{\r\n \\uline{1}\r\n \\setcounter{CCG@counter}{1} \r\n \\whiledo{\\value{CCG@counter} < #1} { \r\n \\CCG@amp \\uline{1}\r\n \\stepcounter{CCG@counter} \r\n }\r\n}\r\n\r\n" }, { "path": "docs/realizer/refs.bib", "content": "\n@PhdThesis{Baldridge:PhD,\n author = \t {Jason Baldridge},\n title = \t {Lexically Specified Derivational Control in Combinatory Categorial Grammar},\n school = \t {School of Informatics, University of Edinburgh},\n year = \t 2002\n}\n\n@InProceedings{Baldridge/Kruijff:2002,\n author = \t {Baldridge, Jason and Kruijff, Geert-Jan},\n title = \t \"{Coupling {CCG} and Hybrid Logic Dependency Semantics}\",\n booktitle =\t {Proc.\\ ACL-02},\n year =\t 2002\n}\n% booktitle =\t {Proc.\\ of 40th Annual Meeting of the Association for Computational Linguistics},\n% pages = {319--326},\n% address =\t {Philadelphia, Pennsylvania},\n\n@InProceedings{Baldridge/Kruijff:2003,\n author = \t {Baldridge, Jason and Kruijff, Geert-Jan},\n title = \t \"{Multi-Modal Combinatory Categorial Grammar}\",\n booktitle = {Proc.\\ ACL-03},\n year =\t 2003\n}\n% booktitle = {Proc.\\ of 10th Annual Meeting of the European Association for Computational Linguistics},\n% address =\t {Budapest, Hungary}\n\n\n@phdthesis{Kruijff:PhD,\n author={Kruijff, Geert-Jan M.},\n title={A Categorial Modal Architecture of Informativity: \n\t Dependency Grammar Logic \\& Information Structure},\n school={Charles University},\n year=2001\n}\n\n\n@InProceedings{Carroll-and-co:1999,\nauthor = {John Carroll and Ann Copestake and Dan Flickinger and Victor Pozna\\'nski},\ntitle = {An efficient chart generator for (semi-) lexicalist grammars},\nbooktitle = {Proc.\\ EWNLG-99},\nyear = 1999\n}\n% booktitle = {Proc.\\ of the 7th European Workshop on Natural Language Generation}, \n% Toulouse, France.\n% pages = {86-95}\n\n@InProceedings{Kay:1996,\nauthor = {Martin Kay},\ntitle = {Chart generation},\nbooktitle = {Proc.\\ ACL-96},\nyear = 1996\n}\n% booktitle = {Proc.\\ of the 34th Annual Meeting of the Association for Computational Linguistics}, \n% pages = {200-204}\n% Santa Cruz, California.\n\n\n@InProceedings{Shieber:1988,\nauthor = {Stuart Shieber},\nyear = 1988,\ntitle = {A uniform architecture for parsing and generation},\nbooktitle = {Proc.\\ COLING-88}\n}\n% booktitle = {Proc.\\ of the 14th International Conference on Computational Linguistics}, \n% pages = {614-619}\n% Budapest, Hungary.\n\n\n@InProceedings{Bob-Moore:2002,\nauthor = {Robert C. Moore},\ntitle = {A Complete, Efficient Sentence-Realization Algorithm for Unification Grammar},\nyear = 2002,\nbooktitle = {Proc.\\ INLG-02}\n}\n% booktitle = {Proc.\\ of the 2nd International Natural Language Generation Conference}\n%pages?\n%New York\n\n\n@InProceedings{Knight/Hatzi:1995,\nauthor = {Kevin Knight and Vasileios Hatzivassiloglou},\nyear = 1995,\ntitle = {Two-level, many-paths generation},\nbooktitle = {Proc.\\ ACL-95}\n}\n\n%I. Langkilde and K. Knight. 1998a. Generation that\n%exploits corpus-based statistical knowledge. In Proc.\n%COLING-ACL.\n\n@InProceedings{Langkilde/Knight:1998,\n author = \t {Irene Langkilde and Kevin Knight}, \n title = \t {The practical value of n-grams in generation},\n booktitle = \t {Proc.\\ INLG-98},\n year =\t 1998\n}\n % booktitle = \t {Proc.\\ of the Ninth International Workshop on Natural Language Generation},\n\n@InProceedings{Langkilde:2000,\nauthor = {Irene Langkilde},\nyear = 2000,\ntitle = {Forest-based statistical sentence generation}, \nbooktitle = {Proc.\\ NAACL-00}\n}\n\n@InProceedings{Langkilde-Geary:2002,\nauthor = {Irene Langkilde-Geary}, \ntitle = {An Empirical Verification of Coverage and Correctness for a General-Purpose Sentence Generator}, \nyear = 2002,\nbooktitle = {Proc.\\ INLG-02}\n}\n% booktitle = {Proc.\\ of the Second International Natural Language Generation Conference}\n%pages?\n%New York\n\n\n@InProceedings{Srini/Owen:2000,\nauthor = {Srinivas Bangalore and Owen Rambow}, \ntitle = {Exploiting a Probabilistic Hierarchical Model for Generation}, \nyear = 2000, \nbooktitle = {Proc.\\ COLING-00}\n}\n% Saarbrucken, Germany, August 2000.\n\n\n@InProceedings{Varges/Mellish:2001,\nauthor = {Sebastian Varges and Chris Mellish},\nyear = 2001,\ntitle = {Instance-based Natural Language Generation},\nbooktitle = {Proc.\\ NAACL-01}\n}\n% booktitle = {Proc.\\ of the 2nd Meeting of the North American Chapter of the Association for Computational Linguistics},\n% pages = {1--8}\n% (NAACL-2001)\n% Carnegie Mellon University, Pittsburgh, PA, USA. \n\n\n@Article{Steedman-LI:2000,\nauthor = {Mark Steedman}, \nyear = 2000,\ntitle = \"Information Structure and the Syntax-Phonology Interface\", \njournal = \"Linguistic Inquiry\", \nvolume = 31,\nnumber = 4, \npages = {649--689}\n}\n \n\n@Book{Steedman:SynProc,\n author = \t {Mark Steedman},\n title = \t {The Syntactic Process},\n publisher = \t{MIT Press},\n year = \t 2000,\n}\n% address=\"Cambridge Mass.\",\n\n\n@InProceedings{White/Baldridge:2003,\nauthor = {Michael White and Jason Baldridge}, \nyear = 2003,\ntitle = \"{Adapting Chart Realization to {CCG}}\",\nbooktitle = {Proc.\\ EWNLG-03}\n} \n% booktitle = {Proc.\\ of the 9th European Workshop on Natural Language Generation}\n\n@Article{White-RLAC:2004,\nauthor = {Michael White}, \nyear = 2004, \ntitle = \"{Efficient Realization of Coordinate Structures in Combinatory Categorial Grammar}\",\njournal = {Research on Language and Computation}, \nnote = {To appear}\n}\n\n@InProceedings{White-INLG:2004,\nauthor = {Michael White}, \nyear = 2004, \ntitle = \"{Reining in CCG Chart Realization}\",\nbooktitle = {Proc.\\ INLG-04}\n}\n\n@InProceedings{White-ACLSoft:2005,\nauthor = {Michael White}, \nyear = 2005, \ntitle = \"Designing an Extensible {API} for Integrating Language Modeling and Realization\",\nbooktitle = {Proc.\\ ACL-05 Workshop on Software}\n}\n\n\n@InCollection{Kruijff:2003,\n author = {Geert-Jan M. Kruijff},\n title = {Binding Across Boundaries},\n booktitle = {Resource-Sensitivity in Binding and Anaphora},\n editor = {Geert-Jan M. Kruijff and Richard T. Oehrle},\n publisher = {Kluwer Academic Publishers},\n year = 2003\n}\n\n@PhdThesis{Shemtov:PhD,\nyear=1997,\nauthor={Hadar Shemtov},\ntitle={Ambiguity Management in Natural Language Generation},\nschool={Stanford University}\n}\n\n \n@TechReport{Bleu:2001,\nauthor = {Kishore Papineni and Salim Roukos and Todd Ward and Wei-Jing Zhu}, \nyear = 2001,\ntitle = \"{Bleu: a Method for Automatic Evaluation of Machine Translation}\",\ninstitution = {IBM}, \nnumber = {RC22176}\n}\n\n\n@InProceedings{FLIGHTS-FLAIRS:2004, \n author = \t {Johanna Moore and Mary Ellen Foster and Oliver Lemon and Michael White}, \n title = \t \"Generating Tailored, Comparative Descriptions in Spoken Dialogue\", \n booktitle = \t {Proc.\\ FLAIRS-04},\n year = {2004}\n}\n\n@inproceedings{COMIC-eChallenges:2003,\n title = {Towards Ambient Intelligence: Multimodal Computers that Understand our Intentions},\n author = {Els den Os and Lou Boves},\n booktitle = {Proc.\\ eChallenges-03},\n year = {2003}\n}\n\n@InProceedings{SRILM-ICSLP:2002,\nauthor = {Andreas Stolcke}, \ntitle = \"{SRILM} --- {A}n extensible language modeling toolkit\", \nbooktitle = {Proc.\\ ICSLP-02}, \nyear = 2002}\n\n@TechReport{Ristad:1995,\nauthor = {Eric S.\\ Ristad},\nyear = 1995,\ntitle = \"{A Natural Law of Succession}\",\ninstitution = {Princeton Univ.}, \nnumber = {CS-TR-495-95}\n}\n\n\n@InProceedings{Bilmes-Kirchoff:2003,\nauthor = {Jeff Bilmes and Katrin Kirchhoff}, \ntitle = \"Factored language models and general parallelized backoff\",\nbooktitle = {Proc.\\ HLT-03}, \nyear = 2003}\n\n@Misc{FLM-JHSW:2002,\nauthor = {Katrin Kirchhoff and Jeff Bilmes and Sourin Das and Nicolae Duta and Melissa Egan and Gang Ji and Feng He and John Henderson and Daben Liu and Mohamed Noamany and Pat Schone and Richard Schwartz and Dimitra Vergyri},\ntitle = \"{Novel Approaches to Arabic Speech Recognition: Report from the 2002 Johns-Hopkins Summer Workshop}\", \nyear = 2002} \n\n\n@Article{Daelemans:1999,\n author = {Walter Daelemans},\n title = \"Introduction to the special issue on memory-based language processing\",\n journal = {Journal of Experimental \\& Theoretical Artificial Intelligence},\n volume = 11, \n number = 3, \n year = 1999, \n pages = {287--296}\n}\n\n@InProceedings{Shaw/Hatzi:1999,\nauthor = {James Shaw and Vasileios Hatzivassiloglou}, \ntitle = \"Ordering Among Premodifiers\",\nbooktitle = {Proc.\\ ACL-99}, \nyear = 1999}\n% Proc. of the 37th Association for Computational Linguistics, pages 135-143, College Park, Maryland, June 1999\n\n@InProceedings{Malouf:2000,\nauthor = {Robert Malouf},\ntitle = \"The order of prenominal adjectives in natural language generation\",\nbooktitle = {Proc.\\ ACL-00}, \nyear = 2000}\n% Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics. Pages 85-92.\n\n@Article{Adwait:2002,\n author = {Adwait Ratnaparkhi},\n title = \"Trainable approaches to surface natural language generation and their application to conversational dialog systems\",\n journal = {Computer, Speech \\& Language}, \n volume = 16, \n number = {3/4}, \n year = 2002, \n pages = {435--455}\n}\n\n@Article{Oh/Rudnicky:2002,\n author = {Alice H. Oh and Alexander I. Rudnicky},\n title = \"Stochastic natural language generation for spoken dialog systems\", \n journal = {Computer, Speech \\& Language}, \n volume = 16, \n number = {3/4}, \n year = 2002, \n pages = {387--407}\n}\n\n\n@InProceedings{Carsten-Alignment:2005,\nauthor = {Carsten Brockmann and Amy Isard and Jon Oberlander and Michael White},\ntitle = \"Variable alignment in affective dialogue\",\nbooktitle = {Proc.\\ UM-05 Workshop on Affective Dialogue Systems},\nyear = 2005}\n\n@Unpublished{COMIC-D7.4:2004,\nauthor = {Michael White},\ntitle = \"Experiments with Multimodal Output in Human-Machine Interaction\",\nnote = {IST Project COMIC Public Deliverable 7.4},\nyear = 2004}\n%\\texttt{http://www.hcrc.ed.ac.uk/comic/documents/deliverables/D7.4-final.pdf}\n\n@InProceedings{Foster-White-NLPXML:2004,\nauthor = {Mary Ellen Foster and Michael White},\ntitle = \"{Techniques for Text Planning with XSLT}\", \nbooktitle = {Proc.\\ 4th NLPXML Workshop}, \nyear = 2004}\n\n\n" }, { "path": "docs/style.css", "content": "body{\nfont-family: Arial, Helvetica, sans-serif;\nsize: 12pt;\nbackground: white;\ncolor: black;\n}\n\nh1{\nfont-family: Verdana, Arial, Helvetica, sans-serif;\nfont-size: 18pt;\ncolor: #006699;\nfont-weight: italic;\ntext-align: center;\n}\n\nh2{\nfont-family: Verdana, Arial, Helvetica, sans-serif;\nfont-size: 16pt;\nfont-weight: bold;\ncolor: #006699;\n}\n\nh3{\nfont-family: Verdana, Arial, Helvetica, sans-serif;\nfont-size: 10pt;\ncolor: #006699;\ntext-align: right;\n}\n\na{\nfont-family: arial,times;\n}\n\np{\nfont-family: Verdana, Arial, Helvetica, sans-serif;\ncolor: #000000;\nfont-size: 12pt;\n}\n\ntable{\nborder: 0;\nmargin: 0;\n}\n\ntd{\nfont-family: Bookman,Lucida, Helvetica,arial, times;\nfont-size: 12pt;\n}\n\ntd.header1{\nfont-family: Bookman,Lucida, Helvetica,arial, times;\nfont-size: 12pt;\nbackground: #52A0EF;\n}\n\ntd.banner{\nfont-family: Bookman,Lucida, Helvetica,arial, times;\nfont-size: 12pt;\nbackground: white;\n}\n\ntd.header2{\nfont-family: Bookman,Lucida, Helvetica,arial, times;\nfont-size: 12pt;\nbackground: #52A0EF;\n}\n" }, { "path": "docs/taggers-README", "content": "\nThe OpenCCG POS- and supertaggers can be used off-the-shelf, as well\nas with the OpenCCG parser. This README describes how to train and\nuse the taggers off-the-shelf; for their use with the OpenCCG parser,\nsee ccgbank-README.\n\nThe training process has been implemented in an ant build file, so\nit's a bit easier now. Before you get started, you'll need to\nconfigure your environment variables as described in the main README\n(ie, $OPENCCG_HOME/README). After that, you'll need to install SRILM and Zhang\nLe's maxent toolkit, as described below. Note that this should be\nrelatively straightforward on linux, but potentially difficult on\nother platforms.\n\nOnce you've taken care of these prerequisites, simply cd into\n$OPENCCG_HOME/ccgbank, and then you can simply train the POS- and\nsupertagger as follows:\n\n$ cd $OPENCCG_HOME/ccgbank\n$ ccg-build -f build-original.xml &> logs/log.original &\n\nNote that ccg-build is simply a front-end to ant that sets some\nenvironment variables, so you can pass through ant flags such as -f,\nwhich indicates that the build file to use is build-original.xml\n(rather than the default, build.xml).\n\nThe training will take a while; you can check the progess by peeking\nat the log file (logs/log.original), as well as the log files for\nindividual steps in the process, which you can see by looking in the\nbuild-original.xml file.\n\nOnce training is done, you can try it out as follows:\n\n$ ccg-build -f build-original.xml test &> logs/log.original.test &\n\nThis task will test the POS- and supertaggers on the dev section, and\nshould only take a matter of minutes. Note that you will likely have\nto increase the Java memory limit, if you haven't already done so;\nthis can be done by choosing a higher value for JAVA_MEM at the end of\nthe script $OPENCCG_HOME/bin/ccg-env, which is invoked by all the\nother OpenCCG scripts (including ccg-build).\n\nThe supertagger output file will be in the following format (with each\nsentence bracketed by ...):\n\n\n[tab][tab]...[tab]...[tab][tab]...[tab]...\n...\n[tab][tab]...[tab]...[tab][tab]...[tab]...\n\n\nAdjust the '-beta' option in the build file's test-st-model target to\ntag at greater tagging ambiguity levels.\n\nHave fun, and do let us know if anything in these instructions\nis flawed. \n\n\n----------------- REQUIRED SOFTWARE -----------------\n\nTo train the models, you'll need to have SRILM command-line\ntools (or some in-house stand-in) and Zhang Le's maxent\ntoolkit working.\n\nInstall SRILM as per the directions, and do the following\nto install the patched version of Zhang Le's toolkit:\n\n$ cd\n$ wget http://homepages.inf.ed.ac.uk/lzhang10/software/maxent/maxent-20061005.tar.bz2\n\nUnpack and patch the maxent.cpp file (it doesn't\ncover the case where ':' can be part of the feature\nsymbol itself, and not just a delimiter that separates\nstring repr's of features from their real-valued\nactivations).\n\n$ bunzip2 maxent-20061005.tar.bz2\n$ tar xf maxent-20061005.tar\n$ cd maxent-20061005/src\n$ patch maxent.cpp $OPENCCG_HOME/docs/maxent.cpp.patched\n\nNow compile the maxent code.\n\n$ cd \n$ cd maxent-20061005\n$ make clean all unittest\n\nTest to make sure it (more or less) works (I always only\npass 7 of the 8 tests, but the training seems to work):\n\n$ cd test\n$ ./runall.py\n\nFinally, add the 'maxent' binary (under 'maxent-20061005/src/opt')\nto your PATH environment variable.\n\nGood. Now we're ready to train some taggers!\n\n\n---------------- NOTES ------------------------\n\nAll the taggers in OpenCCG perform forward-backward tagging. To\nsimplify the implementation (and to take advantage of arbitrarily long\nn-gram histories of tags) we take a hybrid approach, simply\nmultiplying the non-sequence-aware maxent tagging model with a\nSRILM-trained (i.e., ARPA-formatted) model of tag sequences. This was\nthe alternative to the orthodox MEMM (Maximum Entropy Markov Model)\napproach suggested in (McCallum, et al., 2000, section 2.6).\nOtherwise, the approach closely follows that of Curran, Clark and\nVadas (2006), including the use of beta-best POS tags as features.\n\nThe POS- and supertaggers can make use of a prior model instead of a\ntag dictionary. The idea is to train a prior model to give\nprobabilistic features to a downstream maxent model and let it sort it\nout, rather than using tagging dictionaries (which use crude frequency\ncut-offs to determine which tags a word may be assigned).\nUnfortunately, empirical testing of this idea has been inconclusive,\nso it's been put on the back burner (by default, a prior model is used\nwith the POS tagger but not the supertagger); it remains for future\nwork to better test the idea, and in particular, whether the prior\nmodel feature approach can take better advantage of self-trained data\nby just re-training the prior model on such data.\n\n" }, { "path": "grammars/add-chunks.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n\r\n\r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n \r\n \r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/add-family-members.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n Warning, no family with name '\r\n \r\n ' found for entry with stem '\r\n \r\n '. \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n Warning, no entry with name '\r\n \r\n ' found in family with name '\r\n \r\n ' for entry with stem '\r\n \r\n '. \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/append.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/categories.xsd", "content": "\n\n\n\n \n \n Elements and types for categories and feature structures.\n\n By and large, there are no constraints on the names used in defining categories, \n with just a few exceptions:\n \n - '[*DEFAULT*]' proposition or feature value: this reserved value is replaced with \n the predicate associated with a lexical item (defaulting to the stem) \n when instantiating the lexical categories associated with a word\n \n - 'index' feature: this feature receives special treatment in the realizer; see \n Mike and Jason's paper on the realizer for details, at \n http://www.iccs.informatics.ed.ac.uk/~mwhite/White-Baldridge-ENLG-2003-to-appear.pdf \n\n - lists: lists are defined with the 'First' and 'Last' relations, which is \n (optionally) converted to just the 'List' relation by simplify-lists.xsl; \n elements in the list are represented using nodes with the 'elem' predicate, \n together with 'Item' and 'Next' relations to point to the actual list item \n and the next element, respectively\n \n - tuples: pairs (for argument clusters and gapping) are represented using \n nodes with the 'tup' (for tuple) predicate, together with 'Item1' and 'Item2' \n relations to point to the paired items (in principle, further 'ItemN' \n relations could be used for tuples of length greater than 2); \n paired items receive special treatment in the realizer\n \n - 'BoundVar' relation: this relation indicates that the subordinate nominal is \n a bound var; as a result, feature-based instantiation is disabled in the realizer\n \n - 'mark' feature: this semantic attribute indicates that the realizer should label \n the phrase in the XML output headed by the index associated with this feature\n \n \n\n \n \n \n \n \n \n \n \n \n A category, either atomic or complex.\n \n \n \n \n \n \n \n An atomic category.\n Atomic categories must have a type, and may contain \n a feature structure and LF.\n The possible LF predications are given by the \"hldsPreds\" type in hlds.xsd.\n \n \n \n \n \n \n \n \n \n \n The type of the category, e.g. \"np\".\n \n \n \n \n \n \n \n \n \n \n A complex category, consisting of a target category, \n an argument stack, and an optional LF.\n Since a complex cat is not curried, the target category will \n always be atomic.\n An argument stack is one or more basic args, dollar args or set args.\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n A simple, non-recursive feature structure, consisting of a set \n of features, i.e. attribute-value pairs.\n If there is only a single, string-valued feature, it can be \n specified using the \"attr\" and \"val\" attributes on this element.\n \n \n \n \n \n \n \n \n \n An integer id for the feature structure.\n The id is used for coindexation, \"inheritsFrom\" feature propagation, \n and macro access.\n \n \n \n \n \n \n This attribute is used to specify feature propagation with exceptions \n\t\t\t\t\t(i.e., default unification).\n At run time, feature equations are added in order to propagate feature values.\n In particular, a feature variable is added to this feature structure and \n a corresponding one is added to the referenced feature structure, \n for all appropriate attributes except those with explicit values already \n on this feature structure.\n The appropriate attributes are determined by the type of the atomic category, \n and consist of all the attributes seen with this category type in the lexicon \n file. NB: Attributes appearing in macros in the morph file are not included, \n since their association with category types is not available statically.\n \n \n \n \n \n \n The name of the single, string-valued feature (when appropriate).\n \n \n \n \n \n \n The value of the feature.\n \n \n \n \n \n \n \n \n \n A feature, i.e. an attribute-value pair.\n Features can be either syntactic or semantic (LF-valued).\n Syntactic features can be either ground or variable. \n If ground, they must be string-valued, and specified via the \"val\" attribute; \n if variable, the value is a variable named by a \"featvar\" element.\n Semantic features can also be either ground or variable; \n their possible values are given by the \"hldsFeatVals\" type in hlds.xsd.\n \n \n \n \n \n \n \n \n \n \n The name of the feature.\n \n \n \n \n \n \n The value of the feature, when string-valued.\n \n \n \n \n \n \n \n \n \n A variable over syntactic feature values.\n \n \n \n \n \n \n The name of the feature variable.\n \n \n \n \n \n \n \n \n \n \n \n \n A basic arg pairs a slash and a category.\n \n \n \n \n \n \n \n \n \n \n \n A dollar arg pairs an optional slash and a dollar variable.\n The slash defaults to the most general slash.\n \n \n \n \n \n \n \n \n \n \n \n A dollar variable in the definition of a complex category, \n i.e. a variable over any sublist of args in an arg stack.\n \n \n \n \n \n \n The name of the dollar variable, for coindexation purposes.\n \n \n \n \n \n \n \n \n \n A set arg, i.e. an unordered set of basic args.\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n A slash in the definition of a complex category.\n A slash has a direction, a mode and an ability; \n a variable can also be given for the mode, using the \n \"varmodality\" attribute.\n For discussion, see Jason's dissertation, esp. Ch. 8 on \n the implementation; the dissertation is downloadable from \n http://www.iccs.inf.ed.ac.uk/~jmb/dissertation\n \n \n \n \n \n \n The direction of the slash.\n The direction can be forward (/), backward (\\) or both (|).\n Defaults to both.\n \n \n \n \n \n \n \n \n \n \n \n \n \n The mode of the slash.\n The possible values are: \n all (.), application only (*), \n associative (^), permutative (x), \n permutative right (x>), permutative left (<x), \n associative permutative right (>), and associative permutative left (<).\n Defaults to all.\n See Jason's dissertation for details.\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n A variable over modalities.\n \n \n \n \n \n \n The ability of the slash, either inert or active.\n Defaults to either.\n This is used to implement antecedent government; \n see Jason's dissertation for details.\n \n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/comic/build.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n\n \n\n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n builds the grammar (default)\"/>\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "grammars/comic/dict.xml", "content": "\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/comic/dict.xsl", "content": "\n\n\n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n\n\n" }, { "path": "grammars/comic/grammar.xml", "content": "\n\n\n\n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/comic/lexicon-base.xsl", "content": "\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n Creator\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n Source\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/comic/lexicon.xml", "content": "\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/comic/morph.xml", "content": "\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/comic/rules-base.xml", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/comic/rules.xml", "content": "\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/comic/testbed.xml", "content": "\r\n\r\n\r\n\r\n Describing the style.\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n Manufacturer and series.\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n Decorative tiles.\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n Appositive 'with' now uses an elab-rel.\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n Colour scheme.\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n Added pitch accent onto canned text bits.\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n Combining things.\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n The shared argument (the colours) no longer has to be separated out, since the VPing is now an appositive.\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n NB: The relative clause 'that features ...' is now an appositive, and has theme status here; \r\n this would be appropriate in the context of the question 'do you have any more designs with \r\n blue in the colour scheme?'.\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n Transitions.\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n Further options (in thumbnails).\r\n\r\n \r\n NB: this clause from the sample dialogue gets its own sentence.\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n Questions.\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n Changed pitch accent and phrase to theme, for clarification context.\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n Changed pitch accent and phrase to theme, for clarification context.\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n User tailoring.\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/comic/types-extras.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/comic/types.xml", "content": "\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/convert-lists.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/convert-to-graph.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/convert-to-hlds.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n Error: node with id = should be a reference (with idref)\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\t \r\n\t \r\n\t \r\n \r\n \r\n \r\n \r\n \r\n\t \r\n\t \r\n\t\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/core-en/add-chunks.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/core-en/add-intonation-info.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n '\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/core-en/adj.xsl", "content": "\n\n\n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n\n\n\n\n" }, { "path": "grammars/core-en/adv.xsl", "content": "\n\n\n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "grammars/core-en/auxv.xsl", "content": "\n\n\n\n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "grammars/core-en/cats.xsl", "content": "\n\n\n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/core-en/conj.xsl", "content": "\n\n\n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n\n\n\n\n" }, { "path": "grammars/core-en/derive-features.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n rh\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n h\r\n \r\n \r\n \r\n s\r\n \r\n \r\n \r\n \r\n \r\n -\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/core-en/det.xsl", "content": "\n\n\n\n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n\n\n\n\n" }, { "path": "grammars/core-en/dict.xsl", "content": "\n\n\n\n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n\n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n\n\n\n\n" }, { "path": "grammars/core-en/drop-features.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/core-en/lexicon.xsl", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/core-en/misc.xsl", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n\n \n\n\n\n\n" }, { "path": "grammars/core-en/np.xsl", "content": "\n\n\n\n \n \n\n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n\n\n\n\n" }, { "path": "grammars/core-en/pp.xsl", "content": "\n\n\n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n Poss\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "grammars/core-en/punct.xsl", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n\n\n\n\n" }, { "path": "grammars/core-en/raise-nodes.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n Arg1\r\n \r\n \r\n \r\n\r\n\r\n" }, { "path": "grammars/core-en/templates.xsl", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n\n\n\n\n" }, { "path": "grammars/core-en/types.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/core-en/unary-rules.xsl", "content": "\n\n\n\n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "grammars/core-en/v.xsl", "content": "\n\n\n\n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "grammars/dict.xsd", "content": "\n\n\n\n \n \n Schema for dict file.\n \n A dict file is a more refined version of a morph file that groups \n word forms by their stems (and parts of speech) and lists the mappings \n to lexical categories. See the grammar build targets in build.xml \n for how a dict file can be straightforwardly transformed into the \n morph.xml and lexicon.xml files required by the run-time system.\n \n See morph.xsd for more information on the the mapping between morph \n items and lexical categories, and on the macros.\n \n \n\n \n \n \n \n \n \n The root element, containing a list of dictionary entries and macros.\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n A dictionary entry includes any number of 'member-of' and 'stem-for' mapping \n elements, followed by any number of 'word' elements for different word forms.\n The stem and part of speech (pos) attributes are required.\n If there is just a single word form, it can be specified using the 'word' \n attribute, with the stem as the default.\n The predicate may also be given with the 'pred' attribute, if it \n differs from the stem.\n Any macros that apply to all of the entry's word forms may also be listed.\n See morph.xsd for info on the (semantic) class and coart(iculation) attributes.\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n Specifies a mapping from the entry to a family of lexical categories. \n The 'family' attribute should provide the name of a category family.\n The 'pred' attribute may be used to specify a different predicate \n to use with this family.\n \n \n \n \n \n \n \n\n \n \n \n Specifies a mapping from the entry to a particular entry in a family of \n lexical categories.\n The 'family' and 'entry' attributes should provide the names of a \n category family and entry therein.\n \n \n \n \n \n \n \n\n\n \n \n \n A specific word form, given by the 'form' attribute.\n Any macros that are particular to the word form may be listed.\n Any lexical category families or entries whose mapping should be \n blocked may also be listed.\n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/extract-morph.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/flights/build.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n\n \n\n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n builds the grammar (default)\"/>\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "grammars/flights/dict.xml", "content": "\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/flights/dict.xsl", "content": "\n\n\n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n\n\n" }, { "path": "grammars/flights/flairs.xml", "content": "\r\n\r\n\r\n\r\n\r\n Student example.\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n To make \"on Ryanair\" into a predicative adjective, so that it can serve as the \r\n propositional complement of the copula, a dummy 'has-rel' head is introduced.\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n Frequent Flyer example.\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n Business Class example.\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n Frequent Flyer variant, with preferred airline second (still a bit awkward).\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n Some extra test cases.\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/flights/grammar.xml", "content": "\n\n\n\n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/flights/lexicon-base.xsl", "content": "\n\n\n\n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n Airline\n airline\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/flights/lexicon.xml", "content": "\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/flights/morph.xml", "content": "\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/flights/nina107.xml", "content": "\r\n \r\n skipping to nina107_033 \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n TBD:\r\n nina107_040.wav\t From where?\r\n nina107_041.wav\t And from where?\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/flights/rules-base.xml", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/flights/rules.xml", "content": "\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/flights/testbed.xml", "content": "\n\n\n\n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/flights/types-extras.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/flights/types.xml", "content": "\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/flights/vera.xml", "content": "\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/grammar.xsd", "content": "\n\n\n\n \n \n Schema for grammar file.\n \n A grammar file lists the lexicon, morphology and rules files, \n and optionally lists the types file and sequences of \n XSLT transformations to use in loading/saving LFs from/to XML.\n It may also specify a custom list of features to use with \n supertag names, a custom tokenizer, and/or list semantic classes to \n replace words with for language models.\n \n \n\n \n\n \n \n \n The root element.\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n An element with a file attribute specifying a filename.\n \n \n \n \n\n \n \n \n An element with a file attribute specifying a filename for the\n lexicon file, plus an optional openlex attribute indicating\n whether lexical category assignments are complete (for openlex\n false) or incomplete (for openlex true).\n \n \n \n \n \n \n \n \n\n \n \n \n An element with a file attribute specifying a filename \n for the rules file. An optional combosfile attribute \n gives the filename for the list of observed supercat-rule \n combos. The format of the combos file is one combo per line, \n where each combo is either a supertag and a rule name, or a \n pair of supertags and a rule name, separated by single spaces.\n The optional dynamic-combos flag indicates whether the \n observed combos should be determined dynamically (using the ones \n in the combos file as a starting point, if present). By default, \n dynamic-combos is true if a combos file is given, and false if not.\n \n \n \n \n \n \n \n \n \n\n \n \n \n An element which gives the features to use in constructing \n supertag names. The feature names are given as a space-delimited list \n in the feats attribute.\n \n \n \n \n \n \n \n \n An element which gives the fully qualified class name for \n a custom tokenizer, and/or lists semantic classes to replace words \n with for language models.\n \n \n \n \n \n \n \n \n \n Specifies a sequence of XSLT transformations.\n \n \n \n \n \n \n \n\n" }, { "path": "grammars/hlds.xsd", "content": "\n\n\n\n \n \n Elements and types for hybrid logic dependency semantics (HLDS) constructs.\n \n The subset of hybrid logic terms used in HLDS is defined here.\n (The Java classes currently support a slightly larger subset.)\n To support flattening of these terms into elementary predications, the \n contents of the satisfaction operators and diamond relations (modal ops) \n are restricted, as described in detail below.\n \n Note that where a list of terms is allowed, these are implicitly conjoined; \n a conjunction operator is inserted upon loading, when necessary.\n LFs with disjunctive or optionality operators may also be specified.\n \n Nominals and variables may be given a type (or sort) listed in the \n types file, by appending the type to the name after a colon. \n Propositions whose names appear in the types file are also treated as typed.\n \n \n\n \n\n \n \n \n Typically one or more satisfaction operators, encoding the semantics \n associated with a category.\n Exclusive disjunctions (xor) are also allowed.\n \n \n \n \n \n \n \n \n \n \n \n The possible values of an LF feature, i.e., \n a nominal (atom or variable), proposition, or variable (over HLDS terms).\n \n \n \n \n \n \n \n \n \n \n \n \n A satisfaction operator, with a nominal and an arg.\n The nominal can either be a nominal atom, named by the \"nom\" attribute, \n or a nominal variable, named by the \"nomvar\" attribute.\n The nominal may be optionally marked as \"shared\" (see nominal def below).\n The arg optionally begins with a proposition, followed by any number \n of diamond relations or disjunctive/optionality operators, \n but must be non-empty.\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n A named proposition.\n \n \n \n \n \n \n \n \n \n \n A diamond relation (modal operator), with a mode and an arg.\n The \"mode\" attribute names the relation.\n The arg is either just a proposition or a variable, \n or a nominal followed optionally by a proposition and \n any number of nested diamond relations or disjunctive/optionality operators.\n In the former case, the relation encodes a semantic feature; in the latter case, it encodes \n a relation to another semantic head.\n The arg may also be a disjunctive operator containing satops.\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n A conjunction (conj), exclusive disjunction (xor) \n or optionality (opt) operator.\n Note that conjunctions are usually left implicit.\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n A nominal term, either an atom or a variable.\n The \"shared\" attribute indicates that the nominal term \n provides a reference to a node that is a shared part of\n multiple alternatives, rather than being a subordinated \n or coordinated reference.\n \n \n \n \n \n \n \n A nominal atom.\n \n \n \n \n \n \n \n A nominal variable.\n \n \n \n \n \n \n \n A variable over HLDS terms. \n \n \n \n \n \n \n \n A term with a name.\n \n \n \n \n \n \n \n \n A named term with an optional \"shared\" attribute.\n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/lexicon.xsd", "content": "\n\n\n\n \n \n Schema for lexicon file.\n \n The lexicon file contains the definitions of the lexical categories, \n whose entries are grouped into families (inspired by XTAG tree families).\n \n Families may be open or closed. With open families, all words listed \n in the morph file with the same part of speech are mapped to all entries \n in the family. With closed families, the members of the family must \n be explicitly listed.\n \n The lexicon may specify that certain features should \n be treated specially, either for licensing and instantiating \n\t\tsemantically null or \"marked\" categories in the realizer, \n\t\tor for implementing \"distributive\" behavior.\n See below for further details.\n \n \n\n \n \n \n \n \n \n The root element, which includes the list of category families and \n the declarations of the licensing and distributive attributes.\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\t\t\tLists the names of the attributes with \"distributive\" behavior, i.e. \n\t\t\tones where the feature value is supposed to show up on \n\t\t\tevery atomic category. Attributes that are distributive in \n\t\t\tthis sense can be used to prevent phrasal combinations across \n\t\t\ttheme/rheme boundaries, following Steedman's '00 LI paper.\n\t\t\t\n\t\t\tDistributive attributes receive two kinds of special processing. \n\t\t\tFirst, after instantiating a lexical category for a word, \n\t\t\tif a distributive attribute has a unique value, it is automatically \n\t\t\tspread to all of the atomic categories, thereby alleviating the \n\t\t\tneed to ensure this happens in the definition of the lexical \n\t\t\tcategories and macros.\n\t\t\tSecond, as a stopgap measure, following rule combinations, the \n\t\t\tvalue of any distributive attribute appearing on the result \n\t\t\tcategory is propagated to all argument categories, overwriting \n\t\t\tany values that may already be there. This processing allows \n\t\t\tcategories such as s[eme=phr]$\\s[eme=th]$ to be used for boundary \n\t\t\ttones, where the value of the distributive 'eme' feature is changed \n\t\t\ton the target 's' category (in the lexicon), but the value of this \n\t\t\tattribute on the $-matched categories remain unchanged until the \n\t\t\tspecial processing kicks in. This stopgap measure would not be necessary \n\t\t\tif the 'inheritsFrom' and $ capabilities could be combined, which would \n\t\t\trequire a more dynamic treatment of 'inheritsFrom' (i.e. exceptive feature \n\t\t\tpropagation, or default unification).\n \n\t\t\n \n \n \n \n\t\t\t\t A space-separated list of attribute names. \n \n \n \n\t\t\n\t\n\n\t\n \n \n \n \tLists the features to use to license and/or instantiate \n\t\t\tsemantically null or \"marked\" categories in the realizer, \n in priority order.\n\t\t\tExamples of semantically empty categories may include \n\t\t\tcase-marking prepositions or particles, infinitival 'to', \n\t\t\tor complementizers such as 'that'.\n\t\t\tExamples of categories which may be considered \"marked\" \n\t\t\tinclude inverting categories for auxiliaries which are \n\t\t\tused in questions, but not ordinary declaratives.\n\t\t\t\n\t\t\tNote that by default, the 'lex' feature is included as \n\t\t\tone used to license and instantiate semantically empty \n\t\t\tcategories, for all values, in all locations.\n\t\t\tThus, if (say) a semantically null, case marking preposition \n\t\t\tcategory has a 'lex' attribute which is instantiated with the stem \n\t\t\t(via the '[*DEFAULT*]' mechanism), then this category will be activated \n\t\t\tin the realizer iff any of the categories instantiated for the input \n\t\t\tlogical form have the same value for the 'lex' feature, thereby \n\t\t\tindicating its relevance.\n\t\t\tIf an alternative specification for the 'lex' feature \n\t\t\tis listed, the default specification is not included.\n \n\t\t\n \n \n \n \n \n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\tThe name of the licensing feature, i.e. the attribute.\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n \n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t The value of the feature. If not specified, all attribute values \n\t\t\t\t\t\t\t\tare treated the same for licensing and instantiation purposes.\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n \n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t A list of other values of this feature whose presence may also license \n categories with this feature value.\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\tA flag indicating whether semantically null categories with the \n\t\t\t\t\t\t\t\tlicensing feature need to be licensed.\n\t\t\t\t\t\t\t\tDefaults to true.\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t A flag indicating whether initial categories with the \n\t\t\t\t\t\t\t licensing feature are marked and need to be licensed.\n\t\t\t\t\t\t\t\tDefaults to false.\n\t\t\t\t\t\t\t\tIf set to true, the defaults for the license-empty-cats and instantiate flags \n\t\t\t\t\t\t\t\tchange to false, and the default for the location changes to \n\t\t\t\t\t\t\t\ttarget-only.\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t A flag indicating whether semantically empty categories with the \n\t\t\t\t\t\t\t licensing feature should be instantiated.\n Defaults to true.\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\tThe location of the licensing feature on the category to be licensed.\n\t\t\t\t\t\t\t\tIf target-only, then the value must appear only on the target category. \n\t\t\t\t\t\t\t\tIf args-only, then the value must appear only on the argument categories.\n\t\t\t\t\t\t\t\tDefaults to both.\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n \n\t\t\t\t\n \n\t\t\n\t\n\n\t\n \n \n \n \tOptionally specifies the order in which to sort relations in \n the logical forms.\n \n The default sort order is as follows, where \"*\" indicates all \n other relations (sorted alphabetically):\n \n \"BoundVar\", \"PairedWith\", \n \"Restr\", \"Body\", \"Scope\", \n \"*\", \n \"GenRel\", \"Coord\", \"Append\"\n\n The default order may be overridden using the 'order' attribute, \n which should contain a space-separated list of relation names, \n with \"*\" indicating all other relations.\n \n\t\t\n \n \n \n \n\t\t\t\t A space-separated list of relation names. \n \n \n \n\t\t\n\t\n\n\t\n \n \n \n A category family, which groups lexical category entries, and optionally \n includes a list of the family members, if the family is closed.\n \n Family members are listed by their stems. A 'member' element may also have a\n predicate ('pred') explicitly specified, if it differs from the stem.\n The predicate is used to fill in any propositions in the semantics with \n the reserved name '[*DEFAULT*]'.\n \n The name of the family must be unique.\n \n The part of speech ('pos') is used in the mapping between morphological \n items and families, as described earlier.\n \n A family may also declare a relation used for indexing purposes (the 'indexRel') \n in the realizer, i.e. a relation used to lookup words based on relational \n or featural elementary predications in the input logical form;\n see Mike and Jason's paper on the realizer for discussion, at \n http://www.iccs.informatics.ed.ac.uk/~mwhite/White-Baldridge-ENLG-2003-to-appear.pdf.\n Families with semantically null entries are indicated using the reserved value '*NoSem*'.\n Relations may also be declared at the level of entries.\n \n NB: It would be nicer if the system could figure out which entries were semantically null; \n what makes this non-trivial is that macros can introduce additional semantic \n semantic predications. At present, this is only discovered when lexical \n categories are instantiated for specific words.\n \n As an alternative to the indexRel, a family may declare a relation (semantic feature) \n to use to signal a coarticulation, via the 'coartRel' attribute. For example, the \n coart rel \"kon\" may be used to signal a contrast feature that triggers the addition \n of a pitch accent. Coarticulation are handled as lexical modifiers, and as such \n should be given modifier categories of the form X|X.\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n An entry for a family, which contains the definition of a lexical category, \n and must have a name which is unique within the family.\n \n If there is only stem that is applicable to this entry, it may be specified \n on this element.\n \n A relation to use for indexing purposes in realization may be given \n for this entry (if different from other entries in the family).\n \n It is possible to temporarily disable an entry by setting the 'active' \n attribute to false (defaults to true).\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/mini-basque/build.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n\n \n\n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n builds the grammar (default)\"/>\n \n builds the skeleton file\"/>\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-basque/dict.xml", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n\n" }, { "path": "grammars/mini-basque/grammar.xml", "content": "\n\n\n\n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-basque/lexicon-base.xml", "content": "\n\n \n\n\n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\t\t\n \n \n \n \n \n \n\t\n\n\t \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\t\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\t \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\t \n \n \n \n \n \n \n \n \n \n\n\n\n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n\n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\t\n\t\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\t\n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n\t\t\n\t\t\n\t\t \n\t\t \n \n \n \n \n\t\t \n \n\t\t\n \n \n \n \n \n \n \n \n \n\t\t\n\t\t\n\t\t \n\t\t \n \n \n \n \n\t\t \n\t\t \n \n\t\t\n \n \n \n \n \n \n \n \n \n\t\t\n\t\t\n\t\t \n\t\t \n \n \n \n \n\t\t \n\t\t \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/mini-basque/lexicon.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-basque/morph.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-basque/parameters.xml", "content": "\n\n\n\n \n\t\n \n\n \n\t\t\n \n \n \n \n \n \n \n \n\n \n\t\t\n \n \n \n \n \n \n \n\t\n \n\n\n" }, { "path": "grammars/mini-basque/preset-families.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-basque/rules.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-basque/testbed.out", "content": "Loading grammar from URL: file:/home/bozsahin/openccg/grammars/mini-basque/grammar.xml\n\nParse\tRealize\tString\n-----\t-------\t------\nok\t-\tjoan\nok\t-\tikusi\nok\t-\tzuen\nok\t-\tzen\nok\t-\tklase ra\nok\t-\teskola n\nok\t-\tama aitak ikusi zuen\nok\t-\tama joan\nok\t-\t*ama k joan\nok\t-\t*ama k joan du\n(2)\t-\tama k seme a ikusi du\n(2)\t-\tseme a ama k ikusi du\n(2)\t-\tnik liburu a emakume ari eman nahi dut\nFAILED\t-\t*nik liburu a aita k eman nahi dut\nFAILED\t-\t*nik aita k liburu a eman nahi dut\nok\t-\tama k seme a emakume ari eman dio\nok\t-\tseme a ama k emakume ari eman dio\nok\t-\temakume ari seme a ama k eman dio\nok\t-\tnik joan nahi dut\n(2)\t-\tnik kafe a egin nahi dut\nok\t-\t*nik kafe a egin nahi du\nok\t-\tnik kafe a egin nahi\nFAILED\t-\t*nik aita k ikusi nahi dut\nok\t-\tnik ikusi nahi dud an gizon a joan da\nok\t-\tgizon a ikusi nahi dud an seme a joan da\nok\t-\temakume ari liburu a eman dio n gizon a joan du\nFAILED\t-\temakume ari liburu a eman dio n gizon a nik ikusi dut\nok\t-\t*emakume ari liburu a eman dio n gizon ak ni ikusi dut\nok\t-\temakume ari liburu a eman dio n gizon ak ni ikusi\n(2)\t-\tgizon ak emakume ari eman dio n liburu a aita k ikusi du\n(3)\t-\tgizon ak liburu a eman dio n emakume a\nok\t-\tgizon ak liburu a eman dio n emakume a joan du\n(3)\t-\taita k seme a eskola n utzi zuen eta klase ra joan zen\n(3)\t-\tseme a eskola n utzi zuen eta klase ra joan zen\n(3)\t-\tseme a eskola n utzi eta klase ra joan zen\nok\t-\t*seme a eskola n utzi eta klase ra joan zuen\nok\t-\tama joan zen eta aita k ikusi zuen\nok\t-\taita k ikusi zuen eta klase ra joan zen\nok\t-\t*ama k joan eta aita ikusi zuen\nok\t-\taitak ama ikusi zuen eta joan zen\nok\t-\tama aitak ikusi zuen eta joan zen\nok\t-\tjoan zen eta ama ikusi zuen\nok\t-\tjoan zen eta aitak ikusi zuen\nFAILED\t-\tsagarrak emakume ak egosten ditu eta gizonak jaten ditu\nok\t-\tsagarrak emakume ak egosten ditu eta usteltzen dira\nok\t-\tsagarrak usteltzen dira eta emakume ak egosten ditu\n" }, { "path": "grammars/mini-basque/testbed.xml", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "grammars/mini-basque/types.xml", "content": "\n\n\n\n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-dyirbal/build.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n\n \n\n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n builds the grammar (default)\"/>\n \n builds the skeleton file\"/>\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-dyirbal/dict.xml", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n\n\n \n\n\n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n\n" }, { "path": "grammars/mini-dyirbal/grammar.xml", "content": "\n\n\n\n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-dyirbal/lexicon-base.xml", "content": "\n\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/mini-dyirbal/lexicon.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-dyirbal/morph.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-dyirbal/parameters.xml", "content": "\n\n\n\n \n\t\n \n\n \n\t\t\n \n \n \n \n \n \n \n \n\n \n\t\t\n \n \n \n \n \n \n \n\t\n \n\n\n" }, { "path": "grammars/mini-dyirbal/preset-families.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-dyirbal/rules.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-dyirbal/testbed.out", "content": "Loading grammar from URL: file:/home/bozsahin/openccg/grammars/mini-dyirbal/grammar.xml\n\nParse\tRealize\tString\n-----\t-------\t------\nok\t-\tbayi yara walngarra nabaygu\nok\t-\tbayi yara walngarra bural naygu bagun yibi gu\nok\t-\tbayi yara walngarra bangun yibi ngu burali\nok\t-\tyabu numa ngu gigan banagaygu\nok\t-\tnaja bayi yara gigan gubi ngu mawali\nok\t-\t*naja bayi yara gigan bayi gubi mawali\nok\t-\tbayi yara miyanda nu yanu\nok\t-\tbalan yibi bangul yara ngu miyanda nu ru buran\nok\t-\tbayi yara jilwal na nu bagun guda gu yanu\nok\t-\tbayi burrbula bangul gubi ngu baran , bajigu\nFAILED\t-\t*bangul gubi ngu bayi burrbula baran , bajigu\nok\t-\tbayi yara yanu , bangun yibi ngu buran\nok\t-\tnyurra nanana buran , banaganyu\n" }, { "path": "grammars/mini-dyirbal/testbed.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-dyirbal/types.xml", "content": "\n\n\n\n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-english/build.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n\n \n\n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n builds the grammar (default)\"/>\n \n builds the skeleton file\"/>\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-english/dict.xml", "content": "\n\n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/mini-english/grammar.xml", "content": "\n\n\n\n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-english/lexicon-base.xml", "content": "\n\n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-english/lexicon.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-english/morph.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-english/parameters.xml", "content": "\n\n\n\n \n\t\n \n\n \n\t\t\n \n \n \n \n \n \n \n \n\n \n\t\t\n \n \n \n \n \n \n \n\t\n \n\n\n" }, { "path": "grammars/mini-english/preset-families.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-english/rules.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-english/testbed.out", "content": "Loading grammar from URL: file:/home/bozsahin/openccg/grammars/mini-english/grammar.xml\n\nParse\tRealize\tString\n-----\t-------\t------\nok\t-\tJohn promised him to exercise\nok\t-\tJohn persuaded him to read the book\nok\t-\t*John promised him Sue see\nok\t-\tJohn gave Mary and read a book\nok\t-\tthe doctor punched the patient and fell-down\n" }, { "path": "grammars/mini-english/testbed.xml", "content": "\n\n \n \n \n \n \n\n" }, { "path": "grammars/mini-english/types.xml", "content": "\n\n\n\n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-inuit/build.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n\n \n\n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n builds the grammar (default)\"/>\n \n builds the skeleton file\"/>\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-inuit/dict.xml", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n\n\n" }, { "path": "grammars/mini-inuit/grammar.xml", "content": "\n\n\n\n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-inuit/lexicon-base.xml", "content": "\n\n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/mini-inuit/lexicon.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-inuit/morph.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-inuit/parameters.xml", "content": "\n\n\n\n \n\t\n \n\n \n\t\t\n \n \n \n \n \n \n \n \n\n \n\t\t\n \n \n \n \n \n \n \n\t\n \n\n\n" }, { "path": "grammars/mini-inuit/preset-families.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-inuit/rules.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-inuit/testbed.out", "content": "Loading grammar from URL: file:/home/bozsahin/openccg/grammars/mini-inuit/grammar.xml\n\nParse\tRealize\tString\n-----\t-------\t------\nok\t-\tmiiqqat Juuna ikiussallugu niriursuipput\nok\t-\tmiiqqat qitissallutik niriursuipput\nok\t-\tnanuq Piita p tugu taa\nok\t-\tmiiraq kamat tuq\n" }, { "path": "grammars/mini-inuit/testbed.xml", "content": "\n\n \n \n \n \n\n" }, { "path": "grammars/mini-inuit/types.xml", "content": "\n\n\n\n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-nezperce/build.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n\n \n\n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n builds the grammar (default)\"/>\n \n builds the skeleton file\"/>\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-nezperce/parameters.xml", "content": "\n\n\n\n \n\t\n \n\n \n\t\t\n \n \n \n \n \n \n \n \n\n \n\t\t\n \n \n \n \n \n \n \n\t\n \n\n\n" }, { "path": "grammars/mini-nezperce/preset-families.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-nezperce/types.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-tagalog/build.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n\n \n\n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n builds the grammar (default)\"/>\n \n builds the skeleton file\"/>\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-tagalog/dict.xml", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n\n\t \n\n\n" }, { "path": "grammars/mini-tagalog/grammar.xml", "content": "\n\n\n\n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-tagalog/lexicon-base.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/mini-tagalog/lexicon.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-tagalog/morph.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-tagalog/parameters.xml", "content": "\n\n\n\n \n\t\n \n\n \n\t\t\n \n \n \n \n \n \n \n \n\n \n\t\t\n \n \n \n \n \n \n \n\t\n \n\n\n" }, { "path": "grammars/mini-tagalog/preset-families.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-tagalog/rules.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/mini-tagalog/testbed.out", "content": "Loading grammar from URL: file:/home/bozsahin/openccg/grammars/mini-tagalog/grammar.xml\n\nParse\tRealize\tString\n-----\t-------\t------\n(3)\t-\tiniwasan ko ng tumingin kay Lorna\nok\t-\tnagatubili siya ng humiram ng pera sa bangko\nok\t-\tnagatubili siya ng hiramin ang pera sa bangko\n(2)\t-\tnagatubili siya ng hiraman ng pera ang bangko\nok\t-\tbinawalan ko si Maria ng awitin ang pera\n(3)\t-\tang babae ng bumili ng baro\n(3)\t-\tang baro ng binili ng babae\n(2)\t-\thuhugasan ko at pupunasan mo ang mgapinggan\n(2)\t-\tniluto ang pagkain at hinugasan ang mgapinggan ni Josie\n(4)\t-\tnanghuhuli ang ama at nagtitinda ang ina ng isda\n(2)\t-\tnagbigay ng regalo si Maria at nagpadala ng liham ang mgabat\n\t\ta kay Juan\n(5)\t-\tpumunta sa tindahan at bumili ang kapatid ng bigas\n" }, { "path": "grammars/mini-tagalog/testbed.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-tagalog/types.xml", "content": "\n\n\n\n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-turkish/build.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n\n \n\n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n builds the grammar (default)\"/>\n \n builds the skeleton file\"/>\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-turkish/dict.xml", "content": "\n\n\n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\n \n\n\t\t \n \n\n\n \n \n\n \n\n\n" }, { "path": "grammars/mini-turkish/grammar.xml", "content": "\n\n\n\n \n \n \n \n\n\n\n" }, { "path": "grammars/mini-turkish/lexicon-base.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n\n \n \n \n\t \n\t \n \n \n \n \n\t \n\t\n\t\n\t\n\t \n\t\n\t\n\t\n\t\n\t \n\t \n \n \n \n \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n \n\t \n\t \n\t \n\t \n\t\n\t\n\t\n\t \n\t \n\t \n \n \n \n \n\t \n\t \n\t \n\t \t\n\t \n\t \n\t \n\t \n \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n \n\t\n\t\n\t\n\t \n\t\n\t\n\t \n\t \n\t \n\t \n\t \n\t\t \n\t\t\n\t \n\t \n\t\n\n\n\n \n \n \n\t \n\t \n \n \n \n \n\t \n\t\n\t\n\t\n\t \n\t\n\t\n\t\n\t\n\t \n\t \n \n \n \n \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t\n\t\n\t\n\t \n\t \n\t \n \n \n \n \n\t \n\t \n\t \n\t \t\n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n \n\t\n\t\n\t\n\t \n\t\n \n\t \n\t \n\t \n\t \n\t \n\t\t \n\t\t\n\t \n\t \n\t\n\n\n\n \n \n \n\t \n\t \n \n \n \n \n\t \n\t\n\t\n\t\n\t \n\t\n\t\n\t\n\t\n\t \n\t \n \n \n \n \n\t \n\t \n\t \n\t \n\t \n\t \n\t\n\t\n\t\n\t \n\t \n\t \n \n \n \n \n\t \n\t \n\t \n\t \t\n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n \n\t\n\t\n\t\n\t \n\t\n \n\t \n\t \n\t \n\t \n\t \n\t\t \n\t\t\n\t \n\t \n\t\n \n \n\n \n \n \n\t \n\t \n \n \n \n \n\t \n\t\n\t\n\t\n\t \n\t\n\t\n\t\n\t\n\t \n\t \n \n \n \n \n\t \n\t \n\t \n\t \n\t \n\t \n\t\n\t\n\t\n\t \n\t \n\t \n \n \n \n \n\t \n\t \n\t \n\t \t\n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n\t \n \n\t\n\t\n\t\n\t \n\t\n \n\t \n\t \n\t \n\t \n\t \n\t\t \n\t\t\n\t \n\t \n\t\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/mini-turkish/lexicon.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-turkish/morph.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-turkish/parameters.xml", "content": "\n\n\n\n \n\t\n \n\n \n\t\t\n \n \n \n \n \n \n \n \n\n \n\t\t\n \n \n \n \n \n \n \n\t\n \n\n\n" }, { "path": "grammars/mini-turkish/preset-families.xml", "content": "\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-turkish/rules.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\t\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\t \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-turkish/testbed.out", "content": "Loading grammar from URL: file:/home/bozsahin/openccg/grammars/mini-turkish/grammar.xml\n\nParse\tRealize\tString\n-----\t-------\t------\nok\t-\tcocuk kitab i oku ma ya calisti\nok\t-\t*cocuk kitab i oku ma yi calisti\nok\t-\tcocuk kitab i oku mak istedi\nok\t-\tcocuk kitab i oku ma yi istedi\nok\t-\t*cocuk kitab i oku ma ya istedi\nok\t-\t*cocuk adam kitab i ver me yi istedi\nok\t-\tadam cocug u kitab i oku ma ya zorladi\nok\t-\t*adam cocug u kitab i oku ma yi zorladi\nok\t-\tcocug u adam kitab i oku ma ya zorladi\nok\t-\tadam cocug u kitab i kadin a ver me ye zorladi\nok\t-\tadam cocug u kadin a kitab i ver me ye zorladi\nok\t-\tcocug u adam kadin a kitab i ver me ye zorladi\nok\t-\t*adam cocug u kadin kitab i ver me ye zorladi\nok\t-\t*adam cocug a kadin a kitab i ver me ye zorladi\nok\t-\tkitab i oku yan adam uyudu\nok\t-\tcocuk kitab i oku yan adam i gordu\nok\t-\tadam in gor dugu cocuk uyudu\nok\t-\tcocuk adam in gor dugu kitab i okudu\nok\t-\tcocug un kitab i ver digi adam uyudu\nok\t-\tcocug un adam a ver digi kitap dustu\nok\t-\t*adam cocug un adam a ver en kadin i gordu\nok\t-\t*adam adam a kitab i ver digi kadin i gordu\nok\t-\tadam cocug a carpti ve dustu\nok\t-\tcocug a adam carpti ve dustu\nok\t-\tcocug a carpti ve adam dustu\nok\t-\tkitab i adam cocug a verdi ama kadin almisti\nok\t-\tcocug a kitab i adam verdi ama kadin almisti\nok\t-\tdergi ye adam bakti ama cocuk okudu\n" }, { "path": "grammars/mini-turkish/testbed.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/mini-turkish/types.xml", "content": "\n\n\n\n \n \n \n \n \n \n\n" }, { "path": "grammars/morph.xsd", "content": "\n\n\n\n \n \n Schema for morph file.\n \n A morph file lists all the known word forms (morph items) together with \n their stems, parts of speech, semantic classes, associated macros, and \n excluded lexical categories.\n \n The part of speech is used to help determine the mapping between morph \n items and lexical categories. A morph item is automatically associated \n with all open families with matching parts of speech. With closed \n families, the morph item's stem must be listed as a family member, \n and the parts of speech must match. It is also possible to exclude \n certain lexical category entries or families, by including the entry \n name, qualified entry name or family name in the morph item's list of \n excluded lexical categories.\n \n NB: A limitation of the current implementation is that the mapping between \n morph items and open families can only be done with the stem as the predicate; \n to use different predicates, closed families must be used.\n\n Semantic classes may also be given, for n-gram ranking purposes, and for \n restricting the unification of nominals to compatible types, specified \n in the types file. When a category is instantiated, the semantic class \n is assigned to the nominal var(s) for the proposition with the reserved name \n '[*DEFAULT*]'. The types of all nominal vars are then propagated to all \n other nominal vars with the same name, throughout the category.\n\n The macros are used to add features or semantic predications to a lexical \n category that depend on the particular morph item (e.g. tense, number, case, etc.).\n \n \n\n \n \n \n \n \n The root element, containing a list of entries (morph items) and macros.\n Each macro must have a unique name.\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n A morph item, with its word form, stem, part of speech, and optional semantic class, \n associated macros, and excluded lexical categories. The stem defaults to word form.\n At run time, when the grammar is loaded, a cross-reference check is \n performed on the names in the lists of associated macros and excluded \n lexical categories.\n \n The coart flag indicates that this entry is a coarticulation, eg a pitch accent, \n gesture, or other word-associated element. With coarticulations, the word form \n should be one or more attribute-value pairs, with attributes and values separated \n by hyphens, and multiple pairs separated by colons. When multiple attribute-value \n pairs are given, only the first one is used for indexing purposes. An identifying \n value for the stem should also be given, eg *accent* for a pitch accent.\n \n NB: Lexical attributes supplied by coarticulations must be done so uniformly; that \n is, they cannot appear already on some lexical items. Also, when there are \n multipled, independent coarticulations, they must supply disjoint sets of lexical \n attributes, and their categories must be capable of applying in any order.\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n A named macro, with a set of features and/or semantic predications to add to\n a lexical category.\n \n The features are given by feature structure declarations, where the id is used \n to indicate where the features are to be added (i.e., to the feature structure \n of which atomic category).\n \n The semantic predications are given in an 'lf' element; at run time, these HLDS \n predications are flattened and added to any predications already present in \n the lexical category.\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n Macro names begin with @ (for historical reasons) and are followed \n by at least one non-whitespace character.\n \n \n \n \n \n \n\n \n \n \n\n\n\n" }, { "path": "grammars/parameters.xsd", "content": "\n\n\n\n\n\n \n \n \n Schema for the parameters.xml file.\n\n The parameters file contains definitions of language type, its basic \n word order and category for basic intransitive verbs, and the word order\n skeleton for the basic transitive verbs. \n\n iv and tv are pre-CCG categories. They refer to grammatical roles S,A,P,\n described by XML tags s-argument, a-argument and p-argument.\n They are mapped to CCG categories by parametric-lexicon.xsl.\n\n Word order can be specified as ordered/unordered list of arguments. \n Curried arguments can be described by enclosing them in arg tag, \n and set of arguments as setarg tag (the latter is also a valid category \n in openCCG, cf. categories.xsd and lexicon.xsd. But arg is not).\n\n For further information and how to start up lexicon design from a skeletal\n lexicon, see Bozsahin and Steedman (2003) (draft), and the comments\n generated in the file preset-families.xml by paramatric-lexicon.xsl.\n\n As usual, the best way to learn is to look at an example file. Parameter\n files are provided for Basque, Dyirbal, English, Inuit, Tagalog and \n Turkish (in their respective directories). \n\n \n \n\n \n \n \n p-argument and a-argument must not carry case; they are set from iv \n specs. \n \n \n \n \n \n\n \n \n \n s-argument must carry case; tv category and its grammatical relations\n are set up from this case and language type.\n \n \n \n \n \n \n\n \n \n \n\n \n \n \n The direction of the slash.\n The direction can be forward (/), backward (\\) or both (|).\n \n \n \n \n \n \n \n \n\n \n \n \n What kind of subject is missing from the infinitive (parameter)\n \n \n \n \n \n \n \n \n\n\n \n \n \n The parameters must include infinitival, iv and tv specifications. The latter\n\t\ttwo must\n have a predicate specification. iv has s-argument,\n tv has a-argument and p-argument, wrapped around either arg or setarg\n tags. They can appear in any order (both iv and tv specs, and argument\n and predicate specs).\n\t\tControllee must specify subject type as semantic or syntactic.\n \n \n\n \n\t\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/parametric-lexicon.xsl", "content": "\n\n\n\n\n\n \n\n \n\n \n\n \n\n \n \n \n \n \n \n \n \n \n\n \n \n NB: Word order and directionality of embedded clause's \n|(S|NP) type is an 'educated guess' from the syntactic type of TV. \nChange them accordingly.\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n NB: If infinitive type is syntactic subject (which is the \n\t\t default), there is a built-in type in types.xml file called\n\t\t \"subject-case\". Include in this type all the subjects that\n\t\t can be controlled e.g. nominative subjects only (as in German), \n\t\t dative and nominative subjects (as in Malayalam) etc. In this\n\t\t case, the controllee is not semantically restricted.\n\t\t \n\t\t If infinitive type is semantic subject, you need a larger\n\t\t fragment of Hybrid Logic than HLDS uses to implement identity\n\t\t of two event variables e.g.\n\t\t \n\t\t @_e(Arg1 a) ^ @_e(Arg1 b) --> @_a(b)\n\t\t \n\t\t where a is the event variable for controller verb, and\n\t\t b is the event variable for the controlled verb, and Arg1 is the\n\t\t modality for primary arguments (1s).\n\t\t \n\t\t This constraint is formulable in HL but HLDS does not cover that\n\t\t fragment (yet). When it does, the lf tag should just stick in that\n\t\t constraint.\n\t\t \n\t\t Currently, OpenCCG can generate an LF with TWO non-identical Arg1 modalities\n\t\t in the same event structure, therefore some illicit examples would\n\t\t go through.\n\t\t\n \n \n \n \n \n \n NB: Index is same as Arg1's \n \n \n \n \n \n \n \n \n \n \n \n NB: Index is same as Arg1's \n \n \n \n \n \n \n \n\n \n \n NB: Word order and directionality of embedded clause's |(S|NP) \ntype is an 'educated guess' from the syntactic type of TV. Change them\naccordingly \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n NB: If infinitive type is syntactic subject (which is the \n\t\t default), there is a built-in type in types.xml file called\n\t\t \"subject-case\". Include in this type all the subjects that\n\t\t can be controlled e.g. nominative subjects only (as in German), \n\t\t dative and nominative subjects (as in Malayalam) etc. In this\n\t\t case, the controllee is not semantically restricted.\n\t\t \n\t\t If infinitive type is semantic subject, you need a larger\n\t\t fragment of Hybrid Logic than HLDS uses to implement identity\n\t\t of two event variables e.g.\n\t\t \n\t\t @_e(Arg1 a) ^ @_e(Arg1 b) --> @_a(b)\n\t\t \n\t\t where a is the event variable for controller verb, and\n\t\t b is the event variable for the controlled verb, and Arg1 is the\n\t\t modality for primary arguments (1s).\n\t\t \n\t\t This constraint is formulable in HL but HLDS does not cover that\n\t\t fragment (yet). When it does, the lf tag should just stick in that\n\t\t constraint.\n\t\t \n\t\t Currently, OpenCCG can generate an LF with TWO non-identical Arg1 modalities\n\t\t in the same event structure, therefore some illicit examples would\n\t\t go through.\n\t\t\n \n \n \n \n \n \n NB: Index is same as Arg2's \n \n \n \n \n \n \n \n \n \n \n \n NB: Index is same as Arg2's \n \n \n \n \n \n \n \n\n \n\n \n\n \n - This file is generated by parametric-lexicon.xsl to set up\n accusativity/ergativity parameter for IV and TV primary families\n and control primary families.\n\n NB: pre-CCG categories of parameters.xml are mapped to\n CCG categories in this file. From now on, it's all CCG\n\n Suggestions to start-up lexicon development:\n\n 1) Copy this file to lexicon-base.xml to avoid losing your changes\n to it (remember, this file is auto-generated at the start)\n 2) Edit lexicon-base.xml to modify the preset families and to add your \n own families as needed (merging the entries of same family is\n left to you)\n 3) Use the ccg-build facility of openCCG, which uses lexicon-base to build\n the lexicon.xml, morph.xml and rules.xml files needed by the system.\n \n\n \n\n *** Families derived from language parameters ***\n\n Includes primary entries for IV (unerg and unacc), basic TV, TV-control1, \n TV-control2, IV-control1\n \n \n \n subject-case, s-case, p-case and a-case are value types that set up ergative-\n accusative mapping and surface cases of these argumnents (cf. types.xml file).\n Actual case values for them (e.g. nom for a-case in accusative languages,\n erg for a-case in ergative languages) are defined in types.xml file.\n You can of course refer to actual values since they are types, but if a\n construction is related to GR mapping (ERG or ACC), it's better to use\n `subject-case' (which covers a-case and s-case in ACC; p-case and s-case in ERG).\n Types.xml sets these up from parameter specification.\n \n \n\n \n\n \n \n \n\n\n \n \n \n\n \n\n *** End of derived families ***\n \n Add new families here, and merge the new entries for preset\n families as needed (e.g., you may add an entry to TV family\n for pro-dropping the subject etc.)\n \n\n\n \n\n \n\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n NB: Argument is the PAS of the embedded S|NP (cf. E2 above)\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n NB: Argument is the PAS of the embedded S|NP (cf. E2 above)\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n NB: Argument is the PAS of the embedded S|NP (cf. E2 above)\n\n \n \n \n \n \n \n \n\n \n \n\n \n \n \n \n \n \n\n \n \n\n \n \n \n \n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/parametric-types.xsl", "content": "\n\n\n\n\n\n \n \n \n\n \n \n\t \n\n \n\n \n \n\t \n - This file is generated by parametric-types.xsl from parameter specs\n\t (parameters.xml) to set up types.xml file for development.\n \n - If the language in question has eg. quirky subjects etc., add their \n\t values under appropriate types.\n\t \n\t \n\t \n\t subject-case, s-case, p-case and a-case are pre-defined types that are\n\t used in the automatically generated initial lexicon (preset-families\n\t .xml). We recommend that you dont change them. They handle\n\t accusative/ergative mapping among other things (cf. types.xml initial\n\t hierarchy).\n\t \n\t \n\n \n\t\n \n \n \n\n \n \n \n \n \n \n \n \n\t\n \n\n \n\n \n \n \n\n\n \n \n \n \n\n \n\n \n \n \n\n \n \n \n \n \n\n \n\n\n \n \n \n\n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n\n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\t\t Warning: Please substitute case names as children\n\t\tof a-case and p-case since these cannot be predicted parametrically\n\t\tfor an unaligned language (cf. unknown and unknown2 values)\n \n \n \n \n\n\n" }, { "path": "grammars/raise-nodes.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n Next\r\n \r\n \r\n\r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/routes/build.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n\n \n\n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n builds the grammar (default)\"/>\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "grammars/routes/dict.xml", "content": "\n\n\n\n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/routes/dlf_test.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n\t \n \n \n \n \n \n \n\t \n\t \n\t \n\t \n\t\t\n\t \n\t \n\t \n \n \n \n \n \n \n \n \n \n \n \n \n \n\t \n \n \n \n \n \n \n from mid-Cambridge , drive west on Cambridge_Street for about 1 mile .\n from mid-Cambridge , drive on Cambridge_Street toward Harvard_Yard for a mile .\n drive west on Cambridge_Street for a mile .\n drive on Cambridge_Street toward Harvard_Yard for about 1 mile .\n\n" }, { "path": "grammars/routes/grammar.xml", "content": "\n\n\n\n\n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n\n\n" }, { "path": "grammars/routes/lexicon-base.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\t \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n" }, { "path": "grammars/routes/lexicon.xml", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/routes/morph.xml", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/routes/rules.xml", "content": "\n\n\n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n" }, { "path": "grammars/routes/testbed.xml", "content": "\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/routes/types.xml", "content": "\r\n\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\t \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n\r\n\r\n" }, { "path": "grammars/rules.xsd", "content": "\n\n\n\n \n \n Schema for combinatory rules file.\n \n A rules file specifies the combinatory rules to use.\n These rules include the (potentially) universal set of application, \n composition, type raising and substitution rules; they are only \n \"potentially\" universal because for efficiency one might want to \n use a restricted subset of the rules, and because the type raising \n rules are defined to work with specific, configurable argument and \n result types.\n \n The rules file may also specify unary type changing rules, which \n may be thought of as signs for zero morphemes, and thus conceptually \n belonging to the lexicon rather than to the (potentially) universal \n rule set. They are defined in the rules file since, algorithmically, \n they are handled at the same time as the other combinatory rules.\n \n \n\n \n \n \n \n \n \n The root element.\n \n \n \n \n \n\n \n \n \n Application, e.g. X/Y Y => X.\n \n \n \n \n \n \n\n \n \n \n Composition, e.g. X/Y Y/Z => X/Z.\n \n \n \n \n \n \n \n\n \n\n \n \n \n Substitution, e.g. X/Y/Z Y/Z => X/Z.\n \n \n \n \n \n \n \n\n \n\n \n \n \n \n \n\n \n \n \n \n Specifies the direction of the rule.\n With forward combination, the functor appears on the left; \n with backward combination, it appears on the right.\n \n \n \n \n \n \n \n \n \n \n \n \n \n Specifies whether the rule is harmonic (true) or permutative (false).\n \n \n \n\n \n \n \n \n Type raising, e.g. np => s/(s\\np).\n The default category for the arg is \"np\"; \n the default for the result is \"s\".\n \n \n \n \n \n \n \n \n \n \n Specifies whether a coindexed dollar variable is included on \n the argument and result categories.\n \n \n \n \n\n\n \n \n \n Unary type changing rule, e.g. np => s/(s/np) for topicalization.\n The arg and result categories are required, as is the name attribute.\n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/simplify-lists.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/tiny/build.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n\n \n\n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n builds the grammar (default)\"/>\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\t\n\t\n \n \n\t\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\n\t\n\n\n\n" }, { "path": "grammars/tiny/grammar.xml", "content": "\n\n\n\n\n \n \n \n \n\n\n" }, { "path": "grammars/tiny/lexicon.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/tiny/morph.xml", "content": "\n\n\n\n\n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n\n\n \n \n\n\n \n \n\n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/tiny/rules.xml", "content": "\n\n\n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n \n \n \n\n" }, { "path": "grammars/tiny/testbed.xml", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/tiny/types.xml", "content": "\r\n\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n" }, { "path": "grammars/tokens.xsd", "content": "\n\n\n\n \n \n Definitions for tokens.\n \n \n\n \n \n \n A 'token' as defined here is a string with no white space, for easy parsing, \n but otherwise more flexible than a NMTOKEN.\n \n \n \n \n \n \n\n \n \n \n\n \n \n \n Feature variables with optional type.\n \n \n \n \n \n \n\n\n\n" }, { "path": "grammars/treeify-lists.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/types.xsd", "content": "\n\n\n \n \n Schema for type hierarchy file.\n\n A type hierarchy file specifies the hierarchy of simple types which\n constitute the domain of syntactic feature values.\n \n \n\n \n\n \n \n \n The root element, containing a list of type entries.\n Each type must have a unique name.\n \n \n \n \n \n \n \n A simple type, with its name and immediate parents in the hierarchy.\n Parents are specified as a space-separated list of type names.\n Parents attribute is optional and defaults to the built-in type \"top\".\n Only the immediate parents should be specified. \n The whole hierarchy is constructed during the initial grammar loading process.\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/worldcup/add-chunks.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n\r\n\r\n" }, { "path": "grammars/worldcup/build.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n\n \n\n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n builds the grammar (default)\"/>\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "grammars/worldcup/dict.xml", "content": "\n\n\n\n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n\n" }, { "path": "grammars/worldcup/grammar.xml", "content": "\n\n\n\n \n \n \n \n\n \n \n \n \n\n \n \n \n \n\n\n\n" }, { "path": "grammars/worldcup/lexicon-base.xsl", "content": "\n\n\n\n \n \n\n \n \n \n \n\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n\n\n\n\n" }, { "path": "grammars/worldcup/lexicon.xml", "content": "\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/worldcup/morph.xml", "content": "\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n" }, { "path": "grammars/worldcup/raise-nodes.xsl", "content": "\r\n\r\n\r\n\r\n \r\n \r\n\r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n Restr\r\n \r\n \r\n\r\n\r\n\r\n" }, { "path": "grammars/worldcup/rules.xml", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "grammars/worldcup/testbed.xml", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "lib/ASL", "content": "/*\n * ============================================================================\n * The Apache Software License, Version 1.1\n * ============================================================================\n * \n * Copyright (C) 1999 The Apache Software Foundation. All rights reserved.\n * \n * Redistribution and use in source and binary forms, with or without modifica-\n * tion, are permitted provided that the following conditions are met:\n * \n * 1. Redistributions of source code must retain the above copyright notice,\n * this list of conditions and the following disclaimer.\n * \n * 2. Redistributions in binary form must reproduce the above copyright notice,\n * this list of conditions and the following disclaimer in the documentation\n * and/or other materials provided with the distribution.\n * \n * 3. The end-user documentation included with the redistribution, if any, must\n * include the following acknowledgment: \"This product includes software\n * developed by the Apache Software Foundation (http://www.apache.org/).\"\n * Alternately, this acknowledgment may appear in the software itself, if\n * and wherever such third-party acknowledgments normally appear.\n * \n * 4. The names \"Ant\" and \"Apache Software Foundation\" must not be used to\n * endorse or promote products derived from this software without prior\n * written permission. For written permission, please contact\n * apache@apache.org.\n * \n * 5. Products derived from this software may not be called \"Apache\", nor may\n * \"Apache\" appear in their name, without prior written permission of the\n * Apache Software Foundation.\n * \n * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES,\n * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND\n * FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE\n * APACHE SOFTWARE FOUNDATION OR ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT,\n * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLU-\n * DING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS\n * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON\n * ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF\n * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n * \n * This software consists of voluntary contributions made by many individuals\n * on behalf of the Apache Software Foundation. For more information on the \n * Apache Software Foundation, please see .\n *\n */\n" }, { "path": "lib/LGPL", "content": "\t\t GNU LESSER GENERAL PUBLIC LICENSE\n\t\t Version 2.1, February 1999\n\n Copyright (C) 1991, 1999 Free Software Foundation, Inc.\n 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA\n Everyone is permitted to copy and distribute verbatim copies\n of this license document, but changing it is not allowed.\n\n[This is the first released version of the Lesser GPL. It also counts\n as the successor of the GNU Library Public License, version 2, hence\n the version number 2.1.]\n\n\t\t\t Preamble\n\n The licenses for most software are designed to take away your\nfreedom to share and change it. By contrast, the GNU General Public\nLicenses are intended to guarantee your freedom to share and change\nfree software--to make sure the software is free for all its users.\n\n This license, the Lesser General Public License, applies to some\nspecially designated software packages--typically libraries--of the\nFree Software Foundation and other authors who decide to use it. You\ncan use it too, but we suggest you first think carefully about whether\nthis license or the ordinary General Public License is the better\nstrategy to use in any particular case, based on the explanations below.\n\n When we speak of free software, we are referring to freedom of use,\nnot price. Our General Public Licenses are designed to make sure that\nyou have the freedom to distribute copies of free software (and charge\nfor this service if you wish); that you receive source code or can get\nit if you want it; that you can change the software and use pieces of\nit in new free programs; and that you are informed that you can do\nthese things.\n\n To protect your rights, we need to make restrictions that forbid\ndistributors to deny you these rights or to ask you to surrender these\nrights. These restrictions translate to certain responsibilities for\nyou if you distribute copies of the library or if you modify it.\n\n For example, if you distribute copies of the library, whether gratis\nor for a fee, you must give the recipients all the rights that we gave\nyou. You must make sure that they, too, receive or can get the source\ncode. If you link other code with the library, you must provide\ncomplete object files to the recipients, so that they can relink them\nwith the library after making changes to the library and recompiling\nit. And you must show them these terms so they know their rights.\n\n We protect your rights with a two-step method: (1) we copyright the\nlibrary, and (2) we offer you this license, which gives you legal\npermission to copy, distribute and/or modify the library.\n\n To protect each distributor, we want to make it very clear that\nthere is no warranty for the free library. Also, if the library is\nmodified by someone else and passed on, the recipients should know\nthat what they have is not the original version, so that the original\nauthor's reputation will not be affected by problems that might be\nintroduced by others.\n\f\n Finally, software patents pose a constant threat to the existence of\nany free program. We wish to make sure that a company cannot\neffectively restrict the users of a free program by obtaining a\nrestrictive license from a patent holder. Therefore, we insist that\nany patent license obtained for a version of the library must be\nconsistent with the full freedom of use specified in this license.\n\n Most GNU software, including some libraries, is covered by the\nordinary GNU General Public License. This license, the GNU Lesser\nGeneral Public License, applies to certain designated libraries, and\nis quite different from the ordinary General Public License. We use\nthis license for certain libraries in order to permit linking those\nlibraries into non-free programs.\n\n When a program is linked with a library, whether statically or using\na shared library, the combination of the two is legally speaking a\ncombined work, a derivative of the original library. The ordinary\nGeneral Public License therefore permits such linking only if the\nentire combination fits its criteria of freedom. The Lesser General\nPublic License permits more lax criteria for linking other code with\nthe library.\n\n We call this license the \"Lesser\" General Public License because it\ndoes Less to protect the user's freedom than the ordinary General\nPublic License. It also provides other free software developers Less\nof an advantage over competing non-free programs. These disadvantages\nare the reason we use the ordinary General Public License for many\nlibraries. However, the Lesser license provides advantages in certain\nspecial circumstances.\n\n For example, on rare occasions, there may be a special need to\nencourage the widest possible use of a certain library, so that it becomes\na de-facto standard. To achieve this, non-free programs must be\nallowed to use the library. A more frequent case is that a free\nlibrary does the same job as widely used non-free libraries. In this\ncase, there is little to gain by limiting the free library to free\nsoftware only, so we use the Lesser General Public License.\n\n In other cases, permission to use a particular library in non-free\nprograms enables a greater number of people to use a large body of\nfree software. For example, permission to use the GNU C Library in\nnon-free programs enables many more people to use the whole GNU\noperating system, as well as its variant, the GNU/Linux operating\nsystem.\n\n Although the Lesser General Public License is Less protective of the\nusers' freedom, it does ensure that the user of a program that is\nlinked with the Library has the freedom and the wherewithal to run\nthat program using a modified version of the Library.\n\n The precise terms and conditions for copying, distribution and\nmodification follow. Pay close attention to the difference between a\n\"work based on the library\" and a \"work that uses the library\". The\nformer contains code derived from the library, whereas the latter must\nbe combined with the library in order to run.\n\f\n\t\t GNU LESSER GENERAL PUBLIC LICENSE\n TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION\n\n 0. This License Agreement applies to any software library or other\nprogram which contains a notice placed by the copyright holder or\nother authorized party saying it may be distributed under the terms of\nthis Lesser General Public License (also called \"this License\").\nEach licensee is addressed as \"you\".\n\n A \"library\" means a collection of software functions and/or data\nprepared so as to be conveniently linked with application programs\n(which use some of those functions and data) to form executables.\n\n The \"Library\", below, refers to any such software library or work\nwhich has been distributed under these terms. A \"work based on the\nLibrary\" means either the Library or any derivative work under\ncopyright law: that is to say, a work containing the Library or a\nportion of it, either verbatim or with modifications and/or translated\nstraightforwardly into another language. (Hereinafter, translation is\nincluded without limitation in the term \"modification\".)\n\n \"Source code\" for a work means the preferred form of the work for\nmaking modifications to it. For a library, complete source code means\nall the source code for all modules it contains, plus any associated\ninterface definition files, plus the scripts used to control compilation\nand installation of the library.\n\n Activities other than copying, distribution and modification are not\ncovered by this License; they are outside its scope. The act of\nrunning a program using the Library is not restricted, and output from\nsuch a program is covered only if its contents constitute a work based\non the Library (independent of the use of the Library in a tool for\nwriting it). Whether that is true depends on what the Library does\nand what the program that uses the Library does.\n \n 1. You may copy and distribute verbatim copies of the Library's\ncomplete source code as you receive it, in any medium, provided that\nyou conspicuously and appropriately publish on each copy an\nappropriate copyright notice and disclaimer of warranty; keep intact\nall the notices that refer to this License and to the absence of any\nwarranty; and distribute a copy of this License along with the\nLibrary.\n\n You may charge a fee for the physical act of transferring a copy,\nand you may at your option offer warranty protection in exchange for a\nfee.\n\f\n 2. You may modify your copy or copies of the Library or any portion\nof it, thus forming a work based on the Library, and copy and\ndistribute such modifications or work under the terms of Section 1\nabove, provided that you also meet all of these conditions:\n\n a) The modified work must itself be a software library.\n\n b) You must cause the files modified to carry prominent notices\n stating that you changed the files and the date of any change.\n\n c) You must cause the whole of the work to be licensed at no\n charge to all third parties under the terms of this License.\n\n d) If a facility in the modified Library refers to a function or a\n table of data to be supplied by an application program that uses\n the facility, other than as an argument passed when the facility\n is invoked, then you must make a good faith effort to ensure that,\n in the event an application does not supply such function or\n table, the facility still operates, and performs whatever part of\n its purpose remains meaningful.\n\n (For example, a function in a library to compute square roots has\n a purpose that is entirely well-defined independent of the\n application. Therefore, Subsection 2d requires that any\n application-supplied function or table used by this function must\n be optional: if the application does not supply it, the square\n root function must still compute square roots.)\n\nThese requirements apply to the modified work as a whole. If\nidentifiable sections of that work are not derived from the Library,\nand can be reasonably considered independent and separate works in\nthemselves, then this License, and its terms, do not apply to those\nsections when you distribute them as separate works. But when you\ndistribute the same sections as part of a whole which is a work based\non the Library, the distribution of the whole must be on the terms of\nthis License, whose permissions for other licensees extend to the\nentire whole, and thus to each and every part regardless of who wrote\nit.\n\nThus, it is not the intent of this section to claim rights or contest\nyour rights to work written entirely by you; rather, the intent is to\nexercise the right to control the distribution of derivative or\ncollective works based on the Library.\n\nIn addition, mere aggregation of another work not based on the Library\nwith the Library (or with a work based on the Library) on a volume of\na storage or distribution medium does not bring the other work under\nthe scope of this License.\n\n 3. You may opt to apply the terms of the ordinary GNU General Public\nLicense instead of this License to a given copy of the Library. To do\nthis, you must alter all the notices that refer to this License, so\nthat they refer to the ordinary GNU General Public License, version 2,\ninstead of to this License. (If a newer version than version 2 of the\nordinary GNU General Public License has appeared, then you can specify\nthat version instead if you wish.) Do not make any other change in\nthese notices.\n\f\n Once this change is made in a given copy, it is irreversible for\nthat copy, so the ordinary GNU General Public License applies to all\nsubsequent copies and derivative works made from that copy.\n\n This option is useful when you wish to copy part of the code of\nthe Library into a program that is not a library.\n\n 4. You may copy and distribute the Library (or a portion or\nderivative of it, under Section 2) in object code or executable form\nunder the terms of Sections 1 and 2 above provided that you accompany\nit with the complete corresponding machine-readable source code, which\nmust be distributed under the terms of Sections 1 and 2 above on a\nmedium customarily used for software interchange.\n\n If distribution of object code is made by offering access to copy\nfrom a designated place, then offering equivalent access to copy the\nsource code from the same place satisfies the requirement to\ndistribute the source code, even though third parties are not\ncompelled to copy the source along with the object code.\n\n 5. A program that contains no derivative of any portion of the\nLibrary, but is designed to work with the Library by being compiled or\nlinked with it, is called a \"work that uses the Library\". Such a\nwork, in isolation, is not a derivative work of the Library, and\ntherefore falls outside the scope of this License.\n\n However, linking a \"work that uses the Library\" with the Library\ncreates an executable that is a derivative of the Library (because it\ncontains portions of the Library), rather than a \"work that uses the\nlibrary\". The executable is therefore covered by this License.\nSection 6 states terms for distribution of such executables.\n\n When a \"work that uses the Library\" uses material from a header file\nthat is part of the Library, the object code for the work may be a\nderivative work of the Library even though the source code is not.\nWhether this is true is especially significant if the work can be\nlinked without the Library, or if the work is itself a library. The\nthreshold for this to be true is not precisely defined by law.\n\n If such an object file uses only numerical parameters, data\nstructure layouts and accessors, and small macros and small inline\nfunctions (ten lines or less in length), then the use of the object\nfile is unrestricted, regardless of whether it is legally a derivative\nwork. (Executables containing this object code plus portions of the\nLibrary will still fall under Section 6.)\n\n Otherwise, if the work is a derivative of the Library, you may\ndistribute the object code for the work under the terms of Section 6.\nAny executables containing that work also fall under Section 6,\nwhether or not they are linked directly with the Library itself.\n\f\n 6. As an exception to the Sections above, you may also combine or\nlink a \"work that uses the Library\" with the Library to produce a\nwork containing portions of the Library, and distribute that work\nunder terms of your choice, provided that the terms permit\nmodification of the work for the customer's own use and reverse\nengineering for debugging such modifications.\n\n You must give prominent notice with each copy of the work that the\nLibrary is used in it and that the Library and its use are covered by\nthis License. You must supply a copy of this License. If the work\nduring execution displays copyright notices, you must include the\ncopyright notice for the Library among them, as well as a reference\ndirecting the user to the copy of this License. Also, you must do one\nof these things:\n\n a) Accompany the work with the complete corresponding\n machine-readable source code for the Library including whatever\n changes were used in the work (which must be distributed under\n Sections 1 and 2 above); and, if the work is an executable linked\n with the Library, with the complete machine-readable \"work that\n uses the Library\", as object code and/or source code, so that the\n user can modify the Library and then relink to produce a modified\n executable containing the modified Library. (It is understood\n that the user who changes the contents of definitions files in the\n Library will not necessarily be able to recompile the application\n to use the modified definitions.)\n\n b) Use a suitable shared library mechanism for linking with the\n Library. A suitable mechanism is one that (1) uses at run time a\n copy of the library already present on the user's computer system,\n rather than copying library functions into the executable, and (2)\n will operate properly with a modified version of the library, if\n the user installs one, as long as the modified version is\n interface-compatible with the version that the work was made with.\n\n c) Accompany the work with a written offer, valid for at\n least three years, to give the same user the materials\n specified in Subsection 6a, above, for a charge no more\n than the cost of performing this distribution.\n\n d) If distribution of the work is made by offering access to copy\n from a designated place, offer equivalent access to copy the above\n specified materials from the same place.\n\n e) Verify that the user has already received a copy of these\n materials or that you have already sent this user a copy.\n\n For an executable, the required form of the \"work that uses the\nLibrary\" must include any data and utility programs needed for\nreproducing the executable from it. However, as a special exception,\nthe materials to be distributed need not include anything that is\nnormally distributed (in either source or binary form) with the major\ncomponents (compiler, kernel, and so on) of the operating system on\nwhich the executable runs, unless that component itself accompanies\nthe executable.\n\n It may happen that this requirement contradicts the license\nrestrictions of other proprietary libraries that do not normally\naccompany the operating system. Such a contradiction means you cannot\nuse both them and the Library together in an executable that you\ndistribute.\n\f\n 7. You may place library facilities that are a work based on the\nLibrary side-by-side in a single library together with other library\nfacilities not covered by this License, and distribute such a combined\nlibrary, provided that the separate distribution of the work based on\nthe Library and of the other library facilities is otherwise\npermitted, and provided that you do these two things:\n\n a) Accompany the combined library with a copy of the same work\n based on the Library, uncombined with any other library\n facilities. This must be distributed under the terms of the\n Sections above.\n\n b) Give prominent notice with the combined library of the fact\n that part of it is a work based on the Library, and explaining\n where to find the accompanying uncombined form of the same work.\n\n 8. You may not copy, modify, sublicense, link with, or distribute\nthe Library except as expressly provided under this License. Any\nattempt otherwise to copy, modify, sublicense, link with, or\ndistribute the Library is void, and will automatically terminate your\nrights under this License. However, parties who have received copies,\nor rights, from you under this License will not have their licenses\nterminated so long as such parties remain in full compliance.\n\n 9. You are not required to accept this License, since you have not\nsigned it. However, nothing else grants you permission to modify or\ndistribute the Library or its derivative works. These actions are\nprohibited by law if you do not accept this License. Therefore, by\nmodifying or distributing the Library (or any work based on the\nLibrary), you indicate your acceptance of this License to do so, and\nall its terms and conditions for copying, distributing or modifying\nthe Library or works based on it.\n\n 10. Each time you redistribute the Library (or any work based on the\nLibrary), the recipient automatically receives a license from the\noriginal licensor to copy, distribute, link with or modify the Library\nsubject to these terms and conditions. You may not impose any further\nrestrictions on the recipients' exercise of the rights granted herein.\nYou are not responsible for enforcing compliance by third parties with\nthis License.\n\f\n 11. If, as a consequence of a court judgment or allegation of patent\ninfringement or for any other reason (not limited to patent issues),\nconditions are imposed on you (whether by court order, agreement or\notherwise) that contradict the conditions of this License, they do not\nexcuse you from the conditions of this License. If you cannot\ndistribute so as to satisfy simultaneously your obligations under this\nLicense and any other pertinent obligations, then as a consequence you\nmay not distribute the Library at all. For example, if a patent\nlicense would not permit royalty-free redistribution of the Library by\nall those who receive copies directly or indirectly through you, then\nthe only way you could satisfy both it and this License would be to\nrefrain entirely from distribution of the Library.\n\nIf any portion of this section is held invalid or unenforceable under any\nparticular circumstance, the balance of the section is intended to apply,\nand the section as a whole is intended to apply in other circumstances.\n\nIt is not the purpose of this section to induce you to infringe any\npatents or other property right claims or to contest validity of any\nsuch claims; this section has the sole purpose of protecting the\nintegrity of the free software distribution system which is\nimplemented by public license practices. Many people have made\ngenerous contributions to the wide range of software distributed\nthrough that system in reliance on consistent application of that\nsystem; it is up to the author/donor to decide if he or she is willing\nto distribute software through any other system and a licensee cannot\nimpose that choice.\n\nThis section is intended to make thoroughly clear what is believed to\nbe a consequence of the rest of this License.\n\n 12. If the distribution and/or use of the Library is restricted in\ncertain countries either by patents or by copyrighted interfaces, the\noriginal copyright holder who places the Library under this License may add\nan explicit geographical distribution limitation excluding those countries,\nso that distribution is permitted only in or among countries not thus\nexcluded. In such case, this License incorporates the limitation as if\nwritten in the body of this License.\n\n 13. The Free Software Foundation may publish revised and/or new\nversions of the Lesser General Public License from time to time.\nSuch new versions will be similar in spirit to the present version,\nbut may differ in detail to address new problems or concerns.\n\nEach version is given a distinguishing version number. If the Library\nspecifies a version number of this License which applies to it and\n\"any later version\", you have the option of following the terms and\nconditions either of that version or of any later version published by\nthe Free Software Foundation. If the Library does not specify a\nlicense version number, you may choose any version ever published by\nthe Free Software Foundation.\n\f\n 14. If you wish to incorporate parts of the Library into other free\nprograms whose distribution conditions are incompatible with these,\nwrite to the author to ask for permission. For software which is\ncopyrighted by the Free Software Foundation, write to the Free\nSoftware Foundation; we sometimes make exceptions for this. Our\ndecision will be guided by the two goals of preserving the free status\nof all derivatives of our free software and of promoting the sharing\nand reuse of software generally.\n\n\t\t\t NO WARRANTY\n\n 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO\nWARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.\nEXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR\nOTHER PARTIES PROVIDE THE LIBRARY \"AS IS\" WITHOUT WARRANTY OF ANY\nKIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\nPURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE\nLIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME\nTHE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.\n\n 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN\nWRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY\nAND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU\nFOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR\nCONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE\nLIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING\nRENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A\nFAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF\nSUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH\nDAMAGES.\n\n\t\t END OF TERMS AND CONDITIONS\n\f\n How to Apply These Terms to Your New Libraries\n\n If you develop a new library, and you want it to be of the greatest\npossible use to the public, we recommend making it free software that\neveryone can redistribute and change. You can do so by permitting\nredistribution under these terms (or, alternatively, under the terms of the\nordinary General Public License).\n\n To apply these terms, attach the following notices to the library. It is\nsafest to attach them to the start of each source file to most effectively\nconvey the exclusion of warranty; and each file should have at least the\n\"copyright\" line and a pointer to where the full notice is found.\n\n \n Copyright (C) \n\n This library is free software; you can redistribute it and/or\n modify it under the terms of the GNU Lesser General Public\n License as published by the Free Software Foundation; either\n version 2.1 of the License, or (at your option) any later version.\n\n This library is distributed in the hope that it will be useful,\n but WITHOUT ANY WARRANTY; without even the implied warranty of\n MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU\n Lesser General Public License for more details.\n\n You should have received a copy of the GNU Lesser General Public\n License along with this library; if not, write to the Free Software\n Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA\n\nAlso add information on how to contact you by electronic and paper mail.\n\nYou should also get your employer (if you work as a programmer) or your\nschool, if any, to sign a \"copyright disclaimer\" for the library, if\nnecessary. Here is a sample; alter the names:\n\n Yoyodyne, Inc., hereby disclaims all copyright interest in the\n library `Frob' (a library for tweaking knobs) written by James Random Hacker.\n\n , 1 April 1990\n Ty Coon, President of Vice\n\nThat's all there is to it!\n\n\n" }, { "path": "lib/LIBNOTES", "content": "\nThis file briefly describes the libraries which are used by the \nOpenNLP CCG Library. See the appropriate license files also in this directory.\n\n------------------------------------------------------------------------\nant.jar, ant-launcher.jar, ant-junit.jar, ant-junit4.jar, ant-contrib.jar\n\nThe Apache Ant Build System, version 1.9.0\nHomepage: http://ant.apache.org/\nANT Contrib, version 1.0b3\nHomepage: http://sourceforge.net/projects/ant-contrib/\nLicense: Apache Software License (ASL)\n\nA Java based build tool. \n\n\n------------------------------------------------------------------------\njdom.jar\n\nJDOM, version 1.1\nHomepage: http://www.jdom.org\nLicense: See jdom.license\n\nA complete, Java-based solution for accessing, manipulating, and\noutputting XML data from Java code.\n\n\n------------------------------------------------------------------------\njgrapht-jdk1.6.jar\n\nJGraphT, version 0.8.3\nHomepage: http://www.jgrapht.org\nLicense: LGPL \n\nA free Java graph library that provides mathematical graph-theory objects \nand algorithms.\n\n\n------------------------------------------------------------------------\njline.jar\n\njLine, version 1.0\nHomepage: http://jline.sourceforge.net/\nLicense: See jline.license\n\nJLine is a Java library for handling console input. \nIt is similar in functionality to BSD editline and GNU readline,\nproviding a command input history and more.\n\n\n------------------------------------------------------------------------\njunit-4.10.jar\n\nJUnit, version 4.10\nHomepage: http://junit.sourceforge.net/\nLicense: Common Public License - v 1.0\n\nJUnit is a simple, open source framework to write and run repeatable\ntests. \n\n\n------------------------------------------------------------------------\nlibken.so\n\nKenLM, version 4\nHomepage: http://kheafield.com/code/kenlm/\nLicense: LGPL primarily (see src/kenlm/LICENSE)\n\nKenLM is a language modeling toolkit supporting large LMs via a\nmemory-mapped binary format. JNI interface for linux adapted from the\nJoshua decoder; other platforms possible in principle.\n\n\n------------------------------------------------------------------------\ntrove.jar\n\nGNU Trove, version 1.0.2\nHomepage: http://trove4j.sf.net\nLicense: LGPL\n\nHigh performance collections for Java.\n\n\n------------------------------------------------------------------------\nxalan.jar, xercesImpl.jar, xml-apis.jar, xlstc.jar, serializer.jar\n\nThe Apache Xalan-J XML transformation processor, version 2.7.1\nThe Apache Xerces-J XML parser, version 2.9.0\nHomepage: http://xml.apache.org\nLicense: Apache Software License (ASL)\n\nXSLT transformation library and associated XML parser,\nwith schema validation support.\n\n\n------------------------------------------------------------------------\njopt-simple.jar\n\nJOpt Simple, version 3.1\nHomepage: http://jopt-simple.sourceforge.net/\nLicense: MIT License\n\nCommand-line options parsing library.\n\n\n------------------------------------------------------------------------\njavacc.jar\n\nJavaCC, version 4.0\nHomepage: https://javacc.dev.java.net/\nLicense: SUN License\n\nJava parser generator.\n\n\n" }, { "path": "lib/MIT", "content": "\r\n\r\nThe MIT License\r\n\r\nCopyright (c) \r\n\r\nPermission is hereby granted, free of charge, to any person obtaining a copy\r\nof this software and associated documentation files (the \"Software\"), to deal\r\nin the Software without restriction, including without limitation the rights\r\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\ncopies of the Software, and to permit persons to whom the Software is\r\nfurnished to do so, subject to the following conditions:\r\n\r\nThe above copyright notice and this permission notice shall be included in\r\nall copies or substantial portions of the Software.\r\n\r\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\r\nTHE SOFTWARE.\r\n\r\n" }, { "path": "lib/SUN", "content": "Copyright (c) 2003 Sun Microsystems, Inc. All Rights Reserved.\n\nRedistribution and use in source and binary forms, with or without modification, \nare permitted provided that the following conditions are met:\n\n-Redistribution of source code must retain the above copyright notice, this list of conditions and the following disclaimer.\n\n-Redistribution in binary form must reproduce the above copyright notice, \nthis list of conditions and the following disclaimer in the documentation and/or \nother materials provided with the distribution.\n\nNeither the name of Sun Microsystems, Inc. or the names of contributors may \nbe used to endorse or promote products derived from this software without \nspecific prior written permission.\n\nThis software is provided \"AS IS,\" without a warranty of any kind. ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE HEREBY EXCLUDED. SUN MIDROSYSTEMS, INC. (\"SUN\") AND ITS LICENSORS SHALL NOT BE LIABLE FOR ANY DAMAGES SUFFERED BY LICENSEE AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THIS SOFTWARE OR ITS DERIVATIVES. IN NO EVENT WILL SUN OR ITS LICENSORS BE LIABLE FOR ANY LOST REVENUE, PROFIT OR DATA, OR FOR DIRECT, INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL OR PUNITIVE DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF THE USE OF OR INABILITY TO USE THIS SOFTWARE, EVEN IF SUN HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.\n\nYou acknowledge that this software is not designed, licensed or intended for use in the design, construction, operation or maintenance of any nuclear facility.\n" }, { "path": "lib/jdom.license", "content": "/*-- \n\n $Id: jdom.license,v 1.1.1.1 2003/02/28 18:02:10 mwhite14850 Exp $\n\n Copyright (C) 2001 Brett McLaughlin & Jason Hunter.\n All rights reserved.\n \n Redistribution and use in source and binary forms, with or without\n modification, are permitted provided that the following conditions\n are met:\n \n 1. Redistributions of source code must retain the above copyright\n notice, this list of conditions, and the following disclaimer.\n \n 2. Redistributions in binary form must reproduce the above copyright\n notice, this list of conditions, and the disclaimer that follows \n these conditions in the documentation and/or other materials \n provided with the distribution.\n\n 3. The name \"JDOM\" must not be used to endorse or promote products\n derived from this software without prior written permission. For\n written permission, please contact license@jdom.org.\n \n 4. Products derived from this software may not be called \"JDOM\", nor\n may \"JDOM\" appear in their name, without prior written permission\n from the JDOM Project Management (pm@jdom.org).\n \n In addition, we request (but do not require) that you include in the \n end-user documentation provided with the redistribution and/or in the \n software itself an acknowledgement equivalent to the following:\n \"This product includes software developed by the\n JDOM Project (http://www.jdom.org/).\"\n Alternatively, the acknowledgment may be graphical using the logos \n available at http://www.jdom.org/images/logos.\n\n THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED\n WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES\n OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n DISCLAIMED. IN NO EVENT SHALL THE JDOM AUTHORS OR THE PROJECT\n CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF\n USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\n ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\n OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT\n OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF\n SUCH DAMAGE.\n\n This software consists of voluntary contributions made by many \n individuals on behalf of the JDOM Project and was originally \n created by Brett McLaughlin and \n Jason Hunter . For more information on the \n JDOM Project, please see .\n \n */\n\n" }, { "path": "lib/jline.license", "content": "Copyright (c) 2002-2006, Marc Prud'hommeaux \nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or\nwithout modification, are permitted provided that the following\nconditions are met:\n\nRedistributions of source code must retain the above copyright\nnotice, this list of conditions and the following disclaimer.\n\nRedistributions in binary form must reproduce the above copyright\nnotice, this list of conditions and the following disclaimer\nin the documentation and/or other materials provided with\nthe distribution.\n\nNeither the name of JLine nor the names of its contributors\nmay be used to endorse or promote products derived from this\nsoftware without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,\nBUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY\nAND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO\nEVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE\nFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,\nOR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,\nPROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\nDATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED\nAND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT\nLIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING\nIN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED\nOF THE POSSIBILITY OF SUCH DAMAGE.\n\n" }, { "path": "pom.xml", "content": " \n\n 4.0.0\n opennlp\n ccg\n 0.10.0\n pom\n\n \n src/\n \n \n \n 1.8\n 1.8\n \n \n \n" }, { "path": "src/ccg2xml/README", "content": "README for ccg2xml\nBen Wing\nOctober 8, 2006\n\nccg2xml is a front-end for more conveniently specifying OpenCCG grammars.\nThe native XML format is not human-friendly and contains much repetition of\ninformation, often involving long-distance dependencies, which are\ndifficult to keep track of. The .ccg format that is accepted by ccg2xml is\ndesigned to be expressive, concise and human-friendly, with as little\nrequired duplication as possible.\n\nccg2xml is run on a .ccg file and produces (by default) the following files\nin the current directory:\n\n-\tlexicon.xml: Categories, lexical insertion rules\n-\tmorph.xml: Morphological forms of words, feature macros\n-\ttypes.xml: Feature type hierarchies\n-\trules.xml: Allowable combinatory, type-raising and type-changing rules\n-\tgrammar.xml: File giving the names and locations of the previous four\n files\n-\ttestbed.xml: File containing test sentences and expected number of\n parses\n\nThe standard OpenCCG tools, such as tccg, can then be run, as if the\ngrammar had been developed directly in XML. ccg2xml will output error\nmessages if it encounters syntax errors, and in such a case will not\ngenerate any XML files. (There are also warnings that it may generate,\nwhich are non-fatal in that they do not prevent generation.) ccg2xml has a\nfew options:\n\n -o controls which XML files are generated; this can be useful for using\n ccg2xml in conjunction with an existing grammar.\n\n -t outputs the file to stdout after all macro substitutions have been\n applied; this can help in debugging complicated macros that aren't\n working as expected.\n\n -d controls which directory the XML files are output to (by default, the\n current directory).\n\n -y, -m, and --super-macro-debug are debug options useful mostly for\n developers of ccg2xml.\n\nFor more information, run `ccg2xml -h' to get the \"usage\" message.\n\nccg2xml is written in Python, and itself a piece of generated code.\nccg2xml makes use of PLY, a Python system for development of non-ambiguous\ncontext-free grammar parsers, similar to lex and yacc. PLY is written by\nDavid Beazley (dave@dabeaz.com), available at http://www.dabeaz.com/ply/;\nPLY 1.6 is included as part of the ccg2cml source (the files lex.py and\nyacc.py). PLY uses a clever trick of introspecting on the source file to\ndetermine the rules, which are stored in documentation strings. However,\nthis ends up requiring a certain amount of redundancy in the source file.\nThis gets awkward for large projects, so I created a front-end format that\nclosely emulates yacc's format, while adding support for regular-expression\noperators on the right-hand side (RHS) of a rule (*, +, and ?); it consists\nof mixed CFG specifications and Python code. The file `convert-ply.py'\nconverts from my local .ply format into standard .py files, and is used\n(see `Makefile') to generate the ccg2xml script.\n\n (It also attempts to work\naround a serious bug in PLY .) The source\nfile ccg.ply is currently 1,727 lines in length (of mixed CFG\nspecifications and Python code), and the auto-generated ccg2xml file is\n2,370 lines.\n\nccg2xml is written in Python. It uses PLY, which is a Python\nimplementation of the standard `yacc' tool. \n\nNOTE: ccg2xml was written using PLY 1.6, which is included as part of\nccg2xml (lex.py and yacc.py). PLY 1.6 contains a serious bug involving\nempty RHS productions, which convert-cgi.py attempts to work around and\nwhich ccg.ply also works around. I was never able to figure out the exact\nrule format that triggers this bug, so I simply had to use trial and error\nin rewriting rules to get a working ccg2xml. It is strongly recommended to\nupgrade to the most recent version of PLY (currently 2.1), which is likely\nto fix this problem, and make it possible to deterministically modify\nccg.ply.\n\n--------------------------------------------------\n The .ccg format\n--------------------------------------------------\n\nThe general feel of the syntax is like C, Java or Perl. Indentation and\nwhitespace is unimportant (The only exception is in macro definitions,\nwhere the text of a macro must either be on the same line as the definition \n-- possibly extended with backslash line-continuation markers -- or be\nenclosed in braces). This is critical for macros, where having to worry\nabout precisely controlling the indentation or whitespace at the beginning\nor end of a macro definition (such as would be required in Python or shell\nscripts) would be a major hassle and impediment.\n\nThe file consists of a number of declarations, which specify features\n(feature {}), word stems and inflected formats (word {}), lexical insertion\nrules (family {}), test sentences (testbed {}), and other information\nrelevant to the grammar. It can also contain macro definitions (def {})\nalong with corresponding macro calls, such as the calls to noun() and\nverb() in the sample grammar above. The macro mechanism is essentially\njust a way of doing parameterized text substitution, and is discussed in\ndetail in sections 2.4 and 3.2.\n\nIn general, declarations can be in any order; any dependencies that must be\ncomputed are done after the entire file has been parsed. Furthermore, any\ndeclaration containing multiple parts (for example, a feature declaration\nencompassing multiple features, a word declaration with multiple\ninflections, or a family declaration with multiple insertion rules) can be\nsplit up into separate declarations. In addition, the syntax tries to be\nvery forgiving of the usage of commas, semicolons, and other terminators\nand separators. (In most lists, in fact, commas are optional, duplicated\ncommas are no problem, and extra commas at the end of a list are\nallowed. The main exception where commas matter is in macro calls. You can\nstill put an extra comma at the end of a macro call, but otherwise you must\nhave exactly one comma (no more, no less) between arguments. The reason is\nthat macro arguments can contain pretty much any text whatsoever (including\nno text at all), so commas are needed to indicate where one argument stops\nand the next one starts.) This simplifies, in a number of ways, the\ncreation of lists using macros.\n\nThe exceptions to these syntactic allowances are all related to macros.\nFor example, macros must be defined before they can be used. (That is,\nbefore they are used in a macro call in the main text of the grammar; this\ndoes not apply to macro calls inside of another macro definition.) Also,\nthe syntax of commas in argument lists of macro calls is much stricter than\nin lists occurring elsewhere in the grammar (see above).\n\nNote that, in general, there is no need to put quotes around literal text,\nsuch as the inflected forms of words. There is also no concept of\n\"reserved words\" in this format. Words like feature, word, and entry that\nhave a special significance as \"declarators\" when in the right place in the\nsyntax can otherwise be freely used as word stems and inflections, macro\nvariable names, part of speech tags, etc. At the same time, however, it is\npossible to put quotes (single or double) around text. This allows special\ncharacters, spaces, etc. to be used in word inflections, feature values,\nand the like; otherwise, only letters, numbers, '_', '-', '+', '%' and\nnon-ASCII characters can be used. Quotes can also be used to protect\nagainst the accidental interpretation of a word as a declarator, in the\nrare case that this is needed.\n \nThe overall syntactic laxity of the language is part of a general\nphilosophy of making the grammar language as tolerant and as expressive as\npossible. This is in keeping with the purpose of the language. Although\nit looks similar to a traditional programming language, its semantics is on\na much higher level, where expressiveness is much more important and\nprecision relatively less important. Furthermore, the language is intended\nfor use by non-programmers as well as programmers, and non-programmers are\nis likely to have difficulty with rigid syntax rules, especially in the\npresence of macros.\n\n\n--------------------------------------------------\n Macros\n--------------------------------------------------\n\nThe mechanism I chose to use to eliminate repetition is parameterized\nmacros, which work like macro functions in C. Their definition includes a\nseries of formal parameters and a section of literal text intermixed with\nthe names of the macro's parameters. When a call to the macro occurs, the\nmacro's text replaces the call, with the actual arguments to the call\nsubstituted for the occurrences of the corresponding formal parameters in\nthe text. Any macro calls occurring in this text are then recursively\nexpanded.\n\nThe macro expansion mechanism generally happens at the level of tokens,\nwhere a token is a single word, a single non-alphanumeric character, or a\nsingle piece of quoted text. However, it is possible to construct a token\nby pasting two tokens together, using the dot operator; this is similar to\nthe ## operator in C. This is useful for constructing inflected forms of\nwords, as shown in the tiny.ccg and arabic.ccg sample files. It is also\npossible to compositionally construct the name of a macro and then call\nthis constructed name; this is used in arabic.ccg, in the macro\n3rd-weak-verb().\n\nThe parser pays attention to matched pairs of parens, brackets, and braces\nin the text of a macro call argument, and will not get confused by commas\ninside of such matched delimiters. Thus, a macro call foo(a, bar(b, c)) is\ncorrectly interpreted as a call to foo() with two arguments, a and bar(b,\nc), and not a call to foo() with three arguments, where the first argument\nis a, second argument is bar(b, and third argument is c).\n\nThe text of both macro definitions and arguments to macro calls can\noptionally be surrounded by braces, to clearly delimit the text boundaries.\nThe braces do not form part of the text itself (to specify text surrounded\nby braces, a second set would have to be added around the text). It is\nconventional to surround macro definitions with braces, except sometimes\nfor very short definitions that fit on one line. However, braces are not\nnormally used around macro call arguments unless necessary for correct\nparsing (for example, a macro argument containing a comma in it that is not\nsurrounded by matching delimiters).\n\n\n--------------------------------\n Sample Files\n--------------------------------\n\nThree sample .ccg-format files are currently included in the directory\n`ccg-format-grammars' off of the top-level OpenCCG\ndirectory:\n\n- `tinytiny.ccg' is a simple file that parses a basic chunk of\n English (including nouns, personal pronouns, articles, and transitive and\n intransitive verbs) and demonstrates the basic features of the .ccg format.\n\n- `tiny.ccg' is a conversion of the original OpenCCG \"tiny\" sample grammar\n to .ccg format. `tiny.ccg' is no longer as small as its name\n implies, especially since it contains a large number of comments describing\n the CCG format in depth, as well as a number of example declarations to\n demonstrate the various features of the CCG format.\n\n- `arabic.ccg' is an Arabic grammar fragment, which demonstrates how to\n effectively use macros in order to efficiently generate the morphology of\n a complex, highly-inflected language.\n" }, { "path": "src/ccg2xml/Tree.py", "content": "# Highly optimized Tkinter tree control\n# by Charles E. \"Gene\" Cash\n#\n# This is documented more fully on my homepage at\n# http://home.cfl.rr.com/genecash/ and if it's not there, look in the Vaults\n# of Parnassus at http://www.vex.net/parnassus/ which I promise to keep\n# updated.\n#\n# Thanks to Laurent Claustre for sending lots of helpful\n# bug reports.\n#\n# This copyright license is intended to be similar to the FreeBSD license. \n#\n# Copyright 1998 Gene Cash All rights reserved. \n#\n# Redistribution and use in source and binary forms, with or without\n# modification, are permitted provided that the following conditions are\n# met:\n#\n# 1. Redistributions of source code must retain the above copyright\n# notice, this list of conditions and the following disclaimer.\n# 2. Redistributions in binary form must reproduce the above copyright\n# notice, this list of conditions and the following disclaimer in the\n# documentation and/or other materials provided with the\n# distribution.\n#\n# THIS SOFTWARE IS PROVIDED BY GENE CASH ``AS IS'' AND ANY EXPRESS OR\n# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES\n# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n# DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR\n# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\n# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS\n# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)\n# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,\n# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN\n# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n# POSSIBILITY OF SUCH DAMAGE.\n#\n# This means you may do anything you want with this code, except claim you\n# wrote it. Also, if it breaks you get to keep both pieces.\n#\n# 02-DEC-98 Started writing code.\n# 22-NOV-99 Changed garbage collection to a better algorithm.\n# 28-AUG-01 Added logic to deal with exceptions in user callbacks.\n# 02-SEP-01 Fixed hang when closing last node.\n# 07-SEP-01 Added binding tracking so nodes got garbage-collected.\n# Also fixed subclass call to initialize Canvas to properly deal\n# with variable arguments and keyword arguments.\n# 11-SEP-01 Bugfix for unbinding code.\n# 13-OCT-01 Added delete & insert methods for nodes (by email request).\n# LOTS of code cleanup.\n# Changed leading double underscores to PVT nomenclature.\n# Added ability to pass Node subclass to Tree constructor.\n# Removed after_callback since subclassing Node is better idea.\n# 15-OCT-01 Finally added drag'n'drop support. It consisted of a simple\n# change to the Node PVT_click method, and addition of logic like\n# the example in Tkdnd.py. It took 3 days to grok the Tkdnd\n# example and 2 hours to make the code changes. Plus another 1/2\n# day to get a working where() function.\n# 16-OCT-01 Incorporated fixes to delete() and dnd_commit() bugs by\n# Laurent Claustre .\n# 17-OCT-01 Added find_full_id() and cursor_node() methods.\n# 18-OCT-01 Fixes to delete() on root during collapse and with\n# drag-in-progress flag by Laurent Claustre .\n# 10-FEB-02 Fix to prev_visible() by Nicolas Pascal .\n# Fixes which made insert_before()/insert_after() actually work.\n# Also added expand/collapse indicators like Internet Explorer\n# as requested by Nicolas.\n# 11-FEB-02 Another fix to prev_visible(). It works this time. Honest.\n# 31-MAY-02 Added documentation strings so the new PYthon 2.2 help function\n# is a little more useful.\n# 19-AUG-02 Minor fix to eliminate crash in \"treedemo-icons.py\" caused by\n# referencing expand/collapse indicators when lines are turned off.\n# 15-OCT-02 Used new idiom for calling Canvas superclass.\n# 18-NOV-02 Fixed bug discovered by Amanjit Gill , where\n# I didn't pass \"master\" properly to the Canvas superclass. Sigh.\n# One step forward, one step back.\n\nimport Tkdnd\nfrom Tkinter import *\n\n#------------------------------------------------------------------------------\ndef report_callback_exception():\n \"\"\"report exception on sys.stderr.\"\"\"\n import traceback\n import sys\n \n sys.stderr.write(\"Exception in Tree control callback\\n\")\n traceback.print_exc()\n \n#------------------------------------------------------------------------------\nclass Struct:\n \"\"\"Helper object for add_node() method\"\"\"\n def __init__(self):\n pass\n\n#------------------------------------------------------------------------------\nclass Node:\n \"\"\"Tree helper class that's instantiated for each element in the tree. It\n has several useful attributes:\n parent_node - immediate parent node\n id - id assigned at creation\n expanded_icon - image displayed when folder is expanded to display\n children\n collapsed_icon - image displayed when node is not a folder or folder is\n collapsed.\n parent_widget - reference to tree widget that contains node.\n expandable_flag - is true when node is a folder that may be expanded or\n collapsed.\n expanded_flag - true to indicate node is currently expanded.\n h_line - canvas line to left of node image.\n v_line - canvas line below node image that connects children.\n indic - expand/collapse canvas image.\n label - canvas text label\n symbol - current canvas image\n\n Please note that methods prefixed PVT_* are not meant to be used by\n client programs.\"\"\"\n \n def __init__(self, parent_node, id, collapsed_icon, x, y,\n parent_widget=None, expanded_icon=None, label=None,\n expandable_flag=0):\n \"\"\"Create node and initialize it. This also displays the node at the\n given position on the canvas, and binds mouseclicks.\"\"\"\n # immediate parent node\n self.parent_node=parent_node\n # internal name used to manipulate things\n self.id=id\n # bitmaps to be displayed\n self.expanded_icon=expanded_icon\n self.collapsed_icon=collapsed_icon\n # tree widget we belong to\n if parent_widget:\n self.widget=parent_widget\n else:\n self.widget=parent_node.widget\n # for speed\n sw=self.widget\n # our list of child nodes\n self.child_nodes=[]\n # flag that node can be expanded\n self.expandable_flag=expandable_flag\n self.expanded_flag=0\n # add line\n if parent_node and sw.line_flag:\n self.h_line=sw.create_line(x, y, x-sw.dist_x, y)\n else:\n self.h_line=None\n self.v_line=None\n # draw approprate image\n self.symbol=sw.create_image(x, y, image=self.collapsed_icon)\n # add expand/collapse indicator\n self.indic=None\n if expandable_flag and sw.line_flag and sw.plus_icon and sw.minus_icon:\n self.indic=sw.create_image(x-sw.dist_x, y, image=sw.plus_icon)\n # add label\n self.label=sw.create_text(x+sw.text_offset, y, text=label, anchor='w')\n # single-click to expand/collapse\n if self.indic:\n sw.tag_bind(self.indic, '<1>', self.PVT_click)\n else:\n sw.tag_bind(self.symbol, '<1>', self.PVT_click)\n # for drag'n'drop target detection\n sw.tag_bind(self.symbol, '', self.PVT_enter)\n sw.tag_bind(self.label, '', self.PVT_enter)\n\n # for testing (gotta make sure nodes get properly GC'ed)\n #def __del__(self):\n # print self.full_id(), 'deleted'\n\n # ----- PUBLIC METHODS -----\n def set_collapsed_icon(self, icon):\n \"\"\"Set node's collapsed image\"\"\"\n self.collapsed_icon=icon\n if not self.expanded_flag:\n self.widget.itemconfig(self.symbol, image=icon)\n\n def set_expanded_icon(self, icon):\n \"\"\"Set node's expanded image\"\"\"\n self.expanded_icon=icon\n if self.expanded_flag:\n self.widget.itemconfig(self.symbol, image=icon)\n\n def parent(self):\n \"\"\"Return node's parent node\"\"\"\n return self.parent_node\n\n def prev_sib(self):\n \"\"\"Return node's previous sibling (the child immediately above it)\"\"\"\n i=self.parent_node.child_nodes.index(self)-1\n if i >= 0:\n return self.parent_node.child_nodes[i]\n else:\n return None\n\n def next_sib(self):\n \"\"\"Return node's next sibling (the child immediately below it)\"\"\"\n i=self.parent_node.child_nodes.index(self)+1\n if i < len(self.parent_node.child_nodes):\n return self.parent_node.child_nodes[i]\n else:\n return None\n \n def next_visible(self):\n \"\"\"Return next lower visible node\"\"\"\n n=self\n if n.child_nodes:\n # if you can go right, do so\n return n.child_nodes[0]\n while n.parent_node:\n # move to next sibling\n i=n.parent_node.child_nodes.index(n)+1\n if i < len(n.parent_node.child_nodes):\n return n.parent_node.child_nodes[i]\n # if no siblings, move to parent's sibling\n n=n.parent_node\n # we're at bottom\n return self\n \n def prev_visible(self):\n \"\"\"Return next higher visible node\"\"\"\n n=self\n if n.parent_node:\n i=n.parent_node.child_nodes.index(n)-1\n if i < 0:\n return n.parent_node\n else:\n j=n.parent_node.child_nodes[i]\n return j.PVT_last()\n else:\n return n\n \n def children(self):\n \"\"\"Return list of node's children\"\"\"\n return self.child_nodes[:]\n\n def get_label(self):\n \"\"\"Return string containing text of current label\"\"\"\n return self.widget.itemcget(self.label, 'text')\n\n def set_label(self, label):\n \"\"\"Set current text label\"\"\"\n self.widget.itemconfig(self.label, text=label)\n\n def expanded(self):\n \"\"\"Returns true if node is currently expanded, false otherwise\"\"\"\n return self.expanded_flag\n\n def expandable(self):\n \"\"\"Returns true if node can be expanded (i.e. if it's a folder)\"\"\"\n return self.expandable_flag\n \n def full_id(self):\n \"\"\"Return list of IDs of all parents and node ID\"\"\"\n if self.parent_node:\n return self.parent_node.full_id()+(self.id,)\n else:\n return (self.id,)\n\n def expand(self):\n \"\"\"Expand node if possible\"\"\"\n if not self.expanded_flag:\n self.PVT_set_state(1)\n \n def collapse(self):\n \"\"\"Collapse node if possible\"\"\"\n if self.expanded_flag:\n self.PVT_set_state(0)\n\n def delete(self, me_too=1):\n \"\"\"Delete node from tree. (\"me_too\" is a hack not to be used by\n external code, please!)\"\"\"\n sw=self.widget\n if not self.parent_node and me_too:\n # can't delete the root node\n raise ValueError, \"can't delete root node\"\n self.PVT_delete_subtree()\n # move everything up so that distance to next subnode is correct\n n=self.next_visible()\n x1, y1=sw.coords(self.symbol)\n x2, y2=sw.coords(n.symbol)\n if me_too:\n dist=y2-y1\n else:\n dist=y2-y1-sw.dist_y\n self.PVT_tag_move(-dist)\n n=self\n if me_too:\n if sw.pos == self:\n # move cursor if it points to current node\n sw.move_cursor(self.parent_node)\n self.PVT_unbind_all()\n sw.delete(self.symbol)\n sw.delete(self.label)\n sw.delete(self.h_line)\n sw.delete(self.v_line)\n sw.delete(self.indic)\n self.parent_node.child_nodes.remove(self)\n # break circular ref now, so parent may be GC'ed later\n n=self.parent_node\n self.parent_node=None\n n.PVT_cleanup_lines()\n n.PVT_update_scrollregion()\n\n def insert_before(self, nodes):\n \"\"\"Insert list of nodes as siblings before this node. Call parent\n node's add_node() function to generate the list of nodes.\"\"\"\n i=self.parent_node.child_nodes.index(self)\n self.parent_node.PVT_insert(nodes, i, self.prev_visible())\n \n def insert_after(self, nodes):\n \"\"\"Insert list of nodes as siblings after this node. Call parent\n node's add_node() function to generate the list of nodes.\"\"\"\n i=self.parent_node.child_nodes.index(self)+1\n self.parent_node.PVT_insert(nodes, i, self.PVT_last())\n \n def insert_children(self, nodes):\n \"\"\"Insert list of nodes as children of this node. Call node's\n add_node() function to generate the list of nodes.\"\"\"\n self.PVT_insert(nodes, 0, self)\n \n def toggle_state(self):\n \"\"\"Toggle node's state between expanded and collapsed, if possible\"\"\"\n if self.expandable_flag:\n if self.expanded_flag:\n self.PVT_set_state(0)\n else:\n self.PVT_set_state(1)\n \n # ----- functions for drag'n'drop support -----\n def PVT_enter(self, event):\n \"\"\"detect mouse hover for drag'n'drop\"\"\"\n self.widget.target=self\n \n def dnd_end(self, target, event):\n \"\"\"Notification that dnd processing has been ended. It DOES NOT imply\n that we've been dropped somewhere useful, we could have just been\n dropped into deep space and nothing happened to any data structures,\n or it could have been just a plain mouse-click w/o any dragging.\"\"\"\n if not self.widget.drag:\n # if there's been no dragging, it was just a mouse click\n self.widget.move_cursor(self)\n self.toggle_state()\n self.widget.drag=0\n\n # ----- PRIVATE METHODS (prefixed with \"PVT_\") -----\n # these methods are subject to change, so please try not to use them\n def PVT_last(self):\n \"\"\"Return bottom-most node in subtree\"\"\"\n n=self\n while n.child_nodes:\n n=n.child_nodes[-1]\n return n\n \n def PVT_find(self, search):\n \"\"\"Used by searching functions\"\"\"\n if self.id != search[0]:\n # this actually only goes tilt if root doesn't match\n return None\n if len(search) == 1:\n return self\n # get list of children IDs\n i=map(lambda x: x.id, self.child_nodes)\n # if there is a child that matches, search it\n try:\n return self.child_nodes[i.index(search[1])].PVT_find(search[1:])\n except:\n return None\n\n def PVT_insert(self, nodes, pos, below):\n \"\"\"Create and insert new children. \"nodes\" is list previously created\n via calls to add_list(). \"pos\" is index in the list of children where\n the new nodes are inserted. \"below\" is node which new children should\n appear immediately below.\"\"\"\n if not self.expandable_flag:\n raise TypeError, 'not an expandable node'\n # for speed\n sw=self.widget\n # expand and insert children\n children=[]\n self.expanded_flag=1\n sw.itemconfig(self.symbol, image=self.expanded_icon)\n if sw.minus_icon and sw.line_flag:\n sw.itemconfig(self.indic, image=sw.minus_icon)\n if len(nodes):\n # move stuff to make room\n below.PVT_tag_move(sw.dist_y*len(nodes))\n # get position of first new child\n xp, dummy=sw.coords(self.symbol)\n dummy, yp=sw.coords(below.symbol)\n xp=xp+sw.dist_x\n yp=yp+sw.dist_y\n # create vertical line\n if sw.line_flag and not self.v_line:\n self.v_line=sw.create_line(\n xp, yp,\n xp, yp+sw.dist_y*len(nodes))\n sw.tag_lower(self.v_line, self.symbol)\n n=sw.node_class\n for i in nodes:\n # add new subnodes, they'll draw themselves\n # this is a very expensive call\n children.append(\n n(parent_node=self, expandable_flag=i.flag, label=i.name,\n id=i.id, collapsed_icon=i.collapsed_icon,\n expanded_icon=i.expanded_icon, x=xp, y=yp))\n yp=yp+sw.dist_y\n self.child_nodes[pos:pos]=children\n self.PVT_cleanup_lines()\n self.PVT_update_scrollregion()\n sw.move_cursor(sw.pos)\n \n def PVT_set_state(self, state):\n \"\"\"Common code forexpanding/collapsing folders. It's not re-entrant,\n and there are certain cases in which we can be called again before\n we're done, so we use a mutex.\"\"\"\n while self.widget.spinlock:\n pass\n self.widget.spinlock=1\n # expand & draw our subtrees\n if state:\n self.child_nodes=[]\n self.widget.new_nodes=[]\n if self.widget.get_contents_callback:\n # this callback needs to make multiple calls to add_node()\n try:\n self.widget.get_contents_callback(self)\n except:\n report_callback_exception()\n self.PVT_insert(self.widget.new_nodes, 0, self)\n # collapse and delete subtrees\n else:\n self.expanded_flag=0\n self.widget.itemconfig(self.symbol, image=self.collapsed_icon)\n if self.indic:\n self.widget.itemconfig(self.indic, image=self.widget.plus_icon)\n self.delete(0)\n # release mutex\n self.widget.spinlock=0\n\n def PVT_cleanup_lines(self):\n \"\"\"Resize connecting lines\"\"\"\n if self.widget.line_flag:\n n=self\n while n:\n if n.child_nodes:\n x1, y1=self.widget.coords(n.symbol)\n x2, y2=self.widget.coords(n.child_nodes[-1].symbol)\n self.widget.coords(n.v_line, x1, y1, x1, y2)\n n=n.parent_node\n\n def PVT_update_scrollregion(self):\n \"\"\"Update scroll region for new size\"\"\"\n x1, y1, x2, y2=self.widget.bbox('all')\n self.widget.configure(scrollregion=(x1, y1, x2+5, y2+5))\n\n def PVT_delete_subtree(self):\n \"\"\"Recursively delete subtree & clean up cyclic references to make\n garbage collection happy\"\"\"\n sw=self.widget\n sw.delete(self.v_line)\n self.v_line=None\n for i in self.child_nodes:\n # delete node's subtree, if any\n i.PVT_delete_subtree()\n i.PVT_unbind_all()\n # delete widgets from canvas\n sw.delete(i.symbol)\n sw.delete(i.label)\n sw.delete(i.h_line)\n sw.delete(i.v_line)\n sw.delete(i.indic)\n # break circular reference\n i.parent_node=None\n # move cursor if it's in deleted subtree\n if sw.pos in self.child_nodes:\n sw.move_cursor(self)\n # now subnodes will be properly garbage collected\n self.child_nodes=[]\n \n def PVT_unbind_all(self):\n \"\"\"Unbind callbacks so node gets garbage-collected. This wasn't easy\n to figure out the proper way to do this. See also tag_bind() for the\n Tree widget itself.\"\"\"\n for j in (self.symbol, self.label, self.indic, self.h_line,\n self.v_line):\n for k in self.widget.bindings.get(j, ()):\n self.widget.tag_unbind(j, k[0], k[1])\n\n def PVT_tag_move(self, dist):\n \"\"\"Move everything below current icon, to make room for subtree using\n the Disney magic of item tags. This is the secret of making\n everything as fast as it is.\"\"\"\n # mark everything below current node as movable\n bbox1=self.widget.bbox(self.widget.root.symbol, self.label)\n bbox2=self.widget.bbox('all')\n self.widget.dtag('move')\n self.widget.addtag('move', 'overlapping', \n bbox2[0], bbox1[3], bbox2[2], bbox2[3])\n # untag cursor & node so they don't get moved too\n self.widget.dtag(self.widget.cursor_box, 'move')\n self.widget.dtag(self.symbol, 'move')\n self.widget.dtag(self.label, 'move')\n # now do the move of all the tagged objects\n self.widget.move('move', 0, dist)\n \n def PVT_click(self, event):\n \"\"\"Handle mouse clicks by kicking off possible drag'n'drop\n processing\"\"\"\n if self.widget.drop_callback:\n if Tkdnd.dnd_start(self, event):\n x1, y1, x2, y2=self.widget.bbox(self.symbol)\n self.x_off=(x1-x2)/2\n self.y_off=(y1-y2)/2\n else:\n # no callback, don't bother with drag'n'drop\n self.widget.drag=0\n self.dnd_end(None, None)\n\n#------------------------------------------------------------------------------\nclass Tree(Canvas):\n # do we have enough possible arguments?!?!?!\n def __init__(self, master, root_id, root_label='',\n get_contents_callback=None, dist_x=15, dist_y=15,\n text_offset=10, line_flag=1, expanded_icon=None,\n collapsed_icon=None, regular_icon=None, plus_icon=None,\n minus_icon=None, node_class=Node, drop_callback=None,\n *args, **kw_args):\n # pass args to superclass (new idiom from Python 2.2)\n Canvas.__init__(self, master, *args, **kw_args)\n \n # this allows to subclass Node and pass our class in\n self.node_class=node_class\n # keep track of node bindings\n self.bindings={}\n # cheap mutex spinlock\n self.spinlock=0\n # flag to see if there's been any d&d dragging\n self.drag=0\n # default images (BASE64-encoded GIF files)\n if expanded_icon == None:\n self.expanded_icon=PhotoImage(\n data='R0lGODlhEAANAKIAAAAAAMDAwICAgP//////ADAwMAAAAAAA' \\\n 'ACH5BAEAAAEALAAAAAAQAA0AAAM6GCrM+jCIQamIbw6ybXNSx3GVB' \\\n 'YRiygnA534Eq5UlO8jUqLYsquuy0+SXap1CxBHr+HoBjoGndDpNAAA7')\n else:\n self.expanded_icon=expanded_icon\n if collapsed_icon == None:\n self.collapsed_icon=PhotoImage(\n data='R0lGODlhDwANAKIAAAAAAMDAwICAgP//////ADAwMAAAAAAA' \\\n 'ACH5BAEAAAEALAAAAAAPAA0AAAMyGCHM+lAMMoeAT9Jtm5NDKI4Wo' \\\n 'FXcJphhipanq7Kvu8b1dLc5tcuom2foAQQAyKRSmQAAOw==')\n else:\n self.collapsed_icon=collapsed_icon\n if regular_icon == None:\n self.regular_icon=PhotoImage(\n data='R0lGODlhCwAOAJEAAAAAAICAgP///8DAwCH5BAEAAAMALAAA' \\\n 'AAALAA4AAAIphA+jA+JuVgtUtMQePJlWCgSN9oSTV5lkKQpo2q5W+' \\\n 'wbzuJrIHgw1WgAAOw==')\n else:\n self.regular_icon=regular_icon\n if plus_icon == None:\n self.plus_icon=PhotoImage(\n data='R0lGODdhCQAJAPEAAAAAAH9/f////wAAACwAAAAACQAJAAAC' \\\n 'FIyPoiu2sJyCyoF7W3hxz850CFIA\\nADs=')\n else:\n self.plus_icon=plus_icon\n if minus_icon == None:\n self.minus_icon=PhotoImage(\n data='R0lGODdhCQAJAPEAAAAAAH9/f////wAAACwAAAAACQAJAAAC' \\\n 'EYyPoivG614LAlg7ZZbxoR8UADs=')\n else:\n self.minus_icon=minus_icon\n # horizontal distance that subtrees are indented\n self.dist_x=dist_x\n # vertical distance between rows\n self.dist_y=dist_y\n # how far to offset text label\n self.text_offset=text_offset\n # flag controlling connecting line display\n self.line_flag=line_flag\n # called just before subtree expand/collapse\n self.get_contents_callback=get_contents_callback\n # called after drag'n'drop\n self.drop_callback=drop_callback\n # create root node to get the ball rolling\n self.root=node_class(parent_node=None, label=root_label,\n id=root_id, expandable_flag=1,\n collapsed_icon=self.collapsed_icon,\n expanded_icon=self.expanded_icon,\n x=dist_x, y=dist_y, parent_widget=self)\n # configure for scrollbar(s)\n x1, y1, x2, y2=self.bbox('all') \n self.configure(scrollregion=(x1, y1, x2+5, y2+5))\n # add a cursor\n self.cursor_box=self.create_rectangle(0, 0, 0, 0)\n self.move_cursor(self.root)\n # make it easy to point to control\n self.bind('', self.PVT_mousefocus)\n # totally arbitrary yet hopefully intuitive default keybindings\n # stole 'em from ones used by microsoft tree control\n # page-up/page-down\n self.bind('', self.pagedown)\n self.bind('', self.pageup)\n # arrow-up/arrow-down\n self.bind('', self.next)\n self.bind('', self.prev)\n # arrow-left/arrow-right\n self.bind('', self.ascend)\n # (hold this down and you expand the entire tree)\n self.bind('', self.descend)\n # home/end\n self.bind('', self.first)\n self.bind('', self.last)\n # space bar\n self.bind('', self.toggle)\n\n # ----- PRIVATE METHODS (prefixed with \"PVT_\") -----\n # these methods are subject to change, so please try not to use them\n def PVT_mousefocus(self, event):\n \"\"\"Soak up event argument when moused-over\"\"\"\n self.focus_set()\n \n # ----- PUBLIC METHODS -----\n def tag_bind(self, tag, seq, *args, **kw_args):\n \"\"\"Keep track of callback bindings so we can delete them later. I\n shouldn't have to do this!!!!\"\"\"\n # pass args to superclass\n func_id=apply(Canvas.tag_bind, (self, tag, seq)+args, kw_args)\n # save references\n self.bindings[tag]=self.bindings.get(tag, [])+[(seq, func_id)]\n\n def add_list(self, list=None, name=None, id=None, flag=0,\n expanded_icon=None, collapsed_icon=None):\n \"\"\"Add node construction info to list\"\"\"\n n=Struct()\n n.name=name\n n.id=id\n n.flag=flag\n if collapsed_icon:\n n.collapsed_icon=collapsed_icon\n else:\n if flag:\n # it's expandable, use closed folder icon\n n.collapsed_icon=self.collapsed_icon\n else:\n # it's not expandable, use regular file icon\n n.collapsed_icon=self.regular_icon\n if flag:\n if expanded_icon:\n n.expanded_icon=expanded_icon\n else:\n n.expanded_icon=self.expanded_icon\n else:\n # not expandable, don't need an icon\n n.expanded_icon=None\n if list == None:\n list=[]\n list.append(n)\n return list\n\n def add_node(self, name=None, id=None, flag=0, expanded_icon=None,\n collapsed_icon=None):\n \"\"\"Add a node during get_contents_callback()\"\"\"\n self.add_list(self.new_nodes, name, id, flag, expanded_icon,\n collapsed_icon)\n\n def find_full_id(self, search):\n \"\"\"Search for a node\"\"\"\n return self.root.PVT_find(search)\n \n def cursor_node(self, search):\n \"\"\"Return node under cursor\"\"\"\n return self.pos\n \n def see(self, *items):\n \"\"\"Scroll (in a series of nudges) so items are visible\"\"\"\n x1, y1, x2, y2=apply(self.bbox, items)\n while x2 > self.canvasx(0)+self.winfo_width():\n old=self.canvasx(0)\n self.xview('scroll', 1, 'units')\n # avoid endless loop if we can't scroll\n if old == self.canvasx(0):\n break\n while y2 > self.canvasy(0)+self.winfo_height():\n old=self.canvasy(0)\n self.yview('scroll', 1, 'units')\n if old == self.canvasy(0):\n break\n # done in this order to ensure upper-left of object is visible\n while x1 < self.canvasx(0):\n old=self.canvasx(0)\n self.xview('scroll', -1, 'units')\n if old == self.canvasx(0):\n break\n while y1 < self.canvasy(0):\n old=self.canvasy(0)\n self.yview('scroll', -1, 'units')\n if old == self.canvasy(0):\n break\n \n def move_cursor(self, node):\n \"\"\"Move cursor to node\"\"\"\n self.pos=node\n x1, y1, x2, y2=self.bbox(node.symbol, node.label)\n self.coords(self.cursor_box, x1-1, y1-1, x2+1, y2+1)\n self.see(node.symbol, node.label)\n \n def toggle(self, event=None):\n \"\"\"Expand/collapse subtree\"\"\"\n self.pos.toggle_state()\n\n def next(self, event=None):\n \"\"\"Move to next lower visible node\"\"\"\n self.move_cursor(self.pos.next_visible())\n \n def prev(self, event=None):\n \"\"\"Move to next higher visible node\"\"\"\n self.move_cursor(self.pos.prev_visible())\n\n def ascend(self, event=None):\n \"\"\"Move to immediate parent\"\"\"\n if self.pos.parent_node:\n # move to parent\n self.move_cursor(self.pos.parent_node)\n\n def descend(self, event=None):\n \"\"\"Move right, expanding as we go\"\"\"\n if self.pos.expandable_flag:\n self.pos.expand()\n if self.pos.child_nodes:\n # move to first subnode\n self.move_cursor(self.pos.child_nodes[0])\n return\n # if no subnodes, move to next sibling\n self.next()\n\n def first(self, event=None):\n \"\"\"Go to root node\"\"\"\n # move to root node\n self.move_cursor(self.root)\n\n def last(self, event=None):\n \"\"\"Go to last visible node\"\"\"\n # move to bottom-most node\n self.move_cursor(self.root.PVT_last())\n\n def pageup(self, event=None):\n \"\"\"Previous page\"\"\"\n n=self.pos\n j=self.winfo_height()/self.dist_y\n for i in range(j-3):\n n=n.prev_visible()\n self.yview('scroll', -1, 'pages')\n self.move_cursor(n)\n\n def pagedown(self, event=None):\n \"\"\"Next page\"\"\"\n n=self.pos\n j=self.winfo_height()/self.dist_y\n for i in range(j-3):\n n=n.next_visible()\n self.yview('scroll', 1, 'pages')\n self.move_cursor(n)\n \n # ----- functions for drag'n'drop support -----\n def where(self, event):\n \"\"\"Determine drag location in canvas coordinates. event.x & event.y\n don't seem to be what we want.\"\"\"\n # where the corner of the canvas is relative to the screen:\n x_org=self.winfo_rootx()\n y_org=self.winfo_rooty()\n # where the pointer is relative to the canvas widget,\n # including scrolling\n x=self.canvasx(event.x_root-x_org)\n y=self.canvasy(event.y_root-y_org)\n return x, y\n \n def dnd_accept(self, source, event):\n \"\"\"Accept dnd messages, i.e. we're a legit drop target, and we do\n implement d&d functions.\"\"\"\n self.target=None\n return self\n\n def dnd_enter(self, source, event):\n \"\"\"Get ready to drag or drag has entered widget (create drag\n object)\"\"\"\n # this flag lets us know there's been drag motion\n self.drag=1\n x, y=self.where(event)\n x1, y1, x2, y2=source.widget.bbox(source.symbol, source.label)\n dx, dy=x2-x1, y2-y1\n # create dragging icon\n if source.expanded_flag:\n self.dnd_symbol=self.create_image(x, y,\n image=source.expanded_icon)\n else:\n self.dnd_symbol=self.create_image(x, y,\n image=source.collapsed_icon)\n self.dnd_label=self.create_text(x+self.text_offset, y, \n text=source.get_label(),\n justify='left',\n anchor='w')\n\n def dnd_motion(self, source, event):\n \"\"\"Move drag icon\"\"\"\n self.drag=1\n x, y=self.where(event)\n x1, y1, x2, y2=self.bbox(self.dnd_symbol, self.dnd_label)\n self.move(self.dnd_symbol, x-x1+source.x_off, y-y1+source.y_off)\n self.move(self.dnd_label, x-x1+source.x_off, y-y1+source.y_off)\n\n def dnd_leave(self, source, event):\n \"\"\"Finish dragging or drag has left widget (destroy drag object)\"\"\"\n self.delete(self.dnd_symbol)\n self.delete(self.dnd_label)\n\n def dnd_commit(self, source, event):\n \"\"\"Object has been dropped here\"\"\"\n # call our own dnd_leave() to clean up\n self.dnd_leave(source, event)\n # process pending events to detect target node\n # update_idletasks() doesn't do the trick if source & target are\n # on different widgets\n self.update()\n if not self.target:\n # no target node\n return\n # we must update data structures based on the drop\n if self.drop_callback:\n try:\n # called with dragged node and target node\n # this is where a file manager would move the actual file\n # it must also move the nodes around as it wishes\n self.drop_callback(source, self.target)\n except:\n report_callback_exception()\n\n#------------------------------------------------------------------------------\n# the good 'ol test/demo code\nif __name__ == '__main__':\n import os\n import sys\n\n # default routine to get contents of subtree\n # supply this for a different type of app\n # argument is the node object being expanded\n # should call add_node()\n def get_contents(node):\n path=apply(os.path.join, node.full_id())\n for filename in os.listdir(path):\n full=os.path.join(path, filename)\n name=filename\n folder=0\n if os.path.isdir(full):\n # it's a directory\n folder=1\n elif not os.path.isfile(full):\n # but it's not a file\n name=name+' (special)'\n if os.path.islink(full):\n # it's a link\n name=name+' (link to '+os.readlink(full)+')'\n node.widget.add_node(name=name, id=filename, flag=folder)\n\n root=Tk()\n root.title(os.path.basename(sys.argv[0]))\n tree=os.sep\n if sys.platform == 'win32':\n # we could call the root \"My Computer\" and mess with get_contents()\n # to return \"A:\", \"B:\", \"C:\", ... etc. as it's children, but that\n # would just be terminally cute and I'd have to shoot myself\n tree='C:'+os.sep\n\n # create the control\n t=Tree(master=root,\n root_id=tree,\n root_label=tree,\n get_contents_callback=get_contents,\n width=300)\n t.grid(row=0, column=0, sticky='nsew')\n\n # make expandable\n root.grid_rowconfigure(0, weight=1)\n root.grid_columnconfigure(0, weight=1)\n\n # add scrollbars\n sb=Scrollbar(root)\n sb.grid(row=0, column=1, sticky='ns')\n t.configure(yscrollcommand=sb.set)\n sb.configure(command=t.yview)\n\n sb=Scrollbar(root, orient=HORIZONTAL)\n sb.grid(row=1, column=0, sticky='ew')\n t.configure(xscrollcommand=sb.set)\n sb.configure(command=t.xview)\n\n # must get focus so keys work for demo\n t.focus_set()\n\n # we could do without this, but it's nice and friendly to have\n Button(root, text='Quit', command=root.quit).grid(row=2, column=0,\n columnspan=2)\n\n # expand out the root\n t.root.expand()\n \n root.mainloop()\n" }, { "path": "src/ccg2xml/arabic.ccg", "content": "#############################################################\n# #\n# arabic.ccg #\n# #\n#############################################################\n\n# Author: Ben Wing \n# Date: April 2006\n\n# This is a grammar for a fragment of Arabic. It's particularly\n# useful for demonstrating the extended use of macros to handle\n# complicated morphological inflections.\n\n# See the `tiny' grammar (tiny.ccg) for more info about the format\n# of this file.\n\nfeature {\n CASE<2>: nom, acc, gen;\n NUM<2>: sg, du, pl;\n GEND<2>: m, f;\n STATE<2>: cons, non-cons {indef, def};\n ANIM<2>: hum, nonhum;\n PERS<2>: 1st, 2nd, 3rd;\n RESUMPTIVE<2>: nonres, res;\n SEM-NUM: sg-X, du-X, pl-X;\n SEM-PERS: 1st-X, 2nd-X, 3rd-X;\n TENSE: past, pres;\n MOOD: indic, subj, juss;\n\n # Here's a more complicated hierarchy, from the original tiny grammar.\n ontology: sem-obj {\n phys-obj {\n animate-being {\n person\n },\n thing\n },\n situation {\n change {\n action\n },\n state\n }\n };\n}\n\nrule {\n no typeraise;\n typeraise +: n => s;\n typeraise - $: n => s;\n typeraise - $: pp => s;\n typeraise - $: pp/n => s;\n typechange: s$1 | n[nom] => s$1 ;\n typechange: n<~2>[cons] => n<2>[3rd,def] /* n[gen,def] ;\n typechange: n<~2>[cons] => n<2>[3rd,indef] /* n[gen,indef] ;\n}\n\n##########################################################################\n# Morphological entries #\n# (morph.xml) #\n##########################################################################\n\nword wa:Conj; # \"and\"\nword anna:Comp; # \"that\", introducing sentential complements\nword inna:Comp; # same, but only after the verb qaal \"say\"\n\nword maa:InterrogPro(thing): 3rd; # \"what\"\nword man:InterrogPro(person): 3rd; # \"who\"\n\nword li:Prep; # \"what\"\nword fii:Prep; # \"who\"\n\n# This word means \"this\". \nword haadhaa {\n *: sg, m;\n haadhihi: sg, f;\n ha_ulaahi: pl;\n\n # Bizarrely, this word declines for case only in the dual.\n haadhaani: du, m, nom;\n haadhayni: du, m, acc;\n haadhayni: du, m, gen;\n\n haataani: du, f, nom;\n haatayni: du, f, acc;\n haatayni: du, f, gen;\n}\n\n# This word means \"that\". \nword dhaalik {\n *: sg, m;\n tilka: sg, f;\n ulaa_ika: pl;\n\n # Bizarrely, this word declines for case only in the dual.\n dhaanika: du, m, nom;\n dhaynika: du, m, acc;\n dhaynika: du, m, gen;\n\n taanika: du, f, nom;\n taynika: du, f, acc;\n taynika: du, f, gen;\n}\n\n# This is the relative pronoun.\nword al-ladhii {\n *: sg, m;\n al-latii: sg, f;\n al-ladhiina: pl, m;\n al-laati: pl, f;\n\n # Bizarrely, this word declines for case only in the dual.\n al-ladhaani: du, m, nom;\n al-ladhayni: du, m, acc;\n al-ladhayni: du, m, gen;\n\n al-lataani: du, f, nom;\n al-latayni: du, f, acc;\n al-latayni: du, f, gen;\n}\n\nword pro:Pro {\n ana: 1st, 1st-X, sg, sg-X;\n anta: 2nd, 2nd-X, sg, sg-X, m;\n anti: 2nd, 2nd-X, sg, sg-X, f;\n huwa: 3rd, 3rd-X, sg, sg-X, m;\n hiya: 3rd, 3rd-X, sg, sg-X, f;\n naHnu: 1st, 1st-X, pl, pl-X;\n antun: 2nd, 2nd-X, pl, pl-X, m;\n antunna: 2nd, 2nd-X, pl, pl-X, f;\n hum: 3rd, 3rd-X, pl, pl-X, m;\n hunna: 3rd, 3rd-X, pl, pl-X, f;\n}\n\nword ii:: 1st, 1st-X, sg, sg-X;\nword nii:: 1st, 1st-X, sg, sg-X;\nword ka:: 2nd, 2nd-X, sg, sg-X, m;\nword ki:: 2nd, 2nd-X, sg, sg-X, f;\nword hu:: 3rd, 3rd-X, sg, sg-X, m;\nword haa:: 3rd, 3rd-X, sg, sg-X, f;\nword naa:: 1st, 1st-X, pl, pl-X;\nword kum:: 2nd, 2nd-X, pl, pl-X, m;\nword kunna:: 2nd, 2nd-X, pl, pl-X, f;\nword hum:: 3rd, 3rd-X, pl, pl-X, m;\nword hunna:: 3rd, 3rd-X, pl, pl-X, f;\n\n#############################################\n# Nouns #\n#############################################\n\n# This shows how a reasonably complicated morphology can be accommodated.\n# It is certainly possible that some of this may (and probably should)\n# be offloaded into a separate morphology-processing engine. However,\n# even in that case there is often a good deal more to the lexicon.\n\n# We show a couple examples of complete paradigms, in order to make it\n# easier to understand what's going on below.\n\n# Here is a typical noun (kitaab \"book\") with a broken plural (kutub \"books\").\n# For nouns with broken plurals, the plural is typically declined like\n# the singular. Note that Arabic nouns are conjugated for three numbers\n# (singular, dual, plural), three cases (nominative, accusative, dative),\n# and three states (indefinite, definite, construct). (The construct state\n# is used for nouns that are modified by other nouns -- e.g. \"book\" in\n# \"the book of Mary\".)\n\n# Form\t\tNominative\tAccusative\tDative\n# ---------------------------------------------------------------\n# sg.indef\tkitaabun\tkitaaban\tkitaabin\n# sg.def\tal-kitaabu\tal-kitaaba\tal-kitaabi\n# sg.cons\tkitaabu\t\tkitaaba\t\tkitaabi\n# \n# du.indef\tkitaabaani\tkitaabayni\tkitaabayni\n# du.def\tal-kitaabaani\tal-kitaabayni\tal-kitaabayni\n# du.cons\tkitaabaa\tkitaabay\tkitaabay\n# \n# pl.indef\tkutubun\t\tkutuban\t\tkutubin\n# pl.def\tal-kutubu\tal-kutuba\tal-kutubi\n# pl.cons\tkutubu\t\tkutuba\t\tkutubi\n\n# Here is a typical noun (mudarris \"teacher\") with a different kind of\n# plural, a so-called \"strong masculine plural\", which has its own declension.\n\n# Form\t\tNominative\tAccusative\tDative\n# ---------------------------------------------------------------\n# sg.indef\tmudarrisun\tmudarrisan\tmudarrisin\n# sg.def\tal-mudarrisu\tal-mudarrisa\tal-mudarrisi\n# sg.cons\tmudarrisu\tmudarrisa\tmudarrisi\n#\n# du.indef\tmudarrisaani\tmudarrisayni\tmudarrisayni\n# du.def\tal-mudarrisaani\tal-mudarrisayni\tal-mudarrisayni\n# du.cons\tmudarrisaa\tmudarrisay\tmudarrisay\n#\n# pl.indef\tmudarrisuuna\tmudarrisiina\tmudarrisiina\n# pl.def\tal-mudarrisuuna\tal-mudarrisiina\tal-mudarrisiina\n# pl.cons\tmudarrisuu\tmudarrisii\tmudarrisii\n\n# Here, we make heavy use of macros.\n\n# This macro says: Every time an expression of the form\n# three-form-decl(...) occurs, replace it with the text that comes after.\n# The parameters will be substituted into the text. The braces that\n# denote the macro's text do *NOT* form part of the text that is substituted.\n# Note that macro substitutions are processed recursively: If the text\n# of a macro substitution contains calls to other macros, they will also\n# be processed. This makes \"inheritance\" very easy to implement.\n\n# This macro is used for a particular paradigm corresponding to a\n# particular number of a word.\n\ndef three-different-form-decl(indef-form, def-form, cons-form,\n\t\t\t indef-nom, indef-acc, indef-gen,\n \t\t\t def-nom, def-acc, def-gen,\n\t \t cons-nom, cons-acc, cons-gen,\n\t \t morph-num, sem-num, gend) {\n indef-form.indef-nom: morph-num, sem-num, gend, nom, indef;\n indef-form.indef-acc: morph-num, sem-num, gend, acc, indef;\n indef-form.indef-gen: morph-num, sem-num, gend, gen, indef;\n\n add-al(def-form.def-nom): morph-num, sem-num, gend, nom, def;\n add-al(def-form.def-acc): morph-num, sem-num, gend, acc, def;\n add-al(def-form.def-gen): morph-num, sem-num, gend, gen, def;\n\n cons-form.cons-nom: morph-num, sem-num, gend, nom, cons;\n cons-form.cons-acc: morph-num, sem-num, gend, acc, cons;\n cons-form.cons-gen: morph-num, sem-num, gend, gen, cons;\n}\n\n# It's questionable whether we should do this. This assimilates al-\n# to a following coronal consonant, e.g. ar-rajul, as-sigaara,\n# ath-thalj, an-nuur, aDH-DHuhr, etc.\ndef add-al(form) regsub('^al-([std]h|DH|[tdszrnTDSZL])', 'a\\1-\\1', al-.form)\n\ndef three-form-decl(form, indef-nom, indef-acc, indef-gen,\n\t\t def-nom, def-acc, def-gen,\n\t\t cons-nom, cons-acc, cons-gen,\n\t\t morph-num, sem-num, gend) {\n three-different-form-decl(form, form, form,\n\t\t\t indef-nom, indef-acc, indef-gen,\n \t\t\t def-nom, def-acc, def-gen,\n\t \t cons-nom, cons-acc, cons-gen,\n\t \t morph-num, sem-num, gend)\n}\n\n# Using the above macro, we create two more macros to handle two common\n# paradigm types: Accusative and genitive are the same, and the\n# definite is either the same as the construct (two-form-decl-1) or\n# the same as the indefinite (two-form-decl-2).\n\ndef two-form-decl-1(form, non-cons-nom, non-cons-obl,\n\t\t cons-nom, cons-obl, morph-num, sem-num, gend) {\n three-form-decl(form, non-cons-nom, non-cons-obl, non-cons-obl,\n\t\t non-cons-nom, non-cons-obl, non-cons-obl,\n\t\t cons-nom, cons-obl, cons-obl,\n\t\t morph-num, sem-num, gend)\n}\n\ndef two-form-decl-2(form, indef-nom, indef-obl,\n\t\t non-indef-nom, non-indef-obl, morph-num, sem-num, gend) {\n three-form-decl(form, indef-nom, indef-obl, indef-obl,\n\t\t non-indef-nom, non-indef-obl, non-indef-obl,\n\t\t non-indef-nom, non-indef-obl, non-indef-obl,\n\t\t morph-num, sem-num, gend)\n}\n\n# In turn we create macros for particular paradigms: strong masculine (\"uun\"),\n# strong feminine (\"aat\"), dual, and basic triptote (the paradigm for\n# \"kitaab\" above and, in general, most singulars).\n\n# Note that an alternative to using braces is to put the macro text on\n# the same line as the `def' part of the macro (backslashes can be used\n# to join multiple lines together).\n\ndef uun-plural(form) two-form-decl-1(form, uuna, iina, uu, ii, pl, pl-X, m)\ndef aat-plural(form) \\\n two-form-decl-2(form, aatun, aatin, aatu, aati, pl, pl-X, f)\ndef dual(form, gend) \\\n two-form-decl-1(form, aani, ayni, aa, ay, du, du-X, gend)\ndef triptote(form, morph-num, sem-num, gend) \\\n three-form-decl(form, un, an, in, u, a, i, u, a, i, morph-num, sem-num, gend)\n\n# Here we define macros for full paradigms for words. Note how semicolons\n# are not used, because they are supplied by the macro text itself.\n# (Consult the text for three-form-decl() above, and remember that the\n# braces denoting the macro text are not actually part of the text. This\n# means that if you really want braces as the outermost thing in some\n# macro text, you'll need to supply two levels of braces.)\n\ndef thing(sing, plur) {\n word sing:N(thing) {\n triptote(sing, sg, sg-X, m)\n dual(sing, m)\n triptote(plur, sg, pl-X, f)\n }\n}\n\ndef fem-thing(sing, plur) {\n word sing:N(thing) {\n triptote(sing.t, sg, sg-X, f)\n dual(sing.t, f)\n triptote(plur, sg, pl-X, f)\n }\n}\n\ndef person(sing, plur, gend) {\n word sing:N(person) {\n triptote(sing, sg, sg-X, gend)\n dual(sing, gend)\n triptote(plur, pl, pl-X, gend)\n }\n}\n\ndef male(sing, plur) person(sing, plur, m)\ndef female(sing, plur) person(sing, plur, f)\n\ndef strong-male(sing) {\n word sing:N(person) {\n triptote(sing, sg, sg-X, m)\n dual(sing, m)\n uun-plural(sing)\n }\n}\n\n# Here we define the actual words. Note how short these definitions are,\n# specifying only what's unpredictable.\n\nthing(kitaab, kutub)\nthing(waqt, _awqaat)\nthing(Harf, Huruuf)\nthing(dars, duruus)\nthing(waqt, _awqaat)\nfem-thing(sigaara, sagaayir)\nfem-thing(madiina, mudun)\nmale(rajul, rijaal)\nmale(walad, _awlaad)\nmale(Taalib, Tullaab)\nfemale(bint, _abnaat)\nstrong-male(mudarris)\n\nword imra_a:N(person) {\n three-different-form-decl(imra_at, mar_at, imra_at,\n\t\t\t un, an, in, u, a, i, u, a, i,\n sg, sg-X, f)\n three-different-form-decl(imra_at, mar_at, imra_at,\n\t\t\t aani, ayni, ayni, aani, ayni, ayni, aa, ay, ay,\n du, du-X, f)\n triptote(nisaa_, pl, pl-X, f)\n}\n\ndef extended_construct_word(stem, plur) {\n word stem:N(person) {\n three-form-decl(stem, un, an, in, u, a, i, uu, aa, ii, sg, sg-X, m)\n dual(stem, m)\n triptote(plur, pl, pl-X, m)\n }\n}\n\nextended_construct_word(_ax, _ixwaan)\nextended_construct_word(_ab, _abnaa_)\n\n# Typical paradigms:\n\n# 1sg 'aktaa 'aktaa 'akta |\n# 2sg.m taktaa taktaa takta |\n# 2sg.f taktayna taktay taktay |\n# 3sg.m yaktaa yaktaa yakta |\n# 3sg.f taktaa taktaa takta |\n# 2du taktayaani taktayaa taktayaa |\n# 3du.m yaktayaani yaktayaa yaktayaa |\n# 3du.f taktayaani taktayaa taktayaa |\n# 1pl naktaa naktaa nakta |\n# 2pl.m taktawna taktaw taktaw |\n# 2pl.f taktayna taktayna taktayna |\n# 3pl.m yaktawna yaktaw yaktaw |\n# 3pl.f yaktayna yaktayna yaktayna |\n# \n# 1sg 'aktuu 'aktuwa 'aktu | 'aktii 'aktiya 'akti \n# 2sg.m taktuu taktuwa taktu | taktii taktiya takti \n# 2sg.f taktiina taktii taktii | taktiina taktii taktii \n# 3sg.m yaktuu yaktuwa yaktu | yaktii yaktiya yakti \n# 3sg.f taktuu taktuwa taktu | taktii taktiya takti \n# 2du taktuwaani taktuwaa taktuwaa | taktiyaani taktiyaa taktiyaa \n# 3du.m yaktuwaani yaktuwaa yaktuwaa | yaktiyaani yaktiyaa yaktiyaa \n# 3du.f taktuwaani taktuwaa taktuwaa | taktiyaani taktiyaa taktiyaa \n# 1pl naktuu naktuwa naktu | naktii naktiya nakti \n# 2pl.m taktuuna taktuu taktuu | taktuuna taktuu taktuu \n# 2pl.f taktuuna taktuuna taktuuna | taktiina taktiina taktiina \n# 3pl.m yaktuuna yaktuu yaktuu | yaktuuna yaktuu yaktuu \n# 3pl.f yaktuuna yaktuuna yaktuuna | yaktiina yaktiina yaktiina \n\n\ndef two-form-past(formv, formc) {\n formc.tu: past, 1st, sg;\n formc.ta: past, 2nd, m, sg;\n formc.ti: past, 2nd, f, sg;\n formv.a: past, 3rd, m, sg;\n formv.at: past, 3rd, f, sg;\n\n formc.tumaa: past, 2nd, du;\n formv.aa: past, 3rd, m, du;\n formv.ataa: past, 3rd, f, du;\n\n formc.naa: past, 1st, pl;\n formc.tum: past, 2nd, m, pl;\n formc.tunna: past, 2nd, f, pl;\n formv.uu: past, 3rd, m, pl;\n formc.na: past, 3rd, f, pl;\n}\n\ndef 3rd-weak-past-ay(form) {\n form.ay.tu: past, 1st, sg;\n form.ay.ta: past, 2nd, m, sg;\n form.ay.ti: past, 2nd, f, sg;\n form.aa: past, 3rd, m, sg;\n form.at: past, 3rd, f, sg;\n\n form.ay.tumaa: past, 2nd, du;\n form.ay.aa: past, 3rd, m, du;\n form.ataa: past, 3rd, f, du;\n\n form.ay.naa: past, 1st, pl;\n form.ay.tum: past, 2nd, m, pl;\n form.ay.tunna: past, 2nd, f, pl;\n form.aw: past, 3rd, m, pl;\n form.ay.na: past, 3rd, f, pl;\n}\n\ndef 3rd-weak-past-aw(form) {\n form.aw.tu: past, 1st, sg;\n form.aw.ta: past, 2nd, m, sg;\n form.aw.ti: past, 2nd, f, sg;\n form.aa: past, 3rd, m, sg;\n form.at: past, 3rd, f, sg;\n\n form.aw.tumaa: past, 2nd, du;\n form.aw.aa: past, 3rd, m, du;\n form.ataa: past, 3rd, f, du;\n\n form.aw.naa: past, 1st, pl;\n form.aw.tum: past, 2nd, m, pl;\n form.aw.tunna: past, 2nd, f, pl;\n form.aw: past, 3rd, m, pl;\n form.aw.na: past, 3rd, f, pl;\n}\n\ndef 3rd-weak-past-ii(form) {\n form.ii.tu: past, 1st, sg;\n form.ii.ta: past, 2nd, m, sg;\n form.ii.ti: past, 2nd, f, sg;\n form.iya: past, 3rd, m, sg;\n form.iyat: past, 3rd, f, sg;\n\n form.ii.tumaa: past, 2nd, du;\n form.iy.aa: past, 3rd, m, du;\n form.iy.ataa: past, 3rd, f, du;\n\n form.ii.naa: past, 1st, pl;\n form.ii.tum: past, 2nd, m, pl;\n form.ii.tunna: past, 2nd, f, pl;\n form.uu: past, 3rd, m, pl;\n form.ii.na: past, 3rd, f, pl;\n}\n\ndef strong-past(form) two-form-past(form, form)\n\n# In general, almost all Arabic present-tense verbs of a particular mood\n# can be defined using five forms. Verbs with a hamza in the first radical\n# have a problem in the first-singular; ideally this should be handled\n# automatically using a regexp or something of that sort, but we don't have\n# such support currently, so we use an optional param.\n\ndef gen-pres(mood, fsing, fsing-fem, fdual, fplur-masc, fplur-fem) {\n # This shows how you can use regular expressions if need be.\n # regsub(string, regex, repl) is a special built-in that does regular-\n # expression substitution on STRING, replacing all occurrences of\n # REGEX with REPL. Regular-expression syntax is as in Python.\n # In this case, Arabic verbs have a phonetic rule that eliminates\n # two glottal stops occurring near each other at the beginning of a\n # word. For example, _a_kulu -> _aakulu, and _u_kalu -> _uukalu.\n # (That is, the vowel is lengthened.)\n # _ . regsub(foo, bar, fsing): pres, mood, 1st, sg;\n _ . regsub('^([aiu])_', '\\1\\1', fsing): pres, mood, 1st, sg;\n # _.fsing: pres, mood, 1st, sg;\n t.fsing: pres, mood, 2nd, m, sg;\n t.fsing-fem: pres, mood, 2nd, f, sg;\n y.fsing: pres, mood, 3rd, m, sg;\n t.fsing: pres, mood, 3rd, f, sg;\n\n t.fdual: pres, mood, 2nd, du;\n y.fdual: pres, mood, 3rd, m, du;\n t.fdual: pres, mood, 3rd, f, du;\n\n n.fsing: pres, mood, 1st, pl;\n t.fplur-masc: pres, mood, 2nd, m, pl;\n t.fplur-fem: pres, mood, 2nd, f, pl;\n y.fplur-masc: pres, mood, 3rd, m, pl;\n y.fplur-fem: pres, mood, 3rd, f, pl;\n}\n\n# The \"two-form\" present uses normal (non-3rd-weak) endings but may\n# have two forms of the root, one form vocalic endings (almost all of them)\n# and one for consonant endings (only the feminine plural). This\n# encompasses 2nd-weak verbs and doubled verbs, and (trivially) strong verbs.\n\ndef two-form-pres-indic(formv, formc) {\n gen-pres(indic, formv.u, formv.iina, formv.aani, formv.uuna, formc.na)\n}\n\ndef two-form-pres-subj(formv, formc) {\n gen-pres(subj, formv.a, formv.ii, formv.aa, formv.uu, formc.na)\n}\n\n# The jussive is different because the base form (fsing) has no ending.\n# This means that it may assume the consonant form instead of the vowel\n# form, or may have a number of variants (in particular, for doubled\n# verbs). So the base form needs to be given explicitly.\n\ndef two-form-pres-juss(base, formv, formc) {\n gen-pres(juss, base, formv.ii, formv.aa, formv.uu, formc.na)\n}\n\ndef strong-pres(form) {\n two-form-pres-indic(form, form)\n two-form-pres-subj(form, form)\n two-form-pres-juss(form, form, form)\n}\n\ndef 2nd-weak-pres(formv, formc) {\n two-form-pres-indic(formv, formc)\n two-form-pres-subj(formv, formc)\n two-form-pres-juss(formc, formv, formc)\n}\n\ndef doubled-pres(formv, formc) {\n two-form-pres-indic(formv, formc)\n two-form-pres-subj(formv, formc)\n two-form-pres-juss(formc, formv, formc)\n two-form-pres-juss(formv.a, formv, formc)\n two-form-pres-juss(formv.i, formv, formc)\n}\n\n# Verbs whose third radical is a /w/ or a /y/ have all manner of exceptional\n# forms; easiest just to list them. In general, there are three types,\n# depending on whether the base singular forms end in -aa, -ii, or -uu.\n\ndef 3rd-weak-pres-aa(form) {\n gen-pres(indic, form.aa, form.ayna, form.ayaani, form.awna, form.ayna)\n gen-pres(subj, form.aa, form.ay, form.ayaa, form.aw, form.ayna)\n # Note the shortened vowel here.\n gen-pres(juss, form.a, form.ay, form.ayaa, form.aw, form.ayna)\n}\n\ndef 3rd-weak-pres-ii(form) {\n gen-pres(indic, form.ii, form.iina, form.iyaani, form.uuna, form.iina)\n gen-pres(subj, form.iya, form.ii, form.iyaa, form.uu, form.iina)\n # Note the shortened vowel here.\n gen-pres(juss, form.i, form.ii, form.iyaa, form.uu, form.iina)\n}\n\ndef 3rd-weak-pres-uu(form) {\n gen-pres(indic, form.uu, form.iina, form.uwaani, form.uuna, form.uuna)\n gen-pres(subj, form.uwa, form.ii, form.uwaa, form.uu, form.uuna)\n # Note the shortened vowel here.\n gen-pres(juss, form.u, form.ii, form.uwaa, form.uu, form.uuna)\n}\n\ndef 2nd-weak-verb(pastv, props, pastc, presv, presc) {\n word pastv: props {\n two-form-past(pastv, pastc)\n 2nd-weak-pres(presv, presc)\n }\n}\n\n# Note the way that macro calls can be constructed as well. Here, the\n# value of PAST_TYPE is the suffix at the end of the macro name.\n\ndef 3rd-weak-verb(past_stem, props, past_type, pres_stem, pres_type) {\n word past_stem . past_type: props {\n 3rd-weak-past- . past_type(past_stem)\n 3rd-weak-pres- . pres_type(pres_stem)\n }\n}\n\ndef strong-verb(past, props, pres) {\n word past: props {\n strong-past(past)\n strong-pres(pres)\n }\n}\n\n2nd-weak-verb(kaan, TransV(pred=be), kun, akuun, akun)\n2nd-weak-verb(naam, IntransV(pred=sleep), nim, anaam, anam)\n2nd-weak-verb(qaal, SayV(pred=say), qul, aquul, aqul)\nstrong-verb(katab, TransV(pred=write), aktub)\nstrong-verb(dhahab, IntransV(pred=go), adhhab)\n# Note that the following verb, which begins with a glottal stop,\n# will have a modification made to it in the first-person singular present.\n# (See above.)\nstrong-verb(_akal, IntransV TransV (pred=eat), a_kul)\n3rd-weak-verb(ra_, TransV(pred=see), ay, ar, aa)\n3rd-weak-verb(_a9T, DitransV(pred=give), ay, u9T, ii)\n3rd-weak-verb(laq, TransV(pred=find), ii, alq, aa)\nstrong-verb(9araf, ThinkV(pred=know), a9rif) # see also 9alam\nstrong-verb(tafakkar, ThinkV(pred=think), atafakkar)\nstrong-verb(ta9allam, ThinkV(pred=learn), ata9allam)\n\nfamily N {\n entry: n<2>[X, 3rd, nonres]: X:sem-obj(*);\n}\n\nfamily InterrogPro(Pro) {\n entry: s/*(s/n<2>[res]);\n entry: s/*(s|n<2>[nonres,nom]);\n entry: s/*(s/n<2>[nonres,acc]);\n member: maa, man;\n}\n\nfamily Pro {\n entry: n<2>[X, nom, def, nonres]: X:sem-obj(*);\n member: pro;\n}\n\nfamily Rel {\n entry: (n<~2>[CASE,nonres]\\n<~2>[nonres])/*(s/n<2>[res]);\n entry: (n<~2>[CASE,nonres]\\n<~2>[nonres])/*(s|n<2>[nonres,nom]);\n entry: (n<~2>[CASE,nonres]\\n<~2>[nonres])/*(s/n<2>[nonres,acc]);\n member: al-ladhii;\n}\n\nfamily AndConj(Conj) {\n entry: n[pl, CASE, STATE] \\* n[CASE, STATE] /* n[CASE, STATE];\n entry: s$1 \\* s$1 /* s$1;\n member: wa;\n}\n\nfamily Det(indexRel=det) {\n entry: n<2>[X, def, nonres] /^ n<2>[X]: X:sem-obj(*);\n member: haadhaa, dhaalik;\n}\n\n# good luck on this one! construct even more complicated ones! \n#\n#ar-rajulu al-ladhii kataba al-kutuba ra_aa wa _a9Taa li binti hu al-kilaaba al-latii akalat sagaayira mudarrisii al-waladi\n\n#\"the man that wrote the books saw and gave to his daughter the dogs that ate the cigarettes of the boy's teachers.\"\n\n# Works, correctly:\n\n#ar-rajulu al-ladhii kataba al-kutuba ra_aa wa _a9Taa li binti hu as-sagaayira al-latii qultu inna al-waladu tafakkara anna al-mar_atu _a9Tat haa li ha_ulaahi al-mudarrisiina\n\n#Bad\n\n#ar-rajula al-ladhii katabat al-kutuba ra_aa wa _a9Taa li binti hu as-sagaayira al-latii qultu inna al-waladu tafakkara anna al-mar_atu _a9Tat haa li ha_ulaahi al-mudarrisiina\n\n#\"the man that wrote the books saw and gave to his daughter the cigarettes that I said that the boy thought that the woman gave them to those teachers\"\n\n# Fails, correctly:\n\n#ar-rajulu al-ladhii kataba al-kutuba ra_aa wa _a9Taa li binti hu as-sagaayira al-latii qultu inna al-waladu tafakkara anna al-mar_atu _a9Tat hu li ha-ulaahi al-mudarrisiina\n\n# Works:\n\n# ar-rajulu al-ladhii al-waladu _a9Taa as-sagaayira li binti hu dhahaba\n# ar-rajulu _a9Taa li binti hu as-sagaayira\n# ar-rajulu al-ladhii waladu hu _a9Taa as-sagaayira li al-binti dhahaba\n\n# Won't work:\n\n# ar-rajulu al-ladhii al-waladu _a9Taa li binti hu as-sagaayira dhahaba\n# ar-rajulu al-ladhii waladu hu _a9Taa li al-binti as-sagaayira dhahaba\n\n\n\nfamily PossClitic(Cli, indexRel=poss) {\n entry: n<~1>[X, def, nonres] \\* n<1>[X, cons]: X:sem-obj(*);\n entry: (n<~1>[X, def, nonres] / n<2>[res]) \\* n<1>[X, cons]: X:sem-obj(*);\n member: ii, ka, ki, hu, haa, naa, kum, kunna, hum, hunna;\n}\n\nfamily ObjClitic(Cli, indexRel=poss) {\n entry: (s$1 | n<3>[nonres]) \\ (s$1 / n<2>[nonres] | n<3>[nom,nonres]);\n entry: (s$1 / n<~2>[res] | n<3>[nonres]) \\ (s$1 / n<2>[nonres] | n<3>[nom,nonres]);\n member: nii, ka, ki, hu, haa, naa, kum, kunna, hum, hunna;\n}\n\nfamily Adj(indexRel=adj) {\n entry: n<2>[X, NUM, GEND, CASE, STATE] \\ n<2>[X]:\n X:sem-obj(*);\n}\n\nfamily Prep-Nom(Prep, indexRel=\"*NoSem*\") {\n # The pp<~3> notation generates an 'inheritsFrom' tag rather than\n # an 'id' tag for the feature structure.\n entry: pp<~3>[lex=*] /< n<3>[gen,nonres];\n member: li, fii;\n}\n\nfamily Comp(indexRel=\"*NoSem*\") {\n entry: sbar<~1>[lex=*] / s<1>;\n member: anna, inna;\n}\n\nfamily IntransV(V) {\n entry: s[E] | n<2>[X,NUM,PERS,GEND,non-cons,nonres,nom]:\n\tE:action(* ^ X:animate-being);\n}\n\nfamily SayV(V) {\n entry: s[E] / sbar[Z, lex=inna] | n<2>[X,NUM,PERS,GEND,non-cons,nonres,nom]:\n E:action(* ^ X:animate-being ^ Z);\n}\n\nfamily ThinkV(V) {\n entry: s[E] / sbar[Z, lex=anna] | n<2>[X,NUM,PERS,GEND,non-cons,nonres,nom]:\n E:action(* ^ X:animate-being ^ Z);\n}\n\nfamily TransV(V) {\n entry: s[E] / n[Y,acc,nonres] | n<2>[X,NUM,PERS,GEND,non-cons,nonres,nom]:\n E:action(* ^ X:animate-being ^ Y:sem-obj);\n}\n\nfamily DitransV(V) {\n # The first slash (on the pp) is marked with a mode allowing backward xcomp.\n entry: s[E] /< pp[Z,lex=li] / n[Y,acc,nonres] | n<2>[X,NUM,PERS,GEND,non-cons,nonres,nom]:\n E:action(* ^ X:animate-being ^ Y:sem-obj ^\n\t Z:animate-being) ;\n}\n\ntestbed {\n # different states of subject\n rajulun dhahaba: 1;\n ar-rajulu dhahaba: 1;\n rajulu dhahaba: 0;\n\n # subject-verb agreement\n rajulun dhahabuu: 0;\n ar-rajulu dhahabuu: 0;\n rajulu dhahabuu: 0;\n\n # gender agreement\n ar-rajulu dhahaba: 1;\n ar-rajulu dhahabat: 0;\n al-bintu dhahaba: 0;\n al-bintu dhahabat: 1;\n\n # possession\n rajulun hu dhahaba: 0;\n ar-rajulu hu dhahaba: 0;\n rajulu hu dhahaba: 1;\n\n # subject case\n ar-rajula dhahaba: 0;\n ar-rajuli dhahaba: 0;\n\n # case in construct phrase\n _axuu ar-rajuli dhahaba: 1;\n _axuu ar-rajula dhahaba: 0;\n _axuu ar-rajulu dhahaba: 0;\n\n # construct state in construct phrase\n _axun ar-rajuli dhahaba: 0;\n al-_axu ar-rajuli dhahaba: 0;\n\n # object case\n ar-rajulu ra_aa al-kitaaba: 1;\n ar-rajulu ra_aa al-kitaabi: 0;\n ar-rajulu ra_aa al-kitaabu: 0;\n\n # preposition case\n ar-rajulu _a9Taa al-kitaaba li al-waladi: 1;\n ar-rajulu _a9Taa al-kitaaba li al-waladu: 0;\n ar-rajulu _a9Taa al-kitaaba li al-walada: 0;\n\n # subcategorization\n ar-rajulu ra_aa al-kitaaba li al-waladi: 0;\n\n # backward xcomp\n ar-rajulu _a9Taa li al-waladi al-kitaaba: 1;\n _a9Taa ar-rajulu li al-waladi al-kitaaba: 1;\n\n # object clitics\n ana ra_aytu hu: 1;\n ra_aytu hu ana: 1;\n ra_aytu ana hu: 0;\n hu ra_aytu ana: 0;\n hu ana ra_aytu: 0;\n\n huwa ra_aa nii: 1;\n huwa ra_aa ii: 0;\n huwa ra_aa ana: 0;\n\n ar-rajulu _a9Taa haa li al-waladi: 1;\n _a9Taa haa ar-rajulu li al-waladi: 1;\n\n # relative clauses\n # \"I gave it to the man that the girl saw him\"\n _a9Taytu haa li ar-rajuli al-ladhii al-bintu ra_at hu: 3;\n # \"I gave it to the man that the girl saw her\"\n _a9Taytu haa li ar-rajuli al-ladhii al-bintu ra_at haa: 0;\n # \"I gave it to the man that the girl saw\"\n _a9Taytu haa li ar-rajuli al-ladhii al-bintu ra_at: 0;\n # \"I gave it to the man that the girl saw the boy\"\n _a9Taytu haa li ar-rajuli al-ladhii al-bintu ra_at al-walada: 0;\n}\n" }, { "path": "src/ccg2xml/build.xml", "content": "\n\n\n\n \n \n \n\n \n\n \n \n \n \n \n \n \n \n\n \n \n\n \n \n\n \n \n\n \n \n\n \n \n\n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "src/ccg2xml/ccg.ply", "content": "\n# Author: Ben Wing \n# Date: November 2006\n\n#############################################################################\n# #\n# ccg.ply #\n# #\n# Convert a .ccg file, specifying a CCG grammar, into files lexicon.xml, #\n# rules.xml, morph.xml, types.xml and grammar.xml. You can't actually #\n# run this file itself; you have to use convert-ply.py to convert it #\n# into a Python file (ccg2xml), which you then run to generate the XML #\n# files from the .ccg input. For a description of the format of this #\n# file, see the comments in convert-ply.py. #\n# #\n#############################################################################\n\n\nimport sys\nimport re\nimport optparse\nimport copy\nimport os\nimport cStringIO\n\n# Local imports\nimport lex, yacc\n\n# Get options\n\nusage = \"\"\"%prog [OPTIONS] FILE ...\n\nGenerate appropriate XML files for input to OpenCCG.\n\"\"\"\n\nparser = optparse.OptionParser(usage=usage)\nparser.add_option(\"-o\", \"--omit-output\",\n help=\"\"\"Omit the specified files from the output.\nValue should be a list separated by commas or spaces. The allowed values are grammar, morph, lexicon, rules, types, and testbed. If you put a + sign before the list, it means output *only* the specified files.\"\"\"\n )\nparser.add_option(\"-p\", \"--prefix\",\n help=\"\"\"Optional prefix to attach to each of the generated files, so that output from different files can occur in the same directory. Defaults to the base name of the input file, minus any extension, plus a hyphen. If you want such a hyphen or similar char, add it yourself.\"\"\",\n metavar=\"DIR\"\n )\nparser.add_option(\"-d\", \"--dir\",\n help=\"\"\"Directory to store files in (defaults to current directory).\"\"\",\n metavar=\"DIR\"\n )\nparser.add_option(\"-q\", \"--quiet\",\n action=\"store_true\",\n help=\"Don't output explanatory messages, but only warnings and errors.\")\nparser.add_option(\"-t\", \"--transformed-input\",\n action=\"store_true\",\n help=\"Output transformed input after macro substitutions have been applied.\")\nparser.add_option(\"-y\", \"--yacc-debug\",\n action=\"store_true\",\n help=\"Show more output about the YACC parser generation. Also probably generate some extra files, e.g. parser.out, containing info about the generated parser.\")\nparser.add_option(\"-m\", \"--macro-debug\",\n action=\"store_true\",\n help=\"Dump macro definitions at end of file.\")\nparser.add_option(\"--super-macro-debug\",\n action=\"store_true\",\n help=\"Show copious output about macro expansions.\")\n\ndef parse_arguments(argv):\n global options, global_args\n (options, global_args) = parser.parse_args(argv)\n\n # Global variables used for debugging; we may move them into the\n # global-state variable\n global lex_debug\n global xml_debug\n global yacc_debug\n global macro_debug\n global super_macro_debug\n\n lex_debug = 0\n xml_debug = 0\n yacc_debug = options.yacc_debug\n macro_debug = options.macro_debug\n super_macro_debug = options.super_macro_debug\n\n########################################################################\n# Utility functions #\n########################################################################\n\n\n# CONVENTIONS:\n#\n# --------- XML ----------\n#\n# Thankfully, the structure of XML is extremely simple. We represent\n# a single XML statement of the form\n#\n# \n# \n# ...\n# gurgle\n# \n#\n# as a list\n#\n# ['biteme', [('foo', '1'), ('blorp', 'baz')],\n# ['bitemetoo', ...],\n# 'gurgle'\n# ]\n#\n# i.e. an XML statement corresponds to a list where the first element\n# is the statement name, the second element lists any properties, and\n# the remaining elements list items inside the statement.\n#\n# ----------- Property lists -------------\n#\n# The second element of an XML statement in list form is a \"property list\",\n# a list of two-element tuples (property and value). Some functions below\n# (e.g. `getprop', `putprop') manipulate property lists.\n#\n# FIXME: Just use a hash table.\n#\n# ---------- Abstract syntax trees -----------\n#\n# We use classes to represent statements and blocks. Below this level, it's\n# simpler to just use the XML that we ultimately have to generate anyway.\n# The conventions for using XML are either to use property lists or lists of\n# XML statements in the list form outlined above.\n\n\n#############################\n# Handling XML #\n#############################\n\ndef xml_sub(crap):\n if type(crap) is not str:\n crap = str(crap)\n crap = crap.replace('<', '<')\n crap = crap.replace('>', '>')\n return crap\n\ndef print_xml_1(file, xml, indent=0):\n if xml_debug > 1:\n errout(\"%sPrinting: %s\\n\" % (' ' * indent, str(xml)))\n if type(xml) is not list:\n file.write('%s%s\\n' % (' ' * indent, xml_sub(xml)))\n else:\n check_arg_type(\"XML statement\", xml[0], str)\n file.write(' ' * indent)\n file.write('<%s' % xml_sub(xml[0]))\n for x in xml[1]:\n check_arg_type(\"XML statement\", x, tuple)\n if len(x) != 2:\n raise TypeError(\"Bad tuple pair: \" + str(x))\n file.write(' %s=\"%s\"' % (xml_sub(x[0]), xml_sub(x[1])))\n subargs = xml[2:]\n if not subargs:\n file.write('/>\\n')\n else:\n file.write('>\\n')\n for x in subargs:\n print_xml_1(file, x, indent + 2)\n file.write(' ' * indent)\n file.write('\\n' % xml_sub(xml[0]))\n\n# Pretty-print a section of XML, in the format above, to FILE.\n# Start at indent INDENT.\n\ndef print_xml(file, xml):\n if xml_debug == 1:\n errout(\"Printing: %s\\n\" % str(xml))\n print_xml_1(file, xml)\n\n# Return True if PROP is seen as a property in PROPLIST, a list of tuples\n# of (prop, value)\ndef property_specified(prop, proplist):\n return not not ['foo' for (x,y) in proplist if x == prop]\n\n# Return value of property PROP in PROPLIST; signal an error if not found.\ndef getprop(prop, proplist):\n for (x,y) in proplist:\n if x == prop:\n return y\n raise ValueError(\"Property %s not found in %s\" % (prop, proplist))\n\n# Return value of property PROP in PROPLIST, or DEFAULT.\ndef getoptprop(prop, proplist, default=None):\n for (x,y) in proplist:\n if x == prop:\n return y\n return default\n\n# Replace value of property PROP with VALUE in PROPLIST.\ndef putprop(prop, value, proplist):\n for i in xrange(len(proplist)):\n if proplist[i][0] == prop:\n proplist[i] = (prop, value)\n return\n else:\n proplist += [(prop, value)]\n \n\n# Replace property named PROP with NEW in PROPLIST. Often this is called with\n# with PROP equal to None; the None occurs when a PROP=VALUE clause is expected\n# but a bare value is supplied. The context will supply a particular default\n# property (e.g. 'name') to be used when the property name is omitted, but the\n# generic code to handle property-value clauses doesn't know what this is.\n# The surrounding code calls property_name_replace() to fill in the proper name.\n\ndef property_name_replace(prop, new, proplist):\n for i in xrange(len(proplist)):\n if proplist[i][0] == prop:\n proplist[i] = (new, proplist[i][1])\n\n#############################\n# Error-handling #\n#############################\n\ndef init_errors(errors_to_string):\n # Count of number of errors seen so far.\n global error_count\n error_count = 0\n\n global write_errors_to_string\n write_errors_to_string = errors_to_string\n\n global stdout_file, stderr_file\n if errors_to_string:\n stdout_file = cStringIO.StringIO()\n stderr_file = cStringIO.StringIO()\n else:\n stdout_file = sys.stdout\n stderr_file = sys.stderr\n\n global message_log\n message_log = []\n\ndef save_errors(cur):\n cur.error_count = error_count\n cur.write_errors_to_string = write_errors_to_string\n cur.stdout_file = stdout_file\n cur.stderr_file = stderr_file\n\nclass InternalError(StandardError):\n pass\n\ndef argformat(format, arg):\n if type(format) is str:\n return format % arg\n else:\n return str(format)\n\n# Throw an error, like fprintf(stderr, ...)\ndef synerr(format, *arg):\n raise SyntaxError(argformat(format, arg))\n\n# Output to stderr, maybe. But output to stdout if our input is being\n# output at the same time, so the two will stay in sync.\ndef maybe_errout(str):\n # Force display of error\n # FIXME: Maybe we could dump all errors into a single \n # window display and show the messages together\n #showerror('Message', str)\n\n if options.transformed_input:\n stdout_file.write(str)\n else:\n stderr_file.write(str)\n\ndef error_or_warning(title, lineno, format, *arg):\n formatted_arg = argformat(format, arg)\n if lineno:\n maybe_errout(\"%s, line %s: %s\\n\" % (title, lineno,\n formatted_arg))\n else:\n maybe_errout(\"%s: %s\\n\" % (title, formatted_arg))\n # Add the message as a tuple, for easy recall in the editor\n # Note: lineno being put in irrespective of the fact\n # of whether it exists or not\n # FIXME!! The purpose of errors_to_string and message_log duplicate\n # each other somewhat. Clean up.\n global message_log\n message_log += [(title, lineno, formatted_arg)]\n\n# Write formatted arguments to stderr, with Error: printed.\ndef error(lineno, format, *arg):\n global error_count\n error_count += 1\n error_or_warning('Error', lineno, format, *arg)\n\n# Write formatted arguments to stderr, with Warning: printed.\ndef warning(lineno, format, *arg):\n global warning_count\n warning_count += 1\n error_or_warning('Warning', lineno, format, *arg)\n\n# Write formatted arguments to stdout.\ndef outout(format, *arg):\n stdout_file.write(argformat(format, arg))\n\n# Write formatted arguments to stderr.\ndef errout(format, *arg):\n stderr_file.write(argformat(format, arg))\n\n# Debugging output: Always to sys.stderr.\ndef debug(format, *arg):\n sys.stderr.write(argformat(format, arg))\n\ndef check_arg_type(errtype, arg, ty):\n if type(arg) is not ty:\n raise TypeError(\"%s: Type is not %s: %s\" % (errtype, ty, arg))\n\n#############################\n# Abstract Syntax Trees #\n#############################\n\n# Classes beginning with CS (= CCG Syntax) are used for constructing the\n# abstract syntax tree corresponding to a CCG source file. (An abstract\n# syntax tree, or AST, is a hierarchical representation of the syntax of a\n# piece of source code text, in this case a CCG-format file.) The source\n# file is made up out of blocks, each of which begins with an identifier\n# and is followed by one or more statements.\n\n# A CSNode corresponds to any unified section of source code -- a single\n# block or statement, a particular part of a statement (e.g. an\n# attribute-value list or a single attribute-value clause), or even the\n# whole file. The basic restriction is that it must correspond to a single\n# YACC production; hence it logically belongs in a unit and is the maximum\n# extent of text that belongs in the unit or possibly statement in a single\n# block. It has some corresponding source text with starting and ending\n# line numbers, a function to generate the XML, and a function to draw the\n# node. If the node is large enough to represent at least one XML\n# statement, it should be a list of XML statements in the XML-statement\n# form described above (a list [TAG, PROPLIST, CHILD ...]); otherwise, the\n# format is undefined, but most likely will be a property list. The CSNode\n# is initialized from the YaccProduction object (stored in variable `p',\n# usually, but accessed as $@) associated with a particular production,\n# which supplies the extent of source code associated with the production.\n\nclass CSNode(object):\n def __init__(self, prod):\n self.prod = prod\n def xml(self):\n # In many cases, it's easiest just to build up the XML at creation time\n # and store it, rather than constructing it dynamically. Note that we\n # intentionally don't initialize self.static_xml, so we get an error\n # if it's not set.\n return self.static_xml\n # draw(self, parent, cfile, vars): Draw the node by returning a new widget\n # containing the drawn representation: Should be defined if node is\n # drawable. It should return a widget that is a child of PARENT,\n # also a widget. It is up to the caller to call pack() or grid()\n # so that the widget's geometry will be set; but the draw() function\n # should appropriately configure any child widgets that it creates.\n # VARS is an object containing Tkinter variables that may control the\n # way that the node is drawn.\n\n# A CSStatement corresponds to a single statement in a single block. Note\n# that, in the interests of simplicity, we don't currently create objects\n# for pieces of CCG code that are smaller than a statement; instead, we\n# just use the XML representation. We usually follow the convention that\n# if we have to make changes to the XML that make it not be in a one-to-one\n# correspondence with the original code, we do this at the level of the\n# statement or block.\n\nclass CSStatement(CSNode):\n def __init__(self, prod):\n super(CSStatement, self).__init__(prod)\n\n# A CSBlock is a single block.\n\nclass CSBlock(CSNode):\n def __init__(self, prod):\n super(CSBlock, self).__init__(prod)\n\n#############################\n# Misc #\n#############################\n\n# Is it identifier material? Input should be a character.\ndef isalnumund(str):\n return str.isalnum() or str in '_+-'\n\n# Prior to Python 2.4, no sorted()\ndef my_sorted(lyst):\n lystcopy = list(lyst)\n lystcopy.sort()\n return lystcopy\n\n########################################################################\n# Tokenizing #\n########################################################################\n\n# The following IDs have a special meaning to the OpenCCG tokenizer if a\n# token has the form [*ID*].\n#magic_names = ('AMT', 'DATE', 'DUR', 'NUM', 'TIME')\n\n# It seems that the tokenizer does not require an [*ID*] token to be a known\n# magic thing (i.e. [*FOO*] is a legal surface form), so line 402 is commented\n# out.\n# If someone decides that [*ID*] tokens should be restricted to the ones\n# listed above, uncomment lines 397 and 503.\n\n# Directives -- These are particular words that are specially handled in\n# an appropriate position and hence need to be tokens for use in the\n# parser. However, they can also be part of a generic \"word\" -- in\n# other words, we have no \"reserved words\".\n\ndirectives = (\n 'FAMILY', 'ENTRY', 'MEMBER', 'FEATURE', 'PROP', 'RULE',\n 'NO', 'APP', 'COMP', 'XCOMP', 'SUB', 'XSUB', 'TYPERAISE', 'TYPECHANGE',\n 'DEF', 'WORD', 'TESTBED', 'RELATION_SORTING'\n )\n\n# Additional tokens that can form part of a word. A bare 'x' can form\n# part of a word as well, except for in a few circumstances.\nbasic_word_no_x_tokens = ('ID', 'QUOTEDID') + directives\nword_no_x_tokens = ('NUMBER',) + basic_word_no_x_tokens\nword_no_number_tokens = ('X',) + basic_word_no_x_tokens\nword_tokens = ('NUMBER',) + word_no_number_tokens + ('MAGIC_ID',)\nbracket_tokens = ('LPAREN', 'RPAREN', 'LBRACKET', 'RBRACKET',\n 'LBRACE', 'RBRACE')\nother_tokens = (\n # String tokens\n 'SLASH', 'BACKSLASH',\n 'LESS', 'GREATER',\n 'CARET', 'STAR', 'DOT', 'AT', 'EQUALS', 'GOESTO', 'PIPE',\n 'COMMA', 'SEMI', 'DOLLAR', 'COLON', 'BANG', 'TILDE',\n # Handled through t_ID\n 'PLUS', 'MINUS', 'PLUSMINUS',\n # Only in a def()\n 'NEWLINE',\n 'BOGUS_VALUE' # Kludge kludge kludge, fuck me harder\n )\n\ntokens = word_tokens + bracket_tokens + other_tokens\n\nt_LPAREN = r'\\('\nt_RPAREN = r'\\)'\nt_LBRACKET = r'\\['\nt_RBRACKET = r'\\]'\nt_LBRACE = r'\\{'\nt_RBRACE = r'\\}'\n\nt_SLASH = r'/'\nt_BACKSLASH = r'\\\\'\nt_LESS = r'<'\nt_GREATER = r'>'\nt_CARET = r'\\^'\nt_STAR = r'\\*'\nt_DOT = r'\\.'\nt_AT = r'@'\nt_EQUALS = r'='\nt_GOESTO = r'=>'\nt_PIPE = r'\\|'\nt_COMMA = r','\nt_SEMI = r';'\nt_DOLLAR = r'\\$'\nt_COLON = r':'\nt_BANG = r'!'\nt_TILDE = r'~'\n\n# Identifiers and directives\n\ndirectives_map = { }\nfor r in directives:\n directives_map[r.lower().replace('_', '-')] = r\ndirectives_map['x'] = 'X';\n# We handle +, -, and +- here because + and - can, in general, form part\n# of a token.\ndirectives_map['+'] = 'PLUS';\ndirectives_map['-'] = 'MINUS';\ndirectives_map['+-'] = 'PLUSMINUS';\n\ndef t_ID(t):\n r'''(([\\-+%a-zA-Z_0-9]|[^\\000-\\177])+|\"[^\"\\n]+\"|\\'[^'\\n]+')'''\n # convert to directive, maybe\n if re.match(r'^\\d+$', t.value):\n t.type = 'NUMBER'\n try:\n t.value = int(t.value)\n except ValueError:\n error(t.lineno, \"Integer value too large: %s\", t.value)\n t.value = 0\n elif t.value in directives_map:\n t.type = directives_map[t.value]\n else:\n t.type = 'ID'\n # remove quotes if they're there\n if t.value[0] == '\"' or t.value[0] == \"'\":\n t.type = 'QUOTEDID'\n t.value = t.value[1:-1]\n return t\n\n# The distinction from ordinary IDs is currently not really needed, i.e. t_ID\n# could to the job, too. However, it leaves open the possibility to handle\n# magic IDs differently from ordinary ones.\ndef t_MAGIC_ID(t):\n r'''(\\[\\*[^*]+\\*\\])'''\n t.type = 'MAGIC_ID'\n return t\n#t_MAGIC_WORD.func_doc = '(\\[\\*(' + '|'.join(magic_names) + ')\\*\\])'\n\nt_ignore = \" \\t\\r\"\n\n#bracketmap = {'(': 'LPAREN', ')': 'RPAREN',\n# '[': 'LBRACKET', ']': 'RBRACKET',\n# '{': 'LBRACE', '}': 'RBRACE'}\n#\n#def t_LBRACKET(t):\n# r'[\\[\\(\\{]'\n# global parendepth\n# parendepth += 1\n# t.type = bracketmap[t.value]\n# return t\n#\n#def t_RBRACKET(t):\n# r'[\\]\\)\\}]'\n# global parendepth\n# parendepth -= 1\n# t.type = bracketmap[t.value]\n# return t\n\ndef t_backslash_newline(t):\n r'\\\\\\r?\\n'\n t.lineno += 1\n # If it's not a line continuation, it's just a normal backslash\n if not lexer_track_newlines:\n t.type = 'BACKSLASH'\n return t\n\ndef t_newline(t):\n r'\\n'\n t.lineno += 1\n if lexer_track_newlines:\n t.type = 'NEWLINE'\n return t\n\n# Comments\ndef t_comment(t):\n r'\\#[^\\n]*\\n'\n t.lineno += 1\n\ndef t_error(t):\n error(t.lineno, \"Illegal character '%s'\", t.value[0])\n t.skip(1)\n \ndef init_lexer():\n # This is a signal to us to go into \"line mode\", where we return a\n # newline as a token and treat backslash at the end of a line as a line\n # continuation device.\n global lexer_track_newlines\n lexer_track_newlines = 0\n\n # Build the lexer. This does introspection, on all the t_*() functions.\n global globallexer\n\n globallexer = lex.lex(debug=lex_debug)\n\ndef save_lexer(cur):\n cur.lexer_track_newlines = lexer_track_newlines\n cur.globallexer = globallexer\n\n########################################################################\n# Parsing #\n########################################################################\n\ndef p_word(p):\n 'FILLED IN BELOW'\n p[0] = p[1]\n# fill in the documentation (i.e. the cfg rule)\np_word.func_doc = 'word : ' + '\\n| '.join(word_tokens)\n\n# hack, to deal with a reduce/reduce conflict\ndef p_word_except_x(p):\n 'FILLED IN BELOW'\n p[0] = p[1]\n# fill in the documentation (i.e. the cfg rule)\np_word_except_x.func_doc = 'word_except_x : ' + '\\n| '.join(word_no_x_tokens)\n\ndef p_word_no_numbers(p):\n 'FILLED IN BELOW'\n p[0] = p[1]\n# fill in the documentation (i.e. the cfg rule)\np_word_no_numbers.func_doc = (\n 'word_no_numbers : ' + '\\n| '.join(word_no_number_tokens))\n\n%y\n#############################\n# Begin Yacc Declarations #\n#############################\n\n#############################\n# Word lists, etc #\n#############################\n\nempty: : $$ = []\n\ncommas: COMMA\n : commas COMMA\n\ntyped_word : word\n : word COLON word : $$ = '%s:%s' % ($1, $3)\n\n# Possibly empty list of words\n\nword_0: word\n : word commas\n\nword_list: word_0 *\n\n# Possibly empty list of words or *\n\nword_or_star: word | STAR\n\nword_or_star_0: word_or_star\n : word_or_star commas\n\n# Non-empty list of words\n\n#nonempty_word_list: word_0 : $$ = [$1]\n# : nonempty_word_list word_0 : $$ = $1 + [$2]\n\n# Attribute lists contain specifications of the form ATTR=VALUE. The\n# return value is a list of (attribute, value) tupes.\n\nattr : word EQUALS word : $$ = ($1, $3)\n\nattr_0: attr\n : attr commas\n\nattr_list: attr_0*\n\nopt_paren_attr_list: empty\n : LPAREN attr_list RPAREN : $$ = $2\n\n# Extended attribute lists contain either VALUE or ATTR=VALUE. The return\n# value is a list of (attribute, value) tupes; when a bare value is given,\n# the attribute is None.\n\next_attr : word : $$ = (None, $1)\n : word EQUALS word : $$ = ($1, $3)\n\next_attr_0: ext_attr\n : ext_attr commas\n\next_attr_list: ext_attr_0*\n\nopt_paren_ext_attr_list: empty\n : LPAREN ext_attr_list RPAREN : $$ = $2\n\n#############################\n# Statements #\n#############################\n\ntop : statement*\n\nstatement : family_block | feature_block | rule_block | macro_def | word_block\n | testbed_block | relation_sorting_block\n : SEMI\n\n#############################\n# Macros #\n#############################\n\n%p\n\ndef init_macros():\n # Used to turn off macro substitution while processing a macro definition.\n global no_macro_sub\n no_macro_sub = 0\n\n # Needed to handle issue where macro def is immediately followed by\n # macro call.\n global return_bogus_value\n return_bogus_value = 0\n\n # Mapping of macro definitions to parameter list and text.\n global macro_defs\n macro_defs = {}\n\n # It doesn't really matter what the parameter names are for built-ins.\n # There just have to be the right number of them.\n macro_defs['regsub'] = MacroDef(['fromre', 'totext', 'str'], regsub)\n macro_defs['ifmatch'] = MacroDef(['regex', 'string', 'doif', 'doelse'],\n ifmatch)\n macro_defs['ifmatch-nocase'] = MacroDef(['regex', 'string', 'doif',\n 'doelse'],\n ifmatch_nocase)\n\ndef save_macros(cur):\n cur.no_macro_sub = no_macro_sub\n cur.return_bogus_value = return_bogus_value\n cur.macro_defs = macro_defs\n\nclass MacroDef:\n def __init__(self, args, text):\n self.args = args\n self.text = text\n\nclass CCGToken(lex.LexToken):\n def __init__(self, type, value):\n self.type = type\n self.value = value\n\ndef arg_to_text(arg):\n return ''.join([str(x.value) for x in arg])\n\n# Implementation of built-in 'regsub()': Concatenate the tokens into\n# text, then do regex substitution.\ndef regsub(fromre, totext, string):\n return re.sub(arg_to_text(fromre), arg_to_text(totext),\n arg_to_text(string))\n\n# If REGEX matches the beginning of STRING, return DOIF, else return DOELSE.\ndef ifmatch(regex, string, doif, doelse):\n if re.match(arg_to_text(regex), arg_to_text(string)):\n return doif\n else:\n return doelse\n\n# Same as ifmatch() but case-insensitive.\ndef ifmatch_nocase(regex, string, doif, doelse):\n if re.match(arg_to_text(regex), arg_to_text(string), re.IGNORECASE):\n return doif\n else:\n return doelse\n\ndef print_macros():\n for (key, value) in macro_defs.iteritems():\n print \"Macro: %s(%s): %s\" % (key, value.args, value.text)\n\n# Given some text, expand the macros in it, recursively (i.e. apply\n# any macros, then apply macros to the resulting text, etc.). After\n# that, combine text that has the . operator applied to it.\ndef macroexpand_text(text):\n if super_macro_debug:\n print \"Text before expanding: %s\" % arg_to_text(text)\n # Now recursively expand macros. The code to actually check for\n # macros is in MacroLexer.\n lexer = MacroLexer(None)\n lexer.pushstack(text)\n newtext = []\n while True:\n tok = lexer.token()\n #print \"Reading token: %s\" % tok\n if not tok:\n break\n newtext.append(tok)\n text = newtext\n l = len(text)\n if super_macro_debug:\n print \"Text after expanding: %s\" % arg_to_text(text)\n # Now directly handle instances with the '.' operator, so that\n # the operator can be used to create new macro calls\n x = 1\n while x < l - 1:\n if (text[x].type == 'DOT' and text[x-1].type in ['ID', 'QUOTEDID']\n and text[x+1].type in ['ID', 'QUOTEDID']):\n tok = CCGToken(text[x-1].type,\n text[x-1].value + text[x+1].value)\n tok.lineno = text[x].lineno\n # If either is quoted, the result should be quoted.\n if text[x+1].type == 'QUOTEDID':\n tok.type = 'QUOTEDID'\n text[x-1] = tok\n text[x:x+2] = []\n x -= 2\n l -= 2\n x += 1\n return text\n\n# Return text of macro, with ARGS substituted for formal parameters of\n# the macro.\ndef macrosub(macdef, args, lineno):\n text = macdef.text\n # If the text definition is a function (for builtins),\n # macro-expand the arguments, then call the function.\n if callable(text):\n args = [macroexpand_text(x) for x in args]\n text = text(*args)\n if type(text) is str:\n text = [CCGToken('QUOTEDID', text)]\n text[0].lineno = lineno\n return text\n else:\n return macroexpand_text(text)\n else:\n # Otherwise, make a copy of the text and substitute the arguments\n # into it.\n text = text[:]\n args = dict(zip(macdef.args, args))\n l = len(text)\n x = 0\n while x < l:\n if (text[x].type == 'ID' or text[x].type in directives) \\\n and text[x].value in args:\n newtext = args[text[x].value]\n text[x:x+1] = newtext\n l += len(newtext) - 1\n x += len(newtext) - 1\n x += 1\n return macroexpand_text(text)\n\n%y\n\n# We need to do some hackery with BOGUS_VALUE in order to avoid problems\n# when a macro definition is immediately followed by a call to that same\n# macro. The problem is that generally the parser wants to read one token\n# ahead. As a result, by the time it's processed the token that ends a\n# macro definition, it's already read the following token -- and if that\n# token begins a macro call, we're screwed. To avoid this, we ensure that\n# there is an extra BOGUS_VALUE token returned after every macro definition.\n# To make this happen, we set a flag return_bogus_value just before the\n# parser processes the token ending the macro definition. At this point,\n# the parser has already read that token from the lexer, and before it\n# reduces that token, it reads the next token from the lexer -- which\n# returns a bogus token, as we instructed it.\n\nmacro_def : macro_def_1 BOGUS_VALUE\n\nturn_off_macro_sub: :\n global no_macro_sub\n no_macro_sub = 1\n\nreturn_bogus_value: :\n global return_bogus_value\n return_bogus_value = 1\n global no_macro_sub\n no_macro_sub = 0\n\nmacro_def_1 : turn_off_macro_sub DEF word LPAREN macro_param_list turn_on_linetrack RPAREN macro_text :\n macdef = MacroDef($5, $8)\n macdef.args = $5\n macdef.text = $8\n if $3 in macro_defs:\n error($@.lineno(0), \"Redefining macro %s\" % $3)\n macro_defs[$3] = macdef\n #print_macros()\n\nmacro_param_list : word_list\n\nmacro_text: bracemacro_text | linemacro_text\n\nbracemacro_text: turn_off_linetrack LBRACE bracemacro_text_list return_bogus_value RBRACE: $$ = $3\n\nbracemacro_text_list: empty : $$ = []\n : bracemacro_text_list bracemacro_text_entry : $$ = $1 + $2\n\n# The key thing about these is that they must be invoked BEFORE the\n# token that tells you whether to turn the mode on or off. If you\n# try to set the global variable after (even directly after) the\n# RPAREN or NEWLINE or whatever has been processed by a rule, it's too\n# late: The parser has already looked ahead, and any newline directly\n# following the token in question already processed the wrong way.\nturn_on_linetrack: :\n global lexer_track_newlines\n lexer_track_newlines = 1\n\nturn_off_linetrack: :\n global lexer_track_newlines\n lexer_track_newlines = 0\n\n%p\n\ndef p_bracemacro_text_entry(p):\n '''bracemacro_text_entry : LPAREN bracemacro_text_list RPAREN\n | LBRACKET bracemacro_text_list RBRACKET\n | LBRACE bracemacro_text_list RBRACE'''\n p[0] = [p.slice[1]] + p[2] + [p.slice[3]]\n\ndef p_bracemacro_text_entry_other(p):\n 'FILLED IN BELOW'\n p[0] = [p.slice[1]]\n# fill in the documentation (i.e. the cfg rule)\np_bracemacro_text_entry_other.func_doc = (\n 'bracemacro_text_entry : ' + '\\n| '.join(other_tokens + word_tokens)\n )\n\n%y\n\nlinemacro_text: turn_off_linetrack return_bogus_value NEWLINE:\n $$ = []\n\nlinemacro_text: linemacro_begin linemacro_next* turn_off_linetrack return_bogus_value NEWLINE:\n $$ = [$1] + $2\n\n%p\n\ndef p_linemacro_begin(p):\n p[0] = p.slice[1]\n\ndef p_linemacro_next(p):\n p[0] = p.slice[1]\n\nlinemacro_begin_tokens = [x for x in tokens\n if x != 'NEWLINE' and x != 'LBRACE']\nlinemacro_next_tokens = [x for x in tokens if x != 'NEWLINE']\n\n\n# fill in the documentation (i.e. the cfg rule)\np_linemacro_begin.func_doc = (\n 'linemacro_begin : ' + '\\n| '.join(linemacro_begin_tokens)\n )\np_linemacro_next.func_doc = (\n 'linemacro_next : ' + '\\n| '.join(linemacro_next_tokens)\n )\n\n%y\n\n#############################\n# Feature blocks #\n#############################\n\n%p\n\ndef init_features():\n # For each feature value, map its name to a CCGFeatval structure\n # describing it.\n global feature_values\n feature_values = {}\n # List of values for a particular feature; each value is a CCGFeatval,\n # listing a value, its parents, licensing info, macro info, and its\n # feature.\n global feature_to_values\n feature_to_values = {}\n # List of distributive features\n global distributive_features\n distributive_features = []\n # List of XML for licensing features\n global licensing_feature_xml\n licensing_feature_xml = []\n # Mapping of the names of feature values to the value inserted into the\n # XML 'val' attribute; usually the same as the name. (YUCK YUCK YUCK)\n global fv_names_to_values\n fv_names_to_values = {}\n\ndef save_features(cur):\n cur.feature_values = feature_values\n cur.feature_to_values = feature_to_values\n cur.distributive_features = distributive_features\n cur.licensing_feature_xml = licensing_feature_xml\n cur.fv_names_to_values = fv_names_to_values\n\n# A feature value: The \"name\" of the feature value (corresponding to a\n# feature macro), the parents of this value, and any licensing info. Also\n# may include a .feature, which is the \"feature\" that this value is a value\n# for.\nclass CCGFeatval:\n def __init__(self, name, parents, licensing):\n self.name = name\n self.parents = parents\n self.licensing = licensing\n def __str__(self):\n return \"CCGFeatval(%s, parents=%s, licensing=%s)\" % (\n (self.name, self.parents, self.licensing))\n def __repr__(self):\n return str(self)\n\n# Encapsulates directly obtained values and values obtained recursively,\n# so we can avoid needlessly adding parents to the latter kind.\n# Used temporarily when building the hierarchy. Both direct and recursive\n# are lists of CCGFeatvals.\nclass CCGFeatvalList:\n def __init__(self, direct, recursive=None):\n recursive = recursive or [] # fuckme!\n self.direct = direct\n self.recursive = recursive\n def __str__(self):\n return \"CCGFeatvalList(direct=%s,recursive=%s)\" % (self.direct,self.recursive)\n def __repr__(self):\n return str(self)\n\n# For the given feature, and list of CCGFeatvals, convert the parents in\n# each CCGFeatval to a list of CCGFeatvals rather just strings, and clean\n# any excess parents. Basically, if a value has multiple parents and one\n# is reachable by following a path starting from another, it needs to be\n# removed.\n\n# FIXME!! Also output warnings when a featvar and featval have the same name.\n\ndef install_feature(feature, lis, lineno):\n # Add names to reverse-feature list and check for duplicates.\n for x in lis:\n if x.name in feature_values:\n warning(lineno, \"Duplicate feature value `%s' (feature `%s', previously in feature `%s')\",\n x.name, feature, feature_values[x.name].feature)\n else:\n feature_values[x.name] = x\n x.feature = feature\n # Change the parents list of each value to point to actual featval objects\n # rather than just strings; check for unrecognized and duplicate values.\n for x in lis:\n newpar = []\n for y in x.parents:\n if y in feature_values:\n if feature_values[y] in newpar:\n synerr(\"Duplicate feature value %s as parent of %s, feature %s\",\n y, x.name, feature)\n else:\n newpar.append(feature_values[y])\n else:\n synerr(\"Unrecognized feature value %s as parent of %s, feature %s\",\n y, x.name, feature)\n x.parents = newpar\n\n # Check NODE and its parents to make sure it hasn't been seen before in\n # LIST, adding NODE to LIST as soon as it's seen.\n def check_cycles(node, list):\n if node in list:\n synerr(\"Cycle seen involving feature value %s\", node.name)\n for x in node.parents:\n check_cycles(x, list + [node])\n \n # Check for cycles.\n for x in lis:\n check_cycles(x, [])\n\n # Check NODE and its parents to make sure that ORIGNODE is not reachable.\n def check_reachable(node, orignode):\n if node == orignode:\n return True\n for x in node.parents:\n if check_reachable(x, orignode):\n return True\n return False\n \n # Clean excess parents.\n for x in lis:\n newpar = []\n for y in x.parents:\n for z in x.parents:\n if z != y and check_reachable(z, y):\n break\n else:\n newpar.append(y)\n x.parents = newpar\n\n # Finally: Add to feature list.\n feature_to_values[feature] = lis\n\n# Return XML to go in types.xml.\ndef make_feature_types_xml():\n xml = []\n for (x, featvals) in feature_to_values.iteritems():\n # FIXME! Figure out what's going wrong here.\n# typename = x\n# print \"fv_names_to_values: %s\" % fv_names_to_values\n# if x in fv_names_to_values:\n# typename = fv_names_to_values[x]\n# xml += [['type', [('name', typename)]]]\n xml += [['type', [('name', x)]]]\n for y in featvals:\n if y.parents:\n xml += [['type',\n [('name', y.name),\n ('parents', ' '.join([z.name for z in y.parents]))]]]\n else:\n xml += [['type', [('name', y.name), ('parents', x)]]]\n return xml\n\n# Return XML to go in morph.xml.\ndef make_feature_morph_xml():\n xml = []\n for x in my_sorted(feature_values):\n featval = feature_values[x]\n if featval.macrotie:\n entry = ['macro', [('name', '@%s' % x)]]\n for y in featval.macrotie:\n if type(y) is int:\n entry += [['fs', [('id', y)],\n ['feat',\n [('attr', featval.feature),\n ('val', fv_names_to_values[x])]]]]\n else:\n (wordtie, typename) = y\n entry += [['lf', [],\n ['satop', [('nomvar', wordtie)],\n ['diamond', [('mode', typename)],\n ['prop',\n [('name', fv_names_to_values[x])]]]]]]\n xml += [entry]\n return xml\n\n# Return XML to go in lexicon.xml.\ndef make_feature_lexicon_xml():\n xml = []\n if distributive_features:\n xml.append(['distributive-features',\n [('attrs', ' '.join(distributive_features))]])\n if licensing_feature_xml:\n xml.append(['licensing-features', []] + licensing_feature_xml)\n return xml\n\n%y\n\n# Allow you to override the value inserted by a feature macro, if you\n# really want to (requested by Fred).\n\nfeatval_2: word: $$ = ($1, $1)\n : word COLON word: $$ = ($1, $3)\n\nfeatval_1: featval_2 : $$ = $1 + ([], [])\n : featval_2 LBRACKET word_list RBRACKET : $$ = $1 + ($3, [])\n : featval_2 LPAREN attr_list RPAREN : $$ = $1 + ([], $3)\n : featval_2 LBRACKET word_list RBRACKET LPAREN attr_list RPAREN :\n $$ = $1 + ($3, $6)\n\nfeatval: featval_1:\n (name, value, parents, licensing) = $1\n fv_names_to_values[name] = value\n $$ = CCGFeatval(name, parents, licensing)\n\nset_featval: featval: $$ = CCGFeatvalList([$1])\n : featval LBRACE set_featval_list RBRACE:\n # The set_featval_list returns a CCGFeatvalList, where the direct entries\n # are those actually in the list itself, and the recursive entries\n # are descendants of them. First add ourself as parent to the direct\n # entries. Then move direct into recursive and put ourself as the only\n # direct entry.\n for x in $3.direct:\n x.parents += [$1.name]\n $$ = CCGFeatvalList([$1], $3.direct + $3.recursive)\n\nset_featval_0: set_featval\n : set_featval commas\n\nset_featval_list: set_featval_0\n : set_featval_list set_featval_0:\n $1.direct += $2.direct; $1.recursive += $2.recursive; $$ = $1\n\nfeatvar: NUMBER: $$ = int($1)\n : word_no_numbers: $$ = ($1, $1)\n : word_no_numbers COLON word: $$ = ($1, $3)\n\nfeatvar_0 : featvar\n : featvar commas\n\nopt_featspec: empty : $$ = (None, None)\n : LESS featvar_0+ GREATER : $$ = ($2, None)\n : LPAREN attr_list RPAREN : $$ = (None, $2)\n : LESS featvar_0+ GREATER LPAREN attr_list RPAREN: $$ = ($2, $5)\n\nopt_feature_bang: empty | BANG\n\n# We declare this in a slightly strange way to work around the awful bug\n# involving non-recognition of empty RHS rules.\nfeature_decl_tail: SEMI: $$ = []\n : COLON set_featval_list SEMI: $$ = $2\n\nfeature_decl : opt_feature_bang word opt_featspec feature_decl_tail:\n if $1:\n global distributive_features\n distributive_features.append($2)\n if $4:\n feature_values = $4.direct + $4.recursive\n else:\n feature_values = []\n install_feature($2, feature_values, $@.lineno(0))\n (macrotie, licensing) = $3\n # Add macro-tie info to each feature\n for x in feature_values:\n x.macrotie = macrotie\n # Handle licensing attributes on the feature values\n global licensing_feature_xml\n for x in feature_values:\n if x.licensing:\n licensing_feature_xml.append(\n ['feat',\n [('attr', $2), ('val', fv_names_to_values[x.name])]\n + x.licensing])\n # Handle licensing attributes on the feature itself rather than\n # on a feature value\n if licensing:\n licensing_feature_xml.append(['feat', [('attr', $2)] + licensing])\n\nfeature_block : FEATURE LBRACE feature_decl* RBRACE\n\n#############################\n# Atomic categories #\n#############################\n\nunification_id: NUMBER: $$ = ('id', $1)\n : TILDE NUMBER: $$ = ('inheritsFrom', $2)\n\nunification_id_0: unification_id\n : unification_id commas\n\nunification_id_spec: LESS unification_id_0* GREATER: $$ = $2\n\natomcat_bracket_entry : word EQUALS word :\n $$ = ['feat', [('attr', $1), ('val', $3)]]\n\natomcat_bracket_entry : word EQUALS STAR :\n $$ = ['feat', [('attr', $1), ('val', '[*DEFAULT*]')]]\n\n# FIXME!!!! Be more intelligent in determining how to separate nomvars\n# and featvars, instead of just using some isupper() hacks. We should\n# check to see if the nomvars are represented in the corresponding\n# hylo spec. We should also output warnings if a bare word occurs and\n# it is not identified anywhere as either a nomvar (should appear in hylo),\n# a featvar (should appear in feature {}), or a featval (likewise).\n\natomcat_bracket_entry : word :\n if $1[0].isupper() and (len($1) == 1 or not $1[1].isupper()):\n $$ = ['feat', [('attr', 'index')],\n ['lf', [], ['nomvar', [('name', $1)]]]]\n elif $1 in feature_values:\n $$ = ['feat', [('attr', feature_values[$1].feature), ('val', $1)]]\n else:\n $$ = ['feat', [('attr', $1)],\n ['featvar', [('name', \"%s\" % $1.upper())]]]\n\natomcat_bracket_entry : word COLON word:\n $$ = ['feat', [('attr', $1)],\n ['featvar', [('name', \"%s:%s\" % ($1.upper(), $3))]]]\n\natomcat_bracket_entry_0: atomcat_bracket_entry\n : atomcat_bracket_entry commas\n\natomcat_bracket : LBRACKET atomcat_bracket_entry_0* RBRACKET : $$ = $2\n\n# The use of word_except_x here is a hack to avoid a reduce/reduce conflict\n# due to the use of x as an operator as well as a word. Without it, the\n# parser doesn't know, e.g., how to disambiguate something beginning\n# FOO/x(... -- is x an operator or a category? The parser only looks one\n# character ahead, so it can't figure this out. With this hack, you cannot\n# use a single lowercase x as a category name without putting it in quotes,\n# e.g. 'x'.\n\natomcat : word_except_x unification_id_spec? atomcat_bracket? :\n $$ = ['atomcat', [('type', $1)], ['fs', $2] + $3]\n\n#############################\n# Slashes #\n#############################\n\n%p\n# Temporary switch to Python mode to insert the needed function\nslash_to_default_mode = {'/':'>', '\\\\':'<', '|':'.'}\nability_to_ability_value = {'@': 'active', '!': 'inert', None:None}\nability_value_to_ability = {'active': '@', 'inert': '!', None:None}\ndef makeslash(direc, mode, ability):\n if not mode:\n mode = slash_to_default_mode[direc]\n if direc == '|':\n direc = None\n ability = ability_to_ability_value[ability]\n return ['slash'] + [(direc and [('dir', direc)] or []) +\n (ability and [('ability', ability)] or []) +\n (mode and [('mode', mode)] or [])]\n%y\n\nbareslash : SLASH | BACKSLASH | PIPE\n\nslash_ability : AT | BANG\n\nslash_mode : X GREATER : $$ = 'x>'\n : LESS X : $$ = '[E] \\ np<2>[X nom] / np<3>[Y acc]\n#\n#\n# XML output:\n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n\n######################\n\n# This is a basic attempt to create a compact (BNF-style) syntax for\n# representing legal XML for categories. The idea is that this could be\n# parsed and used to verify the XML, or perhaps to convert it to some other\n# form. It's not clear this is worth it -- there is already an XML schema\n# notation for describing legal XML (albeit it's extremely obnoxious and\n# verbose), and verifiers for verifying XML given a schema and a piece of\n# XML, and XSLT (again, obnoxiously verbose) for transforming XML.\n\n# category = ( atomcat | complexcat )\n\n# complexcat {\n# atomcat\n# (slash (category | dollar) | dollar | setarg)+\n# lf?\n# }\n\n# basicArg = ( slash | category )\n# dollarArg = ( slash | dollar )\n# dollar(name)\n# setarg { basicArg basicArg+ }\n\n# atomcat(!type=[NMTOKEN]) {\n# fs?\n# lf?\n# }\n \n# fs(id) {\n# (feat(attr='index') {\n# lf { nomvar(name) } }\n# | feat(attr, val)\n# )+\n# }\n\n# slash(dir=('/', '\\\\', '|'), mode=('.', '*', '^', 'x', 'x<', ''),\n# varmodality, ability=('inert', 'active'))\n \n# fs(id): \n# ...\n\n%p\n\n##################\n# Initialization #\n##################\n\n# This maps face names to actual properties. A face is the complete\n# description for how a particular piece of text is to be displayed. The\n# properties can specify the font family, size, bold or not, italic or not,\n# subscript or not, \"scale\" (modify the size by the specified value), and\n# \"inherit\" to inherit from a specified face. If not otherwise given,\n# all faces inherit from the default.\n\nface_mapping_init = {\n # The default mapping should contain a value for all parameters\n 'default':{'family':'times', 'size':16, 'bold':False, 'italic':False,\n 'subscript':False, 'superscript':False, 'scale':100,\n 'foreground':None, 'background':None},\n 'subscript':{'family':'helvetica', 'subscript':True, 'scale':70,\n },\n 'category':{'bold':True, 'family':'helvetica'},\n 'dollar':{},\n 'slash':{},\n 'slash mode':{'subscript':True, 'scale':65},\n 'paren':{},\n 'brace':{},\n 'family name':{'foreground':'blue', 'scale':130, 'bold':True},\n 'lexical item':{'italic':True},\n 'numeric index':{'inherit':'subscript'},\n 'subscript comma':{'inherit':'subscript'},\n 'nomvar':{'inherit':'subscript', 'foreground':'forest green'},\n 'feature':{'inherit':'subscript', 'bold':True, 'foreground':'red'},\n # Ideally the following should be in small caps\n 'caret':{'scale':115},\n 'semname':{'scale':115},\n 'semrole':{'scale':85, 'bold':True},\n 'member heading':{},\n 'member comma':{},\n 'member':{'bold':True},\n }\n\n# Offset to be applied to the baseline to handle subscripts and superscripts,\n# relative to the size of the font of the subscripted/superscripted text.\n# FIXME: Maybe should be relative to the larger size of the base (non-offset)\n# text. This would require that the 'scale' option not be handled in\n# fixup_face_properties() but dealt with at the time that the offset is\n# computed, so that the original text size is still available.\nsubscript_offset = -0.5\nsuperscript_offset = 1\n\n# Factor to scale all: FIXME, not currently working\nzoom_factor = 100\n\n# Merged and fixed up equivalent of the above. This will also have a\n# 'font' property containing the Tk font item corresponding to the\n# family, size, bold, and italic properties.\nface_mapping = {}\n\n# Fix up a derived table of properties. Currently this only handles 'scale'.\n# This destructively modifies the property table.\ndef fixup_face_properties(props):\n if 'scale' in props:\n scale=props['scale']\n del props['scale']\n assert 'size' in props\n # Consider the size to its scaling factor\n props['size'] = int(props['size'] * scale / 100.0 + 0.5)\n family = props['family']\n size = props['size']\n weight = props['bold'] and 'bold' or 'normal'\n slant = props['italic'] and 'italic' or 'roman'\n props['font'] = tkFont.Font(family=family, size=size, weight=weight,\n slant=slant)\n return props\n\n# Merge two tables of properties, with P2 overriding P1. Remove the\n# 'inherit' property in the process. Creates a new table, and does not\n# modify P1 or P2.\ndef merge_face_properties(p1, p2):\n props = {}\n for x in p1:\n if x != 'inherit':\n props[x] = p1[x]\n for x in p2:\n if x != 'inherit':\n props[x] = p2[x]\n return props\n\n# Derive the complete list of properties associated with a face name.\ndef face_properties(name):\n props = face_mapping_init[name]\n # If name is default, return properties directly\n if name == 'default':\n return merge_face_properties(props, {})\n # Else, determine where to inherit from and merge properties with\n # recursively computed value\n if 'inherit' in props:\n inherit = props['inherit']\n else:\n inherit = 'default'\n return merge_face_properties(face_properties(inherit), props)\n\n# Compute the merged properties for all faces.\ndef compute_face_properties():\n for x in face_mapping_init:\n props = fixup_face_properties(face_properties(x))\n face_mapping[x] = props\n\ndef late_init_draw_once():\n compute_face_properties()\n\n#################################\n# Drawing a section of text #\n#################################\n\n# Create tags in a text widget corresponding to the faces and their\n# properties. FIXME: Maybe we should do this only when needed, for each\n# text widget.\ndef create_tags(text):\n for x in face_mapping:\n props = face_mapping[x]\n offs = 0\n if props['subscript']:\n offs = subscript_offset\n elif props['superscript']:\n offs = superscript_offset\n offs = offs*props['size']\n offs = '%sp' % offs # Dimension in points\n text.tag_config(x, font=props['font'], offset=offs)\n fg = props['foreground']\n bg = props['background']\n if fg:\n text.tag_config(x, foreground=fg)\n if bg:\n text.tag_config(x, background=bg)\n \n\n# A \"draw-into\" object, used for incrementally building up some text\n# in various fonts. Initialized with a parent widget and some initial text.\n# Drawing into it is done by calls to text(). When done, call finish()\n# to return a widget containing the text (which can then be packed, gridded,\n# etc.).\nclass draw_into(object):\n def __init__(self, master, width=120):\n self.wid = Text(master, height=3, width=width,\n borderwidth=0, relief=FLAT,\n background='white')\n self.curface = None\n\tself.wid.slash_image = []\n self.curtext = ''\n create_tags(self.wid)\n\n\t# Self.alltext maintains the length of the text printed \n\t# for the current widget\n\tself.alltext = 0\n\n\t# FIXME: the tirgger for bigger height of the Text\n\t# widget is arbitrarily set to 95. This should be \n\t# driven by width of individual fonts and chars\n\tself.expandTrigger = 95\n\n def finish_run(self):\n if self.curtext:\n self.wid.insert(INSERT, self.curtext, (self.curface,))\n #props = face_mapping[self.curface]\n #Label(self.wid, text=self.curtext,\n # font=props['font']).pack(side=LEFT)\n self.curtext = ''\n def text(self, tex, face='default'):\n if self.curface == face:\n self.curtext += tex\n else:\n self.finish_run()\n self.curtext = tex\n self.curface = face\n\n\t# Increase recorded length of text\n\tself.alltext += len(tex)\n\n\t# Increase height if necessary\n\tif (self.alltext > self.expandTrigger):\n\t\theightval = 3* (self.alltext/self.expandTrigger +1)\n\t \tself.wid.config(height= heightval)\n\n\n def finish(self):\n self.finish_run()\n self.wid.config(state=DISABLED)\n return self.wid\n def image(self, img):\n\t# When there is an image to be embedded\n\tself.finish_run()\n\t# Access the OPENCCG_HOME environment variable\n\t# to determine the correct path for the images\n\topenccg_home = os.environ['OPENCCG_HOME']\n\tgifdir = openccg_home+\"/images/slashes/\"\n\timage = PhotoImage(file=gifdir+img)\n\t# We are creating an instantiated variable here\n\t# for the image, because the actual photo object is destroyed once\n\t# the execution leaves the __init__ code. Without building it this way, \n\t# the display was showing only a space for the image but not the image itself\n\tself.wid.slash_image += [image]\n\tself.wid.image_create(INSERT, image=image)\n\n def onHilite(self):\n \tself.wid.config(bg = '#E9FFE3')\n\n def offHilite(self):\n \tself.wid.config(bg = 'white')\n\ndef category_draw_children(into, chils, depth, vars, need_initial_comma=False,\n sep='', sepface='default'):\n for x in chils:\n if sep and need_initial_comma:\n into.text(sep, sepface)\n need_initial_comma = True\n category_draw(into, x, depth=depth + 1, vars=vars)\n\n# Given the XML for a category, draw a graphical representation into the\n# widget INTO. The drawing is done by calling into.text(TEXT, FACE)\ndef category_draw(into, xml, depth, vars):\n ty = xml[0]\n props = xml[1]\n chils = xml[2:]\n if ty == 'complexcat':\n if depth > 0:\n into.text('(', 'paren')\n category_draw_children(into, chils, depth, vars)\n if depth > 0:\n into.text(')', 'paren')\n elif ty == 'atomcat':\n into.text(getprop('type', props), 'category')\n category_draw_children(into, chils, depth, vars)\n elif ty == 'setarg':\n into.text('{', 'brace')\n category_draw_children(into, chils, depth, vars)\n into.text('}', 'brace')\n elif ty == 'fs':\n needcomma = False\n if vars.show_feat_id.get():\n idval = getoptprop('id', props)\n if idval:\n #into.text('<%s>' % idval, 'numeric index')\n into.text('%s' % idval, 'numeric index')\n needcomma = True\n if vars.show_feat_struct.get():\n category_draw_children(into, chils, depth, vars,\n need_initial_comma=needcomma, sep=',',\n sepface='subscript comma')\n elif ty == 'feat':\n attr = getprop('attr', props)\n if attr == 'index':\n assert len(chils) == 1\n assert chils[0][0] == 'lf'\n chils = chils[0][2:]\n assert len(chils) == 1\n assert chils[0][0] == 'nomvar'\n into.text(getprop('name', chils[0][1]), 'nomvar')\n else:\n val = getoptprop('val', props, None)\n if val:\n if vars.show_full_features.get():\n into.text(\"%s=%s\" % (attr, getprop('val', props)),\n 'feature')\n else:\n into.text(\"%s\" % getprop('val', props), 'feature')\n else:\n into.text(\"%s\" % attr, 'feature')\n elif ty == 'slash':\n dir = getoptprop('dir', props, '|')\n mode = getoptprop('mode', props)\n ability = getoptprop('ability', props)\n# into.text('%s' % dir, 'slash')\n# into.text('%s%s' % (mode or '',\n# ability_to_ability_value[ability] or ''),\n# 'slash mode')\n \t\n\t# We create the file name here\n\t# By interpreting various parameters\n\t# and joiing them together as a string\n\t\n\tif dir == '\\\\':\n\t\tslash_string = 'bk'\n\telif dir == '/':\n\t\tslash_string = 'fd'\n\telse:\n\t\tslash_string = 'str'\n\n\t#slash_mode : X GREATER : $$ = 'x>'\n\t# : LESS X : $$ = '':'cross_greater',\n\t\t '':'greater',\n\t\t '<':'lesser',\n\t\t 'x':'cross',\n\t\t '.':'dot',\n\t\t '*':'star',\n\t\t '^':'box'}\n\tif mode == None:\n\t\timage_string = slash_string + '.GIF'\n\telse:\n\t\timage_string = slash_string+ '_' + modelist[mode] + '.GIF'\n\t\t\n\tinto.image(image_string)\n\n elif ty == 'dollar':\n name = getoptprop('name', props)\n into.text('$', 'dollar')\n into.text('%s' % name, 'numeric index')\n else:\n\t# Have commented the following assert Statement\n\t# and the debug statement\n\t# Because of validation errors\n #debug('ty??? %s\\n' % ty)\n #assert False\n\tdummy = 1\n\n%y\n\ncomplexcat_entry : atomcat\n : LPAREN complexcat RPAREN : $$ = $2\n\ncomplexcat_postmod : DOLLAR NUMBER :\n $$ = [['slash', []], ['dollar', [('name', $2)]]]\n\ncomplexcat_postmod : slash complexcat_entry : $$ = [$1, $2]\n\n#complexcat_postmod : slash DOLLAR NUMBER :\n# $$ = [$1, ['dollar', [('name', $3)]]]\n\ncomplexcat : complexcat_entry\n : complexcat complexcat_postmod :\n if $1[0] != 'complexcat':\n $1 = ['complexcat', []] + [$1]\n $$ = $1 + $2\n\ncat_set_entry : slash complexcat_entry : $$ = [$1, $2]\n\ncat_set_entry_0: cat_set_entry\n : cat_set_entry commas\n\ncomplexcat : complexcat LBRACE cat_set_entry_0+ RBRACE :\n # $3 comes as a list of lists of the form [slash, cat]; we need to\n # flatten the list, which is what reduce() does\n if $1[0] != 'complexcat':\n $1 = ['complexcat', []] + [$1]\n $$ = $1 + [['setarg', []] + reduce(lambda x,y:x+y, $3)]\n\n#############################\n# Hybrid logic #\n#############################\n\n# Example:\n# \n# Source:\n# \n# E:action(* X:animate-being Y:sem-obj)\n# \n# XML output:\n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n# \n\n%p\n\ndef hylo_draw_children(into, chils, need_initial_comma=False,\n sep='', sepface='caret'):\n for x in chils:\n if sep and need_initial_comma:\n into.text(sep, sepface)\n need_initial_comma = True\n hylo_draw(into, x)\n\n# Given the XML for a hylo, draw a graphical representation into the\n# widget INTO. The drawing is done by calling into.text(TEXT, FACE)\ndef hylo_draw(into, xml):\n ty = xml[0]\n props = xml[1]\n chils = xml[2:]\n if ty == 'satop':\n into.text('@')\n into.text(getprop('nomvar', props), 'nomvar')\n into.text('(')\n hylo_draw_children(into, chils, sep=' ^ ')\n into.text(')')\n elif ty == 'prop':\n name = getprop('name', props)\n if name == '[*DEFAULT*]':\n into.text('*', 'semname')\n else:\n into.text(name, 'semname')\n elif ty == 'diamond':\n mode = getprop('mode', props)\n # FIXME: Instead of uppercasing, we really want small caps\n into.text('<%s>' % mode.upper(), 'semrole')\n assert len(chils) > 0\n if len(chils) == 1:\n hylo_draw(into, chils[0])\n else:\n into.text('(')\n hylo_draw_children(into, chils, sep='^')\n into.text(')')\n elif ty == 'nomvar':\n into.text(getprop('name', props), 'nomvar')\n else:\n assert False\n%y\n\nhylo_entry : STAR : $$ = ['prop', [('name', '[*DEFAULT*]')]]\n : typed_word :\n if $1[0].isupper():\n $$ = ['nomvar', [('name', $1)]]\n else:\n $$ = ['prop', [('name', $1)]]\n\nhylo_entry : LESS word GREATER hylo_entry :\n $$ = ['diamond', [('mode', $2)], $4]\n\nhylo_entry : LESS word GREATER LPAREN hylo_list RPAREN :\n $$ = ['diamond', [('mode', $2)]] + $5\n\ncarets : CARET\n : carets CARET\n\nhylo_entry_0: hylo_entry\n : hylo_entry carets\n\nhylo_list : empty\n : hylo_list hylo_entry_0 : $$ = $1 + [$2]\n\nhylo_list_0 : hylo_list\n : carets: $$ = []\n : carets hylo_list: $$ = $2\n\nhylo_spec : typed_word LPAREN hylo_list_0 RPAREN :\n $$ = ['satop', [('nomvar', $1)]] + $3\n\nhybrid_logic : hylo_spec*\n : AT hylo_spec*: $$ = $2\n\n#############################\n# Words #\n#############################\n\n%p\n\ndef init_morphology():\n global morph_xml\n morph_xml = []\n\n # List families/parts-of-speech of a word. This comes from the\n # families/parts-of-speech specified in a word {} declaration; hence we\n # can't really tell families from POS's. This also comes from any\n # member declarations inside of a family.\n global word_to_family_pos\n word_to_family_pos = {}\n # List word members of a family/part-of-speech; more or less the\n # inverse of the previous hash. (Not a perfect inverse because it\n # doesn't currently list any members that come from a member\n # declaration inside of a family, but only from word {} declarations.)\n global family_pos_to_word\n family_pos_to_word = {}\n # word->predicate mapping; this comes from pred=foo declarations in the\n # properties of a word. This is needed because this info must be added\n # to tags in a family.\n global word_to_predicate\n word_to_predicate = {}\n # Mapping of families to parts-of-speech; comes from family {}\n # declarations.\n global family_to_pos\n family_to_pos = {}\n # Contains a key for each part-of-speech seen in a family {}\n # declaration.\n global pos_hash\n pos_hash = {}\n # (XML for) list of word members explicitly specified using a member\n # statement.\n global family_members\n family_members = {}\n\ndef save_morphology(cur):\n cur.morph_xml = morph_xml\n cur.word_to_family_pos = word_to_family_pos\n cur.family_pos_to_word = family_pos_to_word\n cur.word_to_predicate = word_to_predicate\n cur.family_to_pos = family_to_pos\n cur.pos_hash = pos_hash\n cur.family_members = family_members\n\n# Assume that hash[key] is a list, add VALUE to the list if not already there.\ndef add_uniquely_to_hash_entry_list(hash, key, value):\n if key not in hash:\n hash[key] = []\n if value not in hash[key]:\n hash[key] += [value]\n\ndef note_family_member(word, families):\n for x in families:\n add_uniquely_to_hash_entry_list(word_to_family_pos, word, x)\n add_uniquely_to_hash_entry_list(family_pos_to_word, x, word)\n\ndef make_word_morph_xml():\n xml = []\n for x in morph_xml:\n word_pos_list = []\n word = getprop('stem', x[1])\n # Each word needs to be listed as many times as it has parts of\n # speech. We collect together all families and POS's associated\n # with a word, either from word {} or member declarations,\n # and determine all POS's from them.\n for y in word_to_family_pos.get(word, []):\n if y in family_to_pos:\n pos = family_to_pos[y]\n elif y in pos_hash:\n pos = y\n else:\n error(None, 'Family/part-of-speech %s not found (word declaration %s)',\n y, word)\n if pos not in word_pos_list:\n word_pos_list += [pos]\n for y in word_pos_list:\n # Make a copy of the word's XML and set the POS appropriately.\n entry = x[:]\n putprop('pos', y, entry[1])\n xml += [entry]\n return xml\n\n%y\n\nword_param: word_list: $$ = ($1, [])\n : word_list LPAREN ext_attr_list RPAREN:\n # WORD(VALUE) is equivalent to WORD(class=VALUE).\n property_name_replace(None, 'class', $3)\n $$ = ($1, $3)\n\nword_spec_1: WORD word COLON word_param:\n (families, params) = $4\n note_family_member($2, families)\n pred = getoptprop('pred', params)\n if pred:\n word_to_predicate[$2] = pred\n $$ = ($2, [('pos', None), ('stem', $2)] + params)\n\nword_spec_1: WORD word COLON: $$ = ($2, [('pos', None), ('stem', $2)])\n\nword_spec: WORD word: $$ = ($2, [('pos', None), ('stem', $2)])\n : word_spec_1\n\nword_block: word_spec SEMI:\n (word, params) = $1\n morph_xml.append(['entry', [('word', word)] + params])\n\nword_block: word_spec_1 COLON word_macros SEMI:\n (word, params) = $1\n morph_xml.append(['entry', [('word', word)] + $3 + params])\n\nword_macros: word_list: $$ = [('macros', ' '.join(['@%s' % x for x in $1]))]\n\nword_form: word_or_star SEMI: $$ = ($1, [])\n\nword_form: word_or_star COLON word_macros SEMI: $$ = ($1, $3)\n\nword_forms: : $$ = []\n : word_forms word_form : $$ = $1 + [$2]\n\nword_block: word_spec LBRACE word_forms RBRACE:\n (word, params) = $1\n for (form, macros) in $3:\n if form == '*':\n form = word\n morph_xml.append(['entry', [('word', form)] + macros + params])\n\n#############################\n# Family blocks #\n#############################\n\n%p\n\ndef init_lexicon():\n global lexicon_xml\n lexicon_xml = []\n\ndef save_lexicon(cur):\n cur.lexicon_xml = lexicon_xml\n\n# lexicon_xml already contains XML for each family and its entries (i.e.\n# lexical insertion rules). We also need to add to each family the words\n# that are members of the family -- these come from both word {}\n# declarations and member statements.\ndef make_family_lexicon_xml():\n for x in lexicon_xml:\n # Make sure that open families don't have member entries, or otherwise\n # [*DATE*], [*NUM*], etc. won't work.\n closed = getprop('closed', x[1])\n if closed == 'false':\n continue\n name = getprop('name', x[1])\n words_seen = []\n # Add each stem explicitly given in a member statement. The\n # predicate comes from any predicate given in the member statement\n # along with the stem, or from the word {} declaration as a backup.\n for y in family_members[name]:\n stem = getprop('stem', y[1])\n words_seen += [stem]\n pred = getoptprop('pred', y[1])\n if not pred:\n pred = word_to_predicate.get(stem, None)\n x += [['member',\n [('stem', stem)] + (pred and [('pred', pred)] or [])]]\n # Add each stem that specifies that it belongs to this family,\n # unless we already added it.\n for y in family_pos_to_word.get(name, []):\n if y not in words_seen:\n words_seen += [y]\n pred = word_to_predicate.get(y, None)\n x += [['member',\n [('stem', y)] + (pred and [('pred', pred)] or [])]]\n return lexicon_xml\n\n\n# A CSFamily is a `family {}' block.\nclass CSFamily(CSBlock):\n def __init__(self, prod, name, props, statements):\n super(CSFamily, self).__init__(prod)\n self.name = name\n self.props = props\n self.statements = statements\n\tself.text = None\n\tself.homeButton = None\n\tself.btnFrame = None\n\tself.menuHolder = None\n\n\tself.childFrame = None\n\tself.cfile = None\n\tself.cf = None\n\tself.vars = None\n\tself.canvas = None\n\tself.mainFrame = None\n\n def draw(self, childFrame, cfile, vars, row, canvas, mainFrame):\n # Draw the family name\n f = Frame(childFrame, bd=1, relief=SUNKEN, background='white')\n cf = draw_into(f, width=20)\n cf.text('%s' % self.name, 'family name')\n\n\tchild_widget=cf.finish()\n\tself.menuHolder = child_widget\n\n child_widget.pack(fill=BOTH, expand=YES)\n\n\tchild_widget.bind(\"\", self.editPopup)\n\n\tself.childFrame = childFrame\n\tself.cfile = cfile\n\tself.cf = cf\n\tself.vars = vars\n\tself.canvas = canvas\n\tself.mainFrame = mainFrame\n\n\n f.grid(row=row, column=0, sticky=NSEW)\n\n # Draw the various statements\n f = Frame(childFrame, bd=1, relief=SUNKEN, background='white')\n for x in self.statements:\n frame = x.draw(f, cfile, vars)\n if frame:\n frame.pack(fill=BOTH, expand=YES)\n f.grid(row=row, column=1, sticky=NSEW)\n\n childFrame.rowconfigure(row, weight=1)\n\n # Define the binding procedure for the right-click for editing an entry\n def editPopup(self, event):\n\tpopup = Menu(self.menuHolder, tearoff =0)\n\tpopup.add_command(label=' Edit ', command = lambda: self.editSection(self.childFrame, \n\t\t\t\tself.cfile, \n\t\t\t\tself.cf, \n\t\t\t\tself.vars, \n\t\t\t\tself.canvas, \n\t\t\t\tself.mainFrame))\n\ttry:\n\t\tpopup.tk_popup(event.x_root+40, event.y_root, 0)\n\tfinally:\n\t\tpopup.grab_release()\n\t\n\t# Now bind the right-click to the saveSection buttons\n\tself.menuHolder.bind(\"\", self.savePopup)\n\n # Define the right click binding for the save entry\n def savePopup(self, event):\n \tpopup = Menu(self.menuHolder, tearoff = 0)\n\tpopup.add_command(label = 'Done', command = lambda: self.saveSection(self.childFrame,\n\t\t\t\t\t\tself.cfile,\n\t\t\t\t\t\tself.cf,\n\t\t\t\t\t\tself.vars,\n\t\t\t\t\t\tself.canvas,\n\t\t\t\t\t\tself.mainFrame))\n\tpopup.add_command(label = 'Home', command = lambda: self.editHome(self.cfile))\n\n\tfileData = self.cfile.getAllText()\n\tpopup.add_command(label = 'Undo All', command = lambda: self.undoEdit(fileData, self.cfile))\n\n\ttry:\n\t\tpopup.tk_popup (event.x_root+40, event.y_root, 0)\n\tfinally:\n\t\tpopup.grab_release()\n \n # Edit a section, i.e. a family of the grammar individually rather than the entire grammar\n # Note that this will have very preliminary editing capabilities and the complete grammar\n # editing should be done through the Edit global view\n def editSection(self, childFrame, cfile, hiliteText, vars, canvas, mainFrame):\n editFrame = Frame(mainFrame, bd=1, background='white')\n\n \tself.text = Text(editFrame, padx=5, wrap=None, undo = YES, background='white', height =10)\n\tvbar = Scrollbar(editFrame)\n\thbar = Scrollbar(editFrame, orient='horizontal')\n\n\tself.text.config(yscrollcommand=vbar.set) # call vbar.set on text move\n self.text.config(xscrollcommand=hbar.set)\n vbar.config(command=self.text.yview) # call text.yview on scroll move\n hbar.config(command=self.text.xview) # or hbar['command']=text.xview\n\n\t# Changing the mode of the cfile object here,\n\t# so that once the uer clicks done,\n\t# the whole object is recompiled and redisplayed\n\tcfile.mode= 'Edit'\n\n\t# Highlight the row being edited\n\thiliteText.onHilite()\n\n\tvbar.pack(side=RIGHT, fill=Y)\n\thbar.pack(side=BOTTOM, fill=X)\n\tself.text.pack(fill= BOTH, expand= YES)\n\n\t# Set a mark at the beginning of the text\n\tself.text.mark_set(\"START\", INSERT)\n\tself.text.mark_gravity(\"START\", LEFT)\n\n\t# Push in the rest of the file's contents\n\tfileData = cfile.getAllText()\n\tself.text.insert(INSERT, fileData)\n\n\t# Move the insert position to the first occurence of the family name\n\t# FIXME: this is poor implementation\n\t# The positioning of the insert cursor should be happening by parsing the \n\t# CFG production rules, using CSFamily.prod.lineno and endlineno\n\tself.text.config(takefocus=True)\n\tidx= self.text.search('family '+ self.name, \"START\")\n\tself.text.mark_set(CURRENT, idx)\n\tself.text.see(CURRENT)\n\n #editFrame.grid(row=row+1, columnspan =3, sticky = NSEW)\n editFrame.grid(row=2, columnspan =2, sticky = NSEW)\n\tchildFrame.update_idletasks()\n\tcanvas.config(scrollregion=canvas.bbox(\"all\"))\n\n # Finished editing\n #def saveSection(self, childFrame, cfile, hiliteText, varset, canvas, mainFrame, homeButton, undoButton):\n def saveSection(self, childFrame, cfile, hiliteText, varset, canvas, mainFrame):\n \t# We force the text contents of the cfile object to copy over \n\t# all that is presently in the current text-box\n \tcfile.setAllText(self.text.get(1.0,END))\n\n\t# Undo the highlight of the row\n\thiliteText.offHilite()\n\n\t# Recompile whatever was edited and redisplay\n\t# Note: changes are not saved hereby!!\n\tcfile.compile_if_needed()\n\tcfile.onLexicon()\n\n\t# Restore the right-click binding to the original\n\tself.menuHolder.bind(\"\", self.editPopup)\n\n # Restore view to original place where you wanted to edit\n def editHome(self, cfile):\n\t# Move the insert position to the first occurence of the family name\n\t# FIXME: this is poor implementation\n\t# The positioning of the insert cursor should be happening by parsing the \n\t# CFG production rules, using CSFamily.prod.lineno and endlineno\n\tself.text.config(takefocus=True)\n\tidx= self.text.search('family '+ self.name, \"START\")\n\n\tif not idx:\n\t\tshowwarning('Error', 'Original entry for '+self.name+ ' not found!')\n\tself.text.mark_set(CURRENT, idx)\n\tself.text.see(CURRENT)\n\n # Undo all editing done till now\n def undoEdit(self, fileData, cfile):\n \taskqn = askokcancel('Warning','Undo all changes till now?')\n\tif askqn:\n\t\tself.text.delete(\"START\", END)\n\t\tself.text.insert(CURRENT, fileData)\n\t\tself.editHome(cfile)\n\n \t\n\n# CSFamilyEntry is an `entry' statement inside a `family' block.\n#\n# PROPS is a property list corresponding to the entry's name ('name') and\n# any other properties, deriving from the form\n#\n# entry NAME(PROP=VAL, ...):\n#\n# Either the name or properties, or both, may be omitted.\n#\n# CAT is the XML corresponding to the entry's category, and LF is the XML for\n# the logical form (hybrid logic).\n\nclass CSFamilyEntry(CSStatement):\n def __init__(self, prod, props, cat, lf=None):\n super(CSFamilyEntry, self).__init__(prod)\n self.props = props\n # NOTE: self.cat is a single XML statement, but self.lf is a list\n # of XML statements. FIXME.\n self.cat = cat\n self.lf = lf\n def xml(self):\n if self.lf:\n lf = [['lf', []] + self.lf]\n else:\n lf = []\n return [['entry', self.props, self.cat + lf]]\n def draw(self, parent, cfile, vars):\n name = getoptprop('name', self.props)\n f = Frame(parent, background='white')\n cf = draw_into(f)\n cf.text(' ')\n if name:\n cf.text('%s: ' % name)\n category_draw(cf, self.cat, depth=0, vars=vars)\n if self.lf and vars.show_semantics.get():\n cf.text(' : ')\n hylo_draw_children(cf, self.lf)\n cf.finish().pack(fill=BOTH, expand=YES, side=LEFT)\n return f\n\n# CSFamilyMember is a `member' statement inside a `family' block. ITEMS\n# lists the items given, in property-list form:\n#\n# STEM --> [('stem', STEM)]\n# STEM(PRED) --> [('stem', STEM), ('pred', PRED)]\n\nclass CSFamilyMember(CSStatement):\n def __init__(self, prod, items):\n super(CSFamilyMember, self).__init__(prod)\n self.items = items\n def xml(self):\n return [['member', x] for x in self.items]\n def draw(self, parent, cfile, vars):\n return None\n f = Frame(parent, background='white', bd=1, relief=SUNKEN)\n cf = draw_into(f)\n cf.text('Members: ', 'member heading')\n first = True\n for x in self.items:\n stem = getprop('stem', x)\n pred = getoptprop('pred', x)\n if not first:\n cf.text(', ', 'member comma')\n cf.text(' %s%s' % (stem, pred and \"(pred=%s)\" % pred or ''),\n 'member')\n first = False\n\n\tprint len (self.items) \n cf.finish().pack(fill=BOTH, expand=YES)\n return f\n\n%y\n\n# Omitting the colon between entry category and hybrid logic doesn't\n# actually cause parsing problems, but it's probably not a good idea to\n# encourage this, because the syntax might change in the future.\n\nentry_name_1: opt_paren_attr_list\n\n# We shouldn't need the first entry below, but we do, due to the bugginess\n# in PLY in handling empty rules.\nentry_name: word: $$ = [('name', $1)]\n : word entry_name_1: $$ = [('name', $1)] + $2\n : entry_name_1\n\nentry : ENTRY entry_name COLON complexcat COLON hybrid_logic SEMI :\n $$ = CSFamilyEntry($@, props=$2, cat=$4, lf=$6)\n\nentry : ENTRY entry_name COLON complexcat SEMI :\n $$ = CSFamilyEntry($@, props=$2, cat=$4)\n\nmember_entry : word : $$ = [('stem', $1)]\nmember_entry : word LPAREN word RPAREN :\n $$ = [('stem', $1), ('pred', $3)]\n\nmember_entry_0: member_entry\n : member_entry commas\n\nmember : MEMBER COLON member_entry_0+ SEMI\t:\n $$ = CSFamilyMember($@, items=$3)\n\nfamily_statement : member | entry\n\nfamily_statement_list : empty\n\t\t : family_statement_list family_statement : $$ = $1 + [$2]\n\nfamily_block : FAMILY word opt_paren_ext_attr_list LBRACE family_statement_list RBRACE :\n # FAMILY(VALUE) is equivalent to FAMILY(pos=VALUE).\n property_name_replace(None, 'pos', $3)\n # Create the AST object -- before adding to $3.\n $$ = CSFamily($@, name=$2, props=$3, statements=$5)\n # 'pos' (part of speech) defaults to the family name; they would only\n # differ when more than one family is used to define a particular part of\n # speech, to handle related characteristics (family Prep-Nom vs. pos Prep).\n if not property_specified('pos', $3):\n $3 += [('pos', $2)]\n\n # Store mappings related to POS.\n pos = getprop('pos', $3)\n family_to_pos[$2] = pos\n pos_hash[pos] = True\n\n # Now construct the XML for the family\n xml = ['family', [('name', $2)] + $3]\n family_members[$2] = []\n for x in $5:\n if type(x) is CSFamilyMember:\n family_members[$2].extend(x.xml())\n else:\n xml.extend(x.xml())\n # If members have been specified ('member' statements) and there is no\n # 'closed' property, make the family closed.\n \n # if family_members[$2] and not property_specified('closed', xml[1]):\n # xml[1] += [('closed', 'true')]\n \n # Actually, we *always* need classes closed, due to a bizarreness in\n # OpenCCG.\n if not property_specified('closed', xml[1]):\n xml[1] += [('closed', 'true')]\n # Add names to entries ('entry' statements) without them.\n primcount = 0\n for x in xml[2:]:\n if not property_specified('name', x[1]):\n primcount += 1\n x[1] = [('name', 'Entry-%s' % primcount)] + x[1]\n # For each specified member, note the family it's in so that its part\n # of speech can be calculated.\n for x in family_members[$2]:\n add_uniquely_to_hash_entry_list(word_to_family_pos,\n getprop('stem', x[1]),\n $2)\n lexicon_xml.append(xml)\n $$.static_xml = [xml]\n\n#############################\n# Rule blocks #\n#############################\n\n%p\ndef init_rules():\n global rules\n rules = {\n ('app', '+') : True,\n ('app', '-') : True,\n ('comp', '+') : True,\n ('comp', '-') : True,\n ('xcomp', '+') : True,\n ('xcomp', '-') : True,\n ('sub', '+') : False,\n ('sub', '-') : False,\n ('xsub', '+') : False,\n ('xsub', '-') : False,\n ('typeraise', '+') : [(False, True, True)],\n ('typeraise', '-') : [(True, True, True)],\n 'typechange' : [],\n }\n\n global rules_to_xml_mapping\n rules_to_xml_mapping = {\n 'app' : ['application', []],\n 'comp' : ['composition', [('harmonic', 'true')]],\n 'xcomp' : ['composition', [('harmonic', 'false')]],\n 'sub' : ['substitution', [('harmonic', 'true')]],\n 'xsub' : ['substitution', [('harmonic', 'false')]],\n }\n\ndef save_rules(cur):\n cur.rules = rules\n cur.rules_to_xml_mapping = rules_to_xml_mapping\n\ndef make_rules_xml():\n xml = []\n unique = 0\n for (key, value) in my_sorted(rules.items()):\n if type(key) is tuple and key[0] in rules_to_xml_mapping:\n rx = copy.deepcopy(rules_to_xml_mapping[key[0]])\n rx[1] += [('dir', key[1] == '+' and 'forward' or 'backward')]\n xml.append(rx)\n elif type(key) is tuple and key[0] == 'typeraise':\n for (dollar, arg, result) in value:\n xml.append(['typeraising',\n [('dir', key[1] == '+' and 'forward'\n or 'backward'),\n ('useDollar', dollar and 'true' or 'false')]]\n + (arg != True and\n [['arg', [], arg]]\n or [])\n + (result != True and\n [['result', [], result]]\n or []))\n elif key == 'typechange':\n for (arg, result, lf) in value:\n unique += 1\n \tif lf:\n\t lf = [['lf', []] + lf]\n\t else:\n\t lf = []\n xml.append(['typechanging', [('name', 'typechange-%d' % unique)],\n ['arg', [], arg],\n ['result', [], result + lf]])\n else:\n raise InternalError(\"Invalid element in rules hash: %s\" % str(key))\n return xml\n\ndef dotyperaise(plusminus, dollarp, arg, result):\n if plusminus == '+' or plusminus == '+-':\n rules[('typeraise', '+')] += [(dollarp, arg, result)]\n if plusminus == '-' or plusminus == '+-':\n rules[('typeraise', '-')] += [(dollarp, arg, result)]\n\ndef rulesreinit():\n rules.clear()\n rules[('typeraise', '+')] = []\n rules[('typeraise', '-')] = []\n rules['typechange'] = []\n%y\n\nruletype : APP | COMP | XCOMP | SUB | XSUB\n\nopt_dollar : DOLLAR : $$ = True\n : empty : $$ = False\n\nopt_atomcat : atomcat\n : empty : $$ = True\n\nopt_complexcat : COLON complexcat : $$ = $2\n : empty : $$ = True\n\nplusminus_spec : PLUS | MINUS | PLUSMINUS\n\nrule : NO SEMI : rulesreinit()\n : NO ruletype SEMI | NO ruletype PLUSMINUS SEMI : \\\n del rules[($2, '+')]; del rules[($2, '-')]\n : NO ruletype PLUS SEMI : del rules[($2, '+')]\n : NO ruletype MINUS SEMI : del rules[($2, '-')]\n : NO TYPERAISE SEMI | NO TYPERAISE PLUSMINUS SEMI : \\\n rules[('typeraise', '+')] = []; rules[('typeraise', '-')] = []\n : NO TYPERAISE PLUS SEMI : rules[('typeraise', '+')] = []\n : NO TYPERAISE MINUS SEMI : rules[('typeraise', '-')] = []\n : NO TYPECHANGE SEMI : rules['typechange'] = []\n : ruletype PLUSMINUS SEMI : \\\n rules[($1, '+')] = True; rules[($1, '-')] = True\n : ruletype PLUS SEMI : rules[($1, '+')] = True\n : ruletype MINUS SEMI : rules[($1, '-')] = True\n : TYPERAISE plusminus_spec opt_dollar\n opt_complexcat SEMI: dotyperaise($2, $3, $4, True)\n : TYPERAISE plusminus_spec opt_dollar\n COLON complexcat GOESTO opt_atomcat SEMI: dotyperaise($2, $3, $5, $7)\n : TYPECHANGE COLON complexcat GOESTO complexcat SEMI: \\\n rules['typechange'] += [($3, $5, None)]\n : TYPECHANGE COLON complexcat GOESTO complexcat COLON hybrid_logic SEMI: \\\n rules['typechange'] += [($3, $5, $7)]\n\nrule_list : rule_list rule\n : empty\n\nrule_block : RULE LBRACE rule_list RBRACE\n\n#############################\n# Testbed #\n#############################\n\n%p\n\ndef init_testbed():\n global testbed_statements\n testbed_statements = []\n\ndef save_testbed(cur):\n cur.testbed_statements = testbed_statements\n\ndef add_testbed_statement(bang, words, number):\n testbed_statements.append(['item',\n [('string', ' '.join(words))] + bang +\n number])\n\ndef make_testbed_xml():\n return testbed_statements\n\n%y\n\nopt_testbed_bang: BANG: $$ = [('known', 'true')]\n : empty\n\ntestbed_entry: opt_testbed_bang word_list SEMI: \\\n add_testbed_statement($1, $2, [])\n : opt_testbed_bang word_list COLON NUMBER SEMI: \\\n add_testbed_statement($1, $2, [('numOfParses', $4)])\n\ntestbed_block: TESTBED LBRACE testbed_entry* RBRACE\n\n#############################\n# Relation-sorting #\n#############################\n\n%p\n\ndef init_relation_sorting():\n global relation_sorting\n relation_sorting = []\n\ndef save_relation_sorting(cur):\n cur.relation_sorting = relation_sorting\n\ndef make_relation_sorting_lexicon_xml():\n if relation_sorting:\n return [['relation-sorting',\n [('order', ' '.join(relation_sorting))]]]\n else:\n return []\n\n%y\n\nrelation_sorting_block: RELATION_SORTING COLON word_or_star_0 * SEMI:\n global relation_sorting\n relation_sorting += $3\n\n#############################\n# End Yacc Declarations #\n#############################\n%p\n\ndef p_error(p):\n if p:\n error(p.lineno, \"Syntax error at '%s'\", p.value)\n else:\n error(None, \"Unexpected end of file\")\n\n#############################\n# Lexer classes #\n#############################\n\n# A Lexer that allows for a list of tokens to be pushed onto the front of\n# the list of tokens to be returned. Any number of such lists can be\n# pushed.\nclass StackLexer(object):\n def __init__(self, lexer):\n self.lexer = lexer\n self.tokenstack = []\n self.tokenstackind = []\n self.lineno = 1\n\n def input(self, s):\n self.lexer.input(s)\n\n def pushstack(self, stack):\n self.tokenstack.append(stack)\n self.tokenstackind.append(0)\n\n def token(self):\n global return_bogus_value\n if return_bogus_value:\n return_bogus_value = 0\n tok = CCGToken('BOGUS_VALUE', 'BOGUS_VALUE')\n tok.lineno = self.lineno\n return tok\n while self.tokenstack:\n try:\n tok = self.tokenstack[-1][self.tokenstackind[-1]]\n self.tokenstackind[-1] += 1\n return tok\n except IndexError:\n self.tokenstack.pop()\n self.tokenstackind.pop()\n if self.lexer:\n tok = self.lexer.token()\n if tok:\n self.lineno = tok.lineno\n return tok\n return None\n\n# A Lexer that checks for macro calls and expands them appropriately.\nclass MacroLexer(StackLexer):\n def __init__(self, lexer):\n self.last_token = None\n self.indentlevel = 0\n super(MacroLexer, self).__init__(lexer)\n\n def simpletoken(self):\n return super(MacroLexer, self).token()\n\n def noeoftoken(self):\n tok = self.innertoken()\n if not tok:\n raise SyntaxError(\"Unexpected EOF\")\n return tok\n \n def innertoken(self):\n macrotok = self.simpletoken()\n if not macrotok or no_macro_sub or \\\n not (macrotok.type == 'ID' and macrotok.value in macro_defs):\n return macrotok\n else:\n newtok = self.simpletoken()\n if not newtok or newtok.type != 'LPAREN':\n self.pushstack([newtok])\n return macrotok\n macrodef = macro_defs[macrotok.value]\n args = []\n stop = False\n while not stop:\n thisarg = []\n parencount = 0\n expect_rbrace = 0\n newtok = self.noeoftoken()\n if newtok.type == 'LBRACE':\n parencount += 1\n expect_rbrace = 1\n newtok = self.noeoftoken()\n while True:\n if newtok.type in ['LBRACE', 'LBRACKET', 'LPAREN']:\n parencount += 1\n if newtok.type in ['RBRACE', 'RBRACKET', 'RPAREN']:\n parencount -= 1\n if parencount < 0:\n if newtok.type == 'RPAREN':\n stop = True\n break\n error(newtok.lineno, \"Syntax error at %s\",\n newtok.value)\n parencount = 0\n if parencount == 0 and newtok.type == 'RBRACE' and \\\n expect_rbrace:\n expect_rbrace = 0\n newtok = self.noeoftoken()\n continue\n if parencount == 0 and newtok.type == 'COMMA':\n break\n thisarg.append(newtok)\n newtok = self.noeoftoken()\n args.append(thisarg)\n # Allow extra trailing comma\n if len(args) == len(macrodef.args) + 1 and not args[-1]:\n args.pop()\n if len(args) != len(macrodef.args):\n error(macrotok.lineno,\n \"Invalid number of arguments to macro %s\",\n macrotok.value)\n else:\n if super_macro_debug:\n print \"Processing macro: %s\" % macrotok.value\n self.pushstack(macrosub(macrodef, args, self.lineno))\n return self.innertoken()\n\n def token(self):\n def pretty_output_transformed(token):\n def newline(num=1):\n outout('\\n' * num)\n outout(' ' * 2 * self.indentlevel)\n \n if tok.lineno and self.lineno < tok.lineno:\n if tok.lineno - self.lineno == 1:\n newline()\n else:\n newline(2)\n elif self.last_token and (self.last_token.type == 'RBRACE' or\n self.last_token.type == 'SEMI'):\n newline()\n elif tok.type == 'LBRACE':\n newline(2)\n value = str(tok.value)\n lastval = self.last_token and str(self.last_token.value)\n if value and lastval and ((isalnumund(lastval[0]) and\n isalnumund(value[0]))\n or self.last_token.type\n in ('COLON', 'COMMA')):\n outout(' ')\n if tok.type == 'QUOTEDID':\n outout('\"%s\"', value)\n else:\n outout('%s', value)\n if tok.type == 'LBRACE':\n self.indentlevel += 1\n elif tok.type == 'RBRACE':\n self.indentlevel -= 1\n return tok\n\n # Beginning of actual function\n tok = self.innertoken()\n if options.transformed_input and self.lexer and \\\n tok and tok.type != 'BOGUS_VALUE':\n pretty_output_transformed(tok)\n self.last_token = tok\n # print \"Saw token: %s\" % tok\n return tok\n\n#############################\n# Parsing #\n#############################\n\ndef init_parse_once():\n # Initialize the parser once, at beginning. This does introspection on\n # the rules (i.e. p_*() functions) in this file.\n yacc.yacc(start='top', debug=yacc_debug, method='LALR', write_tables=0)\n\n# Parse a .CCG file whose contents are in STR.\n\nclass parse_results:\n pass\n\ndef parse_string(str):\n retval = parse_results()\n if str:\n retval.parse = yacc.parse(str, lexer=MacroLexer(globallexer))\n else:\n retval.parse = []\n save_global_state(retval)\n return retval\n\n#############################\n# Graphics #\n#############################\n\n# Given the return value from parsing (a list of abstract syntax tree-related\n# objects), draw them into the given frame.\n\ndef draw_parse(parse, cfile, childFrame, vars, canvas, mainFrame):\n row = 0\n\n if parse:\n \tfor x in parse:\n \tif hasattr(x, 'draw'):\n \t\tx.draw(childFrame, cfile, vars, row, canvas, mainFrame)\n \t\trow += 1\n \t# Make the column containing the lexical entries expand as necessary\n \tchildFrame.columnconfigure(1, weight=1)\n \t#frame.grid(column=0)\n\n\n\n#############################\n# Initialization #\n#############################\n\n# We encapsulate all global-variable initialization into a function that\n# can be called repeatedly so we can reinitialize our state and parse more\n# than one file. ARGV is the command-line arguments to parse (normally\n# sys.argv[1:]) and ERRORS_TO_STRING indicates whether to write stdout and\n# stderr output to strings or to the normal output locations.\n\ndef init_global_state(errors_to_string=False):\n init_errors(errors_to_string)\n init_lexer()\n init_macros()\n init_features()\n init_morphology()\n init_lexicon()\n init_testbed()\n init_rules()\n init_relation_sorting()\n\n# When we're finished parsing, save the global state to the specified\n# object, so we can track the parse results for more than one file.\ndef save_global_state(cur):\n save_errors(cur)\n save_lexer(cur)\n save_macros(cur)\n save_features(cur)\n save_morphology(cur)\n save_lexicon(cur)\n save_testbed(cur)\n save_rules(cur)\n save_relation_sorting(cur)\n\ndef init_global_state_once():\n init_parse_once()\n\nlate_init_graphics_done = 0\n# Graphics-related initialization that must be done late, after the first\n# Tk top-level window has been created.\ndef late_init_graphics():\n global late_init_graphics_done\n if not late_init_graphics_done:\n late_init_draw_once()\n late_init_graphics_done = 1\n \n\n#############################\n# Main driver #\n#############################\n\n# Function to output a particular XML file\ndef output_xml_file(prefix, grammar_name, filebase, top_level_tag, xml):\n xml_file = os.path.join(options.dir, '%s%s.xml' % (prefix, filebase))\n if not options.quiet:\n errout('Outputting XML file: %s\\n' % xml_file)\n xml = [top_level_tag, [('name', grammar_name),\n ('xmlns:xsi',\n 'http://www.w3.org/2001/XMLSchema-instance'),\n ('xsi:noNamespaceSchemaLocation',\n '../%s.xsd' % filebase)]] + xml\n fil = open(xml_file, 'w')\n fil.write('\\n')\n print_xml(fil, xml)\n fil.close()\n\ndef make_grammar_xml(prefix):\n return [['lexicon', [('file', '%slexicon.xml' % prefix)]],\n ['morphology', [('file', '%smorph.xml' % prefix)]],\n ['rules', [('file', '%srules.xml' % prefix)]],\n ['types', [('file', '%stypes.xml' % prefix)]]]\n\n# Map saying how to output the specified XML file\noutput_file_map = {\n 'lexicon': ('ccg-lexicon',\n lambda pref: make_feature_lexicon_xml() +\n make_relation_sorting_lexicon_xml() +\n make_family_lexicon_xml()),\n 'rules': ('rules', lambda pref: make_rules_xml()),\n 'morph': ('morph',\n lambda pref: make_word_morph_xml() + make_feature_morph_xml()),\n 'types': ('types', lambda pref: make_feature_types_xml()),\n 'grammar': ('grammar', make_grammar_xml),\n 'testbed': ('regression', lambda pref: make_testbed_xml()),\n }\n\n# Process the --omit-output list.\n\ndef split_output_files(arg):\n files = re.split('[,\\s]+', arg)\n for x in files:\n if x not in output_file_map:\n parser.error('Unknown file in --omit-output argument')\n return files\n\ndef main():\n parse_arguments(sys.argv[1:])\n init_global_state_once()\n init_global_state()\n\n if options.omit_output:\n if options.omit_output[0] == '+':\n output_files = split_output_files(options.omit_output[1:])\n else:\n suppress_output = split_output_files(options.omit_output)\n output_files = []\n for x in output_file_map:\n if x not in suppress_output:\n output_files.append(x)\n else:\n output_files = [x for x in output_file_map]\n \n # Now actually parse the input arguments\n\n prefix = options.prefix\n lastfile = '-'\n args = global_args or ['-']\n \n for arg in args:\n if arg == '-':\n if not options.quiet:\n errout(\"ccg2xml: Processing standard input\\n\")\n fil = sys.stdin\n else:\n if not options.quiet:\n errout(\"ccg2xml: Processing %s\\n\" % arg)\n fil = file(arg)\n lastfile = arg\n if prefix == None:\n (phead, ptail) = os.path.split(arg)\n (pbase, pext) = os.path.splitext(ptail)\n prefix = '%s-' % pbase\n retval = parse_string(fil.read())\n # print \"Retval: %s\\n\" % retval\n \n if macro_debug:\n print_macros()\n \n # Make output directory if needed, and output files\n\n if error_count > 0:\n if not options.quiet:\n maybe_errout('Errors during compilation, files not output.\\n')\n sys.exit(1)\n else:\n if options.dir:\n if not os.path.isdir(options.dir):\n os.makedirs(options.dir)\n else:\n options.dir = '.'\n \n for x in output_files:\n file_info = output_file_map[x]\n output_xml_file(prefix, lastfile, x, file_info[0],\n file_info[1](prefix))\n\nif __name__ == '__main__': # when run as a script\n main()\n\n# Local Variables:\n# mode: python\n# end:\n" }, { "path": "src/ccg2xml/ccg_editor.py", "content": "#!/usr/bin/python\n\n# Author: Ben Wing \n# Date: April 2006\n\n#############################################################################\n# #\n# ccg_editor.ply #\n# #\n# Edit a CCG-format file, graphically. Will have a mode for displaying #\n# CCG files in a friendly fashion and allowing for editing of parts or #\n# all of the file. Will also have a mode for testing a CCG grammar, and #\n# allow for compilation and error-finding under control of the editor. #\n# #\n#############################################################################\n\n# This code is based on PyEdit version 1.1, from Oreilly's Programming\n# Python, 2nd Edition, 2001, by Mark Lutz.\n\nfrom Tkinter import * # base widgets, constants\nfrom tkFileDialog import * # standard dialogs\nfrom tkMessageBox import *\nfrom tkSimpleDialog import *\nfrom tkColorChooser import askcolor\nfrom string import split, atoi\nimport sys, os, string, md5\nimport ccg2xml\nimport Tree\nimport re\n\nSTART = '1.0' # index of first char: row=1,col=0\nSEL_FIRST = SEL + '.first' # map sel tag to index\nSEL_LAST = SEL + '.last' # same as 'sel.last'\n\nFontScale = 0 # use bigger font on linux\nif sys.platform[:3] != 'win': # and other non-windows boxes\n FontScale = 3\n\n# Initial top-level window; it's not clear we need this.\n# FIXME: It sucks that we have to call Tk() to get the first top-level window\n# but Toplevel() for all others. We should be able to call Tk() initially,\n# and then Toplevel() to create all top-level windows, including the first.\nroot = None\n\n# List of all open CFile objects\nopenfiles = {}\nfilenames = []\n\n\nclass CTab(Frame):\n # Initialize this tab. Usually called from a subclass. PARENT is\n # the parent widget, CFILE the CFile object associated with the\n # top-level window, and TABNAME is the name of this tab (that tab\n # will be removed from the toolbar).\n def __init__(self, parent, cfile, tabname):\n Frame.__init__(self, parent)\n self.parent = parent\n self.cfile = cfile\n self.toolbar = None\n self.checkbar = None\n self.menubar = [\n ('File', 0,\n [('Open...', 0, self.cfile.onOpen),\n ('New', 0, self.cfile.onNew),\n ('Save', 0, self.onSave),\n ('Save As...', 5, self.onSaveAs),\n ('Close', 0, self.cfile.onClose),\n 'separator',\n ('Quit VisCCG', 0, self.cfile.onQuit)]\n ),\n ('Tools', 0,\n [('Font List', 0, self.cfile.onFontList),\n ('Pick Bg...', 4, self.cfile.onPickBg),\n ('Pick Fg...', 0, self.cfile.onPickFg),\n ('Color List', 0, self.cfile.onColorList),\n 'separator',\n ('Info...', 0, self.cfile.onInfo)]\n )\n ]\n self.toolbar = [\n # ('Display', self.cfile.onDisplay, {'side': LEFT}),\n ('Edit', self.cfile.onEdit, {'side': LEFT}),\n ('Lexicon', self.cfile.onLexicon, {'side': LEFT}),\n ('Testbed', self.cfile.onTestbed, {'side': LEFT}),\n ('Features', self.cfile.onFeatures, {'side': LEFT}),\n ('Words', self.cfile.onWords, {'side': LEFT}),\n ('Rules', self.cfile.onRules, {'side': LEFT}),\n ('Quit', self.cfile.onClose, {'side': RIGHT}),\n ('Help', self.cfile.help, {'side': RIGHT}),\n ('Save', self.onSave, {'side': RIGHT}),\n ]\n# self.remove_toolbar_button(tabname)\n\n # Add MENU (a tuple corresponding to a single top-level menu item)\n # after the item with the name AFTER.\n def add_menu(self, after, menu):\n newmenu = []\n for x in self.menubar:\n newmenu += [x]\n if x[0] == after:\n newmenu += [menu]\n self.menubar = newmenu\n\n # Remove the toolbar button named NAME.\n def remove_toolbar_button(self, name):\n newtoolbar = []\n for x in self.toolbar:\n if x[0] != name:\n newtoolbar += [x]\n self.toolbar = newtoolbar\n\n def reinit(self):\n pass\n\n #####################\n # File menu commands\n #####################\n\n def onSave(self):\n self.onSaveAs(self.cfile.currfile) # may be None\n\n def onSaveAs(self, forcefile=None):\n file = forcefile or self.cfile.my_asksaveasfilename()\n if file:\n text = self.cfile.getAllText()\n try:\n open(file, 'w').write(text)\n except:\n showerror('CCG Editor', 'Could not write file ' + file)\n else:\n self.cfile.setFileName(file) # may be newly created\n self.cfile.edit_modified(NO)\n self.cfile.last_save_signature = self.cfile.getSignature(text)\n\n\nclass CEdit(CTab):\n def __init__(self, parent, cfile):\n CTab.__init__(self, parent, cfile, 'Edit')\n self.debugFrame= None\n\n # Add a frame here, so that debug mode can be enabled\n # by embedding other objects within this frame\n editFrame = Frame(self, bd=1, bg= 'white')\n editFrame.pack(fill=BOTH, expand=YES, side=TOP)\n\n # Add a button frame, embed the button and\n # link to command for the debug mode\n btnFrame = Frame(editFrame, bd = 1)\n btnFrame.grid (row=0, columnspan=3, sticky=NSEW)\n\n vldButton = Button (btnFrame, text='Validate', command = lambda: self.onValidate(editFrame, cfile))\n vldButton.pack(side=RIGHT)\n\n # Put the main edit window in the row below this\n vbar = Scrollbar(editFrame)\n hbar = Scrollbar(editFrame, orient='horizontal')\n self.text = Text(editFrame, padx=5, wrap='none', undo=YES)\n\n vbar.grid(row=1, column=2, sticky=NS)\n hbar.grid(row=2, columnspan=2, sticky=EW) # pack text last\n self.text.grid(row=1, column=1, sticky=NSEW) # else sbars clipped\n\n editFrame.columnconfigure(1, weight=1)\n editFrame.rowconfigure(1, weight=1)\n\n # Add a list containing line numbers\n self.lineList = Text(editFrame, relief=SUNKEN, bg='white', bd=2, yscrollcommand = vbar.set, width=3)\n self.lineList.grid(row=1, column=0, sticky=NS)\n self.lineList.config(font=self.cfile.fonts[0],\n bg=self.cfile.colors[0]['bg'], fg=self.cfile.colors[0]['fg'])\n\n # TODO: The first time the display of the line numbers\n # strangely doesn't go through --- somehow cfile\n # isn't initialized. However, it works properly in the display.\n # Need to understand why this happens.\n\n try:\n self.showLineNums()\n except KeyError:\n self.text.config(yscrollcommand=vbar.set) # call vbar.set on text move\n self.text.config(xscrollcommand=hbar.set)\n #vbar.config(command=text.yview) # call text.yview on scroll move\n hbar.config(command=self.text.xview) # or hbar['command']=text.xview\n\n self.text.config(font=self.cfile.fonts[0],\n bg=self.cfile.colors[0]['bg'], fg=self.cfile.colors[0]['fg'])\n\n #Setting the movement of the listbox and the text\n #together to be controlled by the scrollbar\n vbar.config(command=self.scrollSet)\n\n self.add_menu('File',\n ('Edit', 0,\n [('Cut', 0, self.onCut),\n ('Copy', 1, self.onCopy),\n ('Paste', 0, self.onPaste),\n 'separator',\n ('Delete', 0, self.onDelete),\n ('Select All', 0, self.onSelectAll)]\n ))\n self.add_menu('Edit',\n ('Search', 0,\n [('Goto...', 0, self.cfile.onGoto),\n ('Find...', 0, self.cfile.onFind),\n ('Refind', 0, self.cfile.onRefind),\n ('Change...', 0, self.onChange)]\n ))\n\n def scrollSet(self, *args):\n self.lineList.yview(*args)\n self.text.yview(*args)\n\n def reinit(self):\n self.showLineNums()\n self.text.focus()\n\n def showLineNums(self):\n #Make the list of lines editable\n self.lineList.config(state=NORMAL)\n textData = self.cfile.getAllText()\n listOfLines = textData.split('\\n')\n for num in range(1,len(listOfLines)):\n self.lineList.insert(END,\"%s\\n\" % num)\n #Now that we are done changing the number of lines,\n #we reset the text to be uneditable\n self.lineList.config(state=NORMAL)\n\n def onValidate(self, editFrame, cfile):\n #showwarning(title= 'Sorry', message='Validate and debug feature coming soon!')\n # Destroy previous display of debug or error messages\n # if present\n if self.debugFrame:\n self.debugFrame.grid_forget()\n\n # Compile if file signature has changed\n cfile.compile_if_needed()\n\n # Now, call the error debug routine if errors are found\n if (ccg2xml.error_count > 0):\n self.debugError(editFrame, cfile)\n else:\n showinfo(title='VisCCG: Success', message='No validation errors!')\n\n def debugError(self, editFrame, cfile):\n self.debugFrame = Frame(editFrame, bg='white', bd=2)\n self.debugFrame.grid(row=3, columnspan=2, sticky=NSEW)\n\n # Create Listbox and scrollbars\n sbar = Scrollbar(self.debugFrame)\n list = Listbox(self.debugFrame, relief=SUNKEN, bg='white', bd=2, yscrollcommand = sbar.set)\n sbar.config(command=list.yview)\n list.pack(fill=BOTH, side=LEFT, expand=YES)\n sbar.pack(fill=Y, side=RIGHT)\n\n # Display each message in the log\n for mesg in ccg2xml.message_log:\n type = mesg[0]\n lineno = mesg[1]\n errwarn = mesg[2]\n\n if lineno:\n dispError = type+' at Line '+str(lineno)+': '+errwarn\n else:\n dispError = type+': '+errwarn\n\n list.insert(END, dispError)\n\n #####################\n # Edit menu commands\n #####################\n\n def onCopy(self): # get text selected by mouse,etc\n if not self.text.tag_ranges(SEL): # save in cross-app clipboard\n showerror('CCG Editor', 'No text selected')\n else:\n text = self.text.get(SEL_FIRST, SEL_LAST)\n self.clipboard_clear()\n self.clipboard_append(text)\n\n def onDelete(self): # delete selected text, no save\n if not self.text.tag_ranges(SEL):\n showerror('CCG Editor', 'No text selected')\n else:\n self.text.delete(SEL_FIRST, SEL_LAST)\n\n def onCut(self):\n if not self.text.tag_ranges(SEL):\n showerror('CCG Editor', 'No text selected')\n else:\n self.onCopy() # save and delete selected text\n self.onDelete()\n\n def onPaste(self):\n try:\n text = self.selection_get(selection='CLIPBOARD')\n except TclError:\n showerror('CCG Editor', 'Nothing to paste')\n return\n self.text.insert(INSERT, text) # add at current insert cursor\n self.text.tag_remove(SEL, '1.0', END)\n self.text.tag_add(SEL, INSERT+'-%dc' % len(text), INSERT)\n self.text.see(INSERT) # select it, so it can be cut\n\n def onSelectAll(self):\n self.text.tag_add(SEL, '1.0', END+'-1c') # select entire text\n self.text.mark_set(INSERT, '1.0') # move insert point to top\n self.text.see(INSERT) # scroll to top\n\n #######################\n # Search menu commands\n #######################\n\n def onChange(self):\n new = Toplevel(self)\n Label(new, text='Find text:').grid(row=0, column=0)\n Label(new, text='Change to:').grid(row=1, column=0)\n self.change1 = Entry(new)\n self.change2 = Entry(new)\n self.change1.grid(row=0, column=1, sticky=EW)\n self.change2.grid(row=1, column=1, sticky=EW)\n Button(new, text='Find',\n command=self.onDoFind).grid(row=0, column=2, sticky=EW)\n Button(new, text='Apply',\n command=self.onDoChange).grid(row=1, column=2, sticky=EW)\n new.columnconfigure(1, weight=1) # expandable entrys\n\n def onDoFind(self):\n self.onFind(self.change1.get()) # Find in change box\n\n def onDoChange(self):\n if self.text.tag_ranges(SEL): # must find first\n self.text.delete(SEL_FIRST, SEL_LAST) # Apply in change\n self.text.insert(INSERT, self.change2.get()) # deletes if empty\n self.text.see(INSERT)\n self.onFind(self.change1.get()) # goto next appear\n self.text.update() # force refresh\n\n ####################################\n # Others, useful outside this class\n ####################################\n\n def isEmpty(self):\n return not self.getAllText()\n\n def getAllText(self):\n return self.text.get('1.0', END+'-1c') # extract text as a string\n\n def setAllText(self, text):\n self.text.delete('1.0', END) # store text string in widget\n self.text.insert(END, text) # or '1.0'\n self.text.mark_set(INSERT, '1.0') # move insert point to top\n self.text.see(INSERT) # scroll to top, insert set\n self.cfile.edit_modified(NO)\n\n def clearAllText(self):\n self.text.delete('1.0', END) # clear text in widget\n\n\nclass CWords(CTab):\n def __init__(self, parent, cfile):\n CTab.__init__(self, parent, cfile, 'Words')\n self.child=None\n self.wordList = None\n self.cfile = cfile\n\n # Called when we switch to this mode using the toolbar at top.\n def reinit(self):\n if self.child:\n self.child.pack_forget()\n\n self.child = Frame(self, background='white')\n self.child.pack(expand=YES, fill=BOTH)\n\n scrollbar = Scrollbar(self.child, orient=VERTICAL)\n self.wordList = Listbox(self.child, yscrollcommand=scrollbar.set)\n self.wordList.grid(row=0, column=0, sticky=N+S+E+W)\n\n scrollbar.config(command= self.wordList.yview)\n scrollbar.grid(row=0, column=1, sticky=N+S)\n\n self.child.grid_rowconfigure(0, weight=1)\n self.child.grid_columnconfigure(0, weight=1)\n\n #If the data hasn't been compiled yet, then do so\n try:\n dummy = ccg2xml.morph_xml\n except:\n self.cfile.compile_if_needed()\n #Adding dummy code for all words\n for x in ccg2xml.morph_xml:\n assert x[0] == 'entry'\n self.wordList.insert (END, ccg2xml.getprop('word', x[1]))\n #print ccg2xml.getprop('word', x[1])\n\nclass CLexicon(CTab):\n class lexicon_vars(object):\n def __init__(self):\n self.show_feat_id = IntVar()\n self.show_feat_id.set(1)\n self.show_feat_struct = IntVar()\n self.show_feat_struct.set(1)\n self.show_full_features = IntVar()\n self.show_full_features.set(0)\n self.show_semantics = IntVar()\n self.show_semantics.set(1)\n\n def __init__(self, parent, cfile):\n CTab.__init__(self, parent, cfile, 'Lexicon')\n self.child = None\n self.cnv = None\n self.mainFrame = None\n\n self.vars = self.lexicon_vars()\n # FIXME? It's a bit awkward that ccg.ply has references to the\n # variables below scattered throughout it. But I'm not sure what\n # a better solution would be.\n self.checkbar = [\n (\"Show feature ID's\", self.vars.show_feat_id),\n (\"Show features\", self.vars.show_feat_struct),\n ('Full-form features', self.vars.show_full_features),\n ('Show semantics', self.vars.show_semantics),\n ]\n\n # Called when we switch to this mode using the toolbar at top.\n def reinit(self):\n self.redraw()\n\n def redraw(self):\n self.cfile.compile_if_needed()\n if self.child:\n self.child.pack_forget()\n if self.mainFrame:\n self.mainFrame.pack_forget()\n\n self.mainFrame = Frame(self, bd=1, bg='white')\n self.mainFrame.pack_propagate(0)\n self.mainFrame.pack(expand=YES, fill=BOTH)\n\n self.mainFrame.grid_rowconfigure(0, weight=1)\n self.mainFrame.grid_columnconfigure(0, weight=1)\n\n xscrollbar = Scrollbar(self.mainFrame, orient=HORIZONTAL)\n xscrollbar.grid(row=1, column=0, sticky=E+W)\n\n yscrollbar = Scrollbar(self.mainFrame)\n yscrollbar.grid(row=0, column=1, sticky=N+S)\n\n self.cnv = Canvas(self.mainFrame, bd=2, xscrollcommand=xscrollbar.set,\n yscrollcommand=yscrollbar.set, width = 847, height=369)\n\n xscrollbar.config(command= self.cnv.xview)\n yscrollbar.config(command= self.cnv.yview)\n\n self.child = Frame(self.cnv, bd=2, relief=SUNKEN, background='white')\n\n self.cnv.create_window(0, 0, anchor='nw', window=self.child)\n\n ccg2xml.draw_parse(self.cfile.curparse.parse, self.cfile, self.child, self.vars, self.cnv, self.mainFrame)\n\n self.child.update_idletasks()\n\n self.cnv.config(scrollregion=self.cnv.bbox(\"all\"))\n self.cnv.grid(row=0, column=0, sticky='NSEW')\n\n\nclass CRules(CTab):\n def __init__(self, parent, cfile):\n CTab.__init__(self, parent, cfile, 'Rules')\n\nclass CFeatures(CTab):\n def __init__(self, parent, cfile):\n CTab.__init__(self, parent, cfile, 'Features')\n self.child=None\n self.checkbar=None\n self.edit=None\n self.text=None\n\n # Called when we switch to this mode using the toolbar at top.\n def reinit(self):\n if self.child:\n self.child.pack_forget()\n\n self.child = Frame(self, background='white', width = 847, height = 369)\n self.child.pack(expand=YES, fill=BOTH)\n butframe = Frame(self.child, cursor='hand2',\n relief=SUNKEN, bd=2)\n butframe.pack(fill=X)\n but1 = Button(butframe, text='Expand All', command=self.expand_all)\n but1.pack(side=LEFT)\n but2 = Button(butframe, text='Contract All', command=self.contract_all)\n but2.pack(side=LEFT)\n # Force editing in the same frame: but a lower view:\n # pass self.child as the parent frame\n self.edit = Button(butframe, text='Edit', command= lambda:self.edit_tree(self.child))\n self.edit.pack(side=RIGHT)\n featframe = Frame(self.child, bd=2, relief=SUNKEN,\n background='white')\n featframe.pack(expand=YES, fill=BOTH)\n self.cfile.compile_if_needed()\n\n # Build the tree\n self.tree={}\n self.root_name = re.sub(r'^(.*)\\.(.*)$', r'\\1', self.cfile.file)\n self.tree[self.root_name]=[]\n for feat in self.cfile.curparse.feature_to_values:\n self.tree[self.root_name] += [str(feat)]\n for feat in self.cfile.curparse.feature_to_values:\n self.tree[feat] = []\n\n for x in self.cfile.curparse.feature_to_values[feat]:\n if x.name not in self.tree:\n self.tree[x.name] = []\n\n for x in self.cfile.curparse.feature_to_values[feat]:\n if x.parents:\n par = x.parents[0]\n self.tree[par.name] += [x.name]\n else:\n self.tree[feat] += [x.name]\n\n # Define the images for opened and closed categories\n shut_icon=PhotoImage(data='R0lGODlhCQAQAJH/AMDAwAAAAGnD/wAAACH5BAEAAAAALAAA'\n 'AAAJABAAQAIdhI8hu2EqXIroyQrb\\nyRf0VG0UxnSZ5jFjulrhaxQ'\n 'AO6olVwAAOw==')\n open_icon=PhotoImage(data='R0lGODlhEAAJAJH/AMDAwAAAAGnD/wAAACH5BAEAAAAALAAA'\n 'AAAQAAkAQAIahI+pyyEPg3KwPrko\\nTqH7/yGUJWxcZTapUQAAO8b'\n 'yUgAAOw==')\n\n # Create the tree\n self.t=Tree.Tree(master=featframe,\n root_id='',\n root_label=self.root_name,\n collapsed_icon=shut_icon,\n expanded_icon=open_icon,\n get_contents_callback=self.get_treedata,\n line_flag=False)\n\n self.t.grid(row=0, column=0, sticky = 'nsew')\n\n featframe.grid_rowconfigure(0, weight=1)\n featframe.grid_columnconfigure(0, weight=1)\n\n sb=Scrollbar(featframe)\n sb.grid(row=0, column=1, sticky='ns')\n self.t.configure(yscrollcommand=sb.set)\n sb.configure(command=self.t.yview)\n\n sb=Scrollbar(featframe, orient=HORIZONTAL)\n sb.grid(row=1, column=0, sticky='ew')\n self.t.configure(xscrollcommand=sb.set)\n sb.configure(command=self.t.xview)\n\n # Expand the whole tree out\n self.expand_tree(self.t.root)\n\n # Returns the nodes rooted at the node passed and adds them to the tree\n def get_treedata(self,node):\n lbl = str(node.get_label())\n children = self.tree[lbl]\n for x in children:\n if self.tree[x]:\n expands=1\n else:\n expands=0\n self.t.add_node(name=x,flag=expands)\n\n # Expand the tree rooted at node recursively\n def expand_tree(self, node):\n node.expand()\n for child in node.children():\n if child.expandable():\n self.expand_tree(child)\n\n def expand_all(self):\n self.expand_tree(self.t.root)\n\n def contract_all(self):\n self.t.root.collapse()\n\n def edit_tree(self, parent):\n editFrame = Frame(parent, bd=1, background='white')\n\n\n self.text = Text(editFrame, padx=5, wrap=None, undo = YES, background='white')\n vbar = Scrollbar(editFrame)\n hbar = Scrollbar(editFrame, orient='horizontal')\n\n self.text.config(yscrollcommand=vbar.set) # call vbar.set on text move\n self.text.config(xscrollcommand=hbar.set)\n vbar.config(command=self.text.yview) # call text.yview on scroll move\n hbar.config(command=self.text.xview) # or hbar['command']=text.xview\n\n # Change the text on the button, and also pass the rest\n # of the arguments so that the grid for the statements can be reset\n self.edit.config(text='Done', command= lambda:self.save_tree(parent))\n\n # Changing the mode of the cfile object here,\n # so that once the user clicks done,\n # the whole object is recompiled and redisplayed\n self.cfile.mode= 'Edit'\n\n vbar.pack(side=RIGHT, fill=Y)\n hbar.pack(side=BOTTOM, fill=X)\n self.text.pack(fill= BOTH, expand= YES)\n\n # Set a mark at the beginning of the text\n self.text.mark_set(\"START\", INSERT)\n self.text.mark_gravity(\"START\", LEFT)\n\n # Push in the rest of the file's contents\n fileData = self.cfile.getAllText()\n self.text.insert(INSERT, fileData)\n\n # Move the insert position to the first occurence of the family name\n # FIXME: this is poor implementation\n # The positioning of the insert cursor should be happening by parsing the\n # CFG production rules, using CSFamily.prod.lineno and endlineno\n self.text.config(takefocus=True)\n idx= self.text.search('feature', \"START\")\n if idx:\n self.text.mark_set(CURRENT, idx)\n self.text.see(CURRENT)\n else:\n showwarning('Warning','Features not located in text')\n\n editFrame.pack(expand=YES, fill=BOTH)\n\n def save_tree(self, parent):\n # We force the text contents of the cfile object to copy over\n # all that is presently in the current text-box\n self.cfile.setAllText(self.text.get(1.0,END))\n self.edit.config(text='Edit', command= lambda:self.edit_tree(parent))\n\n # Recompile whatever was edited and redisplay\n # Note: changes are not saved hereby!!\n self.cfile.compile_if_needed()\n self.cfile.onFeatures()\n\nclass CTestbed(CTab):\n def __init__(self, parent, cfile):\n CTab.__init__(self, parent, cfile, 'Testbed')\n self.child = None\n self.edit = None\n self.text = None\n self.editFrame = None\n self.cnv = None\n self.mainFrame = None\n self.newInsert = None\n\n def makelab(self, text, row, col, **props):\n lab = Label(self.child, text=text, background='white', **props)\n # Make the label grow to fill all space allocated for the column\n lab.grid(row=row, column=col, sticky='NSEW')\n\n # Called when we switch to this mode using the toolbar at top.\n def reinit(self):\n if self.child:\n self.child.pack_forget()\n if self.mainFrame:\n self.mainFrame.pack_forget()\n\n self.mainFrame = Frame(self, bd=1, bg='white')\n self.mainFrame.pack(expand=YES, fill=BOTH)\n\n self.mainFrame.grid_rowconfigure(0, weight=1)\n self.mainFrame.grid_columnconfigure(0, weight=1)\n\n xscrollbar = Scrollbar(self.mainFrame, orient=HORIZONTAL)\n xscrollbar.grid(row=1, column=0, sticky=E+W)\n\n yscrollbar = Scrollbar(self.mainFrame)\n yscrollbar.grid(row=0, column=1, sticky=N+S)\n\n self.cnv= Canvas(self.mainFrame, bd=2, xscrollcommand=xscrollbar.set,\n yscrollcommand=yscrollbar.set, width = 847, height=369)\n\n xscrollbar.config(command=self.cnv.xview)\n yscrollbar.config(command=self.cnv.yview)\n\n self.child = Frame(self.cnv, bd=2, relief=SUNKEN, background='white')\n\n\n self.child.rowconfigure(1, weight=1)\n self.child.columnconfigure(1, weight=1)\n self.child.pack(expand=YES, fill=BOTH)\n\n butnFrame = Frame(self.child, relief=SUNKEN, bd=2)\n butnFrame.grid(row=0, sticky='NSEW', columnspan=2)\n\n self.edit = Button(butnFrame, text='Edit', command= self.edit_testbed)\n self.edit.pack(side=RIGHT)\n self.newInsert = Button(butnFrame, text='New Sentence', command= self.new_sentence)\n self.newInsert.pack(side=RIGHT)\n\n self.cfile.compile_if_needed()\n\n self.makelab(\"Num Parses\", 1, 0, bd=1, relief=SUNKEN, fg=\"#77AA77\", font = (\"Helvetica\", FontScale +12))\n self.makelab(\"Sentence\", 1, 1, bd=1, relief=SUNKEN, fg=\"#77AA77\", font = (\"Helvetica\", FontScale +12))\n\n # Make the column containing the sentences grow to include all\n # extra space\n self.child.columnconfigure(1, weight=1)\n for i in xrange(len(self.cfile.curparse.testbed_statements)):\n x = self.cfile.curparse.testbed_statements[i]\n assert x[0] == 'item'\n x = x[1]\n # Left-justify the text\n numparse = ccg2xml.getprop('numOfParses', x)\n string = ccg2xml.getprop('string', x)\n\n # How many parses of the sentence are produced?\n self.makelab('%s' % numparse, i+2, 0)\n\n # Print the sentence itself\n self.makelab('%s%s' % (numparse == 0 and '*' or '', string),\n i+2, 1, anchor=W)\n\n self.cnv.create_window(0, 0, anchor='nw', window=self.child)\n\n self.child.update_idletasks()\n #self.child.grid(row=0, column=0, sticky=NSEW)\n\n self.cnv.config(scrollregion=self.cnv.bbox(\"all\"))\n self.cnv.grid(row=0, column=0, sticky='NSEW')\n\n\n # Edit the testbed\n def edit_testbed(self):\n self.editFrame = Frame(self.mainFrame, bd=1, background='white')\n #self.editFrame.grid(row=len(self.cfile.curparse.testbed_statements)+3, columnspan=2, sticky='NSEW')\n self.editFrame.grid(row=2, columnspan=2, sticky='NSEW')\n\n self.text = Text(self.editFrame, padx=5, wrap=None, undo = YES, background='white')\n vbar = Scrollbar(self.editFrame)\n hbar = Scrollbar(self.editFrame, orient='horizontal')\n\n self.text.config(yscrollcommand=vbar.set) # call vbar.set on text move\n self.text.config(xscrollcommand=hbar.set)\n vbar.config(command=self.text.yview) # call text.yview on scroll move\n hbar.config(command=self.text.xview) # or hbar['command']=text.xview\n\n # Change the text on the button, and also pass the rest\n # of the arguments so that the grid for the statements can be reset\n self.edit.config(text='Done', command= self.save_testbed)\n\n # Changing the mode of the cfile object here,\n # so that once the user clicks done,\n # the whole object is recompiled and redisplayed\n self.cfile.mode= 'Edit'\n\n vbar.pack(side=RIGHT, fill=Y)\n hbar.pack(side=BOTTOM, fill=X)\n self.text.pack(fill= BOTH, expand= YES)\n\n # Set a mark at the beginning of the text\n self.text.mark_set(\"START\", INSERT)\n self.text.mark_gravity(\"START\", LEFT)\n\n # Push in the rest of the file's contents\n fileData = self.cfile.getAllText()\n self.text.insert(INSERT, fileData)\n\n # Move the insert position to the first occurence of the family name\n # FIXME: this is poor implementation\n # The positioning of the insert cursor should be happening by parsing the\n # CFG production rules, using CSFamily.prod.lineno and endlineno\n self.text.config(takefocus=True)\n idx= self.text.search('testbed', \"START\")\n if idx:\n self.text.mark_set(CURRENT, idx)\n self.text.see(CURRENT)\n else:\n showwarning(title= 'VisCCG: Warning', message='No initial testbed found')\n\n #self.editFrame.pack(expand=YES, fill=BOTH)\n self.child.update_idletasks()\n self.cnv.config(scrollregion=self.cnv.bbox(\"all\"))\n\n # Save the edited text\n def save_testbed(self):\n # We force the text contents of the cfile object to copy over\n # all that is presently in the current text-box\n self.cfile.setAllText(self.text.get(1.0,END))\n self.edit.config(text='Edit', command= self.edit_testbed)\n self.editFrame.pack_forget()\n\n # Recompile whatever was edited and redisplay\n # Note: changes are not saved hereby!!\n self.cfile.compile_if_needed()\n self.cfile.onTestbed()\n\n # Enter a new sentence\n def new_sentence(self):\n master = Tk()\n master.title('VisCCG: New Sentence for the testbed')\n sent = Entry(master, bg='#FFFFFF', width = 100)\n nParses = Entry(master, bg='#FFFFFF', width = 2)\n\n sLabel = Label (master, text = 'Sentence:')\n nLabel = Label (master, text = 'Number of parses:')\n\n sent.focus_set()\n\n b = Button(master, text=\"Add sentence\", width=10, command= lambda:self.editNew(master, sent, nParses))\n c = Button(master, text=\"Cancel\", command= master.destroy)\n\n sent.grid (row=1, column=0, sticky = W)\n nParses.grid (row=1, column=1, sticky= W)\n sLabel.grid (row=0, column=0, sticky=W)\n nLabel.grid (row=0, column=1, sticky = W)\n b.grid (row=2, column = 0)\n c.grid (row=2, column = 1)\n\n # Print from the new sentence\n def editNew(self, master, sent, nParses):\n # Prepare the file's contents for editing\n fileData = self.cfile.getAllText()\n self.text = Text(master)\n\n self.text.mark_set(\"START\", INSERT)\n self.text.mark_gravity(\"START\", LEFT)\n self.text.insert(INSERT, fileData)\n\n testSent = sent.get()\n npSent = nParses.get()\n\n self.text.config(takefocus=True)\n idx= self.text.search('testbed', \"START\")\n if idx:\n self.text.mark_set(\"START\", idx)\n idx = self.text.search('{', \"START\", forwards = True)\n self.text.mark_set(\"START\", idx)\n idx = self.text.search('\\n', \"START\", forwards = True)\n # FIXME: really poor search for locating the right position\n # to insert text here. Needs correction!\n self.text.mark_set(INSERT, idx)\n self.text.mark_gravity(INSERT, RIGHT)\n\n self.text.insert (INSERT, '\\n\\t'+ testSent+ ':\\t'+ npSent+ ';')\n\n else:\n showwarning(title= 'VisCCG: Warning', message='No initial testbed found, creating new')\n self.text.mark_set(INSERT, END)\n self.text.mark_gravity(INSERT, RIGHT)\n\n self.text.insert (INSERT, ' testbed {\\n')\n self.text.insert (INSERT, '\\n\\t'+ testSent+ ':\\t'+ npSent+ ';')\n self.text.insert (INSERT, '}\\n')\n\n\n # Set the original file's data to be this\n fileData= self.text.get(1.0, END)\n self.cfile.setAllText(fileData)\n\n # Destroy the entry window\n master.destroy()\n\n # Update the display\n self.cfile.mode= 'Edit'\n self.cfile.compile_if_needed()\n self.cfile.onTestbed()\n\n# Creates the top-level window and populates the widgets below it.\nclass CFile(object):\n #### NOTE NOTE NOTE! Variables declared like this, in the class itself,\n #### are class variables (not instance variables) until they are\n #### assigned to. If you want pure instance variables, you need to\n #### initialize them inside of __init__().\n\n # Hash table describing modes and the associated class\n modelist = {'Edit':CEdit, 'Lexicon':CLexicon, 'Features':CFeatures,\n 'Words':CWords, 'Testbed':CTestbed, 'Rules':CRules}\n\n startfiledir = '.'\n ftypes = [('All files', '*'), # for file open dialog\n ('Text files', '.txt'), # customize in subclass\n ('Python files', '.py')] # or set in each instance\n\n colors = [{'fg':'black', 'bg':'white'}, # color pick list\n {'fg':'yellow', 'bg':'black'}, # first item is default\n {'fg':'white', 'bg':'blue'}, # tailor me as desired\n {'fg':'black', 'bg':'beige'}, # or do PickBg/Fg chooser\n {'fg':'yellow', 'bg':'purple'},\n {'fg':'black', 'bg':'brown'},\n {'fg':'lightgreen', 'bg':'darkgreen'},\n {'fg':'darkblue', 'bg':'orange'},\n {'fg':'orange', 'bg':'darkblue'}]\n\n fonts = [('courier', 9+FontScale, 'normal'), # platform-neutral fonts\n ('courier', 12+FontScale, 'normal'), # (family, size, style)\n ('courier', 10+FontScale, 'bold'), # or popup a listbox\n ('courier', 10+FontScale, 'italic'), # make bigger on linux\n ('times', 10+FontScale, 'normal'),\n ('helvetica', 10+FontScale, 'normal'),\n ('ariel', 10+FontScale, 'normal'),\n ('system', 10+FontScale, 'normal'),\n ('courier', 20+FontScale, 'normal')]\n\n def __init__(self, file=None):\n self.file = file\n\n self.openDialog = None\n self.saveDialog = None\n self.lastfind = None\n self.current_parse = None\n self.mode = None\n self.last_save_signature = None\n self.last_compile_signature = None\n\n # First top-level window is Tk(); rest are Toplevel()\n global root\n if not root:\n root = Tk()\n self.top = root\n else:\n self.top = Toplevel(root)\n\n ccg2xml.late_init_graphics()\n openfiles[self] = True\n self.top.protocol('WM_DELETE_WINDOW', self.onClose)\n\n # We create an outer frame to hold the toolbar and the main widget.\n # Create all the different kinds of main widget.\n # FIXME: Maybe outer isn't necessary?\n self.outer = Frame(self.top)\n self.outer.pack(expand=YES, fill=BOTH) # make frame stretchable\n self.modes = {}\n for mode in self.modelist:\n self.modes[mode] = self.modelist[mode](self.outer, self)\n self.main = None\n self.toolbar_widget = None\n self.checkbar_widget = None\n #self.switch_to('Edit')\n self.setFileName(None)\n if file:\n self.onFirstOpen(file)\n else:\n # When the user has just opened a new file\n # Need to load template from the src folder\n openccg_home = os.environ['OPENCCG_HOME']\n template = open(openccg_home + '/src/ccg2xml/grammar_template.ccg', 'r').read()\n self.setAllText(template)\n\n # Save the MD5 signature for future comparison\n self.last_save_signature = self.getSignature(self.getAllText())\n self.switch_to('Edit')\n\n def switch_to(self, mode):\n # Switch to a different mode (display, edit, test). Remove the\n # existing main and toolbar widgets, if existing. Redo the menubar\n # and toolbar widgets according to the new mode and then display\n # the new widgets.\n #\n # FIXME: We should probably create the menubar and toolbar widgets\n # only once, and remember them.\n if self.mode != mode:\n if self.main:\n self.main.pack_forget()\n if self.toolbar_widget:\n self.toolbar_widget.pack_forget()\n if self.checkbar_widget:\n self.checkbar_widget.pack_forget()\n self.mode = mode\n self.main = self.modes[mode]\n self.makeMenubar()\n self.makeToolbar(mode)\n self.makeCheckbar()\n #print \"Reinit being called now... \"\n self.main.reinit()\n # Pack the main widget after the toolbar, so it goes below it.\n self.main.pack(side=TOP, expand=YES, fill=BOTH)\n\n # Create the menubar; assumes that self.menubar has been set to the\n # appropriate menubar description. Note that the menubar has to be a\n # child of the top-level window itself rather than any child of it, so\n # that it can be correctly displayed at the top of the window -- or\n # possibly in its decoration (Windows) or at top of screen (Mac).\n #\n # From PP2E guimaker.py.\n def makeMenubar(self):\n menubar = Menu(self.top)\n self.top.config(menu=menubar)\n\n for (name, key, items) in self.main.menubar:\n pulldown = Menu(menubar)\n self.addMenuItems(pulldown, items)\n menubar.add_cascade(label=name, underline=key, menu=pulldown)\n\n if sys.platform[:3] == 'win':\n menubar.add_command(label='Help', command=self.help)\n else:\n pulldown = Menu(menubar) # linux needs real pulldown\n pulldown.add_command(label='About', command=self.help)\n menubar.add_cascade(label='Help', menu=pulldown)\n\n # Add items to a menu or submenu. From PP2E guimaker.py.\n def addMenuItems(self, menu, items):\n for item in items: # scan nested items list\n if item == 'separator': # string: add separator\n menu.add_separator({})\n elif type(item) is list: # list: disabled item list\n for num in item:\n menu.entryconfig(num, state=DISABLED)\n elif type(item[2]) is not list:\n menu.add_command(label = item[0], # command:\n underline = item[1], # add command\n command = item[2]) # cmd=callable\n else:\n pullover = Menu(menu)\n self.addMenuItems(pullover, item[2]) # sublist:\n menu.add_cascade(label = item[0], # make submenu\n underline = item[1], # add cascade\n menu = pullover)\n\n def makeToolbar(self, selected):\n \"\"\"\n make toolbar (of buttons) at top, if any\n expand=no, fill=x so same width on resize\n \"\"\"\n if self.main.toolbar:\n self.toolbar_widget = Frame(self.outer, cursor='hand2',\n relief=SUNKEN, bd=2)\n self.toolbar_widget.pack(side=TOP, fill=X)\n for (name, action, where) in self.main.toolbar:\n but = Button(self.toolbar_widget, text=name,\n command=action)\n if name == selected:\n but.config(relief=SUNKEN)\n but.pack(where)\n\n def makeCheckbar(self):\n \"\"\"\n make check-button bar at top, if any\n expand=no, fill=x so same width on resize\n \"\"\"\n if self.main.checkbar:\n self.checkbar_widget = Frame(self.outer, cursor='hand2',\n relief=SUNKEN, bd=2)\n self.checkbar_widget.pack(side=TOP, fill=X)\n for (name, var) in self.main.checkbar:\n Checkbutton(self.checkbar_widget, text=name,\n variable=var,\n command=self.main.redraw).pack(side=LEFT)\n\n def getAllText(self):\n return self.modes['Edit'].getAllText()\n\n def setAllText(self, text):\n self.modes['Edit'].setAllText(text)\n #self.modes['Display'].setAllText(text)\n\n def _getints(self, string):\n \"\"\"Internal function.\"\"\"\n if string:\n if type(string) is str:\n textwid = self.modes['Edit'].text\n return tuple(map(getint, textwid.tk.splitlist(string)))\n else:\n return string\n\n def edit(self, *args):\n \"\"\"Internal method\n\n This method controls the undo mechanism and\n the modified flag. The exact behavior of the\n command depends on the option argument that\n follows the edit argument. The following forms\n of the command are currently supported:\n\n edit_modified, edit_redo, edit_reset, edit_separator\n and edit_undo\n\n \"\"\"\n textwid = self.modes['Edit'].text\n return self._getints(\n textwid.tk.call((textwid._w, 'edit') + args)) or ()\n\n def edit_modified(self, arg=None):\n \"\"\"Get or Set the modified flag\n\n If arg is not specified, returns the modified\n flag of the widget. The insert, delete, edit undo and\n edit redo commands or the user can set or clear the\n modified flag. If boolean is specified, sets the\n modified flag of the widget to arg.\n \"\"\"\n\n # Added to use md5 functionality to watch for changed data\n if arg is None:\n alltext = self.getAllText()\n if (self.last_save_signature != self.getSignature(alltext)):\n return YES\n return self.edit(\"modified\", arg)\n\n def onInfo(self):\n text = self.getAllText() # added on 5/3/00 in 15 mins\n bytes = len(text) # words uses a simple guess:\n lines = len(string.split(text, '\\n')) # any separated by whitespace\n words = len(string.split(text))\n index = self.main.text.index(INSERT)\n where = tuple(string.split(index, '.'))\n\n showinfo('CCG Editor Information',\n 'Current location:\\n\\n' +\n 'line:\\t%s\\ncolumn:\\t%s\\n\\n' % where +\n 'File text statistics:\\n\\n' +\n 'Modified: %s\\n\\n' % self.edit_modified()+\n 'bytes:\\t%d\\nlines:\\t%d\\nwords:\\t%d\\n' %\n (bytes, lines, words))\n\n #######################\n # Search menu commands\n #######################\n\n def onGoto(self, line=None):\n if not line:\n line = askinteger('CCG Editor', 'Enter line number')\n self.main.text.update()\n self.main.text.focus()\n if line is not None:\n maxindex = self.main.text.index(END+'-1c')\n maxline = atoi(split(maxindex, '.')[0])\n if line > 0 and line <= maxline:\n self.main.text.mark_set(INSERT, '%d.0' % line) # goto line\n self.main.text.tag_remove(SEL, '1.0', END) # delete selects\n self.main.text.tag_add(SEL, INSERT, 'insert + 1l') # select line\n self.main.text.see(INSERT) # scroll to line\n else:\n showerror('CCG Editor', 'Bad line number')\n\n def onFind(self, lastkey=None):\n key = lastkey or askstring('CCG Editor', 'Enter search string')\n self.main.text.update()\n self.main.text.focus()\n self.lastfind = key\n if key:\n where = self.main.text.search(key, INSERT, END) # don't wrap\n if not where:\n showerror('CCG Editor', 'String not found')\n else:\n pastkey = where + '+%dc' % len(key) # index past key\n self.main.text.tag_remove(SEL, '1.0', END) # remove any sel\n self.main.text.tag_add(SEL, where, pastkey) # select key\n self.main.text.mark_set(INSERT, pastkey) # for next find\n self.main.text.see(where) # scroll display\n\n def onRefind(self):\n self.onFind(self.lastfind)\n\n ######################\n # Tools menu commands\n ######################\n\n def onFontList(self):\n self.fonts.append(self.fonts[0]) # pick next font in list\n del self.fonts[0] # resizes the text area\n self.modes['Edit'].text.config(font=self.fonts[0])\n self.modes['Display'].text.config(font=self.fonts[0])\n\n def onColorList(self):\n self.colors.append(self.colors[0]) # pick next color in list\n del self.colors[0] # move current to end\n self.modes['Edit'].text.config(fg=self.colors[0]['fg'], bg=self.colors[0]['bg'])\n self.modes['Display'].text.config(fg=self.colors[0]['fg'], bg=self.colors[0]['bg'])\n\n def onPickFg(self):\n self.pickColor('fg')\n def onPickBg(self):\n self.pickColor('bg')\n def pickColor(self, part):\n (triple, hexstr) = askcolor()\n if hexstr:\n apply(self.modes['Edit'].text.config, (), {part: hexstr})\n apply(self.modes['Display'].text.config, (), {part: hexstr})\n\n# def onRunCode(self, parallelmode=1):\n# \"\"\"\n# run Python code being edited--not an ide, but handy;\n# tries to run in file's dir, not cwd (may be pp2e root);\n# inputs and adds command-line arguments for script files;\n# code's stdin/out/err = editor's start window, if any;\n# but parallelmode uses start to open a dos box for i/o;\n# \"\"\"\n# from PP2E.launchmodes import System, Start, Fork\n# filemode = 0\n# thefile = str(self.getFileName())\n# cmdargs = askstring('CCG Editor', 'Commandline arguments?') or ''\n# if os.path.exists(thefile):\n# filemode = askyesno('CCG Editor', 'Run from file?')\n# if not filemode: # run text string\n# namespace = {'__name__': '__main__'} # run as top-level\n# sys.argv = [thefile] + string.split(cmdargs) # could use threads\n# exec self.getAllText() + '\\n' in namespace # exceptions ignored\n# elif askyesno('CCG Editor', 'Text saved in file?'):\n# mycwd = os.getcwd() # cwd may be root\n# os.chdir(os.path.dirname(thefile) or mycwd) # cd for filenames\n# thecmd = thefile + ' ' + cmdargs\n# if not parallelmode: # run as file\n# System(thecmd, thecmd)() # block editor\n# else:\n# if sys.platform[:3] == 'win': # spawn in parallel\n# Start(thecmd, thecmd)() # or use os.spawnv\n# else:\n# Fork(thecmd, thecmd)() # spawn in parallel\n# os.chdir(mycwd)\n\n #####################\n # File menu commands\n #####################\n\n def getSignature(self, contents):\n return md5.md5(contents).digest()\n\n def my_askopenfilename(self): # objects remember last result dir/file\n if not self.openDialog:\n self.openDialog = Open(initialdir=self.startfiledir,\n filetypes=self.ftypes)\n return self.openDialog.show()\n\n def my_asksaveasfilename(self): # objects remember last result dir/file\n if not self.saveDialog:\n self.saveDialog = SaveAs(initialdir=self.startfiledir,\n filetypes=self.ftypes)\n self.last_save_signature = self.getSignature(self.getAllText())\n return self.saveDialog.show()\n\n def onOpen(self):\n file = self.my_askopenfilename()\n # FIXME! Only create new window if file exists and is readable\n if file:\n CFile(file)\n\n def onFirstOpen(self, file):\n try:\n text = open(file, 'r').read()\n except:\n showerror('CCG Editor', 'Could not open file ' + file)\n else:\n self.setAllText(text)\n self.setFileName(file)\n\n def compile_if_needed(self):\n # Compare the last compiled MD5 signature and present one\n # and compile if needed.\n # To force compilation, set this signature to None\n text = self.getAllText()\n textSign = self.getSignature(text)\n if textSign != self.last_compile_signature:\n # Now compile\n ccg2xml.init_global_state(errors_to_string=True)\n ccg2xml.options.quiet = True\n self.curparse = ccg2xml.parse_string(text)\n self.last_compiled_signature = textSign\n\n def onDisplay(self):\n self.switch_to('Display')\n\n def onEdit(self):\n self.switch_to('Edit')\n\n def onLexicon(self):\n self.switch_to('Lexicon')\n\n def onTestbed(self):\n self.switch_to('Testbed')\n\n def onRules(self):\n self.switch_to('Rules')\n\n def onWords(self):\n self.switch_to('Words')\n\n def onFeatures(self):\n self.switch_to('Features')\n\n def onNew(self):\n CFile()\n\n def getFileName(self):\n return self.currfile\n\n def setFileName(self, name):\n self.currfile = name # for save\n if name:\n title = 'VisCCG Editor: %s' % name\n else:\n title = 'VisCCG Editor'\n self.top.title(title)\n self.top.iconname(title)\n\n def help(self):\n showinfo('Help', 'Sorry, no help for ' + self.__class__.__name__)\n\n # Close this window; if this is the last window, quit\n def onClose(self):\n assert self in openfiles\n if len(openfiles) == 1 or self.top == root:\n self.onQuit()\n # If we got this far, the user refused to quit, so do nothing\n else:\n ccg2xml.debug(\"fooooo\\n\")\n del openfiles[self]\n self.top.destroy()\n\n def onQuit(self):\n modfiles = False\n for f in openfiles:\n if f.edit_modified() == YES:\n modfiles = True\n break\n if not modfiles or askyesno('CCG Editor', 'Files are modified, Really quit?'):\n self.top.quit()\n\n\n\ndef main():\n ccg2xml.parse_arguments(sys.argv[1:])\n ccg2xml.init_global_state_once()\n if ccg2xml.global_args and len(ccg2xml.global_args) > 0:\n # file name:\n fname = ccg2xml.global_args[0]\n else:\n fname = None\n\n CFile(fname)\n mainloop()\n\nif __name__ == '__main__': # when run as a script\n main()\n" }, { "path": "src/ccg2xml/convert-ply.py", "content": "#!/usr/bin/python\n\nimport sys\nimport re\nimport optparse\nimport os.path\nimport time\n\n# Author: Ben Wing \n# Date: April 2006\n\n#############################################################################\n# #\n# convert-ply.py #\n# #\n# Convert a .ply file into a .py file which can be run to generate a #\n# compiler for a language and use it to parse a specified file. This #\n# program is something like a compiler-compiler-compiler -- it uses a #\n# script to generate a compiler-compiler script, which in turn is used #\n# to generate a compiler, which in turn processes a program in some #\n# arbitrary syntax and does whatever it wants with it. Very meta!!! #\n# #\n#############################################################################\n\n# The format of a .ply file is that of a .py file with YACC-like directives\n# interspersed. %y on a line by itself switches to YACC mode, and %p\n# switches back to Python mode. In YACC mode, lines should look like this:\n\n# lhs: rhs1 rhs2 ...: python code # first alternative\n# : rhs3 rhs4 ...: more python code # second alternative\n# : rhs5 rhs6 # alternative with default code of $$ = $1\n# : rhs7 rhs8 ...:\n# # If no code follows the colon on the same line, the code consists\n# # of all following indented lines.\n# python code\n# python code\n# ...\n#\n# This represents a context-free rule where LHS expands to one or more\n# rules (e.g. RHS1 RHS2 ... or RHS3 RHS4 ... or RHS5 RHS6 ... etc.).\n# Associated with each rule is some code, which will be invoked when the\n# parser finds an appropriately matching right-hand side somewhere in\n# the stream of tokens being parsed and proceeds to \"reduce\" the relevant\n# tokens into the single left-hand side token LHS. Associated with each\n# token is a value. For terminal tokens, the value is the string in the\n# text that produced this token. For non-terminal tokens, the value is\n# determined by the code associated with the rule that produced this token\n# (i.e. one of the rules with this token as its LHS). The value of a\n# non-terminal token is set by assigning to $$; values of RHS tokens are\n# referenced using $1, $2, etc., where $1 is the first RHS token, $2\n# is the second, etc. Hence, the default code of $$ = $1 assigns the\n# value of the first RHS token to the LHS token. The code can also\n# reference $@, which is a structure encapsulating all RHS values; this\n# is mostly useful for getting at $@.lineno, a function referring to the\n# starting line number of a particular token (especially $@.lineno(0), the\n# starting line number of the set of RHS tokens).\n\n# WARNING!!! Currently, PLY has a serious bug in its handling of empty\n# RHS rules; often it reports a syntax error in place of properly reducing\n# an empty RHS rule. If this happens, you must rewrite the appropriate\n# rules without the use of an empty RHS rule. (As of yet, I'm not sure\n# exactly what the circumstances are that trigger this bug.)\n#\n# Note that this bug exists as of PLY 1.6, which is what we are currently\n# using. It's quite possible that later versions of PLY (especially the\n# new PLY 2.x series) fix the bug.\n\n###########################################################################\n#\n# Command-line options and usage\n#\n###########################################################################\n\nusage = \"\"\"%prog [OPTIONS] FILE ...\n\nConvert from .ply format to a .py file, for lex/yacc.\n\"\"\"\n\nparser = optparse.OptionParser(usage=usage)\nparser.add_option(\"-o\", \"--outfile\",\n default=None,\n help=\"\"\"Specify the output file.\nDefault is y.INFILE.py, where INFILE is the source file's name minus any\n.ply extension.\"\"\",\n metavar=\"FILE\")\n\n(options, args) = parser.parse_args()\n\ndef syntax_error(err, line):\n global errors\n errors += 1\n if errors > maxerr:\n raise SyntaxError(\"Too many errors (more than %s) when compiling\" %\n maxerr)\n sys.stderr.write(\"%s in file %s, line %d: %s\\n\" %\n (err, current_file, current_lineno, line))\n \n\nwordrange = r'\\-a-zA-Z0-9_%'\noperrange = r'\\+\\*\\|\\?'\nwordre = '[%s]+' % wordrange\n\ndef make_name_python_safe(name):\n return re.sub('[^A-Za-z0-9_]', '_', name)\n\n# Replace dollar signs in CODE to point to the actual array of RHS values.\n# If RENUMBER_AT is given, however, convert $RENUMBER_AT to None, and\n# subtract one from all $ references above this value.\n\ndef replace_dollar_signs(code, renumber_at=None):\n newcode = \"\"\n prevright = 0\n for match in re.finditer(r\"\"\"('''([^\\\\\\n]|\\\\(.|\\n))*?'''|\n \\\"\\\"\\\"([^\\\\\\n]|\\\\(.|\\n))*?\\\"\\\"\\\"|\n '([^\\\\\\n]|\\\\(.|\\n))*?'|\n \\\"([^\\\\\\n]|\\\\(.|\\n))*?\\\"|\n [#][^\\n]*\\n?|\n ([^\\\\'\\\"#]|\\\\(.|\\n))*)\"\"\", code, re.VERBOSE):\n errored = 0\n if prevright != match.start(0):\n syntax_error(\"Apparent syntax error in code at position %d\"\n % prevright, code)\n errored = 1\n newcode += code[prevright:match.start(0)]\n prevright = match.end(0)\n matchstr = match.group(0)\n if not matchstr:\n if match.start(0) == len(code):\n break\n elif not errored:\n errored = 1\n syntax_error(\"Apparent syntax error in code at position %d\"\n % match.start(0), code)\n elif matchstr[0] in '\\'\"#':\n # A comment or literal; don't substitute in it\n pass\n elif renumber_at:\n # Renumber\n def replace_dollar_def(match):\n str = match.group()\n ref = int(str[1:])\n if ref == renumber_at:\n return '[]'\n elif ref > renumber_at:\n return '$%d' % (ref - 1)\n else:\n return str\n matchstr = re.sub(r'\\$([0-9]+)', replace_dollar_def, matchstr)\n else:\n # Apply substitutions\n matchstr = re.sub(r'\\$@', 'p', matchstr)\n matchstr = re.sub(r'\\$\\$', 'p[0]', matchstr)\n matchstr = re.sub(r'\\$([0-9]+)', r'p[\\1]', matchstr)\n\n newcode += matchstr\n return newcode\n\n# Output a single CFG rule\ndef output_python_cfg_rule(fil, lhs, rhs, code):\n global unique_no\n\n # Look for occurrences of foo* or foo+; handle all of them by\n # adding appropriate list rules.\n newrhs = \"\"\n prevright = 0\n for match in re.finditer(r'(%s)\\s*([+*])' % wordre, rhs):\n matchstr = match.group(1)\n newrhs += rhs[prevright:match.start(0)]\n prevright = match.end(0)\n unique_no += 1\n matchlhs = '%s_%s_list_%s' % (lhs, matchstr, unique_no)\n newrhs += matchlhs\n if match.group(2) == '+':\n output_python_cfg_rule(fil, matchlhs, matchstr, \" $$ = [$1]\\n\")\n else:\n output_python_cfg_rule(fil, matchlhs, \" \", \" $$ = []\\n\")\n output_python_cfg_rule(fil, matchlhs, '%s %s' % (matchlhs, matchstr),\n \" $$ = $1 + [$2]\\n\")\n if prevright:\n newrhs += rhs[prevright:]\n rhs = newrhs\n\n # Look for occurrences of foo?; handle by splitting into two rules\n # (It should be possible to handle by using empty rules, but this\n # is currently broken in PLY and more-or-less randomly doesn't work,\n # with the empty rule not being recognized and a syntax error\n # resulting)\n match = re.search(r'(%s)\\s*[?]' % wordre, rhs)\n if match:\n matchtoken = match.group(1) # matched token, w/o following '?'\n leftrhs = rhs[0:match.start(0)] # text before match\n rightrhs = rhs[match.end(0):] # text after match\n # Output the \"it's there\" alternative\n output_python_cfg_rule(fil, lhs, leftrhs + matchtoken + rightrhs, code)\n # compute token ID, as would be referenced by a $# reference, based\n # on the text before the match. first delete everything before a\n # '|' (alternatives) then count the number of words.\n tokennum = len(re.sub('.*\\|', '', leftrhs).split()) + 1\n code = replace_dollar_signs(code, tokennum)\n # Output the \"it's not there\" alternative, with the dollar references\n # renumbered. We should *not* attempt any tail-recursion elimination\n # here, in case there are further foo? occurrences later in the file.\n output_python_cfg_rule(fil, lhs, leftrhs + rightrhs, code)\n else:\n unique_no += 1\n print >> fil, \"def p_%s_%d(p):\" % (make_name_python_safe(lhs),\n unique_no)\n rhs = rhs.strip()\n rhs = re.sub(r'\\s*\\|\\s*', r'\\n | ', rhs)\n rhs = re.sub(r'\\n\\s*\\n', '\\n', rhs)\n if rhs.find('\\n') >= 0:\n print >> fil, \" '''%s : %s'''\" % (lhs, rhs)\n else:\n print >> fil, \" '%s : %s'\" % (lhs, rhs)\n code = replace_dollar_signs(code)\n print >> fil, code\n\ndef output_default_python_cfg_rule(fil, lhs, rhs):\n output_python_cfg_rule(fil, lhs, rhs, \" $$ = $1\\n\")\n\ndef finish_any_cfg(fil):\n global curlhs, currhs, yacc_python_mode, yacc_python_code\n if currhs:\n # A RHS not yet finished; finish it\n if yacc_python_code:\n output_python_cfg_rule(fil, curlhs, currhs, yacc_python_code)\n else:\n output_default_python_cfg_rule(fil, curlhs, currhs)\n clear_rule_context()\n\ndef clear_rule_context():\n global curlhs, currhs, yacc_python_mode, yacc_python_code\n curlhs = None\n currhs = None\n yacc_python_mode = False\n yacc_python_code = None\n\n## Process file(s)\n\nargs = args or ['-']\nfor arg in args:\n global current_file\n current_file = arg\n # Open input and output files\n if arg == '-':\n fil = sys.stdin\n else:\n fil = open(arg)\n if options.outfile:\n outarg = options.outfile\n else:\n if arg == '-':\n outarg = 'y.ccg.py'\n else:\n fname = arg\n if fname.endswith('.ply'):\n fname = fname[0:-4]\n (fdir, ffile) = os.path.split(fname)\n outarg = os.path.join(fdir, 'y.%s.py' % ffile)\n outfil = open(outarg, 'w')\n\n # Initialize state\n errors = 0\n maxerr = 5\n unique_no = 0\n clear_rule_context()\n mode = 'python'\n contline = None\n\n print >> outfil, \"\"\"#!/usr/bin/python\n\n################## NOTE NOTE NOTE ##################\n#\n# This file (%s) was automatically generated from %s.\n# Generated by %s at %s.\n#\n# DO NOT MODIFY THIS FILE DIRECTLY. YOUR CHANGES WILL BE LOST.\n# Instead, modify the file `%s' that generated this file, and then\n# rerun `%s%s %s'.\n#\n################## NOTE NOTE NOTE ##################\n\"\"\" % (outarg, current_file, sys.argv[0], time.asctime(), current_file,\n sys.argv[0],\n options.outfile and \" -o %s\" % options.outfile or \"\",\n current_file)\n\n global current_lineno\n current_lineno = 0\n for line in fil:\n current_lineno += 1\n line = line.rstrip(\"\\r\\n\")\n if contline:\n line = contline + line\n contline = None\n if line == '%y':\n mode = 'yacc'\n elif line == '%p':\n mode = 'python'\n finish_any_cfg(outfil)\n elif line == '%l':\n mode = 'lex'\n else:\n if mode == 'python':\n print >> outfil, line\n else:\n if yacc_python_mode:\n if re.match(r'\\S', line):\n yacc_python_mode = False\n else:\n yacc_python_code += line + '\\n'\n continue\n if re.match(r'\\s*#.*$', line):\n print >> outfil, line\n continue\n elif line and line[-1] == '\\\\':\n contline = line[0:-1]\n continue\n elif re.match(r'\\s*$', line):\n print >> outfil, line\n continue\n # Eliminate comments, but conservatively, to avoid any\n # possibility of removing comments inside of quotes (which\n # should occur only in Python code, anyway, in which case\n # it doesn't really matter)\n line = re.sub(r'''^([^\\'\\\"#]*)#.*$''', '\\1', line)\n if mode == 'yacc':\n match = re.match(r'(%s)\\s*(:.*)$' % wordre, line)\n if match:\n # We are starting a new rule\n finish_any_cfg(outfil)\n curlhs = match.group(1)\n line = ' ' + match.group(2)\n match = re.match(r'(\\s*)(:)?([^:]*)(:.*)?$', line)\n if not match or not match.group(1):\n syntax_error(\"Unrecognized rule beginning\", line)\n if re.match(r'^.*[^%s\\s%s].*$' % (operrange, wordrange),\n match.group(3)):\n syntax_error(\"Illegal characters in RHS\", line)\n if not match.group(2):\n # We are continuing an RHS\n if currhs == None:\n syntax_error(\"Invalid RHS continuation\", line)\n else:\n currhs += match.expand(r' \\3')\n else:\n # We are starting an RHS\n if curlhs == None:\n syntax_error(\"Invalid RHS without LHS\", line)\n else:\n if currhs:\n output_default_python_cfg_rule(outfil, curlhs,\n currhs)\n currhs = match.group(3)\n if match.group(4):\n # strip colon, spaces\n code = match.group(4)[1:].strip()\n if not code:\n # Start eating the rest of the code until new rule\n yacc_python_mode = True\n yacc_python_code = \"\"\n else:\n code = ' ' + code + '\\n'\n output_python_cfg_rule(outfil, curlhs, currhs,\n code)\n currhs = None\n finish_any_cfg(outfil)\n fil.close()\n outfil.close()\n" }, { "path": "src/ccg2xml/grammar_template.ccg", "content": "############ [name of the grammar] ####################\n#\n## [a brief description of your grammar]\n## [your name, Month Year]\n#\n#\n# For help on how to create grammars with VisCCG, see the tutorial:\n# http://comp.ling.utexas.edu/wiki/doku.php/openccg/ccggui_tut\n\n####################### Features #######################\n\nfeature {\n}\n\n######################### Words #########################\n\n\n######################### Rules #########################\n\n\n################## Lexicon/Categories ####################\n\n\n####################### Testbed #########################\n\ntestbed {\n}\n" }, { "path": "src/ccg2xml/lex.py", "content": "#-----------------------------------------------------------------------------\n# ply: lex.py\n#\n# Author: David M. Beazley (dave@dabeaz.com)\n#\n# Copyright (C) 2001-2005, David M. Beazley\n#\n# $Header: /cvsroot/openccg/openccg/src/ccg2xml/lex.py,v 1.1 2006/09/30 08:11:29 benwing Exp $\n#\n# This library is free software; you can redistribute it and/or\n# modify it under the terms of the GNU Lesser General Public\n# License as published by the Free Software Foundation; either\n# version 2.1 of the License, or (at your option) any later version.\n#\n# This library is distributed in the hope that it will be useful,\n# but WITHOUT ANY WARRANTY; without even the implied warranty of\n# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU\n# Lesser General Public License for more details.\n#\n# You should have received a copy of the GNU Lesser General Public\n# License along with this library; if not, write to the Free Software\n# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA\n#\n# See the file COPYING for a complete copy of the LGPL.\n#\n#\n# This module automatically constructs a lexical analysis module from regular\n# expression rules defined in a user-defined module. The idea is essentially the same\n# as that used in John Aycock's Spark framework, but the implementation works\n# at the module level rather than requiring the use of classes.\n#\n# This module tries to provide an interface that is closely modeled after\n# the traditional lex interface in Unix. It also differs from Spark\n# in that:\n#\n# - It provides more extensive error checking and reporting if\n# the user supplies a set of regular expressions that can't\n# be compiled or if there is any other kind of a problem in\n# the specification.\n#\n# - The interface is geared towards LALR(1) and LR(1) parser\n# generators. That is tokens are generated one at a time\n# rather than being generated in advanced all in one step.\n#\n# There are a few limitations of this module\n#\n# - The module interface makes it somewhat awkward to support more\n# than one lexer at a time. Although somewhat inelegant from a\n# design perspective, this is rarely a practical concern for\n# most compiler projects.\n#\n# - The lexer requires that the entire input text be read into\n# a string before scanning. I suppose that most machines have\n# enough memory to make this a minor issues, but it makes\n# the lexer somewhat difficult to use in interactive sessions\n# or with streaming data.\n#\n#-----------------------------------------------------------------------------\n\nr\"\"\"\nlex.py\n\nThis module builds lex-like scanners based on regular expression rules.\nTo use the module, simply write a collection of regular expression rules\nand actions like this:\n\n# lexer.py\nimport lex\n\n# Define a list of valid tokens\ntokens = (\n 'IDENTIFIER', 'NUMBER', 'PLUS', 'MINUS'\n )\n\n# Define tokens as functions\ndef t_IDENTIFIER(t):\n r' ([a-zA-Z_](\\w|_)* '\n return t\n\ndef t_NUMBER(t):\n r' \\d+ '\n return t\n\n# Some simple tokens with no actions\nt_PLUS = r'\\+'\nt_MINUS = r'-'\n\n# Initialize the lexer\nlex.lex()\n\nThe tokens list is required and contains a complete list of all valid\ntoken types that the lexer is allowed to produce. Token types are\nrestricted to be valid identifiers. This means that 'MINUS' is a valid\ntoken type whereas '-' is not.\n\nRules are defined by writing a function with a name of the form\nt_rulename. Each rule must accept a single argument which is\na token object generated by the lexer. This token has the following\nattributes:\n\n t.type = type string of the token. This is initially set to the\n name of the rule without the leading t_\n t.value = The value of the lexeme.\n t.lineno = The value of the line number where the token was encountered\n\nFor example, the t_NUMBER() rule above might be called with the following:\n\n t.type = 'NUMBER'\n t.value = '42'\n t.lineno = 3\n\nEach rule returns the token object it would like to supply to the\nparser. In most cases, the token t is returned with few, if any\nmodifications. To discard a token for things like whitespace or\ncomments, simply return nothing. For instance:\n\ndef t_whitespace(t):\n r' \\s+ '\n pass\n\nFor faster lexing, you can also define this in terms of the ignore set like this:\n\nt_ignore = ' \\t'\n\nThe characters in this string are ignored by the lexer. Use of this feature can speed\nup parsing significantly since scanning will immediately proceed to the next token.\n\nlex requires that the token returned by each rule has an attribute\nt.type. Other than this, rules are free to return any kind of token\nobject that they wish and may construct a new type of token object\nfrom the attributes of t (provided the new object has the required\ntype attribute).\n\nIf illegal characters are encountered, the scanner executes the\nfunction t_error(t) where t is a token representing the rest of the\nstring that hasn't been matched. If this function isn't defined, a\nLexError exception is raised. The .text attribute of this exception\nobject contains the part of the string that wasn't matched.\n\nThe t.skip(n) method can be used to skip ahead n characters in the\ninput stream. This is usually only used in the error handling rule.\nFor instance, the following rule would print an error message and\ncontinue:\n\ndef t_error(t):\n print \"Illegal character in input %s\" % t.value[0]\n t.skip(1)\n\nOf course, a nice scanner might wish to skip more than one character\nif the input looks very corrupted.\n\nThe lex module defines a t.lineno attribute on each token that can be used\nto track the current line number in the input. The value of this\nvariable is not modified by lex so it is up to your lexer module\nto correctly update its value depending on the lexical properties\nof the input language. To do this, you might write rules such as\nthe following:\n\ndef t_newline(t):\n r' \\n+ '\n t.lineno += t.value.count(\"\\n\")\n\nTo initialize your lexer so that it can be used, simply call the lex.lex()\nfunction in your rule file. If there are any errors in your\nspecification, warning messages or an exception will be generated to\nalert you to the problem.\n\n(dave: this needs to be rewritten)\nTo use the newly constructed lexer from another module, simply do\nthis:\n\n import lex\n import lexer\n plex.input(\"position = initial + rate*60\")\n\n while 1:\n token = plex.token() # Get a token\n if not token: break # No more tokens\n ... do whatever ...\n\nAssuming that the module 'lexer' has initialized plex as shown\nabove, parsing modules can safely import 'plex' without having\nto import the rule file or any additional imformation about the\nscanner you have defined.\n\"\"\"\n\n# -----------------------------------------------------------------------------\n\n\n__version__ = \"1.6\"\n\nimport re, types, sys, copy\n\n# Exception thrown when invalid token encountered and no default\nclass LexError(Exception):\n def __init__(self,message,s):\n self.args = (message,)\n self.text = s\n\n# Token class\nclass LexToken:\n def __str__(self):\n return \"LexToken(%s,%r,%d)\" % (self.type,self.value,self.lineno)\n def __repr__(self):\n return str(self)\n def skip(self,n):\n try:\n self._skipn += n\n except AttributeError:\n self._skipn = n\n\n# -----------------------------------------------------------------------------\n# Lexer class\n#\n# input() - Store a new string in the lexer\n# token() - Get the next token\n# -----------------------------------------------------------------------------\n\nclass Lexer:\n def __init__(self):\n self.lexre = None # Master regular expression\n self.lexdata = None # Actual input data (as a string)\n self.lexpos = 0 # Current position in input text\n self.lexlen = 0 # Length of the input text\n self.lexindexfunc = [ ] # Reverse mapping of groups to functions and types\n self.lexerrorf = None # Error rule (if any)\n self.lextokens = None # List of valid tokens\n self.lexignore = None # Ignored characters\n self.lineno = 1 # Current line number\n self.debug = 0 # Debugging mode\n self.optimize = 0 # Optimized mode\n self.token = self.errtoken\n\n def __copy__(self):\n c = Lexer()\n c.lexre = self.lexre\n c.lexdata = self.lexdata\n c.lexpos = self.lexpos\n c.lexlen = self.lexlen\n c.lexindexfunc = self.lexindexfunc\n c.lexerrorf = self.lexerrorf\n c.lextokens = self.lextokens\n c.lexignore = self.lexignore\n c.debug = self.debug\n c.lineno = self.lineno\n c.optimize = self.optimize\n c.token = c.realtoken\n return c\n\n # ------------------------------------------------------------\n # input() - Push a new string into the lexer\n # ------------------------------------------------------------\n def input(self,s):\n if not isinstance(s, (unicode, types.StringType)):\n raise ValueError, \"Expected a string\"\n self.lexdata = s\n self.lexpos = 0\n self.lexlen = len(s)\n self.token = self.realtoken\n\n # Change the token routine to point to realtoken()\n global token\n if token == self.errtoken:\n token = self.token\n\n # ------------------------------------------------------------\n # errtoken() - Return error if token is called with no data\n # ------------------------------------------------------------\n def errtoken(self):\n raise RuntimeError, \"No input string given with input()\"\n\n # ------------------------------------------------------------\n # token() - Return the next token from the Lexer\n #\n # Note: This function has been carefully implemented to be as fast\n # as possible. Don't make changes unless you really know what\n # you are doing\n # ------------------------------------------------------------\n def realtoken(self):\n # Make local copies of frequently referenced attributes\n lexpos = self.lexpos\n lexlen = self.lexlen\n lexignore = self.lexignore\n lexdata = self.lexdata\n\n while lexpos < lexlen:\n # This code provides some short-circuit code for whitespace, tabs, and other ignored characters\n if lexdata[lexpos] in lexignore:\n lexpos += 1\n continue\n\n # Look for a regular expression match\n m = self.lexre.match(lexdata,lexpos)\n if m:\n i = m.lastindex\n lexpos = m.end()\n tok = LexToken()\n tok.value = m.group()\n tok.lineno = self.lineno\n tok.lexer = self\n func,tok.type = self.lexindexfunc[i]\n if not func:\n self.lexpos = lexpos\n return tok\n\n # If token is processed by a function, call it\n self.lexpos = lexpos\n newtok = func(tok)\n self.lineno = tok.lineno # Update line number\n\n # Every function must return a token, if nothing, we just move to next token\n if not newtok: continue\n\n # Verify type of the token. If not in the token map, raise an error\n if not self.optimize:\n if not self.lextokens.has_key(newtok.type):\n raise LexError, (\"%s:%d: Rule '%s' returned an unknown token type '%s'\" % (\n func.func_code.co_filename, func.func_code.co_firstlineno,\n func.__name__, newtok.type),lexdata[lexpos:])\n\n return newtok\n\n # No match. Call t_error() if defined.\n if self.lexerrorf:\n tok = LexToken()\n tok.value = self.lexdata[lexpos:]\n tok.lineno = self.lineno\n tok.type = \"error\"\n tok.lexer = self\n oldpos = lexpos\n newtok = self.lexerrorf(tok)\n lexpos += getattr(tok,\"_skipn\",0)\n if oldpos == lexpos:\n # Error method didn't change text position at all. This is an error.\n self.lexpos = lexpos\n raise LexError, (\"Scanning error. Illegal character '%s'\" % (lexdata[lexpos]), lexdata[lexpos:])\n if not newtok: continue\n self.lexpos = lexpos\n return newtok\n\n self.lexpos = lexpos\n raise LexError, (\"No match found\", lexdata[lexpos:])\n\n # No more input data\n self.lexpos = lexpos + 1\n return None\n\n\n# -----------------------------------------------------------------------------\n# validate_file()\n#\n# This checks to see if there are duplicated t_rulename() functions or strings\n# in the parser input file. This is done using a simple regular expression\n# match on each line in the filename.\n# -----------------------------------------------------------------------------\n\ndef validate_file(filename):\n import os.path\n base,ext = os.path.splitext(filename)\n if ext != '.py': return 1 # No idea what the file is. Return OK\n\n try:\n f = open(filename)\n lines = f.readlines()\n f.close()\n except IOError:\n return 1 # Oh well\n\n fre = re.compile(r'\\s*def\\s+(t_[a-zA-Z_0-9]*)\\(')\n sre = re.compile(r'\\s*(t_[a-zA-Z_0-9]*)\\s*=')\n counthash = { }\n linen = 1\n noerror = 1\n for l in lines:\n m = fre.match(l)\n if not m:\n m = sre.match(l)\n if m:\n name = m.group(1)\n prev = counthash.get(name)\n if not prev:\n counthash[name] = linen\n else:\n print \"%s:%d: Rule %s redefined. Previously defined on line %d\" % (filename,linen,name,prev)\n noerror = 0\n linen += 1\n return noerror\n\n# -----------------------------------------------------------------------------\n# _read_lextab(module)\n#\n# Reads lexer table from a lextab file instead of using introspection.\n# -----------------------------------------------------------------------------\n\ndef _read_lextab(lexer, fdict, module):\n exec \"import %s as lextab\" % module\n lexer.lexre = re.compile(lextab._lexre, re.VERBOSE)\n lexer.lexindexfunc = lextab._lextab\n for i in range(len(lextab._lextab)):\n t = lexer.lexindexfunc[i]\n if t:\n if t[0]:\n lexer.lexindexfunc[i] = (fdict[t[0]],t[1])\n lexer.lextokens = lextab._lextokens\n lexer.lexignore = lextab._lexignore\n if lextab._lexerrorf:\n lexer.lexerrorf = fdict[lextab._lexerrorf]\n\n# -----------------------------------------------------------------------------\n# lex(module)\n#\n# Build all of the regular expression rules from definitions in the supplied module\n# -----------------------------------------------------------------------------\ndef lex(module=None,debug=0,optimize=0,lextab=\"lextab\"):\n ldict = None\n regex = \"\"\n error = 0\n files = { }\n lexer = Lexer()\n lexer.debug = debug\n lexer.optimize = optimize\n global token,input\n\n if module:\n # User supplied a module object.\n if isinstance(module, types.ModuleType):\n ldict = module.__dict__\n elif isinstance(module, types.InstanceType):\n _items = [(k,getattr(module,k)) for k in dir(module)]\n ldict = { }\n for (i,v) in _items:\n ldict[i] = v\n else:\n raise ValueError,\"Expected a module or instance\"\n\n else:\n # No module given. We might be able to get information from the caller.\n try:\n raise RuntimeError\n except RuntimeError:\n e,b,t = sys.exc_info()\n f = t.tb_frame\n f = f.f_back # Walk out to our calling function\n ldict = f.f_globals # Grab its globals dictionary\n\n if optimize and lextab:\n try:\n _read_lextab(lexer,ldict, lextab)\n if not lexer.lexignore: lexer.lexignore = \"\"\n token = lexer.token\n input = lexer.input\n return lexer\n\n except ImportError:\n pass\n\n # Get the tokens map\n if (module and isinstance(module,types.InstanceType)):\n tokens = getattr(module,\"tokens\",None)\n else:\n try:\n tokens = ldict[\"tokens\"]\n except KeyError:\n tokens = None\n\n if not tokens:\n raise SyntaxError,\"lex: module does not define 'tokens'\"\n if not (isinstance(tokens,types.ListType) or isinstance(tokens,types.TupleType)):\n raise SyntaxError,\"lex: tokens must be a list or tuple.\"\n\n # Build a dictionary of valid token names\n lexer.lextokens = { }\n if not optimize:\n\n # Utility function for verifying tokens\n def is_identifier(s):\n for c in s:\n if not (c.isalnum() or c == '_'): return 0\n return 1\n\n for n in tokens:\n if not is_identifier(n):\n print \"lex: Bad token name '%s'\" % n\n error = 1\n if lexer.lextokens.has_key(n):\n print \"lex: Warning. Token '%s' multiply defined.\" % n\n lexer.lextokens[n] = None\n else:\n for n in tokens: lexer.lextokens[n] = None\n\n\n if debug:\n print \"lex: tokens = '%s'\" % lexer.lextokens.keys()\n\n # Get a list of symbols with the t_ prefix\n tsymbols = [f for f in ldict.keys() if f[:2] == 't_']\n\n # Now build up a list of functions and a list of strings\n fsymbols = [ ]\n ssymbols = [ ]\n for f in tsymbols:\n if callable(ldict[f]):\n fsymbols.append(ldict[f])\n elif isinstance(ldict[f], types.StringType):\n ssymbols.append((f,ldict[f]))\n else:\n print \"lex: %s not defined as a function or string\" % f\n error = 1\n\n # Sort the functions by line number\n fsymbols.sort(lambda x,y: cmp(x.func_code.co_firstlineno,y.func_code.co_firstlineno))\n\n # Sort the strings by regular expression length\n ssymbols.sort(lambda x,y: (len(x[1]) < len(y[1])) - (len(x[1]) > len(y[1])))\n\n # Check for non-empty symbols\n if len(fsymbols) == 0 and len(ssymbols) == 0:\n raise SyntaxError,\"lex: no rules of the form t_rulename are defined.\"\n\n # Add all of the rules defined with actions first\n for f in fsymbols:\n\n line = f.func_code.co_firstlineno\n file = f.func_code.co_filename\n files[file] = None\n\n ismethod = isinstance(f, types.MethodType)\n\n if not optimize:\n nargs = f.func_code.co_argcount\n if ismethod:\n reqargs = 2\n else:\n reqargs = 1\n if nargs > reqargs:\n print \"%s:%d: Rule '%s' has too many arguments.\" % (file,line,f.__name__)\n error = 1\n continue\n\n if nargs < reqargs:\n print \"%s:%d: Rule '%s' requires an argument.\" % (file,line,f.__name__)\n error = 1\n continue\n\n if f.__name__ == 't_ignore':\n print \"%s:%d: Rule '%s' must be defined as a string.\" % (file,line,f.__name__)\n error = 1\n continue\n\n if f.__name__ == 't_error':\n lexer.lexerrorf = f\n continue\n\n if f.__doc__:\n if not optimize:\n try:\n c = re.compile(f.__doc__, re.VERBOSE)\n except re.error,e:\n print \"%s:%d: Invalid regular expression for rule '%s'. %s\" % (file,line,f.__name__,e)\n error = 1\n continue\n\n if debug:\n print \"lex: Adding rule %s -> '%s'\" % (f.__name__,f.__doc__)\n\n # Okay. The regular expression seemed okay. Let's append it to the master regular\n # expression we're building\n\n if (regex): regex += \"|\"\n regex += \"(?P<%s>%s)\" % (f.__name__,f.__doc__)\n else:\n print \"%s:%d: No regular expression defined for rule '%s'\" % (file,line,f.__name__)\n\n # Now add all of the simple rules\n for name,r in ssymbols:\n\n if name == 't_ignore':\n lexer.lexignore = r\n continue\n\n if not optimize:\n if name == 't_error':\n raise SyntaxError,\"lex: Rule 't_error' must be defined as a function\"\n error = 1\n continue\n\n if not lexer.lextokens.has_key(name[2:]):\n print \"lex: Rule '%s' defined for an unspecified token %s.\" % (name,name[2:])\n error = 1\n continue\n try:\n c = re.compile(r,re.VERBOSE)\n except re.error,e:\n print \"lex: Invalid regular expression for rule '%s'. %s\" % (name,e)\n error = 1\n continue\n if debug:\n print \"lex: Adding rule %s -> '%s'\" % (name,r)\n\n if regex: regex += \"|\"\n regex += \"(?P<%s>%s)\" % (name,r)\n\n if not optimize:\n for f in files.keys():\n if not validate_file(f):\n error = 1\n try:\n if debug:\n print \"lex: regex = '%s'\" % regex\n lexer.lexre = re.compile(regex, re.VERBOSE)\n\n # Build the index to function map for the matching engine\n lexer.lexindexfunc = [ None ] * (max(lexer.lexre.groupindex.values())+1)\n for f,i in lexer.lexre.groupindex.items():\n handle = ldict[f]\n if type(handle) in (types.FunctionType, types.MethodType):\n lexer.lexindexfunc[i] = (handle,handle.__name__[2:])\n else:\n # If rule was specified as a string, we build an anonymous\n # callback function to carry out the action\n lexer.lexindexfunc[i] = (None,f[2:])\n\n # If a lextab was specified, we create a file containing the precomputed\n # regular expression and index table\n\n if lextab and optimize:\n lt = open(lextab+\".py\",\"w\")\n lt.write(\"# %s.py. This file automatically created by PLY. Don't edit.\\n\" % lextab)\n lt.write(\"_lexre = %s\\n\" % repr(regex))\n lt.write(\"_lextab = [\\n\");\n for i in range(0,len(lexer.lexindexfunc)):\n t = lexer.lexindexfunc[i]\n if t:\n if t[0]:\n lt.write(\" ('%s',%s),\\n\"% (t[0].__name__, repr(t[1])))\n else:\n lt.write(\" (None,%s),\\n\" % repr(t[1]))\n else:\n lt.write(\" None,\\n\")\n\n lt.write(\"]\\n\");\n lt.write(\"_lextokens = %s\\n\" % repr(lexer.lextokens))\n lt.write(\"_lexignore = %s\\n\" % repr(lexer.lexignore))\n if (lexer.lexerrorf):\n lt.write(\"_lexerrorf = %s\\n\" % repr(lexer.lexerrorf.__name__))\n else:\n lt.write(\"_lexerrorf = None\\n\")\n lt.close()\n\n except re.error,e:\n print \"lex: Fatal error. Unable to compile regular expression rules. %s\" % e\n error = 1\n if error:\n raise SyntaxError,\"lex: Unable to build lexer.\"\n if not lexer.lexerrorf:\n print \"lex: Warning. no t_error rule is defined.\"\n\n if not lexer.lexignore: lexer.lexignore = \"\"\n\n # Create global versions of the token() and input() functions\n token = lexer.token\n input = lexer.input\n\n return lexer\n\n# -----------------------------------------------------------------------------\n# run()\n#\n# This runs the lexer as a main program\n# -----------------------------------------------------------------------------\n\ndef runmain(lexer=None,data=None):\n if not data:\n try:\n filename = sys.argv[1]\n f = open(filename)\n data = f.read()\n f.close()\n except IndexError:\n print \"Reading from standard input (type EOF to end):\"\n data = sys.stdin.read()\n\n if lexer:\n _input = lexer.input\n else:\n _input = input\n _input(data)\n if lexer:\n _token = lexer.token\n else:\n _token = token\n\n while 1:\n tok = _token()\n if not tok: break\n print \"(%s,'%s',%d)\" % (tok.type, tok.value, tok.lineno)\n" }, { "path": "src/ccg2xml/yacc.py", "content": "#-----------------------------------------------------------------------------\n# ply: yacc.py\n#\n# Author(s): David M. Beazley (dave@dabeaz.com)\n#\n# Copyright (C) 2001-2005, David M. Beazley\n#\n# $Header: /cvsroot/openccg/openccg/src/ccg2xml/yacc.py,v 1.1 2006/09/30 08:11:29 benwing Exp $\n#\n# This library is free software; you can redistribute it and/or\n# modify it under the terms of the GNU Lesser General Public\n# License as published by the Free Software Foundation; either\n# version 2.1 of the License, or (at your option) any later version.\n# \n# This library is distributed in the hope that it will be useful,\n# but WITHOUT ANY WARRANTY; without even the implied warranty of\n# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU\n# Lesser General Public License for more details.\n# \n# You should have received a copy of the GNU Lesser General Public\n# License along with this library; if not, write to the Free Software\n# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA\n# \n# See the file COPYING for a complete copy of the LGPL.\n#\n#\n# This implements an LR parser that is constructed from grammar rules defined\n# as Python functions. Roughly speaking, this module is a cross between\n# John Aycock's Spark system and the GNU bison utility.\n#\n# The current implementation is only somewhat object-oriented. The\n# LR parser itself is defined in terms of an object (which allows multiple\n# parsers to co-exist). However, most of the variables used during table\n# construction are defined in terms of global variables. Users shouldn't\n# notice unless they are trying to define multiple parsers at the same\n# time using threads (in which case they should have their head examined).\n#\n# This implementation supports both SLR and LALR(1) parsing. LALR(1)\n# support was implemented by Elias Ioup (ezioup@alumni.uchicago.edu)\n# and hacked abit by Dave to run faster.\n#\n# :::::::: WARNING :::::::\n#\n# Construction of LR parsing tables is fairly complicated and expensive.\n# To make this module run fast, a *LOT* of work has been put into\n# optimization---often at the expensive of readability and what might\n# consider to be good Python \"coding style.\" Modify the code at your\n# own risk!\n# ----------------------------------------------------------------------------\n\n__version__ = \"1.6\"\n\n#-----------------------------------------------------------------------------\n# === User configurable parameters ===\n#\n# Change these to modify the default behavior of yacc (if you wish)\n#-----------------------------------------------------------------------------\n\nyaccdebug = 1 # Debugging mode. If set, yacc generates a\n # a 'parser.out' file in the current directory\n\ndebug_file = 'parser.out' # Default name of the debugging file\ntab_module = 'parsetab' # Default name of the table module\ndefault_lr = 'SLR' # Default LR table generation method\n\nerror_count = 3 # Number of symbols that must be shifted to leave recovery mode\n\nimport re, types, sys, cStringIO, md5, os.path\n\n# Exception raised for yacc-related errors\nclass YaccError(Exception): pass\n\n#-----------------------------------------------------------------------------\n# === LR Parsing Engine ===\n#\n# The following classes are used for the LR parser itself. These are not\n# used during table construction and are independent of the actual LR\n# table generation algorithm\n#-----------------------------------------------------------------------------\n\n# This class is used to hold non-terminal grammar symbols during parsing.\n# It normally has the following attributes set:\n# .type = Grammar symbol type\n# .value = Symbol value\n# .lineno = Starting line number\n# .endlineno = Ending line number (optional, set automatically)\n\nclass YaccSymbol:\n def __str__(self): return self.type\n def __repr__(self): return str(self)\n\n# This class is a wrapper around the objects actually passed to each\n# grammar rule. Index lookup and assignment actually assign the\n# .value attribute of the underlying YaccSymbol object.\n# The lineno() method returns the line number of a given\n# item (or 0 if not defined). The linespan() method returns\n# a tuple of (startline,endline) representing the range of lines\n# for a symbol.\n\nclass YaccProduction:\n def __init__(self,s):\n self.slice = s\n self.pbstack = []\n\n def __getitem__(self,n):\n return self.slice[n].value\n\n def __setitem__(self,n,v):\n self.slice[n].value = v\n\n def __len__(self):\n return len(self.slice)\n \n def lineno(self,n):\n return getattr(self.slice[n],\"lineno\",0)\n\n def linespan(self,n):\n startline = getattr(self.slice[n],\"lineno\",0)\n endline = getattr(self.slice[n],\"endlineno\",startline)\n return startline,endline\n\n def pushback(self,n):\n if n <= 0:\n raise ValueError, \"Expected a positive value\"\n if n > (len(self.slice)-1):\n raise ValueError, \"Can't push %d tokens. Only %d are available.\" % (n,len(self.slice)-1)\n for i in range(0,n):\n self.pbstack.append(self.slice[-i-1])\n\n# The LR Parsing engine. This is defined as a class so that multiple parsers\n# can exist in the same process. A user never instantiates this directly.\n# Instead, the global yacc() function should be used to create a suitable Parser\n# object. \n\nclass Parser:\n def __init__(self,magic=None):\n\n # This is a hack to keep users from trying to instantiate a Parser\n # object directly.\n\n if magic != \"xyzzy\":\n raise YaccError, \"Can't instantiate Parser. Use yacc() instead.\"\n\n # Reset internal state\n self.productions = None # List of productions\n self.errorfunc = None # Error handling function\n self.action = { } # LR Action table\n self.goto = { } # LR goto table\n self.require = { } # Attribute require table\n self.method = \"Unknown LR\" # Table construction method used\n\n def errok(self):\n self.errorcount = 0\n\n def restart(self):\n del self.statestack[:]\n del self.symstack[:]\n sym = YaccSymbol()\n sym.type = '$'\n self.symstack.append(sym)\n self.statestack.append(0)\n \n def parse(self,input=None,lexer=None,debug=0):\n lookahead = None # Current lookahead symbol\n lookaheadstack = [ ] # Stack of lookahead symbols\n actions = self.action # Local reference to action table\n goto = self.goto # Local reference to goto table\n prod = self.productions # Local reference to production list\n pslice = YaccProduction(None) # Production object passed to grammar rules\n pslice.parser = self # Parser object\n self.errorcount = 0 # Used during error recovery\n\n # If no lexer was given, we will try to use the lex module\n if not lexer:\n import lex as lexer\n\n pslice.lexer = lexer\n \n # If input was supplied, pass to lexer\n if input:\n lexer.input(input)\n\n # Tokenize function\n get_token = lexer.token\n\n statestack = [ ] # Stack of parsing states\n self.statestack = statestack\n symstack = [ ] # Stack of grammar symbols\n self.symstack = symstack\n\n errtoken = None # Err token\n\n # The start state is assumed to be (0,$)\n statestack.append(0)\n sym = YaccSymbol()\n sym.type = '$'\n symstack.append(sym)\n \n while 1:\n # Get the next symbol on the input. If a lookahead symbol\n # is already set, we just use that. Otherwise, we'll pull\n # the next token off of the lookaheadstack or from the lexer\n if not lookahead:\n if not lookaheadstack:\n lookahead = get_token() # Get the next token\n else:\n lookahead = lookaheadstack.pop()\n if not lookahead:\n lookahead = YaccSymbol()\n lookahead.type = '$'\n if debug:\n errorlead = (\"%s . %s\" % (\" \".join([xx.type for xx in symstack][1:]), str(lookahead))).lstrip()\n\n # Check the action table\n s = statestack[-1]\n ltype = lookahead.type\n t = actions.get((s,ltype),None)\n\n if t is not None:\n if t > 0:\n # shift a symbol on the stack\n if ltype == '$':\n # Error, end of input\n sys.stderr.write(\"yacc: Parse error. EOF\\n\")\n return\n statestack.append(t)\n if debug > 1:\n sys.stderr.write(\"%-60s shift state %s\\n\" % (errorlead, t))\n symstack.append(lookahead)\n lookahead = None\n\n # Decrease error count on successful shift\n if self.errorcount > 0:\n self.errorcount -= 1\n \n continue\n \n if t < 0:\n # reduce a symbol on the stack, emit a production\n p = prod[-t]\n pname = p.name\n plen = p.len\n\n # Get production function\n sym = YaccSymbol()\n sym.type = pname # Production name\n sym.value = None\n if debug > 1:\n sys.stderr.write(\"%-60s reduce %d\\n\" % (errorlead, -t))\n\n if plen:\n targ = symstack[-plen-1:]\n targ[0] = sym\n try:\n sym.lineno = targ[1].lineno\n sym.endlineno = getattr(targ[-1],\"endlineno\",targ[-1].lineno)\n except AttributeError:\n sym.lineno = 0\n del symstack[-plen:]\n del statestack[-plen:]\n else:\n sym.lineno = 0\n targ = [ sym ]\n pslice.slice = targ\n pslice.pbstack = []\n # Call the grammar rule with our special slice object\n p.func(pslice)\n\n # If there was a pushback, put that on the stack\n if pslice.pbstack:\n lookaheadstack.append(lookahead)\n for _t in pslice.pbstack:\n lookaheadstack.append(_t)\n lookahead = None\n\n symstack.append(sym)\n statestack.append(goto[statestack[-1],pname])\n continue\n\n if t == 0:\n n = symstack[-1]\n return getattr(n,\"value\",None)\n sys.stderr.write(errorlead, \"\\n\")\n\n if t == None:\n if debug:\n sys.stderr.write(errorlead + \"\\n\")\n # We have some kind of parsing error here. To handle\n # this, we are going to push the current token onto\n # the tokenstack and replace it with an 'error' token.\n # If there are any synchronization rules, they may\n # catch it.\n #\n # In addition to pushing the error token, we call call\n # the user defined p_error() function if this is the\n # first syntax error. This function is only called if\n # errorcount == 0.\n if not self.errorcount:\n self.errorcount = error_count\n errtoken = lookahead\n if errtoken.type == '$':\n errtoken = None # End of file!\n if self.errorfunc:\n global errok,token,restart\n errok = self.errok # Set some special functions available in error recovery\n token = get_token\n restart = self.restart\n tok = self.errorfunc(errtoken)\n del errok, token, restart # Delete special functions\n \n if not self.errorcount:\n # User must have done some kind of panic\n # mode recovery on their own. The\n # returned token is the next lookahead\n lookahead = tok\n errtoken = None\n continue\n else:\n if errtoken:\n if hasattr(errtoken,\"lineno\"): lineno = lookahead.lineno\n else: lineno = 0\n if lineno:\n sys.stderr.write(\"yacc: Syntax error at line %d, token=%s\\n\" % (lineno, errtoken.type))\n else:\n sys.stderr.write(\"yacc: Syntax error, token=%s\" % errtoken.type)\n else:\n sys.stderr.write(\"yacc: Parse error in input. EOF\\n\")\n return\n\n else:\n self.errorcount = error_count\n \n # case 1: the statestack only has 1 entry on it. If we're in this state, the\n # entire parse has been rolled back and we're completely hosed. The token is\n # discarded and we just keep going.\n\n if len(statestack) <= 1 and lookahead.type != '$':\n lookahead = None\n errtoken = None\n # Nuke the pushback stack\n del lookaheadstack[:]\n continue\n\n # case 2: the statestack has a couple of entries on it, but we're\n # at the end of the file. nuke the top entry and generate an error token\n\n # Start nuking entries on the stack\n if lookahead.type == '$':\n # Whoa. We're really hosed here. Bail out\n return \n\n if lookahead.type != 'error':\n sym = symstack[-1]\n if sym.type == 'error':\n # Hmmm. Error is on top of stack, we'll just nuke input\n # symbol and continue\n lookahead = None\n continue\n t = YaccSymbol()\n t.type = 'error'\n if hasattr(lookahead,\"lineno\"):\n t.lineno = lookahead.lineno\n t.value = lookahead\n lookaheadstack.append(lookahead)\n lookahead = t\n else:\n symstack.pop()\n statestack.pop()\n\n continue\n\n # Call an error function here\n raise RuntimeError, \"yacc: internal parser error!!!\\n\"\n\n# -----------------------------------------------------------------------------\n# === Parser Construction ===\n#\n# The following functions and variables are used to implement the yacc() function\n# itself. This is pretty hairy stuff involving lots of error checking,\n# construction of LR items, kernels, and so forth. Although a lot of\n# this work is done using global variables, the resulting Parser object\n# is completely self contained--meaning that it is safe to repeatedly\n# call yacc() with different grammars in the same application.\n# -----------------------------------------------------------------------------\n \n# -----------------------------------------------------------------------------\n# validate_file()\n#\n# This function checks to see if there are duplicated p_rulename() functions\n# in the parser module file. Without this function, it is really easy for\n# users to make mistakes by cutting and pasting code fragments (and it's a real\n# bugger to try and figure out why the resulting parser doesn't work). Therefore,\n# we just do a little regular expression pattern matching of def statements\n# to try and detect duplicates.\n# -----------------------------------------------------------------------------\n\ndef validate_file(filename):\n base,ext = os.path.splitext(filename)\n if ext != '.py': return 1 # No idea. Assume it's okay.\n\n try:\n f = open(filename)\n lines = f.readlines()\n f.close()\n except IOError:\n return 1 # Oh well\n\n # Match def p_funcname(\n fre = re.compile(r'\\s*def\\s+(p_[a-zA-Z_0-9]*)\\(')\n counthash = { }\n linen = 1\n noerror = 1\n for l in lines:\n m = fre.match(l)\n if m:\n name = m.group(1)\n prev = counthash.get(name)\n if not prev:\n counthash[name] = linen\n else:\n sys.stderr.write(\"%s:%d: Function %s redefined. Previously defined on line %d\\n\" % (filename,linen,name,prev))\n noerror = 0\n linen += 1\n return noerror\n\n# This function looks for functions that might be grammar rules, but which don't have the proper p_suffix.\ndef validate_dict(d):\n for n,v in d.items(): \n if n[0:2] == 'p_' and type(v) in (types.FunctionType, types.MethodType): continue\n if n[0:2] == 't_': continue\n\n if n[0:2] == 'p_':\n sys.stderr.write(\"yacc: Warning. '%s' not defined as a function\\n\" % n)\n if 1 and isinstance(v,types.FunctionType) and v.func_code.co_argcount == 1:\n try:\n doc = v.__doc__.split(\" \")\n if doc[1] == ':':\n sys.stderr.write(\"%s:%d: Warning. Possible grammar rule '%s' defined without p_ prefix.\\n\" % (v.func_code.co_filename, v.func_code.co_firstlineno,n))\n except StandardError:\n pass\n\n# -----------------------------------------------------------------------------\n# === GRAMMAR FUNCTIONS ===\n#\n# The following global variables and functions are used to store, manipulate,\n# and verify the grammar rules specified by the user.\n# -----------------------------------------------------------------------------\n\n# Initialize all of the global variables used during grammar construction\ndef initialize_vars():\n global Productions, Prodnames, Prodmap, Terminals \n global Nonterminals, First, Follow, Precedence, LRitems\n global Errorfunc, Signature, Requires\n\n # LALR(1) globals\n global Prodempty, TReductions, NTReductions, GotoSetNum, Canonical\n \n Productions = [None] # A list of all of the productions. The first\n # entry is always reserved for the purpose of\n # building an augmented grammar\n \n Prodnames = { } # A dictionary mapping the names of nonterminals to a list of all\n # productions of that nonterminal.\n \n Prodmap = { } # A dictionary that is only used to detect duplicate\n # productions.\n\n Terminals = { } # A dictionary mapping the names of terminal symbols to a\n # list of the rules where they are used.\n\n Nonterminals = { } # A dictionary mapping names of nonterminals to a list\n # of rule numbers where they are used.\n\n First = { } # A dictionary of precomputed FIRST(x) symbols\n \n Follow = { } # A dictionary of precomputed FOLLOW(x) symbols\n\n Precedence = { } # Precedence rules for each terminal. Contains tuples of the\n # form ('right',level) or ('nonassoc', level) or ('left',level)\n\n LRitems = [ ] # A list of all LR items for the grammar. These are the\n # productions with the \"dot\" like E -> E . PLUS E\n\n Errorfunc = None # User defined error handler\n\n Signature = md5.new() # Digital signature of the grammar rules, precedence\n # and other information. Used to determined when a\n # parsing table needs to be regenerated.\n\n Requires = { } # Requires list\n\n # LALR(1) Initialization\n Prodempty = { } # A dictionary of all productions that have an empty rule\n # of the form P : \n\n TReductions = { } # A dictionary of precomputer reductions from\n # nonterminals to terminals\n\n NTReductions = { } # A dictionary of precomputed reductions from\n # nonterminals to nonterminals\n\n GotoSetNum = { } # A dictionary that remembers goto sets based on\n # the state number and symbol\n\n Canonical = { } # A list of LR item sets. A LR item set is a list of LR\n # items that represent the state of the parser\n\n # File objects used when creating the parser.out debugging file\n global _vf, _vfc\n _vf = cStringIO.StringIO()\n _vfc = cStringIO.StringIO()\n\n# -----------------------------------------------------------------------------\n# class Production:\n#\n# This class stores the raw information about a single production or grammar rule.\n# It has a few required attributes:\n#\n# name - Name of the production (nonterminal)\n# prod - A list of symbols making up its production\n# number - Production number.\n#\n# In addition, a few additional attributes are used to help with debugging or\n# optimization of table generation.\n#\n# file - File where production action is defined.\n# lineno - Line number where action is defined\n# func - Action function\n# prec - Precedence level\n# lr_next - Next LR item. Example, if we are ' E -> E . PLUS E'\n# then lr_next refers to 'E -> E PLUS . E' \n# lr_index - LR item index (location of the \".\") in the prod list.\n# lookaheads - LALR lookahead symbols for this item\n# len - Length of the production (number of symbols on right hand side)\n# -----------------------------------------------------------------------------\n\nclass Production:\n def __init__(self,**kw):\n for k,v in kw.items():\n setattr(self,k,v)\n self.lr_index = -1\n self.lr0_added = 0 # Flag indicating whether or not added to LR0 closure\n self.lr1_added = 0 # Flag indicating whether or not added to LR1\n self.usyms = [ ]\n self.lookaheads = { }\n self.lk_added = { }\n self.setnumbers = [ ]\n \n def __str__(self):\n if self.prod:\n s = \"%s -> %s\" % (self.name,\" \".join(self.prod))\n else:\n s = \"%s -> \" % self.name\n return s\n\n def __repr__(self):\n return str(self)\n\n # Compute lr_items from the production\n def lr_item(self,n):\n if n > len(self.prod): return None\n p = Production()\n p.name = self.name\n p.prod = list(self.prod)\n p.number = self.number\n p.lr_index = n\n p.lookaheads = { }\n p.setnumbers = self.setnumbers\n p.prod.insert(n,\".\")\n p.prod = tuple(p.prod)\n p.len = len(p.prod)\n p.usyms = self.usyms\n\n # Precompute list of productions immediately following\n try:\n p.lrafter = Prodnames[p.prod[n+1]]\n except (IndexError,KeyError),e:\n p.lrafter = []\n try:\n p.lrbefore = p.prod[n-1]\n except IndexError:\n p.lrbefore = None\n\n return p\n\nclass MiniProduction:\n pass\n\n# Utility function\ndef is_identifier(s):\n for c in s:\n if not (c.isalnum() or c == '_'): return 0\n return 1\n\n# -----------------------------------------------------------------------------\n# add_production()\n#\n# Given an action function, this function assembles a production rule.\n# The production rule is assumed to be found in the function's docstring.\n# This rule has the general syntax:\n#\n# name1 ::= production1\n# | production2\n# | production3\n# ...\n# | productionn\n# name2 ::= production1\n# | production2\n# ... \n# -----------------------------------------------------------------------------\n\ndef add_production(f,file,line,prodname,syms):\n \n if Terminals.has_key(prodname):\n sys.stderr.write(\"%s:%d: Illegal rule name '%s'. Already defined as a token.\\n\" % (file,line,prodname))\n return -1\n if prodname == 'error':\n sys.stderr.write(\"%s:%d: Illegal rule name '%s'. error is a reserved word.\\n\" % (file,line,prodname))\n return -1\n \n if not is_identifier(prodname):\n sys.stderr.write(\"%s:%d: Illegal rule name '%s'\\n\" % (file,line,prodname))\n return -1\n\n for s in syms:\n if not is_identifier(s) and s != '%prec':\n sys.stderr.write(\"%s:%d: Illegal name '%s' in rule '%s'\\n\" % (file,line,s, prodname))\n return -1\n\n # See if the rule is already in the rulemap\n map = \"%s -> %s\" % (prodname,syms)\n if Prodmap.has_key(map):\n m = Prodmap[map]\n sys.stderr.write(\"%s:%d: Duplicate rule %s.\\n\" % (file,line, m))\n sys.stderr.write(\"%s:%d: Previous definition at %s:%d\\n\" % (file,line, m.file, m.line))\n return -1\n\n p = Production()\n p.name = prodname\n p.prod = syms\n p.file = file\n p.line = line\n p.func = f\n p.number = len(Productions)\n\n \n Productions.append(p)\n Prodmap[map] = p\n if not Nonterminals.has_key(prodname):\n Nonterminals[prodname] = [ ]\n \n # Add all terminals to Terminals\n i = 0\n while i < len(p.prod):\n t = p.prod[i]\n if t == '%prec':\n try:\n precname = p.prod[i+1]\n except IndexError:\n sys.stderr.write(\"%s:%d: Syntax error. Nothing follows %%prec.\\n\" % (p.file,p.line))\n return -1\n\n prec = Precedence.get(precname,None)\n if not prec:\n sys.stderr.write(\"%s:%d: Nothing known about the precedence of '%s'\\n\" % (p.file,p.line,precname))\n return -1\n else:\n p.prec = prec\n del p.prod[i]\n del p.prod[i]\n continue\n\n if Terminals.has_key(t):\n Terminals[t].append(p.number)\n # Is a terminal. We'll assign a precedence to p based on this\n if not hasattr(p,\"prec\"):\n p.prec = Precedence.get(t,('right',0))\n else:\n if not Nonterminals.has_key(t):\n Nonterminals[t] = [ ]\n Nonterminals[t].append(p.number)\n i += 1\n\n if not hasattr(p,\"prec\"):\n p.prec = ('right',0)\n \n # Set final length of productions\n p.len = len(p.prod)\n p.prod = tuple(p.prod)\n\n # Calculate unique syms in the production\n p.usyms = [ ]\n for s in p.prod:\n if s not in p.usyms:\n p.usyms.append(s)\n \n # Add to the global productions list\n try:\n Prodnames[p.name].append(p)\n except KeyError:\n Prodnames[p.name] = [ p ]\n return 0\n\n# Given a raw rule function, this function rips out its doc string\n# and adds rules to the grammar\n\ndef add_function(f):\n line = f.func_code.co_firstlineno\n file = f.func_code.co_filename\n error = 0\n\n if isinstance(f,types.MethodType):\n reqdargs = 2\n else:\n reqdargs = 1\n \n if f.func_code.co_argcount > reqdargs:\n sys.stderr.write(\"%s:%d: Rule '%s' has too many arguments.\\n\" % (file,line,f.__name__))\n return -1\n\n if f.func_code.co_argcount < reqdargs:\n sys.stderr.write(\"%s:%d: Rule '%s' requires an argument.\\n\" % (file,line,f.__name__))\n return -1\n \n if f.__doc__:\n # Split the doc string into lines\n pstrings = f.__doc__.splitlines()\n lastp = None\n dline = line\n for ps in pstrings:\n dline += 1\n p = ps.split()\n if not p: continue\n try:\n if p[0] == '|':\n # This is a continuation of a previous rule\n if not lastp:\n sys.stderr.write(\"%s:%d: Misplaced '|'.\\n\" % (file,dline))\n return -1\n prodname = lastp\n if len(p) > 1:\n syms = p[1:]\n else:\n syms = [ ]\n else:\n prodname = p[0]\n lastp = prodname\n assign = p[1]\n if len(p) > 2:\n syms = p[2:]\n else:\n syms = [ ]\n if assign != ':' and assign != '::=':\n sys.stderr.write(\"%s:%d: Syntax error. Expected ':'\\n\" % (file,dline))\n return -1\n e = add_production(f,file,dline,prodname,syms)\n error += e\n except StandardError:\n sys.stderr.write(\"%s:%d: Syntax error in rule '%s'\\n\" % (file,dline,ps))\n error -= 1\n else:\n sys.stderr.write(\"%s:%d: No documentation string specified in function '%s'\\n\" % (file,line,f.__name__))\n return error\n\n\n# Cycle checking code (Michael Dyck)\n\ndef compute_reachable():\n '''\n Find each symbol that can be reached from the start symbol.\n Print a warning for any nonterminals that can't be reached.\n (Unused terminals have already had their warning.)\n '''\n Reachable = { }\n for s in Terminals.keys() + Nonterminals.keys():\n Reachable[s] = 0\n\n mark_reachable_from( Productions[0].prod[0], Reachable )\n\n for s in Nonterminals.keys():\n if not Reachable[s]:\n sys.stderr.write(\"yacc: Symbol '%s' is unreachable.\\n\" % s)\n\ndef mark_reachable_from(s, Reachable):\n '''\n Mark all symbols that are reachable from symbol s.\n '''\n if Reachable[s]:\n # We've already reached symbol s.\n return\n Reachable[s] = 1\n for p in Prodnames.get(s,[]):\n for r in p.prod:\n mark_reachable_from(r, Reachable)\n\n# -----------------------------------------------------------------------------\n# compute_terminates()\n#\n# This function looks at the various parsing rules and tries to detect\n# infinite recursion cycles (grammar rules where there is no possible way\n# to derive a string of only terminals).\n# -----------------------------------------------------------------------------\ndef compute_terminates():\n '''\n Raise an error for any symbols that don't terminate.\n '''\n Terminates = {}\n\n # Terminals:\n for t in Terminals.keys():\n Terminates[t] = 1\n\n Terminates['$'] = 1\n\n # Nonterminals:\n\n # Initialize to false:\n for n in Nonterminals.keys():\n Terminates[n] = 0\n\n # Then propagate termination until no change:\n while 1:\n some_change = 0\n for (n,pl) in Prodnames.items():\n # Nonterminal n terminates iff any of its productions terminates.\n for p in pl:\n # Production p terminates iff all of its rhs symbols terminate.\n for s in p.prod:\n if not Terminates[s]:\n # The symbol s does not terminate,\n # so production p does not terminate.\n p_terminates = 0\n break\n else:\n # didn't break from the loop,\n # so every symbol s terminates\n # so production p terminates.\n p_terminates = 1\n\n if p_terminates:\n # symbol n terminates!\n if not Terminates[n]:\n Terminates[n] = 1\n some_change = 1\n # Don't need to consider any more productions for this n.\n break\n\n if not some_change:\n break\n\n some_error = 0\n for (s,terminates) in Terminates.items():\n if not terminates:\n if not Prodnames.has_key(s) and not Terminals.has_key(s) and s != 'error':\n # s is used-but-not-defined, and we've already warned of that,\n # so it would be overkill to say that it's also non-terminating.\n pass\n else:\n sys.stderr.write(\"yacc: Infinite recursion detected for symbol '%s'.\\n\" % s)\n some_error = 1\n\n return some_error\n\n# -----------------------------------------------------------------------------\n# verify_productions()\n#\n# This function examines all of the supplied rules to see if they seem valid.\n# -----------------------------------------------------------------------------\ndef verify_productions(cycle_check=1):\n error = 0\n for p in Productions:\n if not p: continue\n\n for s in p.prod:\n if not Prodnames.has_key(s) and not Terminals.has_key(s) and s != 'error':\n sys.stderr.write(\"%s:%d: Symbol '%s' used, but not defined as a token or a rule.\\n\" % (p.file,p.line,s))\n error = 1\n continue\n\n unused_tok = 0 \n # Now verify all of the tokens\n if yaccdebug:\n _vf.write(\"Unused terminals:\\n\\n\")\n for s,v in Terminals.items():\n if s != 'error' and not v:\n sys.stderr.write(\"yacc: Warning. Token '%s' defined, but not used.\\n\" % s)\n if yaccdebug: _vf.write(\" %s\\n\"% s)\n unused_tok += 1\n\n # Print out all of the productions\n if yaccdebug:\n _vf.write(\"\\nGrammar\\n\\n\")\n for i in range(1,len(Productions)):\n _vf.write(\"Rule %-5d %s\\n\" % (i, Productions[i]))\n \n unused_prod = 0\n # Verify the use of all productions\n for s,v in Nonterminals.items():\n if not v:\n p = Prodnames[s][0]\n sys.stderr.write(\"%s:%d: Warning. Rule '%s' defined, but not used.\\n\" % (p.file,p.line, s))\n unused_prod += 1\n\n \n if unused_tok == 1:\n sys.stderr.write(\"yacc: Warning. There is 1 unused token.\\n\")\n if unused_tok > 1:\n sys.stderr.write(\"yacc: Warning. There are %d unused tokens.\\n\" % unused_tok)\n\n if unused_prod == 1:\n sys.stderr.write(\"yacc: Warning. There is 1 unused rule.\\n\")\n if unused_prod > 1:\n sys.stderr.write(\"yacc: Warning. There are %d unused rules.\\n\" % unused_prod)\n\n if yaccdebug:\n _vf.write(\"\\nTerminals, with rules where they appear\\n\\n\")\n ks = Terminals.keys()\n ks.sort()\n for k in ks:\n _vf.write(\"%-20s : %s\\n\" % (k, \" \".join([str(s) for s in Terminals[k]])))\n _vf.write(\"\\nNonterminals, with rules where they appear\\n\\n\")\n ks = Nonterminals.keys()\n ks.sort()\n for k in ks:\n _vf.write(\"%-20s : %s\\n\" % (k, \" \".join([str(s) for s in Nonterminals[k]])))\n\n if (cycle_check):\n compute_reachable()\n error += compute_terminates()\n# error += check_cycles()\n return error\n\n# -----------------------------------------------------------------------------\n# build_lritems()\n#\n# This function walks the list of productions and builds a complete set of the\n# LR items. The LR items are stored in two ways: First, they are uniquely\n# numbered and placed in the list _lritems. Second, a linked list of LR items\n# is built for each production. For example:\n#\n# E -> E PLUS E\n#\n# Creates the list\n#\n# [E -> . E PLUS E, E -> E . PLUS E, E -> E PLUS . E, E -> E PLUS E . ] \n# -----------------------------------------------------------------------------\n\ndef build_lritems():\n for p in Productions:\n lastlri = p\n lri = p.lr_item(0)\n i = 0\n while 1:\n lri = p.lr_item(i)\n lastlri.lr_next = lri\n if not lri: break\n lri.lr_num = len(LRitems)\n LRitems.append(lri)\n lastlri = lri\n i += 1\n\n # In order for the rest of the parser generator to work, we need to\n # guarantee that no more lritems are generated. Therefore, we nuke\n # the p.lr_item method. (Only used in debugging)\n # Production.lr_item = None\n\n# -----------------------------------------------------------------------------\n# add_precedence()\n#\n# Given a list of precedence rules, add to the precedence table.\n# -----------------------------------------------------------------------------\n\ndef add_precedence(plist):\n plevel = 0\n error = 0\n for p in plist:\n plevel += 1\n try:\n prec = p[0]\n terms = p[1:]\n if prec != 'left' and prec != 'right' and prec != 'nonassoc':\n sys.stderr.write(\"yacc: Invalid precedence '%s'\\n\" % prec)\n return -1\n for t in terms:\n if Precedence.has_key(t):\n sys.stderr.write(\"yacc: Precedence already specified for terminal '%s'\\n\" % t)\n error += 1\n continue\n Precedence[t] = (prec,plevel)\n except:\n sys.stderr.write(\"yacc: Invalid precedence table.\\n\")\n error += 1\n\n return error\n\n# -----------------------------------------------------------------------------\n# augment_grammar()\n#\n# Compute the augmented grammar. This is just a rule S' -> start where start\n# is the starting symbol.\n# -----------------------------------------------------------------------------\n\ndef augment_grammar(start=None):\n if not start:\n start = Productions[1].name\n Productions[0] = Production(name=\"S'\",prod=[start],number=0,len=1,prec=('right',0),func=None)\n Productions[0].usyms = [ start ]\n Nonterminals[start].append(0)\n\n\n# -------------------------------------------------------------------------\n# first()\n#\n# Compute the value of FIRST1(beta) where beta is a tuple of symbols.\n#\n# During execution of compute_first1, the result may be incomplete.\n# Afterward (e.g., when called from compute_follow()), it will be complete.\n# -------------------------------------------------------------------------\ndef first(beta):\n\n # We are computing First(x1,x2,x3,...,xn)\n result = [ ]\n for x in beta:\n x_produces_empty = 0\n\n # Add all the non- symbols of First[x] to the result.\n for f in First[x]:\n if f == '':\n x_produces_empty = 1\n else:\n if f not in result: result.append(f)\n\n if x_produces_empty:\n # We have to consider the next x in beta,\n # i.e. stay in the loop.\n pass\n else:\n # We don't have to consider any further symbols in beta.\n break\n else:\n # There was no 'break' from the loop,\n # so x_produces_empty was true for all x in beta,\n # so beta produces empty as well.\n result.append('')\n\n return result\n\n\n# FOLLOW(x)\n# Given a non-terminal. This function computes the set of all symbols\n# that might follow it. Dragon book, p. 189.\n\ndef compute_follow(start=None):\n # Add '$' to the follow list of the start symbol\n for k in Nonterminals.keys():\n Follow[k] = [ ]\n\n if not start:\n start = Productions[1].name\n \n Follow[start] = [ '$' ]\n \n while 1:\n didadd = 0\n for p in Productions[1:]:\n # Here is the production set\n for i in range(len(p.prod)):\n B = p.prod[i]\n if Nonterminals.has_key(B):\n # Okay. We got a non-terminal in a production\n fst = first(p.prod[i+1:])\n hasempty = 0\n for f in fst:\n if f != '' and f not in Follow[B]:\n Follow[B].append(f)\n didadd = 1\n if f == '':\n hasempty = 1\n if hasempty or i == (len(p.prod)-1):\n # Add elements of follow(a) to follow(b)\n for f in Follow[p.name]:\n if f not in Follow[B]:\n Follow[B].append(f)\n didadd = 1\n if not didadd: break\n\n if 0 and yaccdebug:\n _vf.write('\\nFollow:\\n')\n for k in Nonterminals.keys():\n _vf.write(\"%-20s : %s\\n\" % (k, \" \".join([str(s) for s in Follow[k]])))\n\n# -------------------------------------------------------------------------\n# compute_first1()\n#\n# Compute the value of FIRST1(X) for all symbols\n# -------------------------------------------------------------------------\ndef compute_first1():\n\n # Terminals:\n for t in Terminals.keys():\n First[t] = [t]\n\n First['$'] = ['$']\n First['#'] = ['#'] # what's this for?\n\n # Nonterminals:\n\n # Initialize to the empty set:\n for n in Nonterminals.keys():\n First[n] = []\n\n # Then propagate symbols until no change:\n while 1:\n some_change = 0\n for n in Nonterminals.keys():\n for p in Prodnames[n]:\n for f in first(p.prod):\n if f not in First[n]:\n First[n].append( f )\n some_change = 1\n if not some_change:\n break\n\n if 0 and yaccdebug:\n _vf.write('\\nFirst:\\n')\n for k in Nonterminals.keys():\n _vf.write(\"%-20s : %s\\n\" %\n (k, \" \".join([str(s) for s in First[k]])))\n\n# -----------------------------------------------------------------------------\n# === SLR Generation ===\n#\n# The following functions are used to construct SLR (Simple LR) parsing tables\n# as described on p.221-229 of the dragon book.\n# -----------------------------------------------------------------------------\n\n# Global variables for the LR parsing engine\ndef lr_init_vars():\n global _lr_action, _lr_goto, _lr_method\n global _lr_goto_cache\n \n _lr_action = { } # Action table\n _lr_goto = { } # Goto table\n _lr_method = \"Unknown\" # LR method used\n _lr_goto_cache = { }\n\n# Compute the LR(0) closure operation on I, where I is a set of LR(0) items.\n# prodlist is a list of productions.\n\n_add_count = 0 # Counter used to detect cycles\n\ndef lr0_closure(I):\n global _add_count\n \n _add_count += 1\n prodlist = Productions\n \n # Add everything in I to J \n J = I[:]\n didadd = 1\n while didadd:\n didadd = 0\n for j in J:\n for x in j.lrafter:\n if x.lr0_added == _add_count: continue\n # Add B --> .G to J\n J.append(x.lr_next)\n x.lr0_added = _add_count\n didadd = 1\n \n return J\n\n# Compute the LR(0) goto function goto(I,X) where I is a set\n# of LR(0) items and X is a grammar symbol. This function is written\n# in a way that guarantees uniqueness of the generated goto sets\n# (i.e. the same goto set will never be returned as two different Python\n# objects). With uniqueness, we can later do fast set comparisons using\n# id(obj) instead of element-wise comparison.\n\ndef lr0_goto(I,x):\n # First we look for a previously cached entry\n g = _lr_goto_cache.get((id(I),x),None)\n if g: return g\n\n # Now we generate the goto set in a way that guarantees uniqueness\n # of the result\n \n s = _lr_goto_cache.get(x,None)\n if not s:\n s = { }\n _lr_goto_cache[x] = s\n\n gs = [ ]\n for p in I:\n n = p.lr_next\n if n and n.lrbefore == x:\n s1 = s.get(id(n),None)\n if not s1:\n s1 = { }\n s[id(n)] = s1\n gs.append(n)\n s = s1\n g = s.get('$',None)\n if not g:\n if gs:\n g = lr0_closure(gs)\n s['$'] = g\n else:\n s['$'] = gs\n _lr_goto_cache[(id(I),x)] = g\n return g\n\n# Added for LALR(1)\n\n# Given a setnumber of an lr0 state and a symbol return the setnumber of the goto state \ndef lr0_goto_setnumber(I_setnumber, x):\n global Canonical\n global GotoSetNum\n\n if GotoSetNum.has_key((I_setnumber, x)):\n setnumber = GotoSetNum[(I_setnumber, x)]\n else:\n gset = lr0_goto(Canonical[I_setnumber], x)\n if not gset:\n return -1\n else:\n gsetlen = len(gset) \n for i in xrange(len(gset[0].setnumbers)):\n inall = 1\n for item in gset:\n if not item.setnumbers[i]:\n inall = 0\n break\n if inall and len(Canonical[i]) == gsetlen:\n setnumber = i\n break # Note: DB. I added this to improve performance.\n # Not sure if this breaks the algorithm (it doesn't appear to).\n\n GotoSetNum[(I_setnumber, x)] = setnumber\n \n return setnumber\n\n# Compute the kernel of a set of LR(0) items\ndef lr0_kernel(I):\n KI = [ ]\n for p in I:\n if p.name == \"S'\" or p.lr_index > 0 or p.len == 0:\n KI.append(p)\n\n return KI\n\n_lr0_cidhash = { }\n\n# Compute the LR(0) sets of item function\ndef lr0_items():\n \n C = [ lr0_closure([Productions[0].lr_next]) ]\n i = 0\n for I in C:\n _lr0_cidhash[id(I)] = i\n i += 1\n\n # Loop over the items in C and each grammar symbols\n i = 0\n while i < len(C):\n I = C[i]\n i += 1\n\n # Collect all of the symbols that could possibly be in the goto(I,X) sets\n asyms = { }\n for ii in I:\n for s in ii.usyms:\n asyms[s] = None\n\n for x in asyms.keys():\n g = lr0_goto(I,x)\n if not g: continue\n if _lr0_cidhash.has_key(id(g)): continue\n _lr0_cidhash[id(g)] = len(C) \n C.append(g)\n \n return C\n\n# -----------------------------------------------------------------------------\n# slr_parse_table()\n#\n# This function constructs an SLR table.\n# -----------------------------------------------------------------------------\ndef slr_parse_table():\n global _lr_method\n goto = _lr_goto # Goto array\n action = _lr_action # Action array\n actionp = { } # Action production array (temporary)\n\n _lr_method = \"SLR\"\n \n n_srconflict = 0\n n_rrconflict = 0\n\n if yaccdebug:\n sys.stderr.write(\"yacc: Generating SLR parsing table...\\n\") \n _vf.write(\"\\n\\nParsing method: SLR\\n\\n\")\n \n # Step 1: Construct C = { I0, I1, ... IN}, collection of LR(0) items\n # This determines the number of states\n \n C = lr0_items()\n\n # Build the parser table, state by state\n st = 0\n for I in C:\n # Loop over each production in I\n actlist = [ ] # List of actions\n \n if yaccdebug:\n _vf.write(\"\\nstate %d\\n\\n\" % st)\n for p in I:\n _vf.write(\" (%d) %s\\n\" % (p.number, str(p)))\n _vf.write(\"\\n\")\n\n for p in I:\n try:\n if p.prod[-1] == \".\":\n if p.name == \"S'\":\n # Start symbol. Accept!\n action[st,\"$\"] = 0\n actionp[st,\"$\"] = p\n else:\n # We are at the end of a production. Reduce!\n for a in Follow[p.name]:\n actlist.append((a,p,\"reduce using rule %d (%s)\" % (p.number,p)))\n r = action.get((st,a),None)\n if r is not None:\n # Whoa. Have a shift/reduce or reduce/reduce conflict\n if r > 0:\n # Need to decide on shift or reduce here\n # By default we favor shifting. Need to add\n # some precedence rules here.\n sprec,slevel = Productions[actionp[st,a].number].prec \n rprec,rlevel = Precedence.get(a,('right',0))\n if (slevel < rlevel) or ((slevel == rlevel) and (rprec == 'left')):\n # We really need to reduce here. \n action[st,a] = -p.number\n actionp[st,a] = p\n if not slevel and not rlevel:\n _vfc.write(\"shift/reduce conflict in state %d resolved as reduce.\\n\" % st)\n _vf.write(\" ! shift/reduce conflict for %s resolved as reduce.\\n\" % a)\n n_srconflict += 1\n elif (slevel == rlevel) and (rprec == 'nonassoc'):\n action[st,a] = None\n else:\n # Hmmm. Guess we'll keep the shift\n if not slevel and not rlevel:\n _vfc.write(\"shift/reduce conflict in state %d resolved as shift.\\n\" % st)\n _vf.write(\" ! shift/reduce conflict for %s resolved as shift.\\n\" % a)\n n_srconflict +=1 \n elif r < 0:\n # Reduce/reduce conflict. In this case, we favor the rule\n # that was defined first in the grammar file\n oldp = Productions[-r]\n pp = Productions[p.number]\n if oldp.line > pp.line:\n action[st,a] = -p.number\n actionp[st,a] = p\n # sys.stderr.write(\"Reduce/reduce conflict in state %d\\n\" % st)\n n_rrconflict += 1\n _vfc.write(\"reduce/reduce conflict in state %d resolved using rule %d (%s).\\n\" % (st, actionp[st,a].number, actionp[st,a]))\n _vf.write(\" ! reduce/reduce conflict for %s resolved using rule %d (%s).\\n\" % (a,actionp[st,a].number, actionp[st,a]))\n else:\n sys.stderr.write(\"Unknown conflict in state %d\\n\" % st)\n else:\n action[st,a] = -p.number\n actionp[st,a] = p\n else:\n i = p.lr_index\n a = p.prod[i+1] # Get symbol right after the \".\"\n if Terminals.has_key(a):\n g = lr0_goto(I,a)\n j = _lr0_cidhash.get(id(g),-1)\n if j >= 0:\n # We are in a shift state\n actlist.append((a,p,\"shift and go to state %d\" % j))\n r = action.get((st,a),None)\n if r is not None:\n # Whoa have a shift/reduce or shift/shift conflict\n if r > 0:\n if r != j:\n sys.stderr.write(\"Shift/shift conflict in state %d\\n\" % st)\n elif r < 0:\n # Do a precedence check.\n # - if precedence of reduce rule is higher, we reduce.\n # - if precedence of reduce is same and left assoc, we reduce.\n # - otherwise we shift\n rprec,rlevel = Productions[actionp[st,a].number].prec\n sprec,slevel = Precedence.get(a,('right',0))\n if (slevel > rlevel) or ((slevel == rlevel) and (rprec != 'left')):\n # We decide to shift here... highest precedence to shift\n action[st,a] = j\n actionp[st,a] = p\n if not slevel and not rlevel:\n n_srconflict += 1\n _vfc.write(\"shift/reduce conflict in state %d resolved as shift.\\n\" % st)\n _vf.write(\" ! shift/reduce conflict for %s resolved as shift.\\n\" % a)\n elif (slevel == rlevel) and (rprec == 'nonassoc'):\n action[st,a] = None\n else: \n # Hmmm. Guess we'll keep the reduce\n if not slevel and not rlevel:\n n_srconflict +=1\n _vfc.write(\"shift/reduce conflict in state %d resolved as reduce.\\n\" % st)\n _vf.write(\" ! shift/reduce conflict for %s resolved as reduce.\\n\" % a)\n \n else:\n sys.stderr.write(\"Unknown conflict in state %d\\n\" % st)\n else:\n action[st,a] = j\n actionp[st,a] = p\n \n except StandardError,e:\n raise YaccError, \"Hosed in slr_parse_table\", e\n\n # Print the actions associated with each terminal\n if yaccdebug:\n _actprint = { }\n for a,p,m in actlist:\n if action.has_key((st,a)):\n if p is actionp[st,a]:\n _vf.write(\" %-15s %s\\n\" % (a,m))\n _actprint[(a,m)] = 1\n _vf.write(\"\\n\")\n for a,p,m in actlist:\n if action.has_key((st,a)):\n if p is not actionp[st,a]:\n if not _actprint.has_key((a,m)):\n _vf.write(\" ! %-15s [ %s ]\\n\" % (a,m))\n _actprint[(a,m)] = 1\n \n # Construct the goto table for this state\n if yaccdebug:\n _vf.write(\"\\n\")\n nkeys = { }\n for ii in I:\n for s in ii.usyms:\n if Nonterminals.has_key(s):\n nkeys[s] = None\n for n in nkeys.keys():\n g = lr0_goto(I,n)\n j = _lr0_cidhash.get(id(g),-1) \n if j >= 0:\n goto[st,n] = j\n if yaccdebug:\n _vf.write(\" %-30s shift and go to state %d\\n\" % (n,j))\n\n st += 1\n\n if yaccdebug:\n if n_srconflict == 1:\n sys.stderr.write(\"yacc: %d shift/reduce conflict\\n\" % n_srconflict)\n if n_srconflict > 1:\n sys.stderr.write(\"yacc: %d shift/reduce conflicts\\n\" % n_srconflict)\n if n_rrconflict == 1:\n sys.stderr.write(\"yacc: %d reduce/reduce conflict\\n\" % n_rrconflict)\n if n_rrconflict > 1:\n sys.stderr.write(\"yacc: %d reduce/reduce conflicts\\n\" % n_rrconflict)\n\n\n\n# -----------------------------------------------------------------------------\n# ==== LALR(1) Parsing ====\n# FINISHED! 5/20/2003 by Elias Ioup\n# -----------------------------------------------------------------------------\n\n\n# Compute the lr1_closure of a set I. I is a list of productions and setnumber\n# is the state that you want the lr items that are made from the to come from.\n\n_lr1_add_count = 0\n\ndef lr1_closure(I, setnumber = 0):\n global _add_count\n global Nonterminals\n\n _add_count += 1\n prodlist = Productions\n\n # Add everything in I to J \n J = I[:]\n Jhash = { }\n for j in J:\n Jhash[id(j)] = 1\n \n didadd = 1\n while didadd:\n didadd = 0\n for j in J:\n jprod = j.prod\n jlr_index = j.lr_index\n jprodslice = jprod[jlr_index+2:]\n \n if jlr_index < len(jprod) - 1 and Nonterminals.has_key(jprod[jlr_index+1]):\n first_syms = []\n\n if j.lk_added.setdefault(setnumber, 0) < len(j.lookaheads[setnumber]): \n for a in j.lookaheads[setnumber][j.lk_added[setnumber]:]: \n # find b in FIRST(Xa) if j = [A->a.BX,a]\n temp_first_syms = first(jprodslice + (a,))\n for x in temp_first_syms:\n if x not in first_syms:\n first_syms.append(x)\n\n j.lk_added[setnumber] = len(j.lookaheads[setnumber]) \n\n for x in j.lrafter:\n \n # Add B --> .G to J\n if x.lr_next.lookaheads.has_key(setnumber):\n _xlook = x.lr_next.lookaheads[setnumber] \n for s in first_syms:\n if s not in _xlook:\n _xlook.append(s)\n didadd = 1\n else: \n x.lr_next.lookaheads[setnumber] = first_syms\n didadd = 1\n\n nid = id(x.lr_next)\n if not Jhash.has_key(nid):\n J.append(x.lr_next)\n Jhash[nid] = 1\n \n return J\n\ndef add_lookaheads(K):\n spontaneous = []\n propogate = []\n\n for setnumber in range(len(K)):\n for kitem in K[setnumber]:\n kitem.lookaheads[setnumber] = ['#']\n J = lr1_closure([kitem], setnumber)\n\n # find the lookaheads that are spontaneously created from closures\n # and the propogations of lookaheads between lr items\n for item in J:\n if item.lr_index < len(item.prod)-1:\n for lookahead in item.lookaheads[setnumber]:\n goto_setnumber = lr0_goto_setnumber(setnumber, item.prod[item.lr_index+1]) \n next = None \n if lookahead != '#':\n if item.lr_next in K[goto_setnumber]:\n next = item.lr_next\n if next:\n spontaneous.append((next, (lookahead, goto_setnumber)))\n else:\n if goto_setnumber > -1:\n if item.lr_next in K[goto_setnumber]:\n next = item.lr_next\n \n if next:\n propogate.append(((kitem, setnumber), (next, goto_setnumber)))\n\n \n\n for x in K[setnumber]:\n x.lookaheads[setnumber] = []\n\n for x in spontaneous:\n if x[1][0] not in x[0].lookaheads[x[1][1]]:\n x[0].lookaheads[x[1][1]].append(x[1][0])\n\n K[0][0].lookaheads[0] = ['$']\n\n pitems = {}\n for x in propogate:\n if pitems.has_key(x[0]):\n pitems[x[0]].append(x[1])\n else:\n pitems[x[0]] = []\n pitems[x[0]].append(x[1])\n \n # propogate the lookaheads that were spontaneously generated\n # based on the propogations produced above\n stop = 0\n\n while not stop:\n stop = 1\n kindex = 0\n for set in K:\n for item in set:\n pkey = (item, kindex)\n if pitems.has_key(pkey):\n for propogation in pitems[pkey]:\n gitem = propogation[0]\n gsetnumber = propogation[1]\n glookaheads = gitem.lookaheads[gsetnumber]\n for lookahead in item.lookaheads[kindex]:\n if lookahead not in glookaheads:\n glookaheads.append(lookahead)\n stop = 0\n kindex += 1\n\ndef ReduceNonterminals():\n global Nonterminals\n\n global TReductions\n global NTReductions\n\n for nt in Nonterminals.keys():\n TReductions[nt] = []\n NTReductions[nt] = []\n \n for nt in Nonterminals.keys():\n terms = ReduceToTerminals(nt)\n TReductions[nt].extend(terms)\n if not NTReductions.has_key(nt):\n ReduceToNonterminals(nt)\n \n\n\ndef ReduceToTerminals(nt):\n global Prodnames\n global Terminals\n reducedterminals = []\n\n for p in Prodnames[nt]:\n if len(p.prod) > 0:\n if Terminals.has_key(p.prod[0]):\n if p.prod[0] not in reducedterminals:\n reducedterminals.append(p.prod[0])\n else:\n if p.prod[0] != nt:\n terms = ReduceToTerminals(p.prod[0])\n for t in terms:\n if t not in reducedterminals:\n reducedterminals.append(t)\n\n return reducedterminals\n\n \ndef ReduceToNonterminals(nt):\n global Prodnames\n global Nonterminals\n global NTReductions\n reducednonterminals = []\n\n for p in Prodnames[nt]:\n if len(p.prod) > 0:\n if Nonterminals.has_key(p.prod[0]):\n if p.prod[0] not in reducednonterminals:\n reducednonterminals.append(p.prod[0])\n if p.prod[0] != nt:\n if not NTReductions.has_key(p.prod[0]):\n ReduceToNonterminals(p.prod[0])\n \n nterms = NTReductions[p.prod[0]]\n for nt in nterms:\n if nt not in reducednonterminals:\n reducednonterminals.append(nt)\n \n\n NTReductions[nt] = reducednonterminals\n\n# -----------------------------------------------------------------------------\n# lalr_parse_table()\n#\n# This function constructs an LALR table.\n# -----------------------------------------------------------------------------\ndef lalr_parse_table():\n global _lr_method\n goto = _lr_goto # Goto array\n action = _lr_action # Action array\n actionp = { } # Action production array (temporary)\n goto_cache = _lr_goto_cache\n cid_hash = _lr0_cidhash\n \n _lr_method = \"LALR\"\n \n n_srconflict = 0\n n_rrconflict = 0\n\n if yaccdebug:\n sys.stderr.write(\"yacc: Generating LALR(1) parsing table...\\n\")\n _vf.write(\"\\n\\nParsing method: LALR(1)\\n\\n\")\n \n # Step 1: Construct C = { I0, I1, ... IN}, collection of LR(0) items\n # This determines the number of states\n\n C = lr0_items()\n\n global Canonical\n Canonical = C\n\n ###\n # Create the kernel states.\n ###\n K = []\n setC = [0]*len(C)\n for x in C:\n K.append(lr0_kernel(x))\n for y in x:\n y.setnumbers = setC[:]\n\n _cindex = 0\n for x in C:\n for y in x:\n y.lookaheads[_cindex] = [] \n y.setnumbers[_cindex] = 1\n _cindex = _cindex + 1\n\n ###\n # Add lookaheads to the lr items\n ###\n\n add_lookaheads(K)\n\n ###\n # Do the reductions for parsing first and keep them in globals\n ###\n\n ReduceNonterminals()\n\n global TReductions\n global NTReductions\n global Prodempty\n\n EmptyAncestors = {}\n for y in Prodempty.keys():\n EmptyAncestors[y] = []\n for x in NTReductions.items():\n for y in x[1]:\n if Prodempty.has_key(y):\n EmptyAncestors[y].append(x[0])\n\n\n # Build the parser table, state by state\n st = 0\n for I in C:\n # Loop over each production in I\n actlist = [ ] # List of actions\n acthash = { }\n \n idI = id(I)\n \n if yaccdebug:\n _vf.write(\"\\nstate %d\\n\\n\" % st)\n for p in I:\n _vf.write(\" (%d) %s\\n\" % (p.number, str(p)))\n _vf.write(\"\\n\")\n\n global First\n for p in I:\n try:\n if p.prod[-1] == \".\":\n if p.name == \"S'\":\n # Start symbol. Accept!\n action[st,\"$\"] = 0\n actionp[st,\"$\"] = p\n elif len(p.prod) == 0:\n ancestors = EmptyAncestors[p.name]\n for i in ancestors:\n for s in K:\n if i in s:\n input_list = []\n plist = Productions[i.name]\n for x in plist:\n if len(x.prod) > 0 and x.prod[0] == p.name:\n n = p.prod[1:]\n d = x.prod[lr_index+2:]\n for l in x.lookaheads.items():\n flist = First[tuple(n+d+[l])]\n for f in flist:\n if f not in input_list and f in p.lookaheads[st]:\n input_list.append(f)\n \n # We are at the end of a production. Reduce!\n #print \"input_list: %s\" % input_list\n #print \"Follow[p.name]: %s\" % Follow[p.name]\n for a in input_list:\n actlist.append((a,p,\"reduce using rule %d (%s) \" % (p.number,p)))\n r = action.get((st,a),None)\n if r is not None:\n # Whoa. Have a shift/reduce or reduce/reduce conflict\n if r > 0:\n # Need to decide on shift or reduce here\n # By default we favor shifting. Need to add\n # some precedence rules here.\n sprec,slevel = Productions[actionp[st,a].number].prec \n rprec,rlevel = Precedence.get(a,('right',0))\n if (slevel < rlevel) or ((slevel == rlevel) and (rprec == 'left')):\n # We really need to reduce here. \n action[st,a] = -p.number\n actionp[st,a] = p\n if not slevel and not rlevel:\n _vfc.write(\"shift/reduce conflict in state %d resolved as reduce.\\n\" % st)\n _vf.write(\" ! shift/reduce conflict for %s resolved as reduce.\\n\" % a)\n n_srconflict += 1\n elif (slevel == rlevel) and (rprec == 'nonassoc'):\n action[st,a] = None\n else:\n # Hmmm. Guess we'll keep the shift\n if not slevel and not rlevel:\n _vfc.write(\"shift/reduce conflict in state %d resolved as shift.\\n\" % st)\n _vf.write(\" ! shift/reduce conflict for %s resolved as shift.\\n\" % a)\n n_srconflict +=1 \n elif r < 0:\n # Reduce/reduce conflict. In this case, we favor the rule\n # that was defined first in the grammar file\n oldp = Productions[-r]\n pp = Productions[p.number]\n if oldp.line > pp.line:\n action[st,a] = -p.number\n actionp[st,a] = p\n # print \"Reduce/reduce conflict in state %d\" % st\n n_rrconflict += 1\n _vfc.write(\"reduce/reduce conflict in state %d resolved using rule %d.\\n\" % (st, actionp[st,a].number))\n _vf.write(\" ! reduce/reduce conflict for %s resolved using rule %d.\\n\" % (a,actionp[st,a].number))\n else:\n sys.stderr.write(\"Unknown conflict in state %d\\n\" % st)\n else:\n action[st,a] = -p.number\n actionp[st,a] = p\n\n break # break out of the for s in K loop because we only want to make\n # sure that a production is in the Kernel\n \n else:\n # We are at the end of a production. Reduce!\n\n for a in p.lookaheads[st]:\n actlist.append((a,p,\"reduce using rule %d (%s)\" % (p.number,p)))\n r = action.get((st,a),None)\n if r is not None:\n # Whoa. Have a shift/reduce or reduce/reduce conflict\n if r > 0:\n # Need to decide on shift or reduce here\n # By default we favor shifting. Need to add\n # some precedence rules here.\n sprec,slevel = Productions[actionp[st,a].number].prec \n rprec,rlevel = Precedence.get(a,('right',0)) \n if (slevel < rlevel) or ((slevel == rlevel) and (rprec == 'left')):\n # We really need to reduce here. \n action[st,a] = -p.number\n actionp[st,a] = p\n if not slevel and not rlevel:\n _vfc.write(\"shift/reduce conflict in state %d resolved as reduce.\\n\" % st)\n _vf.write(\" ! shift/reduce conflict for %s resolved as reduce.\\n\" % a)\n n_srconflict += 1\n elif (slevel == rlevel) and (rprec == 'nonassoc'):\n action[st,a] = None\n else:\n # Hmmm. Guess we'll keep the shift\n if not slevel and not rlevel:\n _vfc.write(\"shift/reduce conflict in state %d resolved as shift.\\n\" % st)\n _vf.write(\" ! shift/reduce conflict for %s resolved as shift.\\n\" % a)\n n_srconflict +=1 \n elif r < 0:\n # Reduce/reduce conflict. In this case, we favor the rule\n # that was defined first in the grammar file\n oldp = Productions[-r]\n pp = Productions[p.number]\n if oldp.line > pp.line:\n action[st,a] = -p.number\n actionp[st,a] = p\n # print \"Reduce/reduce conflict in state %d\" % st\n n_rrconflict += 1\n _vfc.write(\"reduce/reduce conflict in state %d resolved using rule %d.\\n\" % (st, actionp[st,a].number))\n _vf.write(\" ! reduce/reduce conflict for %s resolved using rule %d.\\n\" % (a,actionp[st,a].number))\n else:\n print \"Unknown conflict in state %d\" % st\n else:\n action[st,a] = -p.number\n actionp[st,a] = p\n else:\n i = p.lr_index\n a = p.prod[i+1] # Get symbol right after the \".\"\n if Terminals.has_key(a):\n g = goto_cache[(idI,a)]\n j = cid_hash.get(id(g),-1)\n if j >= 0:\n # We are in a shift state\n _k = (a,j)\n if not acthash.has_key(_k):\n actlist.append((a,p,\"shift and go to state %d\" % j))\n acthash[_k] = 1\n r = action.get((st,a),None)\n if r is not None:\n # Whoa have a shift/reduce or shift/shift conflict\n if r > 0:\n if r != j:\n sys.stderr.write(\"Shift/shift conflict in state %d\\n\" % st)\n elif r < 0:\n # Do a precedence check.\n # - if precedence of reduce rule is higher, we reduce.\n # - if precedence of reduce is same and left assoc, we reduce.\n # - otherwise we shift\n rprec,rlevel = Productions[actionp[st,a].number].prec\n sprec,slevel = Precedence.get(a,('right',0))\n if (slevel > rlevel) or ((slevel == rlevel) and (rprec != 'left')):\n # We decide to shift here... highest precedence to shift\n action[st,a] = j\n actionp[st,a] = p\n if not slevel and not rlevel:\n n_srconflict += 1\n _vfc.write(\"shift/reduce conflict in state %d resolved as shift.\\n\" % st)\n _vf.write(\" ! shift/reduce conflict for %s resolved as shift.\\n\" % a)\n elif (slevel == rlevel) and (rprec == 'nonassoc'):\n action[st,a] = None\n else: \n # Hmmm. Guess we'll keep the reduce\n if not slevel and not rlevel:\n n_srconflict +=1\n _vfc.write(\"shift/reduce conflict in state %d resolved as reduce.\\n\" % st)\n _vf.write(\" ! shift/reduce conflict for %s resolved as reduce.\\n\" % a)\n \n else:\n sys.stderr.write(\"Unknown conflict in state %d\\n\" % st)\n else:\n action[st,a] = j\n actionp[st,a] = p\n else:\n nonterminal = a\n term_list = TReductions[nonterminal]\n # DB: This loop gets executed a lot. Try to optimize\n for a in term_list:\n g = goto_cache[(idI,a)]\n j = cid_hash[id(g)]\n if j >= 0:\n # We are in a shift state\n # Don't put repeated shift actions on action list (performance hack)\n _k = (a,j)\n if not acthash.has_key(_k):\n actlist.append((a,p,\"shift and go to state \"+str(j)))\n acthash[_k] = 1\n \n r = action.get((st,a),None)\n if r is not None:\n # Whoa have a shift/reduce or shift/shift conflict\n if r > 0:\n if r != j:\n sys.stderr.write(\"Shift/shift conflict in state %d\\n\" % st)\n continue\n elif r < 0:\n # Do a precedence check.\n # - if precedence of reduce rule is higher, we reduce.\n # - if precedence of reduce is same and left assoc, we reduce.\n # - otherwise we shift\n rprec,rlevel = Productions[actionp[st,a].number].prec\n sprec,slevel = Precedence.get(a,('right',0))\n if (slevel > rlevel) or ((slevel == rlevel) and (rprec != 'left')):\n # We decide to shift here... highest precedence to shift\n action[st,a] = j\n actionp[st,a] = p\n if not slevel and not rlevel:\n n_srconflict += 1\n _vfc.write(\"shift/reduce conflict in state %d resolved as shift.\\n\" % st)\n _vf.write(\" ! shift/reduce conflict for %s resolved as shift.\\n\" % a)\n elif (slevel == rlevel) and (rprec == 'nonassoc'):\n action[st,a] = None\n else: \n # Hmmm. Guess we'll keep the reduce\n if not slevel and not rlevel:\n n_srconflict +=1\n _vfc.write(\"shift/reduce conflict in state %d resolved as reduce.\\n\" % st)\n _vf.write(\" ! shift/reduce conflict for %s resolved as reduce.\\n\" % a)\n \n else:\n sys.stderr.write(\"Unknown conflict in state %d\\n\" % st)\n else:\n action[st,a] = j\n actionp[st,a] = p\n \n except StandardError,e:\n raise YaccError, \"Hosed in lalr_parse_table\", e\n\n # Print the actions associated with each terminal\n if yaccdebug:\n for a,p,m in actlist:\n if action.has_key((st,a)):\n if p is actionp[st,a]:\n _vf.write(\" %-15s %s\\n\" % (a,m))\n _vf.write(\"\\n\")\n\n for a,p,m in actlist:\n if action.has_key((st,a)):\n if p is not actionp[st,a]:\n _vf.write(\" ! %-15s [ %s ]\\n\" % (a,m))\n \n # Construct the goto table for this state\n nkeys = { }\n for ii in I:\n for s in ii.usyms:\n if Nonterminals.has_key(s):\n nkeys[s] = None\n\n # Construct the goto table for this state\n for n in nkeys.keys():\n g = lr0_goto(I,n)\n j = cid_hash.get(id(g),-1) \n if j >= 0:\n goto[st,n] = j\n if yaccdebug:\n _vf.write(\" %-30s shift and go to state %d\\n\" % (n,j))\n\n st += 1\n if yaccdebug:\n if n_srconflict == 1:\n sys.stderr.write(\"yacc: %d shift/reduce conflict\\n\" % n_srconflict)\n if n_srconflict > 1:\n sys.stderr.write(\"yacc: %d shift/reduce conflicts\\n\" % n_srconflict) \n if n_rrconflict == 1:\n sys.stderr.write(\"yacc: %d reduce/reduce conflict\\n\" % n_rrconflict)\n if n_rrconflict > 1:\n sys.stderr.write(\"yacc: %d reduce/reduce conflicts\\n\" % n_rrconflict)\n\n \n# -----------------------------------------------------------------------------\n# ==== LR Utility functions ====\n# -----------------------------------------------------------------------------\n\n# -----------------------------------------------------------------------------\n# _lr_write_tables()\n#\n# This function writes the LR parsing tables to a file\n# -----------------------------------------------------------------------------\n\ndef lr_write_tables(modulename=tab_module,outputdir=''):\n filename = os.path.join(outputdir,modulename) + \".py\"\n try:\n f = open(filename,\"w\")\n\n f.write(\"\"\"\n# %s\n# This file is automatically generated. Do not edit.\n\n_lr_method = %s\n\n_lr_signature = %s\n\"\"\" % (filename, repr(_lr_method), repr(Signature.digest())))\n\n # Change smaller to 0 to go back to original tables\n smaller = 1\n \n # Factor out names to try and make smaller\n if smaller:\n items = { }\n \n for k,v in _lr_action.items():\n i = items.get(k[1])\n if not i:\n i = ([],[])\n items[k[1]] = i\n i[0].append(k[0])\n i[1].append(v)\n\n f.write(\"\\n_lr_action_items = {\")\n for k,v in items.items():\n f.write(\"%r:([\" % k)\n for i in v[0]:\n f.write(\"%r,\" % i)\n f.write(\"],[\")\n for i in v[1]:\n f.write(\"%r,\" % i)\n \n f.write(\"]),\")\n f.write(\"}\\n\")\n\n f.write(\"\"\"\n_lr_action = { }\nfor _k, _v in _lr_action_items.items():\n for _x,_y in zip(_v[0],_v[1]):\n _lr_action[(_x,_k)] = _y\ndel _lr_action_items\n\"\"\")\n \n else:\n f.write(\"\\n_lr_action = { \");\n for k,v in _lr_action.items():\n f.write(\"(%r,%r):%r,\" % (k[0],k[1],v))\n f.write(\"}\\n\");\n\n if smaller:\n # Factor out names to try and make smaller\n items = { }\n \n for k,v in _lr_goto.items():\n i = items.get(k[1])\n if not i:\n i = ([],[])\n items[k[1]] = i\n i[0].append(k[0])\n i[1].append(v)\n\n f.write(\"\\n_lr_goto_items = {\")\n for k,v in items.items():\n f.write(\"%r:([\" % k)\n for i in v[0]:\n f.write(\"%r,\" % i)\n f.write(\"],[\")\n for i in v[1]:\n f.write(\"%r,\" % i)\n \n f.write(\"]),\")\n f.write(\"}\\n\")\n\n f.write(\"\"\"\n_lr_goto = { }\nfor _k, _v in _lr_goto_items.items():\n for _x,_y in zip(_v[0],_v[1]):\n _lr_goto[(_x,_k)] = _y\ndel _lr_goto_items\n\"\"\")\n else:\n f.write(\"\\n_lr_goto = { \");\n for k,v in _lr_goto.items():\n f.write(\"(%r,%r):%r,\" % (k[0],k[1],v)) \n f.write(\"}\\n\");\n\n # Write production table\n f.write(\"_lr_productions = [\\n\")\n for p in Productions:\n if p:\n if (p.func):\n f.write(\" (%r,%d,%r,%r,%d),\\n\" % (p.name, p.len, p.func.__name__,p.file,p.line))\n else:\n f.write(\" (%r,%d,None,None,None),\\n\" % (p.name, p.len))\n else:\n f.write(\" None,\\n\")\n f.write(\"]\\n\")\n f.close()\n\n except IOError,e:\n print \"Unable to create '%s'\" % filename\n print e\n return\n\ndef lr_read_tables(module=tab_module,optimize=0):\n global _lr_action, _lr_goto, _lr_productions, _lr_method\n try:\n exec \"import %s as parsetab\" % module\n \n if (optimize) or (Signature.digest() == parsetab._lr_signature):\n _lr_action = parsetab._lr_action\n _lr_goto = parsetab._lr_goto\n _lr_productions = parsetab._lr_productions\n _lr_method = parsetab._lr_method\n return 1\n else:\n return 0\n \n except (ImportError,AttributeError):\n return 0\n\n# -----------------------------------------------------------------------------\n# yacc(module)\n#\n# Build the parser module\n# -----------------------------------------------------------------------------\n\ndef yacc(method=default_lr, debug=yaccdebug, module=None, tabmodule=tab_module, start=None, check_recursion=1, optimize=0,write_tables=1,debugfile=debug_file,outputdir=''):\n global yaccdebug\n yaccdebug = debug\n \n initialize_vars()\n files = { }\n error = 0\n\n # Add starting symbol to signature\n if start:\n Signature.update(start)\n\n # Add parsing method to signature\n Signature.update(method)\n \n # If a \"module\" parameter was supplied, extract its dictionary.\n # Note: a module may in fact be an instance as well.\n \n if module:\n # User supplied a module object.\n if isinstance(module, types.ModuleType):\n ldict = module.__dict__\n elif isinstance(module, types.InstanceType):\n _items = [(k,getattr(module,k)) for k in dir(module)]\n ldict = { }\n for i in _items:\n ldict[i[0]] = i[1]\n else:\n raise ValueError,\"Expected a module\"\n \n else:\n # No module given. We might be able to get information from the caller.\n # Throw an exception and unwind the traceback to get the globals\n \n try:\n raise RuntimeError\n except RuntimeError:\n e,b,t = sys.exc_info()\n f = t.tb_frame\n f = f.f_back # Walk out to our calling function\n ldict = f.f_globals # Grab its globals dictionary\n\n # If running in optimized mode. We're going to\n\n if (optimize and lr_read_tables(tabmodule,1)):\n # Read parse table\n del Productions[:]\n for p in _lr_productions:\n if not p:\n Productions.append(None)\n else:\n m = MiniProduction()\n m.name = p[0]\n m.len = p[1]\n m.file = p[3]\n m.line = p[4]\n if p[2]:\n m.func = ldict[p[2]]\n Productions.append(m)\n \n else:\n # Get the tokens map\n if (module and isinstance(module,types.InstanceType)):\n tokens = getattr(module,\"tokens\",None)\n else:\n tokens = ldict.get(\"tokens\",None)\n \n if not tokens:\n raise YaccError,\"module does not define a list 'tokens'\"\n if not (isinstance(tokens,types.ListType) or isinstance(tokens,types.TupleType)):\n raise YaccError,\"tokens must be a list or tuple.\"\n\n # Check to see if a requires dictionary is defined.\n requires = ldict.get(\"require\",None)\n if requires:\n if not (isinstance(requires,types.DictType)):\n raise YaccError,\"require must be a dictionary.\"\n\n for r,v in requires.items():\n try:\n if not (isinstance(v,types.ListType)):\n raise TypeError\n v1 = [x.split(\".\") for x in v]\n Requires[r] = v1\n except StandardError:\n print \"Invalid specification for rule '%s' in require. Expected a list of strings\" % r \n\n \n # Build the dictionary of terminals. We a record a 0 in the\n # dictionary to track whether or not a terminal is actually\n # used in the grammar\n\n if 'error' in tokens:\n print \"yacc: Illegal token 'error'. Is a reserved word.\"\n raise YaccError,\"Illegal token name\"\n\n for n in tokens:\n if Terminals.has_key(n):\n print \"yacc: Warning. Token '%s' multiply defined.\" % n\n Terminals[n] = [ ]\n\n Terminals['error'] = [ ]\n\n # Get the precedence map (if any)\n prec = ldict.get(\"precedence\",None)\n if prec:\n if not (isinstance(prec,types.ListType) or isinstance(prec,types.TupleType)):\n raise YaccError,\"precedence must be a list or tuple.\"\n add_precedence(prec)\n Signature.update(repr(prec))\n\n for n in tokens:\n if not Precedence.has_key(n):\n Precedence[n] = ('right',0) # Default, right associative, 0 precedence\n\n # Look for error handler\n ef = ldict.get('p_error',None)\n if ef:\n if isinstance(ef,types.FunctionType):\n ismethod = 0\n elif isinstance(ef, types.MethodType):\n ismethod = 1\n else:\n raise YaccError,\"'p_error' defined, but is not a function or method.\" \n eline = ef.func_code.co_firstlineno\n efile = ef.func_code.co_filename\n files[efile] = None\n\n if (ef.func_code.co_argcount != 1+ismethod):\n raise YaccError,\"%s:%d: p_error() requires 1 argument.\" % (efile,eline)\n global Errorfunc\n Errorfunc = ef\n else:\n print \"yacc: Warning. no p_error() function is defined.\"\n \n # Get the list of built-in functions with p_ prefix\n symbols = [ldict[f] for f in ldict.keys()\n if (type(ldict[f]) in (types.FunctionType, types.MethodType) and ldict[f].__name__[:2] == 'p_'\n and ldict[f].__name__ != 'p_error')]\n\n # Check for non-empty symbols\n if len(symbols) == 0:\n raise YaccError,\"no rules of the form p_rulename are defined.\"\n \n # Sort the symbols by line number\n symbols.sort(lambda x,y: cmp(x.func_code.co_firstlineno,y.func_code.co_firstlineno))\n\n # Add all of the symbols to the grammar\n for f in symbols:\n if (add_function(f)) < 0:\n error += 1\n else:\n files[f.func_code.co_filename] = None\n\n # Make a signature of the docstrings\n for f in symbols:\n if f.__doc__:\n Signature.update(f.__doc__)\n \n lr_init_vars()\n\n if error:\n raise YaccError,\"Unable to construct parser.\"\n\n if not lr_read_tables(tabmodule):\n\n # Validate files\n for filename in files.keys():\n if not validate_file(filename):\n error = 1\n\n # Validate dictionary\n validate_dict(ldict)\n\n if start and not Prodnames.has_key(start):\n raise YaccError,\"Bad starting symbol '%s'\" % start\n \n augment_grammar(start) \n error = verify_productions(cycle_check=check_recursion)\n otherfunc = [ldict[f] for f in ldict.keys()\n if (type(f) in (types.FunctionType,types.MethodType) and ldict[f].__name__[:2] != 'p_')]\n\n if error:\n raise YaccError,\"Unable to construct parser.\"\n \n build_lritems()\n compute_first1()\n compute_follow(start)\n \n if method == 'SLR':\n slr_parse_table()\n elif method == 'LALR':\n lalr_parse_table()\n else:\n raise YaccError, \"Unknown parsing method '%s'\" % method\n\n if write_tables:\n lr_write_tables(tabmodule,outputdir) \n \n if yaccdebug:\n try:\n f = open(os.path.join(outputdir,debugfile),\"w\")\n f.write(_vfc.getvalue())\n f.write(\"\\n\\n\")\n f.write(_vf.getvalue())\n f.close()\n except IOError,e:\n print \"yacc: can't create '%s'\" % debugfile,e\n \n # Made it here. Create a parser object and set up its internal state.\n # Set global parse() method to bound method of parser object.\n\n p = Parser(\"xyzzy\")\n p.productions = Productions\n p.errorfunc = Errorfunc\n p.action = _lr_action\n p.goto = _lr_goto\n p.method = _lr_method\n p.require = Requires\n\n global parse\n parse = p.parse\n\n # Clean up all of the globals we created\n if (not optimize):\n yacc_cleanup()\n return p\n\n# yacc_cleanup function. Delete all of the global variables\n# used during table construction\n\ndef yacc_cleanup():\n global _lr_action, _lr_goto, _lr_method, _lr_goto_cache\n del _lr_action, _lr_goto, _lr_method, _lr_goto_cache\n\n global Productions, Prodnames, Prodmap, Terminals \n global Nonterminals, First, Follow, Precedence, LRitems\n global Errorfunc, Signature, Requires\n global Prodempty, TReductions, NTReductions, GotoSetNum, Canonical\n \n del Productions, Prodnames, Prodmap, Terminals\n del Nonterminals, First, Follow, Precedence, LRitems\n del Errorfunc, Signature, Requires\n del Prodempty, TReductions, NTReductions, GotoSetNum, Canonical\n \n global _vf, _vfc\n del _vf, _vfc\n \n \n# Stub that raises an error if parsing is attempted without first calling yacc()\ndef parse(*args,**kwargs):\n raise YaccError, \"yacc: No parser built with yacc()\"\n\n" }, { "path": "src/kenlm/COPYING", "content": " GNU GENERAL PUBLIC LICENSE\n Version 3, 29 June 2007\n\n Copyright (C) 2007 Free Software Foundation, Inc. \n Everyone is permitted to copy and distribute verbatim copies\n of this license document, but changing it is not allowed.\n\n Preamble\n\n The GNU General Public License is a free, copyleft license for\nsoftware and other kinds of works.\n\n The licenses for most software and other practical works are designed\nto take away your freedom to share and change the works. By contrast,\nthe GNU General Public License is intended to guarantee your freedom to\nshare and change all versions of a program--to make sure it remains free\nsoftware for all its users. We, the Free Software Foundation, use the\nGNU General Public License for most of our software; it applies also to\nany other work released this way by its authors. You can apply it to\nyour programs, too.\n\n When we speak of free software, we are referring to freedom, not\nprice. Our General Public Licenses are designed to make sure that you\nhave the freedom to distribute copies of free software (and charge for\nthem if you wish), that you receive source code or can get it if you\nwant it, that you can change the software or use pieces of it in new\nfree programs, and that you know you can do these things.\n\n To protect your rights, we need to prevent others from denying you\nthese rights or asking you to surrender the rights. Therefore, you have\ncertain responsibilities if you distribute copies of the software, or if\nyou modify it: responsibilities to respect the freedom of others.\n\n For example, if you distribute copies of such a program, whether\ngratis or for a fee, you must pass on to the recipients the same\nfreedoms that you received. You must make sure that they, too, receive\nor can get the source code. And you must show them these terms so they\nknow their rights.\n\n Developers that use the GNU GPL protect your rights with two steps:\n(1) assert copyright on the software, and (2) offer you this License\ngiving you legal permission to copy, distribute and/or modify it.\n\n For the developers' and authors' protection, the GPL clearly explains\nthat there is no warranty for this free software. For both users' and\nauthors' sake, the GPL requires that modified versions be marked as\nchanged, so that their problems will not be attributed erroneously to\nauthors of previous versions.\n\n Some devices are designed to deny users access to install or run\nmodified versions of the software inside them, although the manufacturer\ncan do so. This is fundamentally incompatible with the aim of\nprotecting users' freedom to change the software. The systematic\npattern of such abuse occurs in the area of products for individuals to\nuse, which is precisely where it is most unacceptable. Therefore, we\nhave designed this version of the GPL to prohibit the practice for those\nproducts. If such problems arise substantially in other domains, we\nstand ready to extend this provision to those domains in future versions\nof the GPL, as needed to protect the freedom of users.\n\n Finally, every program is threatened constantly by software patents.\nStates should not allow patents to restrict development and use of\nsoftware on general-purpose computers, but in those that do, we wish to\navoid the special danger that patents applied to a free program could\nmake it effectively proprietary. To prevent this, the GPL assures that\npatents cannot be used to render the program non-free.\n\n The precise terms and conditions for copying, distribution and\nmodification follow.\n\n TERMS AND CONDITIONS\n\n 0. Definitions.\n\n \"This License\" refers to version 3 of the GNU General Public License.\n\n \"Copyright\" also means copyright-like laws that apply to other kinds of\nworks, such as semiconductor masks.\n\n \"The Program\" refers to any copyrightable work licensed under this\nLicense. Each licensee is addressed as \"you\". \"Licensees\" and\n\"recipients\" may be individuals or organizations.\n\n To \"modify\" a work means to copy from or adapt all or part of the work\nin a fashion requiring copyright permission, other than the making of an\nexact copy. The resulting work is called a \"modified version\" of the\nearlier work or a work \"based on\" the earlier work.\n\n A \"covered work\" means either the unmodified Program or a work based\non the Program.\n\n To \"propagate\" a work means to do anything with it that, without\npermission, would make you directly or secondarily liable for\ninfringement under applicable copyright law, except executing it on a\ncomputer or modifying a private copy. Propagation includes copying,\ndistribution (with or without modification), making available to the\npublic, and in some countries other activities as well.\n\n To \"convey\" a work means any kind of propagation that enables other\nparties to make or receive copies. Mere interaction with a user through\na computer network, with no transfer of a copy, is not conveying.\n\n An interactive user interface displays \"Appropriate Legal Notices\"\nto the extent that it includes a convenient and prominently visible\nfeature that (1) displays an appropriate copyright notice, and (2)\ntells the user that there is no warranty for the work (except to the\nextent that warranties are provided), that licensees may convey the\nwork under this License, and how to view a copy of this License. If\nthe interface presents a list of user commands or options, such as a\nmenu, a prominent item in the list meets this criterion.\n\n 1. Source Code.\n\n The \"source code\" for a work means the preferred form of the work\nfor making modifications to it. \"Object code\" means any non-source\nform of a work.\n\n A \"Standard Interface\" means an interface that either is an official\nstandard defined by a recognized standards body, or, in the case of\ninterfaces specified for a particular programming language, one that\nis widely used among developers working in that language.\n\n The \"System Libraries\" of an executable work include anything, other\nthan the work as a whole, that (a) is included in the normal form of\npackaging a Major Component, but which is not part of that Major\nComponent, and (b) serves only to enable use of the work with that\nMajor Component, or to implement a Standard Interface for which an\nimplementation is available to the public in source code form. A\n\"Major Component\", in this context, means a major essential component\n(kernel, window system, and so on) of the specific operating system\n(if any) on which the executable work runs, or a compiler used to\nproduce the work, or an object code interpreter used to run it.\n\n The \"Corresponding Source\" for a work in object code form means all\nthe source code needed to generate, install, and (for an executable\nwork) run the object code and to modify the work, including scripts to\ncontrol those activities. However, it does not include the work's\nSystem Libraries, or general-purpose tools or generally available free\nprograms which are used unmodified in performing those activities but\nwhich are not part of the work. For example, Corresponding Source\nincludes interface definition files associated with source files for\nthe work, and the source code for shared libraries and dynamically\nlinked subprograms that the work is specifically designed to require,\nsuch as by intimate data communication or control flow between those\nsubprograms and other parts of the work.\n\n The Corresponding Source need not include anything that users\ncan regenerate automatically from other parts of the Corresponding\nSource.\n\n The Corresponding Source for a work in source code form is that\nsame work.\n\n 2. Basic Permissions.\n\n All rights granted under this License are granted for the term of\ncopyright on the Program, and are irrevocable provided the stated\nconditions are met. This License explicitly affirms your unlimited\npermission to run the unmodified Program. The output from running a\ncovered work is covered by this License only if the output, given its\ncontent, constitutes a covered work. This License acknowledges your\nrights of fair use or other equivalent, as provided by copyright law.\n\n You may make, run and propagate covered works that you do not\nconvey, without conditions so long as your license otherwise remains\nin force. You may convey covered works to others for the sole purpose\nof having them make modifications exclusively for you, or provide you\nwith facilities for running those works, provided that you comply with\nthe terms of this License in conveying all material for which you do\nnot control copyright. Those thus making or running the covered works\nfor you must do so exclusively on your behalf, under your direction\nand control, on terms that prohibit them from making any copies of\nyour copyrighted material outside their relationship with you.\n\n Conveying under any other circumstances is permitted solely under\nthe conditions stated below. Sublicensing is not allowed; section 10\nmakes it unnecessary.\n\n 3. Protecting Users' Legal Rights From Anti-Circumvention Law.\n\n No covered work shall be deemed part of an effective technological\nmeasure under any applicable law fulfilling obligations under article\n11 of the WIPO copyright treaty adopted on 20 December 1996, or\nsimilar laws prohibiting or restricting circumvention of such\nmeasures.\n\n When you convey a covered work, you waive any legal power to forbid\ncircumvention of technological measures to the extent such circumvention\nis effected by exercising rights under this License with respect to\nthe covered work, and you disclaim any intention to limit operation or\nmodification of the work as a means of enforcing, against the work's\nusers, your or third parties' legal rights to forbid circumvention of\ntechnological measures.\n\n 4. Conveying Verbatim Copies.\n\n You may convey verbatim copies of the Program's source code as you\nreceive it, in any medium, provided that you conspicuously and\nappropriately publish on each copy an appropriate copyright notice;\nkeep intact all notices stating that this License and any\nnon-permissive terms added in accord with section 7 apply to the code;\nkeep intact all notices of the absence of any warranty; and give all\nrecipients a copy of this License along with the Program.\n\n You may charge any price or no price for each copy that you convey,\nand you may offer support or warranty protection for a fee.\n\n 5. Conveying Modified Source Versions.\n\n You may convey a work based on the Program, or the modifications to\nproduce it from the Program, in the form of source code under the\nterms of section 4, provided that you also meet all of these conditions:\n\n a) The work must carry prominent notices stating that you modified\n it, and giving a relevant date.\n\n b) The work must carry prominent notices stating that it is\n released under this License and any conditions added under section\n 7. This requirement modifies the requirement in section 4 to\n \"keep intact all notices\".\n\n c) You must license the entire work, as a whole, under this\n License to anyone who comes into possession of a copy. This\n License will therefore apply, along with any applicable section 7\n additional terms, to the whole of the work, and all its parts,\n regardless of how they are packaged. This License gives no\n permission to license the work in any other way, but it does not\n invalidate such permission if you have separately received it.\n\n d) If the work has interactive user interfaces, each must display\n Appropriate Legal Notices; however, if the Program has interactive\n interfaces that do not display Appropriate Legal Notices, your\n work need not make them do so.\n\n A compilation of a covered work with other separate and independent\nworks, which are not by their nature extensions of the covered work,\nand which are not combined with it such as to form a larger program,\nin or on a volume of a storage or distribution medium, is called an\n\"aggregate\" if the compilation and its resulting copyright are not\nused to limit the access or legal rights of the compilation's users\nbeyond what the individual works permit. Inclusion of a covered work\nin an aggregate does not cause this License to apply to the other\nparts of the aggregate.\n\n 6. Conveying Non-Source Forms.\n\n You may convey a covered work in object code form under the terms\nof sections 4 and 5, provided that you also convey the\nmachine-readable Corresponding Source under the terms of this License,\nin one of these ways:\n\n a) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by the\n Corresponding Source fixed on a durable physical medium\n customarily used for software interchange.\n\n b) Convey the object code in, or embodied in, a physical product\n (including a physical distribution medium), accompanied by a\n written offer, valid for at least three years and valid for as\n long as you offer spare parts or customer support for that product\n model, to give anyone who possesses the object code either (1) a\n copy of the Corresponding Source for all the software in the\n product that is covered by this License, on a durable physical\n medium customarily used for software interchange, for a price no\n more than your reasonable cost of physically performing this\n conveying of source, or (2) access to copy the\n Corresponding Source from a network server at no charge.\n\n c) Convey individual copies of the object code with a copy of the\n written offer to provide the Corresponding Source. This\n alternative is allowed only occasionally and noncommercially, and\n only if you received the object code with such an offer, in accord\n with subsection 6b.\n\n d) Convey the object code by offering access from a designated\n place (gratis or for a charge), and offer equivalent access to the\n Corresponding Source in the same way through the same place at no\n further charge. You need not require recipients to copy the\n Corresponding Source along with the object code. If the place to\n copy the object code is a network server, the Corresponding Source\n may be on a different server (operated by you or a third party)\n that supports equivalent copying facilities, provided you maintain\n clear directions next to the object code saying where to find the\n Corresponding Source. Regardless of what server hosts the\n Corresponding Source, you remain obligated to ensure that it is\n available for as long as needed to satisfy these requirements.\n\n e) Convey the object code using peer-to-peer transmission, provided\n you inform other peers where the object code and Corresponding\n Source of the work are being offered to the general public at no\n charge under subsection 6d.\n\n A separable portion of the object code, whose source code is excluded\nfrom the Corresponding Source as a System Library, need not be\nincluded in conveying the object code work.\n\n A \"User Product\" is either (1) a \"consumer product\", which means any\ntangible personal property which is normally used for personal, family,\nor household purposes, or (2) anything designed or sold for incorporation\ninto a dwelling. In determining whether a product is a consumer product,\ndoubtful cases shall be resolved in favor of coverage. For a particular\nproduct received by a particular user, \"normally used\" refers to a\ntypical or common use of that class of product, regardless of the status\nof the particular user or of the way in which the particular user\nactually uses, or expects or is expected to use, the product. A product\nis a consumer product regardless of whether the product has substantial\ncommercial, industrial or non-consumer uses, unless such uses represent\nthe only significant mode of use of the product.\n\n \"Installation Information\" for a User Product means any methods,\nprocedures, authorization keys, or other information required to install\nand execute modified versions of a covered work in that User Product from\na modified version of its Corresponding Source. The information must\nsuffice to ensure that the continued functioning of the modified object\ncode is in no case prevented or interfered with solely because\nmodification has been made.\n\n If you convey an object code work under this section in, or with, or\nspecifically for use in, a User Product, and the conveying occurs as\npart of a transaction in which the right of possession and use of the\nUser Product is transferred to the recipient in perpetuity or for a\nfixed term (regardless of how the transaction is characterized), the\nCorresponding Source conveyed under this section must be accompanied\nby the Installation Information. But this requirement does not apply\nif neither you nor any third party retains the ability to install\nmodified object code on the User Product (for example, the work has\nbeen installed in ROM).\n\n The requirement to provide Installation Information does not include a\nrequirement to continue to provide support service, warranty, or updates\nfor a work that has been modified or installed by the recipient, or for\nthe User Product in which it has been modified or installed. Access to a\nnetwork may be denied when the modification itself materially and\nadversely affects the operation of the network or violates the rules and\nprotocols for communication across the network.\n\n Corresponding Source conveyed, and Installation Information provided,\nin accord with this section must be in a format that is publicly\ndocumented (and with an implementation available to the public in\nsource code form), and must require no special password or key for\nunpacking, reading or copying.\n\n 7. Additional Terms.\n\n \"Additional permissions\" are terms that supplement the terms of this\nLicense by making exceptions from one or more of its conditions.\nAdditional permissions that are applicable to the entire Program shall\nbe treated as though they were included in this License, to the extent\nthat they are valid under applicable law. If additional permissions\napply only to part of the Program, that part may be used separately\nunder those permissions, but the entire Program remains governed by\nthis License without regard to the additional permissions.\n\n When you convey a copy of a covered work, you may at your option\nremove any additional permissions from that copy, or from any part of\nit. (Additional permissions may be written to require their own\nremoval in certain cases when you modify the work.) You may place\nadditional permissions on material, added by you to a covered work,\nfor which you have or can give appropriate copyright permission.\n\n Notwithstanding any other provision of this License, for material you\nadd to a covered work, you may (if authorized by the copyright holders of\nthat material) supplement the terms of this License with terms:\n\n a) Disclaiming warranty or limiting liability differently from the\n terms of sections 15 and 16 of this License; or\n\n b) Requiring preservation of specified reasonable legal notices or\n author attributions in that material or in the Appropriate Legal\n Notices displayed by works containing it; or\n\n c) Prohibiting misrepresentation of the origin of that material, or\n requiring that modified versions of such material be marked in\n reasonable ways as different from the original version; or\n\n d) Limiting the use for publicity purposes of names of licensors or\n authors of the material; or\n\n e) Declining to grant rights under trademark law for use of some\n trade names, trademarks, or service marks; or\n\n f) Requiring indemnification of licensors and authors of that\n material by anyone who conveys the material (or modified versions of\n it) with contractual assumptions of liability to the recipient, for\n any liability that these contractual assumptions directly impose on\n those licensors and authors.\n\n All other non-permissive additional terms are considered \"further\nrestrictions\" within the meaning of section 10. If the Program as you\nreceived it, or any part of it, contains a notice stating that it is\ngoverned by this License along with a term that is a further\nrestriction, you may remove that term. If a license document contains\na further restriction but permits relicensing or conveying under this\nLicense, you may add to a covered work material governed by the terms\nof that license document, provided that the further restriction does\nnot survive such relicensing or conveying.\n\n If you add terms to a covered work in accord with this section, you\nmust place, in the relevant source files, a statement of the\nadditional terms that apply to those files, or a notice indicating\nwhere to find the applicable terms.\n\n Additional terms, permissive or non-permissive, may be stated in the\nform of a separately written license, or stated as exceptions;\nthe above requirements apply either way.\n\n 8. Termination.\n\n You may not propagate or modify a covered work except as expressly\nprovided under this License. Any attempt otherwise to propagate or\nmodify it is void, and will automatically terminate your rights under\nthis License (including any patent licenses granted under the third\nparagraph of section 11).\n\n However, if you cease all violation of this License, then your\nlicense from a particular copyright holder is reinstated (a)\nprovisionally, unless and until the copyright holder explicitly and\nfinally terminates your license, and (b) permanently, if the copyright\nholder fails to notify you of the violation by some reasonable means\nprior to 60 days after the cessation.\n\n Moreover, your license from a particular copyright holder is\nreinstated permanently if the copyright holder notifies you of the\nviolation by some reasonable means, this is the first time you have\nreceived notice of violation of this License (for any work) from that\ncopyright holder, and you cure the violation prior to 30 days after\nyour receipt of the notice.\n\n Termination of your rights under this section does not terminate the\nlicenses of parties who have received copies or rights from you under\nthis License. If your rights have been terminated and not permanently\nreinstated, you do not qualify to receive new licenses for the same\nmaterial under section 10.\n\n 9. Acceptance Not Required for Having Copies.\n\n You are not required to accept this License in order to receive or\nrun a copy of the Program. Ancillary propagation of a covered work\noccurring solely as a consequence of using peer-to-peer transmission\nto receive a copy likewise does not require acceptance. However,\nnothing other than this License grants you permission to propagate or\nmodify any covered work. These actions infringe copyright if you do\nnot accept this License. Therefore, by modifying or propagating a\ncovered work, you indicate your acceptance of this License to do so.\n\n 10. Automatic Licensing of Downstream Recipients.\n\n Each time you convey a covered work, the recipient automatically\nreceives a license from the original licensors, to run, modify and\npropagate that work, subject to this License. You are not responsible\nfor enforcing compliance by third parties with this License.\n\n An \"entity transaction\" is a transaction transferring control of an\norganization, or substantially all assets of one, or subdividing an\norganization, or merging organizations. If propagation of a covered\nwork results from an entity transaction, each party to that\ntransaction who receives a copy of the work also receives whatever\nlicenses to the work the party's predecessor in interest had or could\ngive under the previous paragraph, plus a right to possession of the\nCorresponding Source of the work from the predecessor in interest, if\nthe predecessor has it or can get it with reasonable efforts.\n\n You may not impose any further restrictions on the exercise of the\nrights granted or affirmed under this License. For example, you may\nnot impose a license fee, royalty, or other charge for exercise of\nrights granted under this License, and you may not initiate litigation\n(including a cross-claim or counterclaim in a lawsuit) alleging that\nany patent claim is infringed by making, using, selling, offering for\nsale, or importing the Program or any portion of it.\n\n 11. Patents.\n\n A \"contributor\" is a copyright holder who authorizes use under this\nLicense of the Program or a work on which the Program is based. The\nwork thus licensed is called the contributor's \"contributor version\".\n\n A contributor's \"essential patent claims\" are all patent claims\nowned or controlled by the contributor, whether already acquired or\nhereafter acquired, that would be infringed by some manner, permitted\nby this License, of making, using, or selling its contributor version,\nbut do not include claims that would be infringed only as a\nconsequence of further modification of the contributor version. For\npurposes of this definition, \"control\" includes the right to grant\npatent sublicenses in a manner consistent with the requirements of\nthis License.\n\n Each contributor grants you a non-exclusive, worldwide, royalty-free\npatent license under the contributor's essential patent claims, to\nmake, use, sell, offer for sale, import and otherwise run, modify and\npropagate the contents of its contributor version.\n\n In the following three paragraphs, a \"patent license\" is any express\nagreement or commitment, however denominated, not to enforce a patent\n(such as an express permission to practice a patent or covenant not to\nsue for patent infringement). To \"grant\" such a patent license to a\nparty means to make such an agreement or commitment not to enforce a\npatent against the party.\n\n If you convey a covered work, knowingly relying on a patent license,\nand the Corresponding Source of the work is not available for anyone\nto copy, free of charge and under the terms of this License, through a\npublicly available network server or other readily accessible means,\nthen you must either (1) cause the Corresponding Source to be so\navailable, or (2) arrange to deprive yourself of the benefit of the\npatent license for this particular work, or (3) arrange, in a manner\nconsistent with the requirements of this License, to extend the patent\nlicense to downstream recipients. \"Knowingly relying\" means you have\nactual knowledge that, but for the patent license, your conveying the\ncovered work in a country, or your recipient's use of the covered work\nin a country, would infringe one or more identifiable patents in that\ncountry that you have reason to believe are valid.\n\n If, pursuant to or in connection with a single transaction or\narrangement, you convey, or propagate by procuring conveyance of, a\ncovered work, and grant a patent license to some of the parties\nreceiving the covered work authorizing them to use, propagate, modify\nor convey a specific copy of the covered work, then the patent license\nyou grant is automatically extended to all recipients of the covered\nwork and works based on it.\n\n A patent license is \"discriminatory\" if it does not include within\nthe scope of its coverage, prohibits the exercise of, or is\nconditioned on the non-exercise of one or more of the rights that are\nspecifically granted under this License. You may not convey a covered\nwork if you are a party to an arrangement with a third party that is\nin the business of distributing software, under which you make payment\nto the third party based on the extent of your activity of conveying\nthe work, and under which the third party grants, to any of the\nparties who would receive the covered work from you, a discriminatory\npatent license (a) in connection with copies of the covered work\nconveyed by you (or copies made from those copies), or (b) primarily\nfor and in connection with specific products or compilations that\ncontain the covered work, unless you entered into that arrangement,\nor that patent license was granted, prior to 28 March 2007.\n\n Nothing in this License shall be construed as excluding or limiting\nany implied license or other defenses to infringement that may\notherwise be available to you under applicable patent law.\n\n 12. No Surrender of Others' Freedom.\n\n If conditions are imposed on you (whether by court order, agreement or\notherwise) that contradict the conditions of this License, they do not\nexcuse you from the conditions of this License. If you cannot convey a\ncovered work so as to satisfy simultaneously your obligations under this\nLicense and any other pertinent obligations, then as a consequence you may\nnot convey it at all. For example, if you agree to terms that obligate you\nto collect a royalty for further conveying from those to whom you convey\nthe Program, the only way you could satisfy both those terms and this\nLicense would be to refrain entirely from conveying the Program.\n\n 13. Use with the GNU Affero General Public License.\n\n Notwithstanding any other provision of this License, you have\npermission to link or combine any covered work with a work licensed\nunder version 3 of the GNU Affero General Public License into a single\ncombined work, and to convey the resulting work. The terms of this\nLicense will continue to apply to the part which is the covered work,\nbut the special requirements of the GNU Affero General Public License,\nsection 13, concerning interaction through a network will apply to the\ncombination as such.\n\n 14. Revised Versions of this License.\n\n The Free Software Foundation may publish revised and/or new versions of\nthe GNU General Public License from time to time. Such new versions will\nbe similar in spirit to the present version, but may differ in detail to\naddress new problems or concerns.\n\n Each version is given a distinguishing version number. If the\nProgram specifies that a certain numbered version of the GNU General\nPublic License \"or any later version\" applies to it, you have the\noption of following the terms and conditions either of that numbered\nversion or of any later version published by the Free Software\nFoundation. If the Program does not specify a version number of the\nGNU General Public License, you may choose any version ever published\nby the Free Software Foundation.\n\n If the Program specifies that a proxy can decide which future\nversions of the GNU General Public License can be used, that proxy's\npublic statement of acceptance of a version permanently authorizes you\nto choose that version for the Program.\n\n Later license versions may give you additional or different\npermissions. However, no additional obligations are imposed on any\nauthor or copyright holder as a result of your choosing to follow a\nlater version.\n\n 15. Disclaimer of Warranty.\n\n THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY\nAPPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT\nHOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM \"AS IS\" WITHOUT WARRANTY\nOF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,\nTHE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\nPURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM\nIS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF\nALL NECESSARY SERVICING, REPAIR OR CORRECTION.\n\n 16. Limitation of Liability.\n\n IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING\nWILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS\nTHE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY\nGENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE\nUSE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF\nDATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD\nPARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),\nEVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF\nSUCH DAMAGES.\n\n 17. Interpretation of Sections 15 and 16.\n\n If the disclaimer of warranty and limitation of liability provided\nabove cannot be given local legal effect according to their terms,\nreviewing courts shall apply local law that most closely approximates\nan absolute waiver of all civil liability in connection with the\nProgram, unless a warranty or assumption of liability accompanies a\ncopy of the Program in return for a fee.\n\n END OF TERMS AND CONDITIONS\n\n How to Apply These Terms to Your New Programs\n\n If you develop a new program, and you want it to be of the greatest\npossible use to the public, the best way to achieve this is to make it\nfree software which everyone can redistribute and change under these terms.\n\n To do so, attach the following notices to the program. It is safest\nto attach them to the start of each source file to most effectively\nstate the exclusion of warranty; and each file should have at least\nthe \"copyright\" line and a pointer to where the full notice is found.\n\n \n Copyright (C) \n\n This program is free software: you can redistribute it and/or modify\n it under the terms of the GNU General Public License as published by\n the Free Software Foundation, either version 3 of the License, or\n (at your option) any later version.\n\n This program is distributed in the hope that it will be useful,\n but WITHOUT ANY WARRANTY; without even the implied warranty of\n MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n GNU General Public License for more details.\n\n You should have received a copy of the GNU General Public License\n along with this program. If not, see .\n\nAlso add information on how to contact you by electronic and paper mail.\n\n If the program does terminal interaction, make it output a short\nnotice like this when it starts in an interactive mode:\n\n Copyright (C) \n This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.\n This is free software, and you are welcome to redistribute it\n under certain conditions; type `show c' for details.\n\nThe hypothetical commands `show w' and `show c' should show the appropriate\nparts of the General Public License. Of course, your program's commands\nmight be different; for a GUI interface, you would use an \"about box\".\n\n You should also get your employer (if you work as a programmer) or school,\nif any, to sign a \"copyright disclaimer\" for the program, if necessary.\nFor more information on this, and how to apply and follow the GNU GPL, see\n.\n\n The GNU General Public License does not permit incorporating your program\ninto proprietary programs. If your program is a subroutine library, you\nmay consider it more useful to permit linking proprietary applications with\nthe library. If this is what you want to do, use the GNU Lesser General\nPublic License instead of this License. But first, please read\n.\n" }, { "path": "src/kenlm/COPYING.LESSER", "content": "\t\t GNU LESSER GENERAL PUBLIC LICENSE\n Version 3, 29 June 2007\n\n Copyright (C) 2007 Free Software Foundation, Inc. \n Everyone is permitted to copy and distribute verbatim copies\n of this license document, but changing it is not allowed.\n\n\n This version of the GNU Lesser General Public License incorporates\nthe terms and conditions of version 3 of the GNU General Public\nLicense, supplemented by the additional permissions listed below.\n\n 0. Additional Definitions.\n\n As used herein, \"this License\" refers to version 3 of the GNU Lesser\nGeneral Public License, and the \"GNU GPL\" refers to version 3 of the GNU\nGeneral Public License.\n\n \"The Library\" refers to a covered work governed by this License,\nother than an Application or a Combined Work as defined below.\n\n An \"Application\" is any work that makes use of an interface provided\nby the Library, but which is not otherwise based on the Library.\nDefining a subclass of a class defined by the Library is deemed a mode\nof using an interface provided by the Library.\n\n A \"Combined Work\" is a work produced by combining or linking an\nApplication with the Library. The particular version of the Library\nwith which the Combined Work was made is also called the \"Linked\nVersion\".\n\n The \"Minimal Corresponding Source\" for a Combined Work means the\nCorresponding Source for the Combined Work, excluding any source code\nfor portions of the Combined Work that, considered in isolation, are\nbased on the Application, and not on the Linked Version.\n\n The \"Corresponding Application Code\" for a Combined Work means the\nobject code and/or source code for the Application, including any data\nand utility programs needed for reproducing the Combined Work from the\nApplication, but excluding the System Libraries of the Combined Work.\n\n 1. Exception to Section 3 of the GNU GPL.\n\n You may convey a covered work under sections 3 and 4 of this License\nwithout being bound by section 3 of the GNU GPL.\n\n 2. Conveying Modified Versions.\n\n If you modify a copy of the Library, and, in your modifications, a\nfacility refers to a function or data to be supplied by an Application\nthat uses the facility (other than as an argument passed when the\nfacility is invoked), then you may convey a copy of the modified\nversion:\n\n a) under this License, provided that you make a good faith effort to\n ensure that, in the event an Application does not supply the\n function or data, the facility still operates, and performs\n whatever part of its purpose remains meaningful, or\n\n b) under the GNU GPL, with none of the additional permissions of\n this License applicable to that copy.\n\n 3. Object Code Incorporating Material from Library Header Files.\n\n The object code form of an Application may incorporate material from\na header file that is part of the Library. You may convey such object\ncode under terms of your choice, provided that, if the incorporated\nmaterial is not limited to numerical parameters, data structure\nlayouts and accessors, or small macros, inline functions and templates\n(ten or fewer lines in length), you do both of the following:\n\n a) Give prominent notice with each copy of the object code that the\n Library is used in it and that the Library and its use are\n covered by this License.\n\n b) Accompany the object code with a copy of the GNU GPL and this license\n document.\n\n 4. Combined Works.\n\n You may convey a Combined Work under terms of your choice that,\ntaken together, effectively do not restrict modification of the\nportions of the Library contained in the Combined Work and reverse\nengineering for debugging such modifications, if you also do each of\nthe following:\n\n a) Give prominent notice with each copy of the Combined Work that\n the Library is used in it and that the Library and its use are\n covered by this License.\n\n b) Accompany the Combined Work with a copy of the GNU GPL and this license\n document.\n\n c) For a Combined Work that displays copyright notices during\n execution, include the copyright notice for the Library among\n these notices, as well as a reference directing the user to the\n copies of the GNU GPL and this license document.\n\n d) Do one of the following:\n\n 0) Convey the Minimal Corresponding Source under the terms of this\n License, and the Corresponding Application Code in a form\n suitable for, and under terms that permit, the user to\n recombine or relink the Application with a modified version of\n the Linked Version to produce a modified Combined Work, in the\n manner specified by section 6 of the GNU GPL for conveying\n Corresponding Source.\n\n 1) Use a suitable shared library mechanism for linking with the\n Library. A suitable mechanism is one that (a) uses at run time\n a copy of the Library already present on the user's computer\n system, and (b) will operate properly with a modified version\n of the Library that is interface-compatible with the Linked\n Version.\n\n e) Provide Installation Information, but only if you would otherwise\n be required to provide such information under section 6 of the\n GNU GPL, and only to the extent that such information is\n necessary to install and execute a modified version of the\n Combined Work produced by recombining or relinking the\n Application with a modified version of the Linked Version. (If\n you use option 4d0, the Installation Information must accompany\n the Minimal Corresponding Source and Corresponding Application\n Code. If you use option 4d1, you must provide the Installation\n Information in the manner specified by section 6 of the GNU GPL\n for conveying Corresponding Source.)\n\n 5. Combined Libraries.\n\n You may place library facilities that are a work based on the\nLibrary side by side in a single library together with other library\nfacilities that are not Applications and are not covered by this\nLicense, and convey such a combined library under terms of your\nchoice, if you do both of the following:\n\n a) Accompany the combined library with a copy of the same work based\n on the Library, uncombined with any other library facilities,\n conveyed under the terms of this License.\n\n b) Give prominent notice with the combined library that part of it\n is a work based on the Library, and explaining where to find the\n accompanying uncombined form of the same work.\n\n 6. Revised Versions of the GNU Lesser General Public License.\n\n The Free Software Foundation may publish revised and/or new versions\nof the GNU Lesser General Public License from time to time. Such new\nversions will be similar in spirit to the present version, but may\ndiffer in detail to address new problems or concerns.\n\n Each version is given a distinguishing version number. If the\nLibrary as you received it specifies that a certain numbered version\nof the GNU Lesser General Public License \"or any later version\"\napplies to it, you have the option of following the terms and\nconditions either of that published version or of any later version\npublished by the Free Software Foundation. If the Library as you\nreceived it does not specify a version number of the GNU Lesser\nGeneral Public License, you may choose any version of the GNU Lesser\nGeneral Public License ever published by the Free Software Foundation.\n\n If the Library as you received it specifies that a proxy can decide\nwhether future versions of the GNU Lesser General Public License shall\napply, that proxy's public statement of acceptance of any version is\npermanent authorization for you to choose that version for the\nLibrary.\n" }, { "path": "src/kenlm/LICENSE", "content": "Most of the code here is licensed under the LGPL. There are exceptions which have their own licenses, listed below. See comments in those files for more details. \n\nutil/murmur_hash.cc is under the MIT license. \nutil/string_piece.hh and util/string_piece.cc are Google code and contains its own license. \n\nFor the rest:\n\n Avenue code is free software: you can redistribute it and/or modify\n it under the terms of the GNU Lesser General Public License as published\n by the Free Software Foundation, either version 3 of the License, or\n (at your option) any later version.\n\n Avenue code is distributed in the hope that it will be useful,\n but WITHOUT ANY WARRANTY; without even the implied warranty of\n MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n GNU Lesser General Public License for more details.\n\n You should have received a copy of the GNU Lesser General Public License\n along with Avenue code. If not, see .\n" }, { "path": "src/kenlm/README", "content": "Language model inference code by Kenneth Heafield \nThe official website is http://kheafield.com/code/kenlm/. If you're a decoder developer, please download the latest version from there instead of copying from another decoder. \n\nTwo data structures are supported: probing and trie. Probing is a probing hash table with keys that ere 64-bit hashes of n-grams and floats as values. Trie is a fairly standard trie but with bit-level packing so it uses the minimum number of bits to store word indices and pointers. The trie node entries are sorted by word index. Probing is the fastest and uses the most memory. Trie uses the least memory and a bit slower. \n\nWith trie, resident memory is 58% of IRST's smallest version and 21% of SRI's compact version. Simultaneously, trie CPU's use is 81% of IRST's fastest version and 84% of SRI's fast version. KenLM's probing hash table implementation goes even faster at the expense of using more memory. See http://kheafield.com/code/kenlm/benchmark/. \n\nBinary format via mmap is supported. Run ./build_binary to make one then pass the binary file name to the appropriate Model constructor. \n\nCurrently, it assumes POSIX APIs for errno, sterror_r, open, close, mmap, munmap, ftruncate, fstat, lseek, and read. This is tested on Linux and the non-UNIX Mac OS X. I welcome submissions porting (via #ifdef) to other systems (e.g. Windows) but proudly have no machine on which to test it. \n\nA brief note to Mac OS X users: your gcc is too old to recognize the pack pragma. The warning effectively means that, on 64-bit machines, the model will use 16 bytes instead of 12 bytes per n-gram of maximum order (those of lower order are already 16 bytes) in the probing and sorted models. The trie is not impacted by this. \n\n\nFOR DEVELOPERS\nCopy the code and distribute with your decoder. \n- It does not depend on Boost or ICU. If you use ICU, define HAVE_ICU in util/have.hh (uncomment the line) to avoid a name conflict. Defining HAVE_BOOST will let you hash StringPiece. \n\n- Most people have zlib. If you don't want to depend on that, comment out #define HAVE_ZLIB in util/have.hh. This will disable loading gzipped ARPA files. \n\n- Look at compile.sh and reimplement using your build system. \n\n- Use either the interface in lm/model.hh or lm/virtual_interface.hh. Interface documentation is in comments of lm/virtual_interface.hh (including for lm/model.hh). \n\n- See lm/config.hh for tuning options. \n\n- I recommend copying the code and distributing it with your decoder. However, please send improvements to me so that they can be integrated into the package. \n\nAlso included: A wrapper to SRI with the same interface. \n\nThe name was Hieu Hoang's idea, not mine. \n" }, { "path": "src/kenlm/build_jnilib.sh", "content": "#!/bin/bash\n# based on compile_query_only.sh\n\necho Compiling source files\n\nrm {lm,util}/*.o 2>/dev/null\nset -e\n\nCXXFLAGS=\"-I. -O3 -DNDEBUG -DKENLM_MAX_ORDER=6 -fPIC $CXXFLAGS\"\n\n#Grab all cc files in these directories except those ending in test.cc or main.cc\nobjects=\"\"\nfor i in util/double-conversion/*.cc util/*.cc lm/*.cc; do\n if [ \"${i%test.cc}\" == \"$i\" ] && [ \"${i%main.cc}\" == \"$i\" ]; then\n g++ $CXXFLAGS -c $i -o ${i%.cc}.o\n objects=\"$objects ${i%.cc}.o\"\n fi\ndone\n\n\n\necho Compiling binaries into bin\n\nmkdir -p bin\ng++ $CXXFLAGS lm/build_binary_main.cc $objects -o bin/build_binary\ng++ $CXXFLAGS lm/query_main.cc $objects -o bin/query\ng++ $CXXFLAGS lm/kenlm_max_order_main.cc -o bin/kenlm_max_order\n\n\n\necho Compiling JNI library and moving it to openccg/lib\n\ng++ $CXXFLAGS jni/wrap.cc -I $JAVA_HOME/include -I $JAVA_HOME/include/linux $objects -shared -Wl,-soname,libken.so -o libken.so -lz -Wno-deprecated -pthread\nmv libken.so ../../lib/.\n" }, { "path": "src/kenlm/clean_query_only.sh", "content": "#!/bin/bash\nrm -rf {lm,util,util/double-conversion}/*.o bin/{query,kenlm_max_order,build_binary}\n" }, { "path": "src/kenlm/jni/wrap.cc", "content": "#include \"lm/enumerate_vocab.hh\"\n#include \"lm/model.hh\"\n#include \"util/murmur_hash.hh\"\n\n#include \n\n#include \n#include \n#include \n#include \n\n// Grr. Everybody's compiler is slightly different and I'm trying to not depend on boost. \n#include \n\n// This is needed to compile on OS X Lion / gcc 4.2.1\nnamespace __gnu_cxx {\ntemplate<>\nstruct hash\n{\n\tsize_t\n\toperator()(unsigned long long int __x) const\n\t\t\t{\n\t\treturn __x;\n\t}\n};\n}\n\n// Verify that jint and lm::ngram::WordIndex are the same size. If this breaks\n// for you, there's a need to revise probString.\nnamespace {\n\ntemplate struct StaticCheck {};\n\ntemplate<> struct StaticCheck {\n\ttypedef bool StaticAssertionPassed;\n};\n\ntypedef StaticCheck::StaticAssertionPassed FloatSize;\n\n// Vocab ids above what the vocabulary knows about are unknown and should\n// be mapped to that.\nvoid MapArray(const std::vector& map, jint *begin, jint *end) {\n\tfor (jint *i = begin; i < end; ++i) {\n\t\t*i = map[*i];\n\t}\n}\n\nchar *PieceCopy(const StringPiece &str) {\n\tchar *ret = (char*) malloc(str.size() + 1);\n\tmemcpy(ret, str.data(), str.size());\n\tret[str.size()] = 0;\n\treturn ret;\n}\n\n// Rather than handle several different instantiations over JNI, we'll just\n// do virtual calls C++-side.\nclass VirtualBase {\npublic:\n\tvirtual ~VirtualBase() {\n\t}\n\n\tvirtual float Prob(jint *begin, jint *end) const = 0;\n\n\tvirtual float ProbString(jint * const begin, jint * const end,\n\t\t\tjint start) const = 0;\n\n\tvirtual uint8_t Order() const = 0;\n\n\tvirtual bool RegisterWord(const StringPiece& word, const int wd_id) = 0;\n\nprotected:\n\tVirtualBase() {\n\t}\n\nprivate:\n};\n\ntemplate class VirtualImpl: public VirtualBase {\npublic:\n\tVirtualImpl(const char *name, float fake_oov_cost) :\n\t\t\tm_(name), fake_oov_cost_(fake_oov_cost) {\n\t\t// Insert unknown id mapping.\n\t\tmap_.push_back(0);\n\t}\n\n\t~VirtualImpl() {\n\t}\n\n\tfloat Prob(jint * const begin, jint * const end) const {\n\t\tMapArray(map_, begin, end);\n\n\t\tstd::reverse(begin, end - 1);\n\t\tlm::ngram::State ignored;\n\t\treturn *(end - 1) ?\n\t\t\t\tm_.FullScoreForgotState(\n\t\t\t\t\t\treinterpret_cast(begin),\n\t\t\t\t\t\treinterpret_cast(end - 1), *(end - 1),\n\t\t\t\t\t\tignored).prob :\n\t\t\t\tfake_oov_cost_;\n\t}\n\n\tfloat ProbString(jint * const begin, jint * const end, jint start) const {\n\t\tMapArray(map_, begin, end);\n\n\t\tfloat prob;\n\t\tlm::ngram::State state;\n\t\tif (start == 0) {\n\t\t\tprob = 0;\n\t\t\tstate = m_.NullContextState();\n\t\t} else {\n\t\t\tstd::reverse(begin, begin + start);\n\t\t\tprob = m_.FullScoreForgotState(\n\t\t\t\t\treinterpret_cast(begin),\n\t\t\t\t\treinterpret_cast(begin + start), begin[start],\n\t\t\t\t\tstate).prob;\n\t\t\tif (begin[start] == 0)\n\t\t\t\tprob = fake_oov_cost_;\n\t\t\t++start;\n\t\t}\n\t\tlm::ngram::State state2;\n\t\tfor (const jint *i = begin + start;;) {\n\t\t\tif (i >= end)\n\t\t\t\tbreak;\n\t\t\tfloat got = m_.Score(state, *i, state2);\n\t\t\tprob += *(i++) ? got : fake_oov_cost_;\n\t\t\tif (i >= end)\n\t\t\t\tbreak;\n\t\t\tgot = m_.Score(state2, *i, state);\n\t\t\tprob += *(i++) ? got : fake_oov_cost_;\n\t\t}\n\t\treturn prob;\n\t}\n\n\tuint8_t Order() const {\n\t\treturn m_.Order();\n\t}\n\n\tbool RegisterWord(const StringPiece& word, const int wd_id) {\n\t\tif (map_.size() <= wd_id) {\n\t\t\tmap_.resize(wd_id + 1, 0);\n\t\t}\n\t\tbool already_present = false;\n\t\tif (map_[wd_id] != 0)\n\t\t\talready_present = true;\n\t\tmap_[wd_id] = m_.GetVocabulary().Index(word);\n\t\treturn already_present;\n\t}\n\nprivate:\n\tModel m_;\n\tfloat fake_oov_cost_;\n\tstd::vector map_;\n};\n\nVirtualBase *ConstructModel(const char *file_name, float fake_oov_cost) {\n\tusing namespace lm::ngram;\n\tModelType model_type;\n\tif (!RecognizeBinary(file_name, model_type))\n\t\tmodel_type = HASH_PROBING;\n\tswitch (model_type) {\n\tcase HASH_PROBING:\n\t\treturn new VirtualImpl(file_name, fake_oov_cost);\n\tcase TRIE_SORTED:\n\t\treturn new VirtualImpl(file_name, fake_oov_cost);\n\tcase ARRAY_TRIE_SORTED:\n\t\treturn new VirtualImpl(file_name, fake_oov_cost);\n\tcase QUANT_TRIE_SORTED:\n\t\treturn new VirtualImpl(file_name, fake_oov_cost);\n\tcase QUANT_ARRAY_TRIE_SORTED:\n\t\treturn new VirtualImpl(file_name, fake_oov_cost);\n\tdefault:\n\t\tUTIL_THROW(\n\t\t\t\tlm::FormatLoadException,\n\t\t\t\t\"Unrecognized file format \" << (unsigned) model_type << \" in file \"\n\t\t\t\t\t\t<< file_name);\n\t}\n}\n\n} // namespace\n\nextern \"C\" {\n\nJNIEXPORT jlong JNICALL Java_opennlp_ccg_ngrams_kenlm_jni_KenLM_construct(\n\t\tJNIEnv *env, jclass, jstring file_name, jfloat fake_oov_cost) {\n\tconst char *str = env->GetStringUTFChars(file_name, 0);\n\tif (!str)\n\t\treturn 0;\n\tjlong ret;\n\ttry {\n\t\tret = reinterpret_cast(ConstructModel(str, fake_oov_cost));\n\t} catch (std::exception &e) {\n\t\tstd::cerr << e.what() << std::endl;\n\t\tabort();\n\t}\n\tenv->ReleaseStringUTFChars(file_name, str);\n\treturn ret;\n}\n\nJNIEXPORT void JNICALL Java_opennlp_ccg_ngrams_kenlm_jni_KenLM_destroy(\n\t\tJNIEnv *env, jclass, jlong pointer) {\n\tdelete reinterpret_cast(pointer);\n}\n\nJNIEXPORT jint JNICALL Java_opennlp_ccg_ngrams_kenlm_jni_KenLM_order(\n\t\tJNIEnv *env, jclass, jlong pointer) {\n\treturn reinterpret_cast(pointer)->Order();\n}\n\nJNIEXPORT jboolean JNICALL Java_opennlp_ccg_ngrams_kenlm_jni_KenLM_registerWord(\n\t\tJNIEnv *env, jclass, jlong pointer, jstring word, jint id) {\n\tconst char *str = env->GetStringUTFChars(word, 0);\n\tif (!str)\n\t\treturn false;\n\tjint ret;\n\ttry {\n\t\tret = reinterpret_cast(pointer)->RegisterWord(str, id);\n\t} catch (std::exception &e) {\n\t\tstd::cerr << e.what() << std::endl;\n\t\tabort();\n\t}\n\tenv->ReleaseStringUTFChars(word, str);\n\treturn ret;\n}\n\nJNIEXPORT jfloat JNICALL Java_opennlp_ccg_ngrams_kenlm_jni_KenLM_prob(\n\t\tJNIEnv *env, jclass, jlong pointer, jintArray arr) {\n\tjint length = env->GetArrayLength(arr);\n\tif (length <= 0)\n\t\treturn 0.0;\n\t// GCC only.\n\tjint values[length];\n\tenv->GetIntArrayRegion(arr, 0, length, values);\n\n\treturn reinterpret_cast(pointer)->Prob(values,\n\t\t\tvalues + length);\n}\n\nJNIEXPORT jfloat JNICALL Java_opennlp_ccg_ngrams_kenlm_jni_KenLM_probString(\n\t\tJNIEnv *env, jclass, jlong pointer, jintArray arr, jint start) {\n\tjint length = env->GetArrayLength(arr);\n\tif (length <= start)\n\t\treturn 0.0;\n\t// GCC only.\n\tjint values[length];\n\tenv->GetIntArrayRegion(arr, 0, length, values);\n\n\treturn reinterpret_cast(pointer)->ProbString(values,\n\t\t\tvalues + length, start);\n}\n\n} // extern\n" }, { "path": "src/kenlm/lm/Jamfile", "content": "# If you need higher order, change this option\n# Having this limit means that State can be\n# (KENLM_MAX_ORDER - 1) * sizeof(float) bytes instead of\n# sizeof(float*) + (KENLM_MAX_ORDER - 1) * sizeof(float) + malloc overhead\nmax-order = [ option.get \"max-kenlm-order\" : 6 : 6 ] ;\nif ( $(max-order) != 6 ) {\n echo \"Setting KenLM maximum n-gram order to $(max-order)\" ;\n}\nmax-order = KENLM_MAX_ORDER=$(max-order) ;\n\npath-constant ORDER-LOG : bin/order.log ;\nupdate-if-changed $(ORDER-LOG) $(max-order) ;\n\nmax-order += $(ORDER-LOG) ;\n\nfakelib kenlm : [ glob *.cc : *main.cc *test.cc ] ../util//kenutil : .. $(max-order) : : .. $(max-order) ;\n\nimport testing ;\n\nrun left_test.cc kenlm /top//boost_unit_test_framework : : test.arpa ;\nrun model_test.cc kenlm /top//boost_unit_test_framework : : test.arpa test_nounk.arpa ;\nrun partial_test.cc kenlm /top//boost_unit_test_framework : : test.arpa ;\n\nexe query : query_main.cc kenlm ../util//kenutil ;\nexe build_binary : build_binary_main.cc kenlm ../util//kenutil ;\nexe fragment : fragment_main.cc kenlm ;\n\nalias programs : query build_binary fragment filter//filter : multi:builder//lmplz ;\n" }, { "path": "src/kenlm/lm/bhiksha.cc", "content": "#include \"lm/bhiksha.hh\"\n#include \"lm/config.hh\"\n#include \"util/file.hh\"\n#include \"util/exception.hh\"\n\n#include \n\nnamespace lm {\nnamespace ngram {\nnamespace trie {\n\nDontBhiksha::DontBhiksha(const void * /*base*/, uint64_t /*max_offset*/, uint64_t max_next, const Config &/*config*/) : \n next_(util::BitsMask::ByMax(max_next)) {}\n\nconst uint8_t kArrayBhikshaVersion = 0;\n\n// TODO: put this in binary file header instead when I change the binary file format again. \nvoid ArrayBhiksha::UpdateConfigFromBinary(int fd, Config &config) {\n uint8_t version;\n uint8_t configured_bits;\n util::ReadOrThrow(fd, &version, 1);\n util::ReadOrThrow(fd, &configured_bits, 1);\n if (version != kArrayBhikshaVersion) UTIL_THROW(FormatLoadException, \"This file has sorted array compression version \" << (unsigned) version << \" but the code expects version \" << (unsigned)kArrayBhikshaVersion);\n config.pointer_bhiksha_bits = configured_bits;\n}\n\nnamespace {\n\n// Find argmin_{chopped \\in [0, RequiredBits(max_next)]} ChoppedDelta(max_offset)\nuint8_t ChopBits(uint64_t max_offset, uint64_t max_next, const Config &config) {\n uint8_t required = util::RequiredBits(max_next);\n uint8_t best_chop = 0;\n int64_t lowest_change = std::numeric_limits::max();\n // There are probably faster ways but I don't care because this is only done once per order at construction time. \n for (uint8_t chop = 0; chop <= std::min(required, config.pointer_bhiksha_bits); ++chop) {\n int64_t change = (max_next >> (required - chop)) * 64 /* table cost in bits */\n - max_offset * static_cast(chop); /* savings in bits*/\n if (change < lowest_change) {\n lowest_change = change;\n best_chop = chop;\n }\n }\n return best_chop;\n}\n\nstd::size_t ArrayCount(uint64_t max_offset, uint64_t max_next, const Config &config) {\n uint8_t required = util::RequiredBits(max_next);\n uint8_t chopping = ChopBits(max_offset, max_next, config);\n return (max_next >> (required - chopping)) + 1 /* we store 0 too */;\n}\n} // namespace\n\nuint64_t ArrayBhiksha::Size(uint64_t max_offset, uint64_t max_next, const Config &config) {\n return sizeof(uint64_t) * (1 /* header */ + ArrayCount(max_offset, max_next, config)) + 7 /* 8-byte alignment */;\n}\n\nuint8_t ArrayBhiksha::InlineBits(uint64_t max_offset, uint64_t max_next, const Config &config) {\n return util::RequiredBits(max_next) - ChopBits(max_offset, max_next, config);\n}\n\nnamespace {\n\nvoid *AlignTo8(void *from) {\n uint8_t *val = reinterpret_cast(from);\n std::size_t remainder = reinterpret_cast(val) & 7;\n if (!remainder) return val;\n return val + 8 - remainder;\n}\n\n} // namespace\n\nArrayBhiksha::ArrayBhiksha(void *base, uint64_t max_offset, uint64_t max_next, const Config &config)\n : next_inline_(util::BitsMask::ByBits(InlineBits(max_offset, max_next, config))),\n offset_begin_(reinterpret_cast(AlignTo8(base)) + 1 /* 8-byte header */),\n offset_end_(offset_begin_ + ArrayCount(max_offset, max_next, config)),\n write_to_(reinterpret_cast(AlignTo8(base)) + 1 /* 8-byte header */ + 1 /* first entry is 0 */),\n original_base_(base) {}\n\nvoid ArrayBhiksha::FinishedLoading(const Config &config) {\n // *offset_begin_ = 0 but without a const_cast.\n *(write_to_ - (write_to_ - offset_begin_)) = 0;\n\n if (write_to_ != offset_end_) UTIL_THROW(util::Exception, \"Did not get all the array entries that were expected.\");\n\n uint8_t *head_write = reinterpret_cast(original_base_);\n *(head_write++) = kArrayBhikshaVersion;\n *(head_write++) = config.pointer_bhiksha_bits;\n}\n\nvoid ArrayBhiksha::LoadedBinary() {\n}\n\n} // namespace trie\n} // namespace ngram\n} // namespace lm\n" }, { "path": "src/kenlm/lm/bhiksha.hh", "content": "/* Simple implementation of\n * @inproceedings{bhikshacompression,\n * author={Bhiksha Raj and Ed Whittaker},\n * year={2003},\n * title={Lossless Compression of Language Model Structure and Word Identifiers},\n * booktitle={Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing},\n * pages={388--391},\n * }\n *\n * Currently only used for next pointers. \n */\n\n#ifndef LM_BHIKSHA__\n#define LM_BHIKSHA__\n\n#include \n#include \n\n#include \"lm/model_type.hh\"\n#include \"lm/trie.hh\"\n#include \"util/bit_packing.hh\"\n#include \"util/sorted_uniform.hh\"\n\nnamespace lm {\nnamespace ngram {\nstruct Config;\n\nnamespace trie {\n\nclass DontBhiksha {\n public:\n static const ModelType kModelTypeAdd = static_cast(0);\n\n static void UpdateConfigFromBinary(int /*fd*/, Config &/*config*/) {}\n\n static uint64_t Size(uint64_t /*max_offset*/, uint64_t /*max_next*/, const Config &/*config*/) { return 0; }\n\n static uint8_t InlineBits(uint64_t /*max_offset*/, uint64_t max_next, const Config &/*config*/) {\n return util::RequiredBits(max_next);\n }\n\n DontBhiksha(const void *base, uint64_t max_offset, uint64_t max_next, const Config &config);\n\n void ReadNext(const void *base, uint64_t bit_offset, uint64_t /*index*/, uint8_t total_bits, NodeRange &out) const {\n out.begin = util::ReadInt57(base, bit_offset, next_.bits, next_.mask);\n out.end = util::ReadInt57(base, bit_offset + total_bits, next_.bits, next_.mask);\n //assert(out.end >= out.begin);\n }\n\n void WriteNext(void *base, uint64_t bit_offset, uint64_t /*index*/, uint64_t value) {\n util::WriteInt57(base, bit_offset, next_.bits, value);\n }\n\n void FinishedLoading(const Config &/*config*/) {}\n\n void LoadedBinary() {}\n\n uint8_t InlineBits() const { return next_.bits; }\n\n private:\n util::BitsMask next_;\n};\n\nclass ArrayBhiksha {\n public:\n static const ModelType kModelTypeAdd = kArrayAdd;\n\n static void UpdateConfigFromBinary(int fd, Config &config);\n\n static uint64_t Size(uint64_t max_offset, uint64_t max_next, const Config &config);\n\n static uint8_t InlineBits(uint64_t max_offset, uint64_t max_next, const Config &config);\n\n ArrayBhiksha(void *base, uint64_t max_offset, uint64_t max_value, const Config &config);\n\n void ReadNext(const void *base, uint64_t bit_offset, uint64_t index, uint8_t total_bits, NodeRange &out) const {\n const uint64_t *begin_it = util::BinaryBelow(util::IdentityAccessor(), offset_begin_, offset_end_, index);\n const uint64_t *end_it;\n for (end_it = begin_it; (end_it < offset_end_) && (*end_it <= index + 1); ++end_it) {}\n --end_it;\n out.begin = ((begin_it - offset_begin_) << next_inline_.bits) | \n util::ReadInt57(base, bit_offset, next_inline_.bits, next_inline_.mask);\n out.end = ((end_it - offset_begin_) << next_inline_.bits) | \n util::ReadInt57(base, bit_offset + total_bits, next_inline_.bits, next_inline_.mask);\n //assert(out.end >= out.begin);\n }\n\n void WriteNext(void *base, uint64_t bit_offset, uint64_t index, uint64_t value) {\n uint64_t encode = value >> next_inline_.bits;\n for (; write_to_ <= offset_begin_ + encode; ++write_to_) *write_to_ = index;\n util::WriteInt57(base, bit_offset, next_inline_.bits, value & next_inline_.mask);\n }\n\n void FinishedLoading(const Config &config);\n\n void LoadedBinary();\n\n uint8_t InlineBits() const { return next_inline_.bits; }\n\n private:\n const util::BitsMask next_inline_;\n\n const uint64_t *const offset_begin_;\n const uint64_t *const offset_end_;\n\n uint64_t *write_to_;\n\n void *original_base_;\n};\n\n} // namespace trie\n} // namespace ngram\n} // namespace lm\n\n#endif // LM_BHIKSHA__\n" }, { "path": "src/kenlm/lm/binary_format.cc", "content": "#include \"lm/binary_format.hh\"\n\n#include \"lm/lm_exception.hh\"\n#include \"util/file.hh\"\n#include \"util/file_piece.hh\"\n\n#include \n#include \n#include \n#include \n\n#include \n\nnamespace lm {\nnamespace ngram {\nnamespace {\nconst char kMagicBeforeVersion[] = \"mmap lm http://kheafield.com/code format version\";\nconst char kMagicBytes[] = \"mmap lm http://kheafield.com/code format version 5\\n\\0\";\n// This must be shorter than kMagicBytes and indicates an incomplete binary file (i.e. build failed).\nconst char kMagicIncomplete[] = \"mmap lm http://kheafield.com/code incomplete\\n\";\nconst long int kMagicVersion = 5;\n\n// Old binary files built on 32-bit machines have this header.\n// TODO: eliminate with next binary release.\nstruct OldSanity {\n char magic[sizeof(kMagicBytes)];\n float zero_f, one_f, minus_half_f;\n WordIndex one_word_index, max_word_index;\n uint64_t one_uint64;\n\n void SetToReference() {\n std::memset(this, 0, sizeof(OldSanity));\n std::memcpy(magic, kMagicBytes, sizeof(magic));\n zero_f = 0.0; one_f = 1.0; minus_half_f = -0.5;\n one_word_index = 1;\n max_word_index = std::numeric_limits::max();\n one_uint64 = 1;\n }\n};\n\n\n// Test values aligned to 8 bytes.\nstruct Sanity {\n char magic[ALIGN8(sizeof(kMagicBytes))];\n float zero_f, one_f, minus_half_f;\n WordIndex one_word_index, max_word_index, padding_to_8;\n uint64_t one_uint64;\n\n void SetToReference() {\n std::memset(this, 0, sizeof(Sanity));\n std::memcpy(magic, kMagicBytes, sizeof(kMagicBytes));\n zero_f = 0.0; one_f = 1.0; minus_half_f = -0.5;\n one_word_index = 1;\n max_word_index = std::numeric_limits::max();\n padding_to_8 = 0;\n one_uint64 = 1;\n }\n};\n\nconst char *kModelNames[6] = {\"probing hash tables\", \"probing hash tables with rest costs\", \"trie\", \"trie with quantization\", \"trie with array-compressed pointers\", \"trie with quantization and array-compressed pointers\"};\n\nstd::size_t TotalHeaderSize(unsigned char order) {\n return ALIGN8(sizeof(Sanity) + sizeof(FixedWidthParameters) + sizeof(uint64_t) * order);\n}\n\nvoid WriteHeader(void *to, const Parameters ¶ms) {\n Sanity header = Sanity();\n header.SetToReference();\n std::memcpy(to, &header, sizeof(Sanity));\n char *out = reinterpret_cast(to) + sizeof(Sanity);\n\n *reinterpret_cast(out) = params.fixed;\n out += sizeof(FixedWidthParameters);\n\n uint64_t *counts = reinterpret_cast(out);\n for (std::size_t i = 0; i < params.counts.size(); ++i) {\n counts[i] = params.counts[i];\n }\n}\n\n} // namespace\n\nuint8_t *SetupJustVocab(const Config &config, uint8_t order, std::size_t memory_size, Backing &backing) {\n if (config.write_mmap) {\n std::size_t total = TotalHeaderSize(order) + memory_size;\n backing.file.reset(util::CreateOrThrow(config.write_mmap));\n if (config.write_method == Config::WRITE_MMAP) {\n backing.vocab.reset(util::MapZeroedWrite(backing.file.get(), total), total, util::scoped_memory::MMAP_ALLOCATED);\n } else {\n util::ResizeOrThrow(backing.file.get(), 0);\n util::MapAnonymous(total, backing.vocab);\n }\n strncpy(reinterpret_cast(backing.vocab.get()), kMagicIncomplete, TotalHeaderSize(order));\n return reinterpret_cast(backing.vocab.get()) + TotalHeaderSize(order);\n } else {\n util::MapAnonymous(memory_size, backing.vocab);\n return reinterpret_cast(backing.vocab.get());\n }\n}\n\nuint8_t *GrowForSearch(const Config &config, std::size_t vocab_pad, std::size_t memory_size, Backing &backing) {\n std::size_t adjusted_vocab = backing.vocab.size() + vocab_pad;\n if (config.write_mmap) {\n // Grow the file to accomodate the search, using zeros.\n try {\n util::ResizeOrThrow(backing.file.get(), adjusted_vocab + memory_size);\n } catch (util::ErrnoException &e) {\n e << \" for file \" << config.write_mmap;\n throw e;\n }\n\n if (config.write_method == Config::WRITE_AFTER) {\n util::MapAnonymous(memory_size, backing.search);\n return reinterpret_cast(backing.search.get());\n }\n // mmap it now.\n // We're skipping over the header and vocab for the search space mmap. mmap likes page aligned offsets, so some arithmetic to round the offset down.\n std::size_t page_size = util::SizePage();\n std::size_t alignment_cruft = adjusted_vocab % page_size;\n backing.search.reset(util::MapOrThrow(alignment_cruft + memory_size, true, util::kFileFlags, false, backing.file.get(), adjusted_vocab - alignment_cruft), alignment_cruft + memory_size, util::scoped_memory::MMAP_ALLOCATED);\n return reinterpret_cast(backing.search.get()) + alignment_cruft;\n } else {\n util::MapAnonymous(memory_size, backing.search);\n return reinterpret_cast(backing.search.get());\n }\n}\n\nvoid FinishFile(const Config &config, ModelType model_type, unsigned int search_version, const std::vector &counts, std::size_t vocab_pad, Backing &backing) {\n if (!config.write_mmap) return;\n switch (config.write_method) {\n case Config::WRITE_MMAP:\n util::SyncOrThrow(backing.vocab.get(), backing.vocab.size());\n util::SyncOrThrow(backing.search.get(), backing.search.size());\n break;\n case Config::WRITE_AFTER:\n util::SeekOrThrow(backing.file.get(), 0);\n util::WriteOrThrow(backing.file.get(), backing.vocab.get(), backing.vocab.size());\n util::SeekOrThrow(backing.file.get(), backing.vocab.size() + vocab_pad);\n util::WriteOrThrow(backing.file.get(), backing.search.get(), backing.search.size());\n util::FSyncOrThrow(backing.file.get());\n break;\n }\n // header and vocab share the same mmap. The header is written here because we know the counts.\n Parameters params = Parameters();\n params.counts = counts;\n params.fixed.order = counts.size();\n params.fixed.probing_multiplier = config.probing_multiplier;\n params.fixed.model_type = model_type;\n params.fixed.has_vocabulary = config.include_vocab;\n params.fixed.search_version = search_version;\n WriteHeader(backing.vocab.get(), params);\n if (config.write_method == Config::WRITE_AFTER) {\n util::SeekOrThrow(backing.file.get(), 0);\n util::WriteOrThrow(backing.file.get(), backing.vocab.get(), TotalHeaderSize(counts.size()));\n }\n}\n\nnamespace detail {\n\nbool IsBinaryFormat(int fd) {\n const uint64_t size = util::SizeFile(fd);\n if (size == util::kBadSize || (size <= static_cast(sizeof(Sanity)))) return false;\n // Try reading the header.\n util::scoped_memory memory;\n try {\n util::MapRead(util::LAZY, fd, 0, sizeof(Sanity), memory);\n } catch (const util::Exception &e) {\n return false;\n }\n Sanity reference_header = Sanity();\n reference_header.SetToReference();\n if (!memcmp(memory.get(), &reference_header, sizeof(Sanity))) return true;\n if (!memcmp(memory.get(), kMagicIncomplete, strlen(kMagicIncomplete))) {\n UTIL_THROW(FormatLoadException, \"This binary file did not finish building\");\n }\n if (!memcmp(memory.get(), kMagicBeforeVersion, strlen(kMagicBeforeVersion))) {\n char *end_ptr;\n const char *begin_version = static_cast(memory.get()) + strlen(kMagicBeforeVersion);\n long int version = strtol(begin_version, &end_ptr, 10);\n if ((end_ptr != begin_version) && version != kMagicVersion) {\n UTIL_THROW(FormatLoadException, \"Binary file has version \" << version << \" but this implementation expects version \" << kMagicVersion << \" so you'll have to use the ARPA to rebuild your binary\");\n }\n\n OldSanity old_sanity = OldSanity();\n old_sanity.SetToReference();\n UTIL_THROW_IF(!memcmp(memory.get(), &old_sanity, sizeof(OldSanity)), FormatLoadException, \"Looks like this is an old 32-bit format. The old 32-bit format has been removed so that 64-bit and 32-bit files are exchangeable.\");\n UTIL_THROW(FormatLoadException, \"File looks like it should be loaded with mmap, but the test values don't match. Try rebuilding the binary format LM using the same code revision, compiler, and architecture\");\n }\n return false;\n}\n\nvoid ReadHeader(int fd, Parameters &out) {\n util::SeekOrThrow(fd, sizeof(Sanity));\n util::ReadOrThrow(fd, &out.fixed, sizeof(out.fixed));\n if (out.fixed.probing_multiplier < 1.0)\n UTIL_THROW(FormatLoadException, \"Binary format claims to have a probing multiplier of \" << out.fixed.probing_multiplier << \" which is < 1.0.\");\n\n out.counts.resize(static_cast(out.fixed.order));\n if (out.fixed.order) util::ReadOrThrow(fd, &*out.counts.begin(), sizeof(uint64_t) * out.fixed.order);\n}\n\nvoid MatchCheck(ModelType model_type, unsigned int search_version, const Parameters ¶ms) {\n if (params.fixed.model_type != model_type) {\n if (static_cast(params.fixed.model_type) >= (sizeof(kModelNames) / sizeof(const char *)))\n UTIL_THROW(FormatLoadException, \"The binary file claims to be model type \" << static_cast(params.fixed.model_type) << \" but this is not implemented for in this inference code.\");\n UTIL_THROW(FormatLoadException, \"The binary file was built for \" << kModelNames[params.fixed.model_type] << \" but the inference code is trying to load \" << kModelNames[model_type]);\n }\n UTIL_THROW_IF(search_version != params.fixed.search_version, FormatLoadException, \"The binary file has \" << kModelNames[params.fixed.model_type] << \" version \" << params.fixed.search_version << \" but this code expects \" << kModelNames[params.fixed.model_type] << \" version \" << search_version);\n}\n\nvoid SeekPastHeader(int fd, const Parameters ¶ms) {\n util::SeekOrThrow(fd, TotalHeaderSize(params.counts.size()));\n}\n\nuint8_t *SetupBinary(const Config &config, const Parameters ¶ms, uint64_t memory_size, Backing &backing) {\n const uint64_t file_size = util::SizeFile(backing.file.get());\n // The header is smaller than a page, so we have to map the whole header as well.\n std::size_t total_map = util::CheckOverflow(TotalHeaderSize(params.counts.size()) + memory_size);\n if (file_size != util::kBadSize && static_cast(file_size) < total_map)\n UTIL_THROW(FormatLoadException, \"Binary file has size \" << file_size << \" but the headers say it should be at least \" << total_map);\n\n util::MapRead(config.load_method, backing.file.get(), 0, total_map, backing.search);\n\n if (config.enumerate_vocab && !params.fixed.has_vocabulary)\n UTIL_THROW(FormatLoadException, \"The decoder requested all the vocabulary strings, but this binary file does not have them. You may need to rebuild the binary file with an updated version of build_binary.\");\n\n // Seek to vocabulary words\n util::SeekOrThrow(backing.file.get(), total_map);\n return reinterpret_cast(backing.search.get()) + TotalHeaderSize(params.counts.size());\n}\n\nvoid ComplainAboutARPA(const Config &config, ModelType model_type) {\n if (config.write_mmap || !config.messages) return;\n if (config.arpa_complain == Config::ALL) {\n *config.messages << \"Loading the LM will be faster if you build a binary file.\" << std::endl;\n } else if (config.arpa_complain == Config::EXPENSIVE &&\n (model_type == TRIE || model_type == QUANT_TRIE || model_type == ARRAY_TRIE || model_type == QUANT_ARRAY_TRIE)) {\n *config.messages << \"Building \" << kModelNames[model_type] << \" from ARPA is expensive. Save time by building a binary format.\" << std::endl;\n }\n}\n\n} // namespace detail\n\nbool RecognizeBinary(const char *file, ModelType &recognized) {\n util::scoped_fd fd(util::OpenReadOrThrow(file));\n if (!detail::IsBinaryFormat(fd.get())) return false;\n Parameters params;\n detail::ReadHeader(fd.get(), params);\n recognized = params.fixed.model_type;\n return true;\n}\n\n} // namespace ngram\n} // namespace lm\n" }, { "path": "src/kenlm/lm/binary_format.hh", "content": "#ifndef LM_BINARY_FORMAT__\n#define LM_BINARY_FORMAT__\n\n#include \"lm/config.hh\"\n#include \"lm/model_type.hh\"\n#include \"lm/read_arpa.hh\"\n\n#include \"util/file_piece.hh\"\n#include \"util/mmap.hh\"\n#include \"util/scoped.hh\"\n\n#include \n#include \n\n#include \n\nnamespace lm {\nnamespace ngram {\n\n/*Inspect a file to determine if it is a binary lm. If not, return false. \n * If so, return true and set recognized to the type. This is the only API in\n * this header designed for use by decoder authors. \n */\nbool RecognizeBinary(const char *file, ModelType &recognized);\n\nstruct FixedWidthParameters {\n unsigned char order;\n float probing_multiplier;\n // What type of model is this? \n ModelType model_type;\n // Does the end of the file have the actual strings in the vocabulary? \n bool has_vocabulary;\n unsigned int search_version;\n};\n\n// This is a macro instead of an inline function so constants can be assigned using it.\n#define ALIGN8(a) ((std::ptrdiff_t(((a)-1)/8)+1)*8)\n\n// Parameters stored in the header of a binary file. \nstruct Parameters {\n FixedWidthParameters fixed;\n std::vector counts;\n};\n\nstruct Backing {\n // File behind memory, if any. \n util::scoped_fd file;\n // Vocabulary lookup table. Not to be confused with the vocab words themselves. \n util::scoped_memory vocab;\n // Raw block of memory backing the language model data structures\n util::scoped_memory search;\n};\n\n// Create just enough of a binary file to write vocabulary to it. \nuint8_t *SetupJustVocab(const Config &config, uint8_t order, std::size_t memory_size, Backing &backing);\n// Grow the binary file for the search data structure and set backing.search, returning the memory address where the search data structure should begin. \nuint8_t *GrowForSearch(const Config &config, std::size_t vocab_pad, std::size_t memory_size, Backing &backing);\n\n// Write header to binary file. This is done last to prevent incomplete files\n// from loading. \nvoid FinishFile(const Config &config, ModelType model_type, unsigned int search_version, const std::vector &counts, std::size_t vocab_pad, Backing &backing);\n\nnamespace detail {\n\nbool IsBinaryFormat(int fd);\n\nvoid ReadHeader(int fd, Parameters ¶ms);\n\nvoid MatchCheck(ModelType model_type, unsigned int search_version, const Parameters ¶ms);\n\nvoid SeekPastHeader(int fd, const Parameters ¶ms);\n\nuint8_t *SetupBinary(const Config &config, const Parameters ¶ms, uint64_t memory_size, Backing &backing);\n\nvoid ComplainAboutARPA(const Config &config, ModelType model_type);\n\n} // namespace detail\n\ntemplate void LoadLM(const char *file, const Config &config, To &to) {\n Backing &backing = to.MutableBacking();\n backing.file.reset(util::OpenReadOrThrow(file));\n\n try {\n if (detail::IsBinaryFormat(backing.file.get())) {\n Parameters params;\n detail::ReadHeader(backing.file.get(), params);\n detail::MatchCheck(To::kModelType, To::kVersion, params);\n // Replace the run-time configured probing_multiplier with the one in the file. \n Config new_config(config);\n new_config.probing_multiplier = params.fixed.probing_multiplier;\n detail::SeekPastHeader(backing.file.get(), params);\n To::UpdateConfigFromBinary(backing.file.get(), params.counts, new_config);\n uint64_t memory_size = To::Size(params.counts, new_config);\n uint8_t *start = detail::SetupBinary(new_config, params, memory_size, backing);\n to.InitializeFromBinary(start, params, new_config, backing.file.get());\n } else {\n detail::ComplainAboutARPA(config, To::kModelType);\n to.InitializeFromARPA(file, config);\n }\n } catch (util::Exception &e) {\n e << \" File: \" << file;\n throw;\n }\n}\n\n} // namespace ngram\n} // namespace lm\n#endif // LM_BINARY_FORMAT__\n" }, { "path": "src/kenlm/lm/blank.hh", "content": "#ifndef LM_BLANK__\n#define LM_BLANK__\n\n#include \n\n#include \n#include \n\nnamespace lm {\nnamespace ngram {\n\n/* Suppose \"foo bar\" appears with zero backoff but there is no trigram\n * beginning with these words. Then, when scoring \"foo bar\", the model could\n * return out_state containing \"bar\" or even null context if \"bar\" also has no\n * backoff and is never followed by another word. Then the backoff is set to\n * kNoExtensionBackoff. If the n-gram might be extended, then out_state must\n * contain the full n-gram, in which case kExtensionBackoff is set. In any\n * case, if an n-gram has non-zero backoff, the full state is returned so\n * backoff can be properly charged. \n * These differ only in sign bit because the backoff is in fact zero in either\n * case. \n */\nconst float kNoExtensionBackoff = -0.0;\nconst float kExtensionBackoff = 0.0;\nconst uint64_t kNoExtensionQuant = 0;\nconst uint64_t kExtensionQuant = 1;\n\ninline void SetExtension(float &backoff) {\n if (backoff == kNoExtensionBackoff) backoff = kExtensionBackoff;\n}\n\n// This compiles down nicely. \ninline bool HasExtension(const float &backoff) {\n typedef union { float f; uint32_t i; } UnionValue;\n UnionValue compare, interpret;\n compare.f = kNoExtensionBackoff;\n interpret.f = backoff;\n return compare.i != interpret.i;\n}\n\n} // namespace ngram\n} // namespace lm\n#endif // LM_BLANK__\n" }, { "path": "src/kenlm/lm/build_binary_main.cc", "content": "#include \"lm/model.hh\"\n#include \"lm/sizes.hh\"\n#include \"util/file_piece.hh\"\n#include \"util/usage.hh\"\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n\n#ifdef WIN32\n#include \"util/getopt.hh\"\n#else\n#include \n#endif\n\nnamespace lm {\nnamespace ngram {\nnamespace {\n\nvoid Usage(const char *name, const char *default_mem) {\n std::cerr << \"Usage: \" << name << \" [-u log10_unknown_probability] [-s] [-i] [-w mmap|after] [-p probing_multiplier] [-T trie_temporary] [-S trie_building_mem] [-q bits] [-b bits] [-a bits] [type] input.arpa [output.mmap]\\n\\n\"\n\"-u sets the log10 probability for if the ARPA file does not have one.\\n\"\n\" Default is -100. The ARPA file will always take precedence.\\n\"\n\"-s allows models to be built even if they do not have and .\\n\"\n\"-i allows buggy models from IRSTLM by mapping positive log probability to 0.\\n\"\n\"-w mmap|after determines how writing is done.\\n\"\n\" mmap maps the binary file and writes to it. Default for trie.\\n\"\n\" after allocates anonymous memory, builds, and writes. Default for probing.\\n\"\n\"-r \\\"order1.arpa order2 order3 order4\\\" adds lower-order rest costs from these\\n\"\n\" model files. order1.arpa must be an ARPA file. All others may be ARPA or\\n\"\n\" the same data structure as being built. All files must have the same\\n\"\n\" vocabulary. For probing, the unigrams must be in the same order.\\n\\n\"\n\"type is either probing or trie. Default is probing.\\n\\n\"\n\"probing uses a probing hash table. It is the fastest but uses the most memory.\\n\"\n\"-p sets the space multiplier and must be >1.0. The default is 1.5.\\n\\n\"\n\"trie is a straightforward trie with bit-level packing. It uses the least\\n\"\n\"memory and is still faster than SRI or IRST. Building the trie format uses an\\n\"\n\"on-disk sort to save memory.\\n\"\n\"-T is the temporary directory prefix. Default is the output file name.\\n\"\n\"-S determines memory use for sorting. Default is \" << default_mem << \". This is compatible\\n\"\n\" with GNU sort. The number is followed by a unit: \\% for percent of physical\\n\"\n\" memory, b for bytes, K for Kilobytes, M for megabytes, then G,T,P,E,Z,Y. \\n\"\n\" Default unit is K for Kilobytes.\\n\"\n\"-q turns quantization on and sets the number of bits (e.g. -q 8).\\n\"\n\"-b sets backoff quantization bits. Requires -q and defaults to that value.\\n\"\n\"-a compresses pointers using an array of offsets. The parameter is the\\n\"\n\" maximum number of bits encoded by the array. Memory is minimized subject\\n\"\n\" to the maximum, so pick 255 to minimize memory.\\n\\n\"\n\"Get a memory estimate by passing an ARPA file without an output file name.\\n\";\n exit(1);\n}\n\n// I could really use boost::lexical_cast right about now. \nfloat ParseFloat(const char *from) {\n char *end;\n float ret = strtod(from, &end);\n if (*end) throw util::ParseNumberException(from);\n return ret;\n}\nunsigned long int ParseUInt(const char *from) {\n char *end;\n unsigned long int ret = strtoul(from, &end, 10);\n if (*end) throw util::ParseNumberException(from);\n return ret;\n}\n\nuint8_t ParseBitCount(const char *from) {\n unsigned long val = ParseUInt(from);\n if (val > 25) {\n util::ParseNumberException e(from);\n e << \" bit counts are limited to 25.\";\n }\n return val;\n}\n\nvoid ParseFileList(const char *from, std::vector &to) {\n to.clear();\n while (true) {\n const char *i;\n for (i = from; *i && *i != ' '; ++i) {}\n to.push_back(std::string(from, i - from));\n if (!*i) break;\n from = i + 1;\n }\n}\n\nvoid ProbingQuantizationUnsupported() {\n std::cerr << \"Quantization is only implemented in the trie data structure.\" << std::endl;\n exit(1);\n}\n\n} // namespace ngram\n} // namespace lm\n} // namespace\n\nint main(int argc, char *argv[]) {\n using namespace lm::ngram;\n\n const char *default_mem = util::GuessPhysicalMemory() ? \"80%\" : \"1G\";\n\n try {\n bool quantize = false, set_backoff_bits = false, bhiksha = false, set_write_method = false, rest = false;\n lm::ngram::Config config;\n config.building_memory = util::ParseSize(default_mem);\n int opt;\n while ((opt = getopt(argc, argv, \"q:b:a:u:p:t:T:m:S:w:sir:\")) != -1) {\n switch(opt) {\n case 'q':\n config.prob_bits = ParseBitCount(optarg);\n if (!set_backoff_bits) config.backoff_bits = config.prob_bits;\n quantize = true;\n break;\n case 'b':\n config.backoff_bits = ParseBitCount(optarg);\n set_backoff_bits = true;\n break;\n case 'a':\n config.pointer_bhiksha_bits = ParseBitCount(optarg);\n bhiksha = true;\n break;\n case 'u':\n config.unknown_missing_logprob = ParseFloat(optarg);\n break;\n case 'p':\n config.probing_multiplier = ParseFloat(optarg);\n break;\n case 't': // legacy\n case 'T':\n config.temporary_directory_prefix = optarg;\n break;\n case 'm': // legacy\n config.building_memory = ParseUInt(optarg) * 1048576;\n break;\n case 'S':\n config.building_memory = std::min(static_cast(std::numeric_limits::max()), util::ParseSize(optarg));\n break;\n case 'w':\n set_write_method = true;\n if (!strcmp(optarg, \"mmap\")) {\n config.write_method = Config::WRITE_MMAP;\n } else if (!strcmp(optarg, \"after\")) {\n config.write_method = Config::WRITE_AFTER;\n } else {\n Usage(argv[0], default_mem);\n }\n break;\n case 's':\n config.sentence_marker_missing = lm::SILENT;\n break;\n case 'i':\n config.positive_log_probability = lm::SILENT;\n break;\n case 'r':\n rest = true;\n ParseFileList(optarg, config.rest_lower_files);\n config.rest_function = Config::REST_LOWER;\n break;\n default:\n Usage(argv[0], default_mem);\n }\n }\n if (!quantize && set_backoff_bits) {\n std::cerr << \"You specified backoff quantization (-b) but not probability quantization (-q)\" << std::endl;\n abort();\n }\n if (optind + 1 == argc) {\n ShowSizes(argv[optind], config);\n return 0;\n }\n const char *model_type;\n const char *from_file;\n\n if (optind + 2 == argc) {\n model_type = \"probing\";\n from_file = argv[optind];\n config.write_mmap = argv[optind + 1];\n } else if (optind + 3 == argc) {\n model_type = argv[optind];\n from_file = argv[optind + 1];\n config.write_mmap = argv[optind + 2];\n } else {\n Usage(argv[0], default_mem);\n }\n if (!strcmp(model_type, \"probing\")) {\n if (!set_write_method) config.write_method = Config::WRITE_AFTER;\n if (quantize || set_backoff_bits) ProbingQuantizationUnsupported();\n if (rest) {\n RestProbingModel(from_file, config);\n } else {\n ProbingModel(from_file, config);\n }\n } else if (!strcmp(model_type, \"trie\")) {\n if (rest) {\n std::cerr << \"Rest + trie is not supported yet.\" << std::endl;\n return 1;\n }\n if (!set_write_method) config.write_method = Config::WRITE_MMAP;\n if (quantize) {\n if (bhiksha) {\n QuantArrayTrieModel(from_file, config);\n } else {\n QuantTrieModel(from_file, config);\n }\n } else {\n if (bhiksha) {\n ArrayTrieModel(from_file, config);\n } else {\n TrieModel(from_file, config);\n }\n }\n } else {\n Usage(argv[0], default_mem);\n }\n }\n catch (const std::exception &e) {\n std::cerr << e.what() << std::endl;\n std::cerr << \"ERROR\" << std::endl;\n return 1;\n }\n std::cerr << \"SUCCESS\" << std::endl;\n return 0;\n}\n" }, { "path": "src/kenlm/lm/config.cc", "content": "#include \"lm/config.hh\"\n\n#include \n\nnamespace lm {\nnamespace ngram {\n\nConfig::Config() :\n show_progress(true),\n messages(&std::cerr),\n enumerate_vocab(NULL),\n unknown_missing(COMPLAIN),\n sentence_marker_missing(THROW_UP),\n positive_log_probability(THROW_UP),\n unknown_missing_logprob(-100.0),\n probing_multiplier(1.5),\n building_memory(1073741824ULL), // 1 GB\n temporary_directory_prefix(NULL),\n arpa_complain(ALL),\n write_mmap(NULL),\n write_method(WRITE_AFTER),\n include_vocab(true),\n rest_function(REST_MAX),\n prob_bits(8),\n backoff_bits(8),\n pointer_bhiksha_bits(22),\n load_method(util::POPULATE_OR_READ) {}\n\n} // namespace ngram\n} // namespace lm\n" }, { "path": "src/kenlm/lm/config.hh", "content": "#ifndef LM_CONFIG__\n#define LM_CONFIG__\n\n#include \"lm/lm_exception.hh\"\n#include \"util/mmap.hh\"\n\n#include \n#include \n#include \n\n/* Configuration for ngram model. Separate header to reduce pollution. */\n\nnamespace lm {\n\nclass EnumerateVocab;\n\nnamespace ngram {\n\nstruct Config {\n // EFFECTIVE FOR BOTH ARPA AND BINARY READS\n\n // (default true) print progress bar to messages\n bool show_progress;\n\n // Where to log messages including the progress bar. Set to NULL for\n // silence.\n std::ostream *messages;\n\n std::ostream *ProgressMessages() const {\n return show_progress ? messages : 0;\n }\n\n // This will be called with every string in the vocabulary. See\n // enumerate_vocab.hh for more detail. Config does not take ownership; you\n // are still responsible for deleting it (or stack allocating).\n EnumerateVocab *enumerate_vocab;\n\n\n // ONLY EFFECTIVE WHEN READING ARPA\n\n // What to do when isn't in the provided model.\n WarningAction unknown_missing;\n // What to do when or is missing from the model.\n // If THROW_UP, the exception will be of type util::SpecialWordMissingException.\n WarningAction sentence_marker_missing;\n\n // What to do with a positive log probability. For COMPLAIN and SILENT, map\n // to 0.\n WarningAction positive_log_probability;\n\n // The probability to substitute for if it's missing from the model.\n // No effect if the model has or unknown_missing == THROW_UP.\n float unknown_missing_logprob;\n\n // Size multiplier for probing hash table. Must be > 1. Space is linear in\n // this. Time is probing_multiplier / (probing_multiplier - 1). No effect\n // for sorted variant.\n // If you find yourself setting this to a low number, consider using the\n // TrieModel which has lower memory consumption.\n float probing_multiplier;\n\n // Amount of memory to use for building. The actual memory usage will be\n // higher since this just sets sort buffer size. Only applies to trie\n // models.\n std::size_t building_memory;\n\n // Template for temporary directory appropriate for passing to mkdtemp.\n // The characters XXXXXX are appended before passing to mkdtemp. Only\n // applies to trie. If NULL, defaults to write_mmap. If that's NULL,\n // defaults to input file name.\n const char *temporary_directory_prefix;\n\n // Level of complaining to do when loading from ARPA instead of binary format.\n enum ARPALoadComplain {ALL, EXPENSIVE, NONE};\n ARPALoadComplain arpa_complain;\n\n // While loading an ARPA file, also write out this binary format file. Set\n // to NULL to disable.\n const char *write_mmap;\n\n enum WriteMethod {\n WRITE_MMAP, // Map the file directly.\n WRITE_AFTER // Write after we're done.\n };\n WriteMethod write_method;\n\n // Include the vocab in the binary file? Only effective if write_mmap != NULL.\n bool include_vocab;\n\n\n // Left rest options. Only used when the model includes rest costs.\n enum RestFunction {\n REST_MAX, // Maximum of any score to the left\n REST_LOWER, // Use lower-order files given below.\n };\n RestFunction rest_function;\n // Only used for REST_LOWER.\n std::vector rest_lower_files;\n\n\n // Quantization options. Only effective for QuantTrieModel. One value is\n // reserved for each of prob and backoff, so 2^bits - 1 buckets will be used\n // to quantize (and one of the remaining backoffs will be 0).\n uint8_t prob_bits, backoff_bits;\n\n // Bhiksha compression (simple form). Only works with trie.\n uint8_t pointer_bhiksha_bits;\n\n\n // ONLY EFFECTIVE WHEN READING BINARY\n\n // How to get the giant array into memory: lazy mmap, populate, read etc.\n // See util/mmap.hh for details of MapMethod.\n util::LoadMethod load_method;\n\n\n // Set defaults.\n Config();\n};\n\n} /* namespace ngram */ } /* namespace lm */\n\n#endif // LM_CONFIG__\n" }, { "path": "src/kenlm/lm/enumerate_vocab.hh", "content": "#ifndef LM_ENUMERATE_VOCAB__\n#define LM_ENUMERATE_VOCAB__\n\n#include \"lm/word_index.hh\"\n#include \"util/string_piece.hh\"\n\nnamespace lm {\n\n/* If you need the actual strings in the vocabulary, inherit from this class\n * and implement Add. Then put a pointer in Config.enumerate_vocab; it does\n * not take ownership. Add is called once per vocab word. index starts at 0\n * and increases by 1 each time. This is only used by the Model constructor;\n * the pointer is not retained by the class. \n */\nclass EnumerateVocab {\n public:\n virtual ~EnumerateVocab() {}\n\n virtual void Add(WordIndex index, const StringPiece &str) = 0;\n\n protected:\n EnumerateVocab() {}\n};\n\n} // namespace lm\n\n#endif // LM_ENUMERATE_VOCAB__\n\n" }, { "path": "src/kenlm/lm/facade.hh", "content": "#ifndef LM_FACADE__\n#define LM_FACADE__\n\n#include \"lm/virtual_interface.hh\"\n#include \"util/string_piece.hh\"\n\n#include \n\nnamespace lm {\nnamespace base {\n\n// Common model interface that depends on knowing the specific classes. \n// Curiously recurring template pattern. \ntemplate class ModelFacade : public Model {\n public:\n typedef StateT State;\n typedef VocabularyT Vocabulary;\n\n // Default Score function calls FullScore. Model can override this. \n float Score(const State &in_state, const WordIndex new_word, State &out_state) const {\n return static_cast(this)->FullScore(in_state, new_word, out_state).prob;\n }\n\n /* Translate from void* to State */\n FullScoreReturn FullScore(const void *in_state, const WordIndex new_word, void *out_state) const {\n return static_cast(this)->FullScore(\n *reinterpret_cast(in_state),\n new_word,\n *reinterpret_cast(out_state));\n }\n float Score(const void *in_state, const WordIndex new_word, void *out_state) const {\n return static_cast(this)->Score(\n *reinterpret_cast(in_state),\n new_word,\n *reinterpret_cast(out_state));\n }\n\n const State &BeginSentenceState() const { return begin_sentence_; }\n const State &NullContextState() const { return null_context_; }\n const Vocabulary &GetVocabulary() const { return *static_cast(&BaseVocabulary()); }\n\n protected:\n ModelFacade() : Model(sizeof(State)) {}\n\n virtual ~ModelFacade() {}\n\n // begin_sentence and null_context can disappear after. vocab should stay. \n void Init(const State &begin_sentence, const State &null_context, const Vocabulary &vocab, unsigned char order) {\n begin_sentence_ = begin_sentence;\n null_context_ = null_context;\n begin_sentence_memory_ = &begin_sentence_;\n null_context_memory_ = &null_context_;\n base_vocab_ = &vocab;\n order_ = order;\n }\n\n private:\n State begin_sentence_, null_context_;\n};\n\n} // mamespace base\n} // namespace lm\n\n#endif // LM_FACADE__\n" }, { "path": "src/kenlm/lm/fragment_main.cc", "content": "#include \"lm/binary_format.hh\"\n#include \"lm/model.hh\"\n#include \"lm/left.hh\"\n#include \"util/tokenize_piece.hh\"\n\ntemplate void Query(const char *name) {\n Model model(name);\n std::string line;\n lm::ngram::ChartState ignored;\n while (getline(std::cin, line)) {\n lm::ngram::RuleScore scorer(model, ignored);\n for (util::TokenIter i(line, ' '); i; ++i) {\n scorer.Terminal(model.GetVocabulary().Index(*i));\n }\n std::cout << scorer.Finish() << '\\n';\n }\n}\n\nint main(int argc, char *argv[]) {\n if (argc != 2) {\n std::cerr << \"Expected model file name.\" << std::endl;\n return 1;\n }\n const char *name = argv[1];\n lm::ngram::ModelType model_type = lm::ngram::PROBING;\n lm::ngram::RecognizeBinary(name, model_type);\n switch (model_type) {\n case lm::ngram::PROBING:\n Query(name);\n break;\n case lm::ngram::REST_PROBING:\n Query(name);\n break;\n default:\n std::cerr << \"Model type not supported yet.\" << std::endl;\n }\n}\n" }, { "path": "src/kenlm/lm/kenlm_max_order_main.cc", "content": "#include \"lm/max_order.hh\"\n#include \n\nint main(int argc, char *argv[]) {\n std::cerr << \"KenLM was compiled with a maximum supported n-gram order set to \" << KENLM_MAX_ORDER << \".\" << std::endl;\n}\n" }, { "path": "src/kenlm/lm/left.hh", "content": "/* Efficient left and right language model state for sentence fragments.\n * Intended usage:\n * Store ChartState with every chart entry. \n * To do a rule application:\n * 1. Make a ChartState object for your new entry. \n * 2. Construct RuleScore. \n * 3. Going from left to right, call Terminal or NonTerminal. \n * For terminals, just pass the vocab id. \n * For non-terminals, pass that non-terminal's ChartState.\n * If your decoder expects scores inclusive of subtree scores (i.e. you\n * label entries with the highest-scoring path), pass the non-terminal's\n * score as prob. \n * If your decoder expects relative scores and will walk the chart later,\n * pass prob = 0.0. \n * In other words, the only effect of prob is that it gets added to the\n * returned log probability. \n * 4. Call Finish. It returns the log probability. \n *\n * There's a couple more details: \n * Do not pass to Terminal as it is formally not a word in the sentence,\n * only context. Instead, call BeginSentence. If called, it should be the\n * first call after RuleScore is constructed (since is always the\n * leftmost).\n *\n * If the leftmost RHS is a non-terminal, it's faster to call BeginNonTerminal.\n *\n * Hashing and sorting comparison operators are provided. All state objects\n * are POD. If you intend to use memcmp on raw state objects, you must call\n * ZeroRemaining first, as the value of array entries beyond length is\n * otherwise undefined. \n *\n * Usage is of course not limited to chart decoding. Anything that generates\n * sentence fragments missing left context could benefit. For example, a\n * phrase-based decoder could pre-score phrases, storing ChartState with each\n * phrase, even if hypotheses are generated left-to-right. \n */\n\n#ifndef LM_LEFT__\n#define LM_LEFT__\n\n#include \"lm/max_order.hh\"\n#include \"lm/state.hh\"\n#include \"lm/return.hh\"\n\n#include \"util/murmur_hash.hh\"\n\n#include \n\nnamespace lm {\nnamespace ngram {\n\ntemplate class RuleScore {\n public:\n explicit RuleScore(const M &model, ChartState &out) : model_(model), out_(&out), left_done_(false), prob_(0.0) {\n out.left.length = 0;\n out.right.length = 0;\n }\n\n void BeginSentence() {\n out_->right = model_.BeginSentenceState();\n // out_->left is empty.\n left_done_ = true;\n }\n\n void Terminal(WordIndex word) {\n State copy(out_->right);\n FullScoreReturn ret(model_.FullScore(copy, word, out_->right));\n if (left_done_) { prob_ += ret.prob; return; }\n if (ret.independent_left) {\n prob_ += ret.prob;\n left_done_ = true;\n return;\n }\n out_->left.pointers[out_->left.length++] = ret.extend_left;\n prob_ += ret.rest;\n if (out_->right.length != copy.length + 1)\n left_done_ = true;\n }\n\n // Faster version of NonTerminal for the case where the rule begins with a non-terminal. \n void BeginNonTerminal(const ChartState &in, float prob = 0.0) {\n prob_ = prob;\n *out_ = in;\n left_done_ = in.left.full;\n }\n\n void NonTerminal(const ChartState &in, float prob = 0.0) {\n prob_ += prob;\n \n if (!in.left.length) {\n if (in.left.full) {\n for (const float *i = out_->right.backoff; i < out_->right.backoff + out_->right.length; ++i) prob_ += *i;\n left_done_ = true;\n out_->right = in.right;\n }\n return;\n }\n\n if (!out_->right.length) {\n out_->right = in.right;\n if (left_done_) {\n prob_ += model_.UnRest(in.left.pointers, in.left.pointers + in.left.length, 1);\n return;\n }\n if (out_->left.length) {\n left_done_ = true;\n } else {\n out_->left = in.left;\n left_done_ = in.left.full;\n }\n return;\n }\n\n float backoffs[KENLM_MAX_ORDER - 1], backoffs2[KENLM_MAX_ORDER - 1];\n float *back = backoffs, *back2 = backoffs2;\n unsigned char next_use = out_->right.length;\n\n // First word\n if (ExtendLeft(in, next_use, 1, out_->right.backoff, back)) return;\n\n // Words after the first, so extending a bigram to begin with\n for (unsigned char extend_length = 2; extend_length <= in.left.length; ++extend_length) {\n if (ExtendLeft(in, next_use, extend_length, back, back2)) return;\n std::swap(back, back2);\n }\n\n if (in.left.full) {\n for (const float *i = back; i != back + next_use; ++i) prob_ += *i;\n left_done_ = true;\n out_->right = in.right;\n return;\n }\n\n // Right state was minimized, so it's already independent of the new words to the left. \n if (in.right.length < in.left.length) {\n out_->right = in.right;\n return;\n }\n\n // Shift exisiting words down. \n for (WordIndex *i = out_->right.words + next_use - 1; i >= out_->right.words; --i) {\n *(i + in.right.length) = *i;\n }\n // Add words from in.right. \n std::copy(in.right.words, in.right.words + in.right.length, out_->right.words);\n // Assemble backoff composed on the existing state's backoff followed by the new state's backoff. \n std::copy(in.right.backoff, in.right.backoff + in.right.length, out_->right.backoff);\n std::copy(back, back + next_use, out_->right.backoff + in.right.length);\n out_->right.length = in.right.length + next_use;\n }\n\n float Finish() {\n // A N-1-gram might extend left and right but we should still set full to true because it's an N-1-gram. \n out_->left.full = left_done_ || (out_->left.length == model_.Order() - 1);\n return prob_;\n }\n\n void Reset() {\n prob_ = 0.0;\n left_done_ = false;\n out_->left.length = 0;\n out_->right.length = 0;\n }\n void Reset(ChartState &replacement) {\n out_ = &replacement;\n Reset();\n }\n\n private:\n bool ExtendLeft(const ChartState &in, unsigned char &next_use, unsigned char extend_length, const float *back_in, float *back_out) {\n ProcessRet(model_.ExtendLeft(\n out_->right.words, out_->right.words + next_use, // Words to extend into\n back_in, // Backoffs to use\n in.left.pointers[extend_length - 1], extend_length, // Words to be extended\n back_out, // Backoffs for the next score\n next_use)); // Length of n-gram to use in next scoring. \n if (next_use != out_->right.length) {\n left_done_ = true;\n if (!next_use) {\n // Early exit. \n out_->right = in.right;\n prob_ += model_.UnRest(in.left.pointers + extend_length, in.left.pointers + in.left.length, extend_length + 1);\n return true;\n }\n }\n // Continue scoring. \n return false;\n }\n\n void ProcessRet(const FullScoreReturn &ret) {\n if (left_done_) {\n prob_ += ret.prob;\n return;\n }\n if (ret.independent_left) {\n prob_ += ret.prob;\n left_done_ = true;\n return;\n }\n out_->left.pointers[out_->left.length++] = ret.extend_left;\n prob_ += ret.rest;\n }\n\n const M &model_;\n\n ChartState *out_;\n\n bool left_done_;\n\n float prob_;\n};\n\n} // namespace ngram\n} // namespace lm\n\n#endif // LM_LEFT__\n" }, { "path": "src/kenlm/lm/left_test.cc", "content": "#include \"lm/left.hh\"\n#include \"lm/model.hh\"\n\n#include \"util/tokenize_piece.hh\"\n\n#include \n\n#define BOOST_TEST_MODULE LeftTest\n#include \n#include \n\nnamespace lm {\nnamespace ngram {\nnamespace {\n\n#define Term(word) score.Terminal(m.GetVocabulary().Index(word));\n#define VCheck(word, value) BOOST_CHECK_EQUAL(m.GetVocabulary().Index(word), value);\n\n// Apparently some Boost versions use templates and are pretty strict about types matching. \n#define SLOPPY_CHECK_CLOSE(ref, value, tol) BOOST_CHECK_CLOSE(static_cast(ref), static_cast(value), static_cast(tol));\n\ntemplate void Short(const M &m) {\n ChartState base;\n {\n RuleScore score(m, base);\n Term(\"more\");\n Term(\"loin\");\n SLOPPY_CHECK_CLOSE(-1.206319 - 0.3561665, score.Finish(), 0.001);\n }\n BOOST_CHECK(base.left.full);\n BOOST_CHECK_EQUAL(2, base.left.length);\n BOOST_CHECK_EQUAL(1, base.right.length);\n VCheck(\"loin\", base.right.words[0]);\n\n ChartState more_left;\n {\n RuleScore score(m, more_left);\n Term(\"little\");\n score.NonTerminal(base, -1.206319 - 0.3561665);\n // p(little more loin | null context)\n SLOPPY_CHECK_CLOSE(-1.56538, score.Finish(), 0.001);\n }\n BOOST_CHECK_EQUAL(3, more_left.left.length);\n BOOST_CHECK_EQUAL(1, more_left.right.length);\n VCheck(\"loin\", more_left.right.words[0]);\n BOOST_CHECK(more_left.left.full);\n\n ChartState shorter;\n {\n RuleScore score(m, shorter);\n Term(\"to\");\n score.NonTerminal(base, -1.206319 - 0.3561665);\n SLOPPY_CHECK_CLOSE(-0.30103 - 1.687872 - 1.206319 - 0.3561665, score.Finish(), 0.01);\n }\n BOOST_CHECK_EQUAL(1, shorter.left.length);\n BOOST_CHECK_EQUAL(1, shorter.right.length);\n VCheck(\"loin\", shorter.right.words[0]);\n BOOST_CHECK(shorter.left.full);\n}\n\ntemplate void Charge(const M &m) {\n ChartState base;\n {\n RuleScore score(m, base);\n Term(\"on\");\n Term(\"more\");\n SLOPPY_CHECK_CLOSE(-1.509559 -0.4771212 -1.206319, score.Finish(), 0.001);\n }\n BOOST_CHECK_EQUAL(1, base.left.length);\n BOOST_CHECK_EQUAL(1, base.right.length);\n VCheck(\"more\", base.right.words[0]);\n BOOST_CHECK(base.left.full);\n\n ChartState extend;\n {\n RuleScore score(m, extend);\n Term(\"looking\");\n score.NonTerminal(base, -1.509559 -0.4771212 -1.206319);\n SLOPPY_CHECK_CLOSE(-3.91039, score.Finish(), 0.001);\n }\n BOOST_CHECK_EQUAL(2, extend.left.length);\n BOOST_CHECK_EQUAL(1, extend.right.length);\n VCheck(\"more\", extend.right.words[0]);\n BOOST_CHECK(extend.left.full);\n\n ChartState tobos;\n {\n RuleScore score(m, tobos);\n score.BeginSentence();\n score.NonTerminal(extend, -3.91039);\n SLOPPY_CHECK_CLOSE(-3.471169, score.Finish(), 0.001);\n }\n BOOST_CHECK_EQUAL(0, tobos.left.length);\n BOOST_CHECK_EQUAL(1, tobos.right.length);\n}\n\ntemplate float LeftToRight(const M &m, const std::vector &words, bool begin_sentence = false) {\n float ret = 0.0;\n State right = begin_sentence ? m.BeginSentenceState() : m.NullContextState();\n for (std::vector::const_iterator i = words.begin(); i != words.end(); ++i) {\n State copy(right);\n ret += m.Score(copy, *i, right);\n }\n return ret;\n}\n\ntemplate float RightToLeft(const M &m, const std::vector &words, bool begin_sentence = false) {\n float ret = 0.0;\n ChartState state;\n state.left.length = 0;\n state.right.length = 0;\n state.left.full = false;\n for (std::vector::const_reverse_iterator i = words.rbegin(); i != words.rend(); ++i) {\n ChartState copy(state);\n RuleScore score(m, state);\n score.Terminal(*i);\n score.NonTerminal(copy, ret);\n ret = score.Finish();\n }\n if (begin_sentence) {\n ChartState copy(state);\n RuleScore score(m, state);\n score.BeginSentence();\n score.NonTerminal(copy, ret);\n ret = score.Finish();\n }\n return ret;\n}\n\ntemplate float TreeMiddle(const M &m, const std::vector &words, bool begin_sentence = false) {\n std::vector > states(words.size());\n for (unsigned int i = 0; i < words.size(); ++i) {\n RuleScore score(m, states[i].first);\n score.Terminal(words[i]);\n states[i].second = score.Finish();\n }\n while (states.size() > 1) {\n std::vector > upper((states.size() + 1) / 2);\n for (unsigned int i = 0; i < states.size() / 2; ++i) {\n RuleScore score(m, upper[i].first);\n score.NonTerminal(states[i*2].first, states[i*2].second);\n score.NonTerminal(states[i*2+1].first, states[i*2+1].second);\n upper[i].second = score.Finish();\n }\n if (states.size() % 2) {\n upper.back() = states.back();\n }\n std::swap(states, upper);\n }\n\n if (states.empty()) return 0.0;\n\n if (begin_sentence) {\n ChartState ignored;\n RuleScore score(m, ignored);\n score.BeginSentence();\n score.NonTerminal(states.front().first, states.front().second);\n return score.Finish();\n } else {\n return states.front().second;\n }\n\n}\n\ntemplate void LookupVocab(const M &m, const StringPiece &str, std::vector &out) {\n out.clear();\n for (util::TokenIter i(str, ' '); i; ++i) {\n out.push_back(m.GetVocabulary().Index(*i));\n }\n}\n\n#define TEXT_TEST(str) \\\n LookupVocab(m, str, words); \\\n expect = LeftToRight(m, words, rest); \\\n SLOPPY_CHECK_CLOSE(expect, RightToLeft(m, words, rest), 0.001); \\\n SLOPPY_CHECK_CLOSE(expect, TreeMiddle(m, words, rest), 0.001); \\\n\n// Build sentences, or parts thereof, from right to left. \ntemplate void GrowBig(const M &m, bool rest = false) {\n std::vector words;\n float expect;\n TEXT_TEST(\"in biarritz watching considering looking . on a little more loin also would consider higher to look good unknown the screening foo bar , unknown however unknown \");\n TEXT_TEST(\"on a little more loin also would consider higher to look good unknown the screening foo bar , unknown however unknown \");\n TEXT_TEST(\"on a little more loin also would consider higher to look good\");\n TEXT_TEST(\"more loin also would consider higher to look good\");\n TEXT_TEST(\"more loin also would consider higher to look\");\n TEXT_TEST(\"also would consider higher to look\");\n TEXT_TEST(\"also would consider higher\");\n TEXT_TEST(\"would consider higher to look\");\n TEXT_TEST(\"consider higher to look\");\n TEXT_TEST(\"consider higher to\");\n TEXT_TEST(\"consider higher\");\n}\n\ntemplate void GrowSmall(const M &m, bool rest = false) {\n std::vector words;\n float expect;\n TEXT_TEST(\"in biarritz watching considering looking . \");\n TEXT_TEST(\"in biarritz watching considering looking .\");\n TEXT_TEST(\"in biarritz\");\n}\n\ntemplate void AlsoWouldConsiderHigher(const M &m) {\n ChartState also;\n {\n RuleScore score(m, also);\n score.Terminal(m.GetVocabulary().Index(\"also\"));\n SLOPPY_CHECK_CLOSE(-1.687872, score.Finish(), 0.001);\n }\n ChartState would;\n {\n RuleScore score(m, would);\n score.Terminal(m.GetVocabulary().Index(\"would\"));\n SLOPPY_CHECK_CLOSE(-1.687872, score.Finish(), 0.001);\n }\n ChartState combine_also_would;\n {\n RuleScore score(m, combine_also_would);\n score.NonTerminal(also, -1.687872);\n score.NonTerminal(would, -1.687872);\n SLOPPY_CHECK_CLOSE(-1.687872 - 2.0, score.Finish(), 0.001);\n }\n BOOST_CHECK_EQUAL(2, combine_also_would.right.length);\n\n ChartState also_would;\n {\n RuleScore score(m, also_would);\n score.Terminal(m.GetVocabulary().Index(\"also\"));\n score.Terminal(m.GetVocabulary().Index(\"would\"));\n SLOPPY_CHECK_CLOSE(-1.687872 - 2.0, score.Finish(), 0.001);\n }\n BOOST_CHECK_EQUAL(2, also_would.right.length);\n\n ChartState consider;\n {\n RuleScore score(m, consider);\n score.Terminal(m.GetVocabulary().Index(\"consider\"));\n SLOPPY_CHECK_CLOSE(-1.687872, score.Finish(), 0.001);\n }\n BOOST_CHECK_EQUAL(1, consider.left.length);\n BOOST_CHECK_EQUAL(1, consider.right.length);\n BOOST_CHECK(!consider.left.full);\n\n ChartState higher;\n float higher_score;\n {\n RuleScore score(m, higher);\n score.Terminal(m.GetVocabulary().Index(\"higher\"));\n higher_score = score.Finish();\n }\n SLOPPY_CHECK_CLOSE(-1.509559, higher_score, 0.001);\n BOOST_CHECK_EQUAL(1, higher.left.length);\n BOOST_CHECK_EQUAL(1, higher.right.length);\n BOOST_CHECK(!higher.left.full);\n VCheck(\"higher\", higher.right.words[0]);\n SLOPPY_CHECK_CLOSE(-0.30103, higher.right.backoff[0], 0.001);\n\n ChartState consider_higher;\n {\n RuleScore score(m, consider_higher);\n score.NonTerminal(consider, -1.687872);\n score.NonTerminal(higher, higher_score);\n SLOPPY_CHECK_CLOSE(-1.509559 - 1.687872 - 0.30103, score.Finish(), 0.001);\n }\n BOOST_CHECK_EQUAL(2, consider_higher.left.length);\n BOOST_CHECK(!consider_higher.left.full);\n\n ChartState full;\n {\n RuleScore score(m, full);\n score.NonTerminal(combine_also_would, -1.687872 - 2.0);\n score.NonTerminal(consider_higher, -1.509559 - 1.687872 - 0.30103);\n SLOPPY_CHECK_CLOSE(-10.6879, score.Finish(), 0.001);\n }\n BOOST_CHECK_EQUAL(4, full.right.length);\n}\n\n#define CHECK_SCORE(str, val) \\\n{ \\\n float got = val; \\\n std::vector indices; \\\n LookupVocab(m, str, indices); \\\n SLOPPY_CHECK_CLOSE(LeftToRight(m, indices), got, 0.001); \\\n}\n\ntemplate void FullGrow(const M &m) {\n std::vector words;\n LookupVocab(m, \"in biarritz watching considering looking . \", words);\n\n ChartState lexical[7];\n float lexical_scores[7];\n for (unsigned int i = 0; i < 7; ++i) {\n RuleScore score(m, lexical[i]);\n score.Terminal(words[i]);\n lexical_scores[i] = score.Finish();\n }\n CHECK_SCORE(\"in\", lexical_scores[0]);\n CHECK_SCORE(\"biarritz\", lexical_scores[1]);\n CHECK_SCORE(\"watching\", lexical_scores[2]);\n CHECK_SCORE(\"\", lexical_scores[6]);\n\n ChartState l1[4];\n float l1_scores[4];\n {\n RuleScore score(m, l1[0]);\n score.NonTerminal(lexical[0], lexical_scores[0]);\n score.NonTerminal(lexical[1], lexical_scores[1]);\n CHECK_SCORE(\"in biarritz\", l1_scores[0] = score.Finish());\n }\n {\n RuleScore score(m, l1[1]);\n score.NonTerminal(lexical[2], lexical_scores[2]);\n score.NonTerminal(lexical[3], lexical_scores[3]);\n CHECK_SCORE(\"watching considering\", l1_scores[1] = score.Finish());\n }\n {\n RuleScore score(m, l1[2]);\n score.NonTerminal(lexical[4], lexical_scores[4]);\n score.NonTerminal(lexical[5], lexical_scores[5]);\n CHECK_SCORE(\"looking .\", l1_scores[2] = score.Finish());\n }\n BOOST_CHECK_EQUAL(l1[2].left.length, 1);\n l1[3] = lexical[6];\n l1_scores[3] = lexical_scores[6];\n\n ChartState l2[2];\n float l2_scores[2];\n {\n RuleScore score(m, l2[0]);\n score.NonTerminal(l1[0], l1_scores[0]);\n score.NonTerminal(l1[1], l1_scores[1]);\n CHECK_SCORE(\"in biarritz watching considering\", l2_scores[0] = score.Finish());\n }\n {\n RuleScore score(m, l2[1]);\n score.NonTerminal(l1[2], l1_scores[2]);\n score.NonTerminal(l1[3], l1_scores[3]);\n CHECK_SCORE(\"looking . \", l2_scores[1] = score.Finish());\n }\n BOOST_CHECK_EQUAL(l2[1].left.length, 1);\n BOOST_CHECK(l2[1].left.full);\n\n ChartState top;\n {\n RuleScore score(m, top);\n score.NonTerminal(l2[0], l2_scores[0]);\n score.NonTerminal(l2[1], l2_scores[1]);\n CHECK_SCORE(\"in biarritz watching considering looking . \", score.Finish());\n }\n}\n\nconst char *FileLocation() {\n if (boost::unit_test::framework::master_test_suite().argc < 2) {\n return \"test.arpa\";\n }\n return boost::unit_test::framework::master_test_suite().argv[1];\n}\n\ntemplate void Everything() {\n Config config;\n config.messages = NULL;\n M m(FileLocation(), config);\n\n Short(m);\n Charge(m);\n GrowBig(m);\n AlsoWouldConsiderHigher(m);\n GrowSmall(m);\n FullGrow(m);\n}\n\nBOOST_AUTO_TEST_CASE(ProbingAll) {\n Everything();\n}\nBOOST_AUTO_TEST_CASE(TrieAll) {\n Everything();\n}\nBOOST_AUTO_TEST_CASE(QuantTrieAll) {\n Everything();\n}\nBOOST_AUTO_TEST_CASE(ArrayQuantTrieAll) {\n Everything();\n}\nBOOST_AUTO_TEST_CASE(ArrayTrieAll) {\n Everything();\n}\n\nBOOST_AUTO_TEST_CASE(RestProbing) {\n Config config;\n config.messages = NULL;\n RestProbingModel m(FileLocation(), config);\n GrowBig(m, true);\n}\n\n} // namespace\n} // namespace ngram\n} // namespace lm\n" }, { "path": "src/kenlm/lm/lm_exception.cc", "content": "#include \"lm/lm_exception.hh\"\n\n#include\n#include\n\nnamespace lm {\n\nConfigException::ConfigException() throw() {}\nConfigException::~ConfigException() throw() {}\n\nLoadException::LoadException() throw() {}\nLoadException::~LoadException() throw() {}\n\nFormatLoadException::FormatLoadException() throw() {}\nFormatLoadException::~FormatLoadException() throw() {}\n\nVocabLoadException::VocabLoadException() throw() {}\nVocabLoadException::~VocabLoadException() throw() {}\n\nSpecialWordMissingException::SpecialWordMissingException() throw() {}\nSpecialWordMissingException::~SpecialWordMissingException() throw() {}\n\n} // namespace lm\n" }, { "path": "src/kenlm/lm/lm_exception.hh", "content": "#ifndef LM_LM_EXCEPTION__\n#define LM_LM_EXCEPTION__\n\n// Named to avoid conflict with util/exception.hh. \n\n#include \"util/exception.hh\"\n#include \"util/string_piece.hh\"\n\n#include \n#include \n\nnamespace lm {\n\ntypedef enum {THROW_UP, COMPLAIN, SILENT} WarningAction;\n\nclass ConfigException : public util::Exception {\n public:\n ConfigException() throw();\n ~ConfigException() throw();\n};\n\nclass LoadException : public util::Exception {\n public:\n virtual ~LoadException() throw();\n\n protected:\n LoadException() throw();\n};\n\nclass FormatLoadException : public LoadException {\n public:\n FormatLoadException() throw();\n ~FormatLoadException() throw();\n};\n\nclass VocabLoadException : public LoadException {\n public:\n virtual ~VocabLoadException() throw();\n VocabLoadException() throw();\n};\n\nclass SpecialWordMissingException : public VocabLoadException {\n public:\n explicit SpecialWordMissingException() throw();\n ~SpecialWordMissingException() throw();\n};\n\n} // namespace lm\n\n#endif // LM_LM_EXCEPTION\n" }, { "path": "src/kenlm/lm/max_order.hh", "content": "/* IF YOUR BUILD SYSTEM PASSES -DKENLM_MAX_ORDER, THEN CHANGE THE BUILD SYSTEM.\n * If not, this is the default maximum order. \n * Having this limit means that State can be\n * (kMaxOrder - 1) * sizeof(float) bytes instead of\n * sizeof(float*) + (kMaxOrder - 1) * sizeof(float) + malloc overhead\n */\n#ifndef KENLM_ORDER_MESSAGE\n#define KENLM_ORDER_MESSAGE \"If your build system supports changing KENLM_MAX_ORDER, change it there and recompile. In the KenLM tarball or Moses, use e.g. `bjam --max-kenlm-order=6 -a'. Otherwise, edit lm/max_order.hh.\"\n#endif\n" }, { "path": "src/kenlm/lm/model.cc", "content": "#include \"lm/model.hh\"\n\n#include \"lm/blank.hh\"\n#include \"lm/lm_exception.hh\"\n#include \"lm/search_hashed.hh\"\n#include \"lm/search_trie.hh\"\n#include \"lm/read_arpa.hh\"\n#include \"util/have.hh\"\n#include \"util/murmur_hash.hh\"\n\n#include \n#include \n#include \n#include \n#include \n\nnamespace lm {\nnamespace ngram {\nnamespace detail {\n\ntemplate const ModelType GenericModel::kModelType = Search::kModelType;\n\ntemplate uint64_t GenericModel::Size(const std::vector &counts, const Config &config) {\n return VocabularyT::Size(counts[0], config) + Search::Size(counts, config);\n}\n\ntemplate void GenericModel::SetupMemory(void *base, const std::vector &counts, const Config &config) {\n size_t goal_size = util::CheckOverflow(Size(counts, config));\n uint8_t *start = static_cast(base);\n size_t allocated = VocabularyT::Size(counts[0], config);\n vocab_.SetupMemory(start, allocated, counts[0], config);\n start += allocated;\n start = search_.SetupMemory(start, counts, config);\n if (static_cast(start - static_cast(base)) != goal_size) UTIL_THROW(FormatLoadException, \"The data structures took \" << (start - static_cast(base)) << \" but Size says they should take \" << goal_size);\n}\n\ntemplate GenericModel::GenericModel(const char *file, const Config &config) {\n LoadLM(file, config, *this);\n\n // g++ prints warnings unless these are fully initialized.\n State begin_sentence = State();\n begin_sentence.length = 1;\n begin_sentence.words[0] = vocab_.BeginSentence();\n typename Search::Node ignored_node;\n bool ignored_independent_left;\n uint64_t ignored_extend_left;\n begin_sentence.backoff[0] = search_.LookupUnigram(begin_sentence.words[0], ignored_node, ignored_independent_left, ignored_extend_left).Backoff();\n State null_context = State();\n null_context.length = 0;\n P::Init(begin_sentence, null_context, vocab_, search_.Order());\n}\n\nnamespace {\nvoid CheckCounts(const std::vector &counts) {\n UTIL_THROW_IF(counts.size() > KENLM_MAX_ORDER, FormatLoadException, \"This model has order \" << counts.size() << \" but KenLM was compiled to support up to \" << KENLM_MAX_ORDER << \". \" << KENLM_ORDER_MESSAGE);\n if (sizeof(uint64_t) > sizeof(std::size_t)) {\n for (std::vector::const_iterator i = counts.begin(); i != counts.end(); ++i) {\n UTIL_THROW_IF(*i > static_cast(std::numeric_limits::max()), util::OverflowException, \"This model has \" << *i << \" \" << (i - counts.begin() + 1) << \"-grams which is too many for 32-bit machines.\");\n }\n }\n}\n} // namespace\n\ntemplate void GenericModel::InitializeFromBinary(void *start, const Parameters ¶ms, const Config &config, int fd) {\n CheckCounts(params.counts);\n SetupMemory(start, params.counts, config);\n vocab_.LoadedBinary(params.fixed.has_vocabulary, fd, config.enumerate_vocab);\n search_.LoadedBinary();\n}\n\ntemplate void GenericModel::InitializeFromARPA(const char *file, const Config &config) {\n // Backing file is the ARPA. Steal it so we can make the backing file the mmap output if any.\n util::FilePiece f(backing_.file.release(), file, config.ProgressMessages());\n try {\n std::vector counts;\n // File counts do not include pruned trigrams that extend to quadgrams etc. These will be fixed by search_.\n ReadARPACounts(f, counts);\n CheckCounts(counts);\n if (counts.size() < 2) UTIL_THROW(FormatLoadException, \"This ngram implementation assumes at least a bigram model.\");\n if (config.probing_multiplier <= 1.0) UTIL_THROW(ConfigException, \"probing multiplier must be > 1.0\");\n\n std::size_t vocab_size = util::CheckOverflow(VocabularyT::Size(counts[0], config));\n // Setup the binary file for writing the vocab lookup table. The search_ is responsible for growing the binary file to its needs.\n vocab_.SetupMemory(SetupJustVocab(config, counts.size(), vocab_size, backing_), vocab_size, counts[0], config);\n\n if (config.write_mmap) {\n WriteWordsWrapper wrap(config.enumerate_vocab);\n vocab_.ConfigureEnumerate(&wrap, counts[0]);\n search_.InitializeFromARPA(file, f, counts, config, vocab_, backing_);\n wrap.Write(backing_.file.get(), backing_.vocab.size() + vocab_.UnkCountChangePadding() + Search::Size(counts, config));\n } else {\n vocab_.ConfigureEnumerate(config.enumerate_vocab, counts[0]);\n search_.InitializeFromARPA(file, f, counts, config, vocab_, backing_);\n }\n\n if (!vocab_.SawUnk()) {\n assert(config.unknown_missing != THROW_UP);\n // Default probabilities for unknown.\n search_.UnknownUnigram().backoff = 0.0;\n search_.UnknownUnigram().prob = config.unknown_missing_logprob;\n }\n FinishFile(config, kModelType, kVersion, counts, vocab_.UnkCountChangePadding(), backing_);\n } catch (util::Exception &e) {\n e << \" Byte: \" << f.Offset();\n throw;\n }\n}\n\ntemplate void GenericModel::UpdateConfigFromBinary(int fd, const std::vector &counts, Config &config) {\n util::AdvanceOrThrow(fd, VocabularyT::Size(counts[0], config));\n Search::UpdateConfigFromBinary(fd, counts, config);\n}\n\ntemplate FullScoreReturn GenericModel::FullScore(const State &in_state, const WordIndex new_word, State &out_state) const {\n FullScoreReturn ret = ScoreExceptBackoff(in_state.words, in_state.words + in_state.length, new_word, out_state);\n for (const float *i = in_state.backoff + ret.ngram_length - 1; i < in_state.backoff + in_state.length; ++i) {\n ret.prob += *i;\n }\n return ret;\n}\n\ntemplate FullScoreReturn GenericModel::FullScoreForgotState(const WordIndex *context_rbegin, const WordIndex *context_rend, const WordIndex new_word, State &out_state) const {\n context_rend = std::min(context_rend, context_rbegin + P::Order() - 1);\n FullScoreReturn ret = ScoreExceptBackoff(context_rbegin, context_rend, new_word, out_state);\n\n // Add the backoff weights for n-grams of order start to (context_rend - context_rbegin).\n unsigned char start = ret.ngram_length;\n if (context_rend - context_rbegin < static_cast(start)) return ret;\n\n bool independent_left;\n uint64_t extend_left;\n typename Search::Node node;\n if (start <= 1) {\n ret.prob += search_.LookupUnigram(*context_rbegin, node, independent_left, extend_left).Backoff();\n start = 2;\n } else if (!search_.FastMakeNode(context_rbegin, context_rbegin + start - 1, node)) {\n return ret;\n }\n // i is the order of the backoff we're looking for.\n unsigned char order_minus_2 = start - 2;\n for (const WordIndex *i = context_rbegin + start - 1; i < context_rend; ++i, ++order_minus_2) {\n typename Search::MiddlePointer p(search_.LookupMiddle(order_minus_2, *i, node, independent_left, extend_left));\n if (!p.Found()) break;\n ret.prob += p.Backoff();\n }\n return ret;\n}\n\ntemplate void GenericModel::GetState(const WordIndex *context_rbegin, const WordIndex *context_rend, State &out_state) const {\n // Generate a state from context.\n context_rend = std::min(context_rend, context_rbegin + P::Order() - 1);\n if (context_rend == context_rbegin) {\n out_state.length = 0;\n return;\n }\n typename Search::Node node;\n bool independent_left;\n uint64_t extend_left;\n out_state.backoff[0] = search_.LookupUnigram(*context_rbegin, node, independent_left, extend_left).Backoff();\n out_state.length = HasExtension(out_state.backoff[0]) ? 1 : 0;\n float *backoff_out = out_state.backoff + 1;\n unsigned char order_minus_2 = 0;\n for (const WordIndex *i = context_rbegin + 1; i < context_rend; ++i, ++backoff_out, ++order_minus_2) {\n typename Search::MiddlePointer p(search_.LookupMiddle(order_minus_2, *i, node, independent_left, extend_left));\n if (!p.Found()) {\n std::copy(context_rbegin, context_rbegin + out_state.length, out_state.words);\n return;\n }\n *backoff_out = p.Backoff();\n if (HasExtension(*backoff_out)) out_state.length = i - context_rbegin + 1;\n }\n std::copy(context_rbegin, context_rbegin + out_state.length, out_state.words);\n}\n\ntemplate FullScoreReturn GenericModel::ExtendLeft(\n const WordIndex *add_rbegin, const WordIndex *add_rend,\n const float *backoff_in,\n uint64_t extend_pointer,\n unsigned char extend_length,\n float *backoff_out,\n unsigned char &next_use) const {\n FullScoreReturn ret;\n typename Search::Node node;\n if (extend_length == 1) {\n typename Search::UnigramPointer ptr(search_.LookupUnigram(static_cast(extend_pointer), node, ret.independent_left, ret.extend_left));\n ret.rest = ptr.Rest();\n ret.prob = ptr.Prob();\n assert(!ret.independent_left);\n } else {\n typename Search::MiddlePointer ptr(search_.Unpack(extend_pointer, extend_length, node));\n ret.rest = ptr.Rest();\n ret.prob = ptr.Prob();\n ret.extend_left = extend_pointer;\n // If this function is called, then it does depend on left words.\n ret.independent_left = false;\n }\n float subtract_me = ret.rest;\n ret.ngram_length = extend_length;\n next_use = extend_length;\n ResumeScore(add_rbegin, add_rend, extend_length - 1, node, backoff_out, next_use, ret);\n next_use -= extend_length;\n // Charge backoffs.\n for (const float *b = backoff_in + ret.ngram_length - extend_length; b < backoff_in + (add_rend - add_rbegin); ++b) ret.prob += *b;\n ret.prob -= subtract_me;\n ret.rest -= subtract_me;\n return ret;\n}\n\nnamespace {\n// Do a paraonoid copy of history, assuming new_word has already been copied\n// (hence the -1). out_state.length could be zero so I avoided using\n// std::copy.\nvoid CopyRemainingHistory(const WordIndex *from, State &out_state) {\n WordIndex *out = out_state.words + 1;\n const WordIndex *in_end = from + static_cast(out_state.length) - 1;\n for (const WordIndex *in = from; in < in_end; ++in, ++out) *out = *in;\n}\n} // namespace\n\n/* Ugly optimized function. Produce a score excluding backoff.\n * The search goes in increasing order of ngram length.\n * Context goes backward, so context_begin is the word immediately preceeding\n * new_word.\n */\ntemplate FullScoreReturn GenericModel::ScoreExceptBackoff(\n const WordIndex *const context_rbegin,\n const WordIndex *const context_rend,\n const WordIndex new_word,\n State &out_state) const {\n assert(new_word < vocab_.Bound());\n FullScoreReturn ret;\n // ret.ngram_length contains the last known non-blank ngram length.\n ret.ngram_length = 1;\n\n typename Search::Node node;\n typename Search::UnigramPointer uni(search_.LookupUnigram(new_word, node, ret.independent_left, ret.extend_left));\n out_state.backoff[0] = uni.Backoff();\n ret.prob = uni.Prob();\n ret.rest = uni.Rest();\n\n // This is the length of the context that should be used for continuation to the right.\n out_state.length = HasExtension(out_state.backoff[0]) ? 1 : 0;\n // We'll write the word anyway since it will probably be used and does no harm being there.\n out_state.words[0] = new_word;\n if (context_rbegin == context_rend) return ret;\n\n ResumeScore(context_rbegin, context_rend, 0, node, out_state.backoff + 1, out_state.length, ret);\n CopyRemainingHistory(context_rbegin, out_state);\n return ret;\n}\n\ntemplate void GenericModel::ResumeScore(const WordIndex *hist_iter, const WordIndex *const context_rend, unsigned char order_minus_2, typename Search::Node &node, float *backoff_out, unsigned char &next_use, FullScoreReturn &ret) const {\n for (; ; ++order_minus_2, ++hist_iter, ++backoff_out) {\n if (hist_iter == context_rend) return;\n if (ret.independent_left) return;\n if (order_minus_2 == P::Order() - 2) break;\n\n typename Search::MiddlePointer pointer(search_.LookupMiddle(order_minus_2, *hist_iter, node, ret.independent_left, ret.extend_left));\n if (!pointer.Found()) return;\n *backoff_out = pointer.Backoff();\n ret.prob = pointer.Prob();\n ret.rest = pointer.Rest();\n ret.ngram_length = order_minus_2 + 2;\n if (HasExtension(*backoff_out)) {\n next_use = ret.ngram_length;\n }\n }\n ret.independent_left = true;\n typename Search::LongestPointer longest(search_.LookupLongest(*hist_iter, node));\n if (longest.Found()) {\n ret.prob = longest.Prob();\n ret.rest = ret.prob;\n // There is no blank in longest_.\n ret.ngram_length = P::Order();\n }\n}\n\ntemplate float GenericModel::InternalUnRest(const uint64_t *pointers_begin, const uint64_t *pointers_end, unsigned char first_length) const {\n float ret;\n typename Search::Node node;\n if (first_length == 1) {\n if (pointers_begin >= pointers_end) return 0.0;\n bool independent_left;\n uint64_t extend_left;\n typename Search::UnigramPointer ptr(search_.LookupUnigram(static_cast(*pointers_begin), node, independent_left, extend_left));\n ret = ptr.Prob() - ptr.Rest();\n ++first_length;\n ++pointers_begin;\n } else {\n ret = 0.0;\n }\n for (const uint64_t *i = pointers_begin; i < pointers_end; ++i, ++first_length) {\n typename Search::MiddlePointer ptr(search_.Unpack(*i, first_length, node));\n ret += ptr.Prob() - ptr.Rest();\n }\n return ret;\n}\n\ntemplate class GenericModel, ProbingVocabulary>;\ntemplate class GenericModel, ProbingVocabulary>;\ntemplate class GenericModel, SortedVocabulary>;\ntemplate class GenericModel, SortedVocabulary>;\ntemplate class GenericModel, SortedVocabulary>;\ntemplate class GenericModel, SortedVocabulary>;\n\n} // namespace detail\n} // namespace ngram\n} // namespace lm\n" }, { "path": "src/kenlm/lm/model.hh", "content": "#ifndef LM_MODEL__\n#define LM_MODEL__\n\n#include \"lm/bhiksha.hh\"\n#include \"lm/binary_format.hh\"\n#include \"lm/config.hh\"\n#include \"lm/facade.hh\"\n#include \"lm/quantize.hh\"\n#include \"lm/search_hashed.hh\"\n#include \"lm/search_trie.hh\"\n#include \"lm/state.hh\"\n#include \"lm/value.hh\"\n#include \"lm/vocab.hh\"\n#include \"lm/weights.hh\"\n\n#include \"util/murmur_hash.hh\"\n\n#include \n#include \n\n#include \n\nnamespace util { class FilePiece; }\n\nnamespace lm {\nnamespace ngram {\nnamespace detail {\n\n// Should return the same results as SRI. \n// ModelFacade typedefs Vocabulary so we use VocabularyT to avoid naming conflicts.\ntemplate class GenericModel : public base::ModelFacade, State, VocabularyT> {\n private:\n typedef base::ModelFacade, State, VocabularyT> P;\n public:\n // This is the model type returned by RecognizeBinary.\n static const ModelType kModelType;\n\n static const unsigned int kVersion = Search::kVersion;\n\n /* Get the size of memory that will be mapped given ngram counts. This\n * does not include small non-mapped control structures, such as this class\n * itself. \n */\n static uint64_t Size(const std::vector &counts, const Config &config = Config());\n\n /* Load the model from a file. It may be an ARPA or binary file. Binary\n * files must have the format expected by this class or you'll get an\n * exception. So TrieModel can only load ARPA or binary created by\n * TrieModel. To classify binary files, call RecognizeBinary in\n * lm/binary_format.hh. \n */\n explicit GenericModel(const char *file, const Config &config = Config());\n\n /* Score p(new_word | in_state) and incorporate new_word into out_state.\n * Note that in_state and out_state must be different references:\n * &in_state != &out_state. \n */\n FullScoreReturn FullScore(const State &in_state, const WordIndex new_word, State &out_state) const;\n\n /* Slower call without in_state. Try to remember state, but sometimes it\n * would cost too much memory or your decoder isn't setup properly. \n * To use this function, make an array of WordIndex containing the context\n * vocabulary ids in reverse order. Then, pass the bounds of the array:\n * [context_rbegin, context_rend). The new_word is not part of the context\n * array unless you intend to repeat words. \n */\n FullScoreReturn FullScoreForgotState(const WordIndex *context_rbegin, const WordIndex *context_rend, const WordIndex new_word, State &out_state) const;\n\n /* Get the state for a context. Don't use this if you can avoid it. Use\n * BeginSentenceState or EmptyContextState and extend from those. If\n * you're only going to use this state to call FullScore once, use\n * FullScoreForgotState. \n * To use this function, make an array of WordIndex containing the context\n * vocabulary ids in reverse order. Then, pass the bounds of the array:\n * [context_rbegin, context_rend). \n */\n void GetState(const WordIndex *context_rbegin, const WordIndex *context_rend, State &out_state) const;\n\n /* More efficient version of FullScore where a partial n-gram has already\n * been scored. \n * NOTE: THE RETURNED .rest AND .prob ARE RELATIVE TO THE .rest RETURNED BEFORE. \n */\n FullScoreReturn ExtendLeft(\n // Additional context in reverse order. This will update add_rend to \n const WordIndex *add_rbegin, const WordIndex *add_rend,\n // Backoff weights to use. \n const float *backoff_in,\n // extend_left returned by a previous query.\n uint64_t extend_pointer,\n // Length of n-gram that the pointer corresponds to. \n unsigned char extend_length,\n // Where to write additional backoffs for [extend_length + 1, min(Order() - 1, return.ngram_length)]\n float *backoff_out,\n // Amount of additional content that should be considered by the next call.\n unsigned char &next_use) const;\n\n /* Return probabilities minus rest costs for an array of pointers. The\n * first length should be the length of the n-gram to which pointers_begin\n * points. \n */\n float UnRest(const uint64_t *pointers_begin, const uint64_t *pointers_end, unsigned char first_length) const {\n // Compiler should optimize this if away. \n return Search::kDifferentRest ? InternalUnRest(pointers_begin, pointers_end, first_length) : 0.0;\n }\n\n private:\n friend void lm::ngram::LoadLM<>(const char *file, const Config &config, GenericModel &to);\n\n static void UpdateConfigFromBinary(int fd, const std::vector &counts, Config &config);\n\n FullScoreReturn ScoreExceptBackoff(const WordIndex *const context_rbegin, const WordIndex *const context_rend, const WordIndex new_word, State &out_state) const;\n\n // Score bigrams and above. Do not include backoff. \n void ResumeScore(const WordIndex *context_rbegin, const WordIndex *const context_rend, unsigned char starting_order_minus_2, typename Search::Node &node, float *backoff_out, unsigned char &next_use, FullScoreReturn &ret) const;\n\n // Appears after Size in the cc file.\n void SetupMemory(void *start, const std::vector &counts, const Config &config);\n\n void InitializeFromBinary(void *start, const Parameters ¶ms, const Config &config, int fd);\n\n void InitializeFromARPA(const char *file, const Config &config);\n\n float InternalUnRest(const uint64_t *pointers_begin, const uint64_t *pointers_end, unsigned char first_length) const;\n\n Backing &MutableBacking() { return backing_; }\n\n Backing backing_;\n \n VocabularyT vocab_;\n\n Search search_;\n};\n\n} // namespace detail\n\n// Instead of typedef, inherit. This allows the Model etc to be forward declared. \n// Oh the joys of C and C++. \n#define LM_COMMA() ,\n#define LM_NAME_MODEL(name, from)\\\nclass name : public from {\\\n public:\\\n name(const char *file, const Config &config = Config()) : from(file, config) {}\\\n};\n\nLM_NAME_MODEL(ProbingModel, detail::GenericModel LM_COMMA() ProbingVocabulary>);\nLM_NAME_MODEL(RestProbingModel, detail::GenericModel LM_COMMA() ProbingVocabulary>);\nLM_NAME_MODEL(TrieModel, detail::GenericModel LM_COMMA() SortedVocabulary>);\nLM_NAME_MODEL(ArrayTrieModel, detail::GenericModel LM_COMMA() SortedVocabulary>);\nLM_NAME_MODEL(QuantTrieModel, detail::GenericModel LM_COMMA() SortedVocabulary>);\nLM_NAME_MODEL(QuantArrayTrieModel, detail::GenericModel LM_COMMA() SortedVocabulary>);\n\n// Default implementation. No real reason for it to be the default. \ntypedef ::lm::ngram::ProbingVocabulary Vocabulary;\ntypedef ProbingModel Model;\n\n} // namespace ngram\n} // namespace lm\n\n#endif // LM_MODEL__\n" }, { "path": "src/kenlm/lm/model_test.cc", "content": "#include \"lm/model.hh\"\n\n#include \n#include \n\n#define BOOST_TEST_MODULE ModelTest\n#include \n#include \n\n// Apparently some Boost versions use templates and are pretty strict about types matching. \n#define SLOPPY_CHECK_CLOSE(ref, value, tol) BOOST_CHECK_CLOSE(static_cast(ref), static_cast(value), static_cast(tol));\n\nnamespace lm {\nnamespace ngram {\n\nstd::ostream &operator<<(std::ostream &o, const State &state) {\n o << \"State length \" << static_cast(state.length) << ':';\n for (const WordIndex *i = state.words; i < state.words + state.length; ++i) {\n o << ' ' << *i;\n }\n return o;\n}\n\nnamespace {\n\n// Stupid bjam reverses the command line arguments randomly.\nconst char *TestLocation() {\n if (boost::unit_test::framework::master_test_suite().argc < 3) {\n return \"test.arpa\";\n }\n char **argv = boost::unit_test::framework::master_test_suite().argv;\n return argv[strstr(argv[1], \"nounk\") ? 2 : 1];\n}\nconst char *TestNoUnkLocation() {\n if (boost::unit_test::framework::master_test_suite().argc < 3) {\n return \"test_nounk.arpa\";\n }\n char **argv = boost::unit_test::framework::master_test_suite().argv;\n return argv[strstr(argv[1], \"nounk\") ? 1 : 2];\n}\n\ntemplate State GetState(const Model &model, const char *word, const State &in) {\n WordIndex context[in.length + 1];\n context[0] = model.GetVocabulary().Index(word);\n std::copy(in.words, in.words + in.length, context + 1);\n State ret;\n model.GetState(context, context + in.length + 1, ret);\n return ret;\n}\n\n#define StartTest(word, ngram, score, indep_left) \\\n ret = model.FullScore( \\\n state, \\\n model.GetVocabulary().Index(word), \\\n out);\\\n SLOPPY_CHECK_CLOSE(score, ret.prob, 0.001); \\\n BOOST_CHECK_EQUAL(static_cast(ngram), ret.ngram_length); \\\n BOOST_CHECK_GE(std::min(ngram, 5 - 1), out.length); \\\n BOOST_CHECK_EQUAL(indep_left, ret.independent_left); \\\n BOOST_CHECK_EQUAL(out, GetState(model, word, state));\n\n#define AppendTest(word, ngram, score, indep_left) \\\n StartTest(word, ngram, score, indep_left) \\\n state = out;\n\ntemplate void Starters(const M &model) {\n FullScoreReturn ret;\n Model::State state(model.BeginSentenceState());\n Model::State out;\n\n StartTest(\"looking\", 2, -0.4846522, true);\n\n // , probability plus backoff\n StartTest(\",\", 1, -1.383514 + -0.4149733, true);\n // probability plus backoff\n StartTest(\"this_is_not_found\", 1, -1.995635 + -0.4149733, true);\n}\n\ntemplate void Continuation(const M &model) {\n FullScoreReturn ret;\n Model::State state(model.BeginSentenceState());\n Model::State out;\n\n AppendTest(\"looking\", 2, -0.484652, true);\n AppendTest(\"on\", 3, -0.348837, true);\n AppendTest(\"a\", 4, -0.0155266, true);\n AppendTest(\"little\", 5, -0.00306122, true);\n State preserve = state;\n AppendTest(\"the\", 1, -4.04005, true);\n AppendTest(\"biarritz\", 1, -1.9889, true);\n AppendTest(\"not_found\", 1, -2.29666, true);\n AppendTest(\"more\", 1, -1.20632 - 20.0, true);\n AppendTest(\".\", 2, -0.51363, true);\n AppendTest(\"\", 3, -0.0191651, true);\n BOOST_CHECK_EQUAL(0, state.length);\n\n state = preserve;\n AppendTest(\"more\", 5, -0.00181395, true);\n BOOST_CHECK_EQUAL(4, state.length);\n AppendTest(\"loin\", 5, -0.0432557, true);\n BOOST_CHECK_EQUAL(1, state.length);\n}\n\ntemplate void Blanks(const M &model) {\n FullScoreReturn ret;\n State state(model.NullContextState());\n State out;\n AppendTest(\"also\", 1, -1.687872, false);\n AppendTest(\"would\", 2, -2, true);\n AppendTest(\"consider\", 3, -3, true);\n State preserve = state;\n AppendTest(\"higher\", 4, -4, true);\n AppendTest(\"looking\", 5, -5, true);\n BOOST_CHECK_EQUAL(1, state.length);\n\n state = preserve;\n // also would consider not_found\n AppendTest(\"not_found\", 1, -1.995635 - 7.0 - 0.30103, true);\n\n state = model.NullContextState();\n // higher looking is a blank. \n AppendTest(\"higher\", 1, -1.509559, false);\n AppendTest(\"looking\", 2, -1.285941 - 0.30103, false);\n\n State higher_looking = state;\n\n BOOST_CHECK_EQUAL(1, state.length);\n AppendTest(\"not_found\", 1, -1.995635 - 0.4771212, true);\n\n state = higher_looking;\n // higher looking consider\n AppendTest(\"consider\", 1, -1.687872 - 0.4771212, true);\n\n state = model.NullContextState();\n AppendTest(\"would\", 1, -1.687872, false);\n BOOST_CHECK_EQUAL(1, state.length);\n AppendTest(\"consider\", 2, -1.687872 -0.30103, false);\n BOOST_CHECK_EQUAL(2, state.length);\n AppendTest(\"higher\", 3, -1.509559 - 0.30103, false);\n BOOST_CHECK_EQUAL(3, state.length);\n AppendTest(\"looking\", 4, -1.285941 - 0.30103, false);\n}\n\ntemplate void Unknowns(const M &model) {\n FullScoreReturn ret;\n State state(model.NullContextState());\n State out;\n\n AppendTest(\"not_found\", 1, -1.995635, false);\n State preserve = state;\n AppendTest(\"not_found2\", 2, -15.0, true);\n AppendTest(\"not_found3\", 2, -15.0 - 2.0, true);\n \n state = preserve;\n AppendTest(\"however\", 2, -4, true);\n AppendTest(\"not_found3\", 3, -6, true);\n}\n\ntemplate void MinimalState(const M &model) {\n FullScoreReturn ret;\n State state(model.NullContextState());\n State out;\n\n AppendTest(\"baz\", 1, -6.535897, true);\n BOOST_CHECK_EQUAL(0, state.length);\n state = model.NullContextState();\n AppendTest(\"foo\", 1, -3.141592, true);\n BOOST_CHECK_EQUAL(1, state.length);\n AppendTest(\"bar\", 2, -6.0, true);\n // Has to include the backoff weight. \n BOOST_CHECK_EQUAL(1, state.length);\n AppendTest(\"bar\", 1, -2.718281 + 3.0, true);\n BOOST_CHECK_EQUAL(1, state.length);\n\n state = model.NullContextState();\n AppendTest(\"to\", 1, -1.687872, false);\n AppendTest(\"look\", 2, -0.2922095, true);\n BOOST_CHECK_EQUAL(2, state.length);\n AppendTest(\"good\", 3, -7, true);\n}\n\ntemplate void ExtendLeftTest(const M &model) {\n State right;\n FullScoreReturn little(model.FullScore(model.NullContextState(), model.GetVocabulary().Index(\"little\"), right));\n const float kLittleProb = -1.285941;\n SLOPPY_CHECK_CLOSE(kLittleProb, little.prob, 0.001);\n unsigned char next_use;\n float backoff_out[4];\n\n FullScoreReturn extend_none(model.ExtendLeft(NULL, NULL, NULL, little.extend_left, 1, NULL, next_use));\n BOOST_CHECK_EQUAL(0, next_use);\n BOOST_CHECK_EQUAL(little.extend_left, extend_none.extend_left);\n SLOPPY_CHECK_CLOSE(little.prob - little.rest, extend_none.prob, 0.001);\n BOOST_CHECK_EQUAL(1, extend_none.ngram_length);\n\n const WordIndex a = model.GetVocabulary().Index(\"a\");\n float backoff_in = 3.14;\n // a little\n FullScoreReturn extend_a(model.ExtendLeft(&a, &a + 1, &backoff_in, little.extend_left, 1, backoff_out, next_use));\n BOOST_CHECK_EQUAL(1, next_use);\n SLOPPY_CHECK_CLOSE(-0.69897, backoff_out[0], 0.001);\n SLOPPY_CHECK_CLOSE(-0.09132547 - little.rest, extend_a.prob, 0.001);\n BOOST_CHECK_EQUAL(2, extend_a.ngram_length);\n BOOST_CHECK(!extend_a.independent_left);\n\n const WordIndex on = model.GetVocabulary().Index(\"on\");\n FullScoreReturn extend_on(model.ExtendLeft(&on, &on + 1, &backoff_in, extend_a.extend_left, 2, backoff_out, next_use));\n BOOST_CHECK_EQUAL(1, next_use);\n SLOPPY_CHECK_CLOSE(-0.4771212, backoff_out[0], 0.001);\n SLOPPY_CHECK_CLOSE(-0.0283603 - (extend_a.rest + little.rest), extend_on.prob, 0.001);\n BOOST_CHECK_EQUAL(3, extend_on.ngram_length);\n BOOST_CHECK(!extend_on.independent_left);\n\n const WordIndex both[2] = {a, on};\n float backoff_in_arr[4];\n FullScoreReturn extend_both(model.ExtendLeft(both, both + 2, backoff_in_arr, little.extend_left, 1, backoff_out, next_use));\n BOOST_CHECK_EQUAL(2, next_use);\n SLOPPY_CHECK_CLOSE(-0.69897, backoff_out[0], 0.001);\n SLOPPY_CHECK_CLOSE(-0.4771212, backoff_out[1], 0.001);\n SLOPPY_CHECK_CLOSE(-0.0283603 - little.rest, extend_both.prob, 0.001);\n BOOST_CHECK_EQUAL(3, extend_both.ngram_length);\n BOOST_CHECK(!extend_both.independent_left);\n BOOST_CHECK_EQUAL(extend_on.extend_left, extend_both.extend_left);\n}\n\n#define StatelessTest(word, provide, ngram, score) \\\n ret = model.FullScoreForgotState(indices + num_words - word, indices + num_words - word + provide, indices[num_words - word - 1], state); \\\n SLOPPY_CHECK_CLOSE(score, ret.prob, 0.001); \\\n BOOST_CHECK_EQUAL(static_cast(ngram), ret.ngram_length); \\\n model.GetState(indices + num_words - word, indices + num_words - word + provide, before); \\\n ret = model.FullScore(before, indices[num_words - word - 1], out); \\\n BOOST_CHECK(state == out); \\\n SLOPPY_CHECK_CLOSE(score, ret.prob, 0.001); \\\n BOOST_CHECK_EQUAL(static_cast(ngram), ret.ngram_length);\n\ntemplate void Stateless(const M &model) {\n const char *words[] = {\"\", \"looking\", \"on\", \"a\", \"little\", \"the\", \"biarritz\", \"not_found\", \"more\", \".\", \"\"};\n const size_t num_words = sizeof(words) / sizeof(const char*);\n // Silience \"array subscript is above array bounds\" when extracting end pointer.\n WordIndex indices[num_words + 1];\n for (unsigned int i = 0; i < num_words; ++i) {\n indices[num_words - 1 - i] = model.GetVocabulary().Index(words[i]);\n }\n FullScoreReturn ret;\n State state, out, before;\n\n ret = model.FullScoreForgotState(indices + num_words - 1, indices + num_words, indices[num_words - 2], state);\n SLOPPY_CHECK_CLOSE(-0.484652, ret.prob, 0.001);\n StatelessTest(1, 1, 2, -0.484652);\n\n // looking\n StatelessTest(1, 2, 2, -0.484652);\n // on\n AppendTest(\"on\", 3, -0.348837, true);\n StatelessTest(2, 3, 3, -0.348837);\n StatelessTest(2, 2, 3, -0.348837);\n StatelessTest(2, 1, 2, -0.4638903);\n // a\n StatelessTest(3, 4, 4, -0.0155266);\n // little\n AppendTest(\"little\", 5, -0.00306122, true);\n StatelessTest(4, 5, 5, -0.00306122);\n // the\n AppendTest(\"the\", 1, -4.04005, true);\n StatelessTest(5, 5, 1, -4.04005);\n // No context of the. \n StatelessTest(5, 0, 1, -1.687872);\n // biarritz\n StatelessTest(6, 1, 1, -1.9889);\n // not found\n StatelessTest(7, 1, 1, -2.29666);\n StatelessTest(7, 0, 1, -1.995635);\n\n WordIndex unk[1];\n unk[0] = 0;\n model.GetState(unk, unk + 1, state);\n BOOST_CHECK_EQUAL(1, state.length);\n BOOST_CHECK_EQUAL(static_cast(0), state.words[0]);\n}\n\ntemplate void NoUnkCheck(const M &model) {\n WordIndex unk_index = 0;\n State state;\n\n FullScoreReturn ret = model.FullScoreForgotState(&unk_index, &unk_index + 1, unk_index, state);\n SLOPPY_CHECK_CLOSE(-100.0, ret.prob, 0.001);\n}\n\ntemplate void Everything(const M &m) {\n Starters(m);\n Continuation(m);\n Blanks(m);\n Unknowns(m);\n MinimalState(m);\n ExtendLeftTest(m);\n Stateless(m);\n}\n\nclass ExpectEnumerateVocab : public EnumerateVocab {\n public:\n ExpectEnumerateVocab() {}\n\n void Add(WordIndex index, const StringPiece &str) {\n BOOST_CHECK_EQUAL(seen.size(), index);\n seen.push_back(std::string(str.data(), str.length()));\n }\n\n void Check(const base::Vocabulary &vocab) {\n BOOST_CHECK_EQUAL(37ULL, seen.size());\n BOOST_REQUIRE(!seen.empty());\n BOOST_CHECK_EQUAL(\"\", seen[0]);\n for (WordIndex i = 0; i < seen.size(); ++i) {\n BOOST_CHECK_EQUAL(i, vocab.Index(seen[i]));\n }\n }\n\n void Clear() {\n seen.clear();\n }\n\n std::vector seen;\n};\n\ntemplate void LoadingTest() {\n Config config;\n config.arpa_complain = Config::NONE;\n config.messages = NULL;\n config.probing_multiplier = 2.0;\n {\n ExpectEnumerateVocab enumerate;\n config.enumerate_vocab = &enumerate;\n ModelT m(TestLocation(), config);\n enumerate.Check(m.GetVocabulary());\n BOOST_CHECK_EQUAL((WordIndex)37, m.GetVocabulary().Bound());\n Everything(m);\n }\n {\n ExpectEnumerateVocab enumerate;\n config.enumerate_vocab = &enumerate;\n ModelT m(TestNoUnkLocation(), config);\n enumerate.Check(m.GetVocabulary());\n BOOST_CHECK_EQUAL((WordIndex)37, m.GetVocabulary().Bound());\n NoUnkCheck(m);\n }\n}\n\nBOOST_AUTO_TEST_CASE(probing) {\n LoadingTest();\n}\nBOOST_AUTO_TEST_CASE(trie) {\n LoadingTest();\n}\nBOOST_AUTO_TEST_CASE(quant_trie) {\n LoadingTest();\n}\nBOOST_AUTO_TEST_CASE(bhiksha_trie) {\n LoadingTest();\n}\nBOOST_AUTO_TEST_CASE(quant_bhiksha_trie) {\n LoadingTest();\n}\n\ntemplate void BinaryTest() {\n Config config;\n config.write_mmap = \"test.binary\";\n config.messages = NULL;\n ExpectEnumerateVocab enumerate;\n config.enumerate_vocab = &enumerate;\n\n {\n ModelT copy_model(TestLocation(), config);\n enumerate.Check(copy_model.GetVocabulary());\n enumerate.Clear();\n Everything(copy_model);\n }\n\n config.write_mmap = NULL;\n\n ModelType type;\n BOOST_REQUIRE(RecognizeBinary(\"test.binary\", type));\n BOOST_CHECK_EQUAL(ModelT::kModelType, type);\n\n {\n ModelT binary(\"test.binary\", config);\n enumerate.Check(binary.GetVocabulary());\n Everything(binary);\n }\n unlink(\"test.binary\");\n\n // Now test without .\n config.write_mmap = \"test_nounk.binary\";\n config.messages = NULL;\n enumerate.Clear();\n {\n ModelT copy_model(TestNoUnkLocation(), config);\n enumerate.Check(copy_model.GetVocabulary());\n enumerate.Clear();\n NoUnkCheck(copy_model);\n }\n config.write_mmap = NULL;\n {\n ModelT binary(TestNoUnkLocation(), config);\n enumerate.Check(binary.GetVocabulary());\n NoUnkCheck(binary);\n }\n unlink(\"test_nounk.binary\");\n}\n\nBOOST_AUTO_TEST_CASE(write_and_read_probing) {\n BinaryTest();\n}\nBOOST_AUTO_TEST_CASE(write_and_read_rest_probing) {\n BinaryTest();\n}\nBOOST_AUTO_TEST_CASE(write_and_read_trie) {\n BinaryTest();\n}\nBOOST_AUTO_TEST_CASE(write_and_read_quant_trie) {\n BinaryTest();\n}\nBOOST_AUTO_TEST_CASE(write_and_read_array_trie) {\n BinaryTest();\n}\nBOOST_AUTO_TEST_CASE(write_and_read_quant_array_trie) {\n BinaryTest();\n}\n\nBOOST_AUTO_TEST_CASE(rest_max) {\n Config config;\n config.arpa_complain = Config::NONE;\n config.messages = NULL;\n\n RestProbingModel model(TestLocation(), config);\n State state, out;\n FullScoreReturn ret(model.FullScore(model.NullContextState(), model.GetVocabulary().Index(\".\"), state));\n SLOPPY_CHECK_CLOSE(-0.2705918, ret.rest, 0.001);\n SLOPPY_CHECK_CLOSE(-0.01916512, model.FullScore(state, model.GetVocabulary().EndSentence(), out).rest, 0.001);\n}\n\n} // namespace\n} // namespace ngram\n} // namespace lm\n" }, { "path": "src/kenlm/lm/model_type.hh", "content": "#ifndef LM_MODEL_TYPE__\n#define LM_MODEL_TYPE__\n\nnamespace lm {\nnamespace ngram {\n\n/* Not the best numbering system, but it grew this way for historical reasons\n * and I want to preserve existing binary files. */\ntypedef enum {PROBING=0, REST_PROBING=1, TRIE=2, QUANT_TRIE=3, ARRAY_TRIE=4, QUANT_ARRAY_TRIE=5} ModelType;\n\n// Historical names. \nconst ModelType HASH_PROBING = PROBING;\nconst ModelType TRIE_SORTED = TRIE;\nconst ModelType QUANT_TRIE_SORTED = QUANT_TRIE;\nconst ModelType ARRAY_TRIE_SORTED = ARRAY_TRIE;\nconst ModelType QUANT_ARRAY_TRIE_SORTED = QUANT_ARRAY_TRIE;\n\nconst static ModelType kQuantAdd = static_cast(QUANT_TRIE - TRIE);\nconst static ModelType kArrayAdd = static_cast(ARRAY_TRIE - TRIE);\n\n} // namespace ngram\n} // namespace lm\n#endif // LM_MODEL_TYPE__\n" }, { "path": "src/kenlm/lm/ngram_query.hh", "content": "#ifndef LM_NGRAM_QUERY__\n#define LM_NGRAM_QUERY__\n\n#include \"lm/enumerate_vocab.hh\"\n#include \"lm/model.hh\"\n#include \"util/usage.hh\"\n\n#include \n#include \n#include \n#include \n#include \n\nnamespace lm {\nnamespace ngram {\n\ntemplate void Query(const Model &model, bool sentence_context, std::istream &in_stream, std::ostream &out_stream) {\n std::cerr << \"Loading statistics:\\n\";\n util::PrintUsage(std::cerr);\n typename Model::State state, out;\n lm::FullScoreReturn ret;\n std::string word;\n\n while (in_stream) {\n state = sentence_context ? model.BeginSentenceState() : model.NullContextState();\n float total = 0.0;\n bool got = false;\n unsigned int oov = 0;\n while (in_stream >> word) {\n got = true;\n lm::WordIndex vocab = model.GetVocabulary().Index(word);\n if (vocab == 0) ++oov;\n ret = model.FullScore(state, vocab, out);\n total += ret.prob;\n out_stream << word << '=' << vocab << ' ' << static_cast(ret.ngram_length) << ' ' << ret.prob << '\\t';\n state = out;\n char c;\n while (true) {\n c = in_stream.get();\n if (!in_stream) break;\n if (c == '\\n') break;\n if (!isspace(c)) {\n in_stream.unget();\n break;\n }\n }\n if (c == '\\n') break;\n }\n if (!got && !in_stream) break;\n if (sentence_context) {\n ret = model.FullScore(state, model.GetVocabulary().EndSentence(), out);\n total += ret.prob;\n out_stream << \"=\" << model.GetVocabulary().EndSentence() << ' ' << static_cast(ret.ngram_length) << ' ' << ret.prob << '\\t';\n }\n out_stream << \"Total: \" << total << \" OOV: \" << oov << '\\n';\n }\n std::cerr << \"After queries:\\n\";\n util::PrintUsage(std::cerr);\n}\n\ntemplate void Query(const char *file, bool sentence_context, std::istream &in_stream, std::ostream &out_stream) {\n Config config;\n M model(file, config);\n Query(model, sentence_context, in_stream, out_stream);\n}\n\n} // namespace ngram\n} // namespace lm\n\n#endif // LM_NGRAM_QUERY__\n\n\n" }, { "path": "src/kenlm/lm/partial.hh", "content": "#ifndef LM_PARTIAL__\n#define LM_PARTIAL__\n\n#include \"lm/return.hh\"\n#include \"lm/state.hh\"\n\n#include \n\n#include \n\nnamespace lm {\nnamespace ngram {\n\nstruct ExtendReturn {\n float adjust;\n bool make_full;\n unsigned char next_use;\n};\n\ntemplate ExtendReturn ExtendLoop(\n const Model &model,\n unsigned char seen, const WordIndex *add_rbegin, const WordIndex *add_rend, const float *backoff_start,\n const uint64_t *pointers, const uint64_t *pointers_end,\n uint64_t *&pointers_write,\n float *backoff_write) {\n unsigned char add_length = add_rend - add_rbegin;\n\n float backoff_buf[2][KENLM_MAX_ORDER - 1];\n float *backoff_in = backoff_buf[0], *backoff_out = backoff_buf[1];\n std::copy(backoff_start, backoff_start + add_length, backoff_in);\n\n ExtendReturn value;\n value.make_full = false;\n value.adjust = 0.0;\n value.next_use = add_length;\n\n unsigned char i = 0;\n unsigned char length = pointers_end - pointers;\n // pointers_write is NULL means that the existing left state is full, so we should use completed probabilities. \n if (pointers_write) {\n // Using full context, writing to new left state. \n for (; i < length; ++i) {\n FullScoreReturn ret(model.ExtendLeft(\n add_rbegin, add_rbegin + value.next_use,\n backoff_in,\n pointers[i], i + seen + 1,\n backoff_out,\n value.next_use));\n std::swap(backoff_in, backoff_out);\n if (ret.independent_left) {\n value.adjust += ret.prob;\n value.make_full = true;\n ++i;\n break;\n }\n value.adjust += ret.rest;\n *pointers_write++ = ret.extend_left;\n if (value.next_use != add_length) {\n value.make_full = true;\n ++i;\n break;\n }\n }\n }\n // Using some of the new context. \n for (; i < length && value.next_use; ++i) {\n FullScoreReturn ret(model.ExtendLeft(\n add_rbegin, add_rbegin + value.next_use,\n backoff_in,\n pointers[i], i + seen + 1,\n backoff_out,\n value.next_use));\n std::swap(backoff_in, backoff_out);\n value.adjust += ret.prob;\n }\n float unrest = model.UnRest(pointers + i, pointers_end, i + seen + 1);\n // Using none of the new context. \n value.adjust += unrest;\n\n std::copy(backoff_in, backoff_in + value.next_use, backoff_write);\n return value;\n}\n\ntemplate float RevealBefore(const Model &model, const Right &reveal, const unsigned char seen, bool reveal_full, Left &left, Right &right) {\n assert(seen < reveal.length || reveal_full);\n uint64_t *pointers_write = reveal_full ? NULL : left.pointers;\n float backoff_buffer[KENLM_MAX_ORDER - 1];\n ExtendReturn value(ExtendLoop(\n model,\n seen, reveal.words + seen, reveal.words + reveal.length, reveal.backoff + seen,\n left.pointers, left.pointers + left.length,\n pointers_write,\n left.full ? backoff_buffer : (right.backoff + right.length)));\n if (reveal_full) {\n left.length = 0;\n value.make_full = true;\n } else {\n left.length = pointers_write - left.pointers;\n value.make_full |= (left.length == model.Order() - 1);\n }\n if (left.full) {\n for (unsigned char i = 0; i < value.next_use; ++i) value.adjust += backoff_buffer[i];\n } else {\n // If left wasn't full when it came in, put words into right state. \n std::copy(reveal.words + seen, reveal.words + seen + value.next_use, right.words + right.length);\n right.length += value.next_use;\n left.full = value.make_full || (right.length == model.Order() - 1);\n }\n return value.adjust;\n}\n\ntemplate float RevealAfter(const Model &model, Left &left, Right &right, const Left &reveal, unsigned char seen) {\n assert(seen < reveal.length || reveal.full);\n uint64_t *pointers_write = left.full ? NULL : (left.pointers + left.length);\n ExtendReturn value(ExtendLoop(\n model,\n seen, right.words, right.words + right.length, right.backoff,\n reveal.pointers + seen, reveal.pointers + reveal.length,\n pointers_write,\n right.backoff));\n if (reveal.full) {\n for (unsigned char i = 0; i < value.next_use; ++i) value.adjust += right.backoff[i];\n right.length = 0;\n value.make_full = true;\n } else {\n right.length = value.next_use;\n value.make_full |= (right.length == model.Order() - 1);\n }\n if (!left.full) {\n left.length = pointers_write - left.pointers;\n left.full = value.make_full || (left.length == model.Order() - 1);\n }\n return value.adjust;\n}\n\ntemplate float Subsume(const Model &model, Left &first_left, const Right &first_right, const Left &second_left, Right &second_right, const unsigned int between_length) {\n assert(first_right.length < KENLM_MAX_ORDER);\n assert(second_left.length < KENLM_MAX_ORDER);\n assert(between_length < KENLM_MAX_ORDER - 1);\n uint64_t *pointers_write = first_left.full ? NULL : (first_left.pointers + first_left.length);\n float backoff_buffer[KENLM_MAX_ORDER - 1];\n ExtendReturn value(ExtendLoop(\n model,\n between_length, first_right.words, first_right.words + first_right.length, first_right.backoff,\n second_left.pointers, second_left.pointers + second_left.length,\n pointers_write,\n second_left.full ? backoff_buffer : (second_right.backoff + second_right.length)));\n if (second_left.full) {\n for (unsigned char i = 0; i < value.next_use; ++i) value.adjust += backoff_buffer[i];\n } else {\n std::copy(first_right.words, first_right.words + value.next_use, second_right.words + second_right.length);\n second_right.length += value.next_use;\n value.make_full |= (second_right.length == model.Order() - 1);\n }\n if (!first_left.full) {\n first_left.length = pointers_write - first_left.pointers;\n first_left.full = value.make_full || second_left.full || (first_left.length == model.Order() - 1);\n }\n assert(first_left.length < KENLM_MAX_ORDER);\n assert(second_right.length < KENLM_MAX_ORDER);\n return value.adjust;\n}\n\n} // namespace ngram\n} // namespace lm\n\n#endif // LM_PARTIAL__\n" }, { "path": "src/kenlm/lm/partial_test.cc", "content": "#include \"lm/partial.hh\"\n\n#include \"lm/left.hh\"\n#include \"lm/model.hh\"\n#include \"util/tokenize_piece.hh\"\n\n#define BOOST_TEST_MODULE PartialTest\n#include \n#include \n\nnamespace lm {\nnamespace ngram {\nnamespace {\n\nconst char *TestLocation() {\n if (boost::unit_test::framework::master_test_suite().argc < 2) {\n return \"test.arpa\";\n }\n return boost::unit_test::framework::master_test_suite().argv[1];\n}\n\nConfig SilentConfig() {\n Config config;\n config.arpa_complain = Config::NONE;\n config.messages = NULL;\n return config;\n}\n\nstruct ModelFixture {\n ModelFixture() : m(TestLocation(), SilentConfig()) {}\n\n RestProbingModel m;\n};\n\nBOOST_FIXTURE_TEST_SUITE(suite, ModelFixture)\n\nBOOST_AUTO_TEST_CASE(SimpleBefore) {\n Left left;\n left.full = false;\n left.length = 0;\n Right right;\n right.length = 0;\n\n Right reveal;\n reveal.length = 1;\n WordIndex period = m.GetVocabulary().Index(\".\");\n reveal.words[0] = period;\n reveal.backoff[0] = -0.845098;\n\n BOOST_CHECK_CLOSE(0.0, RevealBefore(m, reveal, 0, false, left, right), 0.001);\n BOOST_CHECK_EQUAL(0, left.length);\n BOOST_CHECK(!left.full);\n BOOST_CHECK_EQUAL(1, right.length);\n BOOST_CHECK_EQUAL(period, right.words[0]);\n BOOST_CHECK_CLOSE(-0.845098, right.backoff[0], 0.001);\n\n WordIndex more = m.GetVocabulary().Index(\"more\");\n reveal.words[1] = more;\n reveal.backoff[1] = -0.4771212;\n reveal.length = 2;\n BOOST_CHECK_CLOSE(0.0, RevealBefore(m, reveal, 1, false, left, right), 0.001);\n BOOST_CHECK_EQUAL(0, left.length);\n BOOST_CHECK(!left.full);\n BOOST_CHECK_EQUAL(2, right.length);\n BOOST_CHECK_EQUAL(period, right.words[0]);\n BOOST_CHECK_EQUAL(more, right.words[1]);\n BOOST_CHECK_CLOSE(-0.845098, right.backoff[0], 0.001);\n BOOST_CHECK_CLOSE(-0.4771212, right.backoff[1], 0.001);\n}\n\nBOOST_AUTO_TEST_CASE(AlsoWouldConsider) {\n WordIndex would = m.GetVocabulary().Index(\"would\");\n WordIndex consider = m.GetVocabulary().Index(\"consider\");\n\n ChartState current;\n current.left.length = 1;\n current.left.pointers[0] = would;\n current.left.full = false;\n current.right.length = 1;\n current.right.words[0] = would;\n current.right.backoff[0] = -0.30103;\n\n Left after;\n after.full = false;\n after.length = 1;\n after.pointers[0] = consider;\n\n // adjustment for would consider\n BOOST_CHECK_CLOSE(-1.687872 - -0.2922095 - 0.30103, RevealAfter(m, current.left, current.right, after, 0), 0.001);\n\n BOOST_CHECK_EQUAL(2, current.left.length);\n BOOST_CHECK_EQUAL(would, current.left.pointers[0]);\n BOOST_CHECK_EQUAL(false, current.left.full);\n\n WordIndex also = m.GetVocabulary().Index(\"also\");\n Right before;\n before.length = 1;\n before.words[0] = also;\n before.backoff[0] = -0.30103;\n // r(would) = -0.2922095 [i would], r(would -> consider) = -1.988902 [b(would) + p(consider)]\n // p(also -> would) = -2, p(also would -> consider) = -3\n BOOST_CHECK_CLOSE(-2 + 0.2922095 -3 + 1.988902, RevealBefore(m, before, 0, false, current.left, current.right), 0.001);\n BOOST_CHECK_EQUAL(0, current.left.length);\n BOOST_CHECK(current.left.full);\n BOOST_CHECK_EQUAL(2, current.right.length);\n BOOST_CHECK_EQUAL(would, current.right.words[0]);\n BOOST_CHECK_EQUAL(also, current.right.words[1]);\n}\n\nBOOST_AUTO_TEST_CASE(EndSentence) {\n WordIndex loin = m.GetVocabulary().Index(\"loin\");\n WordIndex period = m.GetVocabulary().Index(\".\");\n WordIndex eos = m.GetVocabulary().EndSentence();\n\n ChartState between;\n between.left.length = 1;\n between.left.pointers[0] = eos;\n between.left.full = true;\n between.right.length = 0;\n\n Right before;\n before.words[0] = period;\n before.words[1] = loin;\n before.backoff[0] = -0.845098;\n before.backoff[1] = 0.0;\n \n before.length = 1;\n BOOST_CHECK_CLOSE(-0.0410707, RevealBefore(m, before, 0, true, between.left, between.right), 0.001);\n BOOST_CHECK_EQUAL(0, between.left.length);\n}\n\nfloat ScoreFragment(const RestProbingModel &model, unsigned int *begin, unsigned int *end, ChartState &out) {\n RuleScore scorer(model, out);\n for (unsigned int *i = begin; i < end; ++i) {\n scorer.Terminal(*i);\n }\n return scorer.Finish();\n}\n\nvoid CheckAdjustment(const RestProbingModel &model, float expect, const Right &before_in, bool before_full, ChartState between, const Left &after_in) {\n Right before(before_in);\n Left after(after_in);\n after.full = false;\n float got = 0.0;\n for (unsigned int i = 1; i < 5; ++i) {\n if (before_in.length >= i) {\n before.length = i;\n got += RevealBefore(model, before, i - 1, false, between.left, between.right);\n }\n if (after_in.length >= i) {\n after.length = i;\n got += RevealAfter(model, between.left, between.right, after, i - 1);\n }\n }\n if (after_in.full) {\n after.full = true;\n got += RevealAfter(model, between.left, between.right, after, after.length);\n }\n if (before_full) {\n got += RevealBefore(model, before, before.length, true, between.left, between.right);\n }\n // Sometimes they're zero and BOOST_CHECK_CLOSE fails for this. \n BOOST_CHECK(fabs(expect - got) < 0.001);\n}\n\nvoid FullDivide(const RestProbingModel &model, StringPiece str) {\n std::vector indices;\n for (util::TokenIter i(str, ' '); i; ++i) {\n indices.push_back(model.GetVocabulary().Index(*i));\n }\n ChartState full_state;\n float full = ScoreFragment(model, &indices.front(), &indices.back() + 1, full_state);\n\n ChartState before_state;\n before_state.left.full = false;\n RuleScore before_scorer(model, before_state);\n float before_score = 0.0;\n for (unsigned int before = 0; before < indices.size(); ++before) {\n for (unsigned int after = before; after <= indices.size(); ++after) {\n ChartState after_state, between_state;\n float after_score = ScoreFragment(model, &indices.front() + after, &indices.front() + indices.size(), after_state);\n float between_score = ScoreFragment(model, &indices.front() + before, &indices.front() + after, between_state);\n CheckAdjustment(model, full - before_score - after_score - between_score, before_state.right, before_state.left.full, between_state, after_state.left);\n }\n before_scorer.Terminal(indices[before]);\n before_score = before_scorer.Finish();\n }\n}\n\nBOOST_AUTO_TEST_CASE(Strings) {\n FullDivide(m, \"also would consider\");\n FullDivide(m, \"looking on a little more loin . \");\n FullDivide(m, \"in biarritz watching considering looking . on a little more loin also would consider higher to look good unknown the screening foo bar , unknown however unknown \");\n}\n\nBOOST_AUTO_TEST_SUITE_END()\n} // namespace\n} // namespace ngram\n} // namespace lm\n" }, { "path": "src/kenlm/lm/quantize.cc", "content": "/* Quantize into bins of equal size as described in\n * M. Federico and N. Bertoldi. 2006. How many bits are needed\n * to store probabilities for phrase-based translation? In Proc.\n * of the Workshop on Statistical Machine Translation, pages\n * 94–101, New York City, June. Association for Computa-\n * tional Linguistics.\n */\n\n#include \"lm/quantize.hh\"\n\n#include \"lm/binary_format.hh\"\n#include \"lm/lm_exception.hh\"\n#include \"util/file.hh\"\n\n#include \n#include \n\nnamespace lm {\nnamespace ngram {\n\nnamespace {\n\nvoid MakeBins(std::vector &values, float *centers, uint32_t bins) {\n std::sort(values.begin(), values.end());\n std::vector::const_iterator start = values.begin(), finish;\n for (uint32_t i = 0; i < bins; ++i, ++centers, start = finish) {\n finish = values.begin() + ((values.size() * static_cast(i + 1)) / bins);\n if (finish == start) {\n // zero length bucket.\n *centers = i ? *(centers - 1) : -std::numeric_limits::infinity();\n } else {\n *centers = std::accumulate(start, finish, 0.0) / static_cast(finish - start);\n }\n }\n}\n\nconst char kSeparatelyQuantizeVersion = 2;\n\n} // namespace\n\nvoid SeparatelyQuantize::UpdateConfigFromBinary(int fd, const std::vector &/*counts*/, Config &config) {\n char version;\n util::ReadOrThrow(fd, &version, 1);\n util::ReadOrThrow(fd, &config.prob_bits, 1);\n util::ReadOrThrow(fd, &config.backoff_bits, 1);\n if (version != kSeparatelyQuantizeVersion) UTIL_THROW(FormatLoadException, \"This file has quantization version \" << (unsigned)version << \" but the code expects version \" << (unsigned)kSeparatelyQuantizeVersion);\n util::AdvanceOrThrow(fd, -3);\n}\n\nvoid SeparatelyQuantize::SetupMemory(void *base, unsigned char order, const Config &config) {\n prob_bits_ = config.prob_bits;\n backoff_bits_ = config.backoff_bits;\n // We need the reserved values. \n if (config.prob_bits == 0) UTIL_THROW(ConfigException, \"You can't quantize probability to zero\");\n if (config.backoff_bits == 0) UTIL_THROW(ConfigException, \"You can't quantize backoff to zero\");\n if (config.prob_bits > 25) UTIL_THROW(ConfigException, \"For efficiency reasons, quantizing probability supports at most 25 bits. Currently you have requested \" << static_cast(config.prob_bits) << \" bits.\");\n if (config.backoff_bits > 25) UTIL_THROW(ConfigException, \"For efficiency reasons, quantizing backoff supports at most 25 bits. Currently you have requested \" << static_cast(config.backoff_bits) << \" bits.\");\n // Reserve 8 byte header for bit counts. \n actual_base_ = static_cast(base);\n float *start = reinterpret_cast(actual_base_ + 8);\n for (unsigned char i = 0; i < order - 2; ++i) {\n tables_[i][0] = Bins(prob_bits_, start);\n start += (1ULL << prob_bits_);\n tables_[i][1] = Bins(backoff_bits_, start);\n start += (1ULL << backoff_bits_);\n }\n longest_ = tables_[order - 2][0] = Bins(prob_bits_, start);\n}\n\nvoid SeparatelyQuantize::Train(uint8_t order, std::vector &prob, std::vector &backoff) {\n TrainProb(order, prob);\n\n // Backoff\n float *centers = tables_[order - 2][1].Populate();\n *(centers++) = kNoExtensionBackoff;\n *(centers++) = kExtensionBackoff;\n MakeBins(backoff, centers, (1ULL << backoff_bits_) - 2);\n}\n\nvoid SeparatelyQuantize::TrainProb(uint8_t order, std::vector &prob) {\n float *centers = tables_[order - 2][0].Populate();\n MakeBins(prob, centers, (1ULL << prob_bits_));\n}\n\nvoid SeparatelyQuantize::FinishedLoading(const Config &config) {\n uint8_t *actual_base = actual_base_;\n *(actual_base++) = kSeparatelyQuantizeVersion; // version\n *(actual_base++) = config.prob_bits;\n *(actual_base++) = config.backoff_bits;\n}\n\n} // namespace ngram\n} // namespace lm\n" }, { "path": "src/kenlm/lm/quantize.hh", "content": "#ifndef LM_QUANTIZE_H__\n#define LM_QUANTIZE_H__\n\n#include \"lm/blank.hh\"\n#include \"lm/config.hh\"\n#include \"lm/max_order.hh\"\n#include \"lm/model_type.hh\"\n#include \"util/bit_packing.hh\"\n\n#include \n#include \n\n#include \n\n#include \n\nnamespace lm {\nnamespace ngram {\n\nstruct Config;\n\n/* Store values directly and don't quantize. */\nclass DontQuantize {\n public:\n static const ModelType kModelTypeAdd = static_cast(0);\n static void UpdateConfigFromBinary(int, const std::vector &, Config &) {}\n static uint64_t Size(uint8_t /*order*/, const Config &/*config*/) { return 0; }\n static uint8_t MiddleBits(const Config &/*config*/) { return 63; }\n static uint8_t LongestBits(const Config &/*config*/) { return 31; }\n\n class MiddlePointer {\n public:\n MiddlePointer(const DontQuantize & /*quant*/, unsigned char /*order_minus_2*/, util::BitAddress address) : address_(address) {}\n\n MiddlePointer() : address_(NULL, 0) {}\n\n bool Found() const {\n return address_.base != NULL;\n }\n\n float Prob() const {\n return util::ReadNonPositiveFloat31(address_.base, address_.offset);\n }\n\n float Backoff() const {\n return util::ReadFloat32(address_.base, address_.offset + 31);\n }\n\n float Rest() const { return Prob(); }\n\n void Write(float prob, float backoff) {\n util::WriteNonPositiveFloat31(address_.base, address_.offset, prob);\n util::WriteFloat32(address_.base, address_.offset + 31, backoff);\n }\n\n private:\n util::BitAddress address_;\n };\n\n class LongestPointer {\n public:\n explicit LongestPointer(const DontQuantize &/*quant*/, util::BitAddress address) : address_(address) {}\n\n LongestPointer() : address_(NULL, 0) {}\n\n bool Found() const {\n return address_.base != NULL;\n }\n\n float Prob() const {\n return util::ReadNonPositiveFloat31(address_.base, address_.offset);\n }\n\n void Write(float prob) {\n util::WriteNonPositiveFloat31(address_.base, address_.offset, prob);\n }\n\n private:\n util::BitAddress address_;\n };\n\n DontQuantize() {}\n\n void SetupMemory(void * /*start*/, unsigned char /*order*/, const Config & /*config*/) {}\n\n static const bool kTrain = false;\n // These should never be called because kTrain is false. \n void Train(uint8_t /*order*/, std::vector &/*prob*/, std::vector &/*backoff*/) {}\n void TrainProb(uint8_t, std::vector &/*prob*/) {}\n\n void FinishedLoading(const Config &) {}\n};\n\nclass SeparatelyQuantize {\n private:\n class Bins {\n public:\n // Sigh C++ default constructor\n Bins() {}\n\n Bins(uint8_t bits, float *begin) : begin_(begin), end_(begin_ + (1ULL << bits)), bits_(bits), mask_((1ULL << bits) - 1) {}\n\n float *Populate() { return begin_; }\n\n uint64_t EncodeProb(float value) const {\n return Encode(value, 0);\n }\n\n uint64_t EncodeBackoff(float value) const {\n if (value == 0.0) {\n return HasExtension(value) ? kExtensionQuant : kNoExtensionQuant;\n }\n return Encode(value, 2);\n }\n\n float Decode(std::size_t off) const { return begin_[off]; }\n\n uint8_t Bits() const { return bits_; }\n\n uint64_t Mask() const { return mask_; }\n\n private:\n uint64_t Encode(float value, size_t reserved) const {\n const float *above = std::lower_bound(static_cast(begin_) + reserved, end_, value);\n if (above == begin_ + reserved) return reserved;\n if (above == end_) return end_ - begin_ - 1;\n return above - begin_ - (value - *(above - 1) < *above - value);\n }\n\n float *begin_;\n const float *end_;\n uint8_t bits_;\n uint64_t mask_;\n };\n\n public:\n static const ModelType kModelTypeAdd = kQuantAdd;\n\n static void UpdateConfigFromBinary(int fd, const std::vector &counts, Config &config);\n\n static uint64_t Size(uint8_t order, const Config &config) {\n uint64_t longest_table = (static_cast(1) << static_cast(config.prob_bits)) * sizeof(float);\n uint64_t middle_table = (static_cast(1) << static_cast(config.backoff_bits)) * sizeof(float) + longest_table;\n // unigrams are currently not quantized so no need for a table. \n return (order - 2) * middle_table + longest_table + /* for the bit counts and alignment padding) */ 8;\n }\n\n static uint8_t MiddleBits(const Config &config) { return config.prob_bits + config.backoff_bits; }\n static uint8_t LongestBits(const Config &config) { return config.prob_bits; }\n\n class MiddlePointer {\n public:\n MiddlePointer(const SeparatelyQuantize &quant, unsigned char order_minus_2, const util::BitAddress &address) : bins_(quant.GetTables(order_minus_2)), address_(address) {}\n\n MiddlePointer() : address_(NULL, 0) {}\n\n bool Found() const { return address_.base != NULL; }\n\n float Prob() const {\n return ProbBins().Decode(util::ReadInt25(address_.base, address_.offset + BackoffBins().Bits(), ProbBins().Bits(), ProbBins().Mask()));\n }\n\n float Backoff() const {\n return BackoffBins().Decode(util::ReadInt25(address_.base, address_.offset, BackoffBins().Bits(), BackoffBins().Mask()));\n }\n\n float Rest() const { return Prob(); }\n\n void Write(float prob, float backoff) const {\n util::WriteInt57(address_.base, address_.offset, ProbBins().Bits() + BackoffBins().Bits(), \n (ProbBins().EncodeProb(prob) << BackoffBins().Bits()) | BackoffBins().EncodeBackoff(backoff));\n }\n\n private:\n const Bins &ProbBins() const { return bins_[0]; }\n const Bins &BackoffBins() const { return bins_[1]; }\n const Bins *bins_;\n\n util::BitAddress address_;\n };\n\n class LongestPointer {\n public:\n LongestPointer(const SeparatelyQuantize &quant, const util::BitAddress &address) : table_(&quant.LongestTable()), address_(address) {}\n \n LongestPointer() : address_(NULL, 0) {}\n\n bool Found() const { return address_.base != NULL; }\n\n void Write(float prob) const {\n util::WriteInt25(address_.base, address_.offset, table_->Bits(), table_->EncodeProb(prob));\n }\n\n float Prob() const {\n return table_->Decode(util::ReadInt25(address_.base, address_.offset, table_->Bits(), table_->Mask()));\n }\n\n private:\n const Bins *table_;\n util::BitAddress address_;\n };\n\n SeparatelyQuantize() {}\n\n void SetupMemory(void *start, unsigned char order, const Config &config);\n\n static const bool kTrain = true;\n // Assumes 0.0 is removed from backoff. \n void Train(uint8_t order, std::vector &prob, std::vector &backoff);\n // Train just probabilities (for longest order).\n void TrainProb(uint8_t order, std::vector &prob);\n\n void FinishedLoading(const Config &config);\n\n const Bins *GetTables(unsigned char order_minus_2) const { return tables_[order_minus_2]; }\n\n const Bins &LongestTable() const { return longest_; }\n\n private:\n Bins tables_[KENLM_MAX_ORDER - 1][2];\n\n Bins longest_;\n\n uint8_t *actual_base_;\n\n uint8_t prob_bits_, backoff_bits_;\n};\n\n} // namespace ngram\n} // namespace lm\n\n#endif // LM_QUANTIZE_H__\n" }, { "path": "src/kenlm/lm/query_main.cc", "content": "#include \"lm/ngram_query.hh\"\n\nint main(int argc, char *argv[]) {\n if (!(argc == 2 || (argc == 3 && !strcmp(argv[2], \"null\")))) {\n std::cerr << \"KenLM was compiled with maximum order \" << KENLM_MAX_ORDER << \".\" << std::endl;\n std::cerr << \"Usage: \" << argv[0] << \" lm_file [null]\" << std::endl;\n std::cerr << \"Input is wrapped in and unless null is passed.\" << std::endl;\n return 1;\n }\n try {\n bool sentence_context = (argc == 2);\n using namespace lm::ngram;\n ModelType model_type;\n if (RecognizeBinary(argv[1], model_type)) {\n switch(model_type) {\n case PROBING:\n Query(argv[1], sentence_context, std::cin, std::cout);\n break;\n case REST_PROBING:\n Query(argv[1], sentence_context, std::cin, std::cout);\n break;\n case TRIE:\n Query(argv[1], sentence_context, std::cin, std::cout);\n break;\n case QUANT_TRIE:\n Query(argv[1], sentence_context, std::cin, std::cout);\n break;\n case ARRAY_TRIE:\n Query(argv[1], sentence_context, std::cin, std::cout);\n break;\n case QUANT_ARRAY_TRIE:\n Query(argv[1], sentence_context, std::cin, std::cout);\n break;\n default:\n std::cerr << \"Unrecognized kenlm model type \" << model_type << std::endl;\n abort();\n }\n } else {\n Query(argv[1], sentence_context, std::cin, std::cout);\n }\n std::cerr << \"Total time including destruction:\\n\";\n util::PrintUsage(std::cerr);\n } catch (const std::exception &e) {\n std::cerr << e.what() << std::endl;\n return 1;\n }\n return 0;\n}\n" }, { "path": "src/kenlm/lm/read_arpa.cc", "content": "#include \"lm/read_arpa.hh\"\n\n#include \"lm/blank.hh\"\n#include \"util/file.hh\"\n\n#include \n#include \n#include \n#include \n#include \n\n#include \n#include \n#include \n\n#ifdef WIN32\n#include \n#endif\n\nnamespace lm {\n\n// 1 for '\\t', '\\n', and ' '. This is stricter than isspace. \nconst bool kARPASpaces[256] = {0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};\n\nnamespace {\n\nbool IsEntirelyWhiteSpace(const StringPiece &line) {\n for (size_t i = 0; i < static_cast(line.size()); ++i) {\n if (!isspace(line.data()[i])) return false;\n }\n return true;\n}\n\nconst char kBinaryMagic[] = \"mmap lm http://kheafield.com/code\";\n\n// strtoull isn't portable enough :-(\nuint64_t ReadCount(const std::string &from) {\n std::stringstream stream(from);\n uint64_t ret;\n stream >> ret;\n UTIL_THROW_IF(!stream, FormatLoadException, \"Bad count \" << from);\n return ret;\n}\n\n} // namespace\n\nvoid ReadARPACounts(util::FilePiece &in, std::vector &number) {\n number.clear();\n StringPiece line = in.ReadLine();\n // In general, ARPA files can have arbitrary text before \"\\data\\\"\n // But in KenLM, we require such lines to start with \"#\", so that\n // we can do stricter error checking\n while (IsEntirelyWhiteSpace(line) || line.starts_with(\"#\")) {\n line = in.ReadLine();\n }\n\n if (line != \"\\\\data\\\\\") {\n if ((line.size() >= 2) && (line.data()[0] == 0x1f) && (static_cast(line.data()[1]) == 0x8b)) {\n UTIL_THROW(FormatLoadException, \"Looks like a gzip file. If this is an ARPA file, pipe \" << in.FileName() << \" through zcat. If this already in binary format, you need to decompress it because mmap doesn't work on top of gzip.\");\n }\n if (static_cast(line.size()) >= strlen(kBinaryMagic) && StringPiece(line.data(), strlen(kBinaryMagic)) == kBinaryMagic) \n UTIL_THROW(FormatLoadException, \"This looks like a binary file but got sent to the ARPA parser. Did you compress the binary file or pass a binary file where only ARPA files are accepted?\");\n UTIL_THROW_IF(line.size() >= 4 && StringPiece(line.data(), 4) == \"blmt\", FormatLoadException, \"This looks like an IRSTLM binary file. Did you forget to pass --text yes to compile-lm?\");\n UTIL_THROW_IF(line == \"iARPA\", FormatLoadException, \"This looks like an IRSTLM iARPA file. You need an ARPA file. Run\\n compile-lm --text yes \" << in.FileName() << \" \" << in.FileName() << \".arpa\\nfirst.\");\n UTIL_THROW(FormatLoadException, \"first non-empty line was \\\"\" << line << \"\\\" not \\\\data\\\\.\");\n }\n while (!IsEntirelyWhiteSpace(line = in.ReadLine())) {\n if (line.size() < 6 || strncmp(line.data(), \"ngram \", 6)) UTIL_THROW(FormatLoadException, \"count line \\\"\" << line << \"\\\"doesn't begin with \\\"ngram \\\"\");\n // So strtol doesn't go off the end of line. \n std::string remaining(line.data() + 6, line.size() - 6);\n char *end_ptr;\n unsigned int length = std::strtol(remaining.c_str(), &end_ptr, 10);\n if ((end_ptr == remaining.c_str()) || (length - 1 != number.size())) UTIL_THROW(FormatLoadException, \"ngram count lengths should be consecutive starting with 1: \" << line);\n if (*end_ptr != '=') UTIL_THROW(FormatLoadException, \"Expected = immediately following the first number in the count line \" << line);\n ++end_ptr;\n number.push_back(ReadCount(end_ptr));\n }\n}\n\nvoid ReadNGramHeader(util::FilePiece &in, unsigned int length) {\n StringPiece line;\n while (IsEntirelyWhiteSpace(line = in.ReadLine())) {}\n std::stringstream expected;\n expected << '\\\\' << length << \"-grams:\";\n if (line != expected.str()) UTIL_THROW(FormatLoadException, \"Was expecting n-gram header \" << expected.str() << \" but got \" << line << \" instead\");\n}\n\nvoid ReadBackoff(util::FilePiece &in, Prob &/*weights*/) {\n switch (in.get()) {\n case '\\t':\n {\n float got = in.ReadFloat();\n if (got != 0.0)\n UTIL_THROW(FormatLoadException, \"Non-zero backoff \" << got << \" provided for an n-gram that should have no backoff\");\n }\n break;\n case '\\n':\n break;\n default:\n UTIL_THROW(FormatLoadException, \"Expected tab or newline for backoff\");\n }\n}\n\nvoid ReadBackoff(util::FilePiece &in, float &backoff) {\n // Always make zero negative. \n // Negative zero means that no (n+1)-gram has this n-gram as context. \n // Therefore the hypothesis state can be shorter. Of course, many n-grams\n // are context for (n+1)-grams. An algorithm in the data structure will go\n // back and set the backoff to positive zero in these cases.\n switch (in.get()) {\n case '\\t':\n backoff = in.ReadFloat();\n if (backoff == ngram::kExtensionBackoff) backoff = ngram::kNoExtensionBackoff;\n {\n#ifdef WIN32\n\t\tint float_class = _fpclass(backoff);\n UTIL_THROW_IF(float_class == _FPCLASS_SNAN || float_class == _FPCLASS_QNAN || float_class == _FPCLASS_NINF || float_class == _FPCLASS_PINF, FormatLoadException, \"Bad backoff \" << backoff);\n#else\n int float_class = std::fpclassify(backoff);\n UTIL_THROW_IF(float_class == FP_NAN || float_class == FP_INFINITE, FormatLoadException, \"Bad backoff \" << backoff);\n#endif\n }\n UTIL_THROW_IF(in.get() != '\\n', FormatLoadException, \"Expected newline after backoff\");\n break;\n case '\\n':\n backoff = ngram::kNoExtensionBackoff;\n break;\n default:\n UTIL_THROW(FormatLoadException, \"Expected tab or newline for backoff\");\n }\n}\n\nvoid ReadEnd(util::FilePiece &in) {\n StringPiece line;\n do {\n line = in.ReadLine();\n } while (IsEntirelyWhiteSpace(line));\n if (line != \"\\\\end\\\\\") UTIL_THROW(FormatLoadException, \"Expected \\\\end\\\\ but the ARPA file has \" << line);\n\n try {\n while (true) {\n line = in.ReadLine();\n if (!IsEntirelyWhiteSpace(line)) UTIL_THROW(FormatLoadException, \"Trailing line \" << line);\n }\n } catch (const util::EndOfFileException &e) {}\n}\n\nvoid PositiveProbWarn::Warn(float prob) {\n switch (action_) {\n case THROW_UP:\n UTIL_THROW(FormatLoadException, \"Positive log probability \" << prob << \" in the model. This is a bug in IRSTLM; you can set config.positive_log_probability = SILENT or pass -i to build_binary to substitute 0.0 for the log probability. Error\");\n case COMPLAIN:\n std::cerr << \"There's a positive log probability \" << prob << \" in the APRA file, probably because of a bug in IRSTLM. This and subsequent entires will be mapepd to 0 log probability.\" << std::endl;\n action_ = SILENT;\n break;\n case SILENT:\n break;\n }\n}\n\n} // namespace lm\n" }, { "path": "src/kenlm/lm/read_arpa.hh", "content": "#ifndef LM_READ_ARPA__\n#define LM_READ_ARPA__\n\n#include \"lm/lm_exception.hh\"\n#include \"lm/word_index.hh\"\n#include \"lm/weights.hh\"\n#include \"util/file_piece.hh\"\n\n#include \n#include \n#include \n\nnamespace lm {\n\nvoid ReadARPACounts(util::FilePiece &in, std::vector &number);\nvoid ReadNGramHeader(util::FilePiece &in, unsigned int length);\n\nvoid ReadBackoff(util::FilePiece &in, Prob &weights);\nvoid ReadBackoff(util::FilePiece &in, float &backoff);\ninline void ReadBackoff(util::FilePiece &in, ProbBackoff &weights) {\n ReadBackoff(in, weights.backoff);\n}\ninline void ReadBackoff(util::FilePiece &in, RestWeights &weights) {\n ReadBackoff(in, weights.backoff);\n}\n\nvoid ReadEnd(util::FilePiece &in);\n\nextern const bool kARPASpaces[256];\n\n// Positive log probability warning. \nclass PositiveProbWarn {\n public:\n PositiveProbWarn() : action_(THROW_UP) {}\n\n explicit PositiveProbWarn(WarningAction action) : action_(action) {}\n\n void Warn(float prob);\n\n private:\n WarningAction action_;\n};\n\ntemplate void Read1Gram(util::FilePiece &f, Voc &vocab, Weights *unigrams, PositiveProbWarn &warn) {\n try {\n float prob = f.ReadFloat();\n if (prob > 0.0) {\n warn.Warn(prob);\n prob = 0.0;\n }\n if (f.get() != '\\t') UTIL_THROW(FormatLoadException, \"Expected tab after probability\");\n Weights &value = unigrams[vocab.Insert(f.ReadDelimited(kARPASpaces))];\n value.prob = prob;\n ReadBackoff(f, value);\n } catch(util::Exception &e) {\n e << \" in the 1-gram at byte \" << f.Offset();\n throw;\n }\n}\n\n// Return true if a positive log probability came out.\ntemplate void Read1Grams(util::FilePiece &f, std::size_t count, Voc &vocab, Weights *unigrams, PositiveProbWarn &warn) {\n ReadNGramHeader(f, 1);\n for (std::size_t i = 0; i < count; ++i) {\n Read1Gram(f, vocab, unigrams, warn);\n }\n vocab.FinishedLoading(unigrams);\n}\n\n// Return true if a positive log probability came out.\ntemplate void ReadNGram(util::FilePiece &f, const unsigned char n, const Voc &vocab, WordIndex *const reverse_indices, Weights &weights, PositiveProbWarn &warn) {\n try {\n weights.prob = f.ReadFloat();\n if (weights.prob > 0.0) {\n warn.Warn(weights.prob);\n weights.prob = 0.0;\n }\n for (WordIndex *vocab_out = reverse_indices + n - 1; vocab_out >= reverse_indices; --vocab_out) {\n *vocab_out = vocab.Index(f.ReadDelimited(kARPASpaces));\n }\n ReadBackoff(f, weights);\n } catch(util::Exception &e) {\n e << \" in the \" << static_cast(n) << \"-gram at byte \" << f.Offset();\n throw;\n }\n}\n\n} // namespace lm\n\n#endif // LM_READ_ARPA__\n" }, { "path": "src/kenlm/lm/return.hh", "content": "#ifndef LM_RETURN__\n#define LM_RETURN__\n\n#include \n\nnamespace lm {\n/* Structure returned by scoring routines. */\nstruct FullScoreReturn {\n // log10 probability\n float prob;\n\n /* The length of n-gram matched. Do not use this for recombination. \n * Consider a model containing only the following n-grams:\n * -1 foo\n * -3.14 bar\n * -2.718 baz -5\n * -6 foo bar\n *\n * If you score ``bar'' then ngram_length is 1 and recombination state is the\n * empty string because bar has zero backoff and does not extend to the\n * right. \n * If you score ``foo'' then ngram_length is 1 and recombination state is \n * ``foo''. \n *\n * Ideally, keep output states around and compare them. Failing that,\n * get out_state.ValidLength() and use that length for recombination.\n */\n unsigned char ngram_length;\n\n /* Left extension information. If independent_left is set, then prob is\n * independent of words to the left (up to additional backoff). Otherwise,\n * extend_left indicates how to efficiently extend further to the left. \n */\n bool independent_left;\n uint64_t extend_left; // Defined only if independent_left\n\n // Rest cost for extension to the left.\n float rest;\n};\n\n} // namespace lm\n#endif // LM_RETURN__\n" }, { "path": "src/kenlm/lm/search_hashed.cc", "content": "#include \"lm/search_hashed.hh\"\n\n#include \"lm/binary_format.hh\"\n#include \"lm/blank.hh\"\n#include \"lm/lm_exception.hh\"\n#include \"lm/model.hh\"\n#include \"lm/read_arpa.hh\"\n#include \"lm/value.hh\"\n#include \"lm/vocab.hh\"\n\n#include \"util/bit_packing.hh\"\n#include \"util/file_piece.hh\"\n\n#include \n\nnamespace lm {\nnamespace ngram {\n\nclass ProbingModel;\n\nnamespace {\n\n/* These are passed to ReadNGrams so that n-grams with zero backoff that appear as context will still be used in state. */\ntemplate class ActivateLowerMiddle {\n public:\n explicit ActivateLowerMiddle(Middle &middle) : modify_(middle) {}\n\n void operator()(const WordIndex *vocab_ids, const unsigned int n) {\n uint64_t hash = static_cast(vocab_ids[1]);\n for (const WordIndex *i = vocab_ids + 2; i < vocab_ids + n; ++i) {\n hash = detail::CombineWordHash(hash, *i);\n }\n typename Middle::MutableIterator i;\n // TODO: somehow get text of n-gram for this error message.\n if (!modify_.UnsafeMutableFind(hash, i))\n UTIL_THROW(FormatLoadException, \"The context of every \" << n << \"-gram should appear as a \" << (n-1) << \"-gram\");\n SetExtension(i->value.backoff);\n }\n\n private:\n Middle &modify_;\n};\n\ntemplate class ActivateUnigram {\n public:\n explicit ActivateUnigram(Weights *unigram) : modify_(unigram) {}\n\n void operator()(const WordIndex *vocab_ids, const unsigned int /*n*/) {\n // assert(n == 2);\n SetExtension(modify_[vocab_ids[1]].backoff);\n }\n\n private:\n Weights *modify_;\n};\n\n// Find the lower order entry, inserting blanks along the way as necessary. \ntemplate void FindLower(\n const std::vector &keys,\n typename Value::Weights &unigram,\n std::vector > &middle,\n std::vector &between) {\n typename util::ProbingHashTable::MutableIterator iter;\n typename Value::ProbingEntry entry;\n // Backoff will always be 0.0. We'll get the probability and rest in another pass.\n entry.value.backoff = kNoExtensionBackoff;\n // Go back and find the longest right-aligned entry, informing it that it extends left. Normally this will match immediately, but sometimes SRI is dumb. \n for (int lower = keys.size() - 2; ; --lower) {\n if (lower == -1) {\n between.push_back(&unigram);\n return;\n }\n entry.key = keys[lower];\n bool found = middle[lower].FindOrInsert(entry, iter);\n between.push_back(&iter->value);\n if (found) return;\n }\n}\n\n// Between usually has single entry, the value to adjust. But sometimes SRI stupidly pruned entries so it has unitialized blank values to be set here. \ntemplate void AdjustLower(\n const Added &added,\n const Build &build,\n std::vector &between, \n const unsigned int n,\n const std::vector &vocab_ids,\n typename Build::Value::Weights *unigrams,\n std::vector > &middle) {\n typedef typename Build::Value Value;\n if (between.size() == 1) {\n build.MarkExtends(*between.front(), added);\n return;\n }\n typedef util::ProbingHashTable Middle;\n float prob = -fabs(between.back()->prob);\n // Order of the n-gram on which probabilities are based. \n unsigned char basis = n - between.size();\n assert(basis != 0);\n typename Build::Value::Weights **change = &between.back();\n // Skip the basis.\n --change;\n if (basis == 1) {\n // Hallucinate a bigram based on a unigram's backoff and a unigram probability. \n float &backoff = unigrams[vocab_ids[1]].backoff;\n SetExtension(backoff);\n prob += backoff;\n (*change)->prob = prob;\n build.SetRest(&*vocab_ids.begin(), 2, **change);\n basis = 2;\n --change;\n }\n uint64_t backoff_hash = static_cast(vocab_ids[1]);\n for (unsigned char i = 2; i <= basis; ++i) {\n backoff_hash = detail::CombineWordHash(backoff_hash, vocab_ids[i]);\n }\n for (; basis < n - 1; ++basis, --change) {\n typename Middle::MutableIterator gotit;\n if (middle[basis - 2].UnsafeMutableFind(backoff_hash, gotit)) {\n float &backoff = gotit->value.backoff;\n SetExtension(backoff);\n prob += backoff;\n }\n (*change)->prob = prob;\n build.SetRest(&*vocab_ids.begin(), basis + 1, **change);\n backoff_hash = detail::CombineWordHash(backoff_hash, vocab_ids[basis+1]);\n }\n\n typename std::vector::const_iterator i(between.begin());\n build.MarkExtends(**i, added);\n const typename Value::Weights *longer = *i;\n // Everything has probability but is not marked as extending. \n for (++i; i != between.end(); ++i) {\n build.MarkExtends(**i, *longer);\n longer = *i;\n }\n}\n\n// Continue marking lower entries even they know that they extend left. This is used for upper/lower bounds. \ntemplate void MarkLower(\n const std::vector &keys,\n const Build &build,\n typename Build::Value::Weights &unigram,\n std::vector > &middle,\n int start_order,\n const typename Build::Value::Weights &longer) {\n if (start_order == 0) return;\n typename util::ProbingHashTable::MutableIterator iter;\n // Hopefully the compiler will realize that if MarkExtends always returns false, it can simplify this code. \n for (int even_lower = start_order - 2 /* index in middle */; ; --even_lower) {\n if (even_lower == -1) {\n build.MarkExtends(unigram, longer);\n return;\n }\n middle[even_lower].UnsafeMutableFind(keys[even_lower], iter);\n if (!build.MarkExtends(iter->value, longer)) return;\n }\n}\n\ntemplate void ReadNGrams(\n util::FilePiece &f,\n const unsigned int n,\n const size_t count,\n const ProbingVocabulary &vocab,\n const Build &build,\n typename Build::Value::Weights *unigrams,\n std::vector > &middle,\n Activate activate,\n Store &store,\n PositiveProbWarn &warn) {\n typedef typename Build::Value Value;\n typedef util::ProbingHashTable Middle;\n assert(n >= 2);\n ReadNGramHeader(f, n);\n\n // Both vocab_ids and keys are non-empty because n >= 2.\n // vocab ids of words in reverse order.\n std::vector vocab_ids(n);\n std::vector keys(n-1);\n typename Store::Entry entry;\n std::vector between;\n for (size_t i = 0; i < count; ++i) {\n ReadNGram(f, n, vocab, &*vocab_ids.begin(), entry.value, warn);\n build.SetRest(&*vocab_ids.begin(), n, entry.value);\n\n keys[0] = detail::CombineWordHash(static_cast(vocab_ids.front()), vocab_ids[1]);\n for (unsigned int h = 1; h < n - 1; ++h) {\n keys[h] = detail::CombineWordHash(keys[h-1], vocab_ids[h+1]);\n }\n // Initially the sign bit is on, indicating it does not extend left. Most already have this but there might +0.0. \n util::SetSign(entry.value.prob);\n entry.key = keys[n-2];\n\n store.Insert(entry);\n between.clear();\n FindLower(keys, unigrams[vocab_ids.front()], middle, between);\n AdjustLower(entry.value, build, between, n, vocab_ids, unigrams, middle);\n if (Build::kMarkEvenLower) MarkLower(keys, build, unigrams[vocab_ids.front()], middle, n - between.size() - 1, *between.back());\n activate(&*vocab_ids.begin(), n);\n }\n\n store.FinishedInserting();\n}\n\n} // namespace\nnamespace detail {\n \ntemplate uint8_t *HashedSearch::SetupMemory(uint8_t *start, const std::vector &counts, const Config &config) {\n std::size_t allocated = Unigram::Size(counts[0]);\n unigram_ = Unigram(start, counts[0], allocated);\n start += allocated;\n for (unsigned int n = 2; n < counts.size(); ++n) {\n allocated = Middle::Size(counts[n - 1], config.probing_multiplier);\n middle_.push_back(Middle(start, allocated));\n start += allocated;\n }\n allocated = Longest::Size(counts.back(), config.probing_multiplier);\n longest_ = Longest(start, allocated);\n start += allocated;\n return start;\n}\n\ntemplate void HashedSearch::InitializeFromARPA(const char * /*file*/, util::FilePiece &f, const std::vector &counts, const Config &config, ProbingVocabulary &vocab, Backing &backing) {\n // TODO: fix sorted.\n SetupMemory(GrowForSearch(config, vocab.UnkCountChangePadding(), Size(counts, config), backing), counts, config);\n\n PositiveProbWarn warn(config.positive_log_probability);\n Read1Grams(f, counts[0], vocab, unigram_.Raw(), warn);\n CheckSpecials(config, vocab);\n DispatchBuild(f, counts, config, vocab, warn);\n}\n\ntemplate <> void HashedSearch::DispatchBuild(util::FilePiece &f, const std::vector &counts, const Config &config, const ProbingVocabulary &vocab, PositiveProbWarn &warn) {\n NoRestBuild build;\n ApplyBuild(f, counts, vocab, warn, build);\n}\n\ntemplate <> void HashedSearch::DispatchBuild(util::FilePiece &f, const std::vector &counts, const Config &config, const ProbingVocabulary &vocab, PositiveProbWarn &warn) {\n switch (config.rest_function) {\n case Config::REST_MAX:\n {\n MaxRestBuild build;\n ApplyBuild(f, counts, vocab, warn, build);\n }\n break;\n case Config::REST_LOWER:\n {\n LowerRestBuild build(config, counts.size(), vocab);\n ApplyBuild(f, counts, vocab, warn, build);\n }\n break;\n }\n}\n\ntemplate template void HashedSearch::ApplyBuild(util::FilePiece &f, const std::vector &counts, const ProbingVocabulary &vocab, PositiveProbWarn &warn, const Build &build) {\n for (WordIndex i = 0; i < counts[0]; ++i) {\n build.SetRest(&i, (unsigned int)1, unigram_.Raw()[i]);\n }\n\n try {\n if (counts.size() > 2) {\n ReadNGrams, Middle>(\n f, 2, counts[1], vocab, build, unigram_.Raw(), middle_, ActivateUnigram(unigram_.Raw()), middle_[0], warn);\n }\n for (unsigned int n = 3; n < counts.size(); ++n) {\n ReadNGrams, Middle>(\n f, n, counts[n-1], vocab, build, unigram_.Raw(), middle_, ActivateLowerMiddle(middle_[n-3]), middle_[n-2], warn);\n }\n if (counts.size() > 2) {\n ReadNGrams, Longest>(\n f, counts.size(), counts[counts.size() - 1], vocab, build, unigram_.Raw(), middle_, ActivateLowerMiddle(middle_.back()), longest_, warn);\n } else {\n ReadNGrams, Longest>(\n f, counts.size(), counts[counts.size() - 1], vocab, build, unigram_.Raw(), middle_, ActivateUnigram(unigram_.Raw()), longest_, warn);\n }\n } catch (util::ProbingSizeException &e) {\n UTIL_THROW(util::ProbingSizeException, \"Avoid pruning n-grams like \\\"bar baz quux\\\" when \\\"foo bar baz quux\\\" is still in the model. KenLM will work when this pruning happens, but the probing model assumes these events are rare enough that using blank space in the probing hash table will cover all of them. Increase probing_multiplier (-p to build_binary) to add more blank spaces.\\n\");\n }\n ReadEnd(f);\n}\n\ntemplate void HashedSearch::LoadedBinary() {\n unigram_.LoadedBinary();\n for (typename std::vector::iterator i = middle_.begin(); i != middle_.end(); ++i) {\n i->LoadedBinary();\n }\n longest_.LoadedBinary();\n}\n\ntemplate class HashedSearch;\ntemplate class HashedSearch;\n\n} // namespace detail\n} // namespace ngram\n} // namespace lm\n" }, { "path": "src/kenlm/lm/search_hashed.hh", "content": "#ifndef LM_SEARCH_HASHED__\n#define LM_SEARCH_HASHED__\n\n#include \"lm/model_type.hh\"\n#include \"lm/config.hh\"\n#include \"lm/read_arpa.hh\"\n#include \"lm/return.hh\"\n#include \"lm/weights.hh\"\n\n#include \"util/bit_packing.hh\"\n#include \"util/probing_hash_table.hh\"\n\n#include \n#include \n#include \n\nnamespace util { class FilePiece; }\n\nnamespace lm {\nnamespace ngram {\nstruct Backing;\nclass ProbingVocabulary;\nnamespace detail {\n\ninline uint64_t CombineWordHash(uint64_t current, const WordIndex next) {\n uint64_t ret = (current * 8978948897894561157ULL) ^ (static_cast(1 + next) * 17894857484156487943ULL);\n return ret;\n}\n\n#pragma pack(push)\n#pragma pack(4)\nstruct ProbEntry {\n uint64_t key;\n Prob value;\n typedef uint64_t Key;\n typedef Prob Value;\n uint64_t GetKey() const {\n return key;\n }\n};\n\n#pragma pack(pop)\n\nclass LongestPointer {\n public:\n explicit LongestPointer(const float &to) : to_(&to) {}\n\n LongestPointer() : to_(NULL) {}\n\n bool Found() const {\n return to_ != NULL;\n }\n\n float Prob() const {\n return *to_;\n }\n\n private:\n const float *to_;\n};\n\ntemplate class HashedSearch {\n public:\n typedef uint64_t Node;\n\n typedef typename Value::ProbingProxy UnigramPointer;\n typedef typename Value::ProbingProxy MiddlePointer;\n typedef ::lm::ngram::detail::LongestPointer LongestPointer;\n\n static const ModelType kModelType = Value::kProbingModelType;\n static const bool kDifferentRest = Value::kDifferentRest;\n static const unsigned int kVersion = 0;\n\n // TODO: move probing_multiplier here with next binary file format update. \n static void UpdateConfigFromBinary(int, const std::vector &, Config &) {}\n\n static uint64_t Size(const std::vector &counts, const Config &config) {\n uint64_t ret = Unigram::Size(counts[0]);\n for (unsigned char n = 1; n < counts.size() - 1; ++n) {\n ret += Middle::Size(counts[n], config.probing_multiplier);\n }\n return ret + Longest::Size(counts.back(), config.probing_multiplier);\n }\n\n uint8_t *SetupMemory(uint8_t *start, const std::vector &counts, const Config &config);\n\n void InitializeFromARPA(const char *file, util::FilePiece &f, const std::vector &counts, const Config &config, ProbingVocabulary &vocab, Backing &backing);\n\n void LoadedBinary();\n\n unsigned char Order() const {\n return middle_.size() + 2;\n }\n\n typename Value::Weights &UnknownUnigram() { return unigram_.Unknown(); }\n\n UnigramPointer LookupUnigram(WordIndex word, Node &next, bool &independent_left, uint64_t &extend_left) const {\n extend_left = static_cast(word);\n next = extend_left;\n UnigramPointer ret(unigram_.Lookup(word));\n independent_left = ret.IndependentLeft();\n return ret;\n }\n\n#pragma GCC diagnostic ignored \"-Wuninitialized\"\n MiddlePointer Unpack(uint64_t extend_pointer, unsigned char extend_length, Node &node) const {\n node = extend_pointer;\n typename Middle::ConstIterator found;\n bool got = middle_[extend_length - 2].Find(extend_pointer, found);\n assert(got);\n (void)got;\n return MiddlePointer(found->value);\n }\n\n MiddlePointer LookupMiddle(unsigned char order_minus_2, WordIndex word, Node &node, bool &independent_left, uint64_t &extend_pointer) const {\n node = CombineWordHash(node, word);\n typename Middle::ConstIterator found;\n if (!middle_[order_minus_2].Find(node, found)) {\n independent_left = true;\n return MiddlePointer();\n }\n extend_pointer = node;\n MiddlePointer ret(found->value);\n independent_left = ret.IndependentLeft();\n return ret;\n }\n\n LongestPointer LookupLongest(WordIndex word, const Node &node) const {\n // Sign bit is always on because longest n-grams do not extend left. \n typename Longest::ConstIterator found;\n if (!longest_.Find(CombineWordHash(node, word), found)) return LongestPointer();\n return LongestPointer(found->value.prob);\n }\n\n // Generate a node without necessarily checking that it actually exists. \n // Optionally return false if it's know to not exist. \n bool FastMakeNode(const WordIndex *begin, const WordIndex *end, Node &node) const {\n assert(begin != end);\n node = static_cast(*begin);\n for (const WordIndex *i = begin + 1; i < end; ++i) {\n node = CombineWordHash(node, *i);\n }\n return true;\n }\n\n private:\n // Interpret config's rest cost build policy and pass the right template argument to ApplyBuild. \n void DispatchBuild(util::FilePiece &f, const std::vector &counts, const Config &config, const ProbingVocabulary &vocab, PositiveProbWarn &warn);\n\n template void ApplyBuild(util::FilePiece &f, const std::vector &counts, const ProbingVocabulary &vocab, PositiveProbWarn &warn, const Build &build);\n\n class Unigram {\n public:\n Unigram() {}\n\n Unigram(void *start, uint64_t count, std::size_t /*allocated*/) : \n unigram_(static_cast(start))\n#ifdef DEBUG\n , count_(count)\n#endif\n {}\n\n static uint64_t Size(uint64_t count) {\n return (count + 1) * sizeof(typename Value::Weights); // +1 for hallucinate \n }\n\n const typename Value::Weights &Lookup(WordIndex index) const {\n#ifdef DEBUG\n assert(index < count_);\n#endif\n return unigram_[index];\n }\n\n typename Value::Weights &Unknown() { return unigram_[0]; }\n\n void LoadedBinary() {}\n\n // For building.\n typename Value::Weights *Raw() { return unigram_; }\n\n private:\n typename Value::Weights *unigram_;\n#ifdef DEBUG\n uint64_t count_;\n#endif\n };\n\n Unigram unigram_;\n\n typedef util::ProbingHashTable Middle;\n std::vector middle_;\n\n typedef util::ProbingHashTable Longest;\n Longest longest_;\n};\n\n} // namespace detail\n} // namespace ngram\n} // namespace lm\n\n#endif // LM_SEARCH_HASHED__\n" }, { "path": "src/kenlm/lm/search_trie.cc", "content": "/* This is where the trie is built. It's on-disk. */\n#include \"lm/search_trie.hh\"\n\n#include \"lm/bhiksha.hh\"\n#include \"lm/binary_format.hh\"\n#include \"lm/blank.hh\"\n#include \"lm/lm_exception.hh\"\n#include \"lm/max_order.hh\"\n#include \"lm/quantize.hh\"\n#include \"lm/trie.hh\"\n#include \"lm/trie_sort.hh\"\n#include \"lm/vocab.hh\"\n#include \"lm/weights.hh\"\n#include \"lm/word_index.hh\"\n#include \"util/ersatz_progress.hh\"\n#include \"util/mmap.hh\"\n#include \"util/proxy_iterator.hh\"\n#include \"util/scoped.hh\"\n#include \"util/sized_iterator.hh\"\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#if defined(_WIN32) || defined(_WIN64)\n#include \n#endif\n\nnamespace lm {\nnamespace ngram {\nnamespace trie {\nnamespace {\n\nvoid ReadOrThrow(FILE *from, void *data, size_t size) {\n UTIL_THROW_IF(1 != std::fread(data, size, 1, from), util::ErrnoException, \"Short read\");\n}\n\nint Compare(unsigned char order, const void *first_void, const void *second_void) {\n const WordIndex *first = reinterpret_cast(first_void), *second = reinterpret_cast(second_void);\n const WordIndex *end = first + order;\n for (; first != end; ++first, ++second) {\n if (*first < *second) return -1;\n if (*first > *second) return 1;\n }\n return 0;\n}\n\nstruct ProbPointer {\n unsigned char array;\n uint64_t index;\n};\n\n// Array of n-grams and float indices.\nclass BackoffMessages {\n public:\n void Init(std::size_t entry_size) {\n current_ = NULL;\n allocated_ = NULL;\n entry_size_ = entry_size;\n }\n\n void Add(const WordIndex *to, ProbPointer index) {\n while (current_ + entry_size_ > allocated_) {\n std::size_t allocated_size = allocated_ - (uint8_t*)backing_.get();\n Resize(std::max(allocated_size * 2, entry_size_));\n }\n memcpy(current_, to, entry_size_ - sizeof(ProbPointer));\n *reinterpret_cast(current_ + entry_size_ - sizeof(ProbPointer)) = index;\n current_ += entry_size_;\n }\n\n void Apply(float *const *const base, FILE *unigrams) {\n FinishedAdding();\n if (current_ == allocated_) return;\n rewind(unigrams);\n ProbBackoff weights;\n WordIndex unigram = 0;\n ReadOrThrow(unigrams, &weights, sizeof(weights));\n for (; current_ != allocated_; current_ += entry_size_) {\n const WordIndex &cur_word = *reinterpret_cast(current_);\n for (; unigram < cur_word; ++unigram) {\n ReadOrThrow(unigrams, &weights, sizeof(weights));\n }\n if (!HasExtension(weights.backoff)) {\n weights.backoff = kExtensionBackoff;\n UTIL_THROW_IF(fseek(unigrams, -sizeof(weights), SEEK_CUR), util::ErrnoException, \"Seeking backwards to denote unigram extension failed.\");\n util::WriteOrThrow(unigrams, &weights, sizeof(weights));\n }\n const ProbPointer &write_to = *reinterpret_cast(current_ + sizeof(WordIndex));\n base[write_to.array][write_to.index] += weights.backoff;\n }\n backing_.reset();\n }\n\n void Apply(float *const *const base, RecordReader &reader) {\n FinishedAdding();\n if (current_ == allocated_) return;\n // We'll also use the same buffer to record messages to blanks that they extend.\n WordIndex *extend_out = reinterpret_cast(current_);\n const unsigned char order = (entry_size_ - sizeof(ProbPointer)) / sizeof(WordIndex);\n for (reader.Rewind(); reader && (current_ != allocated_); ) {\n switch (Compare(order, reader.Data(), current_)) {\n case -1:\n ++reader;\n break;\n case 1:\n // Message but nobody to receive it. Write it down at the beginning of the buffer so we can inform this blank that it extends.\n for (const WordIndex *w = reinterpret_cast(current_); w != reinterpret_cast(current_) + order; ++w, ++extend_out) *extend_out = *w;\n current_ += entry_size_;\n break;\n case 0:\n float &backoff = reinterpret_cast((uint8_t*)reader.Data() + order * sizeof(WordIndex))->backoff;\n if (!HasExtension(backoff)) {\n backoff = kExtensionBackoff;\n reader.Overwrite(&backoff, sizeof(float));\n } else {\n const ProbPointer &write_to = *reinterpret_cast(current_ + entry_size_ - sizeof(ProbPointer));\n base[write_to.array][write_to.index] += backoff;\n }\n current_ += entry_size_;\n break;\n }\n }\n // Now this is a list of blanks that extend right.\n entry_size_ = sizeof(WordIndex) * order;\n Resize(sizeof(WordIndex) * (extend_out - (const WordIndex*)backing_.get()));\n current_ = (uint8_t*)backing_.get();\n }\n\n // Call after Apply\n bool Extends(unsigned char order, const WordIndex *words) {\n if (current_ == allocated_) return false;\n assert(order * sizeof(WordIndex) == entry_size_);\n while (true) {\n switch(Compare(order, words, current_)) {\n case 1:\n current_ += entry_size_;\n if (current_ == allocated_) return false;\n break;\n case -1:\n return false;\n case 0:\n return true;\n }\n }\n }\n\n private:\n void FinishedAdding() {\n Resize(current_ - (uint8_t*)backing_.get());\n // Sort requests in same order as files.\n std::sort(\n util::SizedIterator(util::SizedProxy(backing_.get(), entry_size_)),\n util::SizedIterator(util::SizedProxy(current_, entry_size_)),\n util::SizedCompare(EntryCompare((entry_size_ - sizeof(ProbPointer)) / sizeof(WordIndex))));\n current_ = (uint8_t*)backing_.get();\n }\n\n void Resize(std::size_t to) {\n std::size_t current = current_ - (uint8_t*)backing_.get();\n backing_.call_realloc(to);\n current_ = (uint8_t*)backing_.get() + current;\n allocated_ = (uint8_t*)backing_.get() + to;\n }\n\n util::scoped_malloc backing_;\n\n uint8_t *current_, *allocated_;\n\n std::size_t entry_size_;\n};\n\nconst float kBadProb = std::numeric_limits::infinity();\n\nclass SRISucks {\n public:\n SRISucks() {\n for (BackoffMessages *i = messages_; i != messages_ + KENLM_MAX_ORDER - 1; ++i)\n i->Init(sizeof(ProbPointer) + sizeof(WordIndex) * (i - messages_ + 1));\n }\n\n void Send(unsigned char begin, unsigned char order, const WordIndex *to, float prob_basis) {\n assert(prob_basis != kBadProb);\n ProbPointer pointer;\n pointer.array = order - 1;\n pointer.index = values_[order - 1].size();\n for (unsigned char i = begin; i < order; ++i) {\n messages_[i - 1].Add(to, pointer);\n }\n values_[order - 1].push_back(prob_basis);\n }\n\n void ObtainBackoffs(unsigned char total_order, FILE *unigram_file, RecordReader *reader) {\n for (unsigned char i = 0; i < KENLM_MAX_ORDER - 1; ++i) {\n it_[i] = values_[i].empty() ? NULL : &*values_[i].begin();\n }\n messages_[0].Apply(it_, unigram_file);\n BackoffMessages *messages = messages_ + 1;\n const RecordReader *end = reader + total_order - 2 /* exclude unigrams and longest order */;\n for (; reader != end; ++messages, ++reader) {\n messages->Apply(it_, *reader);\n }\n }\n\n ProbBackoff GetBlank(unsigned char total_order, unsigned char order, const WordIndex *indices) {\n assert(order > 1);\n ProbBackoff ret;\n ret.prob = *(it_[order - 1]++);\n ret.backoff = ((order != total_order - 1) && messages_[order - 1].Extends(order, indices)) ? kExtensionBackoff : kNoExtensionBackoff;\n return ret;\n }\n\n const std::vector &Values(unsigned char order) const {\n return values_[order - 1];\n }\n\n private:\n // This used to be one array. Then I needed to separate it by order for quantization to work.\n std::vector values_[KENLM_MAX_ORDER - 1];\n BackoffMessages messages_[KENLM_MAX_ORDER - 1];\n\n float *it_[KENLM_MAX_ORDER - 1];\n};\n\nclass FindBlanks {\n public:\n FindBlanks(unsigned char order, const ProbBackoff *unigrams, SRISucks &messages)\n : counts_(order), unigrams_(unigrams), sri_(messages) {}\n\n float UnigramProb(WordIndex index) const {\n return unigrams_[index].prob;\n }\n\n void Unigram(WordIndex /*index*/) {\n ++counts_[0];\n }\n\n void MiddleBlank(const unsigned char order, const WordIndex *indices, unsigned char lower, float prob_basis) {\n sri_.Send(lower, order, indices + 1, prob_basis);\n ++counts_[order - 1];\n }\n\n void Middle(const unsigned char order, const void * /*data*/) {\n ++counts_[order - 1];\n }\n\n void Longest(const void * /*data*/) {\n ++counts_.back();\n }\n\n // Unigrams wrote one past.\n void Cleanup() {\n --counts_[0];\n }\n\n const std::vector &Counts() const {\n return counts_;\n }\n\n private:\n std::vector counts_;\n\n const ProbBackoff *unigrams_;\n\n SRISucks &sri_;\n};\n\n// Phase to actually write n-grams to the trie.\ntemplate class WriteEntries {\n public:\n WriteEntries(RecordReader *contexts, const Quant &quant, UnigramValue *unigrams, BitPackedMiddle *middle, BitPackedLongest &longest, unsigned char order, SRISucks &sri) :\n contexts_(contexts),\n quant_(quant),\n unigrams_(unigrams),\n middle_(middle),\n longest_(longest),\n bigram_pack_((order == 2) ? static_cast(longest_) : static_cast(*middle_)),\n order_(order),\n sri_(sri) {}\n\n float UnigramProb(WordIndex index) const { return unigrams_[index].weights.prob; }\n\n void Unigram(WordIndex word) {\n unigrams_[word].next = bigram_pack_.InsertIndex();\n }\n\n void MiddleBlank(const unsigned char order, const WordIndex *indices, unsigned char /*lower*/, float /*prob_base*/) {\n ProbBackoff weights = sri_.GetBlank(order_, order, indices);\n typename Quant::MiddlePointer(quant_, order - 2, middle_[order - 2].Insert(indices[order - 1])).Write(weights.prob, weights.backoff);\n }\n\n void Middle(const unsigned char order, const void *data) {\n RecordReader &context = contexts_[order - 1];\n const WordIndex *words = reinterpret_cast(data);\n ProbBackoff weights = *reinterpret_cast(words + order);\n if (context && !memcmp(data, context.Data(), sizeof(WordIndex) * order)) {\n SetExtension(weights.backoff);\n ++context;\n }\n typename Quant::MiddlePointer(quant_, order - 2, middle_[order - 2].Insert(words[order - 1])).Write(weights.prob, weights.backoff);\n }\n\n void Longest(const void *data) {\n const WordIndex *words = reinterpret_cast(data);\n typename Quant::LongestPointer(quant_, longest_.Insert(words[order_ - 1])).Write(reinterpret_cast(words + order_)->prob);\n }\n\n void Cleanup() {}\n\n private:\n RecordReader *contexts_;\n const Quant &quant_;\n UnigramValue *const unigrams_;\n BitPackedMiddle *const middle_;\n BitPackedLongest &longest_;\n BitPacked &bigram_pack_;\n const unsigned char order_;\n SRISucks &sri_;\n};\n\nstruct Gram {\n Gram(const WordIndex *in_begin, unsigned char order) : begin(in_begin), end(in_begin + order) {}\n\n const WordIndex *begin, *end;\n\n // For queue, this is the direction we want.\n bool operator<(const Gram &other) const {\n return std::lexicographical_compare(other.begin, other.end, begin, end);\n }\n};\n\ntemplate class BlankManager {\n public:\n BlankManager(unsigned char total_order, Doing &doing) : total_order_(total_order), been_length_(0), doing_(doing) {\n for (float *i = basis_; i != basis_ + KENLM_MAX_ORDER - 1; ++i) *i = kBadProb;\n }\n\n void Visit(const WordIndex *to, unsigned char length, float prob) {\n basis_[length - 1] = prob;\n unsigned char overlap = std::min(length - 1, been_length_);\n const WordIndex *cur;\n WordIndex *pre;\n for (cur = to, pre = been_; cur != to + overlap; ++cur, ++pre) {\n if (*pre != *cur) break;\n }\n if (cur == to + length - 1) {\n *pre = *cur;\n been_length_ = length;\n return;\n }\n // There are blanks to insert starting with order blank.\n unsigned char blank = cur - to + 1;\n UTIL_THROW_IF(blank == 1, FormatLoadException, \"Missing a unigram that appears as context.\");\n const float *lower_basis;\n for (lower_basis = basis_ + blank - 2; *lower_basis == kBadProb; --lower_basis) {}\n unsigned char based_on = lower_basis - basis_ + 1;\n for (; cur != to + length - 1; ++blank, ++cur, ++pre) {\n assert(*lower_basis != kBadProb);\n doing_.MiddleBlank(blank, to, based_on, *lower_basis);\n *pre = *cur;\n // Mark that the probability is a blank so it shouldn't be used as the basis for a later n-gram.\n basis_[blank - 1] = kBadProb;\n }\n *pre = *cur;\n been_length_ = length;\n }\n\n private:\n const unsigned char total_order_;\n\n WordIndex been_[KENLM_MAX_ORDER];\n unsigned char been_length_;\n\n float basis_[KENLM_MAX_ORDER];\n\n Doing &doing_;\n};\n\ntemplate void RecursiveInsert(const unsigned char total_order, const WordIndex unigram_count, RecordReader *input, std::ostream *progress_out, const char *message, Doing &doing) {\n util::ErsatzProgress progress(unigram_count + 1, progress_out, message);\n WordIndex unigram = 0;\n std::priority_queue grams;\n grams.push(Gram(&unigram, 1));\n for (unsigned char i = 2; i <= total_order; ++i) {\n if (input[i-2]) grams.push(Gram(reinterpret_cast(input[i-2].Data()), i));\n }\n\n BlankManager blank(total_order, doing);\n\n while (true) {\n Gram top = grams.top();\n grams.pop();\n unsigned char order = top.end - top.begin;\n if (order == 1) {\n blank.Visit(&unigram, 1, doing.UnigramProb(unigram));\n doing.Unigram(unigram);\n progress.Set(unigram);\n if (++unigram == unigram_count + 1) break;\n grams.push(top);\n } else {\n if (order == total_order) {\n blank.Visit(top.begin, order, reinterpret_cast(top.end)->prob);\n doing.Longest(top.begin);\n } else {\n blank.Visit(top.begin, order, reinterpret_cast(top.end)->prob);\n doing.Middle(order, top.begin);\n }\n RecordReader &reader = input[order - 2];\n if (++reader) grams.push(top);\n }\n }\n assert(grams.empty());\n doing.Cleanup();\n}\n\nvoid SanityCheckCounts(const std::vector &initial, const std::vector &fixed) {\n if (fixed[0] != initial[0]) UTIL_THROW(util::Exception, \"Unigram count should be constant but initial is \" << initial[0] << \" and recounted is \" << fixed[0]);\n if (fixed.back() != initial.back()) UTIL_THROW(util::Exception, \"Longest count should be constant but it changed from \" << initial.back() << \" to \" << fixed.back());\n for (unsigned char i = 0; i < initial.size(); ++i) {\n if (fixed[i] < initial[i]) UTIL_THROW(util::Exception, \"Counts came out lower than expected. This shouldn't happen\");\n }\n}\n\ntemplate void TrainQuantizer(uint8_t order, uint64_t count, const std::vector &additional, RecordReader &reader, util::ErsatzProgress &progress, Quant &quant) {\n std::vector probs(additional), backoffs;\n probs.reserve(count + additional.size());\n backoffs.reserve(count);\n for (reader.Rewind(); reader; ++reader) {\n const ProbBackoff &weights = *reinterpret_cast(reinterpret_cast(reader.Data()) + sizeof(WordIndex) * order);\n probs.push_back(weights.prob);\n if (weights.backoff != 0.0) backoffs.push_back(weights.backoff);\n ++progress;\n }\n quant.Train(order, probs, backoffs);\n}\n\ntemplate void TrainProbQuantizer(uint8_t order, uint64_t count, RecordReader &reader, util::ErsatzProgress &progress, Quant &quant) {\n std::vector probs, backoffs;\n probs.reserve(count);\n for (reader.Rewind(); reader; ++reader) {\n const Prob &weights = *reinterpret_cast(reinterpret_cast(reader.Data()) + sizeof(WordIndex) * order);\n probs.push_back(weights.prob);\n ++progress;\n }\n quant.TrainProb(order, probs);\n}\n\nvoid PopulateUnigramWeights(FILE *file, WordIndex unigram_count, RecordReader &contexts, UnigramValue *unigrams) {\n // Fill unigram probabilities.\n try {\n rewind(file);\n for (WordIndex i = 0; i < unigram_count; ++i) {\n ReadOrThrow(file, &unigrams[i].weights, sizeof(ProbBackoff));\n if (contexts && *reinterpret_cast(contexts.Data()) == i) {\n SetExtension(unigrams[i].weights.backoff);\n ++contexts;\n }\n }\n } catch (util::Exception &e) {\n e << \" while re-reading unigram probabilities\";\n throw;\n }\n}\n\n} // namespace\n\ntemplate void BuildTrie(SortedFiles &files, std::vector &counts, const Config &config, TrieSearch &out, Quant &quant, const SortedVocabulary &vocab, Backing &backing) {\n RecordReader inputs[KENLM_MAX_ORDER - 1];\n RecordReader contexts[KENLM_MAX_ORDER - 1];\n\n for (unsigned char i = 2; i <= counts.size(); ++i) {\n inputs[i-2].Init(files.Full(i), i * sizeof(WordIndex) + (i == counts.size() ? sizeof(Prob) : sizeof(ProbBackoff)));\n contexts[i-2].Init(files.Context(i), (i-1) * sizeof(WordIndex));\n }\n\n SRISucks sri;\n std::vector fixed_counts;\n util::scoped_FILE unigram_file;\n util::scoped_fd unigram_fd(files.StealUnigram());\n {\n util::scoped_memory unigrams;\n MapRead(util::POPULATE_OR_READ, unigram_fd.get(), 0, counts[0] * sizeof(ProbBackoff), unigrams);\n FindBlanks finder(counts.size(), reinterpret_cast(unigrams.get()), sri);\n RecursiveInsert(counts.size(), counts[0], inputs, config.ProgressMessages(), \"Identifying n-grams omitted by SRI\", finder);\n fixed_counts = finder.Counts();\n }\n unigram_file.reset(util::FDOpenOrThrow(unigram_fd));\n for (const RecordReader *i = inputs; i != inputs + counts.size() - 2; ++i) {\n if (*i) UTIL_THROW(FormatLoadException, \"There's a bug in the trie implementation: the \" << (i - inputs + 2) << \"-gram table did not complete reading\");\n }\n SanityCheckCounts(counts, fixed_counts);\n counts = fixed_counts;\n\n sri.ObtainBackoffs(counts.size(), unigram_file.get(), inputs);\n\n out.SetupMemory(GrowForSearch(config, vocab.UnkCountChangePadding(), TrieSearch::Size(fixed_counts, config), backing), fixed_counts, config);\n\n for (unsigned char i = 2; i <= counts.size(); ++i) {\n inputs[i-2].Rewind();\n }\n if (Quant::kTrain) {\n util::ErsatzProgress progress(std::accumulate(counts.begin() + 1, counts.end(), 0),\n config.ProgressMessages(), \"Quantizing\");\n for (unsigned char i = 2; i < counts.size(); ++i) {\n TrainQuantizer(i, counts[i-1], sri.Values(i), inputs[i-2], progress, quant);\n }\n TrainProbQuantizer(counts.size(), counts.back(), inputs[counts.size() - 2], progress, quant);\n quant.FinishedLoading(config);\n }\n\n UnigramValue *unigrams = out.unigram_.Raw();\n PopulateUnigramWeights(unigram_file.get(), counts[0], contexts[0], unigrams);\n unigram_file.reset();\n\n for (unsigned char i = 2; i <= counts.size(); ++i) {\n inputs[i-2].Rewind();\n }\n // Fill entries except unigram probabilities.\n {\n WriteEntries writer(contexts, quant, unigrams, out.middle_begin_, out.longest_, counts.size(), sri);\n RecursiveInsert(counts.size(), counts[0], inputs, config.ProgressMessages(), \"Writing trie\", writer);\n }\n\n // Do not disable this error message or else too little state will be returned. Both WriteEntries::Middle and returning state based on found n-grams will need to be fixed to handle this situation.\n for (unsigned char order = 2; order <= counts.size(); ++order) {\n const RecordReader &context = contexts[order - 2];\n if (context) {\n FormatLoadException e;\n e << \"A \" << static_cast(order) << \"-gram has context\";\n const WordIndex *ctx = reinterpret_cast(context.Data());\n for (const WordIndex *i = ctx; i != ctx + order - 1; ++i) {\n e << ' ' << *i;\n }\n e << \" so this context must appear in the model as a \" << static_cast(order - 1) << \"-gram but it does not\";\n throw e;\n }\n }\n\n /* Set ending offsets so the last entry will be sized properly */\n // Last entry for unigrams was already set.\n if (out.middle_begin_ != out.middle_end_) {\n for (typename TrieSearch::Middle *i = out.middle_begin_; i != out.middle_end_ - 1; ++i) {\n i->FinishedLoading((i+1)->InsertIndex(), config);\n }\n (out.middle_end_ - 1)->FinishedLoading(out.longest_.InsertIndex(), config);\n }\n}\n\ntemplate uint8_t *TrieSearch::SetupMemory(uint8_t *start, const std::vector &counts, const Config &config) {\n quant_.SetupMemory(start, counts.size(), config);\n start += Quant::Size(counts.size(), config);\n unigram_.Init(start);\n start += Unigram::Size(counts[0]);\n FreeMiddles();\n middle_begin_ = static_cast(malloc(sizeof(Middle) * (counts.size() - 2)));\n middle_end_ = middle_begin_ + (counts.size() - 2);\n std::vector middle_starts(counts.size() - 2);\n for (unsigned char i = 2; i < counts.size(); ++i) {\n middle_starts[i-2] = start;\n start += Middle::Size(Quant::MiddleBits(config), counts[i-1], counts[0], counts[i], config);\n }\n // Crazy backwards thing so we initialize using pointers to ones that have already been initialized\n for (unsigned char i = counts.size() - 1; i >= 2; --i) {\n new (middle_begin_ + i - 2) Middle(\n middle_starts[i-2],\n quant_.MiddleBits(config),\n counts[i-1],\n counts[0],\n counts[i],\n (i == counts.size() - 1) ? static_cast(longest_) : static_cast(middle_begin_[i-1]),\n config);\n }\n longest_.Init(start, quant_.LongestBits(config), counts[0]);\n return start + Longest::Size(Quant::LongestBits(config), counts.back(), counts[0]);\n}\n\ntemplate void TrieSearch::LoadedBinary() {\n unigram_.LoadedBinary();\n for (Middle *i = middle_begin_; i != middle_end_; ++i) {\n i->LoadedBinary();\n }\n longest_.LoadedBinary();\n}\n\ntemplate void TrieSearch::InitializeFromARPA(const char *file, util::FilePiece &f, std::vector &counts, const Config &config, SortedVocabulary &vocab, Backing &backing) {\n std::string temporary_prefix;\n if (config.temporary_directory_prefix) {\n temporary_prefix = config.temporary_directory_prefix;\n } else if (config.write_mmap) {\n temporary_prefix = config.write_mmap;\n } else {\n temporary_prefix = file;\n }\n // At least 1MB sorting memory.\n SortedFiles sorted(config, f, counts, std::max(config.building_memory, 1048576), temporary_prefix, vocab);\n\n BuildTrie(sorted, counts, config, *this, quant_, vocab, backing);\n}\n\ntemplate class TrieSearch;\ntemplate class TrieSearch;\ntemplate class TrieSearch;\ntemplate class TrieSearch;\n\n} // namespace trie\n} // namespace ngram\n} // namespace lm\n" }, { "path": "src/kenlm/lm/search_trie.hh", "content": "#ifndef LM_SEARCH_TRIE__\n#define LM_SEARCH_TRIE__\n\n#include \"lm/config.hh\"\n#include \"lm/model_type.hh\"\n#include \"lm/return.hh\"\n#include \"lm/trie.hh\"\n#include \"lm/weights.hh\"\n\n#include \"util/file.hh\"\n#include \"util/file_piece.hh\"\n\n#include \n\n#include \n\nnamespace lm {\nnamespace ngram {\nstruct Backing;\nclass SortedVocabulary;\nnamespace trie {\n\ntemplate class TrieSearch;\nclass SortedFiles;\ntemplate void BuildTrie(SortedFiles &files, std::vector &counts, const Config &config, TrieSearch &out, Quant &quant, const SortedVocabulary &vocab, Backing &backing);\n\ntemplate class TrieSearch {\n public:\n typedef NodeRange Node;\n\n typedef ::lm::ngram::trie::UnigramPointer UnigramPointer;\n typedef typename Quant::MiddlePointer MiddlePointer;\n typedef typename Quant::LongestPointer LongestPointer;\n\n static const bool kDifferentRest = false;\n\n static const ModelType kModelType = static_cast(TRIE_SORTED + Quant::kModelTypeAdd + Bhiksha::kModelTypeAdd);\n\n static const unsigned int kVersion = 1;\n\n static void UpdateConfigFromBinary(int fd, const std::vector &counts, Config &config) {\n Quant::UpdateConfigFromBinary(fd, counts, config);\n util::AdvanceOrThrow(fd, Quant::Size(counts.size(), config) + Unigram::Size(counts[0]));\n Bhiksha::UpdateConfigFromBinary(fd, config);\n }\n\n static uint64_t Size(const std::vector &counts, const Config &config) {\n uint64_t ret = Quant::Size(counts.size(), config) + Unigram::Size(counts[0]);\n for (unsigned char i = 1; i < counts.size() - 1; ++i) {\n ret += Middle::Size(Quant::MiddleBits(config), counts[i], counts[0], counts[i+1], config);\n }\n return ret + Longest::Size(Quant::LongestBits(config), counts.back(), counts[0]);\n }\n\n TrieSearch() : middle_begin_(NULL), middle_end_(NULL) {}\n\n ~TrieSearch() { FreeMiddles(); }\n\n uint8_t *SetupMemory(uint8_t *start, const std::vector &counts, const Config &config);\n\n void LoadedBinary();\n\n void InitializeFromARPA(const char *file, util::FilePiece &f, std::vector &counts, const Config &config, SortedVocabulary &vocab, Backing &backing);\n\n unsigned char Order() const {\n return middle_end_ - middle_begin_ + 2;\n }\n\n ProbBackoff &UnknownUnigram() { return unigram_.Unknown(); }\n\n UnigramPointer LookupUnigram(WordIndex word, Node &next, bool &independent_left, uint64_t &extend_left) const {\n extend_left = static_cast(word);\n UnigramPointer ret(unigram_.Find(word, next));\n independent_left = (next.begin == next.end);\n return ret;\n }\n\n MiddlePointer Unpack(uint64_t extend_pointer, unsigned char extend_length, Node &node) const {\n return MiddlePointer(quant_, extend_length - 2, middle_begin_[extend_length - 2].ReadEntry(extend_pointer, node));\n }\n\n MiddlePointer LookupMiddle(unsigned char order_minus_2, WordIndex word, Node &node, bool &independent_left, uint64_t &extend_left) const {\n util::BitAddress address(middle_begin_[order_minus_2].Find(word, node, extend_left));\n independent_left = (address.base == NULL) || (node.begin == node.end);\n return MiddlePointer(quant_, order_minus_2, address);\n }\n\n LongestPointer LookupLongest(WordIndex word, const Node &node) const {\n return LongestPointer(quant_, longest_.Find(word, node));\n }\n\n bool FastMakeNode(const WordIndex *begin, const WordIndex *end, Node &node) const {\n assert(begin != end);\n bool independent_left;\n uint64_t ignored;\n LookupUnigram(*begin, node, independent_left, ignored);\n for (const WordIndex *i = begin + 1; i < end; ++i) {\n if (independent_left || !LookupMiddle(i - begin - 1, *i, node, independent_left, ignored).Found()) return false;\n }\n return true;\n }\n\n private:\n friend void BuildTrie(SortedFiles &files, std::vector &counts, const Config &config, TrieSearch &out, Quant &quant, const SortedVocabulary &vocab, Backing &backing);\n\n // Middles are managed manually so we can delay construction and they don't have to be copyable. \n void FreeMiddles() {\n for (const Middle *i = middle_begin_; i != middle_end_; ++i) {\n i->~Middle();\n }\n free(middle_begin_);\n }\n\n typedef trie::BitPackedMiddle Middle;\n\n typedef trie::BitPackedLongest Longest;\n Longest longest_;\n\n Middle *middle_begin_, *middle_end_;\n Quant quant_;\n\n typedef ::lm::ngram::trie::Unigram Unigram;\n Unigram unigram_;\n};\n\n} // namespace trie\n} // namespace ngram\n} // namespace lm\n\n#endif // LM_SEARCH_TRIE__\n" }, { "path": "src/kenlm/lm/sizes.cc", "content": "#include \"lm/sizes.hh\"\n#include \"lm/model.hh\"\n#include \"util/file_piece.hh\"\n\n#include \n#include \n\nnamespace lm {\nnamespace ngram {\n\nvoid ShowSizes(const std::vector &counts, const lm::ngram::Config &config) {\n uint64_t sizes[6];\n sizes[0] = ProbingModel::Size(counts, config);\n sizes[1] = RestProbingModel::Size(counts, config);\n sizes[2] = TrieModel::Size(counts, config);\n sizes[3] = QuantTrieModel::Size(counts, config);\n sizes[4] = ArrayTrieModel::Size(counts, config);\n sizes[5] = QuantArrayTrieModel::Size(counts, config);\n uint64_t max_length = *std::max_element(sizes, sizes + sizeof(sizes) / sizeof(uint64_t));\n uint64_t min_length = *std::min_element(sizes, sizes + sizeof(sizes) / sizeof(uint64_t));\n uint64_t divide;\n char prefix;\n if (min_length < (1 << 10) * 10) {\n prefix = ' ';\n divide = 1;\n } else if (min_length < (1 << 20) * 10) {\n prefix = 'k';\n divide = 1 << 10;\n } else if (min_length < (1ULL << 30) * 10) {\n prefix = 'M';\n divide = 1 << 20;\n } else {\n prefix = 'G';\n divide = 1 << 30;\n }\n long int length = std::max(2, static_cast(ceil(log10((double) max_length / divide))));\n std::cerr << \"Memory estimate for binary LM:\\ntype \";\n\n // right align bytes. \n for (long int i = 0; i < length - 2; ++i) std::cerr << ' ';\n\n std::cerr << prefix << \"B\\n\"\n \"probing \" << std::setw(length) << (sizes[0] / divide) << \" assuming -p \" << config.probing_multiplier << \"\\n\"\n \"probing \" << std::setw(length) << (sizes[1] / divide) << \" assuming -r models -p \" << config.probing_multiplier << \"\\n\"\n \"trie \" << std::setw(length) << (sizes[2] / divide) << \" without quantization\\n\"\n \"trie \" << std::setw(length) << (sizes[3] / divide) << \" assuming -q \" << (unsigned)config.prob_bits << \" -b \" << (unsigned)config.backoff_bits << \" quantization \\n\"\n \"trie \" << std::setw(length) << (sizes[4] / divide) << \" assuming -a \" << (unsigned)config.pointer_bhiksha_bits << \" array pointer compression\\n\"\n \"trie \" << std::setw(length) << (sizes[5] / divide) << \" assuming -a \" << (unsigned)config.pointer_bhiksha_bits << \" -q \" << (unsigned)config.prob_bits << \" -b \" << (unsigned)config.backoff_bits<< \" array pointer compression and quantization\\n\";\n}\n\nvoid ShowSizes(const std::vector &counts) {\n lm::ngram::Config config;\n ShowSizes(counts, config);\n}\n\nvoid ShowSizes(const char *file, const lm::ngram::Config &config) {\n std::vector counts;\n util::FilePiece f(file);\n lm::ReadARPACounts(f, counts);\n ShowSizes(counts, config);\n}\n\n}} //namespaces\n" }, { "path": "src/kenlm/lm/sizes.hh", "content": "#ifndef LM_SIZES__\n#define LM_SIZES__\n\n#include \n\n#include \n\nnamespace lm { namespace ngram {\n\nstruct Config;\n\nvoid ShowSizes(const std::vector &counts, const lm::ngram::Config &config);\nvoid ShowSizes(const std::vector &counts);\nvoid ShowSizes(const char *file, const lm::ngram::Config &config);\n\n}} // namespaces\n#endif // LM_SIZES__\n" }, { "path": "src/kenlm/lm/state.hh", "content": "#ifndef LM_STATE__\n#define LM_STATE__\n\n#include \"lm/max_order.hh\"\n#include \"lm/word_index.hh\"\n#include \"util/murmur_hash.hh\"\n\n#include \n\nnamespace lm {\nnamespace ngram {\n\n// This is a POD but if you want memcmp to return the same as operator==, call\n// ZeroRemaining first. \nclass State {\n public:\n bool operator==(const State &other) const {\n if (length != other.length) return false;\n return !memcmp(words, other.words, length * sizeof(WordIndex));\n }\n\n // Three way comparison function. \n int Compare(const State &other) const {\n if (length != other.length) return length < other.length ? -1 : 1;\n return memcmp(words, other.words, length * sizeof(WordIndex));\n }\n\n bool operator<(const State &other) const {\n if (length != other.length) return length < other.length;\n return memcmp(words, other.words, length * sizeof(WordIndex)) < 0;\n }\n\n // Call this before using raw memcmp. \n void ZeroRemaining() {\n for (unsigned char i = length; i < KENLM_MAX_ORDER - 1; ++i) {\n words[i] = 0;\n backoff[i] = 0.0;\n }\n }\n\n unsigned char Length() const { return length; }\n\n // You shouldn't need to touch anything below this line, but the members are public so FullState will qualify as a POD. \n // This order minimizes total size of the struct if WordIndex is 64 bit, float is 32 bit, and alignment of 64 bit integers is 64 bit. \n WordIndex words[KENLM_MAX_ORDER - 1];\n float backoff[KENLM_MAX_ORDER - 1];\n unsigned char length;\n};\n\ntypedef State Right;\n\ninline uint64_t hash_value(const State &state, uint64_t seed = 0) {\n return util::MurmurHashNative(state.words, sizeof(WordIndex) * state.length, seed);\n}\n\nstruct Left {\n bool operator==(const Left &other) const {\n return \n length == other.length &&\n (!length || (pointers[length - 1] == other.pointers[length - 1] && full == other.full));\n }\n\n int Compare(const Left &other) const {\n if (length < other.length) return -1;\n if (length > other.length) return 1;\n if (length == 0) return 0; // Must be full.\n if (pointers[length - 1] > other.pointers[length - 1]) return 1;\n if (pointers[length - 1] < other.pointers[length - 1]) return -1;\n return (int)full - (int)other.full;\n }\n\n bool operator<(const Left &other) const {\n return Compare(other) == -1;\n }\n\n void ZeroRemaining() {\n for (uint64_t * i = pointers + length; i < pointers + KENLM_MAX_ORDER - 1; ++i)\n *i = 0;\n }\n\n uint64_t pointers[KENLM_MAX_ORDER - 1];\n unsigned char length;\n bool full;\n};\n\ninline uint64_t hash_value(const Left &left) {\n unsigned char add[2];\n add[0] = left.length;\n add[1] = left.full;\n return util::MurmurHashNative(add, 2, left.length ? left.pointers[left.length - 1] : 0);\n}\n\nstruct ChartState {\n bool operator==(const ChartState &other) {\n return (right == other.right) && (left == other.left);\n }\n\n int Compare(const ChartState &other) const {\n int lres = left.Compare(other.left);\n if (lres) return lres;\n return right.Compare(other.right);\n }\n\n bool operator<(const ChartState &other) const {\n return Compare(other) == -1;\n }\n\n void ZeroRemaining() {\n left.ZeroRemaining();\n right.ZeroRemaining();\n }\n\n Left left;\n State right;\n};\n\ninline uint64_t hash_value(const ChartState &state) {\n return hash_value(state.right, hash_value(state.left));\n}\n\n\n} // namespace ngram\n} // namespace lm\n\n#endif // LM_STATE__\n" }, { "path": "src/kenlm/lm/test.arpa", "content": "\n\\data\\\nngram 1=37\nngram 2=47\nngram 3=11\nngram 4=6\nngram 5=4\n\n\\1-grams:\n-1.383514\t,\t-0.30103\n-1.139057\t.\t-0.845098\n-1.029493\t\n-99\t\t-0.4149733\n-1.995635\t\t-20\n-1.285941\ta\t-0.69897\n-1.687872\talso\t-0.30103\n-1.687872\tbeyond\t-0.30103\n-1.687872\tbiarritz\t-0.30103\n-1.687872\tcall\t-0.30103\n-1.687872\tconcerns\t-0.30103\n-1.687872\tconsider\t-0.30103\n-1.687872\tconsidering\t-0.30103\n-1.687872\tfor\t-0.30103\n-1.509559\thigher\t-0.30103\n-1.687872\thowever\t-0.30103\n-1.687872\ti\t-0.30103\n-1.687872\timmediate\t-0.30103\n-1.687872\tin\t-0.30103\n-1.687872\tis\t-0.30103\n-1.285941\tlittle\t-0.69897\n-1.383514\tloin\t-0.30103\n-1.687872\tlook\t-0.30103\n-1.285941\tlooking\t-0.4771212\n-1.206319\tmore\t-0.544068\n-1.509559\ton\t-0.4771212\n-1.509559\tscreening\t-0.4771212\n-1.687872\tsmall\t-0.30103\n-1.687872\tthe\t-0.30103\n-1.687872\tto\t-0.30103\n-1.687872\twatch\t-0.30103\n-1.687872\twatching\t-0.30103\n-1.687872\twhat\t-0.30103\n-1.687872\twould\t-0.30103\n-3.141592\tfoo\n-2.718281\tbar\t3.0\n-6.535897\tbaz\t-0.0\n\n\\2-grams:\n-0.6925742\t, .\n-0.7522095\t, however\n-0.7522095\t, is\n-0.0602359\t. \n-0.4846522\t looking\t-0.4771214\n-1.051485\t screening\n-1.07153\t the\n-1.07153\t watching\n-1.07153\t what\n-0.09132547\ta little\t-0.69897\n-0.2922095\talso call\n-0.2922095\tbeyond immediate\n-0.2705918\tbiarritz .\n-0.2922095\tcall for\n-0.2922095\tconcerns in\n-0.2922095\tconsider watch\n-0.2922095\tconsidering consider\n-0.2834328\tfor ,\n-0.5511513\thigher more\n-0.5845945\thigher small\n-0.2834328\thowever ,\n-0.2922095\ti would\n-0.2922095\timmediate concerns\n-0.2922095\tin biarritz\n-0.2922095\tis to\n-0.09021038\tlittle more\t-0.1998621\n-0.7273645\tloin ,\n-0.6925742\tloin .\n-0.6708385\tloin \n-0.2922095\tlook beyond\n-0.4638903\tlooking higher\n-0.4638903\tlooking on\t-0.4771212\n-0.5136299\tmore .\t-0.4771212\n-0.3561665\tmore loin\n-0.1649931\ton a\t-0.4771213\n-0.1649931\tscreening a\t-0.4771213\n-0.2705918\tsmall .\n-0.287799\tthe screening\n-0.2922095\tto look\n-0.2622373\twatch \n-0.2922095\twatching considering\n-0.2922095\twhat i\n-0.2922095\twould also\n-2\talso would\t-6\n-15\t \t-2\n-4\t however\t-1\n-6\tfoo bar\n\n\\3-grams:\n-0.01916512\tmore . \n-0.0283603\ton a little\t-0.4771212\n-0.0283603\tscreening a little\t-0.4771212\n-0.01660496\ta little more\t-0.09409451\n-0.3488368\t looking higher\n-0.3488368\t looking on\t-0.4771212\n-0.1892331\tlittle more loin\n-0.04835128\tlooking on a\t-0.4771212\n-3\talso would consider\t-7\n-6\t however \t-12\n-7\tto look good\n\n\\4-grams:\n-0.009249173\tlooking on a little\t-0.4771212\n-0.005464747\ton a little more\t-0.4771212\n-0.005464747\tscreening a little more\n-0.1453306\ta little more loin\n-0.01552657\t looking on a\t-0.4771212\n-4\talso would consider higher\t-8\n\n\\5-grams:\n-0.003061223\t looking on a little\n-0.001813953\tlooking on a little more\n-0.0432557\ton a little more loin\n-5\talso would consider higher looking\n\n\\end\\\n" }, { "path": "src/kenlm/lm/test_nounk.arpa", "content": "\n\\data\\\nngram 1=36\nngram 2=45\nngram 3=10\nngram 4=6\nngram 5=4\n\n\\1-grams:\n-1.383514\t,\t-0.30103\n-1.139057\t.\t-0.845098\n-1.029493\t\n-99\t\t-0.4149733\n-1.285941\ta\t-0.69897\n-1.687872\talso\t-0.30103\n-1.687872\tbeyond\t-0.30103\n-1.687872\tbiarritz\t-0.30103\n-1.687872\tcall\t-0.30103\n-1.687872\tconcerns\t-0.30103\n-1.687872\tconsider\t-0.30103\n-1.687872\tconsidering\t-0.30103\n-1.687872\tfor\t-0.30103\n-1.509559\thigher\t-0.30103\n-1.687872\thowever\t-0.30103\n-1.687872\ti\t-0.30103\n-1.687872\timmediate\t-0.30103\n-1.687872\tin\t-0.30103\n-1.687872\tis\t-0.30103\n-1.285941\tlittle\t-0.69897\n-1.383514\tloin\t-0.30103\n-1.687872\tlook\t-0.30103\n-1.285941\tlooking\t-0.4771212\n-1.206319\tmore\t-0.544068\n-1.509559\ton\t-0.4771212\n-1.509559\tscreening\t-0.4771212\n-1.687872\tsmall\t-0.30103\n-1.687872\tthe\t-0.30103\n-1.687872\tto\t-0.30103\n-1.687872\twatch\t-0.30103\n-1.687872\twatching\t-0.30103\n-1.687872\twhat\t-0.30103\n-1.687872\twould\t-0.30103\n-3.141592\tfoo\n-2.718281\tbar\t3.0\n-6.535897\tbaz\t-0.0\n\n\\2-grams:\n-0.6925742\t, .\n-0.7522095\t, however\n-0.7522095\t, is\n-0.0602359\t. \n-0.4846522\t looking\t-0.4771214\n-1.051485\t screening\n-1.07153\t the\n-1.07153\t watching\n-1.07153\t what\n-0.09132547\ta little\t-0.69897\n-0.2922095\talso call\n-0.2922095\tbeyond immediate\n-0.2705918\tbiarritz .\n-0.2922095\tcall for\n-0.2922095\tconcerns in\n-0.2922095\tconsider watch\n-0.2922095\tconsidering consider\n-0.2834328\tfor ,\n-0.5511513\thigher more\n-0.5845945\thigher small\n-0.2834328\thowever ,\n-0.2922095\ti would\n-0.2922095\timmediate concerns\n-0.2922095\tin biarritz\n-0.2922095\tis to\n-0.09021038\tlittle more\t-0.1998621\n-0.7273645\tloin ,\n-0.6925742\tloin .\n-0.6708385\tloin \n-0.2922095\tlook beyond\n-0.4638903\tlooking higher\n-0.4638903\tlooking on\t-0.4771212\n-0.5136299\tmore .\t-0.4771212\n-0.3561665\tmore loin\n-0.1649931\ton a\t-0.4771213\n-0.1649931\tscreening a\t-0.4771213\n-0.2705918\tsmall .\n-0.287799\tthe screening\n-0.2922095\tto look\n-0.2622373\twatch \n-0.2922095\twatching considering\n-0.2922095\twhat i\n-0.2922095\twould also\n-2\talso would\t-6\n-6\tfoo bar\n\n\\3-grams:\n-0.01916512\tmore . \n-0.0283603\ton a little\t-0.4771212\n-0.0283603\tscreening a little\t-0.4771212\n-0.01660496\ta little more\t-0.09409451\n-0.3488368\t looking higher\n-0.3488368\t looking on\t-0.4771212\n-0.1892331\tlittle more loin\n-0.04835128\tlooking on a\t-0.4771212\n-3\talso would consider\t-7\n-7\tto look good\n\n\\4-grams:\n-0.009249173\tlooking on a little\t-0.4771212\n-0.005464747\ton a little more\t-0.4771212\n-0.005464747\tscreening a little more\n-0.1453306\ta little more loin\n-0.01552657\t looking on a\t-0.4771212\n-4\talso would consider higher\t-8\n\n\\5-grams:\n-0.003061223\t looking on a little\n-0.001813953\tlooking on a little more\n-0.0432557\ton a little more loin\n-5\talso would consider higher looking\n\n\\end\\\n" }, { "path": "src/kenlm/lm/trie.cc", "content": "#include \"lm/trie.hh\"\n\n#include \"lm/bhiksha.hh\"\n#include \"util/bit_packing.hh\"\n#include \"util/exception.hh\"\n#include \"util/sorted_uniform.hh\"\n\n#include \n\nnamespace lm {\nnamespace ngram {\nnamespace trie {\nnamespace {\n\nclass KeyAccessor {\n public:\n KeyAccessor(const void *base, uint64_t key_mask, uint8_t key_bits, uint8_t total_bits) \n : base_(reinterpret_cast(base)), key_mask_(key_mask), key_bits_(key_bits), total_bits_(total_bits) {}\n\n typedef uint64_t Key;\n\n Key operator()(uint64_t index) const {\n return util::ReadInt57(base_, index * static_cast(total_bits_), key_bits_, key_mask_);\n }\n\n private:\n const uint8_t *const base_;\n const WordIndex key_mask_;\n const uint8_t key_bits_, total_bits_;\n};\n\nbool FindBitPacked(const void *base, uint64_t key_mask, uint8_t key_bits, uint8_t total_bits, uint64_t begin_index, uint64_t end_index, const uint64_t max_vocab, const uint64_t key, uint64_t &at_index) {\n KeyAccessor accessor(base, key_mask, key_bits, total_bits);\n if (!util::BoundedSortedUniformFind::T>(accessor, begin_index - 1, (uint64_t)0, end_index, max_vocab, key, at_index)) return false;\n return true;\n}\n} // namespace\n\nuint64_t BitPacked::BaseSize(uint64_t entries, uint64_t max_vocab, uint8_t remaining_bits) {\n uint8_t total_bits = util::RequiredBits(max_vocab) + remaining_bits;\n // Extra entry for next pointer at the end. \n // +7 then / 8 to round up bits and convert to bytes\n // +sizeof(uint64_t) so that ReadInt57 etc don't go segfault. \n // Note that this waste is O(order), not O(number of ngrams).\n return ((1 + entries) * total_bits + 7) / 8 + sizeof(uint64_t);\n}\n\nvoid BitPacked::BaseInit(void *base, uint64_t max_vocab, uint8_t remaining_bits) {\n util::BitPackingSanity();\n word_bits_ = util::RequiredBits(max_vocab);\n word_mask_ = (1ULL << word_bits_) - 1ULL;\n if (word_bits_ > 57) UTIL_THROW(util::Exception, \"Sorry, word indices more than \" << (1ULL << 57) << \" are not implemented. Edit util/bit_packing.hh and fix the bit packing functions.\");\n total_bits_ = word_bits_ + remaining_bits;\n\n base_ = static_cast(base);\n insert_index_ = 0;\n max_vocab_ = max_vocab;\n}\n\ntemplate uint64_t BitPackedMiddle::Size(uint8_t quant_bits, uint64_t entries, uint64_t max_vocab, uint64_t max_ptr, const Config &config) {\n return Bhiksha::Size(entries + 1, max_ptr, config) + BaseSize(entries, max_vocab, quant_bits + Bhiksha::InlineBits(entries + 1, max_ptr, config));\n}\n\ntemplate BitPackedMiddle::BitPackedMiddle(void *base, uint8_t quant_bits, uint64_t entries, uint64_t max_vocab, uint64_t max_next, const BitPacked &next_source, const Config &config) :\n BitPacked(),\n quant_bits_(quant_bits),\n // If the offset of the method changes, also change TrieSearch::UpdateConfigFromBinary.\n bhiksha_(base, entries + 1, max_next, config),\n next_source_(&next_source) {\n if (entries + 1 >= (1ULL << 57) || (max_next >= (1ULL << 57))) UTIL_THROW(util::Exception, \"Sorry, this does not support more than \" << (1ULL << 57) << \" n-grams of a particular order. Edit util/bit_packing.hh and fix the bit packing functions.\");\n BaseInit(reinterpret_cast(base) + Bhiksha::Size(entries + 1, max_next, config), max_vocab, quant_bits_ + bhiksha_.InlineBits());\n}\n\ntemplate util::BitAddress BitPackedMiddle::Insert(WordIndex word) {\n assert(word <= word_mask_);\n uint64_t at_pointer = insert_index_ * total_bits_;\n\n util::WriteInt57(base_, at_pointer, word_bits_, word);\n at_pointer += word_bits_;\n util::BitAddress ret(base_, at_pointer);\n at_pointer += quant_bits_;\n uint64_t next = next_source_->InsertIndex();\n bhiksha_.WriteNext(base_, at_pointer, insert_index_, next);\n ++insert_index_;\n return ret;\n}\n\ntemplate util::BitAddress BitPackedMiddle::Find(WordIndex word, NodeRange &range, uint64_t &pointer) const {\n uint64_t at_pointer;\n if (!FindBitPacked(base_, word_mask_, word_bits_, total_bits_, range.begin, range.end, max_vocab_, word, at_pointer)) {\n return util::BitAddress(NULL, 0);\n }\n pointer = at_pointer;\n at_pointer *= total_bits_;\n at_pointer += word_bits_;\n bhiksha_.ReadNext(base_, at_pointer + quant_bits_, pointer, total_bits_, range);\n\n return util::BitAddress(base_, at_pointer);\n}\n\ntemplate void BitPackedMiddle::FinishedLoading(uint64_t next_end, const Config &config) {\n uint64_t last_next_write = (insert_index_ + 1) * total_bits_ - bhiksha_.InlineBits();\n bhiksha_.WriteNext(base_, last_next_write, insert_index_ + 1, next_end);\n bhiksha_.FinishedLoading(config);\n}\n\nutil::BitAddress BitPackedLongest::Insert(WordIndex index) {\n assert(index <= word_mask_);\n uint64_t at_pointer = insert_index_ * total_bits_;\n util::WriteInt57(base_, at_pointer, word_bits_, index);\n at_pointer += word_bits_;\n ++insert_index_;\n return util::BitAddress(base_, at_pointer);\n}\n\nutil::BitAddress BitPackedLongest::Find(WordIndex word, const NodeRange &range) const {\n uint64_t at_pointer;\n if (!FindBitPacked(base_, word_mask_, word_bits_, total_bits_, range.begin, range.end, max_vocab_, word, at_pointer)) return util::BitAddress(NULL, 0);\n at_pointer = at_pointer * total_bits_ + word_bits_;\n return util::BitAddress(base_, at_pointer);\n}\n\ntemplate class BitPackedMiddle;\ntemplate class BitPackedMiddle;\n\n} // namespace trie\n} // namespace ngram\n} // namespace lm\n" }, { "path": "src/kenlm/lm/trie.hh", "content": "#ifndef LM_TRIE__\n#define LM_TRIE__\n\n#include \"lm/weights.hh\"\n#include \"lm/word_index.hh\"\n#include \"util/bit_packing.hh\"\n\n#include \n\n#include \n\nnamespace lm {\nnamespace ngram {\nstruct Config;\nnamespace trie {\n\nstruct NodeRange {\n uint64_t begin, end;\n};\n\n// TODO: if the number of unigrams is a concern, also bit pack these records. \nstruct UnigramValue {\n ProbBackoff weights;\n uint64_t next;\n uint64_t Next() const { return next; }\n};\n\nclass UnigramPointer {\n public:\n explicit UnigramPointer(const ProbBackoff &to) : to_(&to) {}\n\n UnigramPointer() : to_(NULL) {}\n\n bool Found() const { return to_ != NULL; }\n\n float Prob() const { return to_->prob; }\n float Backoff() const { return to_->backoff; }\n float Rest() const { return Prob(); }\n\n private:\n const ProbBackoff *to_;\n};\n\nclass Unigram {\n public:\n Unigram() {}\n \n void Init(void *start) {\n unigram_ = static_cast(start);\n }\n \n static uint64_t Size(uint64_t count) {\n // +1 in case unknown doesn't appear. +1 for the final next. \n return (count + 2) * sizeof(UnigramValue);\n }\n \n const ProbBackoff &Lookup(WordIndex index) const { return unigram_[index].weights; }\n \n ProbBackoff &Unknown() { return unigram_[0].weights; }\n\n UnigramValue *Raw() {\n return unigram_;\n }\n \n void LoadedBinary() {}\n\n UnigramPointer Find(WordIndex word, NodeRange &next) const {\n UnigramValue *val = unigram_ + word;\n next.begin = val->next;\n next.end = (val+1)->next;\n return UnigramPointer(val->weights);\n }\n\n private:\n UnigramValue *unigram_;\n}; \n\nclass BitPacked {\n public:\n BitPacked() {}\n\n uint64_t InsertIndex() const {\n return insert_index_;\n }\n\n protected:\n static uint64_t BaseSize(uint64_t entries, uint64_t max_vocab, uint8_t remaining_bits);\n\n void BaseInit(void *base, uint64_t max_vocab, uint8_t remaining_bits);\n\n uint8_t word_bits_;\n uint8_t total_bits_;\n uint64_t word_mask_;\n\n uint8_t *base_;\n\n uint64_t insert_index_, max_vocab_;\n};\n\ntemplate class BitPackedMiddle : public BitPacked {\n public:\n static uint64_t Size(uint8_t quant_bits, uint64_t entries, uint64_t max_vocab, uint64_t max_next, const Config &config);\n\n // next_source need not be initialized. \n BitPackedMiddle(void *base, uint8_t quant_bits, uint64_t entries, uint64_t max_vocab, uint64_t max_next, const BitPacked &next_source, const Config &config);\n\n util::BitAddress Insert(WordIndex word);\n\n void FinishedLoading(uint64_t next_end, const Config &config);\n\n void LoadedBinary() { bhiksha_.LoadedBinary(); }\n\n util::BitAddress Find(WordIndex word, NodeRange &range, uint64_t &pointer) const;\n\n util::BitAddress ReadEntry(uint64_t pointer, NodeRange &range) {\n uint64_t addr = pointer * total_bits_;\n addr += word_bits_;\n bhiksha_.ReadNext(base_, addr + quant_bits_, pointer, total_bits_, range);\n return util::BitAddress(base_, addr);\n }\n\n private:\n uint8_t quant_bits_;\n Bhiksha bhiksha_;\n\n const BitPacked *next_source_;\n};\n\nclass BitPackedLongest : public BitPacked {\n public:\n static uint64_t Size(uint8_t quant_bits, uint64_t entries, uint64_t max_vocab) {\n return BaseSize(entries, max_vocab, quant_bits);\n }\n\n BitPackedLongest() {}\n\n void Init(void *base, uint8_t quant_bits, uint64_t max_vocab) {\n BaseInit(base, max_vocab, quant_bits);\n }\n\n void LoadedBinary() {}\n\n util::BitAddress Insert(WordIndex word);\n\n util::BitAddress Find(WordIndex word, const NodeRange &node) const;\n\n private:\n uint8_t quant_bits_;\n};\n\n} // namespace trie\n} // namespace ngram\n} // namespace lm\n\n#endif // LM_TRIE__\n" }, { "path": "src/kenlm/lm/trie_sort.cc", "content": "#include \"lm/trie_sort.hh\"\n\n#include \"lm/config.hh\"\n#include \"lm/lm_exception.hh\"\n#include \"lm/read_arpa.hh\"\n#include \"lm/vocab.hh\"\n#include \"lm/weights.hh\"\n#include \"lm/word_index.hh\"\n#include \"util/file_piece.hh\"\n#include \"util/mmap.hh\"\n#include \"util/proxy_iterator.hh\"\n#include \"util/sized_iterator.hh\"\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\nnamespace lm {\nnamespace ngram {\nnamespace trie {\nnamespace {\n\ntypedef util::SizedIterator NGramIter;\n\n// Proxy for an entry except there is some extra cruft between the entries. This is used to sort (n-1)-grams using the same memory as the sorted n-grams. \nclass PartialViewProxy {\n public:\n PartialViewProxy() : attention_size_(0), inner_() {}\n\n PartialViewProxy(void *ptr, std::size_t block_size, std::size_t attention_size) : attention_size_(attention_size), inner_(ptr, block_size) {}\n\n operator std::string() const {\n return std::string(reinterpret_cast(inner_.Data()), attention_size_);\n }\n\n PartialViewProxy &operator=(const PartialViewProxy &from) {\n memcpy(inner_.Data(), from.inner_.Data(), attention_size_);\n return *this;\n }\n\n PartialViewProxy &operator=(const std::string &from) {\n memcpy(inner_.Data(), from.data(), attention_size_);\n return *this;\n }\n\n const void *Data() const { return inner_.Data(); }\n void *Data() { return inner_.Data(); }\n\n private:\n friend class util::ProxyIterator;\n\n typedef std::string value_type;\n\n const std::size_t attention_size_;\n\n typedef util::SizedInnerIterator InnerIterator;\n InnerIterator &Inner() { return inner_; }\n const InnerIterator &Inner() const { return inner_; } \n InnerIterator inner_;\n};\n\ntypedef util::ProxyIterator PartialIter;\n\nFILE *DiskFlush(const void *mem_begin, const void *mem_end, const std::string &temp_prefix) {\n util::scoped_fd file(util::MakeTemp(temp_prefix));\n util::WriteOrThrow(file.get(), mem_begin, (uint8_t*)mem_end - (uint8_t*)mem_begin);\n return util::FDOpenOrThrow(file);\n}\n\nFILE *WriteContextFile(uint8_t *begin, uint8_t *end, const std::string &temp_prefix, std::size_t entry_size, unsigned char order) {\n const size_t context_size = sizeof(WordIndex) * (order - 1);\n // Sort just the contexts using the same memory. \n PartialIter context_begin(PartialViewProxy(begin + sizeof(WordIndex), entry_size, context_size));\n PartialIter context_end(PartialViewProxy(end + sizeof(WordIndex), entry_size, context_size));\n\n#if defined(_WIN32) || defined(_WIN64)\n std::stable_sort\n#else\n std::sort\n#endif\n (context_begin, context_end, util::SizedCompare(EntryCompare(order - 1)));\n\n util::scoped_FILE out(util::FMakeTemp(temp_prefix));\n\n // Write out to file and uniqueify at the same time. Could have used unique_copy if there was an appropriate OutputIterator. \n if (context_begin == context_end) return out.release();\n PartialIter i(context_begin);\n util::WriteOrThrow(out.get(), i->Data(), context_size);\n const void *previous = i->Data();\n ++i;\n for (; i != context_end; ++i) {\n if (memcmp(previous, i->Data(), context_size)) {\n util::WriteOrThrow(out.get(), i->Data(), context_size);\n previous = i->Data();\n }\n }\n return out.release();\n}\n\nstruct ThrowCombine {\n void operator()(std::size_t /*entry_size*/, const void * /*first*/, const void * /*second*/, FILE * /*out*/) const {\n UTIL_THROW(FormatLoadException, \"Duplicate n-gram detected.\");\n }\n};\n\n// Useful for context files that just contain records with no value. \nstruct FirstCombine {\n void operator()(std::size_t entry_size, const void *first, const void * /*second*/, FILE *out) const {\n util::WriteOrThrow(out, first, entry_size);\n }\n};\n\ntemplate FILE *MergeSortedFiles(FILE *first_file, FILE *second_file, const std::string &temp_prefix, std::size_t weights_size, unsigned char order, const Combine &combine) {\n std::size_t entry_size = sizeof(WordIndex) * order + weights_size;\n RecordReader first, second;\n first.Init(first_file, entry_size);\n second.Init(second_file, entry_size);\n util::scoped_FILE out_file(util::FMakeTemp(temp_prefix));\n EntryCompare less(order);\n while (first && second) {\n if (less(first.Data(), second.Data())) {\n util::WriteOrThrow(out_file.get(), first.Data(), entry_size);\n ++first;\n } else if (less(second.Data(), first.Data())) {\n util::WriteOrThrow(out_file.get(), second.Data(), entry_size);\n ++second;\n } else {\n combine(entry_size, first.Data(), second.Data(), out_file.get());\n ++first; ++second;\n }\n }\n for (RecordReader &remains = (first ? first : second); remains; ++remains) {\n util::WriteOrThrow(out_file.get(), remains.Data(), entry_size);\n }\n return out_file.release();\n}\n\n} // namespace\n\nvoid RecordReader::Init(FILE *file, std::size_t entry_size) {\n entry_size_ = entry_size;\n data_.reset(malloc(entry_size));\n UTIL_THROW_IF(!data_.get(), util::ErrnoException, \"Failed to malloc read buffer\");\n file_ = file;\n if (file) {\n rewind(file);\n remains_ = true;\n ++*this;\n } else {\n remains_ = false;\n }\n}\n\nvoid RecordReader::Overwrite(const void *start, std::size_t amount) {\n long internal = (uint8_t*)start - (uint8_t*)data_.get();\n UTIL_THROW_IF(fseek(file_, internal - entry_size_, SEEK_CUR), util::ErrnoException, \"Couldn't seek backwards for revision\");\n util::WriteOrThrow(file_, start, amount);\n long forward = entry_size_ - internal - amount;\n#if !defined(_WIN32) && !defined(_WIN64)\n if (forward) \n#endif\n UTIL_THROW_IF(fseek(file_, forward, SEEK_CUR), util::ErrnoException, \"Couldn't seek forwards past revision\");\n}\n\nvoid RecordReader::Rewind() {\n if (file_) {\n rewind(file_);\n remains_ = true;\n ++*this;\n } else {\n remains_ = false;\n }\n}\n\nSortedFiles::SortedFiles(const Config &config, util::FilePiece &f, std::vector &counts, size_t buffer, const std::string &file_prefix, SortedVocabulary &vocab) {\n PositiveProbWarn warn(config.positive_log_probability);\n unigram_.reset(util::MakeTemp(file_prefix));\n {\n // In case appears. \n size_t size_out = (counts[0] + 1) * sizeof(ProbBackoff);\n util::scoped_mmap unigram_mmap(util::MapZeroedWrite(unigram_.get(), size_out), size_out);\n Read1Grams(f, counts[0], vocab, reinterpret_cast(unigram_mmap.get()), warn);\n CheckSpecials(config, vocab);\n if (!vocab.SawUnk()) ++counts[0];\n }\n\n // Only use as much buffer as we need. \n size_t buffer_use = 0;\n for (unsigned int order = 2; order < counts.size(); ++order) {\n buffer_use = std::max(buffer_use, static_cast((sizeof(WordIndex) * order + 2 * sizeof(float)) * counts[order - 1]));\n }\n buffer_use = std::max(buffer_use, static_cast((sizeof(WordIndex) * counts.size() + sizeof(float)) * counts.back()));\n buffer = std::min(buffer, buffer_use);\n\n util::scoped_malloc mem;\n mem.reset(malloc(buffer));\n if (!mem.get()) UTIL_THROW(util::ErrnoException, \"malloc failed for sort buffer size \" << buffer);\n\n for (unsigned char order = 2; order <= counts.size(); ++order) {\n ConvertToSorted(f, vocab, counts, file_prefix, order, warn, mem.get(), buffer);\n }\n ReadEnd(f);\n}\n\nnamespace {\nclass Closer {\n public:\n explicit Closer(std::deque &files) : files_(files) {}\n\n ~Closer() {\n for (std::deque::iterator i = files_.begin(); i != files_.end(); ++i) {\n util::scoped_FILE deleter(*i);\n }\n }\n\n void PopFront() {\n util::scoped_FILE deleter(files_.front());\n files_.pop_front();\n }\n private:\n std::deque &files_;\n};\n} // namespace\n\nvoid SortedFiles::ConvertToSorted(util::FilePiece &f, const SortedVocabulary &vocab, const std::vector &counts, const std::string &file_prefix, unsigned char order, PositiveProbWarn &warn, void *mem, std::size_t mem_size) {\n ReadNGramHeader(f, order);\n const size_t count = counts[order - 1];\n // Size of weights. Does it include backoff? \n const size_t words_size = sizeof(WordIndex) * order;\n const size_t weights_size = sizeof(float) + ((order == counts.size()) ? 0 : sizeof(float));\n const size_t entry_size = words_size + weights_size;\n const size_t batch_size = std::min(count, mem_size / entry_size);\n uint8_t *const begin = reinterpret_cast(mem);\n\n std::deque files, contexts;\n Closer files_closer(files), contexts_closer(contexts);\n\n for (std::size_t batch = 0, done = 0; done < count; ++batch) {\n uint8_t *out = begin;\n uint8_t *out_end = out + std::min(count - done, batch_size) * entry_size;\n if (order == counts.size()) {\n for (; out != out_end; out += entry_size) {\n ReadNGram(f, order, vocab, reinterpret_cast(out), *reinterpret_cast(out + words_size), warn);\n }\n } else {\n for (; out != out_end; out += entry_size) {\n ReadNGram(f, order, vocab, reinterpret_cast(out), *reinterpret_cast(out + words_size), warn);\n }\n }\n // Sort full records by full n-gram. \n util::SizedProxy proxy_begin(begin, entry_size), proxy_end(out_end, entry_size);\n // parallel_sort uses too much RAM. TODO: figure out why windows sort doesn't like my proxies. \n#if defined(_WIN32) || defined(_WIN64)\n std::stable_sort\n#else\n std::sort\n#endif\n (NGramIter(proxy_begin), NGramIter(proxy_end), util::SizedCompare(EntryCompare(order)));\n files.push_back(DiskFlush(begin, out_end, file_prefix));\n contexts.push_back(WriteContextFile(begin, out_end, file_prefix, entry_size, order));\n\n done += (out_end - begin) / entry_size;\n }\n\n // All individual files created. Merge them. \n\n while (files.size() > 1) {\n files.push_back(MergeSortedFiles(files[0], files[1], file_prefix, weights_size, order, ThrowCombine()));\n files_closer.PopFront();\n files_closer.PopFront();\n contexts.push_back(MergeSortedFiles(contexts[0], contexts[1], file_prefix, 0, order - 1, FirstCombine()));\n contexts_closer.PopFront();\n contexts_closer.PopFront();\n }\n\n if (!files.empty()) {\n // Steal from closers.\n full_[order - 2].reset(files.front());\n files.pop_front();\n context_[order - 2].reset(contexts.front());\n contexts.pop_front();\n }\n}\n\n} // namespace trie\n} // namespace ngram\n} // namespace lm\n" }, { "path": "src/kenlm/lm/trie_sort.hh", "content": "// Step of trie builder: create sorted files. \n\n#ifndef LM_TRIE_SORT__\n#define LM_TRIE_SORT__\n\n#include \"lm/max_order.hh\"\n#include \"lm/word_index.hh\"\n\n#include \"util/file.hh\"\n#include \"util/scoped.hh\"\n\n#include \n#include \n#include \n#include \n\n#include \n\nnamespace util {\nclass FilePiece;\n} // namespace util\n\nnamespace lm {\nclass PositiveProbWarn;\nnamespace ngram {\nclass SortedVocabulary;\nstruct Config;\n\nnamespace trie {\n\nclass EntryCompare : public std::binary_function {\n public:\n explicit EntryCompare(unsigned char order) : order_(order) {}\n\n bool operator()(const void *first_void, const void *second_void) const {\n const WordIndex *first = static_cast(first_void);\n const WordIndex *second = static_cast(second_void);\n const WordIndex *end = first + order_;\n for (; first != end; ++first, ++second) {\n if (*first < *second) return true;\n if (*first > *second) return false;\n }\n return false;\n }\n private:\n unsigned char order_;\n};\n\nclass RecordReader {\n public:\n RecordReader() : remains_(true) {}\n\n void Init(FILE *file, std::size_t entry_size);\n\n void *Data() { return data_.get(); }\n const void *Data() const { return data_.get(); }\n\n RecordReader &operator++() {\n std::size_t ret = fread(data_.get(), entry_size_, 1, file_);\n if (!ret) {\n UTIL_THROW_IF(!feof(file_), util::ErrnoException, \"Error reading temporary file\");\n remains_ = false;\n }\n return *this;\n }\n\n operator bool() const { return remains_; }\n\n void Rewind();\n\n std::size_t EntrySize() const { return entry_size_; }\n\n void Overwrite(const void *start, std::size_t amount);\n\n private:\n FILE *file_;\n\n util::scoped_malloc data_;\n\n bool remains_;\n\n std::size_t entry_size_;\n};\n\nclass SortedFiles {\n public:\n // Build from ARPA\n SortedFiles(const Config &config, util::FilePiece &f, std::vector &counts, std::size_t buffer, const std::string &file_prefix, SortedVocabulary &vocab);\n\n int StealUnigram() {\n return unigram_.release();\n }\n\n FILE *Full(unsigned char order) {\n return full_[order - 2].get();\n }\n\n FILE *Context(unsigned char of_order) {\n return context_[of_order - 2].get();\n }\n\n private:\n void ConvertToSorted(util::FilePiece &f, const SortedVocabulary &vocab, const std::vector &counts, const std::string &prefix, unsigned char order, PositiveProbWarn &warn, void *mem, std::size_t mem_size);\n \n util::scoped_fd unigram_;\n\n util::scoped_FILE full_[KENLM_MAX_ORDER - 1], context_[KENLM_MAX_ORDER - 1];\n};\n\n} // namespace trie\n} // namespace ngram\n} // namespace lm\n\n#endif // LM_TRIE_SORT__\n" }, { "path": "src/kenlm/lm/value.hh", "content": "#ifndef LM_VALUE__\n#define LM_VALUE__\n\n#include \"lm/model_type.hh\"\n#include \"lm/value_build.hh\"\n#include \"lm/weights.hh\"\n#include \"util/bit_packing.hh\"\n\n#include \n\nnamespace lm {\nnamespace ngram {\n\n// Template proxy for probing unigrams and middle.\ntemplate class GenericProbingProxy {\n public:\n explicit GenericProbingProxy(const Weights &to) : to_(&to) {}\n\n GenericProbingProxy() : to_(0) {}\n\n bool Found() const { return to_ != 0; }\n\n float Prob() const {\n util::FloatEnc enc;\n enc.f = to_->prob;\n enc.i |= util::kSignBit;\n return enc.f;\n }\n\n float Backoff() const { return to_->backoff; }\n\n bool IndependentLeft() const {\n util::FloatEnc enc;\n enc.f = to_->prob;\n return enc.i & util::kSignBit;\n }\n\n protected:\n const Weights *to_;\n};\n\n// Basic proxy for trie unigrams. \ntemplate class GenericTrieUnigramProxy {\n public:\n explicit GenericTrieUnigramProxy(const Weights &to) : to_(&to) {}\n\n GenericTrieUnigramProxy() : to_(0) {}\n\n bool Found() const { return to_ != 0; }\n float Prob() const { return to_->prob; }\n float Backoff() const { return to_->backoff; }\n float Rest() const { return Prob(); }\n\n protected:\n const Weights *to_;\n};\n\nstruct BackoffValue {\n typedef ProbBackoff Weights;\n static const ModelType kProbingModelType = PROBING;\n\n class ProbingProxy : public GenericProbingProxy {\n public:\n explicit ProbingProxy(const Weights &to) : GenericProbingProxy(to) {}\n ProbingProxy() {}\n float Rest() const { return Prob(); }\n };\n\n class TrieUnigramProxy : public GenericTrieUnigramProxy {\n public:\n explicit TrieUnigramProxy(const Weights &to) : GenericTrieUnigramProxy(to) {}\n TrieUnigramProxy() {}\n float Rest() const { return Prob(); }\n };\n\n struct ProbingEntry {\n typedef uint64_t Key;\n typedef Weights Value;\n uint64_t key;\n ProbBackoff value;\n uint64_t GetKey() const { return key; }\n };\n\n struct TrieUnigramValue {\n Weights weights;\n uint64_t next;\n uint64_t Next() const { return next; }\n };\n\n const static bool kDifferentRest = false;\n\n template void Callback(const Config &, unsigned int, typename Model::Vocabulary &, C &callback) {\n NoRestBuild build;\n callback(build);\n }\n};\n\nstruct RestValue {\n typedef RestWeights Weights;\n static const ModelType kProbingModelType = REST_PROBING;\n\n class ProbingProxy : public GenericProbingProxy {\n public:\n explicit ProbingProxy(const Weights &to) : GenericProbingProxy(to) {}\n ProbingProxy() {}\n float Rest() const { return to_->rest; }\n };\n\n class TrieUnigramProxy : public GenericTrieUnigramProxy {\n public:\n explicit TrieUnigramProxy(const Weights &to) : GenericTrieUnigramProxy(to) {}\n TrieUnigramProxy() {}\n float Rest() const { return to_->rest; }\n };\n\n// gcc 4.1 doesn't properly back dependent types :-(. \n#pragma pack(push)\n#pragma pack(4)\n struct ProbingEntry {\n typedef uint64_t Key;\n typedef Weights Value;\n Key key;\n Value value;\n Key GetKey() const { return key; }\n };\n\n struct TrieUnigramValue {\n Weights weights;\n uint64_t next;\n uint64_t Next() const { return next; }\n };\n#pragma pack(pop)\n\n const static bool kDifferentRest = true;\n\n template void Callback(const Config &config, unsigned int order, typename Model::Vocabulary &vocab, C &callback) {\n switch (config.rest_function) {\n case Config::REST_MAX:\n {\n MaxRestBuild build;\n callback(build);\n }\n break;\n case Config::REST_LOWER:\n {\n LowerRestBuild build(config, order, vocab);\n callback(build);\n }\n break;\n }\n }\n};\n\n} // namespace ngram\n} // namespace lm\n\n#endif // LM_VALUE__\n" }, { "path": "src/kenlm/lm/value_build.cc", "content": "#include \"lm/value_build.hh\"\n\n#include \"lm/model.hh\"\n#include \"lm/read_arpa.hh\"\n\nnamespace lm { \nnamespace ngram {\n\ntemplate LowerRestBuild::LowerRestBuild(const Config &config, unsigned int order, const typename Model::Vocabulary &vocab) {\n UTIL_THROW_IF(config.rest_lower_files.size() != order - 1, ConfigException, \"This model has order \" << order << \" so there should be \" << (order - 1) << \" lower-order models for rest cost purposes.\");\n Config for_lower = config;\n for_lower.rest_lower_files.clear();\n\n // Unigram models aren't supported, so this is a custom loader. \n // TODO: optimize the unigram loading? \n {\n util::FilePiece uni(config.rest_lower_files[0].c_str());\n std::vector number;\n ReadARPACounts(uni, number);\n UTIL_THROW_IF(number.size() != 1, FormatLoadException, \"Expected the unigram model to have order 1, not \" << number.size());\n ReadNGramHeader(uni, 1);\n unigrams_.resize(number[0]);\n unigrams_[0] = config.unknown_missing_logprob;\n PositiveProbWarn warn;\n for (uint64_t i = 0; i < number[0]; ++i) {\n WordIndex w;\n Prob entry;\n ReadNGram(uni, 1, vocab, &w, entry, warn);\n unigrams_[w] = entry.prob;\n }\n }\n\n try {\n for (unsigned int i = 2; i < order; ++i) {\n models_.push_back(new Model(config.rest_lower_files[i - 1].c_str(), for_lower));\n UTIL_THROW_IF(models_.back()->Order() != i, FormatLoadException, \"Lower order file \" << config.rest_lower_files[i-1] << \" should have order \" << i);\n }\n } catch (...) {\n for (typename std::vector::const_iterator i = models_.begin(); i != models_.end(); ++i) {\n delete *i;\n }\n models_.clear();\n throw;\n }\n\n // TODO: force/check same vocab. \n}\n\ntemplate LowerRestBuild::~LowerRestBuild() {\n for (typename std::vector::const_iterator i = models_.begin(); i != models_.end(); ++i) {\n delete *i;\n }\n}\n\ntemplate class LowerRestBuild;\n\n} // namespace ngram\n} // namespace lm\n" }, { "path": "src/kenlm/lm/value_build.hh", "content": "#ifndef LM_VALUE_BUILD__\n#define LM_VALUE_BUILD__\n\n#include \"lm/weights.hh\"\n#include \"lm/word_index.hh\"\n#include \"util/bit_packing.hh\"\n\n#include \n\nnamespace lm {\nnamespace ngram {\n\nstruct Config;\nstruct BackoffValue;\nstruct RestValue;\n\nclass NoRestBuild {\n public:\n typedef BackoffValue Value;\n\n NoRestBuild() {}\n\n void SetRest(const WordIndex *, unsigned int, const Prob &/*prob*/) const {}\n void SetRest(const WordIndex *, unsigned int, const ProbBackoff &) const {}\n\n template bool MarkExtends(ProbBackoff &weights, const Second &) const {\n util::UnsetSign(weights.prob);\n return false;\n }\n\n // Probing doesn't need to go back to unigram.\n const static bool kMarkEvenLower = false;\n};\n\nclass MaxRestBuild {\n public:\n typedef RestValue Value;\n\n MaxRestBuild() {}\n\n void SetRest(const WordIndex *, unsigned int, const Prob &/*prob*/) const {}\n void SetRest(const WordIndex *, unsigned int, RestWeights &weights) const {\n weights.rest = weights.prob;\n util::SetSign(weights.rest);\n }\n\n bool MarkExtends(RestWeights &weights, const RestWeights &to) const {\n util::UnsetSign(weights.prob);\n if (weights.rest >= to.rest) return false;\n weights.rest = to.rest;\n return true;\n }\n bool MarkExtends(RestWeights &weights, const Prob &to) const {\n util::UnsetSign(weights.prob);\n if (weights.rest >= to.prob) return false;\n weights.rest = to.prob;\n return true;\n }\n\n // Probing does need to go back to unigram. \n const static bool kMarkEvenLower = true;\n};\n\ntemplate class LowerRestBuild {\n public:\n typedef RestValue Value;\n\n LowerRestBuild(const Config &config, unsigned int order, const typename Model::Vocabulary &vocab);\n\n ~LowerRestBuild();\n\n void SetRest(const WordIndex *, unsigned int, const Prob &/*prob*/) const {}\n void SetRest(const WordIndex *vocab_ids, unsigned int n, RestWeights &weights) const {\n typename Model::State ignored;\n if (n == 1) {\n weights.rest = unigrams_[*vocab_ids];\n } else {\n weights.rest = models_[n-2]->FullScoreForgotState(vocab_ids + 1, vocab_ids + n, *vocab_ids, ignored).prob;\n }\n }\n\n template bool MarkExtends(RestWeights &weights, const Second &) const {\n util::UnsetSign(weights.prob);\n return false;\n }\n\n const static bool kMarkEvenLower = false;\n\n std::vector unigrams_;\n\n std::vector models_;\n};\n\n} // namespace ngram\n} // namespace lm\n\n#endif // LM_VALUE_BUILD__\n" }, { "path": "src/kenlm/lm/virtual_interface.cc", "content": "#include \"lm/virtual_interface.hh\"\n\n#include \"lm/lm_exception.hh\"\n\nnamespace lm {\nnamespace base {\n\nVocabulary::~Vocabulary() {}\n\nvoid Vocabulary::SetSpecial(WordIndex begin_sentence, WordIndex end_sentence, WordIndex not_found) {\n begin_sentence_ = begin_sentence;\n end_sentence_ = end_sentence;\n not_found_ = not_found;\n}\n\nModel::~Model() {}\n\n} // namespace base\n} // namespace lm\n" }, { "path": "src/kenlm/lm/virtual_interface.hh", "content": "#ifndef LM_VIRTUAL_INTERFACE__\n#define LM_VIRTUAL_INTERFACE__\n\n#include \"lm/return.hh\"\n#include \"lm/word_index.hh\"\n#include \"util/string_piece.hh\"\n\n#include \n\nnamespace lm {\nnamespace base {\n\ntemplate class ModelFacade;\n\n/* Vocabulary interface. Call Index(string) and get a word index for use in\n * calling Model. It provides faster convenience functions for , , and\n * although you can also find these using Index. \n *\n * Some models do not load the mapping from index to string. If you need this,\n * check if the model Vocabulary class implements such a function and access it\n * directly. \n *\n * The Vocabulary object is always owned by the Model and can be retrieved from\n * the Model using BaseVocabulary() for this abstract interface or\n * GetVocabulary() for the actual implementation (in which case you'll need the\n * actual implementation of the Model too). \n */\nclass Vocabulary {\n public:\n virtual ~Vocabulary();\n\n WordIndex BeginSentence() const { return begin_sentence_; }\n WordIndex EndSentence() const { return end_sentence_; }\n WordIndex NotFound() const { return not_found_; }\n\n /* Most implementations allow StringPiece lookups and need only override\n * Index(StringPiece). SRI requires null termination and overrides all\n * three methods. \n */\n virtual WordIndex Index(const StringPiece &str) const = 0;\n virtual WordIndex Index(const std::string &str) const {\n return Index(StringPiece(str));\n }\n virtual WordIndex Index(const char *str) const {\n return Index(StringPiece(str));\n }\n\n protected:\n // Call SetSpecial afterward. \n Vocabulary() {}\n\n Vocabulary(WordIndex begin_sentence, WordIndex end_sentence, WordIndex not_found) {\n SetSpecial(begin_sentence, end_sentence, not_found);\n }\n\n void SetSpecial(WordIndex begin_sentence, WordIndex end_sentence, WordIndex not_found);\n\n WordIndex begin_sentence_, end_sentence_, not_found_;\n\n private:\n // Disable copy constructors. They're private and undefined. \n // Ersatz boost::noncopyable.\n Vocabulary(const Vocabulary &);\n Vocabulary &operator=(const Vocabulary &);\n};\n\n/* There are two ways to access a Model. \n *\n *\n * OPTION 1: Access the Model directly (e.g. lm::ngram::Model in model.hh).\n *\n * Every Model implements the scoring function:\n * float Score(\n * const Model::State &in_state,\n * const WordIndex new_word,\n * Model::State &out_state) const;\n *\n * It can also return the length of n-gram matched by the model:\n * FullScoreReturn FullScore(\n * const Model::State &in_state,\n * const WordIndex new_word,\n * Model::State &out_state) const;\n *\n *\n * There are also accessor functions:\n * const State &BeginSentenceState() const;\n * const State &NullContextState() const;\n * const Vocabulary &GetVocabulary() const;\n * unsigned int Order() const;\n *\n * NB: In case you're wondering why the model implementation looks like it's\n * missing these methods, see facade.hh. \n *\n * This is the fastest way to use a model and presents a normal State class to\n * be included in a hypothesis state structure. \n *\n *\n * OPTION 2: Use the virtual interface below. \n *\n * The virtual interface allow you to decide which Model to use at runtime \n * without templatizing everything on the Model type. However, each Model has\n * its own State class, so a single State cannot be efficiently provided (it\n * would require using the maximum memory of any Model's State or memory\n * allocation with each lookup). This means you become responsible for\n * allocating memory with size StateSize() and passing it to the Score or \n * FullScore functions provided here. \n *\n * For example, cdec has a std::string containing the entire state of a\n * hypothesis. It can reserve StateSize bytes in this string for the model\n * state. \n *\n * All the State objects are POD, so it's ok to use raw memory for storing\n * State.\n * in_state and out_state must not have the same address. \n */\nclass Model {\n public:\n virtual ~Model();\n\n size_t StateSize() const { return state_size_; }\n const void *BeginSentenceMemory() const { return begin_sentence_memory_; }\n const void *NullContextMemory() const { return null_context_memory_; }\n\n // Requires in_state != out_state\n virtual float Score(const void *in_state, const WordIndex new_word, void *out_state) const = 0;\n\n // Requires in_state != out_state\n virtual FullScoreReturn FullScore(const void *in_state, const WordIndex new_word, void *out_state) const = 0;\n\n unsigned char Order() const { return order_; }\n\n const Vocabulary &BaseVocabulary() const { return *base_vocab_; }\n\n private:\n template friend class ModelFacade;\n explicit Model(size_t state_size) : state_size_(state_size) {}\n\n const size_t state_size_;\n const void *begin_sentence_memory_, *null_context_memory_;\n\n const Vocabulary *base_vocab_;\n\n unsigned char order_;\n\n // Disable copy constructors. They're private and undefined. \n // Ersatz boost::noncopyable.\n Model(const Model &);\n Model &operator=(const Model &);\n};\n\n} // mamespace base\n} // namespace lm\n\n#endif // LM_VIRTUAL_INTERFACE__\n" }, { "path": "src/kenlm/lm/weights.hh", "content": "#ifndef LM_WEIGHTS__\n#define LM_WEIGHTS__\n\n// Weights for n-grams. Probability and possibly a backoff. \n\nnamespace lm {\nstruct Prob {\n float prob;\n};\n// No inheritance so this will be a POD. \nstruct ProbBackoff {\n float prob;\n float backoff;\n};\nstruct RestWeights {\n float prob;\n float backoff;\n float rest;\n};\n\n} // namespace lm\n#endif // LM_WEIGHTS__\n" }, { "path": "src/kenlm/lm/word_index.hh", "content": "// Separate header because this is used often.\n#ifndef LM_WORD_INDEX__\n#define LM_WORD_INDEX__\n\n#include \n\nnamespace lm {\ntypedef unsigned int WordIndex;\nconst WordIndex kMaxWordIndex = UINT_MAX;\n} // namespace lm\n\ntypedef lm::WordIndex LMWordIndex;\n\n#endif\n" }, { "path": "src/kenlm/util/Jamfile", "content": "local compressed_flags = .. HAVE_ZLIB ;\nlocal compressed_deps = /top//z ;\nif [ test_library \"bz2\" ] && [ test_header \"bzlib.h\" ] {\n external-lib bz2 ;\n compressed_flags += HAVE_BZLIB ;\n compressed_deps += bz2 ;\n}\nif [ test_library \"lzma\" ] && [ test_header \"lzma.h\" ] {\n external-lib lzma ;\n compressed_flags += HAVE_XZLIB ;\n compressed_deps += lzma ;\n}\n\nobj read_compressed.o : read_compressed.cc : $(compressed_flags) ;\nalias read_compressed : read_compressed.o $(compressed_deps) ;\nobj read_compressed_test.o : read_compressed_test.cc /top//boost_unit_test_framework : $(compressed_flags) ;\nobj file_piece_test.o : file_piece_test.cc /top//boost_unit_test_framework : $(compressed_flags) ;\n\nfakelib kenutil : bit_packing.cc ersatz_progress.cc exception.cc file.cc file_piece.cc mmap.cc murmur_hash.cc pool.cc read_compressed scoped.cc string_piece.cc usage.cc double-conversion//double-conversion : .. : : .. ;\n\nimport testing ;\n\nunit-test bit_packing_test : bit_packing_test.cc kenutil /top//boost_unit_test_framework ;\nrun file_piece_test.o kenutil /top//boost_unit_test_framework : : file_piece.cc ;\nunit-test read_compressed_test : read_compressed_test.o kenutil /top//boost_unit_test_framework ;\nunit-test joint_sort_test : joint_sort_test.cc kenutil /top//boost_unit_test_framework ;\nunit-test probing_hash_table_test : probing_hash_table_test.cc kenutil /top//boost_unit_test_framework ;\nunit-test sorted_uniform_test : sorted_uniform_test.cc kenutil /top//boost_unit_test_framework ;\nunit-test tokenize_piece_test : tokenize_piece_test.cc kenutil /top//boost_unit_test_framework ;\nunit-test multi_intersection_test : multi_intersection_test.cc kenutil /top//boost_unit_test_framework ;\n" }, { "path": "src/kenlm/util/bit_packing.cc", "content": "#include \"util/bit_packing.hh\"\n#include \"util/exception.hh\"\n\n#include \n\nnamespace util {\n\nnamespace {\ntemplate struct StaticCheck {};\ntemplate <> struct StaticCheck { typedef bool StaticAssertionPassed; };\n\n// If your float isn't 4 bytes, we're hosed. \ntypedef StaticCheck::StaticAssertionPassed FloatSize;\n\n} // namespace\n\nuint8_t RequiredBits(uint64_t max_value) {\n if (!max_value) return 0;\n uint8_t ret = 1;\n while (max_value >>= 1) ++ret;\n return ret;\n}\n\nvoid BitPackingSanity() {\n const FloatEnc neg1 = { -1.0 }, pos1 = { 1.0 };\n if ((neg1.i ^ pos1.i) != 0x80000000) UTIL_THROW(Exception, \"Sign bit is not 0x80000000\");\n char mem[57+8];\n memset(mem, 0, sizeof(mem));\n const uint64_t test57 = 0x123456789abcdefULL;\n for (uint64_t b = 0; b < 57 * 8; b += 57) {\n WriteInt57(mem, b, 57, test57);\n }\n for (uint64_t b = 0; b < 57 * 8; b += 57) {\n if (test57 != ReadInt57(mem, b, 57, (1ULL << 57) - 1))\n UTIL_THROW(Exception, \"The bit packing routines are failing for your architecture. Please send a bug report with your architecture, operating system, and compiler.\");\n }\n // TODO: more checks. \n}\n\n} // namespace util\n" }, { "path": "src/kenlm/util/bit_packing.hh", "content": "#ifndef UTIL_BIT_PACKING__\n#define UTIL_BIT_PACKING__\n\n/* Bit-level packing routines \n *\n * WARNING WARNING WARNING:\n * The write functions assume that memory is zero initially. This makes them\n * faster and is the appropriate case for mmapped language model construction.\n * These routines assume that unaligned access to uint64_t is fast. This is\n * the case on x86_64. I'm not sure how fast unaligned 64-bit access is on\n * x86 but my target audience is large language models for which 64-bit is\n * necessary. \n *\n * Call the BitPackingSanity function to sanity check. Calling once suffices,\n * but it may be called multiple times when that's inconvenient. \n *\n * ARM and MinGW ports contributed by Hideo Okuma and Tomoyuki Yoshimura at\n * NICT.\n */\n\n#include \n#ifdef __APPLE__\n#include \n#elif __linux__\n#include \n#elif !defined(_WIN32) && !defined(_WIN64)\n#include \n#endif \n\n#include \n\n#include \n\nnamespace util {\n\n// Fun fact: __BYTE_ORDER is wrong on Solaris Sparc, but the version without __ is correct. \n#if BYTE_ORDER == LITTLE_ENDIAN\ninline uint8_t BitPackShift(uint8_t bit, uint8_t /*length*/) {\n return bit;\n}\n#elif BYTE_ORDER == BIG_ENDIAN\ninline uint8_t BitPackShift(uint8_t bit, uint8_t length) {\n return 64 - length - bit;\n}\n#else\n#error \"Bit packing code isn't written for your byte order.\"\n#endif\n\ninline uint64_t ReadOff(const void *base, uint64_t bit_off) {\n#if defined(__arm) || defined(__arm__)\n const uint8_t *base_off = reinterpret_cast(base) + (bit_off >> 3);\n uint64_t value64;\n memcpy(&value64, base_off, sizeof(value64));\n return value64;\n#else\n return *reinterpret_cast(reinterpret_cast(base) + (bit_off >> 3));\n#endif\n}\n\n/* Pack integers up to 57 bits using their least significant digits. \n * The length is specified using mask:\n * Assumes mask == (1 << length) - 1 where length <= 57. \n */\ninline uint64_t ReadInt57(const void *base, uint64_t bit_off, uint8_t length, uint64_t mask) {\n return (ReadOff(base, bit_off) >> BitPackShift(bit_off & 7, length)) & mask;\n}\n/* Assumes value < (1 << length) and length <= 57.\n * Assumes the memory is zero initially. \n */\ninline void WriteInt57(void *base, uint64_t bit_off, uint8_t length, uint64_t value) {\n#if defined(__arm) || defined(__arm__)\n uint8_t *base_off = reinterpret_cast(base) + (bit_off >> 3);\n uint64_t value64;\n memcpy(&value64, base_off, sizeof(value64));\n value64 |= (value << BitPackShift(bit_off & 7, length));\n memcpy(base_off, &value64, sizeof(value64));\n#else\n *reinterpret_cast(reinterpret_cast(base) + (bit_off >> 3)) |= \n (value << BitPackShift(bit_off & 7, length));\n#endif\n}\n\n/* Same caveats as above, but for a 25 bit limit. */\ninline uint32_t ReadInt25(const void *base, uint64_t bit_off, uint8_t length, uint32_t mask) {\n#if defined(__arm) || defined(__arm__)\n const uint8_t *base_off = reinterpret_cast(base) + (bit_off >> 3);\n uint32_t value32;\n memcpy(&value32, base_off, sizeof(value32));\n return (value32 >> BitPackShift(bit_off & 7, length)) & mask;\n#else\n return (*reinterpret_cast(reinterpret_cast(base) + (bit_off >> 3)) >> BitPackShift(bit_off & 7, length)) & mask;\n#endif\n}\n\ninline void WriteInt25(void *base, uint64_t bit_off, uint8_t length, uint32_t value) {\n#if defined(__arm) || defined(__arm__)\n uint8_t *base_off = reinterpret_cast(base) + (bit_off >> 3);\n uint32_t value32;\n memcpy(&value32, base_off, sizeof(value32));\n value32 |= (value << BitPackShift(bit_off & 7, length));\n memcpy(base_off, &value32, sizeof(value32));\n#else\n *reinterpret_cast(reinterpret_cast(base) + (bit_off >> 3)) |= \n (value << BitPackShift(bit_off & 7, length));\n#endif\n}\n\ntypedef union { float f; uint32_t i; } FloatEnc;\n\ninline float ReadFloat32(const void *base, uint64_t bit_off) {\n FloatEnc encoded;\n encoded.i = ReadOff(base, bit_off) >> BitPackShift(bit_off & 7, 32);\n return encoded.f;\n}\ninline void WriteFloat32(void *base, uint64_t bit_off, float value) {\n FloatEnc encoded;\n encoded.f = value;\n WriteInt57(base, bit_off, 32, encoded.i);\n}\n\nconst uint32_t kSignBit = 0x80000000;\n\ninline void SetSign(float &to) {\n FloatEnc enc;\n enc.f = to;\n enc.i |= kSignBit;\n to = enc.f;\n}\n\ninline void UnsetSign(float &to) {\n FloatEnc enc;\n enc.f = to;\n enc.i &= ~kSignBit;\n to = enc.f;\n}\n\ninline float ReadNonPositiveFloat31(const void *base, uint64_t bit_off) {\n FloatEnc encoded;\n encoded.i = ReadOff(base, bit_off) >> BitPackShift(bit_off & 7, 31);\n // Sign bit set means negative. \n encoded.i |= kSignBit;\n return encoded.f;\n}\ninline void WriteNonPositiveFloat31(void *base, uint64_t bit_off, float value) {\n FloatEnc encoded;\n encoded.f = value;\n encoded.i &= ~kSignBit;\n WriteInt57(base, bit_off, 31, encoded.i);\n}\n\nvoid BitPackingSanity();\n\n// Return bits required to store integers upto max_value. Not the most\n// efficient implementation, but this is only called a few times to size tries. \nuint8_t RequiredBits(uint64_t max_value);\n\nstruct BitsMask {\n static BitsMask ByMax(uint64_t max_value) {\n BitsMask ret;\n ret.FromMax(max_value);\n return ret;\n }\n static BitsMask ByBits(uint8_t bits) {\n BitsMask ret;\n ret.bits = bits;\n ret.mask = (1ULL << bits) - 1;\n return ret;\n }\n void FromMax(uint64_t max_value) {\n bits = RequiredBits(max_value);\n mask = (1ULL << bits) - 1;\n }\n uint8_t bits;\n uint64_t mask;\n};\n\nstruct BitAddress {\n BitAddress(void *in_base, uint64_t in_offset) : base(in_base), offset(in_offset) {}\n\n void *base;\n uint64_t offset;\n};\n\n} // namespace util\n\n#endif // UTIL_BIT_PACKING__\n" }, { "path": "src/kenlm/util/bit_packing_test.cc", "content": "#include \"util/bit_packing.hh\"\n\n#define BOOST_TEST_MODULE BitPackingTest\n#include \n\n#include \n\nnamespace util {\nnamespace {\n\nconst uint64_t test57 = 0x123456789abcdefULL;\nconst uint32_t test25 = 0x1234567;\n\nBOOST_AUTO_TEST_CASE(ZeroBit57) {\n char mem[16];\n memset(mem, 0, sizeof(mem));\n WriteInt57(mem, 0, 57, test57);\n BOOST_CHECK_EQUAL(test57, ReadInt57(mem, 0, 57, (1ULL << 57) - 1));\n}\n\nBOOST_AUTO_TEST_CASE(EachBit57) {\n char mem[16];\n for (uint8_t b = 0; b < 8; ++b) {\n memset(mem, 0, sizeof(mem));\n WriteInt57(mem, b, 57, test57);\n BOOST_CHECK_EQUAL(test57, ReadInt57(mem, b, 57, (1ULL << 57) - 1));\n }\n}\n\nBOOST_AUTO_TEST_CASE(Consecutive57) {\n char mem[57+8];\n memset(mem, 0, sizeof(mem));\n for (uint64_t b = 0; b < 57 * 8; b += 57) {\n WriteInt57(mem, b, 57, test57);\n BOOST_CHECK_EQUAL(test57, ReadInt57(mem, b, 57, (1ULL << 57) - 1));\n }\n for (uint64_t b = 0; b < 57 * 8; b += 57) {\n BOOST_CHECK_EQUAL(test57, ReadInt57(mem, b, 57, (1ULL << 57) - 1));\n }\n}\n\nBOOST_AUTO_TEST_CASE(Consecutive25) {\n char mem[25+8];\n memset(mem, 0, sizeof(mem));\n for (uint64_t b = 0; b < 25 * 8; b += 25) {\n WriteInt25(mem, b, 25, test25);\n BOOST_CHECK_EQUAL(test25, ReadInt25(mem, b, 25, (1ULL << 25) - 1));\n }\n for (uint64_t b = 0; b < 25 * 8; b += 25) {\n BOOST_CHECK_EQUAL(test25, ReadInt25(mem, b, 25, (1ULL << 25) - 1));\n }\n}\n\nBOOST_AUTO_TEST_CASE(Sanity) {\n BitPackingSanity();\n}\n\n} // namespace\n} // namespace util\n" }, { "path": "src/kenlm/util/double-conversion/Jamfile", "content": "fakelib double-conversion : [ glob *.cc ] : : : . ;\n" }, { "path": "src/kenlm/util/double-conversion/LICENSE", "content": "Copyright 2006-2011, the V8 project authors. All rights reserved.\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are\nmet:\n\n * Redistributions of source code must retain the above copyright\n notice, this list of conditions and the following disclaimer.\n * Redistributions in binary form must reproduce the above\n copyright notice, this list of conditions and the following\n disclaimer in the documentation and/or other materials provided\n with the distribution.\n * Neither the name of Google Inc. nor the names of its\n contributors may be used to endorse or promote products derived\n from this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\nLIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\nA PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\nOWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\nSPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\nLIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\nDATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\nTHEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n" }, { "path": "src/kenlm/util/double-conversion/bignum-dtoa.cc", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#include \n\n#include \"bignum-dtoa.h\"\n\n#include \"bignum.h\"\n#include \"ieee.h\"\n\nnamespace double_conversion {\n\nstatic int NormalizedExponent(uint64_t significand, int exponent) {\n ASSERT(significand != 0);\n while ((significand & Double::kHiddenBit) == 0) {\n significand = significand << 1;\n exponent = exponent - 1;\n }\n return exponent;\n}\n\n\n// Forward declarations:\n// Returns an estimation of k such that 10^(k-1) <= v < 10^k.\nstatic int EstimatePower(int exponent);\n// Computes v / 10^estimated_power exactly, as a ratio of two bignums, numerator\n// and denominator.\nstatic void InitialScaledStartValues(uint64_t significand,\n int exponent,\n bool lower_boundary_is_closer,\n int estimated_power,\n bool need_boundary_deltas,\n Bignum* numerator,\n Bignum* denominator,\n Bignum* delta_minus,\n Bignum* delta_plus);\n// Multiplies numerator/denominator so that its values lies in the range 1-10.\n// Returns decimal_point s.t.\n// v = numerator'/denominator' * 10^(decimal_point-1)\n// where numerator' and denominator' are the values of numerator and\n// denominator after the call to this function.\nstatic void FixupMultiply10(int estimated_power, bool is_even,\n int* decimal_point,\n Bignum* numerator, Bignum* denominator,\n Bignum* delta_minus, Bignum* delta_plus);\n// Generates digits from the left to the right and stops when the generated\n// digits yield the shortest decimal representation of v.\nstatic void GenerateShortestDigits(Bignum* numerator, Bignum* denominator,\n Bignum* delta_minus, Bignum* delta_plus,\n bool is_even,\n Vector buffer, int* length);\n// Generates 'requested_digits' after the decimal point.\nstatic void BignumToFixed(int requested_digits, int* decimal_point,\n Bignum* numerator, Bignum* denominator,\n Vector(buffer), int* length);\n// Generates 'count' digits of numerator/denominator.\n// Once 'count' digits have been produced rounds the result depending on the\n// remainder (remainders of exactly .5 round upwards). Might update the\n// decimal_point when rounding up (for example for 0.9999).\nstatic void GenerateCountedDigits(int count, int* decimal_point,\n Bignum* numerator, Bignum* denominator,\n Vector(buffer), int* length);\n\n\nvoid BignumDtoa(double v, BignumDtoaMode mode, int requested_digits,\n Vector buffer, int* length, int* decimal_point) {\n ASSERT(v > 0);\n ASSERT(!Double(v).IsSpecial());\n uint64_t significand;\n int exponent;\n bool lower_boundary_is_closer;\n if (mode == BIGNUM_DTOA_SHORTEST_SINGLE) {\n float f = static_cast(v);\n ASSERT(f == v);\n significand = Single(f).Significand();\n exponent = Single(f).Exponent();\n lower_boundary_is_closer = Single(f).LowerBoundaryIsCloser();\n } else {\n significand = Double(v).Significand();\n exponent = Double(v).Exponent();\n lower_boundary_is_closer = Double(v).LowerBoundaryIsCloser();\n }\n bool need_boundary_deltas =\n (mode == BIGNUM_DTOA_SHORTEST || mode == BIGNUM_DTOA_SHORTEST_SINGLE);\n\n bool is_even = (significand & 1) == 0;\n int normalized_exponent = NormalizedExponent(significand, exponent);\n // estimated_power might be too low by 1.\n int estimated_power = EstimatePower(normalized_exponent);\n\n // Shortcut for Fixed.\n // The requested digits correspond to the digits after the point. If the\n // number is much too small, then there is no need in trying to get any\n // digits.\n if (mode == BIGNUM_DTOA_FIXED && -estimated_power - 1 > requested_digits) {\n buffer[0] = '\\0';\n *length = 0;\n // Set decimal-point to -requested_digits. This is what Gay does.\n // Note that it should not have any effect anyways since the string is\n // empty.\n *decimal_point = -requested_digits;\n return;\n }\n\n Bignum numerator;\n Bignum denominator;\n Bignum delta_minus;\n Bignum delta_plus;\n // Make sure the bignum can grow large enough. The smallest double equals\n // 4e-324. In this case the denominator needs fewer than 324*4 binary digits.\n // The maximum double is 1.7976931348623157e308 which needs fewer than\n // 308*4 binary digits.\n ASSERT(Bignum::kMaxSignificantBits >= 324*4);\n InitialScaledStartValues(significand, exponent, lower_boundary_is_closer,\n estimated_power, need_boundary_deltas,\n &numerator, &denominator,\n &delta_minus, &delta_plus);\n // We now have v = (numerator / denominator) * 10^estimated_power.\n FixupMultiply10(estimated_power, is_even, decimal_point,\n &numerator, &denominator,\n &delta_minus, &delta_plus);\n // We now have v = (numerator / denominator) * 10^(decimal_point-1), and\n // 1 <= (numerator + delta_plus) / denominator < 10\n switch (mode) {\n case BIGNUM_DTOA_SHORTEST:\n case BIGNUM_DTOA_SHORTEST_SINGLE:\n GenerateShortestDigits(&numerator, &denominator,\n &delta_minus, &delta_plus,\n is_even, buffer, length);\n break;\n case BIGNUM_DTOA_FIXED:\n BignumToFixed(requested_digits, decimal_point,\n &numerator, &denominator,\n buffer, length);\n break;\n case BIGNUM_DTOA_PRECISION:\n GenerateCountedDigits(requested_digits, decimal_point,\n &numerator, &denominator,\n buffer, length);\n break;\n default:\n UNREACHABLE();\n }\n buffer[*length] = '\\0';\n}\n\n\n// The procedure starts generating digits from the left to the right and stops\n// when the generated digits yield the shortest decimal representation of v. A\n// decimal representation of v is a number lying closer to v than to any other\n// double, so it converts to v when read.\n//\n// This is true if d, the decimal representation, is between m- and m+, the\n// upper and lower boundaries. d must be strictly between them if !is_even.\n// m- := (numerator - delta_minus) / denominator\n// m+ := (numerator + delta_plus) / denominator\n//\n// Precondition: 0 <= (numerator+delta_plus) / denominator < 10.\n// If 1 <= (numerator+delta_plus) / denominator < 10 then no leading 0 digit\n// will be produced. This should be the standard precondition.\nstatic void GenerateShortestDigits(Bignum* numerator, Bignum* denominator,\n Bignum* delta_minus, Bignum* delta_plus,\n bool is_even,\n Vector buffer, int* length) {\n // Small optimization: if delta_minus and delta_plus are the same just reuse\n // one of the two bignums.\n if (Bignum::Equal(*delta_minus, *delta_plus)) {\n delta_plus = delta_minus;\n }\n *length = 0;\n while (true) {\n uint16_t digit;\n digit = numerator->DivideModuloIntBignum(*denominator);\n ASSERT(digit <= 9); // digit is a uint16_t and therefore always positive.\n // digit = numerator / denominator (integer division).\n // numerator = numerator % denominator.\n buffer[(*length)++] = digit + '0';\n\n // Can we stop already?\n // If the remainder of the division is less than the distance to the lower\n // boundary we can stop. In this case we simply round down (discarding the\n // remainder).\n // Similarly we test if we can round up (using the upper boundary).\n bool in_delta_room_minus;\n bool in_delta_room_plus;\n if (is_even) {\n in_delta_room_minus = Bignum::LessEqual(*numerator, *delta_minus);\n } else {\n in_delta_room_minus = Bignum::Less(*numerator, *delta_minus);\n }\n if (is_even) {\n in_delta_room_plus =\n Bignum::PlusCompare(*numerator, *delta_plus, *denominator) >= 0;\n } else {\n in_delta_room_plus =\n Bignum::PlusCompare(*numerator, *delta_plus, *denominator) > 0;\n }\n if (!in_delta_room_minus && !in_delta_room_plus) {\n // Prepare for next iteration.\n numerator->Times10();\n delta_minus->Times10();\n // We optimized delta_plus to be equal to delta_minus (if they share the\n // same value). So don't multiply delta_plus if they point to the same\n // object.\n if (delta_minus != delta_plus) {\n delta_plus->Times10();\n }\n } else if (in_delta_room_minus && in_delta_room_plus) {\n // Let's see if 2*numerator < denominator.\n // If yes, then the next digit would be < 5 and we can round down.\n int compare = Bignum::PlusCompare(*numerator, *numerator, *denominator);\n if (compare < 0) {\n // Remaining digits are less than .5. -> Round down (== do nothing).\n } else if (compare > 0) {\n // Remaining digits are more than .5 of denominator. -> Round up.\n // Note that the last digit could not be a '9' as otherwise the whole\n // loop would have stopped earlier.\n // We still have an assert here in case the preconditions were not\n // satisfied.\n ASSERT(buffer[(*length) - 1] != '9');\n buffer[(*length) - 1]++;\n } else {\n // Halfway case.\n // TODO(floitsch): need a way to solve half-way cases.\n // For now let's round towards even (since this is what Gay seems to\n // do).\n\n if ((buffer[(*length) - 1] - '0') % 2 == 0) {\n // Round down => Do nothing.\n } else {\n ASSERT(buffer[(*length) - 1] != '9');\n buffer[(*length) - 1]++;\n }\n }\n return;\n } else if (in_delta_room_minus) {\n // Round down (== do nothing).\n return;\n } else { // in_delta_room_plus\n // Round up.\n // Note again that the last digit could not be '9' since this would have\n // stopped the loop earlier.\n // We still have an ASSERT here, in case the preconditions were not\n // satisfied.\n ASSERT(buffer[(*length) -1] != '9');\n buffer[(*length) - 1]++;\n return;\n }\n }\n}\n\n\n// Let v = numerator / denominator < 10.\n// Then we generate 'count' digits of d = x.xxxxx... (without the decimal point)\n// from left to right. Once 'count' digits have been produced we decide wether\n// to round up or down. Remainders of exactly .5 round upwards. Numbers such\n// as 9.999999 propagate a carry all the way, and change the\n// exponent (decimal_point), when rounding upwards.\nstatic void GenerateCountedDigits(int count, int* decimal_point,\n Bignum* numerator, Bignum* denominator,\n Vector(buffer), int* length) {\n ASSERT(count >= 0);\n for (int i = 0; i < count - 1; ++i) {\n uint16_t digit;\n digit = numerator->DivideModuloIntBignum(*denominator);\n ASSERT(digit <= 9); // digit is a uint16_t and therefore always positive.\n // digit = numerator / denominator (integer division).\n // numerator = numerator % denominator.\n buffer[i] = digit + '0';\n // Prepare for next iteration.\n numerator->Times10();\n }\n // Generate the last digit.\n uint16_t digit;\n digit = numerator->DivideModuloIntBignum(*denominator);\n if (Bignum::PlusCompare(*numerator, *numerator, *denominator) >= 0) {\n digit++;\n }\n buffer[count - 1] = digit + '0';\n // Correct bad digits (in case we had a sequence of '9's). Propagate the\n // carry until we hat a non-'9' or til we reach the first digit.\n for (int i = count - 1; i > 0; --i) {\n if (buffer[i] != '0' + 10) break;\n buffer[i] = '0';\n buffer[i - 1]++;\n }\n if (buffer[0] == '0' + 10) {\n // Propagate a carry past the top place.\n buffer[0] = '1';\n (*decimal_point)++;\n }\n *length = count;\n}\n\n\n// Generates 'requested_digits' after the decimal point. It might omit\n// trailing '0's. If the input number is too small then no digits at all are\n// generated (ex.: 2 fixed digits for 0.00001).\n//\n// Input verifies: 1 <= (numerator + delta) / denominator < 10.\nstatic void BignumToFixed(int requested_digits, int* decimal_point,\n Bignum* numerator, Bignum* denominator,\n Vector(buffer), int* length) {\n // Note that we have to look at more than just the requested_digits, since\n // a number could be rounded up. Example: v=0.5 with requested_digits=0.\n // Even though the power of v equals 0 we can't just stop here.\n if (-(*decimal_point) > requested_digits) {\n // The number is definitively too small.\n // Ex: 0.001 with requested_digits == 1.\n // Set decimal-point to -requested_digits. This is what Gay does.\n // Note that it should not have any effect anyways since the string is\n // empty.\n *decimal_point = -requested_digits;\n *length = 0;\n return;\n } else if (-(*decimal_point) == requested_digits) {\n // We only need to verify if the number rounds down or up.\n // Ex: 0.04 and 0.06 with requested_digits == 1.\n ASSERT(*decimal_point == -requested_digits);\n // Initially the fraction lies in range (1, 10]. Multiply the denominator\n // by 10 so that we can compare more easily.\n denominator->Times10();\n if (Bignum::PlusCompare(*numerator, *numerator, *denominator) >= 0) {\n // If the fraction is >= 0.5 then we have to include the rounded\n // digit.\n buffer[0] = '1';\n *length = 1;\n (*decimal_point)++;\n } else {\n // Note that we caught most of similar cases earlier.\n *length = 0;\n }\n return;\n } else {\n // The requested digits correspond to the digits after the point.\n // The variable 'needed_digits' includes the digits before the point.\n int needed_digits = (*decimal_point) + requested_digits;\n GenerateCountedDigits(needed_digits, decimal_point,\n numerator, denominator,\n buffer, length);\n }\n}\n\n\n// Returns an estimation of k such that 10^(k-1) <= v < 10^k where\n// v = f * 2^exponent and 2^52 <= f < 2^53.\n// v is hence a normalized double with the given exponent. The output is an\n// approximation for the exponent of the decimal approimation .digits * 10^k.\n//\n// The result might undershoot by 1 in which case 10^k <= v < 10^k+1.\n// Note: this property holds for v's upper boundary m+ too.\n// 10^k <= m+ < 10^k+1.\n// (see explanation below).\n//\n// Examples:\n// EstimatePower(0) => 16\n// EstimatePower(-52) => 0\n//\n// Note: e >= 0 => EstimatedPower(e) > 0. No similar claim can be made for e<0.\nstatic int EstimatePower(int exponent) {\n // This function estimates log10 of v where v = f*2^e (with e == exponent).\n // Note that 10^floor(log10(v)) <= v, but v <= 10^ceil(log10(v)).\n // Note that f is bounded by its container size. Let p = 53 (the double's\n // significand size). Then 2^(p-1) <= f < 2^p.\n //\n // Given that log10(v) == log2(v)/log2(10) and e+(len(f)-1) is quite close\n // to log2(v) the function is simplified to (e+(len(f)-1)/log2(10)).\n // The computed number undershoots by less than 0.631 (when we compute log3\n // and not log10).\n //\n // Optimization: since we only need an approximated result this computation\n // can be performed on 64 bit integers. On x86/x64 architecture the speedup is\n // not really measurable, though.\n //\n // Since we want to avoid overshooting we decrement by 1e10 so that\n // floating-point imprecisions don't affect us.\n //\n // Explanation for v's boundary m+: the computation takes advantage of\n // the fact that 2^(p-1) <= f < 2^p. Boundaries still satisfy this requirement\n // (even for denormals where the delta can be much more important).\n\n const double k1Log10 = 0.30102999566398114; // 1/lg(10)\n\n // For doubles len(f) == 53 (don't forget the hidden bit).\n const int kSignificandSize = Double::kSignificandSize;\n double estimate = ceil((exponent + kSignificandSize - 1) * k1Log10 - 1e-10);\n return static_cast(estimate);\n}\n\n\n// See comments for InitialScaledStartValues.\nstatic void InitialScaledStartValuesPositiveExponent(\n uint64_t significand, int exponent,\n int estimated_power, bool need_boundary_deltas,\n Bignum* numerator, Bignum* denominator,\n Bignum* delta_minus, Bignum* delta_plus) {\n // A positive exponent implies a positive power.\n ASSERT(estimated_power >= 0);\n // Since the estimated_power is positive we simply multiply the denominator\n // by 10^estimated_power.\n\n // numerator = v.\n numerator->AssignUInt64(significand);\n numerator->ShiftLeft(exponent);\n // denominator = 10^estimated_power.\n denominator->AssignPowerUInt16(10, estimated_power);\n\n if (need_boundary_deltas) {\n // Introduce a common denominator so that the deltas to the boundaries are\n // integers.\n denominator->ShiftLeft(1);\n numerator->ShiftLeft(1);\n // Let v = f * 2^e, then m+ - v = 1/2 * 2^e; With the common\n // denominator (of 2) delta_plus equals 2^e.\n delta_plus->AssignUInt16(1);\n delta_plus->ShiftLeft(exponent);\n // Same for delta_minus. The adjustments if f == 2^p-1 are done later.\n delta_minus->AssignUInt16(1);\n delta_minus->ShiftLeft(exponent);\n }\n}\n\n\n// See comments for InitialScaledStartValues\nstatic void InitialScaledStartValuesNegativeExponentPositivePower(\n uint64_t significand, int exponent,\n int estimated_power, bool need_boundary_deltas,\n Bignum* numerator, Bignum* denominator,\n Bignum* delta_minus, Bignum* delta_plus) {\n // v = f * 2^e with e < 0, and with estimated_power >= 0.\n // This means that e is close to 0 (have a look at how estimated_power is\n // computed).\n\n // numerator = significand\n // since v = significand * 2^exponent this is equivalent to\n // numerator = v * / 2^-exponent\n numerator->AssignUInt64(significand);\n // denominator = 10^estimated_power * 2^-exponent (with exponent < 0)\n denominator->AssignPowerUInt16(10, estimated_power);\n denominator->ShiftLeft(-exponent);\n\n if (need_boundary_deltas) {\n // Introduce a common denominator so that the deltas to the boundaries are\n // integers.\n denominator->ShiftLeft(1);\n numerator->ShiftLeft(1);\n // Let v = f * 2^e, then m+ - v = 1/2 * 2^e; With the common\n // denominator (of 2) delta_plus equals 2^e.\n // Given that the denominator already includes v's exponent the distance\n // to the boundaries is simply 1.\n delta_plus->AssignUInt16(1);\n // Same for delta_minus. The adjustments if f == 2^p-1 are done later.\n delta_minus->AssignUInt16(1);\n }\n}\n\n\n// See comments for InitialScaledStartValues\nstatic void InitialScaledStartValuesNegativeExponentNegativePower(\n uint64_t significand, int exponent,\n int estimated_power, bool need_boundary_deltas,\n Bignum* numerator, Bignum* denominator,\n Bignum* delta_minus, Bignum* delta_plus) {\n // Instead of multiplying the denominator with 10^estimated_power we\n // multiply all values (numerator and deltas) by 10^-estimated_power.\n\n // Use numerator as temporary container for power_ten.\n Bignum* power_ten = numerator;\n power_ten->AssignPowerUInt16(10, -estimated_power);\n\n if (need_boundary_deltas) {\n // Since power_ten == numerator we must make a copy of 10^estimated_power\n // before we complete the computation of the numerator.\n // delta_plus = delta_minus = 10^estimated_power\n delta_plus->AssignBignum(*power_ten);\n delta_minus->AssignBignum(*power_ten);\n }\n\n // numerator = significand * 2 * 10^-estimated_power\n // since v = significand * 2^exponent this is equivalent to\n // numerator = v * 10^-estimated_power * 2 * 2^-exponent.\n // Remember: numerator has been abused as power_ten. So no need to assign it\n // to itself.\n ASSERT(numerator == power_ten);\n numerator->MultiplyByUInt64(significand);\n\n // denominator = 2 * 2^-exponent with exponent < 0.\n denominator->AssignUInt16(1);\n denominator->ShiftLeft(-exponent);\n\n if (need_boundary_deltas) {\n // Introduce a common denominator so that the deltas to the boundaries are\n // integers.\n numerator->ShiftLeft(1);\n denominator->ShiftLeft(1);\n // With this shift the boundaries have their correct value, since\n // delta_plus = 10^-estimated_power, and\n // delta_minus = 10^-estimated_power.\n // These assignments have been done earlier.\n // The adjustments if f == 2^p-1 (lower boundary is closer) are done later.\n }\n}\n\n\n// Let v = significand * 2^exponent.\n// Computes v / 10^estimated_power exactly, as a ratio of two bignums, numerator\n// and denominator. The functions GenerateShortestDigits and\n// GenerateCountedDigits will then convert this ratio to its decimal\n// representation d, with the required accuracy.\n// Then d * 10^estimated_power is the representation of v.\n// (Note: the fraction and the estimated_power might get adjusted before\n// generating the decimal representation.)\n//\n// The initial start values consist of:\n// - a scaled numerator: s.t. numerator/denominator == v / 10^estimated_power.\n// - a scaled (common) denominator.\n// optionally (used by GenerateShortestDigits to decide if it has the shortest\n// decimal converting back to v):\n// - v - m-: the distance to the lower boundary.\n// - m+ - v: the distance to the upper boundary.\n//\n// v, m+, m-, and therefore v - m- and m+ - v all share the same denominator.\n//\n// Let ep == estimated_power, then the returned values will satisfy:\n// v / 10^ep = numerator / denominator.\n// v's boundarys m- and m+:\n// m- / 10^ep == v / 10^ep - delta_minus / denominator\n// m+ / 10^ep == v / 10^ep + delta_plus / denominator\n// Or in other words:\n// m- == v - delta_minus * 10^ep / denominator;\n// m+ == v + delta_plus * 10^ep / denominator;\n//\n// Since 10^(k-1) <= v < 10^k (with k == estimated_power)\n// or 10^k <= v < 10^(k+1)\n// we then have 0.1 <= numerator/denominator < 1\n// or 1 <= numerator/denominator < 10\n//\n// It is then easy to kickstart the digit-generation routine.\n//\n// The boundary-deltas are only filled if the mode equals BIGNUM_DTOA_SHORTEST\n// or BIGNUM_DTOA_SHORTEST_SINGLE.\n\nstatic void InitialScaledStartValues(uint64_t significand,\n int exponent,\n bool lower_boundary_is_closer,\n int estimated_power,\n bool need_boundary_deltas,\n Bignum* numerator,\n Bignum* denominator,\n Bignum* delta_minus,\n Bignum* delta_plus) {\n if (exponent >= 0) {\n InitialScaledStartValuesPositiveExponent(\n significand, exponent, estimated_power, need_boundary_deltas,\n numerator, denominator, delta_minus, delta_plus);\n } else if (estimated_power >= 0) {\n InitialScaledStartValuesNegativeExponentPositivePower(\n significand, exponent, estimated_power, need_boundary_deltas,\n numerator, denominator, delta_minus, delta_plus);\n } else {\n InitialScaledStartValuesNegativeExponentNegativePower(\n significand, exponent, estimated_power, need_boundary_deltas,\n numerator, denominator, delta_minus, delta_plus);\n }\n\n if (need_boundary_deltas && lower_boundary_is_closer) {\n // The lower boundary is closer at half the distance of \"normal\" numbers.\n // Increase the common denominator and adapt all but the delta_minus.\n denominator->ShiftLeft(1); // *2\n numerator->ShiftLeft(1); // *2\n delta_plus->ShiftLeft(1); // *2\n }\n}\n\n\n// This routine multiplies numerator/denominator so that its values lies in the\n// range 1-10. That is after a call to this function we have:\n// 1 <= (numerator + delta_plus) /denominator < 10.\n// Let numerator the input before modification and numerator' the argument\n// after modification, then the output-parameter decimal_point is such that\n// numerator / denominator * 10^estimated_power ==\n// numerator' / denominator' * 10^(decimal_point - 1)\n// In some cases estimated_power was too low, and this is already the case. We\n// then simply adjust the power so that 10^(k-1) <= v < 10^k (with k ==\n// estimated_power) but do not touch the numerator or denominator.\n// Otherwise the routine multiplies the numerator and the deltas by 10.\nstatic void FixupMultiply10(int estimated_power, bool is_even,\n int* decimal_point,\n Bignum* numerator, Bignum* denominator,\n Bignum* delta_minus, Bignum* delta_plus) {\n bool in_range;\n if (is_even) {\n // For IEEE doubles half-way cases (in decimal system numbers ending with 5)\n // are rounded to the closest floating-point number with even significand.\n in_range = Bignum::PlusCompare(*numerator, *delta_plus, *denominator) >= 0;\n } else {\n in_range = Bignum::PlusCompare(*numerator, *delta_plus, *denominator) > 0;\n }\n if (in_range) {\n // Since numerator + delta_plus >= denominator we already have\n // 1 <= numerator/denominator < 10. Simply update the estimated_power.\n *decimal_point = estimated_power + 1;\n } else {\n *decimal_point = estimated_power;\n numerator->Times10();\n if (Bignum::Equal(*delta_minus, *delta_plus)) {\n delta_minus->Times10();\n delta_plus->AssignBignum(*delta_minus);\n } else {\n delta_minus->Times10();\n delta_plus->Times10();\n }\n }\n}\n\n} // namespace double_conversion\n" }, { "path": "src/kenlm/util/double-conversion/bignum-dtoa.h", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#ifndef DOUBLE_CONVERSION_BIGNUM_DTOA_H_\n#define DOUBLE_CONVERSION_BIGNUM_DTOA_H_\n\n#include \"utils.h\"\n\nnamespace double_conversion {\n\nenum BignumDtoaMode {\n // Return the shortest correct representation.\n // For example the output of 0.299999999999999988897 is (the less accurate but\n // correct) 0.3.\n BIGNUM_DTOA_SHORTEST,\n // Same as BIGNUM_DTOA_SHORTEST but for single-precision floats.\n BIGNUM_DTOA_SHORTEST_SINGLE,\n // Return a fixed number of digits after the decimal point.\n // For instance fixed(0.1, 4) becomes 0.1000\n // If the input number is big, the output will be big.\n BIGNUM_DTOA_FIXED,\n // Return a fixed number of digits, no matter what the exponent is.\n BIGNUM_DTOA_PRECISION\n};\n\n// Converts the given double 'v' to ascii.\n// The result should be interpreted as buffer * 10^(point-length).\n// The buffer will be null-terminated.\n//\n// The input v must be > 0 and different from NaN, and Infinity.\n//\n// The output depends on the given mode:\n// - SHORTEST: produce the least amount of digits for which the internal\n// identity requirement is still satisfied. If the digits are printed\n// (together with the correct exponent) then reading this number will give\n// 'v' again. The buffer will choose the representation that is closest to\n// 'v'. If there are two at the same distance, than the number is round up.\n// In this mode the 'requested_digits' parameter is ignored.\n// - FIXED: produces digits necessary to print a given number with\n// 'requested_digits' digits after the decimal point. The produced digits\n// might be too short in which case the caller has to fill the gaps with '0's.\n// Example: toFixed(0.001, 5) is allowed to return buffer=\"1\", point=-2.\n// Halfway cases are rounded up. The call toFixed(0.15, 2) thus returns\n// buffer=\"2\", point=0.\n// Note: the length of the returned buffer has no meaning wrt the significance\n// of its digits. That is, just because it contains '0's does not mean that\n// any other digit would not satisfy the internal identity requirement.\n// - PRECISION: produces 'requested_digits' where the first digit is not '0'.\n// Even though the length of produced digits usually equals\n// 'requested_digits', the function is allowed to return fewer digits, in\n// which case the caller has to fill the missing digits with '0's.\n// Halfway cases are again rounded up.\n// 'BignumDtoa' expects the given buffer to be big enough to hold all digits\n// and a terminating null-character.\nvoid BignumDtoa(double v, BignumDtoaMode mode, int requested_digits,\n Vector buffer, int* length, int* point);\n\n} // namespace double_conversion\n\n#endif // DOUBLE_CONVERSION_BIGNUM_DTOA_H_\n" }, { "path": "src/kenlm/util/double-conversion/bignum.cc", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#include \"bignum.h\"\n#include \"utils.h\"\n\nnamespace double_conversion {\n\nBignum::Bignum()\n : bigits_(bigits_buffer_, kBigitCapacity), used_digits_(0), exponent_(0) {\n for (int i = 0; i < kBigitCapacity; ++i) {\n bigits_[i] = 0;\n }\n}\n\n\ntemplate\nstatic int BitSize(S value) {\n return 8 * sizeof(value);\n}\n\n// Guaranteed to lie in one Bigit.\nvoid Bignum::AssignUInt16(uint16_t value) {\n ASSERT(kBigitSize >= BitSize(value));\n Zero();\n if (value == 0) return;\n\n EnsureCapacity(1);\n bigits_[0] = value;\n used_digits_ = 1;\n}\n\n\nvoid Bignum::AssignUInt64(uint64_t value) {\n const int kUInt64Size = 64;\n\n Zero();\n if (value == 0) return;\n\n int needed_bigits = kUInt64Size / kBigitSize + 1;\n EnsureCapacity(needed_bigits);\n for (int i = 0; i < needed_bigits; ++i) {\n bigits_[i] = value & kBigitMask;\n value = value >> kBigitSize;\n }\n used_digits_ = needed_bigits;\n Clamp();\n}\n\n\nvoid Bignum::AssignBignum(const Bignum& other) {\n exponent_ = other.exponent_;\n for (int i = 0; i < other.used_digits_; ++i) {\n bigits_[i] = other.bigits_[i];\n }\n // Clear the excess digits (if there were any).\n for (int i = other.used_digits_; i < used_digits_; ++i) {\n bigits_[i] = 0;\n }\n used_digits_ = other.used_digits_;\n}\n\n\nstatic uint64_t ReadUInt64(Vector buffer,\n int from,\n int digits_to_read) {\n uint64_t result = 0;\n for (int i = from; i < from + digits_to_read; ++i) {\n int digit = buffer[i] - '0';\n ASSERT(0 <= digit && digit <= 9);\n result = result * 10 + digit;\n }\n return result;\n}\n\n\nvoid Bignum::AssignDecimalString(Vector value) {\n // 2^64 = 18446744073709551616 > 10^19\n const int kMaxUint64DecimalDigits = 19;\n Zero();\n int length = value.length();\n int pos = 0;\n // Let's just say that each digit needs 4 bits.\n while (length >= kMaxUint64DecimalDigits) {\n uint64_t digits = ReadUInt64(value, pos, kMaxUint64DecimalDigits);\n pos += kMaxUint64DecimalDigits;\n length -= kMaxUint64DecimalDigits;\n MultiplyByPowerOfTen(kMaxUint64DecimalDigits);\n AddUInt64(digits);\n }\n uint64_t digits = ReadUInt64(value, pos, length);\n MultiplyByPowerOfTen(length);\n AddUInt64(digits);\n Clamp();\n}\n\n\nstatic int HexCharValue(char c) {\n if ('0' <= c && c <= '9') return c - '0';\n if ('a' <= c && c <= 'f') return 10 + c - 'a';\n if ('A' <= c && c <= 'F') return 10 + c - 'A';\n UNREACHABLE();\n return 0; // To make compiler happy.\n}\n\n\nvoid Bignum::AssignHexString(Vector value) {\n Zero();\n int length = value.length();\n\n int needed_bigits = length * 4 / kBigitSize + 1;\n EnsureCapacity(needed_bigits);\n int string_index = length - 1;\n for (int i = 0; i < needed_bigits - 1; ++i) {\n // These bigits are guaranteed to be \"full\".\n Chunk current_bigit = 0;\n for (int j = 0; j < kBigitSize / 4; j++) {\n current_bigit += HexCharValue(value[string_index--]) << (j * 4);\n }\n bigits_[i] = current_bigit;\n }\n used_digits_ = needed_bigits - 1;\n\n Chunk most_significant_bigit = 0; // Could be = 0;\n for (int j = 0; j <= string_index; ++j) {\n most_significant_bigit <<= 4;\n most_significant_bigit += HexCharValue(value[j]);\n }\n if (most_significant_bigit != 0) {\n bigits_[used_digits_] = most_significant_bigit;\n used_digits_++;\n }\n Clamp();\n}\n\n\nvoid Bignum::AddUInt64(uint64_t operand) {\n if (operand == 0) return;\n Bignum other;\n other.AssignUInt64(operand);\n AddBignum(other);\n}\n\n\nvoid Bignum::AddBignum(const Bignum& other) {\n ASSERT(IsClamped());\n ASSERT(other.IsClamped());\n\n // If this has a greater exponent than other append zero-bigits to this.\n // After this call exponent_ <= other.exponent_.\n Align(other);\n\n // There are two possibilities:\n // aaaaaaaaaaa 0000 (where the 0s represent a's exponent)\n // bbbbb 00000000\n // ----------------\n // ccccccccccc 0000\n // or\n // aaaaaaaaaa 0000\n // bbbbbbbbb 0000000\n // -----------------\n // cccccccccccc 0000\n // In both cases we might need a carry bigit.\n\n EnsureCapacity(1 + Max(BigitLength(), other.BigitLength()) - exponent_);\n Chunk carry = 0;\n int bigit_pos = other.exponent_ - exponent_;\n ASSERT(bigit_pos >= 0);\n for (int i = 0; i < other.used_digits_; ++i) {\n Chunk sum = bigits_[bigit_pos] + other.bigits_[i] + carry;\n bigits_[bigit_pos] = sum & kBigitMask;\n carry = sum >> kBigitSize;\n bigit_pos++;\n }\n\n while (carry != 0) {\n Chunk sum = bigits_[bigit_pos] + carry;\n bigits_[bigit_pos] = sum & kBigitMask;\n carry = sum >> kBigitSize;\n bigit_pos++;\n }\n used_digits_ = Max(bigit_pos, used_digits_);\n ASSERT(IsClamped());\n}\n\n\nvoid Bignum::SubtractBignum(const Bignum& other) {\n ASSERT(IsClamped());\n ASSERT(other.IsClamped());\n // We require this to be bigger than other.\n ASSERT(LessEqual(other, *this));\n\n Align(other);\n\n int offset = other.exponent_ - exponent_;\n Chunk borrow = 0;\n int i;\n for (i = 0; i < other.used_digits_; ++i) {\n ASSERT((borrow == 0) || (borrow == 1));\n Chunk difference = bigits_[i + offset] - other.bigits_[i] - borrow;\n bigits_[i + offset] = difference & kBigitMask;\n borrow = difference >> (kChunkSize - 1);\n }\n while (borrow != 0) {\n Chunk difference = bigits_[i + offset] - borrow;\n bigits_[i + offset] = difference & kBigitMask;\n borrow = difference >> (kChunkSize - 1);\n ++i;\n }\n Clamp();\n}\n\n\nvoid Bignum::ShiftLeft(int shift_amount) {\n if (used_digits_ == 0) return;\n exponent_ += shift_amount / kBigitSize;\n int local_shift = shift_amount % kBigitSize;\n EnsureCapacity(used_digits_ + 1);\n BigitsShiftLeft(local_shift);\n}\n\n\nvoid Bignum::MultiplyByUInt32(uint32_t factor) {\n if (factor == 1) return;\n if (factor == 0) {\n Zero();\n return;\n }\n if (used_digits_ == 0) return;\n\n // The product of a bigit with the factor is of size kBigitSize + 32.\n // Assert that this number + 1 (for the carry) fits into double chunk.\n ASSERT(kDoubleChunkSize >= kBigitSize + 32 + 1);\n DoubleChunk carry = 0;\n for (int i = 0; i < used_digits_; ++i) {\n DoubleChunk product = static_cast(factor) * bigits_[i] + carry;\n bigits_[i] = static_cast(product & kBigitMask);\n carry = (product >> kBigitSize);\n }\n while (carry != 0) {\n EnsureCapacity(used_digits_ + 1);\n bigits_[used_digits_] = carry & kBigitMask;\n used_digits_++;\n carry >>= kBigitSize;\n }\n}\n\n\nvoid Bignum::MultiplyByUInt64(uint64_t factor) {\n if (factor == 1) return;\n if (factor == 0) {\n Zero();\n return;\n }\n ASSERT(kBigitSize < 32);\n uint64_t carry = 0;\n uint64_t low = factor & 0xFFFFFFFF;\n uint64_t high = factor >> 32;\n for (int i = 0; i < used_digits_; ++i) {\n uint64_t product_low = low * bigits_[i];\n uint64_t product_high = high * bigits_[i];\n uint64_t tmp = (carry & kBigitMask) + product_low;\n bigits_[i] = tmp & kBigitMask;\n carry = (carry >> kBigitSize) + (tmp >> kBigitSize) +\n (product_high << (32 - kBigitSize));\n }\n while (carry != 0) {\n EnsureCapacity(used_digits_ + 1);\n bigits_[used_digits_] = carry & kBigitMask;\n used_digits_++;\n carry >>= kBigitSize;\n }\n}\n\n\nvoid Bignum::MultiplyByPowerOfTen(int exponent) {\n const uint64_t kFive27 = UINT64_2PART_C(0x6765c793, fa10079d);\n const uint16_t kFive1 = 5;\n const uint16_t kFive2 = kFive1 * 5;\n const uint16_t kFive3 = kFive2 * 5;\n const uint16_t kFive4 = kFive3 * 5;\n const uint16_t kFive5 = kFive4 * 5;\n const uint16_t kFive6 = kFive5 * 5;\n const uint32_t kFive7 = kFive6 * 5;\n const uint32_t kFive8 = kFive7 * 5;\n const uint32_t kFive9 = kFive8 * 5;\n const uint32_t kFive10 = kFive9 * 5;\n const uint32_t kFive11 = kFive10 * 5;\n const uint32_t kFive12 = kFive11 * 5;\n const uint32_t kFive13 = kFive12 * 5;\n const uint32_t kFive1_to_12[] =\n { kFive1, kFive2, kFive3, kFive4, kFive5, kFive6,\n kFive7, kFive8, kFive9, kFive10, kFive11, kFive12 };\n\n ASSERT(exponent >= 0);\n if (exponent == 0) return;\n if (used_digits_ == 0) return;\n\n // We shift by exponent at the end just before returning.\n int remaining_exponent = exponent;\n while (remaining_exponent >= 27) {\n MultiplyByUInt64(kFive27);\n remaining_exponent -= 27;\n }\n while (remaining_exponent >= 13) {\n MultiplyByUInt32(kFive13);\n remaining_exponent -= 13;\n }\n if (remaining_exponent > 0) {\n MultiplyByUInt32(kFive1_to_12[remaining_exponent - 1]);\n }\n ShiftLeft(exponent);\n}\n\n\nvoid Bignum::Square() {\n ASSERT(IsClamped());\n int product_length = 2 * used_digits_;\n EnsureCapacity(product_length);\n\n // Comba multiplication: compute each column separately.\n // Example: r = a2a1a0 * b2b1b0.\n // r = 1 * a0b0 +\n // 10 * (a1b0 + a0b1) +\n // 100 * (a2b0 + a1b1 + a0b2) +\n // 1000 * (a2b1 + a1b2) +\n // 10000 * a2b2\n //\n // In the worst case we have to accumulate nb-digits products of digit*digit.\n //\n // Assert that the additional number of bits in a DoubleChunk are enough to\n // sum up used_digits of Bigit*Bigit.\n if ((1 << (2 * (kChunkSize - kBigitSize))) <= used_digits_) {\n UNIMPLEMENTED();\n }\n DoubleChunk accumulator = 0;\n // First shift the digits so we don't overwrite them.\n int copy_offset = used_digits_;\n for (int i = 0; i < used_digits_; ++i) {\n bigits_[copy_offset + i] = bigits_[i];\n }\n // We have two loops to avoid some 'if's in the loop.\n for (int i = 0; i < used_digits_; ++i) {\n // Process temporary digit i with power i.\n // The sum of the two indices must be equal to i.\n int bigit_index1 = i;\n int bigit_index2 = 0;\n // Sum all of the sub-products.\n while (bigit_index1 >= 0) {\n Chunk chunk1 = bigits_[copy_offset + bigit_index1];\n Chunk chunk2 = bigits_[copy_offset + bigit_index2];\n accumulator += static_cast(chunk1) * chunk2;\n bigit_index1--;\n bigit_index2++;\n }\n bigits_[i] = static_cast(accumulator) & kBigitMask;\n accumulator >>= kBigitSize;\n }\n for (int i = used_digits_; i < product_length; ++i) {\n int bigit_index1 = used_digits_ - 1;\n int bigit_index2 = i - bigit_index1;\n // Invariant: sum of both indices is again equal to i.\n // Inner loop runs 0 times on last iteration, emptying accumulator.\n while (bigit_index2 < used_digits_) {\n Chunk chunk1 = bigits_[copy_offset + bigit_index1];\n Chunk chunk2 = bigits_[copy_offset + bigit_index2];\n accumulator += static_cast(chunk1) * chunk2;\n bigit_index1--;\n bigit_index2++;\n }\n // The overwritten bigits_[i] will never be read in further loop iterations,\n // because bigit_index1 and bigit_index2 are always greater\n // than i - used_digits_.\n bigits_[i] = static_cast(accumulator) & kBigitMask;\n accumulator >>= kBigitSize;\n }\n // Since the result was guaranteed to lie inside the number the\n // accumulator must be 0 now.\n ASSERT(accumulator == 0);\n\n // Don't forget to update the used_digits and the exponent.\n used_digits_ = product_length;\n exponent_ *= 2;\n Clamp();\n}\n\n\nvoid Bignum::AssignPowerUInt16(uint16_t base, int power_exponent) {\n ASSERT(base != 0);\n ASSERT(power_exponent >= 0);\n if (power_exponent == 0) {\n AssignUInt16(1);\n return;\n }\n Zero();\n int shifts = 0;\n // We expect base to be in range 2-32, and most often to be 10.\n // It does not make much sense to implement different algorithms for counting\n // the bits.\n while ((base & 1) == 0) {\n base >>= 1;\n shifts++;\n }\n int bit_size = 0;\n int tmp_base = base;\n while (tmp_base != 0) {\n tmp_base >>= 1;\n bit_size++;\n }\n int final_size = bit_size * power_exponent;\n // 1 extra bigit for the shifting, and one for rounded final_size.\n EnsureCapacity(final_size / kBigitSize + 2);\n\n // Left to Right exponentiation.\n int mask = 1;\n while (power_exponent >= mask) mask <<= 1;\n\n // The mask is now pointing to the bit above the most significant 1-bit of\n // power_exponent.\n // Get rid of first 1-bit;\n mask >>= 2;\n uint64_t this_value = base;\n\n bool delayed_multipliciation = false;\n const uint64_t max_32bits = 0xFFFFFFFF;\n while (mask != 0 && this_value <= max_32bits) {\n this_value = this_value * this_value;\n // Verify that there is enough space in this_value to perform the\n // multiplication. The first bit_size bits must be 0.\n if ((power_exponent & mask) != 0) {\n uint64_t base_bits_mask =\n ~((static_cast(1) << (64 - bit_size)) - 1);\n bool high_bits_zero = (this_value & base_bits_mask) == 0;\n if (high_bits_zero) {\n this_value *= base;\n } else {\n delayed_multipliciation = true;\n }\n }\n mask >>= 1;\n }\n AssignUInt64(this_value);\n if (delayed_multipliciation) {\n MultiplyByUInt32(base);\n }\n\n // Now do the same thing as a bignum.\n while (mask != 0) {\n Square();\n if ((power_exponent & mask) != 0) {\n MultiplyByUInt32(base);\n }\n mask >>= 1;\n }\n\n // And finally add the saved shifts.\n ShiftLeft(shifts * power_exponent);\n}\n\n\n// Precondition: this/other < 16bit.\nuint16_t Bignum::DivideModuloIntBignum(const Bignum& other) {\n ASSERT(IsClamped());\n ASSERT(other.IsClamped());\n ASSERT(other.used_digits_ > 0);\n\n // Easy case: if we have less digits than the divisor than the result is 0.\n // Note: this handles the case where this == 0, too.\n if (BigitLength() < other.BigitLength()) {\n return 0;\n }\n\n Align(other);\n\n uint16_t result = 0;\n\n // Start by removing multiples of 'other' until both numbers have the same\n // number of digits.\n while (BigitLength() > other.BigitLength()) {\n // This naive approach is extremely inefficient if the this divided other\n // might be big. This function is implemented for doubleToString where\n // the result should be small (less than 10).\n ASSERT(other.bigits_[other.used_digits_ - 1] >= ((1 << kBigitSize) / 16));\n // Remove the multiples of the first digit.\n // Example this = 23 and other equals 9. -> Remove 2 multiples.\n result += bigits_[used_digits_ - 1];\n SubtractTimes(other, bigits_[used_digits_ - 1]);\n }\n\n ASSERT(BigitLength() == other.BigitLength());\n\n // Both bignums are at the same length now.\n // Since other has more than 0 digits we know that the access to\n // bigits_[used_digits_ - 1] is safe.\n Chunk this_bigit = bigits_[used_digits_ - 1];\n Chunk other_bigit = other.bigits_[other.used_digits_ - 1];\n\n if (other.used_digits_ == 1) {\n // Shortcut for easy (and common) case.\n int quotient = this_bigit / other_bigit;\n bigits_[used_digits_ - 1] = this_bigit - other_bigit * quotient;\n result += quotient;\n Clamp();\n return result;\n }\n\n int division_estimate = this_bigit / (other_bigit + 1);\n result += division_estimate;\n SubtractTimes(other, division_estimate);\n\n if (other_bigit * (division_estimate + 1) > this_bigit) {\n // No need to even try to subtract. Even if other's remaining digits were 0\n // another subtraction would be too much.\n return result;\n }\n\n while (LessEqual(other, *this)) {\n SubtractBignum(other);\n result++;\n }\n return result;\n}\n\n\ntemplate\nstatic int SizeInHexChars(S number) {\n ASSERT(number > 0);\n int result = 0;\n while (number != 0) {\n number >>= 4;\n result++;\n }\n return result;\n}\n\n\nstatic char HexCharOfValue(int value) {\n ASSERT(0 <= value && value <= 16);\n if (value < 10) return value + '0';\n return value - 10 + 'A';\n}\n\n\nbool Bignum::ToHexString(char* buffer, int buffer_size) const {\n ASSERT(IsClamped());\n // Each bigit must be printable as separate hex-character.\n ASSERT(kBigitSize % 4 == 0);\n const int kHexCharsPerBigit = kBigitSize / 4;\n\n if (used_digits_ == 0) {\n if (buffer_size < 2) return false;\n buffer[0] = '0';\n buffer[1] = '\\0';\n return true;\n }\n // We add 1 for the terminating '\\0' character.\n int needed_chars = (BigitLength() - 1) * kHexCharsPerBigit +\n SizeInHexChars(bigits_[used_digits_ - 1]) + 1;\n if (needed_chars > buffer_size) return false;\n int string_index = needed_chars - 1;\n buffer[string_index--] = '\\0';\n for (int i = 0; i < exponent_; ++i) {\n for (int j = 0; j < kHexCharsPerBigit; ++j) {\n buffer[string_index--] = '0';\n }\n }\n for (int i = 0; i < used_digits_ - 1; ++i) {\n Chunk current_bigit = bigits_[i];\n for (int j = 0; j < kHexCharsPerBigit; ++j) {\n buffer[string_index--] = HexCharOfValue(current_bigit & 0xF);\n current_bigit >>= 4;\n }\n }\n // And finally the last bigit.\n Chunk most_significant_bigit = bigits_[used_digits_ - 1];\n while (most_significant_bigit != 0) {\n buffer[string_index--] = HexCharOfValue(most_significant_bigit & 0xF);\n most_significant_bigit >>= 4;\n }\n return true;\n}\n\n\nBignum::Chunk Bignum::BigitAt(int index) const {\n if (index >= BigitLength()) return 0;\n if (index < exponent_) return 0;\n return bigits_[index - exponent_];\n}\n\n\nint Bignum::Compare(const Bignum& a, const Bignum& b) {\n ASSERT(a.IsClamped());\n ASSERT(b.IsClamped());\n int bigit_length_a = a.BigitLength();\n int bigit_length_b = b.BigitLength();\n if (bigit_length_a < bigit_length_b) return -1;\n if (bigit_length_a > bigit_length_b) return +1;\n for (int i = bigit_length_a - 1; i >= Min(a.exponent_, b.exponent_); --i) {\n Chunk bigit_a = a.BigitAt(i);\n Chunk bigit_b = b.BigitAt(i);\n if (bigit_a < bigit_b) return -1;\n if (bigit_a > bigit_b) return +1;\n // Otherwise they are equal up to this digit. Try the next digit.\n }\n return 0;\n}\n\n\nint Bignum::PlusCompare(const Bignum& a, const Bignum& b, const Bignum& c) {\n ASSERT(a.IsClamped());\n ASSERT(b.IsClamped());\n ASSERT(c.IsClamped());\n if (a.BigitLength() < b.BigitLength()) {\n return PlusCompare(b, a, c);\n }\n if (a.BigitLength() + 1 < c.BigitLength()) return -1;\n if (a.BigitLength() > c.BigitLength()) return +1;\n // The exponent encodes 0-bigits. So if there are more 0-digits in 'a' than\n // 'b' has digits, then the bigit-length of 'a'+'b' must be equal to the one\n // of 'a'.\n if (a.exponent_ >= b.BigitLength() && a.BigitLength() < c.BigitLength()) {\n return -1;\n }\n\n Chunk borrow = 0;\n // Starting at min_exponent all digits are == 0. So no need to compare them.\n int min_exponent = Min(Min(a.exponent_, b.exponent_), c.exponent_);\n for (int i = c.BigitLength() - 1; i >= min_exponent; --i) {\n Chunk chunk_a = a.BigitAt(i);\n Chunk chunk_b = b.BigitAt(i);\n Chunk chunk_c = c.BigitAt(i);\n Chunk sum = chunk_a + chunk_b;\n if (sum > chunk_c + borrow) {\n return +1;\n } else {\n borrow = chunk_c + borrow - sum;\n if (borrow > 1) return -1;\n borrow <<= kBigitSize;\n }\n }\n if (borrow == 0) return 0;\n return -1;\n}\n\n\nvoid Bignum::Clamp() {\n while (used_digits_ > 0 && bigits_[used_digits_ - 1] == 0) {\n used_digits_--;\n }\n if (used_digits_ == 0) {\n // Zero.\n exponent_ = 0;\n }\n}\n\n\nbool Bignum::IsClamped() const {\n return used_digits_ == 0 || bigits_[used_digits_ - 1] != 0;\n}\n\n\nvoid Bignum::Zero() {\n for (int i = 0; i < used_digits_; ++i) {\n bigits_[i] = 0;\n }\n used_digits_ = 0;\n exponent_ = 0;\n}\n\n\nvoid Bignum::Align(const Bignum& other) {\n if (exponent_ > other.exponent_) {\n // If \"X\" represents a \"hidden\" digit (by the exponent) then we are in the\n // following case (a == this, b == other):\n // a: aaaaaaXXXX or a: aaaaaXXX\n // b: bbbbbbX b: bbbbbbbbXX\n // We replace some of the hidden digits (X) of a with 0 digits.\n // a: aaaaaa000X or a: aaaaa0XX\n int zero_digits = exponent_ - other.exponent_;\n EnsureCapacity(used_digits_ + zero_digits);\n for (int i = used_digits_ - 1; i >= 0; --i) {\n bigits_[i + zero_digits] = bigits_[i];\n }\n for (int i = 0; i < zero_digits; ++i) {\n bigits_[i] = 0;\n }\n used_digits_ += zero_digits;\n exponent_ -= zero_digits;\n ASSERT(used_digits_ >= 0);\n ASSERT(exponent_ >= 0);\n }\n}\n\n\nvoid Bignum::BigitsShiftLeft(int shift_amount) {\n ASSERT(shift_amount < kBigitSize);\n ASSERT(shift_amount >= 0);\n Chunk carry = 0;\n for (int i = 0; i < used_digits_; ++i) {\n Chunk new_carry = bigits_[i] >> (kBigitSize - shift_amount);\n bigits_[i] = ((bigits_[i] << shift_amount) + carry) & kBigitMask;\n carry = new_carry;\n }\n if (carry != 0) {\n bigits_[used_digits_] = carry;\n used_digits_++;\n }\n}\n\n\nvoid Bignum::SubtractTimes(const Bignum& other, int factor) {\n ASSERT(exponent_ <= other.exponent_);\n if (factor < 3) {\n for (int i = 0; i < factor; ++i) {\n SubtractBignum(other);\n }\n return;\n }\n Chunk borrow = 0;\n int exponent_diff = other.exponent_ - exponent_;\n for (int i = 0; i < other.used_digits_; ++i) {\n DoubleChunk product = static_cast(factor) * other.bigits_[i];\n DoubleChunk remove = borrow + product;\n Chunk difference = bigits_[i + exponent_diff] - (remove & kBigitMask);\n bigits_[i + exponent_diff] = difference & kBigitMask;\n borrow = static_cast((difference >> (kChunkSize - 1)) +\n (remove >> kBigitSize));\n }\n for (int i = other.used_digits_ + exponent_diff; i < used_digits_; ++i) {\n if (borrow == 0) return;\n Chunk difference = bigits_[i] - borrow;\n bigits_[i] = difference & kBigitMask;\n borrow = difference >> (kChunkSize - 1);\n ++i;\n }\n Clamp();\n}\n\n\n} // namespace double_conversion\n" }, { "path": "src/kenlm/util/double-conversion/bignum.h", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#ifndef DOUBLE_CONVERSION_BIGNUM_H_\n#define DOUBLE_CONVERSION_BIGNUM_H_\n\n#include \"utils.h\"\n\nnamespace double_conversion {\n\nclass Bignum {\n public:\n // 3584 = 128 * 28. We can represent 2^3584 > 10^1000 accurately.\n // This bignum can encode much bigger numbers, since it contains an\n // exponent.\n static const int kMaxSignificantBits = 3584;\n\n Bignum();\n void AssignUInt16(uint16_t value);\n void AssignUInt64(uint64_t value);\n void AssignBignum(const Bignum& other);\n\n void AssignDecimalString(Vector value);\n void AssignHexString(Vector value);\n\n void AssignPowerUInt16(uint16_t base, int exponent);\n\n void AddUInt16(uint16_t operand);\n void AddUInt64(uint64_t operand);\n void AddBignum(const Bignum& other);\n // Precondition: this >= other.\n void SubtractBignum(const Bignum& other);\n\n void Square();\n void ShiftLeft(int shift_amount);\n void MultiplyByUInt32(uint32_t factor);\n void MultiplyByUInt64(uint64_t factor);\n void MultiplyByPowerOfTen(int exponent);\n void Times10() { return MultiplyByUInt32(10); }\n // Pseudocode:\n // int result = this / other;\n // this = this % other;\n // In the worst case this function is in O(this/other).\n uint16_t DivideModuloIntBignum(const Bignum& other);\n\n bool ToHexString(char* buffer, int buffer_size) const;\n\n // Returns\n // -1 if a < b,\n // 0 if a == b, and\n // +1 if a > b.\n static int Compare(const Bignum& a, const Bignum& b);\n static bool Equal(const Bignum& a, const Bignum& b) {\n return Compare(a, b) == 0;\n }\n static bool LessEqual(const Bignum& a, const Bignum& b) {\n return Compare(a, b) <= 0;\n }\n static bool Less(const Bignum& a, const Bignum& b) {\n return Compare(a, b) < 0;\n }\n // Returns Compare(a + b, c);\n static int PlusCompare(const Bignum& a, const Bignum& b, const Bignum& c);\n // Returns a + b == c\n static bool PlusEqual(const Bignum& a, const Bignum& b, const Bignum& c) {\n return PlusCompare(a, b, c) == 0;\n }\n // Returns a + b <= c\n static bool PlusLessEqual(const Bignum& a, const Bignum& b, const Bignum& c) {\n return PlusCompare(a, b, c) <= 0;\n }\n // Returns a + b < c\n static bool PlusLess(const Bignum& a, const Bignum& b, const Bignum& c) {\n return PlusCompare(a, b, c) < 0;\n }\n private:\n typedef uint32_t Chunk;\n typedef uint64_t DoubleChunk;\n\n static const int kChunkSize = sizeof(Chunk) * 8;\n static const int kDoubleChunkSize = sizeof(DoubleChunk) * 8;\n // With bigit size of 28 we loose some bits, but a double still fits easily\n // into two chunks, and more importantly we can use the Comba multiplication.\n static const int kBigitSize = 28;\n static const Chunk kBigitMask = (1 << kBigitSize) - 1;\n // Every instance allocates kBigitLength chunks on the stack. Bignums cannot\n // grow. There are no checks if the stack-allocated space is sufficient.\n static const int kBigitCapacity = kMaxSignificantBits / kBigitSize;\n\n void EnsureCapacity(int size) {\n if (size > kBigitCapacity) {\n UNREACHABLE();\n }\n }\n void Align(const Bignum& other);\n void Clamp();\n bool IsClamped() const;\n void Zero();\n // Requires this to have enough capacity (no tests done).\n // Updates used_digits_ if necessary.\n // shift_amount must be < kBigitSize.\n void BigitsShiftLeft(int shift_amount);\n // BigitLength includes the \"hidden\" digits encoded in the exponent.\n int BigitLength() const { return used_digits_ + exponent_; }\n Chunk BigitAt(int index) const;\n void SubtractTimes(const Bignum& other, int factor);\n\n Chunk bigits_buffer_[kBigitCapacity];\n // A vector backed by bigits_buffer_. This way accesses to the array are\n // checked for out-of-bounds errors.\n Vector bigits_;\n int used_digits_;\n // The Bignum's value equals value(bigits_) * 2^(exponent_ * kBigitSize).\n int exponent_;\n\n DISALLOW_COPY_AND_ASSIGN(Bignum);\n};\n\n} // namespace double_conversion\n\n#endif // DOUBLE_CONVERSION_BIGNUM_H_\n" }, { "path": "src/kenlm/util/double-conversion/cached-powers.cc", "content": "// Copyright 2006-2008 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#include \n#include \n#include \n\n#include \"utils.h\"\n\n#include \"cached-powers.h\"\n\nnamespace double_conversion {\n\nstruct CachedPower {\n uint64_t significand;\n int16_t binary_exponent;\n int16_t decimal_exponent;\n};\n\nstatic const CachedPower kCachedPowers[] = {\n {UINT64_2PART_C(0xfa8fd5a0, 081c0288), -1220, -348},\n {UINT64_2PART_C(0xbaaee17f, a23ebf76), -1193, -340},\n {UINT64_2PART_C(0x8b16fb20, 3055ac76), -1166, -332},\n {UINT64_2PART_C(0xcf42894a, 5dce35ea), -1140, -324},\n {UINT64_2PART_C(0x9a6bb0aa, 55653b2d), -1113, -316},\n {UINT64_2PART_C(0xe61acf03, 3d1a45df), -1087, -308},\n {UINT64_2PART_C(0xab70fe17, c79ac6ca), -1060, -300},\n {UINT64_2PART_C(0xff77b1fc, bebcdc4f), -1034, -292},\n {UINT64_2PART_C(0xbe5691ef, 416bd60c), -1007, -284},\n {UINT64_2PART_C(0x8dd01fad, 907ffc3c), -980, -276},\n {UINT64_2PART_C(0xd3515c28, 31559a83), -954, -268},\n {UINT64_2PART_C(0x9d71ac8f, ada6c9b5), -927, -260},\n {UINT64_2PART_C(0xea9c2277, 23ee8bcb), -901, -252},\n {UINT64_2PART_C(0xaecc4991, 4078536d), -874, -244},\n {UINT64_2PART_C(0x823c1279, 5db6ce57), -847, -236},\n {UINT64_2PART_C(0xc2109436, 4dfb5637), -821, -228},\n {UINT64_2PART_C(0x9096ea6f, 3848984f), -794, -220},\n {UINT64_2PART_C(0xd77485cb, 25823ac7), -768, -212},\n {UINT64_2PART_C(0xa086cfcd, 97bf97f4), -741, -204},\n {UINT64_2PART_C(0xef340a98, 172aace5), -715, -196},\n {UINT64_2PART_C(0xb23867fb, 2a35b28e), -688, -188},\n {UINT64_2PART_C(0x84c8d4df, d2c63f3b), -661, -180},\n {UINT64_2PART_C(0xc5dd4427, 1ad3cdba), -635, -172},\n {UINT64_2PART_C(0x936b9fce, bb25c996), -608, -164},\n {UINT64_2PART_C(0xdbac6c24, 7d62a584), -582, -156},\n {UINT64_2PART_C(0xa3ab6658, 0d5fdaf6), -555, -148},\n {UINT64_2PART_C(0xf3e2f893, dec3f126), -529, -140},\n {UINT64_2PART_C(0xb5b5ada8, aaff80b8), -502, -132},\n {UINT64_2PART_C(0x87625f05, 6c7c4a8b), -475, -124},\n {UINT64_2PART_C(0xc9bcff60, 34c13053), -449, -116},\n {UINT64_2PART_C(0x964e858c, 91ba2655), -422, -108},\n {UINT64_2PART_C(0xdff97724, 70297ebd), -396, -100},\n {UINT64_2PART_C(0xa6dfbd9f, b8e5b88f), -369, -92},\n {UINT64_2PART_C(0xf8a95fcf, 88747d94), -343, -84},\n {UINT64_2PART_C(0xb9447093, 8fa89bcf), -316, -76},\n {UINT64_2PART_C(0x8a08f0f8, bf0f156b), -289, -68},\n {UINT64_2PART_C(0xcdb02555, 653131b6), -263, -60},\n {UINT64_2PART_C(0x993fe2c6, d07b7fac), -236, -52},\n {UINT64_2PART_C(0xe45c10c4, 2a2b3b06), -210, -44},\n {UINT64_2PART_C(0xaa242499, 697392d3), -183, -36},\n {UINT64_2PART_C(0xfd87b5f2, 8300ca0e), -157, -28},\n {UINT64_2PART_C(0xbce50864, 92111aeb), -130, -20},\n {UINT64_2PART_C(0x8cbccc09, 6f5088cc), -103, -12},\n {UINT64_2PART_C(0xd1b71758, e219652c), -77, -4},\n {UINT64_2PART_C(0x9c400000, 00000000), -50, 4},\n {UINT64_2PART_C(0xe8d4a510, 00000000), -24, 12},\n {UINT64_2PART_C(0xad78ebc5, ac620000), 3, 20},\n {UINT64_2PART_C(0x813f3978, f8940984), 30, 28},\n {UINT64_2PART_C(0xc097ce7b, c90715b3), 56, 36},\n {UINT64_2PART_C(0x8f7e32ce, 7bea5c70), 83, 44},\n {UINT64_2PART_C(0xd5d238a4, abe98068), 109, 52},\n {UINT64_2PART_C(0x9f4f2726, 179a2245), 136, 60},\n {UINT64_2PART_C(0xed63a231, d4c4fb27), 162, 68},\n {UINT64_2PART_C(0xb0de6538, 8cc8ada8), 189, 76},\n {UINT64_2PART_C(0x83c7088e, 1aab65db), 216, 84},\n {UINT64_2PART_C(0xc45d1df9, 42711d9a), 242, 92},\n {UINT64_2PART_C(0x924d692c, a61be758), 269, 100},\n {UINT64_2PART_C(0xda01ee64, 1a708dea), 295, 108},\n {UINT64_2PART_C(0xa26da399, 9aef774a), 322, 116},\n {UINT64_2PART_C(0xf209787b, b47d6b85), 348, 124},\n {UINT64_2PART_C(0xb454e4a1, 79dd1877), 375, 132},\n {UINT64_2PART_C(0x865b8692, 5b9bc5c2), 402, 140},\n {UINT64_2PART_C(0xc83553c5, c8965d3d), 428, 148},\n {UINT64_2PART_C(0x952ab45c, fa97a0b3), 455, 156},\n {UINT64_2PART_C(0xde469fbd, 99a05fe3), 481, 164},\n {UINT64_2PART_C(0xa59bc234, db398c25), 508, 172},\n {UINT64_2PART_C(0xf6c69a72, a3989f5c), 534, 180},\n {UINT64_2PART_C(0xb7dcbf53, 54e9bece), 561, 188},\n {UINT64_2PART_C(0x88fcf317, f22241e2), 588, 196},\n {UINT64_2PART_C(0xcc20ce9b, d35c78a5), 614, 204},\n {UINT64_2PART_C(0x98165af3, 7b2153df), 641, 212},\n {UINT64_2PART_C(0xe2a0b5dc, 971f303a), 667, 220},\n {UINT64_2PART_C(0xa8d9d153, 5ce3b396), 694, 228},\n {UINT64_2PART_C(0xfb9b7cd9, a4a7443c), 720, 236},\n {UINT64_2PART_C(0xbb764c4c, a7a44410), 747, 244},\n {UINT64_2PART_C(0x8bab8eef, b6409c1a), 774, 252},\n {UINT64_2PART_C(0xd01fef10, a657842c), 800, 260},\n {UINT64_2PART_C(0x9b10a4e5, e9913129), 827, 268},\n {UINT64_2PART_C(0xe7109bfb, a19c0c9d), 853, 276},\n {UINT64_2PART_C(0xac2820d9, 623bf429), 880, 284},\n {UINT64_2PART_C(0x80444b5e, 7aa7cf85), 907, 292},\n {UINT64_2PART_C(0xbf21e440, 03acdd2d), 933, 300},\n {UINT64_2PART_C(0x8e679c2f, 5e44ff8f), 960, 308},\n {UINT64_2PART_C(0xd433179d, 9c8cb841), 986, 316},\n {UINT64_2PART_C(0x9e19db92, b4e31ba9), 1013, 324},\n {UINT64_2PART_C(0xeb96bf6e, badf77d9), 1039, 332},\n {UINT64_2PART_C(0xaf87023b, 9bf0ee6b), 1066, 340},\n};\n\nstatic const int kCachedPowersLength = ARRAY_SIZE(kCachedPowers);\nstatic const int kCachedPowersOffset = 348; // -1 * the first decimal_exponent.\nstatic const double kD_1_LOG2_10 = 0.30102999566398114; // 1 / lg(10)\n// Difference between the decimal exponents in the table above.\nconst int PowersOfTenCache::kDecimalExponentDistance = 8;\nconst int PowersOfTenCache::kMinDecimalExponent = -348;\nconst int PowersOfTenCache::kMaxDecimalExponent = 340;\n\nvoid PowersOfTenCache::GetCachedPowerForBinaryExponentRange(\n int min_exponent,\n int max_exponent,\n DiyFp* power,\n int* decimal_exponent) {\n int kQ = DiyFp::kSignificandSize;\n double k = ceil((min_exponent + kQ - 1) * kD_1_LOG2_10);\n int foo = kCachedPowersOffset;\n int index =\n (foo + static_cast(k) - 1) / kDecimalExponentDistance + 1;\n ASSERT(0 <= index && index < kCachedPowersLength);\n CachedPower cached_power = kCachedPowers[index];\n ASSERT(min_exponent <= cached_power.binary_exponent);\n ASSERT(cached_power.binary_exponent <= max_exponent);\n *decimal_exponent = cached_power.decimal_exponent;\n *power = DiyFp(cached_power.significand, cached_power.binary_exponent);\n}\n\n\nvoid PowersOfTenCache::GetCachedPowerForDecimalExponent(int requested_exponent,\n DiyFp* power,\n int* found_exponent) {\n ASSERT(kMinDecimalExponent <= requested_exponent);\n ASSERT(requested_exponent < kMaxDecimalExponent + kDecimalExponentDistance);\n int index =\n (requested_exponent + kCachedPowersOffset) / kDecimalExponentDistance;\n CachedPower cached_power = kCachedPowers[index];\n *power = DiyFp(cached_power.significand, cached_power.binary_exponent);\n *found_exponent = cached_power.decimal_exponent;\n ASSERT(*found_exponent <= requested_exponent);\n ASSERT(requested_exponent < *found_exponent + kDecimalExponentDistance);\n}\n\n} // namespace double_conversion\n" }, { "path": "src/kenlm/util/double-conversion/cached-powers.h", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#ifndef DOUBLE_CONVERSION_CACHED_POWERS_H_\n#define DOUBLE_CONVERSION_CACHED_POWERS_H_\n\n#include \"diy-fp.h\"\n\nnamespace double_conversion {\n\nclass PowersOfTenCache {\n public:\n\n // Not all powers of ten are cached. The decimal exponent of two neighboring\n // cached numbers will differ by kDecimalExponentDistance.\n static const int kDecimalExponentDistance;\n\n static const int kMinDecimalExponent;\n static const int kMaxDecimalExponent;\n\n // Returns a cached power-of-ten with a binary exponent in the range\n // [min_exponent; max_exponent] (boundaries included).\n static void GetCachedPowerForBinaryExponentRange(int min_exponent,\n int max_exponent,\n DiyFp* power,\n int* decimal_exponent);\n\n // Returns a cached power of ten x ~= 10^k such that\n // k <= decimal_exponent < k + kCachedPowersDecimalDistance.\n // The given decimal_exponent must satisfy\n // kMinDecimalExponent <= requested_exponent, and\n // requested_exponent < kMaxDecimalExponent + kDecimalExponentDistance.\n static void GetCachedPowerForDecimalExponent(int requested_exponent,\n DiyFp* power,\n int* found_exponent);\n};\n\n} // namespace double_conversion\n\n#endif // DOUBLE_CONVERSION_CACHED_POWERS_H_\n" }, { "path": "src/kenlm/util/double-conversion/diy-fp.cc", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n\n#include \"diy-fp.h\"\n#include \"utils.h\"\n\nnamespace double_conversion {\n\nvoid DiyFp::Multiply(const DiyFp& other) {\n // Simply \"emulates\" a 128 bit multiplication.\n // However: the resulting number only contains 64 bits. The least\n // significant 64 bits are only used for rounding the most significant 64\n // bits.\n const uint64_t kM32 = 0xFFFFFFFFU;\n uint64_t a = f_ >> 32;\n uint64_t b = f_ & kM32;\n uint64_t c = other.f_ >> 32;\n uint64_t d = other.f_ & kM32;\n uint64_t ac = a * c;\n uint64_t bc = b * c;\n uint64_t ad = a * d;\n uint64_t bd = b * d;\n uint64_t tmp = (bd >> 32) + (ad & kM32) + (bc & kM32);\n // By adding 1U << 31 to tmp we round the final result.\n // Halfway cases will be round up.\n tmp += 1U << 31;\n uint64_t result_f = ac + (ad >> 32) + (bc >> 32) + (tmp >> 32);\n e_ += other.e_ + 64;\n f_ = result_f;\n}\n\n} // namespace double_conversion\n" }, { "path": "src/kenlm/util/double-conversion/diy-fp.h", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#ifndef DOUBLE_CONVERSION_DIY_FP_H_\n#define DOUBLE_CONVERSION_DIY_FP_H_\n\n#include \"utils.h\"\n\nnamespace double_conversion {\n\n// This \"Do It Yourself Floating Point\" class implements a floating-point number\n// with a uint64 significand and an int exponent. Normalized DiyFp numbers will\n// have the most significant bit of the significand set.\n// Multiplication and Subtraction do not normalize their results.\n// DiyFp are not designed to contain special doubles (NaN and Infinity).\nclass DiyFp {\n public:\n static const int kSignificandSize = 64;\n\n DiyFp() : f_(0), e_(0) {}\n DiyFp(uint64_t f, int e) : f_(f), e_(e) {}\n\n // this = this - other.\n // The exponents of both numbers must be the same and the significand of this\n // must be bigger than the significand of other.\n // The result will not be normalized.\n void Subtract(const DiyFp& other) {\n ASSERT(e_ == other.e_);\n ASSERT(f_ >= other.f_);\n f_ -= other.f_;\n }\n\n // Returns a - b.\n // The exponents of both numbers must be the same and this must be bigger\n // than other. The result will not be normalized.\n static DiyFp Minus(const DiyFp& a, const DiyFp& b) {\n DiyFp result = a;\n result.Subtract(b);\n return result;\n }\n\n\n // this = this * other.\n void Multiply(const DiyFp& other);\n\n // returns a * b;\n static DiyFp Times(const DiyFp& a, const DiyFp& b) {\n DiyFp result = a;\n result.Multiply(b);\n return result;\n }\n\n void Normalize() {\n ASSERT(f_ != 0);\n uint64_t f = f_;\n int e = e_;\n\n // This method is mainly called for normalizing boundaries. In general\n // boundaries need to be shifted by 10 bits. We thus optimize for this case.\n const uint64_t k10MSBits = UINT64_2PART_C(0xFFC00000, 00000000);\n while ((f & k10MSBits) == 0) {\n f <<= 10;\n e -= 10;\n }\n while ((f & kUint64MSB) == 0) {\n f <<= 1;\n e--;\n }\n f_ = f;\n e_ = e;\n }\n\n static DiyFp Normalize(const DiyFp& a) {\n DiyFp result = a;\n result.Normalize();\n return result;\n }\n\n uint64_t f() const { return f_; }\n int e() const { return e_; }\n\n void set_f(uint64_t new_value) { f_ = new_value; }\n void set_e(int new_value) { e_ = new_value; }\n\n private:\n static const uint64_t kUint64MSB = UINT64_2PART_C(0x80000000, 00000000);\n\n uint64_t f_;\n int e_;\n};\n\n} // namespace double_conversion\n\n#endif // DOUBLE_CONVERSION_DIY_FP_H_\n" }, { "path": "src/kenlm/util/double-conversion/double-conversion.cc", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#include \n#include \n\n#include \"double-conversion.h\"\n\n#include \"bignum-dtoa.h\"\n#include \"fast-dtoa.h\"\n#include \"fixed-dtoa.h\"\n#include \"ieee.h\"\n#include \"strtod.h\"\n#include \"utils.h\"\n\nnamespace double_conversion {\n\nconst DoubleToStringConverter& DoubleToStringConverter::EcmaScriptConverter() {\n int flags = UNIQUE_ZERO | EMIT_POSITIVE_EXPONENT_SIGN;\n static DoubleToStringConverter converter(flags,\n \"Infinity\",\n \"NaN\",\n 'e',\n -6, 21,\n 6, 0);\n return converter;\n}\n\n\nbool DoubleToStringConverter::HandleSpecialValues(\n double value,\n StringBuilder* result_builder) const {\n Double double_inspect(value);\n if (double_inspect.IsInfinite()) {\n if (infinity_symbol_ == NULL) return false;\n if (value < 0) {\n result_builder->AddCharacter('-');\n }\n result_builder->AddString(infinity_symbol_);\n return true;\n }\n if (double_inspect.IsNan()) {\n if (nan_symbol_ == NULL) return false;\n result_builder->AddString(nan_symbol_);\n return true;\n }\n return false;\n}\n\n\nvoid DoubleToStringConverter::CreateExponentialRepresentation(\n const char* decimal_digits,\n int length,\n int exponent,\n StringBuilder* result_builder) const {\n ASSERT(length != 0);\n result_builder->AddCharacter(decimal_digits[0]);\n if (length != 1) {\n result_builder->AddCharacter('.');\n result_builder->AddSubstring(&decimal_digits[1], length-1);\n }\n result_builder->AddCharacter(exponent_character_);\n if (exponent < 0) {\n result_builder->AddCharacter('-');\n exponent = -exponent;\n } else {\n if ((flags_ & EMIT_POSITIVE_EXPONENT_SIGN) != 0) {\n result_builder->AddCharacter('+');\n }\n }\n if (exponent == 0) {\n result_builder->AddCharacter('0');\n return;\n }\n ASSERT(exponent < 1e4);\n const int kMaxExponentLength = 5;\n char buffer[kMaxExponentLength + 1];\n buffer[kMaxExponentLength] = '\\0';\n int first_char_pos = kMaxExponentLength;\n while (exponent > 0) {\n buffer[--first_char_pos] = '0' + (exponent % 10);\n exponent /= 10;\n }\n result_builder->AddSubstring(&buffer[first_char_pos],\n kMaxExponentLength - first_char_pos);\n}\n\n\nvoid DoubleToStringConverter::CreateDecimalRepresentation(\n const char* decimal_digits,\n int length,\n int decimal_point,\n int digits_after_point,\n StringBuilder* result_builder) const {\n // Create a representation that is padded with zeros if needed.\n if (decimal_point <= 0) {\n // \"0.00000decimal_rep\".\n result_builder->AddCharacter('0');\n if (digits_after_point > 0) {\n result_builder->AddCharacter('.');\n result_builder->AddPadding('0', -decimal_point);\n ASSERT(length <= digits_after_point - (-decimal_point));\n result_builder->AddSubstring(decimal_digits, length);\n int remaining_digits = digits_after_point - (-decimal_point) - length;\n result_builder->AddPadding('0', remaining_digits);\n }\n } else if (decimal_point >= length) {\n // \"decimal_rep0000.00000\" or \"decimal_rep.0000\"\n result_builder->AddSubstring(decimal_digits, length);\n result_builder->AddPadding('0', decimal_point - length);\n if (digits_after_point > 0) {\n result_builder->AddCharacter('.');\n result_builder->AddPadding('0', digits_after_point);\n }\n } else {\n // \"decima.l_rep000\"\n ASSERT(digits_after_point > 0);\n result_builder->AddSubstring(decimal_digits, decimal_point);\n result_builder->AddCharacter('.');\n ASSERT(length - decimal_point <= digits_after_point);\n result_builder->AddSubstring(&decimal_digits[decimal_point],\n length - decimal_point);\n int remaining_digits = digits_after_point - (length - decimal_point);\n result_builder->AddPadding('0', remaining_digits);\n }\n if (digits_after_point == 0) {\n if ((flags_ & EMIT_TRAILING_DECIMAL_POINT) != 0) {\n result_builder->AddCharacter('.');\n }\n if ((flags_ & EMIT_TRAILING_ZERO_AFTER_POINT) != 0) {\n result_builder->AddCharacter('0');\n }\n }\n}\n\n\nbool DoubleToStringConverter::ToShortestIeeeNumber(\n double value,\n StringBuilder* result_builder,\n DoubleToStringConverter::DtoaMode mode) const {\n ASSERT(mode == SHORTEST || mode == SHORTEST_SINGLE);\n if (Double(value).IsSpecial()) {\n return HandleSpecialValues(value, result_builder);\n }\n\n int decimal_point;\n bool sign;\n const int kDecimalRepCapacity = kBase10MaximalLength + 1;\n char decimal_rep[kDecimalRepCapacity];\n int decimal_rep_length;\n\n DoubleToAscii(value, mode, 0, decimal_rep, kDecimalRepCapacity,\n &sign, &decimal_rep_length, &decimal_point);\n\n bool unique_zero = (flags_ & UNIQUE_ZERO) != 0;\n if (sign && (value != 0.0 || !unique_zero)) {\n result_builder->AddCharacter('-');\n }\n\n int exponent = decimal_point - 1;\n if ((decimal_in_shortest_low_ <= exponent) &&\n (exponent < decimal_in_shortest_high_)) {\n CreateDecimalRepresentation(decimal_rep, decimal_rep_length,\n decimal_point,\n Max(0, decimal_rep_length - decimal_point),\n result_builder);\n } else {\n CreateExponentialRepresentation(decimal_rep, decimal_rep_length, exponent,\n result_builder);\n }\n return true;\n}\n\n\nbool DoubleToStringConverter::ToFixed(double value,\n int requested_digits,\n StringBuilder* result_builder) const {\n ASSERT(kMaxFixedDigitsBeforePoint == 60);\n const double kFirstNonFixed = 1e60;\n\n if (Double(value).IsSpecial()) {\n return HandleSpecialValues(value, result_builder);\n }\n\n if (requested_digits > kMaxFixedDigitsAfterPoint) return false;\n if (value >= kFirstNonFixed || value <= -kFirstNonFixed) return false;\n\n // Find a sufficiently precise decimal representation of n.\n int decimal_point;\n bool sign;\n // Add space for the '\\0' byte.\n const int kDecimalRepCapacity =\n kMaxFixedDigitsBeforePoint + kMaxFixedDigitsAfterPoint + 1;\n char decimal_rep[kDecimalRepCapacity];\n int decimal_rep_length;\n DoubleToAscii(value, FIXED, requested_digits,\n decimal_rep, kDecimalRepCapacity,\n &sign, &decimal_rep_length, &decimal_point);\n\n bool unique_zero = ((flags_ & UNIQUE_ZERO) != 0);\n if (sign && (value != 0.0 || !unique_zero)) {\n result_builder->AddCharacter('-');\n }\n\n CreateDecimalRepresentation(decimal_rep, decimal_rep_length, decimal_point,\n requested_digits, result_builder);\n return true;\n}\n\n\nbool DoubleToStringConverter::ToExponential(\n double value,\n int requested_digits,\n StringBuilder* result_builder) const {\n if (Double(value).IsSpecial()) {\n return HandleSpecialValues(value, result_builder);\n }\n\n if (requested_digits < -1) return false;\n if (requested_digits > kMaxExponentialDigits) return false;\n\n int decimal_point;\n bool sign;\n // Add space for digit before the decimal point and the '\\0' character.\n const int kDecimalRepCapacity = kMaxExponentialDigits + 2;\n ASSERT(kDecimalRepCapacity > kBase10MaximalLength);\n char decimal_rep[kDecimalRepCapacity];\n int decimal_rep_length;\n\n if (requested_digits == -1) {\n DoubleToAscii(value, SHORTEST, 0,\n decimal_rep, kDecimalRepCapacity,\n &sign, &decimal_rep_length, &decimal_point);\n } else {\n DoubleToAscii(value, PRECISION, requested_digits + 1,\n decimal_rep, kDecimalRepCapacity,\n &sign, &decimal_rep_length, &decimal_point);\n ASSERT(decimal_rep_length <= requested_digits + 1);\n\n for (int i = decimal_rep_length; i < requested_digits + 1; ++i) {\n decimal_rep[i] = '0';\n }\n decimal_rep_length = requested_digits + 1;\n }\n\n bool unique_zero = ((flags_ & UNIQUE_ZERO) != 0);\n if (sign && (value != 0.0 || !unique_zero)) {\n result_builder->AddCharacter('-');\n }\n\n int exponent = decimal_point - 1;\n CreateExponentialRepresentation(decimal_rep,\n decimal_rep_length,\n exponent,\n result_builder);\n return true;\n}\n\n\nbool DoubleToStringConverter::ToPrecision(double value,\n int precision,\n StringBuilder* result_builder) const {\n if (Double(value).IsSpecial()) {\n return HandleSpecialValues(value, result_builder);\n }\n\n if (precision < kMinPrecisionDigits || precision > kMaxPrecisionDigits) {\n return false;\n }\n\n // Find a sufficiently precise decimal representation of n.\n int decimal_point;\n bool sign;\n // Add one for the terminating null character.\n const int kDecimalRepCapacity = kMaxPrecisionDigits + 1;\n char decimal_rep[kDecimalRepCapacity];\n int decimal_rep_length;\n\n DoubleToAscii(value, PRECISION, precision,\n decimal_rep, kDecimalRepCapacity,\n &sign, &decimal_rep_length, &decimal_point);\n ASSERT(decimal_rep_length <= precision);\n\n bool unique_zero = ((flags_ & UNIQUE_ZERO) != 0);\n if (sign && (value != 0.0 || !unique_zero)) {\n result_builder->AddCharacter('-');\n }\n\n // The exponent if we print the number as x.xxeyyy. That is with the\n // decimal point after the first digit.\n int exponent = decimal_point - 1;\n\n int extra_zero = ((flags_ & EMIT_TRAILING_ZERO_AFTER_POINT) != 0) ? 1 : 0;\n if ((-decimal_point + 1 > max_leading_padding_zeroes_in_precision_mode_) ||\n (decimal_point - precision + extra_zero >\n max_trailing_padding_zeroes_in_precision_mode_)) {\n // Fill buffer to contain 'precision' digits.\n // Usually the buffer is already at the correct length, but 'DoubleToAscii'\n // is allowed to return less characters.\n for (int i = decimal_rep_length; i < precision; ++i) {\n decimal_rep[i] = '0';\n }\n\n CreateExponentialRepresentation(decimal_rep,\n precision,\n exponent,\n result_builder);\n } else {\n CreateDecimalRepresentation(decimal_rep, decimal_rep_length, decimal_point,\n Max(0, precision - decimal_point),\n result_builder);\n }\n return true;\n}\n\n\nstatic BignumDtoaMode DtoaToBignumDtoaMode(\n DoubleToStringConverter::DtoaMode dtoa_mode) {\n switch (dtoa_mode) {\n case DoubleToStringConverter::SHORTEST: return BIGNUM_DTOA_SHORTEST;\n case DoubleToStringConverter::SHORTEST_SINGLE:\n return BIGNUM_DTOA_SHORTEST_SINGLE;\n case DoubleToStringConverter::FIXED: return BIGNUM_DTOA_FIXED;\n case DoubleToStringConverter::PRECISION: return BIGNUM_DTOA_PRECISION;\n default:\n UNREACHABLE();\n return BIGNUM_DTOA_SHORTEST; // To silence compiler.\n }\n}\n\n\nvoid DoubleToStringConverter::DoubleToAscii(double v,\n DtoaMode mode,\n int requested_digits,\n char* buffer,\n int buffer_length,\n bool* sign,\n int* length,\n int* point) {\n Vector vector(buffer, buffer_length);\n ASSERT(!Double(v).IsSpecial());\n ASSERT(mode == SHORTEST || mode == SHORTEST_SINGLE || requested_digits >= 0);\n\n if (Double(v).Sign() < 0) {\n *sign = true;\n v = -v;\n } else {\n *sign = false;\n }\n\n if (mode == PRECISION && requested_digits == 0) {\n vector[0] = '\\0';\n *length = 0;\n return;\n }\n\n if (v == 0) {\n vector[0] = '0';\n vector[1] = '\\0';\n *length = 1;\n *point = 1;\n return;\n }\n\n bool fast_worked;\n switch (mode) {\n case SHORTEST:\n fast_worked = FastDtoa(v, FAST_DTOA_SHORTEST, 0, vector, length, point);\n break;\n case SHORTEST_SINGLE:\n fast_worked = FastDtoa(v, FAST_DTOA_SHORTEST_SINGLE, 0,\n vector, length, point);\n break;\n case FIXED:\n fast_worked = FastFixedDtoa(v, requested_digits, vector, length, point);\n break;\n case PRECISION:\n fast_worked = FastDtoa(v, FAST_DTOA_PRECISION, requested_digits,\n vector, length, point);\n break;\n default:\n UNREACHABLE();\n fast_worked = false;\n }\n if (fast_worked) return;\n\n // If the fast dtoa didn't succeed use the slower bignum version.\n BignumDtoaMode bignum_mode = DtoaToBignumDtoaMode(mode);\n BignumDtoa(v, bignum_mode, requested_digits, vector, length, point);\n vector[*length] = '\\0';\n}\n\n\n// Consumes the given substring from the iterator.\n// Returns false, if the substring does not match.\nstatic bool ConsumeSubString(const char** current,\n const char* end,\n const char* substring) {\n ASSERT(**current == *substring);\n for (substring++; *substring != '\\0'; substring++) {\n ++*current;\n if (*current == end || **current != *substring) return false;\n }\n ++*current;\n return true;\n}\n\n\n// Maximum number of significant digits in decimal representation.\n// The longest possible double in decimal representation is\n// (2^53 - 1) * 2 ^ -1074 that is (2 ^ 53 - 1) * 5 ^ 1074 / 10 ^ 1074\n// (768 digits). If we parse a number whose first digits are equal to a\n// mean of 2 adjacent doubles (that could have up to 769 digits) the result\n// must be rounded to the bigger one unless the tail consists of zeros, so\n// we don't need to preserve all the digits.\nconst int kMaxSignificantDigits = 772;\n\n\n// Returns true if a nonspace found and false if the end has reached.\nstatic inline bool AdvanceToNonspace(const char** current, const char* end) {\n while (*current != end) {\n if (**current != ' ') return true;\n ++*current;\n }\n return false;\n}\n\n\nstatic bool isDigit(int x, int radix) {\n return (x >= '0' && x <= '9' && x < '0' + radix)\n || (radix > 10 && x >= 'a' && x < 'a' + radix - 10)\n || (radix > 10 && x >= 'A' && x < 'A' + radix - 10);\n}\n\n\nstatic double SignedZero(bool sign) {\n return sign ? -0.0 : 0.0;\n}\n\n\n// Parsing integers with radix 2, 4, 8, 16, 32. Assumes current != end.\ntemplate \nstatic double RadixStringToIeee(const char* current,\n const char* end,\n bool sign,\n bool allow_trailing_junk,\n double junk_string_value,\n bool read_as_double,\n const char** trailing_pointer) {\n ASSERT(current != end);\n\n const int kDoubleSize = Double::kSignificandSize;\n const int kSingleSize = Single::kSignificandSize;\n const int kSignificandSize = read_as_double? kDoubleSize: kSingleSize;\n\n // Skip leading 0s.\n while (*current == '0') {\n ++current;\n if (current == end) {\n *trailing_pointer = end;\n return SignedZero(sign);\n }\n }\n\n int64_t number = 0;\n int exponent = 0;\n const int radix = (1 << radix_log_2);\n\n do {\n int digit;\n if (*current >= '0' && *current <= '9' && *current < '0' + radix) {\n digit = static_cast(*current) - '0';\n } else if (radix > 10 && *current >= 'a' && *current < 'a' + radix - 10) {\n digit = static_cast(*current) - 'a' + 10;\n } else if (radix > 10 && *current >= 'A' && *current < 'A' + radix - 10) {\n digit = static_cast(*current) - 'A' + 10;\n } else {\n if (allow_trailing_junk || !AdvanceToNonspace(¤t, end)) {\n break;\n } else {\n return junk_string_value;\n }\n }\n\n number = number * radix + digit;\n int overflow = static_cast(number >> kSignificandSize);\n if (overflow != 0) {\n // Overflow occurred. Need to determine which direction to round the\n // result.\n int overflow_bits_count = 1;\n while (overflow > 1) {\n overflow_bits_count++;\n overflow >>= 1;\n }\n\n int dropped_bits_mask = ((1 << overflow_bits_count) - 1);\n int dropped_bits = static_cast(number) & dropped_bits_mask;\n number >>= overflow_bits_count;\n exponent = overflow_bits_count;\n\n bool zero_tail = true;\n while (true) {\n ++current;\n if (current == end || !isDigit(*current, radix)) break;\n zero_tail = zero_tail && *current == '0';\n exponent += radix_log_2;\n }\n\n if (!allow_trailing_junk && AdvanceToNonspace(¤t, end)) {\n return junk_string_value;\n }\n\n int middle_value = (1 << (overflow_bits_count - 1));\n if (dropped_bits > middle_value) {\n number++; // Rounding up.\n } else if (dropped_bits == middle_value) {\n // Rounding to even to consistency with decimals: half-way case rounds\n // up if significant part is odd and down otherwise.\n if ((number & 1) != 0 || !zero_tail) {\n number++; // Rounding up.\n }\n }\n\n // Rounding up may cause overflow.\n if ((number & ((int64_t)1 << kSignificandSize)) != 0) {\n exponent++;\n number >>= 1;\n }\n break;\n }\n ++current;\n } while (current != end);\n\n ASSERT(number < ((int64_t)1 << kSignificandSize));\n ASSERT(static_cast(static_cast(number)) == number);\n\n *trailing_pointer = current;\n\n if (exponent == 0) {\n if (sign) {\n if (number == 0) return -0.0;\n number = -number;\n }\n return static_cast(number);\n }\n\n ASSERT(number != 0);\n return Double(DiyFp(number, exponent)).value();\n}\n\n\ndouble StringToDoubleConverter::StringToIeee(\n const char* input,\n int length,\n int* processed_characters_count,\n bool read_as_double) const {\n const char* current = input;\n const char* end = input + length;\n\n *processed_characters_count = 0;\n\n const bool allow_trailing_junk = (flags_ & ALLOW_TRAILING_JUNK) != 0;\n const bool allow_leading_spaces = (flags_ & ALLOW_LEADING_SPACES) != 0;\n const bool allow_trailing_spaces = (flags_ & ALLOW_TRAILING_SPACES) != 0;\n const bool allow_spaces_after_sign = (flags_ & ALLOW_SPACES_AFTER_SIGN) != 0;\n\n // To make sure that iterator dereferencing is valid the following\n // convention is used:\n // 1. Each '++current' statement is followed by check for equality to 'end'.\n // 2. If AdvanceToNonspace returned false then current == end.\n // 3. If 'current' becomes equal to 'end' the function returns or goes to\n // 'parsing_done'.\n // 4. 'current' is not dereferenced after the 'parsing_done' label.\n // 5. Code before 'parsing_done' may rely on 'current != end'.\n if (current == end) return empty_string_value_;\n\n if (allow_leading_spaces || allow_trailing_spaces) {\n if (!AdvanceToNonspace(¤t, end)) {\n *processed_characters_count = current - input;\n return empty_string_value_;\n }\n if (!allow_leading_spaces && (input != current)) {\n // No leading spaces allowed, but AdvanceToNonspace moved forward.\n return junk_string_value_;\n }\n }\n\n // The longest form of simplified number is: \"-.1eXXX\\0\".\n const int kBufferSize = kMaxSignificantDigits + 10;\n char buffer[kBufferSize]; // NOLINT: size is known at compile time.\n int buffer_pos = 0;\n\n // Exponent will be adjusted if insignificant digits of the integer part\n // or insignificant leading zeros of the fractional part are dropped.\n int exponent = 0;\n int significant_digits = 0;\n int insignificant_digits = 0;\n bool nonzero_digit_dropped = false;\n\n bool sign = false;\n\n if (*current == '+' || *current == '-') {\n sign = (*current == '-');\n ++current;\n const char* next_non_space = current;\n // Skip following spaces (if allowed).\n if (!AdvanceToNonspace(&next_non_space, end)) return junk_string_value_;\n if (!allow_spaces_after_sign && (current != next_non_space)) {\n return junk_string_value_;\n }\n current = next_non_space;\n }\n\n if (infinity_symbol_ != NULL) {\n if (*current == infinity_symbol_[0]) {\n if (!ConsumeSubString(¤t, end, infinity_symbol_)) {\n return junk_string_value_;\n }\n\n if (!(allow_trailing_spaces || allow_trailing_junk) && (current != end)) {\n return junk_string_value_;\n }\n if (!allow_trailing_junk && AdvanceToNonspace(¤t, end)) {\n return junk_string_value_;\n }\n\n ASSERT(buffer_pos == 0);\n *processed_characters_count = current - input;\n return sign ? -Double::Infinity() : Double::Infinity();\n }\n }\n\n if (nan_symbol_ != NULL) {\n if (*current == nan_symbol_[0]) {\n if (!ConsumeSubString(¤t, end, nan_symbol_)) {\n return junk_string_value_;\n }\n\n if (!(allow_trailing_spaces || allow_trailing_junk) && (current != end)) {\n return junk_string_value_;\n }\n if (!allow_trailing_junk && AdvanceToNonspace(¤t, end)) {\n return junk_string_value_;\n }\n\n ASSERT(buffer_pos == 0);\n *processed_characters_count = current - input;\n return sign ? -Double::NaN() : Double::NaN();\n }\n }\n\n bool leading_zero = false;\n if (*current == '0') {\n ++current;\n if (current == end) {\n *processed_characters_count = current - input;\n return SignedZero(sign);\n }\n\n leading_zero = true;\n\n // It could be hexadecimal value.\n if ((flags_ & ALLOW_HEX) && (*current == 'x' || *current == 'X')) {\n ++current;\n if (current == end || !isDigit(*current, 16)) {\n return junk_string_value_; // \"0x\".\n }\n\n const char* tail_pointer = NULL;\n double result = RadixStringToIeee<4>(current,\n end,\n sign,\n allow_trailing_junk,\n junk_string_value_,\n read_as_double,\n &tail_pointer);\n if (tail_pointer != NULL) {\n if (allow_trailing_spaces) AdvanceToNonspace(&tail_pointer, end);\n *processed_characters_count = tail_pointer - input;\n }\n return result;\n }\n\n // Ignore leading zeros in the integer part.\n while (*current == '0') {\n ++current;\n if (current == end) {\n *processed_characters_count = current - input;\n return SignedZero(sign);\n }\n }\n }\n\n bool octal = leading_zero && (flags_ & ALLOW_OCTALS) != 0;\n\n // Copy significant digits of the integer part (if any) to the buffer.\n while (*current >= '0' && *current <= '9') {\n if (significant_digits < kMaxSignificantDigits) {\n ASSERT(buffer_pos < kBufferSize);\n buffer[buffer_pos++] = static_cast(*current);\n significant_digits++;\n // Will later check if it's an octal in the buffer.\n } else {\n insignificant_digits++; // Move the digit into the exponential part.\n nonzero_digit_dropped = nonzero_digit_dropped || *current != '0';\n }\n octal = octal && *current < '8';\n ++current;\n if (current == end) goto parsing_done;\n }\n\n if (significant_digits == 0) {\n octal = false;\n }\n\n if (*current == '.') {\n if (octal && !allow_trailing_junk) return junk_string_value_;\n if (octal) goto parsing_done;\n\n ++current;\n if (current == end) {\n if (significant_digits == 0 && !leading_zero) {\n return junk_string_value_;\n } else {\n goto parsing_done;\n }\n }\n\n if (significant_digits == 0) {\n // octal = false;\n // Integer part consists of 0 or is absent. Significant digits start after\n // leading zeros (if any).\n while (*current == '0') {\n ++current;\n if (current == end) {\n *processed_characters_count = current - input;\n return SignedZero(sign);\n }\n exponent--; // Move this 0 into the exponent.\n }\n }\n\n // There is a fractional part.\n // We don't emit a '.', but adjust the exponent instead.\n while (*current >= '0' && *current <= '9') {\n if (significant_digits < kMaxSignificantDigits) {\n ASSERT(buffer_pos < kBufferSize);\n buffer[buffer_pos++] = static_cast(*current);\n significant_digits++;\n exponent--;\n } else {\n // Ignore insignificant digits in the fractional part.\n nonzero_digit_dropped = nonzero_digit_dropped || *current != '0';\n }\n ++current;\n if (current == end) goto parsing_done;\n }\n }\n\n if (!leading_zero && exponent == 0 && significant_digits == 0) {\n // If leading_zeros is true then the string contains zeros.\n // If exponent < 0 then string was [+-]\\.0*...\n // If significant_digits != 0 the string is not equal to 0.\n // Otherwise there are no digits in the string.\n return junk_string_value_;\n }\n\n // Parse exponential part.\n if (*current == 'e' || *current == 'E') {\n if (octal && !allow_trailing_junk) return junk_string_value_;\n if (octal) goto parsing_done;\n ++current;\n if (current == end) {\n if (allow_trailing_junk) {\n goto parsing_done;\n } else {\n return junk_string_value_;\n }\n }\n char sign = '+';\n if (*current == '+' || *current == '-') {\n sign = static_cast(*current);\n ++current;\n if (current == end) {\n if (allow_trailing_junk) {\n goto parsing_done;\n } else {\n return junk_string_value_;\n }\n }\n }\n\n if (current == end || *current < '0' || *current > '9') {\n if (allow_trailing_junk) {\n goto parsing_done;\n } else {\n return junk_string_value_;\n }\n }\n\n const int max_exponent = INT_MAX / 2;\n ASSERT(-max_exponent / 2 <= exponent && exponent <= max_exponent / 2);\n int num = 0;\n do {\n // Check overflow.\n int digit = *current - '0';\n if (num >= max_exponent / 10\n && !(num == max_exponent / 10 && digit <= max_exponent % 10)) {\n num = max_exponent;\n } else {\n num = num * 10 + digit;\n }\n ++current;\n } while (current != end && *current >= '0' && *current <= '9');\n\n exponent += (sign == '-' ? -num : num);\n }\n\n if (!(allow_trailing_spaces || allow_trailing_junk) && (current != end)) {\n return junk_string_value_;\n }\n if (!allow_trailing_junk && AdvanceToNonspace(¤t, end)) {\n return junk_string_value_;\n }\n if (allow_trailing_spaces) {\n AdvanceToNonspace(¤t, end);\n }\n\n parsing_done:\n exponent += insignificant_digits;\n\n if (octal) {\n double result;\n const char* tail_pointer = NULL;\n result = RadixStringToIeee<3>(buffer,\n buffer + buffer_pos,\n sign,\n allow_trailing_junk,\n junk_string_value_,\n read_as_double,\n &tail_pointer);\n ASSERT(tail_pointer != NULL);\n *processed_characters_count = current - input;\n return result;\n }\n\n if (nonzero_digit_dropped) {\n buffer[buffer_pos++] = '1';\n exponent--;\n }\n\n ASSERT(buffer_pos < kBufferSize);\n buffer[buffer_pos] = '\\0';\n\n double converted;\n if (read_as_double) {\n converted = Strtod(Vector(buffer, buffer_pos), exponent);\n } else {\n converted = Strtof(Vector(buffer, buffer_pos), exponent);\n }\n *processed_characters_count = current - input;\n return sign? -converted: converted;\n}\n\n} // namespace double_conversion\n" }, { "path": "src/kenlm/util/double-conversion/double-conversion.h", "content": "// Copyright 2012 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#ifndef DOUBLE_CONVERSION_DOUBLE_CONVERSION_H_\n#define DOUBLE_CONVERSION_DOUBLE_CONVERSION_H_\n\n#include \"utils.h\"\n\nnamespace double_conversion {\n\nclass DoubleToStringConverter {\n public:\n // When calling ToFixed with a double > 10^kMaxFixedDigitsBeforePoint\n // or a requested_digits parameter > kMaxFixedDigitsAfterPoint then the\n // function returns false.\n static const int kMaxFixedDigitsBeforePoint = 60;\n static const int kMaxFixedDigitsAfterPoint = 60;\n\n // When calling ToExponential with a requested_digits\n // parameter > kMaxExponentialDigits then the function returns false.\n static const int kMaxExponentialDigits = 120;\n\n // When calling ToPrecision with a requested_digits\n // parameter < kMinPrecisionDigits or requested_digits > kMaxPrecisionDigits\n // then the function returns false.\n static const int kMinPrecisionDigits = 1;\n static const int kMaxPrecisionDigits = 120;\n\n enum Flags {\n NO_FLAGS = 0,\n EMIT_POSITIVE_EXPONENT_SIGN = 1,\n EMIT_TRAILING_DECIMAL_POINT = 2,\n EMIT_TRAILING_ZERO_AFTER_POINT = 4,\n UNIQUE_ZERO = 8\n };\n\n // Flags should be a bit-or combination of the possible Flags-enum.\n // - NO_FLAGS: no special flags.\n // - EMIT_POSITIVE_EXPONENT_SIGN: when the number is converted into exponent\n // form, emits a '+' for positive exponents. Example: 1.2e+2.\n // - EMIT_TRAILING_DECIMAL_POINT: when the input number is an integer and is\n // converted into decimal format then a trailing decimal point is appended.\n // Example: 2345.0 is converted to \"2345.\".\n // - EMIT_TRAILING_ZERO_AFTER_POINT: in addition to a trailing decimal point\n // emits a trailing '0'-character. This flag requires the\n // EXMIT_TRAILING_DECIMAL_POINT flag.\n // Example: 2345.0 is converted to \"2345.0\".\n // - UNIQUE_ZERO: \"-0.0\" is converted to \"0.0\".\n //\n // Infinity symbol and nan_symbol provide the string representation for these\n // special values. If the string is NULL and the special value is encountered\n // then the conversion functions return false.\n //\n // The exponent_character is used in exponential representations. It is\n // usually 'e' or 'E'.\n //\n // When converting to the shortest representation the converter will\n // represent input numbers in decimal format if they are in the interval\n // [10^decimal_in_shortest_low; 10^decimal_in_shortest_high[\n // (lower boundary included, greater boundary excluded).\n // Example: with decimal_in_shortest_low = -6 and\n // decimal_in_shortest_high = 21:\n // ToShortest(0.000001) -> \"0.000001\"\n // ToShortest(0.0000001) -> \"1e-7\"\n // ToShortest(111111111111111111111.0) -> \"111111111111111110000\"\n // ToShortest(100000000000000000000.0) -> \"100000000000000000000\"\n // ToShortest(1111111111111111111111.0) -> \"1.1111111111111111e+21\"\n //\n // When converting to precision mode the converter may add\n // max_leading_padding_zeroes before returning the number in exponential\n // format.\n // Example with max_leading_padding_zeroes_in_precision_mode = 6.\n // ToPrecision(0.0000012345, 2) -> \"0.0000012\"\n // ToPrecision(0.00000012345, 2) -> \"1.2e-7\"\n // Similarily the converter may add up to\n // max_trailing_padding_zeroes_in_precision_mode in precision mode to avoid\n // returning an exponential representation. A zero added by the\n // EMIT_TRAILING_ZERO_AFTER_POINT flag is counted for this limit.\n // Examples for max_trailing_padding_zeroes_in_precision_mode = 1:\n // ToPrecision(230.0, 2) -> \"230\"\n // ToPrecision(230.0, 2) -> \"230.\" with EMIT_TRAILING_DECIMAL_POINT.\n // ToPrecision(230.0, 2) -> \"2.3e2\" with EMIT_TRAILING_ZERO_AFTER_POINT.\n DoubleToStringConverter(int flags,\n const char* infinity_symbol,\n const char* nan_symbol,\n char exponent_character,\n int decimal_in_shortest_low,\n int decimal_in_shortest_high,\n int max_leading_padding_zeroes_in_precision_mode,\n int max_trailing_padding_zeroes_in_precision_mode)\n : flags_(flags),\n infinity_symbol_(infinity_symbol),\n nan_symbol_(nan_symbol),\n exponent_character_(exponent_character),\n decimal_in_shortest_low_(decimal_in_shortest_low),\n decimal_in_shortest_high_(decimal_in_shortest_high),\n max_leading_padding_zeroes_in_precision_mode_(\n max_leading_padding_zeroes_in_precision_mode),\n max_trailing_padding_zeroes_in_precision_mode_(\n max_trailing_padding_zeroes_in_precision_mode) {\n // When 'trailing zero after the point' is set, then 'trailing point'\n // must be set too.\n ASSERT(((flags & EMIT_TRAILING_DECIMAL_POINT) != 0) ||\n !((flags & EMIT_TRAILING_ZERO_AFTER_POINT) != 0));\n }\n\n // Returns a converter following the EcmaScript specification.\n static const DoubleToStringConverter& EcmaScriptConverter();\n\n // Computes the shortest string of digits that correctly represent the input\n // number. Depending on decimal_in_shortest_low and decimal_in_shortest_high\n // (see constructor) it then either returns a decimal representation, or an\n // exponential representation.\n // Example with decimal_in_shortest_low = -6,\n // decimal_in_shortest_high = 21,\n // EMIT_POSITIVE_EXPONENT_SIGN activated, and\n // EMIT_TRAILING_DECIMAL_POINT deactived:\n // ToShortest(0.000001) -> \"0.000001\"\n // ToShortest(0.0000001) -> \"1e-7\"\n // ToShortest(111111111111111111111.0) -> \"111111111111111110000\"\n // ToShortest(100000000000000000000.0) -> \"100000000000000000000\"\n // ToShortest(1111111111111111111111.0) -> \"1.1111111111111111e+21\"\n //\n // Note: the conversion may round the output if the returned string\n // is accurate enough to uniquely identify the input-number.\n // For example the most precise representation of the double 9e59 equals\n // \"899999999999999918767229449717619953810131273674690656206848\", but\n // the converter will return the shorter (but still correct) \"9e59\".\n //\n // Returns true if the conversion succeeds. The conversion always succeeds\n // except when the input value is special and no infinity_symbol or\n // nan_symbol has been given to the constructor.\n bool ToShortest(double value, StringBuilder* result_builder) const {\n return ToShortestIeeeNumber(value, result_builder, SHORTEST);\n }\n\n // Same as ToShortest, but for single-precision floats.\n bool ToShortestSingle(float value, StringBuilder* result_builder) const {\n return ToShortestIeeeNumber(value, result_builder, SHORTEST_SINGLE);\n }\n\n\n // Computes a decimal representation with a fixed number of digits after the\n // decimal point. The last emitted digit is rounded.\n //\n // Examples:\n // ToFixed(3.12, 1) -> \"3.1\"\n // ToFixed(3.1415, 3) -> \"3.142\"\n // ToFixed(1234.56789, 4) -> \"1234.5679\"\n // ToFixed(1.23, 5) -> \"1.23000\"\n // ToFixed(0.1, 4) -> \"0.1000\"\n // ToFixed(1e30, 2) -> \"1000000000000000019884624838656.00\"\n // ToFixed(0.1, 30) -> \"0.100000000000000005551115123126\"\n // ToFixed(0.1, 17) -> \"0.10000000000000001\"\n //\n // If requested_digits equals 0, then the tail of the result depends on\n // the EMIT_TRAILING_DECIMAL_POINT and EMIT_TRAILING_ZERO_AFTER_POINT.\n // Examples, for requested_digits == 0,\n // let EMIT_TRAILING_DECIMAL_POINT and EMIT_TRAILING_ZERO_AFTER_POINT be\n // - false and false: then 123.45 -> 123\n // 0.678 -> 1\n // - true and false: then 123.45 -> 123.\n // 0.678 -> 1.\n // - true and true: then 123.45 -> 123.0\n // 0.678 -> 1.0\n //\n // Returns true if the conversion succeeds. The conversion always succeeds\n // except for the following cases:\n // - the input value is special and no infinity_symbol or nan_symbol has\n // been provided to the constructor,\n // - 'value' > 10^kMaxFixedDigitsBeforePoint, or\n // - 'requested_digits' > kMaxFixedDigitsAfterPoint.\n // The last two conditions imply that the result will never contain more than\n // 1 + kMaxFixedDigitsBeforePoint + 1 + kMaxFixedDigitsAfterPoint characters\n // (one additional character for the sign, and one for the decimal point).\n bool ToFixed(double value,\n int requested_digits,\n StringBuilder* result_builder) const;\n\n // Computes a representation in exponential format with requested_digits\n // after the decimal point. The last emitted digit is rounded.\n // If requested_digits equals -1, then the shortest exponential representation\n // is computed.\n //\n // Examples with EMIT_POSITIVE_EXPONENT_SIGN deactivated, and\n // exponent_character set to 'e'.\n // ToExponential(3.12, 1) -> \"3.1e0\"\n // ToExponential(5.0, 3) -> \"5.000e0\"\n // ToExponential(0.001, 2) -> \"1.00e-3\"\n // ToExponential(3.1415, -1) -> \"3.1415e0\"\n // ToExponential(3.1415, 4) -> \"3.1415e0\"\n // ToExponential(3.1415, 3) -> \"3.142e0\"\n // ToExponential(123456789000000, 3) -> \"1.235e14\"\n // ToExponential(1000000000000000019884624838656.0, -1) -> \"1e30\"\n // ToExponential(1000000000000000019884624838656.0, 32) ->\n // \"1.00000000000000001988462483865600e30\"\n // ToExponential(1234, 0) -> \"1e3\"\n //\n // Returns true if the conversion succeeds. The conversion always succeeds\n // except for the following cases:\n // - the input value is special and no infinity_symbol or nan_symbol has\n // been provided to the constructor,\n // - 'requested_digits' > kMaxExponentialDigits.\n // The last condition implies that the result will never contain more than\n // kMaxExponentialDigits + 8 characters (the sign, the digit before the\n // decimal point, the decimal point, the exponent character, the\n // exponent's sign, and at most 3 exponent digits).\n bool ToExponential(double value,\n int requested_digits,\n StringBuilder* result_builder) const;\n\n // Computes 'precision' leading digits of the given 'value' and returns them\n // either in exponential or decimal format, depending on\n // max_{leading|trailing}_padding_zeroes_in_precision_mode (given to the\n // constructor).\n // The last computed digit is rounded.\n //\n // Example with max_leading_padding_zeroes_in_precision_mode = 6.\n // ToPrecision(0.0000012345, 2) -> \"0.0000012\"\n // ToPrecision(0.00000012345, 2) -> \"1.2e-7\"\n // Similarily the converter may add up to\n // max_trailing_padding_zeroes_in_precision_mode in precision mode to avoid\n // returning an exponential representation. A zero added by the\n // EMIT_TRAILING_ZERO_AFTER_POINT flag is counted for this limit.\n // Examples for max_trailing_padding_zeroes_in_precision_mode = 1:\n // ToPrecision(230.0, 2) -> \"230\"\n // ToPrecision(230.0, 2) -> \"230.\" with EMIT_TRAILING_DECIMAL_POINT.\n // ToPrecision(230.0, 2) -> \"2.3e2\" with EMIT_TRAILING_ZERO_AFTER_POINT.\n // Examples for max_trailing_padding_zeroes_in_precision_mode = 3, and no\n // EMIT_TRAILING_ZERO_AFTER_POINT:\n // ToPrecision(123450.0, 6) -> \"123450\"\n // ToPrecision(123450.0, 5) -> \"123450\"\n // ToPrecision(123450.0, 4) -> \"123500\"\n // ToPrecision(123450.0, 3) -> \"123000\"\n // ToPrecision(123450.0, 2) -> \"1.2e5\"\n //\n // Returns true if the conversion succeeds. The conversion always succeeds\n // except for the following cases:\n // - the input value is special and no infinity_symbol or nan_symbol has\n // been provided to the constructor,\n // - precision < kMinPericisionDigits\n // - precision > kMaxPrecisionDigits\n // The last condition implies that the result will never contain more than\n // kMaxPrecisionDigits + 7 characters (the sign, the decimal point, the\n // exponent character, the exponent's sign, and at most 3 exponent digits).\n bool ToPrecision(double value,\n int precision,\n StringBuilder* result_builder) const;\n\n enum DtoaMode {\n // Produce the shortest correct representation.\n // For example the output of 0.299999999999999988897 is (the less accurate\n // but correct) 0.3.\n SHORTEST,\n // Same as SHORTEST, but for single-precision floats.\n SHORTEST_SINGLE,\n // Produce a fixed number of digits after the decimal point.\n // For instance fixed(0.1, 4) becomes 0.1000\n // If the input number is big, the output will be big.\n FIXED,\n // Fixed number of digits (independent of the decimal point).\n PRECISION\n };\n\n // The maximal number of digits that are needed to emit a double in base 10.\n // A higher precision can be achieved by using more digits, but the shortest\n // accurate representation of any double will never use more digits than\n // kBase10MaximalLength.\n // Note that DoubleToAscii null-terminates its input. So the given buffer\n // should be at least kBase10MaximalLength + 1 characters long.\n static const int kBase10MaximalLength = 17;\n\n // Converts the given double 'v' to ascii. 'v' must not be NaN, +Infinity, or\n // -Infinity. In SHORTEST_SINGLE-mode this restriction also applies to 'v'\n // after it has been casted to a single-precision float. That is, in this\n // mode static_cast(v) must not be NaN, +Infinity or -Infinity.\n //\n // The result should be interpreted as buffer * 10^(point-length).\n //\n // The output depends on the given mode:\n // - SHORTEST: produce the least amount of digits for which the internal\n // identity requirement is still satisfied. If the digits are printed\n // (together with the correct exponent) then reading this number will give\n // 'v' again. The buffer will choose the representation that is closest to\n // 'v'. If there are two at the same distance, than the one farther away\n // from 0 is chosen (halfway cases - ending with 5 - are rounded up).\n // In this mode the 'requested_digits' parameter is ignored.\n // - SHORTEST_SINGLE: same as SHORTEST but with single-precision.\n // - FIXED: produces digits necessary to print a given number with\n // 'requested_digits' digits after the decimal point. The produced digits\n // might be too short in which case the caller has to fill the remainder\n // with '0's.\n // Example: toFixed(0.001, 5) is allowed to return buffer=\"1\", point=-2.\n // Halfway cases are rounded towards +/-Infinity (away from 0). The call\n // toFixed(0.15, 2) thus returns buffer=\"2\", point=0.\n // The returned buffer may contain digits that would be truncated from the\n // shortest representation of the input.\n // - PRECISION: produces 'requested_digits' where the first digit is not '0'.\n // Even though the length of produced digits usually equals\n // 'requested_digits', the function is allowed to return fewer digits, in\n // which case the caller has to fill the missing digits with '0's.\n // Halfway cases are again rounded away from 0.\n // DoubleToAscii expects the given buffer to be big enough to hold all\n // digits and a terminating null-character. In SHORTEST-mode it expects a\n // buffer of at least kBase10MaximalLength + 1. In all other modes the\n // requested_digits parameter and the padding-zeroes limit the size of the\n // output. Don't forget the decimal point, the exponent character and the\n // terminating null-character when computing the maximal output size.\n // The given length is only used in debug mode to ensure the buffer is big\n // enough.\n static void DoubleToAscii(double v,\n DtoaMode mode,\n int requested_digits,\n char* buffer,\n int buffer_length,\n bool* sign,\n int* length,\n int* point);\n\n private:\n // Implementation for ToShortest and ToShortestSingle.\n bool ToShortestIeeeNumber(double value,\n StringBuilder* result_builder,\n DtoaMode mode) const;\n\n // If the value is a special value (NaN or Infinity) constructs the\n // corresponding string using the configured infinity/nan-symbol.\n // If either of them is NULL or the value is not special then the\n // function returns false.\n bool HandleSpecialValues(double value, StringBuilder* result_builder) const;\n // Constructs an exponential representation (i.e. 1.234e56).\n // The given exponent assumes a decimal point after the first decimal digit.\n void CreateExponentialRepresentation(const char* decimal_digits,\n int length,\n int exponent,\n StringBuilder* result_builder) const;\n // Creates a decimal representation (i.e 1234.5678).\n void CreateDecimalRepresentation(const char* decimal_digits,\n int length,\n int decimal_point,\n int digits_after_point,\n StringBuilder* result_builder) const;\n\n const int flags_;\n const char* const infinity_symbol_;\n const char* const nan_symbol_;\n const char exponent_character_;\n const int decimal_in_shortest_low_;\n const int decimal_in_shortest_high_;\n const int max_leading_padding_zeroes_in_precision_mode_;\n const int max_trailing_padding_zeroes_in_precision_mode_;\n\n DISALLOW_IMPLICIT_CONSTRUCTORS(DoubleToStringConverter);\n};\n\n\nclass StringToDoubleConverter {\n public:\n // Enumeration for allowing octals and ignoring junk when converting\n // strings to numbers.\n enum Flags {\n NO_FLAGS = 0,\n ALLOW_HEX = 1,\n ALLOW_OCTALS = 2,\n ALLOW_TRAILING_JUNK = 4,\n ALLOW_LEADING_SPACES = 8,\n ALLOW_TRAILING_SPACES = 16,\n ALLOW_SPACES_AFTER_SIGN = 32\n };\n\n // Flags should be a bit-or combination of the possible Flags-enum.\n // - NO_FLAGS: no special flags.\n // - ALLOW_HEX: recognizes the prefix \"0x\". Hex numbers may only be integers.\n // Ex: StringToDouble(\"0x1234\") -> 4660.0\n // In StringToDouble(\"0x1234.56\") the characters \".56\" are trailing\n // junk. The result of the call is hence dependent on\n // the ALLOW_TRAILING_JUNK flag and/or the junk value.\n // With this flag \"0x\" is a junk-string. Even with ALLOW_TRAILING_JUNK,\n // the string will not be parsed as \"0\" followed by junk.\n //\n // - ALLOW_OCTALS: recognizes the prefix \"0\" for octals:\n // If a sequence of octal digits starts with '0', then the number is\n // read as octal integer. Octal numbers may only be integers.\n // Ex: StringToDouble(\"01234\") -> 668.0\n // StringToDouble(\"012349\") -> 12349.0 // Not a sequence of octal\n // // digits.\n // In StringToDouble(\"01234.56\") the characters \".56\" are trailing\n // junk. The result of the call is hence dependent on\n // the ALLOW_TRAILING_JUNK flag and/or the junk value.\n // In StringToDouble(\"01234e56\") the characters \"e56\" are trailing\n // junk, too.\n // - ALLOW_TRAILING_JUNK: ignore trailing characters that are not part of\n // a double literal.\n // - ALLOW_LEADING_SPACES: skip over leading spaces.\n // - ALLOW_TRAILING_SPACES: ignore trailing spaces.\n // - ALLOW_SPACES_AFTER_SIGN: ignore spaces after the sign.\n // Ex: StringToDouble(\"- 123.2\") -> -123.2.\n // StringToDouble(\"+ 123.2\") -> 123.2\n //\n // empty_string_value is returned when an empty string is given as input.\n // If ALLOW_LEADING_SPACES or ALLOW_TRAILING_SPACES are set, then a string\n // containing only spaces is converted to the 'empty_string_value', too.\n //\n // junk_string_value is returned when\n // a) ALLOW_TRAILING_JUNK is not set, and a junk character (a character not\n // part of a double-literal) is found.\n // b) ALLOW_TRAILING_JUNK is set, but the string does not start with a\n // double literal.\n //\n // infinity_symbol and nan_symbol are strings that are used to detect\n // inputs that represent infinity and NaN. They can be null, in which case\n // they are ignored.\n // The conversion routine first reads any possible signs. Then it compares the\n // following character of the input-string with the first character of\n // the infinity, and nan-symbol. If either matches, the function assumes, that\n // a match has been found, and expects the following input characters to match\n // the remaining characters of the special-value symbol.\n // This means that the following restrictions apply to special-value symbols:\n // - they must not start with signs ('+', or '-'),\n // - they must not have the same first character.\n // - they must not start with digits.\n //\n // Examples:\n // flags = ALLOW_HEX | ALLOW_TRAILING_JUNK,\n // empty_string_value = 0.0,\n // junk_string_value = NaN,\n // infinity_symbol = \"infinity\",\n // nan_symbol = \"nan\":\n // StringToDouble(\"0x1234\") -> 4660.0.\n // StringToDouble(\"0x1234K\") -> 4660.0.\n // StringToDouble(\"\") -> 0.0 // empty_string_value.\n // StringToDouble(\" \") -> NaN // junk_string_value.\n // StringToDouble(\" 1\") -> NaN // junk_string_value.\n // StringToDouble(\"0x\") -> NaN // junk_string_value.\n // StringToDouble(\"-123.45\") -> -123.45.\n // StringToDouble(\"--123.45\") -> NaN // junk_string_value.\n // StringToDouble(\"123e45\") -> 123e45.\n // StringToDouble(\"123E45\") -> 123e45.\n // StringToDouble(\"123e+45\") -> 123e45.\n // StringToDouble(\"123E-45\") -> 123e-45.\n // StringToDouble(\"123e\") -> 123.0 // trailing junk ignored.\n // StringToDouble(\"123e-\") -> 123.0 // trailing junk ignored.\n // StringToDouble(\"+NaN\") -> NaN // NaN string literal.\n // StringToDouble(\"-infinity\") -> -inf. // infinity literal.\n // StringToDouble(\"Infinity\") -> NaN // junk_string_value.\n //\n // flags = ALLOW_OCTAL | ALLOW_LEADING_SPACES,\n // empty_string_value = 0.0,\n // junk_string_value = NaN,\n // infinity_symbol = NULL,\n // nan_symbol = NULL:\n // StringToDouble(\"0x1234\") -> NaN // junk_string_value.\n // StringToDouble(\"01234\") -> 668.0.\n // StringToDouble(\"\") -> 0.0 // empty_string_value.\n // StringToDouble(\" \") -> 0.0 // empty_string_value.\n // StringToDouble(\" 1\") -> 1.0\n // StringToDouble(\"0x\") -> NaN // junk_string_value.\n // StringToDouble(\"0123e45\") -> NaN // junk_string_value.\n // StringToDouble(\"01239E45\") -> 1239e45.\n // StringToDouble(\"-infinity\") -> NaN // junk_string_value.\n // StringToDouble(\"NaN\") -> NaN // junk_string_value.\n StringToDoubleConverter(int flags,\n double empty_string_value,\n double junk_string_value,\n const char* infinity_symbol,\n const char* nan_symbol)\n : flags_(flags),\n empty_string_value_(empty_string_value),\n junk_string_value_(junk_string_value),\n infinity_symbol_(infinity_symbol),\n nan_symbol_(nan_symbol) {\n }\n\n // Performs the conversion.\n // The output parameter 'processed_characters_count' is set to the number\n // of characters that have been processed to read the number.\n // Spaces than are processed with ALLOW_{LEADING|TRAILING}_SPACES are included\n // in the 'processed_characters_count'. Trailing junk is never included.\n double StringToDouble(const char* buffer,\n int length,\n int* processed_characters_count) const {\n return StringToIeee(buffer, length, processed_characters_count, true);\n }\n\n // Same as StringToDouble but reads a float.\n // Note that this is not equivalent to static_cast(StringToDouble(...))\n // due to potential double-rounding.\n float StringToFloat(const char* buffer,\n int length,\n int* processed_characters_count) const {\n return static_cast(StringToIeee(buffer, length,\n processed_characters_count, false));\n }\n\n private:\n const int flags_;\n const double empty_string_value_;\n const double junk_string_value_;\n const char* const infinity_symbol_;\n const char* const nan_symbol_;\n\n double StringToIeee(const char* buffer,\n int length,\n int* processed_characters_count,\n bool read_as_double) const;\n\n DISALLOW_IMPLICIT_CONSTRUCTORS(StringToDoubleConverter);\n};\n\n} // namespace double_conversion\n\n#endif // DOUBLE_CONVERSION_DOUBLE_CONVERSION_H_\n" }, { "path": "src/kenlm/util/double-conversion/fast-dtoa.cc", "content": "// Copyright 2012 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#include \"fast-dtoa.h\"\n\n#include \"cached-powers.h\"\n#include \"diy-fp.h\"\n#include \"ieee.h\"\n\nnamespace double_conversion {\n\n// The minimal and maximal target exponent define the range of w's binary\n// exponent, where 'w' is the result of multiplying the input by a cached power\n// of ten.\n//\n// A different range might be chosen on a different platform, to optimize digit\n// generation, but a smaller range requires more powers of ten to be cached.\nstatic const int kMinimalTargetExponent = -60;\nstatic const int kMaximalTargetExponent = -32;\n\n\n// Adjusts the last digit of the generated number, and screens out generated\n// solutions that may be inaccurate. A solution may be inaccurate if it is\n// outside the safe interval, or if we cannot prove that it is closer to the\n// input than a neighboring representation of the same length.\n//\n// Input: * buffer containing the digits of too_high / 10^kappa\n// * the buffer's length\n// * distance_too_high_w == (too_high - w).f() * unit\n// * unsafe_interval == (too_high - too_low).f() * unit\n// * rest = (too_high - buffer * 10^kappa).f() * unit\n// * ten_kappa = 10^kappa * unit\n// * unit = the common multiplier\n// Output: returns true if the buffer is guaranteed to contain the closest\n// representable number to the input.\n// Modifies the generated digits in the buffer to approach (round towards) w.\nstatic bool RoundWeed(Vector buffer,\n int length,\n uint64_t distance_too_high_w,\n uint64_t unsafe_interval,\n uint64_t rest,\n uint64_t ten_kappa,\n uint64_t unit) {\n uint64_t small_distance = distance_too_high_w - unit;\n uint64_t big_distance = distance_too_high_w + unit;\n // Let w_low = too_high - big_distance, and\n // w_high = too_high - small_distance.\n // Note: w_low < w < w_high\n //\n // The real w (* unit) must lie somewhere inside the interval\n // ]w_low; w_high[ (often written as \"(w_low; w_high)\")\n\n // Basically the buffer currently contains a number in the unsafe interval\n // ]too_low; too_high[ with too_low < w < too_high\n //\n // too_high - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -\n // ^v 1 unit ^ ^ ^ ^\n // boundary_high --------------------- . . . .\n // ^v 1 unit . . . .\n // - - - - - - - - - - - - - - - - - - - + - - + - - - - - - . .\n // . . ^ . .\n // . big_distance . . .\n // . . . . rest\n // small_distance . . . .\n // v . . . .\n // w_high - - - - - - - - - - - - - - - - - - . . . .\n // ^v 1 unit . . . .\n // w ---------------------------------------- . . . .\n // ^v 1 unit v . . .\n // w_low - - - - - - - - - - - - - - - - - - - - - . . .\n // . . v\n // buffer --------------------------------------------------+-------+--------\n // . .\n // safe_interval .\n // v .\n // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - .\n // ^v 1 unit .\n // boundary_low ------------------------- unsafe_interval\n // ^v 1 unit v\n // too_low - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -\n //\n //\n // Note that the value of buffer could lie anywhere inside the range too_low\n // to too_high.\n //\n // boundary_low, boundary_high and w are approximations of the real boundaries\n // and v (the input number). They are guaranteed to be precise up to one unit.\n // In fact the error is guaranteed to be strictly less than one unit.\n //\n // Anything that lies outside the unsafe interval is guaranteed not to round\n // to v when read again.\n // Anything that lies inside the safe interval is guaranteed to round to v\n // when read again.\n // If the number inside the buffer lies inside the unsafe interval but not\n // inside the safe interval then we simply do not know and bail out (returning\n // false).\n //\n // Similarly we have to take into account the imprecision of 'w' when finding\n // the closest representation of 'w'. If we have two potential\n // representations, and one is closer to both w_low and w_high, then we know\n // it is closer to the actual value v.\n //\n // By generating the digits of too_high we got the largest (closest to\n // too_high) buffer that is still in the unsafe interval. In the case where\n // w_high < buffer < too_high we try to decrement the buffer.\n // This way the buffer approaches (rounds towards) w.\n // There are 3 conditions that stop the decrementation process:\n // 1) the buffer is already below w_high\n // 2) decrementing the buffer would make it leave the unsafe interval\n // 3) decrementing the buffer would yield a number below w_high and farther\n // away than the current number. In other words:\n // (buffer{-1} < w_high) && w_high - buffer{-1} > buffer - w_high\n // Instead of using the buffer directly we use its distance to too_high.\n // Conceptually rest ~= too_high - buffer\n // We need to do the following tests in this order to avoid over- and\n // underflows.\n ASSERT(rest <= unsafe_interval);\n while (rest < small_distance && // Negated condition 1\n unsafe_interval - rest >= ten_kappa && // Negated condition 2\n (rest + ten_kappa < small_distance || // buffer{-1} > w_high\n small_distance - rest >= rest + ten_kappa - small_distance)) {\n buffer[length - 1]--;\n rest += ten_kappa;\n }\n\n // We have approached w+ as much as possible. We now test if approaching w-\n // would require changing the buffer. If yes, then we have two possible\n // representations close to w, but we cannot decide which one is closer.\n if (rest < big_distance &&\n unsafe_interval - rest >= ten_kappa &&\n (rest + ten_kappa < big_distance ||\n big_distance - rest > rest + ten_kappa - big_distance)) {\n return false;\n }\n\n // Weeding test.\n // The safe interval is [too_low + 2 ulp; too_high - 2 ulp]\n // Since too_low = too_high - unsafe_interval this is equivalent to\n // [too_high - unsafe_interval + 4 ulp; too_high - 2 ulp]\n // Conceptually we have: rest ~= too_high - buffer\n return (2 * unit <= rest) && (rest <= unsafe_interval - 4 * unit);\n}\n\n\n// Rounds the buffer upwards if the result is closer to v by possibly adding\n// 1 to the buffer. If the precision of the calculation is not sufficient to\n// round correctly, return false.\n// The rounding might shift the whole buffer in which case the kappa is\n// adjusted. For example \"99\", kappa = 3 might become \"10\", kappa = 4.\n//\n// If 2*rest > ten_kappa then the buffer needs to be round up.\n// rest can have an error of +/- 1 unit. This function accounts for the\n// imprecision and returns false, if the rounding direction cannot be\n// unambiguously determined.\n//\n// Precondition: rest < ten_kappa.\nstatic bool RoundWeedCounted(Vector buffer,\n int length,\n uint64_t rest,\n uint64_t ten_kappa,\n uint64_t unit,\n int* kappa) {\n ASSERT(rest < ten_kappa);\n // The following tests are done in a specific order to avoid overflows. They\n // will work correctly with any uint64 values of rest < ten_kappa and unit.\n //\n // If the unit is too big, then we don't know which way to round. For example\n // a unit of 50 means that the real number lies within rest +/- 50. If\n // 10^kappa == 40 then there is no way to tell which way to round.\n if (unit >= ten_kappa) return false;\n // Even if unit is just half the size of 10^kappa we are already completely\n // lost. (And after the previous test we know that the expression will not\n // over/underflow.)\n if (ten_kappa - unit <= unit) return false;\n // If 2 * (rest + unit) <= 10^kappa we can safely round down.\n if ((ten_kappa - rest > rest) && (ten_kappa - 2 * rest >= 2 * unit)) {\n return true;\n }\n // If 2 * (rest - unit) >= 10^kappa, then we can safely round up.\n if ((rest > unit) && (ten_kappa - (rest - unit) <= (rest - unit))) {\n // Increment the last digit recursively until we find a non '9' digit.\n buffer[length - 1]++;\n for (int i = length - 1; i > 0; --i) {\n if (buffer[i] != '0' + 10) break;\n buffer[i] = '0';\n buffer[i - 1]++;\n }\n // If the first digit is now '0'+ 10 we had a buffer with all '9's. With the\n // exception of the first digit all digits are now '0'. Simply switch the\n // first digit to '1' and adjust the kappa. Example: \"99\" becomes \"10\" and\n // the power (the kappa) is increased.\n if (buffer[0] == '0' + 10) {\n buffer[0] = '1';\n (*kappa) += 1;\n }\n return true;\n }\n return false;\n}\n\n// Returns the biggest power of ten that is less than or equal to the given\n// number. We furthermore receive the maximum number of bits 'number' has.\n//\n// Returns power == 10^(exponent_plus_one-1) such that\n// power <= number < power * 10.\n// If number_bits == 0 then 0^(0-1) is returned.\n// The number of bits must be <= 32.\n// Precondition: number < (1 << (number_bits + 1)).\n\n// Inspired by the method for finding an integer log base 10 from here:\n// http://graphics.stanford.edu/~seander/bithacks.html#IntegerLog10\nstatic unsigned int const kSmallPowersOfTen[] =\n {0, 1, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000,\n 1000000000};\n\nstatic void BiggestPowerTen(uint32_t number,\n int number_bits,\n uint32_t* power,\n int* exponent_plus_one) {\n ASSERT(number < (1u << (number_bits + 1)));\n // 1233/4096 is approximately 1/lg(10).\n int exponent_plus_one_guess = ((number_bits + 1) * 1233 >> 12);\n // We increment to skip over the first entry in the kPowersOf10 table.\n // Note: kPowersOf10[i] == 10^(i-1).\n exponent_plus_one_guess++;\n // We don't have any guarantees that 2^number_bits <= number.\n // TODO(floitsch): can we change the 'while' into an 'if'? We definitely see\n // number < (2^number_bits - 1), but I haven't encountered\n // number < (2^number_bits - 2) yet.\n while (number < kSmallPowersOfTen[exponent_plus_one_guess]) {\n exponent_plus_one_guess--;\n }\n *power = kSmallPowersOfTen[exponent_plus_one_guess];\n *exponent_plus_one = exponent_plus_one_guess;\n}\n\n// Generates the digits of input number w.\n// w is a floating-point number (DiyFp), consisting of a significand and an\n// exponent. Its exponent is bounded by kMinimalTargetExponent and\n// kMaximalTargetExponent.\n// Hence -60 <= w.e() <= -32.\n//\n// Returns false if it fails, in which case the generated digits in the buffer\n// should not be used.\n// Preconditions:\n// * low, w and high are correct up to 1 ulp (unit in the last place). That\n// is, their error must be less than a unit of their last digits.\n// * low.e() == w.e() == high.e()\n// * low < w < high, and taking into account their error: low~ <= high~\n// * kMinimalTargetExponent <= w.e() <= kMaximalTargetExponent\n// Postconditions: returns false if procedure fails.\n// otherwise:\n// * buffer is not null-terminated, but len contains the number of digits.\n// * buffer contains the shortest possible decimal digit-sequence\n// such that LOW < buffer * 10^kappa < HIGH, where LOW and HIGH are the\n// correct values of low and high (without their error).\n// * if more than one decimal representation gives the minimal number of\n// decimal digits then the one closest to W (where W is the correct value\n// of w) is chosen.\n// Remark: this procedure takes into account the imprecision of its input\n// numbers. If the precision is not enough to guarantee all the postconditions\n// then false is returned. This usually happens rarely (~0.5%).\n//\n// Say, for the sake of example, that\n// w.e() == -48, and w.f() == 0x1234567890abcdef\n// w's value can be computed by w.f() * 2^w.e()\n// We can obtain w's integral digits by simply shifting w.f() by -w.e().\n// -> w's integral part is 0x1234\n// w's fractional part is therefore 0x567890abcdef.\n// Printing w's integral part is easy (simply print 0x1234 in decimal).\n// In order to print its fraction we repeatedly multiply the fraction by 10 and\n// get each digit. Example the first digit after the point would be computed by\n// (0x567890abcdef * 10) >> 48. -> 3\n// The whole thing becomes slightly more complicated because we want to stop\n// once we have enough digits. That is, once the digits inside the buffer\n// represent 'w' we can stop. Everything inside the interval low - high\n// represents w. However we have to pay attention to low, high and w's\n// imprecision.\nstatic bool DigitGen(DiyFp low,\n DiyFp w,\n DiyFp high,\n Vector buffer,\n int* length,\n int* kappa) {\n ASSERT(low.e() == w.e() && w.e() == high.e());\n ASSERT(low.f() + 1 <= high.f() - 1);\n ASSERT(kMinimalTargetExponent <= w.e() && w.e() <= kMaximalTargetExponent);\n // low, w and high are imprecise, but by less than one ulp (unit in the last\n // place).\n // If we remove (resp. add) 1 ulp from low (resp. high) we are certain that\n // the new numbers are outside of the interval we want the final\n // representation to lie in.\n // Inversely adding (resp. removing) 1 ulp from low (resp. high) would yield\n // numbers that are certain to lie in the interval. We will use this fact\n // later on.\n // We will now start by generating the digits within the uncertain\n // interval. Later we will weed out representations that lie outside the safe\n // interval and thus _might_ lie outside the correct interval.\n uint64_t unit = 1;\n DiyFp too_low = DiyFp(low.f() - unit, low.e());\n DiyFp too_high = DiyFp(high.f() + unit, high.e());\n // too_low and too_high are guaranteed to lie outside the interval we want the\n // generated number in.\n DiyFp unsafe_interval = DiyFp::Minus(too_high, too_low);\n // We now cut the input number into two parts: the integral digits and the\n // fractionals. We will not write any decimal separator though, but adapt\n // kappa instead.\n // Reminder: we are currently computing the digits (stored inside the buffer)\n // such that: too_low < buffer * 10^kappa < too_high\n // We use too_high for the digit_generation and stop as soon as possible.\n // If we stop early we effectively round down.\n DiyFp one = DiyFp(static_cast(1) << -w.e(), w.e());\n // Division by one is a shift.\n uint32_t integrals = static_cast(too_high.f() >> -one.e());\n // Modulo by one is an and.\n uint64_t fractionals = too_high.f() & (one.f() - 1);\n uint32_t divisor;\n int divisor_exponent_plus_one;\n BiggestPowerTen(integrals, DiyFp::kSignificandSize - (-one.e()),\n &divisor, &divisor_exponent_plus_one);\n *kappa = divisor_exponent_plus_one;\n *length = 0;\n // Loop invariant: buffer = too_high / 10^kappa (integer division)\n // The invariant holds for the first iteration: kappa has been initialized\n // with the divisor exponent + 1. And the divisor is the biggest power of ten\n // that is smaller than integrals.\n while (*kappa > 0) {\n int digit = integrals / divisor;\n buffer[*length] = '0' + digit;\n (*length)++;\n integrals %= divisor;\n (*kappa)--;\n // Note that kappa now equals the exponent of the divisor and that the\n // invariant thus holds again.\n uint64_t rest =\n (static_cast(integrals) << -one.e()) + fractionals;\n // Invariant: too_high = buffer * 10^kappa + DiyFp(rest, one.e())\n // Reminder: unsafe_interval.e() == one.e()\n if (rest < unsafe_interval.f()) {\n // Rounding down (by not emitting the remaining digits) yields a number\n // that lies within the unsafe interval.\n return RoundWeed(buffer, *length, DiyFp::Minus(too_high, w).f(),\n unsafe_interval.f(), rest,\n static_cast(divisor) << -one.e(), unit);\n }\n divisor /= 10;\n }\n\n // The integrals have been generated. We are at the point of the decimal\n // separator. In the following loop we simply multiply the remaining digits by\n // 10 and divide by one. We just need to pay attention to multiply associated\n // data (like the interval or 'unit'), too.\n // Note that the multiplication by 10 does not overflow, because w.e >= -60\n // and thus one.e >= -60.\n ASSERT(one.e() >= -60);\n ASSERT(fractionals < one.f());\n ASSERT(UINT64_2PART_C(0xFFFFFFFF, FFFFFFFF) / 10 >= one.f());\n while (true) {\n fractionals *= 10;\n unit *= 10;\n unsafe_interval.set_f(unsafe_interval.f() * 10);\n // Integer division by one.\n int digit = static_cast(fractionals >> -one.e());\n buffer[*length] = '0' + digit;\n (*length)++;\n fractionals &= one.f() - 1; // Modulo by one.\n (*kappa)--;\n if (fractionals < unsafe_interval.f()) {\n return RoundWeed(buffer, *length, DiyFp::Minus(too_high, w).f() * unit,\n unsafe_interval.f(), fractionals, one.f(), unit);\n }\n }\n}\n\n\n\n// Generates (at most) requested_digits digits of input number w.\n// w is a floating-point number (DiyFp), consisting of a significand and an\n// exponent. Its exponent is bounded by kMinimalTargetExponent and\n// kMaximalTargetExponent.\n// Hence -60 <= w.e() <= -32.\n//\n// Returns false if it fails, in which case the generated digits in the buffer\n// should not be used.\n// Preconditions:\n// * w is correct up to 1 ulp (unit in the last place). That\n// is, its error must be strictly less than a unit of its last digit.\n// * kMinimalTargetExponent <= w.e() <= kMaximalTargetExponent\n//\n// Postconditions: returns false if procedure fails.\n// otherwise:\n// * buffer is not null-terminated, but length contains the number of\n// digits.\n// * the representation in buffer is the most precise representation of\n// requested_digits digits.\n// * buffer contains at most requested_digits digits of w. If there are less\n// than requested_digits digits then some trailing '0's have been removed.\n// * kappa is such that\n// w = buffer * 10^kappa + eps with |eps| < 10^kappa / 2.\n//\n// Remark: This procedure takes into account the imprecision of its input\n// numbers. If the precision is not enough to guarantee all the postconditions\n// then false is returned. This usually happens rarely, but the failure-rate\n// increases with higher requested_digits.\nstatic bool DigitGenCounted(DiyFp w,\n int requested_digits,\n Vector buffer,\n int* length,\n int* kappa) {\n ASSERT(kMinimalTargetExponent <= w.e() && w.e() <= kMaximalTargetExponent);\n ASSERT(kMinimalTargetExponent >= -60);\n ASSERT(kMaximalTargetExponent <= -32);\n // w is assumed to have an error less than 1 unit. Whenever w is scaled we\n // also scale its error.\n uint64_t w_error = 1;\n // We cut the input number into two parts: the integral digits and the\n // fractional digits. We don't emit any decimal separator, but adapt kappa\n // instead. Example: instead of writing \"1.2\" we put \"12\" into the buffer and\n // increase kappa by 1.\n DiyFp one = DiyFp(static_cast(1) << -w.e(), w.e());\n // Division by one is a shift.\n uint32_t integrals = static_cast(w.f() >> -one.e());\n // Modulo by one is an and.\n uint64_t fractionals = w.f() & (one.f() - 1);\n uint32_t divisor;\n int divisor_exponent_plus_one;\n BiggestPowerTen(integrals, DiyFp::kSignificandSize - (-one.e()),\n &divisor, &divisor_exponent_plus_one);\n *kappa = divisor_exponent_plus_one;\n *length = 0;\n\n // Loop invariant: buffer = w / 10^kappa (integer division)\n // The invariant holds for the first iteration: kappa has been initialized\n // with the divisor exponent + 1. And the divisor is the biggest power of ten\n // that is smaller than 'integrals'.\n while (*kappa > 0) {\n int digit = integrals / divisor;\n buffer[*length] = '0' + digit;\n (*length)++;\n requested_digits--;\n integrals %= divisor;\n (*kappa)--;\n // Note that kappa now equals the exponent of the divisor and that the\n // invariant thus holds again.\n if (requested_digits == 0) break;\n divisor /= 10;\n }\n\n if (requested_digits == 0) {\n uint64_t rest =\n (static_cast(integrals) << -one.e()) + fractionals;\n return RoundWeedCounted(buffer, *length, rest,\n static_cast(divisor) << -one.e(), w_error,\n kappa);\n }\n\n // The integrals have been generated. We are at the point of the decimal\n // separator. In the following loop we simply multiply the remaining digits by\n // 10 and divide by one. We just need to pay attention to multiply associated\n // data (the 'unit'), too.\n // Note that the multiplication by 10 does not overflow, because w.e >= -60\n // and thus one.e >= -60.\n ASSERT(one.e() >= -60);\n ASSERT(fractionals < one.f());\n ASSERT(UINT64_2PART_C(0xFFFFFFFF, FFFFFFFF) / 10 >= one.f());\n while (requested_digits > 0 && fractionals > w_error) {\n fractionals *= 10;\n w_error *= 10;\n // Integer division by one.\n int digit = static_cast(fractionals >> -one.e());\n buffer[*length] = '0' + digit;\n (*length)++;\n requested_digits--;\n fractionals &= one.f() - 1; // Modulo by one.\n (*kappa)--;\n }\n if (requested_digits != 0) return false;\n return RoundWeedCounted(buffer, *length, fractionals, one.f(), w_error,\n kappa);\n}\n\n\n// Provides a decimal representation of v.\n// Returns true if it succeeds, otherwise the result cannot be trusted.\n// There will be *length digits inside the buffer (not null-terminated).\n// If the function returns true then\n// v == (double) (buffer * 10^decimal_exponent).\n// The digits in the buffer are the shortest representation possible: no\n// 0.09999999999999999 instead of 0.1. The shorter representation will even be\n// chosen even if the longer one would be closer to v.\n// The last digit will be closest to the actual v. That is, even if several\n// digits might correctly yield 'v' when read again, the closest will be\n// computed.\nstatic bool Grisu3(double v,\n FastDtoaMode mode,\n Vector buffer,\n int* length,\n int* decimal_exponent) {\n DiyFp w = Double(v).AsNormalizedDiyFp();\n // boundary_minus and boundary_plus are the boundaries between v and its\n // closest floating-point neighbors. Any number strictly between\n // boundary_minus and boundary_plus will round to v when convert to a double.\n // Grisu3 will never output representations that lie exactly on a boundary.\n DiyFp boundary_minus, boundary_plus;\n if (mode == FAST_DTOA_SHORTEST) {\n Double(v).NormalizedBoundaries(&boundary_minus, &boundary_plus);\n } else {\n ASSERT(mode == FAST_DTOA_SHORTEST_SINGLE);\n float single_v = static_cast(v);\n Single(single_v).NormalizedBoundaries(&boundary_minus, &boundary_plus);\n }\n ASSERT(boundary_plus.e() == w.e());\n DiyFp ten_mk; // Cached power of ten: 10^-k\n int mk; // -k\n int ten_mk_minimal_binary_exponent =\n kMinimalTargetExponent - (w.e() + DiyFp::kSignificandSize);\n int ten_mk_maximal_binary_exponent =\n kMaximalTargetExponent - (w.e() + DiyFp::kSignificandSize);\n PowersOfTenCache::GetCachedPowerForBinaryExponentRange(\n ten_mk_minimal_binary_exponent,\n ten_mk_maximal_binary_exponent,\n &ten_mk, &mk);\n ASSERT((kMinimalTargetExponent <= w.e() + ten_mk.e() +\n DiyFp::kSignificandSize) &&\n (kMaximalTargetExponent >= w.e() + ten_mk.e() +\n DiyFp::kSignificandSize));\n // Note that ten_mk is only an approximation of 10^-k. A DiyFp only contains a\n // 64 bit significand and ten_mk is thus only precise up to 64 bits.\n\n // The DiyFp::Times procedure rounds its result, and ten_mk is approximated\n // too. The variable scaled_w (as well as scaled_boundary_minus/plus) are now\n // off by a small amount.\n // In fact: scaled_w - w*10^k < 1ulp (unit in the last place) of scaled_w.\n // In other words: let f = scaled_w.f() and e = scaled_w.e(), then\n // (f-1) * 2^e < w*10^k < (f+1) * 2^e\n DiyFp scaled_w = DiyFp::Times(w, ten_mk);\n ASSERT(scaled_w.e() ==\n boundary_plus.e() + ten_mk.e() + DiyFp::kSignificandSize);\n // In theory it would be possible to avoid some recomputations by computing\n // the difference between w and boundary_minus/plus (a power of 2) and to\n // compute scaled_boundary_minus/plus by subtracting/adding from\n // scaled_w. However the code becomes much less readable and the speed\n // enhancements are not terriffic.\n DiyFp scaled_boundary_minus = DiyFp::Times(boundary_minus, ten_mk);\n DiyFp scaled_boundary_plus = DiyFp::Times(boundary_plus, ten_mk);\n\n // DigitGen will generate the digits of scaled_w. Therefore we have\n // v == (double) (scaled_w * 10^-mk).\n // Set decimal_exponent == -mk and pass it to DigitGen. If scaled_w is not an\n // integer than it will be updated. For instance if scaled_w == 1.23 then\n // the buffer will be filled with \"123\" und the decimal_exponent will be\n // decreased by 2.\n int kappa;\n bool result = DigitGen(scaled_boundary_minus, scaled_w, scaled_boundary_plus,\n buffer, length, &kappa);\n *decimal_exponent = -mk + kappa;\n return result;\n}\n\n\n// The \"counted\" version of grisu3 (see above) only generates requested_digits\n// number of digits. This version does not generate the shortest representation,\n// and with enough requested digits 0.1 will at some point print as 0.9999999...\n// Grisu3 is too imprecise for real halfway cases (1.5 will not work) and\n// therefore the rounding strategy for halfway cases is irrelevant.\nstatic bool Grisu3Counted(double v,\n int requested_digits,\n Vector buffer,\n int* length,\n int* decimal_exponent) {\n DiyFp w = Double(v).AsNormalizedDiyFp();\n DiyFp ten_mk; // Cached power of ten: 10^-k\n int mk; // -k\n int ten_mk_minimal_binary_exponent =\n kMinimalTargetExponent - (w.e() + DiyFp::kSignificandSize);\n int ten_mk_maximal_binary_exponent =\n kMaximalTargetExponent - (w.e() + DiyFp::kSignificandSize);\n PowersOfTenCache::GetCachedPowerForBinaryExponentRange(\n ten_mk_minimal_binary_exponent,\n ten_mk_maximal_binary_exponent,\n &ten_mk, &mk);\n ASSERT((kMinimalTargetExponent <= w.e() + ten_mk.e() +\n DiyFp::kSignificandSize) &&\n (kMaximalTargetExponent >= w.e() + ten_mk.e() +\n DiyFp::kSignificandSize));\n // Note that ten_mk is only an approximation of 10^-k. A DiyFp only contains a\n // 64 bit significand and ten_mk is thus only precise up to 64 bits.\n\n // The DiyFp::Times procedure rounds its result, and ten_mk is approximated\n // too. The variable scaled_w (as well as scaled_boundary_minus/plus) are now\n // off by a small amount.\n // In fact: scaled_w - w*10^k < 1ulp (unit in the last place) of scaled_w.\n // In other words: let f = scaled_w.f() and e = scaled_w.e(), then\n // (f-1) * 2^e < w*10^k < (f+1) * 2^e\n DiyFp scaled_w = DiyFp::Times(w, ten_mk);\n\n // We now have (double) (scaled_w * 10^-mk).\n // DigitGen will generate the first requested_digits digits of scaled_w and\n // return together with a kappa such that scaled_w ~= buffer * 10^kappa. (It\n // will not always be exactly the same since DigitGenCounted only produces a\n // limited number of digits.)\n int kappa;\n bool result = DigitGenCounted(scaled_w, requested_digits,\n buffer, length, &kappa);\n *decimal_exponent = -mk + kappa;\n return result;\n}\n\n\nbool FastDtoa(double v,\n FastDtoaMode mode,\n int requested_digits,\n Vector buffer,\n int* length,\n int* decimal_point) {\n ASSERT(v > 0);\n ASSERT(!Double(v).IsSpecial());\n\n bool result = false;\n int decimal_exponent = 0;\n switch (mode) {\n case FAST_DTOA_SHORTEST:\n case FAST_DTOA_SHORTEST_SINGLE:\n result = Grisu3(v, mode, buffer, length, &decimal_exponent);\n break;\n case FAST_DTOA_PRECISION:\n result = Grisu3Counted(v, requested_digits,\n buffer, length, &decimal_exponent);\n break;\n default:\n UNREACHABLE();\n }\n if (result) {\n *decimal_point = *length + decimal_exponent;\n buffer[*length] = '\\0';\n }\n return result;\n}\n\n} // namespace double_conversion\n" }, { "path": "src/kenlm/util/double-conversion/fast-dtoa.h", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#ifndef DOUBLE_CONVERSION_FAST_DTOA_H_\n#define DOUBLE_CONVERSION_FAST_DTOA_H_\n\n#include \"utils.h\"\n\nnamespace double_conversion {\n\nenum FastDtoaMode {\n // Computes the shortest representation of the given input. The returned\n // result will be the most accurate number of this length. Longer\n // representations might be more accurate.\n FAST_DTOA_SHORTEST,\n // Same as FAST_DTOA_SHORTEST but for single-precision floats.\n FAST_DTOA_SHORTEST_SINGLE,\n // Computes a representation where the precision (number of digits) is\n // given as input. The precision is independent of the decimal point.\n FAST_DTOA_PRECISION\n};\n\n// FastDtoa will produce at most kFastDtoaMaximalLength digits. This does not\n// include the terminating '\\0' character.\nstatic const int kFastDtoaMaximalLength = 17;\n// Same for single-precision numbers.\nstatic const int kFastDtoaMaximalSingleLength = 9;\n\n// Provides a decimal representation of v.\n// The result should be interpreted as buffer * 10^(point - length).\n//\n// Precondition:\n// * v must be a strictly positive finite double.\n//\n// Returns true if it succeeds, otherwise the result can not be trusted.\n// There will be *length digits inside the buffer followed by a null terminator.\n// If the function returns true and mode equals\n// - FAST_DTOA_SHORTEST, then\n// the parameter requested_digits is ignored.\n// The result satisfies\n// v == (double) (buffer * 10^(point - length)).\n// The digits in the buffer are the shortest representation possible. E.g.\n// if 0.099999999999 and 0.1 represent the same double then \"1\" is returned\n// with point = 0.\n// The last digit will be closest to the actual v. That is, even if several\n// digits might correctly yield 'v' when read again, the buffer will contain\n// the one closest to v.\n// - FAST_DTOA_PRECISION, then\n// the buffer contains requested_digits digits.\n// the difference v - (buffer * 10^(point-length)) is closest to zero for\n// all possible representations of requested_digits digits.\n// If there are two values that are equally close, then FastDtoa returns\n// false.\n// For both modes the buffer must be large enough to hold the result.\nbool FastDtoa(double d,\n FastDtoaMode mode,\n int requested_digits,\n Vector buffer,\n int* length,\n int* decimal_point);\n\n} // namespace double_conversion\n\n#endif // DOUBLE_CONVERSION_FAST_DTOA_H_\n" }, { "path": "src/kenlm/util/double-conversion/fixed-dtoa.cc", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#include \n\n#include \"fixed-dtoa.h\"\n#include \"ieee.h\"\n\nnamespace double_conversion {\n\n// Represents a 128bit type. This class should be replaced by a native type on\n// platforms that support 128bit integers.\nclass UInt128 {\n public:\n UInt128() : high_bits_(0), low_bits_(0) { }\n UInt128(uint64_t high, uint64_t low) : high_bits_(high), low_bits_(low) { }\n\n void Multiply(uint32_t multiplicand) {\n uint64_t accumulator;\n\n accumulator = (low_bits_ & kMask32) * multiplicand;\n uint32_t part = static_cast(accumulator & kMask32);\n accumulator >>= 32;\n accumulator = accumulator + (low_bits_ >> 32) * multiplicand;\n low_bits_ = (accumulator << 32) + part;\n accumulator >>= 32;\n accumulator = accumulator + (high_bits_ & kMask32) * multiplicand;\n part = static_cast(accumulator & kMask32);\n accumulator >>= 32;\n accumulator = accumulator + (high_bits_ >> 32) * multiplicand;\n high_bits_ = (accumulator << 32) + part;\n ASSERT((accumulator >> 32) == 0);\n }\n\n void Shift(int shift_amount) {\n ASSERT(-64 <= shift_amount && shift_amount <= 64);\n if (shift_amount == 0) {\n return;\n } else if (shift_amount == -64) {\n high_bits_ = low_bits_;\n low_bits_ = 0;\n } else if (shift_amount == 64) {\n low_bits_ = high_bits_;\n high_bits_ = 0;\n } else if (shift_amount <= 0) {\n high_bits_ <<= -shift_amount;\n high_bits_ += low_bits_ >> (64 + shift_amount);\n low_bits_ <<= -shift_amount;\n } else {\n low_bits_ >>= shift_amount;\n low_bits_ += high_bits_ << (64 - shift_amount);\n high_bits_ >>= shift_amount;\n }\n }\n\n // Modifies *this to *this MOD (2^power).\n // Returns *this DIV (2^power).\n int DivModPowerOf2(int power) {\n if (power >= 64) {\n int result = static_cast(high_bits_ >> (power - 64));\n high_bits_ -= static_cast(result) << (power - 64);\n return result;\n } else {\n uint64_t part_low = low_bits_ >> power;\n uint64_t part_high = high_bits_ << (64 - power);\n int result = static_cast(part_low + part_high);\n high_bits_ = 0;\n low_bits_ -= part_low << power;\n return result;\n }\n }\n\n bool IsZero() const {\n return high_bits_ == 0 && low_bits_ == 0;\n }\n\n int BitAt(int position) {\n if (position >= 64) {\n return static_cast(high_bits_ >> (position - 64)) & 1;\n } else {\n return static_cast(low_bits_ >> position) & 1;\n }\n }\n\n private:\n static const uint64_t kMask32 = 0xFFFFFFFF;\n // Value == (high_bits_ << 64) + low_bits_\n uint64_t high_bits_;\n uint64_t low_bits_;\n};\n\n\nstatic const int kDoubleSignificandSize = 53; // Includes the hidden bit.\n\n\nstatic void FillDigits32FixedLength(uint32_t number, int requested_length,\n Vector buffer, int* length) {\n for (int i = requested_length - 1; i >= 0; --i) {\n buffer[(*length) + i] = '0' + number % 10;\n number /= 10;\n }\n *length += requested_length;\n}\n\n\nstatic void FillDigits32(uint32_t number, Vector buffer, int* length) {\n int number_length = 0;\n // We fill the digits in reverse order and exchange them afterwards.\n while (number != 0) {\n int digit = number % 10;\n number /= 10;\n buffer[(*length) + number_length] = '0' + digit;\n number_length++;\n }\n // Exchange the digits.\n int i = *length;\n int j = *length + number_length - 1;\n while (i < j) {\n char tmp = buffer[i];\n buffer[i] = buffer[j];\n buffer[j] = tmp;\n i++;\n j--;\n }\n *length += number_length;\n}\n\n\nstatic void FillDigits64FixedLength(uint64_t number, int requested_length,\n Vector buffer, int* length) {\n const uint32_t kTen7 = 10000000;\n // For efficiency cut the number into 3 uint32_t parts, and print those.\n uint32_t part2 = static_cast(number % kTen7);\n number /= kTen7;\n uint32_t part1 = static_cast(number % kTen7);\n uint32_t part0 = static_cast(number / kTen7);\n\n FillDigits32FixedLength(part0, 3, buffer, length);\n FillDigits32FixedLength(part1, 7, buffer, length);\n FillDigits32FixedLength(part2, 7, buffer, length);\n}\n\n\nstatic void FillDigits64(uint64_t number, Vector buffer, int* length) {\n const uint32_t kTen7 = 10000000;\n // For efficiency cut the number into 3 uint32_t parts, and print those.\n uint32_t part2 = static_cast(number % kTen7);\n number /= kTen7;\n uint32_t part1 = static_cast(number % kTen7);\n uint32_t part0 = static_cast(number / kTen7);\n\n if (part0 != 0) {\n FillDigits32(part0, buffer, length);\n FillDigits32FixedLength(part1, 7, buffer, length);\n FillDigits32FixedLength(part2, 7, buffer, length);\n } else if (part1 != 0) {\n FillDigits32(part1, buffer, length);\n FillDigits32FixedLength(part2, 7, buffer, length);\n } else {\n FillDigits32(part2, buffer, length);\n }\n}\n\n\nstatic void RoundUp(Vector buffer, int* length, int* decimal_point) {\n // An empty buffer represents 0.\n if (*length == 0) {\n buffer[0] = '1';\n *decimal_point = 1;\n *length = 1;\n return;\n }\n // Round the last digit until we either have a digit that was not '9' or until\n // we reached the first digit.\n buffer[(*length) - 1]++;\n for (int i = (*length) - 1; i > 0; --i) {\n if (buffer[i] != '0' + 10) {\n return;\n }\n buffer[i] = '0';\n buffer[i - 1]++;\n }\n // If the first digit is now '0' + 10, we would need to set it to '0' and add\n // a '1' in front. However we reach the first digit only if all following\n // digits had been '9' before rounding up. Now all trailing digits are '0' and\n // we simply switch the first digit to '1' and update the decimal-point\n // (indicating that the point is now one digit to the right).\n if (buffer[0] == '0' + 10) {\n buffer[0] = '1';\n (*decimal_point)++;\n }\n}\n\n\n// The given fractionals number represents a fixed-point number with binary\n// point at bit (-exponent).\n// Preconditions:\n// -128 <= exponent <= 0.\n// 0 <= fractionals * 2^exponent < 1\n// The buffer holds the result.\n// The function will round its result. During the rounding-process digits not\n// generated by this function might be updated, and the decimal-point variable\n// might be updated. If this function generates the digits 99 and the buffer\n// already contained \"199\" (thus yielding a buffer of \"19999\") then a\n// rounding-up will change the contents of the buffer to \"20000\".\nstatic void FillFractionals(uint64_t fractionals, int exponent,\n int fractional_count, Vector buffer,\n int* length, int* decimal_point) {\n ASSERT(-128 <= exponent && exponent <= 0);\n // 'fractionals' is a fixed-point number, with binary point at bit\n // (-exponent). Inside the function the non-converted remainder of fractionals\n // is a fixed-point number, with binary point at bit 'point'.\n if (-exponent <= 64) {\n // One 64 bit number is sufficient.\n ASSERT(fractionals >> 56 == 0);\n int point = -exponent;\n for (int i = 0; i < fractional_count; ++i) {\n if (fractionals == 0) break;\n // Instead of multiplying by 10 we multiply by 5 and adjust the point\n // location. This way the fractionals variable will not overflow.\n // Invariant at the beginning of the loop: fractionals < 2^point.\n // Initially we have: point <= 64 and fractionals < 2^56\n // After each iteration the point is decremented by one.\n // Note that 5^3 = 125 < 128 = 2^7.\n // Therefore three iterations of this loop will not overflow fractionals\n // (even without the subtraction at the end of the loop body). At this\n // time point will satisfy point <= 61 and therefore fractionals < 2^point\n // and any further multiplication of fractionals by 5 will not overflow.\n fractionals *= 5;\n point--;\n int digit = static_cast(fractionals >> point);\n buffer[*length] = '0' + digit;\n (*length)++;\n fractionals -= static_cast(digit) << point;\n }\n // If the first bit after the point is set we have to round up.\n if (((fractionals >> (point - 1)) & 1) == 1) {\n RoundUp(buffer, length, decimal_point);\n }\n } else { // We need 128 bits.\n ASSERT(64 < -exponent && -exponent <= 128);\n UInt128 fractionals128 = UInt128(fractionals, 0);\n fractionals128.Shift(-exponent - 64);\n int point = 128;\n for (int i = 0; i < fractional_count; ++i) {\n if (fractionals128.IsZero()) break;\n // As before: instead of multiplying by 10 we multiply by 5 and adjust the\n // point location.\n // This multiplication will not overflow for the same reasons as before.\n fractionals128.Multiply(5);\n point--;\n int digit = fractionals128.DivModPowerOf2(point);\n buffer[*length] = '0' + digit;\n (*length)++;\n }\n if (fractionals128.BitAt(point - 1) == 1) {\n RoundUp(buffer, length, decimal_point);\n }\n }\n}\n\n\n// Removes leading and trailing zeros.\n// If leading zeros are removed then the decimal point position is adjusted.\nstatic void TrimZeros(Vector buffer, int* length, int* decimal_point) {\n while (*length > 0 && buffer[(*length) - 1] == '0') {\n (*length)--;\n }\n int first_non_zero = 0;\n while (first_non_zero < *length && buffer[first_non_zero] == '0') {\n first_non_zero++;\n }\n if (first_non_zero != 0) {\n for (int i = first_non_zero; i < *length; ++i) {\n buffer[i - first_non_zero] = buffer[i];\n }\n *length -= first_non_zero;\n *decimal_point -= first_non_zero;\n }\n}\n\n\nbool FastFixedDtoa(double v,\n int fractional_count,\n Vector buffer,\n int* length,\n int* decimal_point) {\n const uint32_t kMaxUInt32 = 0xFFFFFFFF;\n uint64_t significand = Double(v).Significand();\n int exponent = Double(v).Exponent();\n // v = significand * 2^exponent (with significand a 53bit integer).\n // If the exponent is larger than 20 (i.e. we may have a 73bit number) then we\n // don't know how to compute the representation. 2^73 ~= 9.5*10^21.\n // If necessary this limit could probably be increased, but we don't need\n // more.\n if (exponent > 20) return false;\n if (fractional_count > 20) return false;\n *length = 0;\n // At most kDoubleSignificandSize bits of the significand are non-zero.\n // Given a 64 bit integer we have 11 0s followed by 53 potentially non-zero\n // bits: 0..11*..0xxx..53*..xx\n if (exponent + kDoubleSignificandSize > 64) {\n // The exponent must be > 11.\n //\n // We know that v = significand * 2^exponent.\n // And the exponent > 11.\n // We simplify the task by dividing v by 10^17.\n // The quotient delivers the first digits, and the remainder fits into a 64\n // bit number.\n // Dividing by 10^17 is equivalent to dividing by 5^17*2^17.\n const uint64_t kFive17 = UINT64_2PART_C(0xB1, A2BC2EC5); // 5^17\n uint64_t divisor = kFive17;\n int divisor_power = 17;\n uint64_t dividend = significand;\n uint32_t quotient;\n uint64_t remainder;\n // Let v = f * 2^e with f == significand and e == exponent.\n // Then need q (quotient) and r (remainder) as follows:\n // v = q * 10^17 + r\n // f * 2^e = q * 10^17 + r\n // f * 2^e = q * 5^17 * 2^17 + r\n // If e > 17 then\n // f * 2^(e-17) = q * 5^17 + r/2^17\n // else\n // f = q * 5^17 * 2^(17-e) + r/2^e\n if (exponent > divisor_power) {\n // We only allow exponents of up to 20 and therefore (17 - e) <= 3\n dividend <<= exponent - divisor_power;\n quotient = static_cast(dividend / divisor);\n remainder = (dividend % divisor) << divisor_power;\n } else {\n divisor <<= divisor_power - exponent;\n quotient = static_cast(dividend / divisor);\n remainder = (dividend % divisor) << exponent;\n }\n FillDigits32(quotient, buffer, length);\n FillDigits64FixedLength(remainder, divisor_power, buffer, length);\n *decimal_point = *length;\n } else if (exponent >= 0) {\n // 0 <= exponent <= 11\n significand <<= exponent;\n FillDigits64(significand, buffer, length);\n *decimal_point = *length;\n } else if (exponent > -kDoubleSignificandSize) {\n // We have to cut the number.\n uint64_t integrals = significand >> -exponent;\n uint64_t fractionals = significand - (integrals << -exponent);\n if (integrals > kMaxUInt32) {\n FillDigits64(integrals, buffer, length);\n } else {\n FillDigits32(static_cast(integrals), buffer, length);\n }\n *decimal_point = *length;\n FillFractionals(fractionals, exponent, fractional_count,\n buffer, length, decimal_point);\n } else if (exponent < -128) {\n // This configuration (with at most 20 digits) means that all digits must be\n // 0.\n ASSERT(fractional_count <= 20);\n buffer[0] = '\\0';\n *length = 0;\n *decimal_point = -fractional_count;\n } else {\n *decimal_point = 0;\n FillFractionals(significand, exponent, fractional_count,\n buffer, length, decimal_point);\n }\n TrimZeros(buffer, length, decimal_point);\n buffer[*length] = '\\0';\n if ((*length) == 0) {\n // The string is empty and the decimal_point thus has no importance. Mimick\n // Gay's dtoa and and set it to -fractional_count.\n *decimal_point = -fractional_count;\n }\n return true;\n}\n\n} // namespace double_conversion\n" }, { "path": "src/kenlm/util/double-conversion/fixed-dtoa.h", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#ifndef DOUBLE_CONVERSION_FIXED_DTOA_H_\n#define DOUBLE_CONVERSION_FIXED_DTOA_H_\n\n#include \"utils.h\"\n\nnamespace double_conversion {\n\n// Produces digits necessary to print a given number with\n// 'fractional_count' digits after the decimal point.\n// The buffer must be big enough to hold the result plus one terminating null\n// character.\n//\n// The produced digits might be too short in which case the caller has to fill\n// the gaps with '0's.\n// Example: FastFixedDtoa(0.001, 5, ...) is allowed to return buffer = \"1\", and\n// decimal_point = -2.\n// Halfway cases are rounded towards +/-Infinity (away from 0). The call\n// FastFixedDtoa(0.15, 2, ...) thus returns buffer = \"2\", decimal_point = 0.\n// The returned buffer may contain digits that would be truncated from the\n// shortest representation of the input.\n//\n// This method only works for some parameters. If it can't handle the input it\n// returns false. The output is null-terminated when the function succeeds.\nbool FastFixedDtoa(double v, int fractional_count,\n Vector buffer, int* length, int* decimal_point);\n\n} // namespace double_conversion\n\n#endif // DOUBLE_CONVERSION_FIXED_DTOA_H_\n" }, { "path": "src/kenlm/util/double-conversion/ieee.h", "content": "// Copyright 2012 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#ifndef DOUBLE_CONVERSION_DOUBLE_H_\n#define DOUBLE_CONVERSION_DOUBLE_H_\n\n#include \"diy-fp.h\"\n\nnamespace double_conversion {\n\n// We assume that doubles and uint64_t have the same endianness.\nstatic uint64_t double_to_uint64(double d) { return BitCast(d); }\nstatic double uint64_to_double(uint64_t d64) { return BitCast(d64); }\nstatic uint32_t float_to_uint32(float f) { return BitCast(f); }\nstatic float uint32_to_float(uint32_t d32) { return BitCast(d32); }\n\n// Helper functions for doubles.\nclass Double {\n public:\n static const uint64_t kSignMask = UINT64_2PART_C(0x80000000, 00000000);\n static const uint64_t kExponentMask = UINT64_2PART_C(0x7FF00000, 00000000);\n static const uint64_t kSignificandMask = UINT64_2PART_C(0x000FFFFF, FFFFFFFF);\n static const uint64_t kHiddenBit = UINT64_2PART_C(0x00100000, 00000000);\n static const int kPhysicalSignificandSize = 52; // Excludes the hidden bit.\n static const int kSignificandSize = 53;\n\n Double() : d64_(0) {}\n explicit Double(double d) : d64_(double_to_uint64(d)) {}\n explicit Double(uint64_t d64) : d64_(d64) {}\n explicit Double(DiyFp diy_fp)\n : d64_(DiyFpToUint64(diy_fp)) {}\n\n // The value encoded by this Double must be greater or equal to +0.0.\n // It must not be special (infinity, or NaN).\n DiyFp AsDiyFp() const {\n ASSERT(Sign() > 0);\n ASSERT(!IsSpecial());\n return DiyFp(Significand(), Exponent());\n }\n\n // The value encoded by this Double must be strictly greater than 0.\n DiyFp AsNormalizedDiyFp() const {\n ASSERT(value() > 0.0);\n uint64_t f = Significand();\n int e = Exponent();\n\n // The current double could be a denormal.\n while ((f & kHiddenBit) == 0) {\n f <<= 1;\n e--;\n }\n // Do the final shifts in one go.\n f <<= DiyFp::kSignificandSize - kSignificandSize;\n e -= DiyFp::kSignificandSize - kSignificandSize;\n return DiyFp(f, e);\n }\n\n // Returns the double's bit as uint64.\n uint64_t AsUint64() const {\n return d64_;\n }\n\n // Returns the next greater double. Returns +infinity on input +infinity.\n double NextDouble() const {\n if (d64_ == kInfinity) return Double(kInfinity).value();\n if (Sign() < 0 && Significand() == 0) {\n // -0.0\n return 0.0;\n }\n if (Sign() < 0) {\n return Double(d64_ - 1).value();\n } else {\n return Double(d64_ + 1).value();\n }\n }\n\n double PreviousDouble() const {\n if (d64_ == (kInfinity | kSignMask)) return -Double::Infinity();\n if (Sign() < 0) {\n return Double(d64_ + 1).value();\n } else {\n if (Significand() == 0) return -0.0;\n return Double(d64_ - 1).value();\n }\n }\n\n int Exponent() const {\n if (IsDenormal()) return kDenormalExponent;\n\n uint64_t d64 = AsUint64();\n int biased_e =\n static_cast((d64 & kExponentMask) >> kPhysicalSignificandSize);\n return biased_e - kExponentBias;\n }\n\n uint64_t Significand() const {\n uint64_t d64 = AsUint64();\n uint64_t significand = d64 & kSignificandMask;\n if (!IsDenormal()) {\n return significand + kHiddenBit;\n } else {\n return significand;\n }\n }\n\n // Returns true if the double is a denormal.\n bool IsDenormal() const {\n uint64_t d64 = AsUint64();\n return (d64 & kExponentMask) == 0;\n }\n\n // We consider denormals not to be special.\n // Hence only Infinity and NaN are special.\n bool IsSpecial() const {\n uint64_t d64 = AsUint64();\n return (d64 & kExponentMask) == kExponentMask;\n }\n\n bool IsNan() const {\n uint64_t d64 = AsUint64();\n return ((d64 & kExponentMask) == kExponentMask) &&\n ((d64 & kSignificandMask) != 0);\n }\n\n bool IsInfinite() const {\n uint64_t d64 = AsUint64();\n return ((d64 & kExponentMask) == kExponentMask) &&\n ((d64 & kSignificandMask) == 0);\n }\n\n int Sign() const {\n uint64_t d64 = AsUint64();\n return (d64 & kSignMask) == 0? 1: -1;\n }\n\n // Precondition: the value encoded by this Double must be greater or equal\n // than +0.0.\n DiyFp UpperBoundary() const {\n ASSERT(Sign() > 0);\n return DiyFp(Significand() * 2 + 1, Exponent() - 1);\n }\n\n // Computes the two boundaries of this.\n // The bigger boundary (m_plus) is normalized. The lower boundary has the same\n // exponent as m_plus.\n // Precondition: the value encoded by this Double must be greater than 0.\n void NormalizedBoundaries(DiyFp* out_m_minus, DiyFp* out_m_plus) const {\n ASSERT(value() > 0.0);\n DiyFp v = this->AsDiyFp();\n DiyFp m_plus = DiyFp::Normalize(DiyFp((v.f() << 1) + 1, v.e() - 1));\n DiyFp m_minus;\n if (LowerBoundaryIsCloser()) {\n m_minus = DiyFp((v.f() << 2) - 1, v.e() - 2);\n } else {\n m_minus = DiyFp((v.f() << 1) - 1, v.e() - 1);\n }\n m_minus.set_f(m_minus.f() << (m_minus.e() - m_plus.e()));\n m_minus.set_e(m_plus.e());\n *out_m_plus = m_plus;\n *out_m_minus = m_minus;\n }\n\n bool LowerBoundaryIsCloser() const {\n // The boundary is closer if the significand is of the form f == 2^p-1 then\n // the lower boundary is closer.\n // Think of v = 1000e10 and v- = 9999e9.\n // Then the boundary (== (v - v-)/2) is not just at a distance of 1e9 but\n // at a distance of 1e8.\n // The only exception is for the smallest normal: the largest denormal is\n // at the same distance as its successor.\n // Note: denormals have the same exponent as the smallest normals.\n bool physical_significand_is_zero = ((AsUint64() & kSignificandMask) == 0);\n return physical_significand_is_zero && (Exponent() != kDenormalExponent);\n }\n\n double value() const { return uint64_to_double(d64_); }\n\n // Returns the significand size for a given order of magnitude.\n // If v = f*2^e with 2^p-1 <= f <= 2^p then p+e is v's order of magnitude.\n // This function returns the number of significant binary digits v will have\n // once it's encoded into a double. In almost all cases this is equal to\n // kSignificandSize. The only exceptions are denormals. They start with\n // leading zeroes and their effective significand-size is hence smaller.\n static int SignificandSizeForOrderOfMagnitude(int order) {\n if (order >= (kDenormalExponent + kSignificandSize)) {\n return kSignificandSize;\n }\n if (order <= kDenormalExponent) return 0;\n return order - kDenormalExponent;\n }\n\n static double Infinity() {\n return Double(kInfinity).value();\n }\n\n static double NaN() {\n return Double(kNaN).value();\n }\n\n private:\n static const int kExponentBias = 0x3FF + kPhysicalSignificandSize;\n static const int kDenormalExponent = -kExponentBias + 1;\n static const int kMaxExponent = 0x7FF - kExponentBias;\n static const uint64_t kInfinity = UINT64_2PART_C(0x7FF00000, 00000000);\n static const uint64_t kNaN = UINT64_2PART_C(0x7FF80000, 00000000);\n\n const uint64_t d64_;\n\n static uint64_t DiyFpToUint64(DiyFp diy_fp) {\n uint64_t significand = diy_fp.f();\n int exponent = diy_fp.e();\n while (significand > kHiddenBit + kSignificandMask) {\n significand >>= 1;\n exponent++;\n }\n if (exponent >= kMaxExponent) {\n return kInfinity;\n }\n if (exponent < kDenormalExponent) {\n return 0;\n }\n while (exponent > kDenormalExponent && (significand & kHiddenBit) == 0) {\n significand <<= 1;\n exponent--;\n }\n uint64_t biased_exponent;\n if (exponent == kDenormalExponent && (significand & kHiddenBit) == 0) {\n biased_exponent = 0;\n } else {\n biased_exponent = static_cast(exponent + kExponentBias);\n }\n return (significand & kSignificandMask) |\n (biased_exponent << kPhysicalSignificandSize);\n }\n};\n\nclass Single {\n public:\n static const uint32_t kSignMask = 0x80000000;\n static const uint32_t kExponentMask = 0x7F800000;\n static const uint32_t kSignificandMask = 0x007FFFFF;\n static const uint32_t kHiddenBit = 0x00800000;\n static const int kPhysicalSignificandSize = 23; // Excludes the hidden bit.\n static const int kSignificandSize = 24;\n\n Single() : d32_(0) {}\n explicit Single(float f) : d32_(float_to_uint32(f)) {}\n explicit Single(uint32_t d32) : d32_(d32) {}\n\n // The value encoded by this Single must be greater or equal to +0.0.\n // It must not be special (infinity, or NaN).\n DiyFp AsDiyFp() const {\n ASSERT(Sign() > 0);\n ASSERT(!IsSpecial());\n return DiyFp(Significand(), Exponent());\n }\n\n // Returns the single's bit as uint64.\n uint32_t AsUint32() const {\n return d32_;\n }\n\n int Exponent() const {\n if (IsDenormal()) return kDenormalExponent;\n\n uint32_t d32 = AsUint32();\n int biased_e =\n static_cast((d32 & kExponentMask) >> kPhysicalSignificandSize);\n return biased_e - kExponentBias;\n }\n\n uint32_t Significand() const {\n uint32_t d32 = AsUint32();\n uint32_t significand = d32 & kSignificandMask;\n if (!IsDenormal()) {\n return significand + kHiddenBit;\n } else {\n return significand;\n }\n }\n\n // Returns true if the single is a denormal.\n bool IsDenormal() const {\n uint32_t d32 = AsUint32();\n return (d32 & kExponentMask) == 0;\n }\n\n // We consider denormals not to be special.\n // Hence only Infinity and NaN are special.\n bool IsSpecial() const {\n uint32_t d32 = AsUint32();\n return (d32 & kExponentMask) == kExponentMask;\n }\n\n bool IsNan() const {\n uint32_t d32 = AsUint32();\n return ((d32 & kExponentMask) == kExponentMask) &&\n ((d32 & kSignificandMask) != 0);\n }\n\n bool IsInfinite() const {\n uint32_t d32 = AsUint32();\n return ((d32 & kExponentMask) == kExponentMask) &&\n ((d32 & kSignificandMask) == 0);\n }\n\n int Sign() const {\n uint32_t d32 = AsUint32();\n return (d32 & kSignMask) == 0? 1: -1;\n }\n\n // Computes the two boundaries of this.\n // The bigger boundary (m_plus) is normalized. The lower boundary has the same\n // exponent as m_plus.\n // Precondition: the value encoded by this Single must be greater than 0.\n void NormalizedBoundaries(DiyFp* out_m_minus, DiyFp* out_m_plus) const {\n ASSERT(value() > 0.0);\n DiyFp v = this->AsDiyFp();\n DiyFp m_plus = DiyFp::Normalize(DiyFp((v.f() << 1) + 1, v.e() - 1));\n DiyFp m_minus;\n if (LowerBoundaryIsCloser()) {\n m_minus = DiyFp((v.f() << 2) - 1, v.e() - 2);\n } else {\n m_minus = DiyFp((v.f() << 1) - 1, v.e() - 1);\n }\n m_minus.set_f(m_minus.f() << (m_minus.e() - m_plus.e()));\n m_minus.set_e(m_plus.e());\n *out_m_plus = m_plus;\n *out_m_minus = m_minus;\n }\n\n // Precondition: the value encoded by this Single must be greater or equal\n // than +0.0.\n DiyFp UpperBoundary() const {\n ASSERT(Sign() > 0);\n return DiyFp(Significand() * 2 + 1, Exponent() - 1);\n }\n\n bool LowerBoundaryIsCloser() const {\n // The boundary is closer if the significand is of the form f == 2^p-1 then\n // the lower boundary is closer.\n // Think of v = 1000e10 and v- = 9999e9.\n // Then the boundary (== (v - v-)/2) is not just at a distance of 1e9 but\n // at a distance of 1e8.\n // The only exception is for the smallest normal: the largest denormal is\n // at the same distance as its successor.\n // Note: denormals have the same exponent as the smallest normals.\n bool physical_significand_is_zero = ((AsUint32() & kSignificandMask) == 0);\n return physical_significand_is_zero && (Exponent() != kDenormalExponent);\n }\n\n float value() const { return uint32_to_float(d32_); }\n\n static float Infinity() {\n return Single(kInfinity).value();\n }\n\n static float NaN() {\n return Single(kNaN).value();\n }\n\n private:\n static const int kExponentBias = 0x7F + kPhysicalSignificandSize;\n static const int kDenormalExponent = -kExponentBias + 1;\n static const int kMaxExponent = 0xFF - kExponentBias;\n static const uint32_t kInfinity = 0x7F800000;\n static const uint32_t kNaN = 0x7FC00000;\n\n const uint32_t d32_;\n};\n\n} // namespace double_conversion\n\n#endif // DOUBLE_CONVERSION_DOUBLE_H_\n" }, { "path": "src/kenlm/util/double-conversion/strtod.cc", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#include \n#include \n\n#include \"strtod.h\"\n#include \"bignum.h\"\n#include \"cached-powers.h\"\n#include \"ieee.h\"\n\nnamespace double_conversion {\n\n// 2^53 = 9007199254740992.\n// Any integer with at most 15 decimal digits will hence fit into a double\n// (which has a 53bit significand) without loss of precision.\nstatic const int kMaxExactDoubleIntegerDecimalDigits = 15;\n// 2^64 = 18446744073709551616 > 10^19\nstatic const int kMaxUint64DecimalDigits = 19;\n\n// Max double: 1.7976931348623157 x 10^308\n// Min non-zero double: 4.9406564584124654 x 10^-324\n// Any x >= 10^309 is interpreted as +infinity.\n// Any x <= 10^-324 is interpreted as 0.\n// Note that 2.5e-324 (despite being smaller than the min double) will be read\n// as non-zero (equal to the min non-zero double).\nstatic const int kMaxDecimalPower = 309;\nstatic const int kMinDecimalPower = -324;\n\n// 2^64 = 18446744073709551616\nstatic const uint64_t kMaxUint64 = UINT64_2PART_C(0xFFFFFFFF, FFFFFFFF);\n\n\nstatic const double exact_powers_of_ten[] = {\n 1.0, // 10^0\n 10.0,\n 100.0,\n 1000.0,\n 10000.0,\n 100000.0,\n 1000000.0,\n 10000000.0,\n 100000000.0,\n 1000000000.0,\n 10000000000.0, // 10^10\n 100000000000.0,\n 1000000000000.0,\n 10000000000000.0,\n 100000000000000.0,\n 1000000000000000.0,\n 10000000000000000.0,\n 100000000000000000.0,\n 1000000000000000000.0,\n 10000000000000000000.0,\n 100000000000000000000.0, // 10^20\n 1000000000000000000000.0,\n // 10^22 = 0x21e19e0c9bab2400000 = 0x878678326eac9 * 2^22\n 10000000000000000000000.0\n};\nstatic const int kExactPowersOfTenSize = ARRAY_SIZE(exact_powers_of_ten);\n\n// Maximum number of significant digits in the decimal representation.\n// In fact the value is 772 (see conversions.cc), but to give us some margin\n// we round up to 780.\nstatic const int kMaxSignificantDecimalDigits = 780;\n\nstatic Vector TrimLeadingZeros(Vector buffer) {\n for (int i = 0; i < buffer.length(); i++) {\n if (buffer[i] != '0') {\n return buffer.SubVector(i, buffer.length());\n }\n }\n return Vector(buffer.start(), 0);\n}\n\n\nstatic Vector TrimTrailingZeros(Vector buffer) {\n for (int i = buffer.length() - 1; i >= 0; --i) {\n if (buffer[i] != '0') {\n return buffer.SubVector(0, i + 1);\n }\n }\n return Vector(buffer.start(), 0);\n}\n\n\nstatic void CutToMaxSignificantDigits(Vector buffer,\n int exponent,\n char* significant_buffer,\n int* significant_exponent) {\n for (int i = 0; i < kMaxSignificantDecimalDigits - 1; ++i) {\n significant_buffer[i] = buffer[i];\n }\n // The input buffer has been trimmed. Therefore the last digit must be\n // different from '0'.\n ASSERT(buffer[buffer.length() - 1] != '0');\n // Set the last digit to be non-zero. This is sufficient to guarantee\n // correct rounding.\n significant_buffer[kMaxSignificantDecimalDigits - 1] = '1';\n *significant_exponent =\n exponent + (buffer.length() - kMaxSignificantDecimalDigits);\n}\n\n\n// Trims the buffer and cuts it to at most kMaxSignificantDecimalDigits.\n// If possible the input-buffer is reused, but if the buffer needs to be\n// modified (due to cutting), then the input needs to be copied into the\n// buffer_copy_space.\nstatic void TrimAndCut(Vector buffer, int exponent,\n char* buffer_copy_space, int space_size,\n Vector* trimmed, int* updated_exponent) {\n Vector left_trimmed = TrimLeadingZeros(buffer);\n Vector right_trimmed = TrimTrailingZeros(left_trimmed);\n exponent += left_trimmed.length() - right_trimmed.length();\n if (right_trimmed.length() > kMaxSignificantDecimalDigits) {\n ASSERT(space_size >= kMaxSignificantDecimalDigits);\n CutToMaxSignificantDigits(right_trimmed, exponent,\n buffer_copy_space, updated_exponent);\n *trimmed = Vector(buffer_copy_space,\n kMaxSignificantDecimalDigits);\n } else {\n *trimmed = right_trimmed;\n *updated_exponent = exponent;\n }\n}\n\n\n// Reads digits from the buffer and converts them to a uint64.\n// Reads in as many digits as fit into a uint64.\n// When the string starts with \"1844674407370955161\" no further digit is read.\n// Since 2^64 = 18446744073709551616 it would still be possible read another\n// digit if it was less or equal than 6, but this would complicate the code.\nstatic uint64_t ReadUint64(Vector buffer,\n int* number_of_read_digits) {\n uint64_t result = 0;\n int i = 0;\n while (i < buffer.length() && result <= (kMaxUint64 / 10 - 1)) {\n int digit = buffer[i++] - '0';\n ASSERT(0 <= digit && digit <= 9);\n result = 10 * result + digit;\n }\n *number_of_read_digits = i;\n return result;\n}\n\n\n// Reads a DiyFp from the buffer.\n// The returned DiyFp is not necessarily normalized.\n// If remaining_decimals is zero then the returned DiyFp is accurate.\n// Otherwise it has been rounded and has error of at most 1/2 ulp.\nstatic void ReadDiyFp(Vector buffer,\n DiyFp* result,\n int* remaining_decimals) {\n int read_digits;\n uint64_t significand = ReadUint64(buffer, &read_digits);\n if (buffer.length() == read_digits) {\n *result = DiyFp(significand, 0);\n *remaining_decimals = 0;\n } else {\n // Round the significand.\n if (buffer[read_digits] >= '5') {\n significand++;\n }\n // Compute the binary exponent.\n int exponent = 0;\n *result = DiyFp(significand, exponent);\n *remaining_decimals = buffer.length() - read_digits;\n }\n}\n\n\nstatic bool DoubleStrtod(Vector trimmed,\n int exponent,\n double* result) {\n#if !defined(DOUBLE_CONVERSION_CORRECT_DOUBLE_OPERATIONS)\n // On x86 the floating-point stack can be 64 or 80 bits wide. If it is\n // 80 bits wide (as is the case on Linux) then double-rounding occurs and the\n // result is not accurate.\n // We know that Windows32 uses 64 bits and is therefore accurate.\n // Note that the ARM simulator is compiled for 32bits. It therefore exhibits\n // the same problem.\n return false;\n#endif\n if (trimmed.length() <= kMaxExactDoubleIntegerDecimalDigits) {\n int read_digits;\n // The trimmed input fits into a double.\n // If the 10^exponent (resp. 10^-exponent) fits into a double too then we\n // can compute the result-double simply by multiplying (resp. dividing) the\n // two numbers.\n // This is possible because IEEE guarantees that floating-point operations\n // return the best possible approximation.\n if (exponent < 0 && -exponent < kExactPowersOfTenSize) {\n // 10^-exponent fits into a double.\n *result = static_cast(ReadUint64(trimmed, &read_digits));\n ASSERT(read_digits == trimmed.length());\n *result /= exact_powers_of_ten[-exponent];\n return true;\n }\n if (0 <= exponent && exponent < kExactPowersOfTenSize) {\n // 10^exponent fits into a double.\n *result = static_cast(ReadUint64(trimmed, &read_digits));\n ASSERT(read_digits == trimmed.length());\n *result *= exact_powers_of_ten[exponent];\n return true;\n }\n int remaining_digits =\n kMaxExactDoubleIntegerDecimalDigits - trimmed.length();\n if ((0 <= exponent) &&\n (exponent - remaining_digits < kExactPowersOfTenSize)) {\n // The trimmed string was short and we can multiply it with\n // 10^remaining_digits. As a result the remaining exponent now fits\n // into a double too.\n *result = static_cast(ReadUint64(trimmed, &read_digits));\n ASSERT(read_digits == trimmed.length());\n *result *= exact_powers_of_ten[remaining_digits];\n *result *= exact_powers_of_ten[exponent - remaining_digits];\n return true;\n }\n }\n return false;\n}\n\n\n// Returns 10^exponent as an exact DiyFp.\n// The given exponent must be in the range [1; kDecimalExponentDistance[.\nstatic DiyFp AdjustmentPowerOfTen(int exponent) {\n ASSERT(0 < exponent);\n ASSERT(exponent < PowersOfTenCache::kDecimalExponentDistance);\n // Simply hardcode the remaining powers for the given decimal exponent\n // distance.\n ASSERT(PowersOfTenCache::kDecimalExponentDistance == 8);\n switch (exponent) {\n case 1: return DiyFp(UINT64_2PART_C(0xa0000000, 00000000), -60);\n case 2: return DiyFp(UINT64_2PART_C(0xc8000000, 00000000), -57);\n case 3: return DiyFp(UINT64_2PART_C(0xfa000000, 00000000), -54);\n case 4: return DiyFp(UINT64_2PART_C(0x9c400000, 00000000), -50);\n case 5: return DiyFp(UINT64_2PART_C(0xc3500000, 00000000), -47);\n case 6: return DiyFp(UINT64_2PART_C(0xf4240000, 00000000), -44);\n case 7: return DiyFp(UINT64_2PART_C(0x98968000, 00000000), -40);\n default:\n UNREACHABLE();\n return DiyFp(0, 0);\n }\n}\n\n\n// If the function returns true then the result is the correct double.\n// Otherwise it is either the correct double or the double that is just below\n// the correct double.\nstatic bool DiyFpStrtod(Vector buffer,\n int exponent,\n double* result) {\n DiyFp input;\n int remaining_decimals;\n ReadDiyFp(buffer, &input, &remaining_decimals);\n // Since we may have dropped some digits the input is not accurate.\n // If remaining_decimals is different than 0 than the error is at most\n // .5 ulp (unit in the last place).\n // We don't want to deal with fractions and therefore keep a common\n // denominator.\n const int kDenominatorLog = 3;\n const int kDenominator = 1 << kDenominatorLog;\n // Move the remaining decimals into the exponent.\n exponent += remaining_decimals;\n int error = (remaining_decimals == 0 ? 0 : kDenominator / 2);\n\n int old_e = input.e();\n input.Normalize();\n error <<= old_e - input.e();\n\n ASSERT(exponent <= PowersOfTenCache::kMaxDecimalExponent);\n if (exponent < PowersOfTenCache::kMinDecimalExponent) {\n *result = 0.0;\n return true;\n }\n DiyFp cached_power;\n int cached_decimal_exponent;\n PowersOfTenCache::GetCachedPowerForDecimalExponent(exponent,\n &cached_power,\n &cached_decimal_exponent);\n\n if (cached_decimal_exponent != exponent) {\n int adjustment_exponent = exponent - cached_decimal_exponent;\n DiyFp adjustment_power = AdjustmentPowerOfTen(adjustment_exponent);\n input.Multiply(adjustment_power);\n if (kMaxUint64DecimalDigits - buffer.length() >= adjustment_exponent) {\n // The product of input with the adjustment power fits into a 64 bit\n // integer.\n ASSERT(DiyFp::kSignificandSize == 64);\n } else {\n // The adjustment power is exact. There is hence only an error of 0.5.\n error += kDenominator / 2;\n }\n }\n\n input.Multiply(cached_power);\n // The error introduced by a multiplication of a*b equals\n // error_a + error_b + error_a*error_b/2^64 + 0.5\n // Substituting a with 'input' and b with 'cached_power' we have\n // error_b = 0.5 (all cached powers have an error of less than 0.5 ulp),\n // error_ab = 0 or 1 / kDenominator > error_a*error_b/ 2^64\n int error_b = kDenominator / 2;\n int error_ab = (error == 0 ? 0 : 1); // We round up to 1.\n int fixed_error = kDenominator / 2;\n error += error_b + error_ab + fixed_error;\n\n old_e = input.e();\n input.Normalize();\n error <<= old_e - input.e();\n\n // See if the double's significand changes if we add/subtract the error.\n int order_of_magnitude = DiyFp::kSignificandSize + input.e();\n int effective_significand_size =\n Double::SignificandSizeForOrderOfMagnitude(order_of_magnitude);\n int precision_digits_count =\n DiyFp::kSignificandSize - effective_significand_size;\n if (precision_digits_count + kDenominatorLog >= DiyFp::kSignificandSize) {\n // This can only happen for very small denormals. In this case the\n // half-way multiplied by the denominator exceeds the range of an uint64.\n // Simply shift everything to the right.\n int shift_amount = (precision_digits_count + kDenominatorLog) -\n DiyFp::kSignificandSize + 1;\n input.set_f(input.f() >> shift_amount);\n input.set_e(input.e() + shift_amount);\n // We add 1 for the lost precision of error, and kDenominator for\n // the lost precision of input.f().\n error = (error >> shift_amount) + 1 + kDenominator;\n precision_digits_count -= shift_amount;\n }\n // We use uint64_ts now. This only works if the DiyFp uses uint64_ts too.\n ASSERT(DiyFp::kSignificandSize == 64);\n ASSERT(precision_digits_count < 64);\n uint64_t one64 = 1;\n uint64_t precision_bits_mask = (one64 << precision_digits_count) - 1;\n uint64_t precision_bits = input.f() & precision_bits_mask;\n uint64_t half_way = one64 << (precision_digits_count - 1);\n precision_bits *= kDenominator;\n half_way *= kDenominator;\n DiyFp rounded_input(input.f() >> precision_digits_count,\n input.e() + precision_digits_count);\n if (precision_bits >= half_way + error) {\n rounded_input.set_f(rounded_input.f() + 1);\n }\n // If the last_bits are too close to the half-way case than we are too\n // inaccurate and round down. In this case we return false so that we can\n // fall back to a more precise algorithm.\n\n *result = Double(rounded_input).value();\n if (half_way - error < precision_bits && precision_bits < half_way + error) {\n // Too imprecise. The caller will have to fall back to a slower version.\n // However the returned number is guaranteed to be either the correct\n // double, or the next-lower double.\n return false;\n } else {\n return true;\n }\n}\n\n\n// Returns\n// - -1 if buffer*10^exponent < diy_fp.\n// - 0 if buffer*10^exponent == diy_fp.\n// - +1 if buffer*10^exponent > diy_fp.\n// Preconditions:\n// buffer.length() + exponent <= kMaxDecimalPower + 1\n// buffer.length() + exponent > kMinDecimalPower\n// buffer.length() <= kMaxDecimalSignificantDigits\nstatic int CompareBufferWithDiyFp(Vector buffer,\n int exponent,\n DiyFp diy_fp) {\n ASSERT(buffer.length() + exponent <= kMaxDecimalPower + 1);\n ASSERT(buffer.length() + exponent > kMinDecimalPower);\n ASSERT(buffer.length() <= kMaxSignificantDecimalDigits);\n // Make sure that the Bignum will be able to hold all our numbers.\n // Our Bignum implementation has a separate field for exponents. Shifts will\n // consume at most one bigit (< 64 bits).\n // ln(10) == 3.3219...\n ASSERT(((kMaxDecimalPower + 1) * 333 / 100) < Bignum::kMaxSignificantBits);\n Bignum buffer_bignum;\n Bignum diy_fp_bignum;\n buffer_bignum.AssignDecimalString(buffer);\n diy_fp_bignum.AssignUInt64(diy_fp.f());\n if (exponent >= 0) {\n buffer_bignum.MultiplyByPowerOfTen(exponent);\n } else {\n diy_fp_bignum.MultiplyByPowerOfTen(-exponent);\n }\n if (diy_fp.e() > 0) {\n diy_fp_bignum.ShiftLeft(diy_fp.e());\n } else {\n buffer_bignum.ShiftLeft(-diy_fp.e());\n }\n return Bignum::Compare(buffer_bignum, diy_fp_bignum);\n}\n\n\n// Returns true if the guess is the correct double.\n// Returns false, when guess is either correct or the next-lower double.\nstatic bool ComputeGuess(Vector trimmed, int exponent,\n double* guess) {\n if (trimmed.length() == 0) {\n *guess = 0.0;\n return true;\n }\n if (exponent + trimmed.length() - 1 >= kMaxDecimalPower) {\n *guess = Double::Infinity();\n return true;\n }\n if (exponent + trimmed.length() <= kMinDecimalPower) {\n *guess = 0.0;\n return true;\n }\n\n if (DoubleStrtod(trimmed, exponent, guess) ||\n DiyFpStrtod(trimmed, exponent, guess)) {\n return true;\n }\n if (*guess == Double::Infinity()) {\n return true;\n }\n return false;\n}\n\ndouble Strtod(Vector buffer, int exponent) {\n char copy_buffer[kMaxSignificantDecimalDigits];\n Vector trimmed;\n int updated_exponent;\n TrimAndCut(buffer, exponent, copy_buffer, kMaxSignificantDecimalDigits,\n &trimmed, &updated_exponent);\n exponent = updated_exponent;\n\n double guess;\n bool is_correct = ComputeGuess(trimmed, exponent, &guess);\n if (is_correct) return guess;\n\n DiyFp upper_boundary = Double(guess).UpperBoundary();\n int comparison = CompareBufferWithDiyFp(trimmed, exponent, upper_boundary);\n if (comparison < 0) {\n return guess;\n } else if (comparison > 0) {\n return Double(guess).NextDouble();\n } else if ((Double(guess).Significand() & 1) == 0) {\n // Round towards even.\n return guess;\n } else {\n return Double(guess).NextDouble();\n }\n}\n\nfloat Strtof(Vector buffer, int exponent) {\n char copy_buffer[kMaxSignificantDecimalDigits];\n Vector trimmed;\n int updated_exponent;\n TrimAndCut(buffer, exponent, copy_buffer, kMaxSignificantDecimalDigits,\n &trimmed, &updated_exponent);\n exponent = updated_exponent;\n\n double double_guess;\n bool is_correct = ComputeGuess(trimmed, exponent, &double_guess);\n\n float float_guess = static_cast(double_guess);\n if (float_guess == double_guess) {\n // This shortcut triggers for integer values.\n return float_guess;\n }\n\n // We must catch double-rounding. Say the double has been rounded up, and is\n // now a boundary of a float, and rounds up again. This is why we have to\n // look at previous too.\n // Example (in decimal numbers):\n // input: 12349\n // high-precision (4 digits): 1235\n // low-precision (3 digits):\n // when read from input: 123\n // when rounded from high precision: 124.\n // To do this we simply look at the neigbors of the correct result and see\n // if they would round to the same float. If the guess is not correct we have\n // to look at four values (since two different doubles could be the correct\n // double).\n\n double double_next = Double(double_guess).NextDouble();\n double double_previous = Double(double_guess).PreviousDouble();\n\n float f1 = static_cast(double_previous);\n#ifndef NDEBUG\n float f2 = float_guess;\n#endif\n float f3 = static_cast(double_next);\n float f4;\n if (is_correct) {\n f4 = f3;\n } else {\n double double_next2 = Double(double_next).NextDouble();\n f4 = static_cast(double_next2);\n }\n#ifndef NDEBUG\n ASSERT(f1 <= f2 && f2 <= f3 && f3 <= f4);\n#endif\n\n // If the guess doesn't lie near a single-precision boundary we can simply\n // return its float-value.\n if (f1 == f4) {\n return float_guess;\n }\n\n ASSERT((f1 != f2 && f2 == f3 && f3 == f4) ||\n (f1 == f2 && f2 != f3 && f3 == f4) ||\n (f1 == f2 && f2 == f3 && f3 != f4));\n\n // guess and next are the two possible canditates (in the same way that\n // double_guess was the lower candidate for a double-precision guess).\n float guess = f1;\n float next = f4;\n DiyFp upper_boundary;\n if (guess == 0.0f) {\n float min_float = 1e-45f;\n upper_boundary = Double(static_cast(min_float) / 2).AsDiyFp();\n } else {\n upper_boundary = Single(guess).UpperBoundary();\n }\n int comparison = CompareBufferWithDiyFp(trimmed, exponent, upper_boundary);\n if (comparison < 0) {\n return guess;\n } else if (comparison > 0) {\n return next;\n } else if ((Single(guess).Significand() & 1) == 0) {\n // Round towards even.\n return guess;\n } else {\n return next;\n }\n}\n\n} // namespace double_conversion\n" }, { "path": "src/kenlm/util/double-conversion/strtod.h", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#ifndef DOUBLE_CONVERSION_STRTOD_H_\n#define DOUBLE_CONVERSION_STRTOD_H_\n\n#include \"utils.h\"\n\nnamespace double_conversion {\n\n// The buffer must only contain digits in the range [0-9]. It must not\n// contain a dot or a sign. It must not start with '0', and must not be empty.\ndouble Strtod(Vector buffer, int exponent);\n\n// The buffer must only contain digits in the range [0-9]. It must not\n// contain a dot or a sign. It must not start with '0', and must not be empty.\nfloat Strtof(Vector buffer, int exponent);\n\n} // namespace double_conversion\n\n#endif // DOUBLE_CONVERSION_STRTOD_H_\n" }, { "path": "src/kenlm/util/double-conversion/utils.h", "content": "// Copyright 2010 the V8 project authors. All rights reserved.\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following\n// disclaimer in the documentation and/or other materials provided\n// with the distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived\n// from this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n\n#ifndef DOUBLE_CONVERSION_UTILS_H_\n#define DOUBLE_CONVERSION_UTILS_H_\n\n#include \n#include \n\n#include \n#ifndef ASSERT\n#define ASSERT(condition) (assert(condition))\n#endif\n#ifndef UNIMPLEMENTED\n#define UNIMPLEMENTED() (abort())\n#endif\n#ifndef UNREACHABLE\n#define UNREACHABLE() (abort())\n#endif\n\n// Double operations detection based on target architecture.\n// Linux uses a 80bit wide floating point stack on x86. This induces double\n// rounding, which in turn leads to wrong results.\n// An easy way to test if the floating-point operations are correct is to\n// evaluate: 89255.0/1e22. If the floating-point stack is 64 bits wide then\n// the result is equal to 89255e-22.\n// The best way to test this, is to create a division-function and to compare\n// the output of the division with the expected result. (Inlining must be\n// disabled.)\n// On Linux,x86 89255e-22 != Div_double(89255.0/1e22)\n#if defined(_M_X64) || defined(__x86_64__) || \\\n defined(__ARMEL__) || defined(__avr32__) || \\\n defined(__hppa__) || defined(__ia64__) || \\\n defined(__mips__) || defined(__powerpc__) || \\\n defined(__sparc__) || defined(__sparc) || defined(__s390__) || \\\n defined(__SH4__) || defined(__alpha__) || \\\n defined(_MIPS_ARCH_MIPS32R2)\n#define DOUBLE_CONVERSION_CORRECT_DOUBLE_OPERATIONS 1\n#elif defined(_M_IX86) || defined(__i386__) || defined(__i386)\n#if defined(_WIN32)\n// Windows uses a 64bit wide floating point stack.\n#define DOUBLE_CONVERSION_CORRECT_DOUBLE_OPERATIONS 1\n#else\n#undef DOUBLE_CONVERSION_CORRECT_DOUBLE_OPERATIONS\n#endif // _WIN32\n#else\n#error Target architecture was not detected as supported by Double-Conversion.\n#endif\n\n\n#if defined(_WIN32) && !defined(__MINGW32__)\n\ntypedef signed char int8_t;\ntypedef unsigned char uint8_t;\ntypedef short int16_t; // NOLINT\ntypedef unsigned short uint16_t; // NOLINT\ntypedef int int32_t;\ntypedef unsigned int uint32_t;\ntypedef __int64 int64_t;\ntypedef unsigned __int64 uint64_t;\n// intptr_t and friends are defined in crtdefs.h through stdio.h.\n\n#else\n\n#include \n\n#endif\n\n// The following macro works on both 32 and 64-bit platforms.\n// Usage: instead of writing 0x1234567890123456\n// write UINT64_2PART_C(0x12345678,90123456);\n#define UINT64_2PART_C(a, b) (((static_cast(a) << 32) + 0x##b##u))\n\n\n// The expression ARRAY_SIZE(a) is a compile-time constant of type\n// size_t which represents the number of elements of the given\n// array. You should only use ARRAY_SIZE on statically allocated\n// arrays.\n#ifndef ARRAY_SIZE\n#define ARRAY_SIZE(a) \\\n ((sizeof(a) / sizeof(*(a))) / \\\n static_cast(!(sizeof(a) % sizeof(*(a)))))\n#endif\n\n// A macro to disallow the evil copy constructor and operator= functions\n// This should be used in the private: declarations for a class\n#ifndef DISALLOW_COPY_AND_ASSIGN\n#define DISALLOW_COPY_AND_ASSIGN(TypeName) \\\n TypeName(const TypeName&); \\\n void operator=(const TypeName&)\n#endif\n\n// A macro to disallow all the implicit constructors, namely the\n// default constructor, copy constructor and operator= functions.\n//\n// This should be used in the private: declarations for a class\n// that wants to prevent anyone from instantiating it. This is\n// especially useful for classes containing only static methods.\n#ifndef DISALLOW_IMPLICIT_CONSTRUCTORS\n#define DISALLOW_IMPLICIT_CONSTRUCTORS(TypeName) \\\n TypeName(); \\\n DISALLOW_COPY_AND_ASSIGN(TypeName)\n#endif\n\nnamespace double_conversion {\n\nstatic const int kCharSize = sizeof(char);\n\n// Returns the maximum of the two parameters.\ntemplate \nstatic T Max(T a, T b) {\n return a < b ? b : a;\n}\n\n\n// Returns the minimum of the two parameters.\ntemplate \nstatic T Min(T a, T b) {\n return a < b ? a : b;\n}\n\n\ninline int StrLength(const char* string) {\n size_t length = strlen(string);\n ASSERT(length == static_cast(static_cast(length)));\n return static_cast(length);\n}\n\n// This is a simplified version of V8's Vector class.\ntemplate \nclass Vector {\n public:\n Vector() : start_(NULL), length_(0) {}\n Vector(T* data, int length) : start_(data), length_(length) {\n ASSERT(length == 0 || (length > 0 && data != NULL));\n }\n\n // Returns a vector using the same backing storage as this one,\n // spanning from and including 'from', to but not including 'to'.\n Vector SubVector(int from, int to) {\n ASSERT(to <= length_);\n ASSERT(from < to);\n ASSERT(0 <= from);\n return Vector(start() + from, to - from);\n }\n\n // Returns the length of the vector.\n int length() const { return length_; }\n\n // Returns whether or not the vector is empty.\n bool is_empty() const { return length_ == 0; }\n\n // Returns the pointer to the start of the data in the vector.\n T* start() const { return start_; }\n\n // Access individual vector elements - checks bounds in debug mode.\n T& operator[](int index) const {\n ASSERT(0 <= index && index < length_);\n return start_[index];\n }\n\n T& first() { return start_[0]; }\n\n T& last() { return start_[length_ - 1]; }\n\n private:\n T* start_;\n int length_;\n};\n\n\n// Helper class for building result strings in a character buffer. The\n// purpose of the class is to use safe operations that checks the\n// buffer bounds on all operations in debug mode.\nclass StringBuilder {\n public:\n StringBuilder(char* buffer, int size)\n : buffer_(buffer, size), position_(0) { }\n\n ~StringBuilder() { if (!is_finalized()) Finalize(); }\n\n int size() const { return buffer_.length(); }\n\n // Get the current position in the builder.\n int position() const {\n ASSERT(!is_finalized());\n return position_;\n }\n\n // Reset the position.\n void Reset() { position_ = 0; }\n\n // Add a single character to the builder. It is not allowed to add\n // 0-characters; use the Finalize() method to terminate the string\n // instead.\n void AddCharacter(char c) {\n ASSERT(c != '\\0');\n ASSERT(!is_finalized() && position_ < buffer_.length());\n buffer_[position_++] = c;\n }\n\n // Add an entire string to the builder. Uses strlen() internally to\n // compute the length of the input string.\n void AddString(const char* s) {\n AddSubstring(s, StrLength(s));\n }\n\n // Add the first 'n' characters of the given string 's' to the\n // builder. The input string must have enough characters.\n void AddSubstring(const char* s, int n) {\n ASSERT(!is_finalized() && position_ + n < buffer_.length());\n ASSERT(static_cast(n) <= strlen(s));\n memmove(&buffer_[position_], s, n * kCharSize);\n position_ += n;\n }\n\n\n // Add character padding to the builder. If count is non-positive,\n // nothing is added to the builder.\n void AddPadding(char c, int count) {\n for (int i = 0; i < count; i++) {\n AddCharacter(c);\n }\n }\n\n // Finalize the string by 0-terminating it and returning the buffer.\n char* Finalize() {\n ASSERT(!is_finalized() && position_ < buffer_.length());\n buffer_[position_] = '\\0';\n // Make sure nobody managed to add a 0-character to the\n // buffer while building the string.\n ASSERT(strlen(buffer_.start()) == static_cast(position_));\n position_ = -1;\n ASSERT(is_finalized());\n return buffer_.start();\n }\n\n private:\n Vector buffer_;\n int position_;\n\n bool is_finalized() const { return position_ < 0; }\n\n DISALLOW_IMPLICIT_CONSTRUCTORS(StringBuilder);\n};\n\n// The type-based aliasing rule allows the compiler to assume that pointers of\n// different types (for some definition of different) never alias each other.\n// Thus the following code does not work:\n//\n// float f = foo();\n// int fbits = *(int*)(&f);\n//\n// The compiler 'knows' that the int pointer can't refer to f since the types\n// don't match, so the compiler may cache f in a register, leaving random data\n// in fbits. Using C++ style casts makes no difference, however a pointer to\n// char data is assumed to alias any other pointer. This is the 'memcpy\n// exception'.\n//\n// Bit_cast uses the memcpy exception to move the bits from a variable of one\n// type of a variable of another type. Of course the end result is likely to\n// be implementation dependent. Most compilers (gcc-4.2 and MSVC 2005)\n// will completely optimize BitCast away.\n//\n// There is an additional use for BitCast.\n// Recent gccs will warn when they see casts that may result in breakage due to\n// the type-based aliasing rule. If you have checked that there is no breakage\n// you can use BitCast to cast one pointer type to another. This confuses gcc\n// enough that it can no longer see that you have cast one pointer type to\n// another thus avoiding the warning.\ntemplate \ninline Dest BitCast(const Source& source) {\n // Compile time assertion: sizeof(Dest) == sizeof(Source)\n // A compile error here means your Dest and Source have different sizes.\n typedef char VerifySizesAreEqual[sizeof(Dest) == sizeof(Source) ? 1 : -1];\n\n Dest dest;\n memmove(&dest, &source, sizeof(dest));\n return dest;\n}\n\ntemplate \ninline Dest BitCast(Source* source) {\n return BitCast(reinterpret_cast(source));\n}\n\n} // namespace double_conversion\n\n#endif // DOUBLE_CONVERSION_UTILS_H_\n" }, { "path": "src/kenlm/util/ersatz_progress.cc", "content": "#include \"util/ersatz_progress.hh\"\n\n#include \n#include \n#include \n#include \n\nnamespace util {\n\nnamespace { const unsigned char kWidth = 100; }\n\nconst char kProgressBanner[] = \"----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100\\n\";\n\nErsatzProgress::ErsatzProgress() : current_(0), next_(std::numeric_limits::max()), complete_(next_), out_(NULL) {}\n\nErsatzProgress::~ErsatzProgress() {\n if (out_) Finished();\n}\n\nErsatzProgress::ErsatzProgress(uint64_t complete, std::ostream *to, const std::string &message) \n : current_(0), next_(complete / kWidth), complete_(complete), stones_written_(0), out_(to) {\n if (!out_) {\n next_ = std::numeric_limits::max();\n return;\n }\n if (!message.empty()) *out_ << message << '\\n';\n *out_ << kProgressBanner;\n}\n\nvoid ErsatzProgress::Milestone() {\n if (!out_) { current_ = 0; return; }\n if (!complete_) return;\n unsigned char stone = std::min(static_cast(kWidth), (current_ * kWidth) / complete_);\n\n for (; stones_written_ < stone; ++stones_written_) {\n (*out_) << '*';\n }\n if (stone == kWidth) {\n (*out_) << std::endl;\n next_ = std::numeric_limits::max();\n out_ = NULL;\n } else {\n next_ = std::max(next_, ((stone + 1) * complete_ + kWidth - 1) / kWidth);\n }\n}\n\n} // namespace util\n" }, { "path": "src/kenlm/util/ersatz_progress.hh", "content": "#ifndef UTIL_ERSATZ_PROGRESS__\n#define UTIL_ERSATZ_PROGRESS__\n\n#include \n#include \n\n#include \n\n// Ersatz version of boost::progress so core language model doesn't depend on\n// boost. Also adds option to print nothing. \n\nnamespace util {\n\nextern const char kProgressBanner[];\n\nclass ErsatzProgress {\n public:\n // No output. \n ErsatzProgress();\n\n // Null means no output. The null value is useful for passing along the ostream pointer from another caller. \n explicit ErsatzProgress(uint64_t complete, std::ostream *to = &std::cerr, const std::string &message = \"\");\n\n ~ErsatzProgress();\n\n ErsatzProgress &operator++() {\n if (++current_ >= next_) Milestone();\n return *this;\n }\n\n ErsatzProgress &operator+=(uint64_t amount) {\n if ((current_ += amount) >= next_) Milestone();\n return *this;\n }\n\n void Set(uint64_t to) {\n if ((current_ = to) >= next_) Milestone();\n }\n\n void Finished() {\n Set(complete_);\n }\n\n private:\n void Milestone();\n\n uint64_t current_, next_, complete_;\n unsigned char stones_written_;\n std::ostream *out_;\n\n // noncopyable\n ErsatzProgress(const ErsatzProgress &other);\n ErsatzProgress &operator=(const ErsatzProgress &other);\n};\n\n} // namespace util\n\n#endif // UTIL_ERSATZ_PROGRESS__\n" }, { "path": "src/kenlm/util/exception.cc", "content": "#include \"util/exception.hh\"\n\n#ifdef __GXX_RTTI\n#include \n#endif\n\n#include \n#include \n\nnamespace util {\n\nException::Exception() throw() {}\nException::~Exception() throw() {}\n\nException::Exception(const Exception &from) : std::exception() {\n stream_ << from.stream_.str();\n}\n\nException &Exception::operator=(const Exception &from) {\n stream_ << from.stream_.str();\n return *this;\n}\n\nconst char *Exception::what() const throw() {\n text_ = stream_.str();\n return text_.c_str();\n}\n\nvoid Exception::SetLocation(const char *file, unsigned int line, const char *func, const char *child_name, const char *condition) {\n /* The child class might have set some text, but we want this to come first.\n * Another option would be passing this information to the constructor, but\n * then child classes would have to accept constructor arguments and pass\n * them down. \n */\n text_ = stream_.str();\n stream_.str(\"\");\n stream_ << file << ':' << line;\n if (func) stream_ << \" in \" << func << \" threw \";\n if (child_name) {\n stream_ << child_name;\n } else {\n#ifdef __GXX_RTTI\n stream_ << typeid(this).name();\n#else\n stream_ << \"an exception\";\n#endif\n }\n if (condition) stream_ << \" because `\" << condition;\n stream_ << \"'.\\n\";\n stream_ << text_;\n}\n\nnamespace {\n// The XOPEN version.\nconst char *HandleStrerror(int ret, const char *buf) {\n if (!ret) return buf;\n return NULL;\n}\n\n// The GNU version.\nconst char *HandleStrerror(const char *ret, const char * /*buf*/) {\n return ret;\n}\n} // namespace\n\nErrnoException::ErrnoException() throw() : errno_(errno) {\n char buf[200];\n buf[0] = 0;\n#if defined(sun) || defined(_WIN32) || defined(_WIN64)\n const char *add = strerror(errno);\n#else\n const char *add = HandleStrerror(strerror_r(errno, buf, 200), buf);\n#endif\n\n if (add) {\n *this << add << ' ';\n }\n}\n\nErrnoException::~ErrnoException() throw() {}\n\nOverflowException::OverflowException() throw() {}\nOverflowException::~OverflowException() throw() {}\n\n} // namespace util\n" }, { "path": "src/kenlm/util/exception.hh", "content": "#ifndef UTIL_EXCEPTION__\n#define UTIL_EXCEPTION__\n\n#include \n#include \n#include \n#include \n\n#include \n\nnamespace util {\n\ntemplate typename Except::template ExceptionTag::Identity operator<<(Except &e, const Data &data);\n\nclass Exception : public std::exception {\n public:\n Exception() throw();\n virtual ~Exception() throw();\n\n Exception(const Exception &from);\n Exception &operator=(const Exception &from);\n\n // Not threadsafe, but probably doesn't matter. FWIW, Boost's exception guidance implies that what() isn't threadsafe. \n const char *what() const throw();\n\n // For use by the UTIL_THROW macros. \n void SetLocation(\n const char *file,\n unsigned int line,\n const char *func,\n const char *child_name,\n const char *condition);\n\n private:\n template friend typename Except::template ExceptionTag::Identity operator<<(Except &e, const Data &data);\n\n // This helps restrict operator<< defined below. \n template struct ExceptionTag {\n typedef T Identity;\n };\n\n std::stringstream stream_;\n mutable std::string text_;\n};\n\n/* This implements the normal operator<< for Exception and all its children. \n * SFINAE means it only applies to Exception. Think of this as an ersatz\n * boost::enable_if. \n */\ntemplate typename Except::template ExceptionTag::Identity operator<<(Except &e, const Data &data) {\n e.stream_ << data;\n return e;\n}\n\n#ifdef __GNUC__\n#define UTIL_FUNC_NAME __PRETTY_FUNCTION__\n#else\n#ifdef _WIN32\n#define UTIL_FUNC_NAME __FUNCTION__\n#else\n#define UTIL_FUNC_NAME NULL\n#endif\n#endif\n\n/* Create an instance of Exception, add the message Modify, and throw it.\n * Modify is appended to the what() message and can contain << for ostream\n * operations. \n *\n * do .. while kludge to swallow trailing ; character\n * http://gcc.gnu.org/onlinedocs/cpp/Swallowing-the-Semicolon.html . \n * Arg can be a constructor argument to the exception.\n */\n#define UTIL_THROW_BACKEND(Condition, Exception, Arg, Modify) do { \\\n Exception UTIL_e Arg; \\\n UTIL_e.SetLocation(__FILE__, __LINE__, UTIL_FUNC_NAME, #Exception, Condition); \\\n UTIL_e << Modify; \\\n throw UTIL_e; \\\n} while (0)\n\n#define UTIL_THROW_ARG(Exception, Arg, Modify) \\\n UTIL_THROW_BACKEND(NULL, Exception, Arg, Modify)\n\n#define UTIL_THROW(Exception, Modify) \\\n UTIL_THROW_BACKEND(NULL, Exception, , Modify);\n\n#if __GNUC__ >= 3\n#define UTIL_UNLIKELY(x) __builtin_expect (!!(x), 0)\n#else\n#define UTIL_UNLIKELY(x) (x)\n#endif\n\n#define UTIL_THROW_IF_ARG(Condition, Exception, Arg, Modify) do { \\\n if (UTIL_UNLIKELY(Condition)) { \\\n UTIL_THROW_BACKEND(#Condition, Exception, Arg, Modify); \\\n } \\\n} while (0)\n\n#define UTIL_THROW_IF(Condition, Exception, Modify) \\\n UTIL_THROW_IF_ARG(Condition, Exception, , Modify)\n\n// Exception that records errno and adds it to the message.\nclass ErrnoException : public Exception {\n public:\n ErrnoException() throw();\n\n virtual ~ErrnoException() throw();\n\n int Error() const throw() { return errno_; }\n\n private:\n int errno_;\n};\n\n// Utilities for overflow checking. \nclass OverflowException : public Exception {\n public:\n OverflowException() throw();\n ~OverflowException() throw();\n};\n\ntemplate inline std::size_t CheckOverflowInternal(uint64_t value) {\n UTIL_THROW_IF(value > static_cast(std::numeric_limits::max()), OverflowException, \"Integer overflow detected. This model is too big for 32-bit code.\");\n return value;\n}\n\ntemplate <> inline std::size_t CheckOverflowInternal<8>(uint64_t value) {\n return value;\n}\n\ninline std::size_t CheckOverflow(uint64_t value) {\n return CheckOverflowInternal(value);\n}\n\n} // namespace util\n\n#endif // UTIL_EXCEPTION__\n" }, { "path": "src/kenlm/util/fake_ofstream.hh", "content": "/* Like std::ofstream but without being incredibly slow. Backed by a raw fd.\n * Does not support many data types. Currently, it's targeted at writing ARPA\n * files quickly.\n */\n#include \"util/double-conversion/double-conversion.h\"\n#include \"util/double-conversion/utils.h\"\n#include \"util/file.hh\"\n#include \"util/scoped.hh\"\n#include \"util/string_piece.hh\"\n\n#define BOOST_LEXICAL_CAST_ASSUME_C_LOCALE\n#include \n\nnamespace util {\nclass FakeOFStream {\n public:\n static const std::size_t kOutBuf = 1048576;\n\n // Does not take ownership of out.\n explicit FakeOFStream(int out)\n : buf_(util::MallocOrThrow(kOutBuf)),\n builder_(static_cast(buf_.get()), kOutBuf),\n // Mostly the default but with inf instead. And no flags.\n convert_(double_conversion::DoubleToStringConverter::NO_FLAGS, \"inf\", \"NaN\", 'e', -6, 21, 6, 0),\n fd_(out) {}\n\n ~FakeOFStream() {\n Flush();\n }\n\n FakeOFStream &operator<<(float value) {\n // Odd, but this is the largest number found in the comments.\n EnsureRemaining(double_conversion::DoubleToStringConverter::kMaxPrecisionDigits + 8);\n convert_.ToShortestSingle(value, &builder_);\n return *this;\n }\n\n FakeOFStream &operator<<(double value) {\n EnsureRemaining(double_conversion::DoubleToStringConverter::kMaxPrecisionDigits + 8);\n convert_.ToShortest(value, &builder_);\n return *this;\n }\n\n FakeOFStream &operator<<(StringPiece str) {\n if (str.size() > kOutBuf) {\n Flush();\n util::WriteOrThrow(fd_, str.data(), str.size());\n } else {\n EnsureRemaining(str.size());\n builder_.AddSubstring(str.data(), str.size());\n }\n return *this;\n }\n\n // Inefficient! TODO: more efficient implementation\n FakeOFStream &operator<<(unsigned value) {\n return *this << boost::lexical_cast(value);\n }\n\n FakeOFStream &operator<<(char c) {\n EnsureRemaining(1);\n builder_.AddCharacter(c);\n return *this;\n }\n\n // Note this does not sync.\n void Flush() {\n util::WriteOrThrow(fd_, buf_.get(), builder_.position());\n builder_.Reset();\n }\n\n private:\n void EnsureRemaining(std::size_t amount) {\n if (static_cast(builder_.size() - builder_.position()) < amount) {\n Flush();\n }\n }\n\n util::scoped_malloc buf_;\n double_conversion::StringBuilder builder_;\n double_conversion::DoubleToStringConverter convert_;\n int fd_;\n};\n\n} // namespace\n" }, { "path": "src/kenlm/util/file.cc", "content": "#define _LARGEFILE64_SOURCE\n#define _FILE_OFFSET_BITS 64\n\n#include \"util/file.hh\"\n\n#include \"util/exception.hh\"\n\n#include \n#include \n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#if defined(_WIN32) || defined(_WIN64)\n#include \n#include \n#include \n#include \n#include \n#else\n#include \n#endif\n\nnamespace util {\n\nscoped_fd::~scoped_fd() {\n if (fd_ != -1 && close(fd_)) {\n std::cerr << \"Could not close file \" << fd_ << std::endl;\n std::abort();\n }\n}\n\nscoped_FILE::~scoped_FILE() {\n if (file_ && std::fclose(file_)) {\n std::cerr << \"Could not close file \" << std::endl;\n std::abort();\n }\n}\n\n// Note that ErrnoException records errno before NameFromFD is called.\nFDException::FDException(int fd) throw() : fd_(fd), name_guess_(NameFromFD(fd)) {\n *this << \"in \" << name_guess_ << ' ';\n}\n\nFDException::~FDException() throw() {}\n\nEndOfFileException::EndOfFileException() throw() {\n *this << \"End of file\";\n}\nEndOfFileException::~EndOfFileException() throw() {}\n\nint OpenReadOrThrow(const char *name) {\n int ret;\n#if defined(_WIN32) || defined(_WIN64)\n UTIL_THROW_IF(-1 == (ret = _open(name, _O_BINARY | _O_RDONLY)), ErrnoException, \"while opening \" << name);\n#else\n UTIL_THROW_IF(-1 == (ret = open(name, O_RDONLY)), ErrnoException, \"while opening \" << name);\n#endif\n return ret;\n}\n\nint CreateOrThrow(const char *name) {\n int ret;\n#if defined(_WIN32) || defined(_WIN64)\n UTIL_THROW_IF(-1 == (ret = _open(name, _O_CREAT | _O_TRUNC | _O_RDWR | _O_BINARY, _S_IREAD | _S_IWRITE)), ErrnoException, \"while creating \" << name);\n#else\n UTIL_THROW_IF(-1 == (ret = open(name, O_CREAT | O_TRUNC | O_RDWR, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH)), ErrnoException, \"while creating \" << name);\n#endif\n return ret;\n}\n\nuint64_t SizeFile(int fd) {\n#if defined(_WIN32) || defined(_WIN64)\n __int64 ret = _filelengthi64(fd);\n return (ret == -1) ? kBadSize : ret;\n#else // Not windows.\n\n#ifdef OS_ANDROID\n struct stat64 sb;\n int ret = fstat64(fd, &sb);\n#else\n struct stat sb;\n int ret = fstat(fd, &sb);\n#endif\n if (ret == -1 || (!sb.st_size && !S_ISREG(sb.st_mode))) return kBadSize;\n return sb.st_size;\n#endif\n}\n\nuint64_t SizeOrThrow(int fd) {\n uint64_t ret = SizeFile(fd);\n UTIL_THROW_IF_ARG(ret == kBadSize, FDException, (fd), \"Failed to size\");\n return ret;\n}\n\nvoid ResizeOrThrow(int fd, uint64_t to) {\n#if defined(_WIN32) || defined(_WIN64)\n errno_t ret = _chsize_s\n#elif defined(OS_ANDROID)\n int ret = ftruncate64\n#else\n int ret = ftruncate\n#endif\n (fd, to);\n UTIL_THROW_IF_ARG(ret, FDException, (fd), \"while resizing to \" << to << \" bytes\");\n}\n\nstd::size_t PartialRead(int fd, void *to, std::size_t amount) {\n#if defined(_WIN32) || defined(_WIN64)\n amount = min(static_cast(INT_MAX), amount);\n int ret = _read(fd, to, amount); \n#else\n errno = 0;\n ssize_t ret;\n do {\n ret = read(fd, to, amount);\n } while (ret == -1 && errno == EINTR);\n#endif\n UTIL_THROW_IF_ARG(ret < 0, FDException, (fd), \"while reading \" << amount << \" bytes\");\n return static_cast(ret);\n}\n\nvoid ReadOrThrow(int fd, void *to_void, std::size_t amount) {\n uint8_t *to = static_cast(to_void);\n while (amount) {\n std::size_t ret = PartialRead(fd, to, amount);\n UTIL_THROW_IF(ret == 0, EndOfFileException, \" in \" << NameFromFD(fd) << \" but there should be \" << amount << \" more bytes to read.\");\n amount -= ret;\n to += ret;\n }\n}\n\nstd::size_t ReadOrEOF(int fd, void *to_void, std::size_t amount) {\n uint8_t *to = static_cast(to_void);\n std::size_t remaining = amount;\n while (remaining) {\n std::size_t ret = PartialRead(fd, to, remaining);\n if (!ret) return amount - remaining;\n remaining -= ret;\n to += ret;\n }\n return amount;\n}\n\nvoid PReadOrThrow(int fd, void *to_void, std::size_t size, uint64_t off) {\n uint8_t *to = static_cast(to_void);\n#if defined(_WIN32) || defined(_WIN64)\n UTIL_THROW(Exception, \"This pread implementation for windows is broken. Please send me a patch that does not change the file pointer. Atomically. Or send me an implementation of pwrite that is allowed to change the file pointer but can be called concurrently with pread.\");\n const std::size_t kMaxDWORD = static_cast(4294967295UL);\n#endif\n for (;size ;) {\n#if defined(_WIN32) || defined(_WIN64)\n /* BROKEN: changes file pointer. Even if you save it and change it back, it won't be safe to use concurrently with write() or read() which lmplz does. */\n // size_t might be 64-bit. DWORD is always 32.\n DWORD reading = static_cast(std::min(kMaxDWORD, size));\n DWORD ret;\n OVERLAPPED overlapped;\n memset(&overlapped, 0, sizeof(OVERLAPPED));\n overlapped.Offset = static_cast(off);\n overlapped.OffsetHigh = static_cast(off >> 32);\n UTIL_THROW_IF(!ReadFile((HANDLE)_get_osfhandle(fd), to, reading, &ret, &overlapped), Exception, \"ReadFile failed for offset \" << off);\n#else\n ssize_t ret;\n errno = 0;\n do {\n#ifdef OS_ANDROID\n ret = pread64(fd, to, size, off);\n#else\n ret = pread(fd, to, size, off);\n#endif\n } while (ret == -1 && errno == EINTR);\n if (ret <= 0) {\n UTIL_THROW_IF(ret == 0, EndOfFileException, \" for reading \" << size << \" bytes at \" << off << \" from \" << NameFromFD(fd));\n UTIL_THROW_ARG(FDException, (fd), \"while reading \" << size << \" bytes at offset \" << off);\n }\n#endif\n size -= ret;\n off += ret;\n to += ret;\n }\n}\n\nvoid WriteOrThrow(int fd, const void *data_void, std::size_t size) {\n const uint8_t *data = static_cast(data_void);\n while (size) {\n#if defined(_WIN32) || defined(_WIN64)\n int ret = write(fd, data, min(static_cast(INT_MAX), size));\n#else\n errno = 0;\n ssize_t ret;\n do {\n ret = write(fd, data, size);\n } while (ret == -1 && errno == EINTR);\n#endif\n UTIL_THROW_IF_ARG(ret < 1, FDException, (fd), \"while writing \" << size << \" bytes\");\n data += ret;\n size -= ret;\n }\n}\n\nvoid WriteOrThrow(FILE *to, const void *data, std::size_t size) {\n if (!size) return;\n UTIL_THROW_IF(1 != std::fwrite(data, size, 1, to), ErrnoException, \"Short write; requested size \" << size);\n}\n\nvoid FSyncOrThrow(int fd) {\n// Apparently windows doesn't have fsync? \n#if !defined(_WIN32) && !defined(_WIN64)\n UTIL_THROW_IF_ARG(-1 == fsync(fd), FDException, (fd), \"while syncing\");\n#endif\n}\n\nnamespace {\n\n// Static assert for 64-bit off_t size.\n#if !defined(_WIN32) && !defined(_WIN64) && !defined(OS_ANDROID)\ntemplate struct CheckOffT;\ntemplate <> struct CheckOffT<8> {\n struct True {};\n};\n// If there's a compiler error on the next line, then off_t isn't 64 bit. And\n// that makes me a sad panda.\ntypedef CheckOffT::True IgnoredType;\n#endif\n\n// Can't we all just get along? \nvoid InternalSeek(int fd, int64_t off, int whence) {\n if (\n#if defined(_WIN32) || defined(_WIN64)\n (__int64)-1 == _lseeki64(fd, off, whence)\n#elif defined(OS_ANDROID)\n (off64_t)-1 == lseek64(fd, off, whence)\n#else\n (off_t)-1 == lseek(fd, off, whence)\n#endif\n ) UTIL_THROW_ARG(FDException, (fd), \"while seeking to \" << off << \" whence \" << whence);\n}\n} // namespace\n\nvoid SeekOrThrow(int fd, uint64_t off) {\n InternalSeek(fd, off, SEEK_SET);\n}\n\nvoid AdvanceOrThrow(int fd, int64_t off) {\n InternalSeek(fd, off, SEEK_CUR);\n}\n\nvoid SeekEnd(int fd) {\n InternalSeek(fd, 0, SEEK_END);\n}\n\nstd::FILE *FDOpenOrThrow(scoped_fd &file) {\n std::FILE *ret = fdopen(file.get(), \"r+b\");\n UTIL_THROW_IF_ARG(!ret, FDException, (file.get()), \"Could not fdopen for write\");\n file.release();\n return ret;\n}\n\nstd::FILE *FDOpenReadOrThrow(scoped_fd &file) {\n std::FILE *ret = fdopen(file.get(), \"rb\");\n UTIL_THROW_IF_ARG(!ret, FDException, (file.get()), \"Could not fdopen for read\");\n file.release();\n return ret;\n}\n\n// Sigh. Windows temporary file creation is full of race conditions.\n#if defined(_WIN32) || defined(_WIN64)\n/* mkstemp extracted from libc/sysdeps/posix/tempname.c. Copyright\n (C) 1991-1999, 2000, 2001, 2006 Free Software Foundation, Inc.\n\n The GNU C Library is free software; you can redistribute it and/or\n modify it under the terms of the GNU Lesser General Public\n License as published by the Free Software Foundation; either\n version 2.1 of the License, or (at your option) any later version. */\n\n/* This has been modified from the original version to rename the function and\n * set the Windows temporary flag. */\n\nstatic const char letters[] =\n\"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789\";\n\n/* Generate a temporary file name based on TMPL. TMPL must match the\n rules for mk[s]temp (i.e. end in \"XXXXXX\"). The name constructed\n does not exist at the time of the call to mkstemp. TMPL is\n overwritten with the result. */\nint\nmkstemp_and_unlink(char *tmpl)\n{\n int len;\n char *XXXXXX;\n static unsigned long long value;\n unsigned long long random_time_bits;\n unsigned int count;\n int fd = -1;\n int save_errno = errno;\n\n /* A lower bound on the number of temporary files to attempt to\n generate. The maximum total number of temporary file names that\n can exist for a given template is 62**6. It should never be\n necessary to try all these combinations. Instead if a reasonable\n number of names is tried (we define reasonable as 62**3) fail to\n give the system administrator the chance to remove the problems. */\n#define ATTEMPTS_MIN (62 * 62 * 62)\n\n /* The number of times to attempt to generate a temporary file. To\n conform to POSIX, this must be no smaller than TMP_MAX. */\n#if ATTEMPTS_MIN < TMP_MAX\n unsigned int attempts = TMP_MAX;\n#else\n unsigned int attempts = ATTEMPTS_MIN;\n#endif\n\n len = strlen (tmpl);\n if (len < 6 || strcmp (&tmpl[len - 6], \"XXXXXX\"))\n {\n errno = EINVAL;\n return -1;\n }\n\n/* This is where the Xs start. */\n XXXXXX = &tmpl[len - 6];\n\n /* Get some more or less random data. */\n {\n SYSTEMTIME stNow;\n FILETIME ftNow;\n\n // get system time\n GetSystemTime(&stNow);\n stNow.wMilliseconds = 500;\n if (!SystemTimeToFileTime(&stNow, &ftNow))\n {\n errno = -1;\n return -1;\n }\n\n random_time_bits = (((unsigned long long)ftNow.dwHighDateTime << 32)\n | (unsigned long long)ftNow.dwLowDateTime);\n }\n value += random_time_bits ^ (unsigned long long)GetCurrentThreadId ();\n\n for (count = 0; count < attempts; value += 7777, ++count)\n {\n unsigned long long v = value;\n\n /* Fill in the random bits. */\n XXXXXX[0] = letters[v % 62];\n v /= 62;\n XXXXXX[1] = letters[v % 62];\n v /= 62;\n XXXXXX[2] = letters[v % 62];\n v /= 62;\n XXXXXX[3] = letters[v % 62];\n v /= 62;\n XXXXXX[4] = letters[v % 62];\n v /= 62;\n XXXXXX[5] = letters[v % 62];\n\n /* Modified for windows and to unlink */\n // fd = open (tmpl, O_RDWR | O_CREAT | O_EXCL, _S_IREAD | _S_IWRITE);\n int flags = _O_RDWR | _O_CREAT | _O_EXCL | _O_BINARY;\n flags |= _O_TEMPORARY;\n fd = _open (tmpl, flags, _S_IREAD | _S_IWRITE);\n if (fd >= 0)\n {\n errno = save_errno;\n return fd;\n }\n else if (errno != EEXIST)\n return -1;\n }\n\n /* We got out of the loop because we ran out of combinations to try. */\n errno = EEXIST;\n return -1;\n}\n#else\nint\nmkstemp_and_unlink(char *tmpl) {\n int ret = mkstemp(tmpl);\n if (ret != -1) {\n UTIL_THROW_IF(unlink(tmpl), ErrnoException, \"while deleting delete \" << tmpl);\n }\n return ret;\n}\n#endif\n\n// If it's a directory, add a /. This lets users say -T /tmp without creating\n// /tmpAAAAAA\nvoid NormalizeTempPrefix(std::string &base) {\n if (base.empty()) return;\n if (base[base.size() - 1] == '/') return;\n struct stat sb;\n // It's fine for it to not exist.\n if (-1 == stat(base.c_str(), &sb)) return;\n if (\n#if defined(_WIN32) || defined(_WIN64)\n sb.st_mode & _S_IFDIR\n#else\n S_ISDIR(sb.st_mode)\n#endif\n ) base += '/';\n}\n\nint MakeTemp(const std::string &base) {\n std::string name(base);\n name += \"XXXXXX\";\n name.push_back(0);\n int ret;\n UTIL_THROW_IF(-1 == (ret = mkstemp_and_unlink(&name[0])), ErrnoException, \"while making a temporary based on \" << base);\n return ret;\n}\n\nstd::FILE *FMakeTemp(const std::string &base) {\n util::scoped_fd file(MakeTemp(base));\n return FDOpenOrThrow(file);\n}\n\nint DupOrThrow(int fd) {\n int ret = dup(fd);\n UTIL_THROW_IF_ARG(ret == -1, FDException, (fd), \"in duplicating the file descriptor\");\n return ret;\n}\n\nnamespace {\n// Try to name things but be willing to fail too.\nbool TryName(int fd, std::string &out) {\n#if defined(_WIN32) || defined(_WIN64)\n return false;\n#else\n std::string name(\"/proc/self/fd/\");\n std::ostringstream convert;\n convert << fd;\n name += convert.str();\n \n struct stat sb;\n if (-1 == lstat(name.c_str(), &sb)) \n return false;\n out.resize(sb.st_size + 1);\n ssize_t ret = readlink(name.c_str(), &out[0], sb.st_size + 1);\n if (-1 == ret)\n return false;\n if (ret > sb.st_size) {\n // Increased in size?!\n return false;\n }\n out.resize(ret);\n // Don't use the non-file names.\n if (!out.empty() && out[0] != '/') \n return false;\n return true;\n#endif\n}\n} // namespace\n\nstd::string NameFromFD(int fd) {\n std::string ret;\n if (TryName(fd, ret)) return ret;\n switch (fd) {\n case 0: return \"stdin\";\n case 1: return \"stdout\";\n case 2: return \"stderr\";\n }\n ret = \"fd \";\n std::ostringstream convert;\n convert << fd;\n ret += convert.str();\n return ret;\n}\n\n} // namespace util\n" }, { "path": "src/kenlm/util/file.hh", "content": "#ifndef UTIL_FILE__\n#define UTIL_FILE__\n\n#include \"util/exception.hh\"\n\n#include \n#include \n#include \n\n#include \n\nnamespace util {\n\nclass scoped_fd {\n public:\n scoped_fd() : fd_(-1) {}\n\n explicit scoped_fd(int fd) : fd_(fd) {}\n\n ~scoped_fd();\n\n void reset(int to = -1) {\n scoped_fd other(fd_);\n fd_ = to;\n }\n\n int get() const { return fd_; }\n\n int operator*() const { return fd_; }\n\n int release() {\n int ret = fd_;\n fd_ = -1;\n return ret;\n }\n\n private:\n int fd_;\n\n scoped_fd(const scoped_fd &);\n scoped_fd &operator=(const scoped_fd &);\n};\n\nclass scoped_FILE {\n public:\n explicit scoped_FILE(std::FILE *file = NULL) : file_(file) {}\n\n ~scoped_FILE();\n\n std::FILE *get() { return file_; }\n const std::FILE *get() const { return file_; }\n\n void reset(std::FILE *to = NULL) {\n scoped_FILE other(file_);\n file_ = to;\n }\n\n std::FILE *release() {\n std::FILE *ret = file_;\n file_ = NULL;\n return ret;\n }\n\n private:\n std::FILE *file_;\n};\n\n/* Thrown for any operation where the fd is known. */\nclass FDException : public ErrnoException {\n public:\n explicit FDException(int fd) throw();\n\n virtual ~FDException() throw();\n\n // This may no longer be valid if the exception was thrown past open.\n int FD() const { return fd_; }\n\n // Guess from NameFromFD.\n const std::string &NameGuess() const { return name_guess_; }\n\n private:\n int fd_;\n\n std::string name_guess_;\n};\n\n// End of file reached.\nclass EndOfFileException : public Exception {\n public:\n EndOfFileException() throw();\n ~EndOfFileException() throw();\n};\n\n// Open for read only. \nint OpenReadOrThrow(const char *name);\n// Create file if it doesn't exist, truncate if it does. Opened for write. \nint CreateOrThrow(const char *name);\n\n// Return value for SizeFile when it can't size properly. \nconst uint64_t kBadSize = (uint64_t)-1;\nuint64_t SizeFile(int fd);\nuint64_t SizeOrThrow(int fd);\n\nvoid ResizeOrThrow(int fd, uint64_t to);\n\nstd::size_t PartialRead(int fd, void *to, std::size_t size);\nvoid ReadOrThrow(int fd, void *to, std::size_t size);\nstd::size_t ReadOrEOF(int fd, void *to_void, std::size_t size);\n// Positioned: unix only for now. \nvoid PReadOrThrow(int fd, void *to, std::size_t size, uint64_t off);\n\nvoid WriteOrThrow(int fd, const void *data_void, std::size_t size);\nvoid WriteOrThrow(FILE *to, const void *data, std::size_t size);\n\nvoid FSyncOrThrow(int fd);\n\n// Seeking\nvoid SeekOrThrow(int fd, uint64_t off);\nvoid AdvanceOrThrow(int fd, int64_t off);\nvoid SeekEnd(int fd);\n\nstd::FILE *FDOpenOrThrow(scoped_fd &file);\nstd::FILE *FDOpenReadOrThrow(scoped_fd &file);\n\n// Temporary files\n// Append a / if base is a directory.\nvoid NormalizeTempPrefix(std::string &base);\nint MakeTemp(const std::string &prefix);\nstd::FILE *FMakeTemp(const std::string &prefix);\n\n// dup an fd.\nint DupOrThrow(int fd);\n\n/* Attempt get file name from fd. This won't always work (i.e. on Windows or\n * a pipe). The file might have been renamed. It's intended for diagnostics\n * and logging only.\n */\nstd::string NameFromFD(int fd);\n\n} // namespace util\n\n#endif // UTIL_FILE__\n" }, { "path": "src/kenlm/util/file_piece.cc", "content": "#include \"util/file_piece.hh\"\n\n#include \"util/double-conversion/double-conversion.h\"\n#include \"util/exception.hh\"\n#include \"util/file.hh\"\n#include \"util/mmap.hh\"\n\n#if defined(_WIN32) || defined(_WIN64)\n#include \n#else\n#include \n#endif\n\n#include \n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n\nnamespace util {\n\nParseNumberException::ParseNumberException(StringPiece value) throw() {\n *this << \"Could not parse \\\"\" << value << \"\\\" into a number\";\n}\n\n// Sigh this is the only way I could come up with to do a _const_ bool. It has ' ', '\\f', '\\n', '\\r', '\\t', and '\\v' (same as isspace on C locale). \nconst bool kSpaces[256] = {0,0,0,0,0,0,0,0,0,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};\n\nFilePiece::FilePiece(const char *name, std::ostream *show_progress, std::size_t min_buffer) : \n file_(OpenReadOrThrow(name)), total_size_(SizeFile(file_.get())), page_(SizePage()),\n progress_(total_size_, total_size_ == kBadSize ? NULL : show_progress, std::string(\"Reading \") + name) {\n Initialize(name, show_progress, min_buffer);\n}\n\nnamespace {\nstd::string NamePossiblyFind(int fd, const char *name) {\n if (name) return name;\n return NameFromFD(fd);\n}\n} // namespace\n\nFilePiece::FilePiece(int fd, const char *name, std::ostream *show_progress, std::size_t min_buffer) : \n file_(fd), total_size_(SizeFile(file_.get())), page_(SizePage()),\n progress_(total_size_, total_size_ == kBadSize ? NULL : show_progress, std::string(\"Reading \") + NamePossiblyFind(fd, name)) {\n Initialize(NamePossiblyFind(fd, name).c_str(), show_progress, min_buffer);\n}\n\nFilePiece::FilePiece(std::istream &stream, const char *name, std::size_t min_buffer) :\n total_size_(kBadSize), page_(SizePage()) {\n InitializeNoRead(\"istream\", min_buffer);\n\n fallback_to_read_ = true;\n data_.reset(MallocOrThrow(default_map_size_), default_map_size_, scoped_memory::MALLOC_ALLOCATED);\n position_ = data_.begin();\n position_end_ = position_;\n \n fell_back_.Reset(stream);\n}\n\nFilePiece::~FilePiece() {}\n\nStringPiece FilePiece::ReadLine(char delim) {\n std::size_t skip = 0;\n while (true) {\n for (const char *i = position_ + skip; i < position_end_; ++i) {\n if (*i == delim) {\n StringPiece ret(position_, i - position_);\n position_ = i + 1;\n return ret;\n }\n }\n if (at_end_) {\n if (position_ == position_end_) Shift();\n return Consume(position_end_);\n }\n skip = position_end_ - position_;\n Shift();\n }\n}\n\nfloat FilePiece::ReadFloat() {\n return ReadNumber();\n}\ndouble FilePiece::ReadDouble() {\n return ReadNumber();\n}\nlong int FilePiece::ReadLong() {\n return ReadNumber();\n}\nunsigned long int FilePiece::ReadULong() {\n return ReadNumber();\n}\n\n// Factored out so that istream can call this.\nvoid FilePiece::InitializeNoRead(const char *name, std::size_t min_buffer) {\n file_name_ = name;\n\n default_map_size_ = page_ * std::max((min_buffer / page_ + 1), 2);\n position_ = NULL;\n position_end_ = NULL;\n mapped_offset_ = 0;\n at_end_ = false;\n}\n\nvoid FilePiece::Initialize(const char *name, std::ostream *show_progress, std::size_t min_buffer) {\n InitializeNoRead(name, min_buffer);\n\n if (total_size_ == kBadSize) {\n // So the assertion passes. \n fallback_to_read_ = false;\n if (show_progress) \n *show_progress << \"File \" << name << \" isn't normal. Using slower read() instead of mmap(). No progress bar.\" << std::endl;\n TransitionToRead();\n } else {\n fallback_to_read_ = false;\n }\n Shift();\n // gzip detect.\n if ((position_end_ >= position_ + ReadCompressed::kMagicSize) && ReadCompressed::DetectCompressedMagic(position_)) {\n if (!fallback_to_read_) {\n at_end_ = false;\n TransitionToRead();\n }\n }\n}\n\nnamespace {\n\nstatic const double_conversion::StringToDoubleConverter kConverter(\n double_conversion::StringToDoubleConverter::ALLOW_TRAILING_JUNK | double_conversion::StringToDoubleConverter::ALLOW_LEADING_SPACES,\n std::numeric_limits::quiet_NaN(),\n std::numeric_limits::quiet_NaN(),\n \"inf\",\n \"NaN\");\n\nvoid ParseNumber(const char *begin, const char *&end, float &out) {\n int count;\n out = kConverter.StringToFloat(begin, end - begin, &count);\n end = begin + count;\n}\nvoid ParseNumber(const char *begin, const char *&end, double &out) {\n int count;\n out = kConverter.StringToDouble(begin, end - begin, &count);\n end = begin + count;\n}\nvoid ParseNumber(const char *begin, const char *&end, long int &out) {\n char *silly_end;\n out = strtol(begin, &silly_end, 10);\n end = silly_end;\n}\nvoid ParseNumber(const char *begin, const char *&end, unsigned long int &out) {\n char *silly_end;\n out = strtoul(begin, &silly_end, 10);\n end = silly_end;\n}\n} // namespace\n\ntemplate T FilePiece::ReadNumber() {\n SkipSpaces();\n while (last_space_ < position_) {\n if (at_end_) {\n // Hallucinate a null off the end of the file.\n std::string buffer(position_, position_end_);\n const char *buf = buffer.c_str();\n const char *end = buf + buffer.size();\n T ret;\n ParseNumber(buf, end, ret);\n if (buf == end) throw ParseNumberException(buffer);\n position_ += end - buf;\n return ret;\n }\n Shift();\n }\n const char *end = last_space_;\n T ret;\n ParseNumber(position_, end, ret);\n if (end == position_) throw ParseNumberException(ReadDelimited());\n position_ = end;\n return ret;\n}\n\nconst char *FilePiece::FindDelimiterOrEOF(const bool *delim) {\n std::size_t skip = 0;\n while (true) {\n for (const char *i = position_ + skip; i < position_end_; ++i) {\n if (delim[static_cast(*i)]) return i;\n }\n if (at_end_) {\n if (position_ == position_end_) Shift();\n return position_end_;\n }\n skip = position_end_ - position_;\n Shift();\n }\n}\n\nvoid FilePiece::Shift() {\n if (at_end_) {\n progress_.Finished();\n throw EndOfFileException();\n }\n uint64_t desired_begin = position_ - data_.begin() + mapped_offset_;\n\n if (!fallback_to_read_) MMapShift(desired_begin);\n // Notice an mmap failure might set the fallback. \n if (fallback_to_read_) ReadShift();\n\n for (last_space_ = position_end_ - 1; last_space_ >= position_; --last_space_) {\n if (kSpaces[static_cast(*last_space_)]) break;\n }\n}\n\nvoid FilePiece::MMapShift(uint64_t desired_begin) {\n // Use mmap. \n uint64_t ignore = desired_begin % page_;\n // Duplicate request for Shift means give more data. \n if (position_ == data_.begin() + ignore && position_) {\n default_map_size_ *= 2;\n }\n // Local version so that in case of failure it doesn't overwrite the class variable. \n uint64_t mapped_offset = desired_begin - ignore;\n\n uint64_t mapped_size;\n if (default_map_size_ >= static_cast(total_size_ - mapped_offset)) {\n at_end_ = true;\n mapped_size = total_size_ - mapped_offset;\n } else {\n mapped_size = default_map_size_;\n }\n\n // Forcibly clear the existing mmap first. \n data_.reset();\n try {\n MapRead(POPULATE_OR_LAZY, *file_, mapped_offset, mapped_size, data_);\n } catch (const util::ErrnoException &e) {\n if (desired_begin) {\n SeekOrThrow(*file_, desired_begin);\n }\n // The mmap was scheduled to end the file, but now we're going to read it. \n at_end_ = false;\n TransitionToRead();\n return;\n }\n mapped_offset_ = mapped_offset;\n position_ = data_.begin() + ignore;\n position_end_ = data_.begin() + mapped_size;\n\n progress_.Set(desired_begin);\n}\n\nvoid FilePiece::TransitionToRead() {\n assert(!fallback_to_read_);\n fallback_to_read_ = true;\n data_.reset();\n data_.reset(MallocOrThrow(default_map_size_), default_map_size_, scoped_memory::MALLOC_ALLOCATED);\n position_ = data_.begin();\n position_end_ = position_;\n\n try {\n fell_back_.Reset(file_.release());\n } catch (util::Exception &e) {\n e << \" in file \" << file_name_;\n throw;\n }\n}\n\nvoid FilePiece::ReadShift() {\n assert(fallback_to_read_);\n // Bytes [data_.begin(), position_) have been consumed. \n // Bytes [position_, position_end_) have been read into the buffer. \n\n // Start at the beginning of the buffer if there's nothing useful in it. \n if (position_ == position_end_) {\n mapped_offset_ += (position_end_ - data_.begin());\n position_ = data_.begin();\n position_end_ = position_;\n }\n\n std::size_t already_read = position_end_ - data_.begin();\n\n if (already_read == default_map_size_) {\n if (position_ == data_.begin()) {\n // Buffer too small. \n std::size_t valid_length = position_end_ - position_;\n default_map_size_ *= 2;\n data_.call_realloc(default_map_size_);\n UTIL_THROW_IF(!data_.get(), ErrnoException, \"realloc failed for \" << default_map_size_);\n position_ = data_.begin();\n position_end_ = position_ + valid_length;\n } else {\n std::size_t moving = position_end_ - position_;\n memmove(data_.get(), position_, moving);\n position_ = data_.begin();\n position_end_ = position_ + moving;\n already_read = moving;\n }\n }\n\n std::size_t read_return = fell_back_.Read(static_cast(data_.get()) + already_read, default_map_size_ - already_read);\n progress_.Set(fell_back_.RawAmount());\n\n if (read_return == 0) {\n at_end_ = true;\n }\n position_end_ += read_return;\n}\n\n} // namespace util\n" }, { "path": "src/kenlm/util/file_piece.hh", "content": "#ifndef UTIL_FILE_PIECE__\n#define UTIL_FILE_PIECE__\n\n#include \"util/ersatz_progress.hh\"\n#include \"util/exception.hh\"\n#include \"util/file.hh\"\n#include \"util/mmap.hh\"\n#include \"util/read_compressed.hh\"\n#include \"util/string_piece.hh\"\n\n#include \n#include \n#include \n\n#include \n\nnamespace util {\n\nclass ParseNumberException : public Exception {\n public:\n explicit ParseNumberException(StringPiece value) throw();\n ~ParseNumberException() throw() {}\n};\n\nextern const bool kSpaces[256];\n\n// Memory backing the returned StringPiece may vanish on the next call.\nclass FilePiece {\n public:\n // 1 MB default.\n explicit FilePiece(const char *file, std::ostream *show_progress = NULL, std::size_t min_buffer = 1048576);\n // Takes ownership of fd. name is used for messages.\n explicit FilePiece(int fd, const char *name = NULL, std::ostream *show_progress = NULL, std::size_t min_buffer = 1048576);\n\n /* Read from an istream. Don't use this if you can avoid it. Raw fd IO is\n * much faster. But sometimes you just have an istream like Boost's HTTP\n * server and want to parse it the same way.\n * name is just used for messages and FileName().\n */\n explicit FilePiece(std::istream &stream, const char *name = NULL, std::size_t min_buffer = 1048576);\n\n ~FilePiece();\n\n char get() {\n if (position_ == position_end_) {\n Shift();\n if (at_end_) throw EndOfFileException();\n }\n return *(position_++);\n }\n\n // Leaves the delimiter, if any, to be returned by get(). Delimiters defined by isspace().\n StringPiece ReadDelimited(const bool *delim = kSpaces) {\n SkipSpaces(delim);\n return Consume(FindDelimiterOrEOF(delim));\n }\n\n // Unlike ReadDelimited, this includes leading spaces and consumes the delimiter.\n // It is similar to getline in that way.\n StringPiece ReadLine(char delim = '\\n');\n\n float ReadFloat();\n double ReadDouble();\n long int ReadLong();\n unsigned long int ReadULong();\n\n // Skip spaces defined by isspace.\n void SkipSpaces(const bool *delim = kSpaces) {\n for (; ; ++position_) {\n if (position_ == position_end_) Shift();\n if (!delim[static_cast(*position_)]) return;\n }\n }\n\n uint64_t Offset() const {\n return position_ - data_.begin() + mapped_offset_;\n }\n\n const std::string &FileName() const { return file_name_; }\n\n private:\n void InitializeNoRead(const char *name, std::size_t min_buffer);\n // Calls InitializeNoRead, so don't call both.\n void Initialize(const char *name, std::ostream *show_progress, std::size_t min_buffer);\n\n template T ReadNumber();\n\n StringPiece Consume(const char *to) {\n StringPiece ret(position_, to - position_);\n position_ = to;\n return ret;\n }\n\n const char *FindDelimiterOrEOF(const bool *delim = kSpaces);\n\n void Shift();\n // Backends to Shift().\n void MMapShift(uint64_t desired_begin);\n\n void TransitionToRead();\n void ReadShift();\n\n const char *position_, *last_space_, *position_end_;\n\n scoped_fd file_;\n const uint64_t total_size_;\n const uint64_t page_;\n\n std::size_t default_map_size_;\n uint64_t mapped_offset_;\n\n // Order matters: file_ should always be destroyed after this.\n scoped_memory data_;\n\n bool at_end_;\n bool fallback_to_read_;\n\n ErsatzProgress progress_;\n\n std::string file_name_;\n\n ReadCompressed fell_back_;\n};\n\n} // namespace util\n\n#endif // UTIL_FILE_PIECE__\n" }, { "path": "src/kenlm/util/file_piece_test.cc", "content": "// Tests might fail if you have creative characters in your path. Sue me. \n#include \"util/file_piece.hh\"\n\n#include \"util/file.hh\"\n#include \"util/scoped.hh\"\n\n#define BOOST_TEST_MODULE FilePieceTest\n#include \n#include \n#include \n\n#include \n#include \n#include \n\nnamespace util {\nnamespace {\n\nstd::string FileLocation() {\n if (boost::unit_test::framework::master_test_suite().argc < 2) {\n return \"file_piece.cc\";\n }\n std::string ret(boost::unit_test::framework::master_test_suite().argv[1]);\n return ret;\n}\n\n/* istream */\nBOOST_AUTO_TEST_CASE(IStream) {\n std::fstream ref(FileLocation().c_str(), std::ios::in);\n std::fstream backing(FileLocation().c_str(), std::ios::in);\n FilePiece test(backing);\n std::string ref_line;\n while (getline(ref, ref_line)) {\n StringPiece test_line(test.ReadLine());\n BOOST_CHECK_EQUAL(ref_line, test_line);\n }\n BOOST_CHECK_THROW(test.get(), EndOfFileException);\n BOOST_CHECK_THROW(test.get(), EndOfFileException);\n}\n\n/* mmap implementation */\nBOOST_AUTO_TEST_CASE(MMapReadLine) {\n std::fstream ref(FileLocation().c_str(), std::ios::in);\n FilePiece test(FileLocation().c_str(), NULL, 1);\n std::string ref_line;\n while (getline(ref, ref_line)) {\n StringPiece test_line(test.ReadLine());\n // I submitted a bug report to ICU: http://bugs.icu-project.org/trac/ticket/7924\n if (!test_line.empty() || !ref_line.empty()) {\n BOOST_CHECK_EQUAL(ref_line, test_line);\n }\n }\n BOOST_CHECK_THROW(test.get(), EndOfFileException);\n}\n\n#if !defined(_WIN32) && !defined(_WIN64) && !defined(__APPLE__)\n/* Apple isn't happy with the popen, fileno, dup. And I don't want to\n * reimplement popen. This is an issue with the test. \n */\n/* read() implementation */\nBOOST_AUTO_TEST_CASE(StreamReadLine) {\n std::fstream ref(FileLocation().c_str(), std::ios::in);\n\n std::string popen_args = \"cat \\\"\";\n popen_args += FileLocation();\n popen_args += '\"';\n\n FILE *catter = popen(popen_args.c_str(), \"r\");\n BOOST_REQUIRE(catter);\n \n FilePiece test(dup(fileno(catter)), \"file_piece.cc\", NULL, 1);\n std::string ref_line;\n while (getline(ref, ref_line)) {\n StringPiece test_line(test.ReadLine());\n // I submitted a bug report to ICU: http://bugs.icu-project.org/trac/ticket/7924\n if (!test_line.empty() || !ref_line.empty()) {\n BOOST_CHECK_EQUAL(ref_line, test_line);\n }\n }\n BOOST_CHECK_THROW(test.get(), EndOfFileException);\n BOOST_REQUIRE(!pclose(catter));\n}\n#endif\n\n#ifdef HAVE_ZLIB\n\n// gzip file\nBOOST_AUTO_TEST_CASE(PlainZipReadLine) {\n std::string location(FileLocation());\n std::fstream ref(location.c_str(), std::ios::in);\n\n std::string command(\"gzip <\\\"\");\n command += location + \"\\\" >\\\"\" + location + \"\\\".gz\";\n\n BOOST_REQUIRE_EQUAL(0, system(command.c_str()));\n FilePiece test((location + \".gz\").c_str(), NULL, 1);\n unlink((location + \".gz\").c_str());\n std::string ref_line;\n while (getline(ref, ref_line)) {\n StringPiece test_line(test.ReadLine());\n // I submitted a bug report to ICU: http://bugs.icu-project.org/trac/ticket/7924\n if (!test_line.empty() || !ref_line.empty()) {\n BOOST_CHECK_EQUAL(ref_line, test_line);\n }\n }\n BOOST_CHECK_THROW(test.get(), EndOfFileException);\n}\n\n// gzip stream. Apple doesn't like popen, fileno, dup. This is an issue with\n// the test. \n#ifndef __APPLE__\nBOOST_AUTO_TEST_CASE(StreamZipReadLine) {\n std::fstream ref(FileLocation().c_str(), std::ios::in);\n\n std::string command(\"gzip <\\\"\");\n command += FileLocation() + \"\\\"\";\n\n FILE * catter = popen(command.c_str(), \"r\");\n BOOST_REQUIRE(catter);\n \n FilePiece test(dup(fileno(catter)), \"file_piece.cc.gz\", NULL, 1);\n std::string ref_line;\n while (getline(ref, ref_line)) {\n StringPiece test_line(test.ReadLine());\n // I submitted a bug report to ICU: http://bugs.icu-project.org/trac/ticket/7924\n if (!test_line.empty() || !ref_line.empty()) {\n BOOST_CHECK_EQUAL(ref_line, test_line);\n }\n }\n BOOST_CHECK_THROW(test.get(), EndOfFileException);\n BOOST_REQUIRE(!pclose(catter));\n}\n#endif // __APPLE__\n\n#endif // HAVE_ZLIB\n\n} // namespace\n} // namespace util\n" }, { "path": "src/kenlm/util/getopt.c", "content": "/*\nPOSIX getopt for Windows\n\nAT&T Public License\n\nCode given out at the 1985 UNIFORUM conference in Dallas. \n*/\n\n#ifndef __GNUC__\n\n#include \"getopt.hh\"\n#include \n#include \n\n#define NULL\t0\n#define EOF\t(-1)\n#define ERR(s, c)\tif(opterr){\\\n\tchar errbuf[2];\\\n\terrbuf[0] = c; errbuf[1] = '\\n';\\\n\tfputs(argv[0], stderr);\\\n\tfputs(s, stderr);\\\n\tfputc(c, stderr);}\n\t//(void) write(2, argv[0], (unsigned)strlen(argv[0]));\\\n\t//(void) write(2, s, (unsigned)strlen(s));\\\n\t//(void) write(2, errbuf, 2);}\n\nint\topterr = 1;\nint\toptind = 1;\nint\toptopt;\nchar\t*optarg;\n\nint\ngetopt(argc, argv, opts)\nint\targc;\nchar\t**argv, *opts;\n{\n\tstatic int sp = 1;\n\tregister int c;\n\tregister char *cp;\n\n\tif(sp == 1)\n\t\tif(optind >= argc ||\n\t\t argv[optind][0] != '-' || argv[optind][1] == '\\0')\n\t\t\treturn(EOF);\n\t\telse if(strcmp(argv[optind], \"--\") == NULL) {\n\t\t\toptind++;\n\t\t\treturn(EOF);\n\t\t}\n\toptopt = c = argv[optind][sp];\n\tif(c == ':' || (cp=strchr(opts, c)) == NULL) {\n\t\tERR(\": illegal option -- \", c);\n\t\tif(argv[optind][++sp] == '\\0') {\n\t\t\toptind++;\n\t\t\tsp = 1;\n\t\t}\n\t\treturn('?');\n\t}\n\tif(*++cp == ':') {\n\t\tif(argv[optind][sp+1] != '\\0')\n\t\t\toptarg = &argv[optind++][sp+1];\n\t\telse if(++optind >= argc) {\n\t\t\tERR(\": option requires an argument -- \", c);\n\t\t\tsp = 1;\n\t\t\treturn('?');\n\t\t} else\n\t\t\toptarg = argv[optind++];\n\t\tsp = 1;\n\t} else {\n\t\tif(argv[optind][++sp] == '\\0') {\n\t\t\tsp = 1;\n\t\t\toptind++;\n\t\t}\n\t\toptarg = NULL;\n\t}\n\treturn(c);\n}\n\n#endif /* __GNUC__ */\n" }, { "path": "src/kenlm/util/getopt.hh", "content": "/*\nPOSIX getopt for Windows\n\nAT&T Public License\n\nCode given out at the 1985 UNIFORUM conference in Dallas. \n*/\n\n#ifdef __GNUC__\n#include \n#endif\n#ifndef __GNUC__\n\n#ifndef _WINGETOPT_H_\n#define _WINGETOPT_H_\n\n#ifdef __cplusplus\nextern \"C\" {\n#endif\n\nextern int opterr;\nextern int optind;\nextern int optopt;\nextern char *optarg;\nextern int getopt(int argc, char **argv, char *opts);\n\n#ifdef __cplusplus\n}\n#endif\n\n#endif /* _GETOPT_H_ */\n#endif /* __GNUC__ */\n\n" }, { "path": "src/kenlm/util/have.hh", "content": "/* Optional packages. You might want to integrate this with your build system e.g. config.h from ./configure. */\n#ifndef UTIL_HAVE__\n#define UTIL_HAVE__\n\n#ifdef HAVE_CONFIG_H\n#include \"config.h\"\n#endif\n\n#ifndef HAVE_ICU\n//#define HAVE_ICU\n#endif\n\n#endif // UTIL_HAVE__\n" }, { "path": "src/kenlm/util/joint_sort.hh", "content": "#ifndef UTIL_JOINT_SORT__\n#define UTIL_JOINT_SORT__\n\n/* A terrifying amount of C++ to coax std::sort into soring one range while\n * also permuting another range the same way.\n */\n\n#include \"util/proxy_iterator.hh\"\n\n#include \n#include \n#include \n\nnamespace util {\n\nnamespace detail {\n\ntemplate class JointProxy;\n\ntemplate class JointIter {\n public:\n JointIter() {}\n\n JointIter(const KeyIter &key_iter, const ValueIter &value_iter) : key_(key_iter), value_(value_iter) {}\n\n bool operator==(const JointIter &other) const { return key_ == other.key_; }\n\n bool operator<(const JointIter &other) const { return (key_ < other.key_); }\n\n std::ptrdiff_t operator-(const JointIter &other) const { return key_ - other.key_; }\n\n JointIter &operator+=(std::ptrdiff_t amount) {\n key_ += amount;\n value_ += amount;\n return *this;\n }\n\n void swap(const JointIter &other) {\n std::swap(key_, other.key_);\n std::swap(value_, other.value_);\n }\n\n private:\n friend class JointProxy;\n KeyIter key_;\n ValueIter value_;\n};\n\ntemplate class JointProxy {\n private:\n typedef JointIter InnerIterator;\n\n public:\n typedef struct {\n typename std::iterator_traits::value_type key;\n typename std::iterator_traits::value_type value;\n const typename std::iterator_traits::value_type &GetKey() const { return key; }\n } value_type;\n\n JointProxy(const KeyIter &key_iter, const ValueIter &value_iter) : inner_(key_iter, value_iter) {}\n JointProxy(const JointProxy &other) : inner_(other.inner_) {}\n\n operator value_type() const {\n value_type ret;\n ret.key = *inner_.key_;\n ret.value = *inner_.value_;\n return ret;\n }\n\n JointProxy &operator=(const JointProxy &other) {\n *inner_.key_ = *other.inner_.key_;\n *inner_.value_ = *other.inner_.value_;\n return *this;\n }\n\n JointProxy &operator=(const value_type &other) {\n *inner_.key_ = other.key;\n *inner_.value_ = other.value;\n return *this;\n }\n\n typename std::iterator_traits::reference GetKey() const {\n return *(inner_.key_);\n }\n\n void swap(JointProxy &other) {\n std::swap(*inner_.key_, *other.inner_.key_);\n std::swap(*inner_.value_, *other.inner_.value_);\n }\n\n private:\n friend class ProxyIterator >;\n\n InnerIterator &Inner() { return inner_; }\n const InnerIterator &Inner() const { return inner_; }\n InnerIterator inner_;\n};\n\ntemplate class LessWrapper : public std::binary_function {\n public:\n explicit LessWrapper(const Less &less) : less_(less) {}\n\n bool operator()(const Proxy &left, const Proxy &right) const {\n return less_(left.GetKey(), right.GetKey());\n }\n bool operator()(const Proxy &left, const typename Proxy::value_type &right) const {\n return less_(left.GetKey(), right.GetKey());\n }\n bool operator()(const typename Proxy::value_type &left, const Proxy &right) const {\n return less_(left.GetKey(), right.GetKey());\n }\n bool operator()(const typename Proxy::value_type &left, const typename Proxy::value_type &right) const {\n return less_(left.GetKey(), right.GetKey());\n }\n\n private:\n const Less less_;\n};\n\n} // namespace detail\n\ntemplate class PairedIterator : public ProxyIterator > {\n public:\n PairedIterator(const KeyIter &key, const ValueIter &value) :\n ProxyIterator >(detail::JointProxy(key, value)) {}\n};\n\ntemplate void JointSort(const KeyIter &key_begin, const KeyIter &key_end, const ValueIter &value_begin, const Less &less) {\n ProxyIterator > full_begin(detail::JointProxy(key_begin, value_begin));\n detail::LessWrapper, Less> less_wrap(less);\n std::sort(full_begin, full_begin + (key_end - key_begin), less_wrap);\n}\n\n\ntemplate void JointSort(const KeyIter &key_begin, const KeyIter &key_end, const ValueIter &value_begin) {\n JointSort(key_begin, key_end, value_begin, std::less::value_type>());\n}\n\n} // namespace util\n\nnamespace std {\ntemplate void swap(util::detail::JointIter &left, util::detail::JointIter &right) {\n left.swap(right);\n}\n\ntemplate void swap(util::detail::JointProxy &left, util::detail::JointProxy &right) {\n left.swap(right);\n}\n} // namespace std\n\n#endif // UTIL_JOINT_SORT__\n" }, { "path": "src/kenlm/util/joint_sort_test.cc", "content": "#include \"util/joint_sort.hh\"\n\n#define BOOST_TEST_MODULE JointSortTest\n#include \n\nnamespace util { namespace {\n\nBOOST_AUTO_TEST_CASE(just_flip) {\n char keys[2];\n int values[2];\n keys[0] = 1; values[0] = 327;\n keys[1] = 0; values[1] = 87897;\n JointSort(keys + 0, keys + 2, values + 0);\n BOOST_CHECK_EQUAL(0, keys[0]);\n BOOST_CHECK_EQUAL(87897, values[0]);\n BOOST_CHECK_EQUAL(1, keys[1]);\n BOOST_CHECK_EQUAL(327, values[1]);\n}\n\nBOOST_AUTO_TEST_CASE(three) {\n char keys[3];\n int values[3];\n keys[0] = 1; values[0] = 327;\n keys[1] = 2; values[1] = 87897;\n keys[2] = 0; values[2] = 10;\n JointSort(keys + 0, keys + 3, values + 0);\n BOOST_CHECK_EQUAL(0, keys[0]);\n BOOST_CHECK_EQUAL(1, keys[1]);\n BOOST_CHECK_EQUAL(2, keys[2]);\n}\n\nBOOST_AUTO_TEST_CASE(char_int) {\n char keys[4];\n int values[4];\n keys[0] = 3; values[0] = 327;\n keys[1] = 1; values[1] = 87897;\n keys[2] = 2; values[2] = 10;\n keys[3] = 0; values[3] = 24347;\n JointSort(keys + 0, keys + 4, values + 0);\n BOOST_CHECK_EQUAL(0, keys[0]);\n BOOST_CHECK_EQUAL(24347, values[0]);\n BOOST_CHECK_EQUAL(1, keys[1]);\n BOOST_CHECK_EQUAL(87897, values[1]);\n BOOST_CHECK_EQUAL(2, keys[2]);\n BOOST_CHECK_EQUAL(10, values[2]);\n BOOST_CHECK_EQUAL(3, keys[3]);\n BOOST_CHECK_EQUAL(327, values[3]);\n}\n\n}} // namespace anonymous util\n" }, { "path": "src/kenlm/util/mmap.cc", "content": "/* Memory mapping wrappers.\n * ARM and MinGW ports contributed by Hideo Okuma and Tomoyuki Yoshimura at\n * NICT.\n */\n#include \"util/mmap.hh\"\n\n#include \"util/exception.hh\"\n#include \"util/file.hh\"\n#include \"util/scoped.hh\"\n\n#include \n\n#include \n#include \n#include \n#include \n#include \n\n#if defined(_WIN32) || defined(_WIN64)\n#include \n#include \n#else\n#include \n#include \n#endif\n\nnamespace util {\n\nlong SizePage() {\n#if defined(_WIN32) || defined(_WIN64)\n SYSTEM_INFO si;\n GetSystemInfo(&si);\n return si.dwAllocationGranularity;\n#else\n return sysconf(_SC_PAGE_SIZE);\n#endif\n}\n\nvoid SyncOrThrow(void *start, size_t length) {\n#if defined(_WIN32) || defined(_WIN64)\n UTIL_THROW_IF(!::FlushViewOfFile(start, length), ErrnoException, \"Failed to sync mmap\");\n#else\n UTIL_THROW_IF(msync(start, length, MS_SYNC), ErrnoException, \"Failed to sync mmap\");\n#endif\n}\n\nvoid UnmapOrThrow(void *start, size_t length) {\n#if defined(_WIN32) || defined(_WIN64)\n UTIL_THROW_IF(!::UnmapViewOfFile(start), ErrnoException, \"Failed to unmap a file\");\n#else\n UTIL_THROW_IF(munmap(start, length), ErrnoException, \"munmap failed\");\n#endif\n}\n\nscoped_mmap::~scoped_mmap() {\n if (data_ != (void*)-1) {\n try {\n // Thanks Denis Filimonov for pointing out NFS likes msync first. \n SyncOrThrow(data_, size_);\n UnmapOrThrow(data_, size_);\n } catch (const util::ErrnoException &e) {\n std::cerr << e.what();\n abort();\n }\n }\n}\n\nvoid scoped_memory::reset(void *data, std::size_t size, Alloc source) {\n switch(source_) {\n case MMAP_ALLOCATED:\n scoped_mmap(data_, size_);\n break;\n case ARRAY_ALLOCATED:\n delete [] reinterpret_cast(data_);\n break;\n case MALLOC_ALLOCATED:\n free(data_);\n break;\n case NONE_ALLOCATED:\n break;\n }\n data_ = data;\n size_ = size;\n source_ = source;\n}\n\nvoid scoped_memory::call_realloc(std::size_t size) {\n assert(source_ == MALLOC_ALLOCATED || source_ == NONE_ALLOCATED);\n void *new_data = realloc(data_, size);\n if (!new_data) {\n reset();\n } else {\n reset(new_data, size, MALLOC_ALLOCATED);\n }\n}\n\nvoid *MapOrThrow(std::size_t size, bool for_write, int flags, bool prefault, int fd, uint64_t offset) {\n#ifdef MAP_POPULATE // Linux specific\n if (prefault) {\n flags |= MAP_POPULATE;\n }\n#endif\n#if defined(_WIN32) || defined(_WIN64)\n int protectC = for_write ? PAGE_READWRITE : PAGE_READONLY;\n int protectM = for_write ? FILE_MAP_WRITE : FILE_MAP_READ;\n uint64_t total_size = size + offset;\n HANDLE hMapping = CreateFileMapping((HANDLE)_get_osfhandle(fd), NULL, protectC, total_size >> 32, static_cast(total_size), NULL);\n UTIL_THROW_IF(!hMapping, ErrnoException, \"CreateFileMapping failed\");\n LPVOID ret = MapViewOfFile(hMapping, protectM, offset >> 32, offset, size);\n CloseHandle(hMapping);\n UTIL_THROW_IF(!ret, ErrnoException, \"MapViewOfFile failed\");\n#else\n int protect = for_write ? (PROT_READ | PROT_WRITE) : PROT_READ;\n void *ret;\n UTIL_THROW_IF((ret = mmap(NULL, size, protect, flags, fd, offset)) == MAP_FAILED, ErrnoException, \"mmap failed for size \" << size << \" at offset \" << offset);\n# ifdef MADV_HUGEPAGE\n /* We like huge pages but it's fine if we can't have them. Note that huge\n * pages are not supported for file-backed mmap on linux.\n */\n madvise(ret, size, MADV_HUGEPAGE);\n# endif\n#endif\n return ret;\n}\n\nconst int kFileFlags =\n#if defined(_WIN32) || defined(_WIN64)\n 0 // MapOrThrow ignores flags on windows\n#elif defined(MAP_FILE)\n MAP_FILE | MAP_SHARED\n#else\n MAP_SHARED\n#endif\n ;\n\nvoid MapRead(LoadMethod method, int fd, uint64_t offset, std::size_t size, scoped_memory &out) {\n switch (method) {\n case LAZY:\n out.reset(MapOrThrow(size, false, kFileFlags, false, fd, offset), size, scoped_memory::MMAP_ALLOCATED);\n break;\n case POPULATE_OR_LAZY:\n#ifdef MAP_POPULATE\n case POPULATE_OR_READ:\n#endif\n out.reset(MapOrThrow(size, false, kFileFlags, true, fd, offset), size, scoped_memory::MMAP_ALLOCATED);\n break;\n#ifndef MAP_POPULATE\n case POPULATE_OR_READ:\n#endif\n case READ:\n out.reset(MallocOrThrow(size), size, scoped_memory::MALLOC_ALLOCATED);\n SeekOrThrow(fd, offset);\n ReadOrThrow(fd, out.get(), size);\n break;\n }\n}\n\n// Allocates zeroed memory in to.\nvoid MapAnonymous(std::size_t size, util::scoped_memory &to) {\n to.reset();\n#if defined(_WIN32) || defined(_WIN64)\n to.reset(calloc(1, size), size, scoped_memory::MALLOC_ALLOCATED);\n#else\n to.reset(MapOrThrow(size, true,\n# if defined(MAP_ANONYMOUS)\n MAP_ANONYMOUS | MAP_PRIVATE // Linux\n# else\n MAP_ANON | MAP_PRIVATE // BSD\n# endif\n , false, -1, 0), size, scoped_memory::MMAP_ALLOCATED);\n#endif\n}\n\nvoid *MapZeroedWrite(int fd, std::size_t size) {\n ResizeOrThrow(fd, 0);\n ResizeOrThrow(fd, size);\n return MapOrThrow(size, true, kFileFlags, false, fd, 0);\n}\n\nvoid *MapZeroedWrite(const char *name, std::size_t size, scoped_fd &file) {\n file.reset(CreateOrThrow(name));\n try {\n return MapZeroedWrite(file.get(), size);\n } catch (ErrnoException &e) {\n e << \" in file \" << name;\n throw;\n }\n}\n\n} // namespace util\n" }, { "path": "src/kenlm/util/mmap.hh", "content": "#ifndef UTIL_MMAP__\n#define UTIL_MMAP__\n// Utilities for mmaped files. \n\n#include \n\n#include \n#include \n\nnamespace util {\n\nclass scoped_fd;\n\nlong SizePage();\n\n// (void*)-1 is MAP_FAILED; this is done to avoid including the mmap header here. \nclass scoped_mmap {\n public:\n scoped_mmap() : data_((void*)-1), size_(0) {}\n scoped_mmap(void *data, std::size_t size) : data_(data), size_(size) {}\n ~scoped_mmap();\n\n void *get() const { return data_; }\n\n const uint8_t *begin() const { return reinterpret_cast(data_); }\n const uint8_t *end() const { return reinterpret_cast(data_) + size_; }\n std::size_t size() const { return size_; }\n\n void reset(void *data, std::size_t size) {\n scoped_mmap other(data_, size_);\n data_ = data;\n size_ = size;\n }\n\n void reset() {\n reset((void*)-1, 0);\n }\n\n private:\n void *data_;\n std::size_t size_;\n\n scoped_mmap(const scoped_mmap &);\n scoped_mmap &operator=(const scoped_mmap &);\n};\n\n/* For when the memory might come from mmap, new char[], or malloc. Uses NULL\n * and 0 for blanks even though mmap signals errors with (void*)-1). The reset\n * function checks that blank for mmap. \n */\nclass scoped_memory {\n public:\n typedef enum {MMAP_ALLOCATED, ARRAY_ALLOCATED, MALLOC_ALLOCATED, NONE_ALLOCATED} Alloc;\n\n scoped_memory() : data_(NULL), size_(0), source_(NONE_ALLOCATED) {}\n\n ~scoped_memory() { reset(); }\n\n void *get() const { return data_; }\n const char *begin() const { return reinterpret_cast(data_); }\n const char *end() const { return reinterpret_cast(data_) + size_; }\n std::size_t size() const { return size_; }\n\n Alloc source() const { return source_; }\n\n void reset() { reset(NULL, 0, NONE_ALLOCATED); }\n\n void reset(void *data, std::size_t size, Alloc from);\n\n // realloc allows the current data to escape hence the need for this call\n // If realloc fails, destroys the original too and get() returns NULL.\n void call_realloc(std::size_t to);\n\n private:\n\n void *data_;\n std::size_t size_;\n\n Alloc source_;\n\n scoped_memory(const scoped_memory &);\n scoped_memory &operator=(const scoped_memory &);\n};\n\ntypedef enum {\n // mmap with no prepopulate\n LAZY,\n // On linux, pass MAP_POPULATE to mmap.\n POPULATE_OR_LAZY,\n // Populate on Linux. malloc and read on non-Linux. \n POPULATE_OR_READ,\n // malloc and read. \n READ\n} LoadMethod;\n\nextern const int kFileFlags;\n\n// Wrapper around mmap to check it worked and hide some platform macros. \nvoid *MapOrThrow(std::size_t size, bool for_write, int flags, bool prefault, int fd, uint64_t offset = 0);\n\nvoid MapRead(LoadMethod method, int fd, uint64_t offset, std::size_t size, scoped_memory &out);\n\nvoid MapAnonymous(std::size_t size, scoped_memory &to);\n\n// Open file name with mmap of size bytes, all of which are initially zero. \nvoid *MapZeroedWrite(int fd, std::size_t size);\nvoid *MapZeroedWrite(const char *name, std::size_t size, scoped_fd &file);\n\n// msync wrapper \nvoid SyncOrThrow(void *start, size_t length);\n\n} // namespace util\n\n#endif // UTIL_MMAP__\n" }, { "path": "src/kenlm/util/multi_intersection.hh", "content": "#ifndef UTIL_MULTI_INTERSECTION__\n#define UTIL_MULTI_INTERSECTION__\n\n#include \n#include \n\n#include \n#include \n#include \n\nnamespace util {\n\nnamespace detail {\ntemplate struct RangeLessBySize : public std::binary_function {\n bool operator()(const Range &left, const Range &right) const {\n return left.size() < right.size();\n }\n};\n\n/* Takes sets specified by their iterators and a boost::optional containing\n * the lowest intersection if any. Each set must be sorted in increasing\n * order. sets is changed to truncate the beginning of each sequence to the\n * location of the match or an empty set. Precondition: sets is not empty\n * since the intersection over null is the universe and this function does not\n * know the universe. \n */\ntemplate boost::optional::value_type> FirstIntersectionSorted(std::vector > &sets, const Less &less = std::less::value_type>()) {\n typedef std::vector > Sets;\n typedef typename std::iterator_traits::value_type Value;\n\n assert(!sets.empty());\n\n if (sets.front().empty()) return boost::optional();\n // Possibly suboptimal to copy for general Value; makes unsigned int go slightly faster. \n Value highest(sets.front().front());\n for (typename Sets::iterator i(sets.begin()); i != sets.end(); ) {\n i->advance_begin(std::lower_bound(i->begin(), i->end(), highest, less) - i->begin());\n if (i->empty()) return boost::optional();\n if (less(highest, i->front())) {\n highest = i->front();\n // start over\n i = sets.begin();\n } else {\n ++i;\n }\n }\n return boost::optional(highest);\n}\n\n} // namespace detail\n\ntemplate boost::optional::value_type> FirstIntersection(std::vector > &sets, const Less less) {\n assert(!sets.empty());\n\n std::sort(sets.begin(), sets.end(), detail::RangeLessBySize >());\n return detail::FirstIntersectionSorted(sets, less);\n}\n\ntemplate boost::optional::value_type> FirstIntersection(std::vector > &sets) {\n return FirstIntersection(sets, std::less::value_type>());\n}\n\ntemplate void AllIntersection(std::vector > &sets, Output &out, const Less less) {\n typedef typename std::iterator_traits::value_type Value;\n assert(!sets.empty());\n\n std::sort(sets.begin(), sets.end(), detail::RangeLessBySize >());\n boost::optional ret;\n for (boost::optional ret; ret = detail::FirstIntersectionSorted(sets, less); sets.front().advance_begin(1)) {\n out(*ret);\n }\n}\n\ntemplate void AllIntersection(std::vector > &sets, Output &out) {\n AllIntersection(sets, out, std::less::value_type>());\n}\n\n} // namespace util\n\n#endif // UTIL_MULTI_INTERSECTION__\n" }, { "path": "src/kenlm/util/multi_intersection_test.cc", "content": "#include \"util/multi_intersection.hh\"\n\n#define BOOST_TEST_MODULE MultiIntersectionTest\n#include \n\nnamespace util {\nnamespace {\n\nBOOST_AUTO_TEST_CASE(Empty) {\n std::vector > sets;\n \n sets.push_back(boost::iterator_range(static_cast(NULL), static_cast(NULL)));\n BOOST_CHECK(!FirstIntersection(sets));\n}\n\nBOOST_AUTO_TEST_CASE(Single) {\n std::vector nums;\n nums.push_back(1);\n nums.push_back(4);\n nums.push_back(100);\n std::vector::const_iterator> > sets;\n sets.push_back(nums);\n\n boost::optional ret(FirstIntersection(sets));\n\n BOOST_REQUIRE(ret);\n BOOST_CHECK_EQUAL(static_cast(1), *ret);\n}\n\ntemplate boost::iterator_range RangeFromArray(const T (&arr)[len]) {\n return boost::iterator_range(arr, arr + len);\n}\n\nBOOST_AUTO_TEST_CASE(MultiNone) {\n unsigned int nums0[] = {1, 3, 4, 22};\n unsigned int nums1[] = {2, 5, 12};\n unsigned int nums2[] = {4, 17};\n\n std::vector > sets;\n sets.push_back(RangeFromArray(nums0));\n sets.push_back(RangeFromArray(nums1));\n sets.push_back(RangeFromArray(nums2));\n\n BOOST_CHECK(!FirstIntersection(sets));\n}\n\nBOOST_AUTO_TEST_CASE(MultiOne) {\n unsigned int nums0[] = {1, 3, 4, 17, 22};\n unsigned int nums1[] = {2, 5, 12, 17};\n unsigned int nums2[] = {4, 17};\n\n std::vector > sets;\n sets.push_back(RangeFromArray(nums0));\n sets.push_back(RangeFromArray(nums1));\n sets.push_back(RangeFromArray(nums2));\n\n boost::optional ret(FirstIntersection(sets));\n BOOST_REQUIRE(ret);\n BOOST_CHECK_EQUAL(static_cast(17), *ret);\n}\n\n} // namespace\n} // namespace util\n" }, { "path": "src/kenlm/util/murmur_hash.cc", "content": "/* Downloaded from http://sites.google.com/site/murmurhash/ which says \"All\n * code is released to the public domain. For business purposes, Murmurhash is\n * under the MIT license.\"\n * This is modified from the original:\n * ULL tag on 0xc6a4a7935bd1e995 so this will compile on 32-bit. \n * length changed to unsigned int. \n * placed in namespace util\n * add MurmurHashNative\n * default option = 0 for seed\n * ARM port from NICT\n */\n\n#include \"util/murmur_hash.hh\"\n#include \n\nnamespace util {\n\n//-----------------------------------------------------------------------------\n// MurmurHash2, 64-bit versions, by Austin Appleby\n\n// The same caveats as 32-bit MurmurHash2 apply here - beware of alignment \n// and endian-ness issues if used across multiple platforms.\n\n// 64-bit hash for 64-bit platforms\n\nuint64_t MurmurHash64A ( const void * key, std::size_t len, uint64_t seed )\n{\n const uint64_t m = 0xc6a4a7935bd1e995ULL;\n const int r = 47;\n\n uint64_t h = seed ^ (len * m);\n\n#if defined(__arm) || defined(__arm__)\n const size_t ksize = sizeof(uint64_t);\n const unsigned char * data = (const unsigned char *)key;\n const unsigned char * end = data + (std::size_t)(len/8) * ksize;\n#else\n const uint64_t * data = (const uint64_t *)key;\n const uint64_t * end = data + (len/8);\n#endif\n\n while(data != end)\n {\n#if defined(__arm) || defined(__arm__)\n uint64_t k;\n memcpy(&k, data, ksize);\n data += ksize;\n#else\n uint64_t k = *data++;\n#endif\n\n k *= m; \n k ^= k >> r; \n k *= m; \n \n h ^= k;\n h *= m; \n }\n\n const unsigned char * data2 = (const unsigned char*)data;\n\n switch(len & 7)\n {\n case 7: h ^= uint64_t(data2[6]) << 48;\n case 6: h ^= uint64_t(data2[5]) << 40;\n case 5: h ^= uint64_t(data2[4]) << 32;\n case 4: h ^= uint64_t(data2[3]) << 24;\n case 3: h ^= uint64_t(data2[2]) << 16;\n case 2: h ^= uint64_t(data2[1]) << 8;\n case 1: h ^= uint64_t(data2[0]);\n h *= m;\n };\n \n h ^= h >> r;\n h *= m;\n h ^= h >> r;\n\n return h;\n} \n\n\n// 64-bit hash for 32-bit platforms\n\nuint64_t MurmurHash64B ( const void * key, std::size_t len, uint64_t seed )\n{\n const unsigned int m = 0x5bd1e995;\n const int r = 24;\n\n unsigned int h1 = seed ^ len;\n unsigned int h2 = 0;\n\n#if defined(__arm) || defined(__arm__)\n size_t ksize = sizeof(unsigned int);\n const unsigned char * data = (const unsigned char *)key;\n#else\n const unsigned int * data = (const unsigned int *)key;\n#endif\n\n unsigned int k1, k2;\n while(len >= 8)\n {\n#if defined(__arm) || defined(__arm__)\n memcpy(&k1, data, ksize);\n data += ksize;\n memcpy(&k2, data, ksize);\n data += ksize;\n#else\n k1 = *data++;\n k2 = *data++;\n#endif\n\n k1 *= m; k1 ^= k1 >> r; k1 *= m;\n h1 *= m; h1 ^= k1;\n len -= 4;\n\n k2 *= m; k2 ^= k2 >> r; k2 *= m;\n h2 *= m; h2 ^= k2;\n len -= 4;\n }\n\n if(len >= 4)\n {\n#if defined(__arm) || defined(__arm__)\n memcpy(&k1, data, ksize);\n data += ksize;\n#else\n k1 = *data++;\n#endif\n k1 *= m; k1 ^= k1 >> r; k1 *= m;\n h1 *= m; h1 ^= k1;\n len -= 4;\n }\n\n switch(len)\n {\n case 3: h2 ^= ((unsigned char*)data)[2] << 16;\n case 2: h2 ^= ((unsigned char*)data)[1] << 8;\n case 1: h2 ^= ((unsigned char*)data)[0];\n h2 *= m;\n };\n\n h1 ^= h2 >> 18; h1 *= m;\n h2 ^= h1 >> 22; h2 *= m;\n h1 ^= h2 >> 17; h1 *= m;\n h2 ^= h1 >> 19; h2 *= m;\n\n uint64_t h = h1;\n\n h = (h << 32) | h2;\n\n return h;\n}\n\n// Trick to test for 64-bit architecture at compile time. \nnamespace {\ntemplate inline uint64_t MurmurHashNativeBackend(const void * key, std::size_t len, uint64_t seed) {\n return MurmurHash64A(key, len, seed);\n}\ntemplate <> inline uint64_t MurmurHashNativeBackend<4>(const void * key, std::size_t len, uint64_t seed) {\n return MurmurHash64B(key, len, seed);\n}\n} // namespace\n\nuint64_t MurmurHashNative(const void * key, std::size_t len, uint64_t seed) {\n return MurmurHashNativeBackend(key, len, seed);\n}\n\n} // namespace util\n" }, { "path": "src/kenlm/util/murmur_hash.hh", "content": "#ifndef UTIL_MURMUR_HASH__\n#define UTIL_MURMUR_HASH__\n#include \n#include \n\nnamespace util {\n\nuint64_t MurmurHash64A(const void * key, std::size_t len, uint64_t seed = 0);\nuint64_t MurmurHash64B(const void * key, std::size_t len, uint64_t seed = 0);\nuint64_t MurmurHashNative(const void * key, std::size_t len, uint64_t seed = 0);\n\n} // namespace util\n\n#endif // UTIL_MURMUR_HASH__\n" }, { "path": "src/kenlm/util/pcqueue.hh", "content": "#ifndef UTIL_PCQUEUE__\n#define UTIL_PCQUEUE__\n\n#include \n#include \n#include \n#include \n\n#include \n\nnamespace util {\n\ninline void WaitSemaphore (boost::interprocess::interprocess_semaphore &on) {\n while (1) {\n try {\n on.wait();\n break;\n }\n catch (boost::interprocess::interprocess_exception &e) {\n if (e.get_native_error() != EINTR) throw;\n }\n }\n}\n\n/* Producer consumer queue safe for multiple producers and multiple consumers.\n * T must be default constructable and have operator=. \n * The value is copied twice for Consume(T &out) or three times for Consume(),\n * so larger objects should be passed via pointer.\n * Strong exception guarantee if operator= throws. Undefined if semaphores throw. \n */\ntemplate class PCQueue : boost::noncopyable {\n public:\n explicit PCQueue(size_t size)\n : empty_(size), used_(0),\n storage_(new T[size]),\n end_(storage_.get() + size),\n produce_at_(storage_.get()),\n consume_at_(storage_.get()) {}\n\n // Add a value to the queue.\n void Produce(const T &val) {\n WaitSemaphore(empty_);\n {\n boost::unique_lock produce_lock(produce_at_mutex_);\n try {\n *produce_at_ = val;\n }\n catch (...) {\n empty_.post();\n throw;\n }\n if (++produce_at_ == end_) produce_at_ = storage_.get();\n }\n used_.post();\n }\n\n // Consume a value, assigning it to out.\n T& Consume(T &out) {\n WaitSemaphore(used_);\n {\n boost::unique_lock consume_lock(consume_at_mutex_);\n try {\n out = *consume_at_;\n }\n catch (...) {\n used_.post();\n throw;\n }\n if (++consume_at_ == end_) consume_at_ = storage_.get();\n }\n empty_.post();\n return out;\n }\n\n // Convenience version of Consume that copies the value to return.\n // The other version is faster.\n T Consume() {\n T ret;\n Consume(ret);\n return ret;\n }\n \n private:\n // Number of empty spaces in storage_.\n boost::interprocess::interprocess_semaphore empty_;\n // Number of occupied spaces in storage_.\n boost::interprocess::interprocess_semaphore used_;\n\n boost::scoped_array storage_;\n\n T *const end_;\n\n // Index for next write in storage_.\n T *produce_at_;\n boost::mutex produce_at_mutex_;\n\n // Index for next read from storage_.\n T *consume_at_;\n boost::mutex consume_at_mutex_;\n\n};\n\n} // namespace util\n\n#endif // UTIL_PCQUEUE__\n" }, { "path": "src/kenlm/util/pool.cc", "content": "#include \"util/pool.hh\"\n\n#include \"util/scoped.hh\"\n\n#include \n\nnamespace util {\n\nPool::Pool() {\n current_ = NULL;\n current_end_ = NULL;\n}\n\nPool::~Pool() {\n FreeAll();\n}\n\nvoid Pool::FreeAll() {\n for (std::vector::const_iterator i(free_list_.begin()); i != free_list_.end(); ++i) {\n free(*i);\n }\n free_list_.clear();\n current_ = NULL;\n current_end_ = NULL;\n}\n\nvoid *Pool::More(std::size_t size) {\n std::size_t amount = std::max(static_cast(32) << free_list_.size(), size);\n uint8_t *ret = static_cast(MallocOrThrow(amount));\n free_list_.push_back(ret);\n current_ = ret + size;\n current_end_ = ret + amount;\n return ret;\n}\n\n} // namespace util\n" }, { "path": "src/kenlm/util/pool.hh", "content": "// Very simple pool. It can only allocate memory. And all of the memory it\n// allocates must be freed at the same time. \n\n#ifndef UTIL_POOL__\n#define UTIL_POOL__\n\n#include \n\n#include \n\nnamespace util {\n\nclass Pool {\n public:\n Pool();\n\n ~Pool();\n\n void *Allocate(std::size_t size) {\n void *ret = current_;\n current_ += size;\n if (current_ < current_end_) {\n return ret;\n } else {\n return More(size);\n }\n }\n\n void FreeAll();\n\n private:\n void *More(std::size_t size);\n\n std::vector free_list_;\n\n uint8_t *current_, *current_end_;\n\n // no copying\n Pool(const Pool &);\n Pool &operator=(const Pool &);\n}; \n\n} // namespace util\n\n#endif // UTIL_POOL__\n" }, { "path": "src/kenlm/util/probing_hash_table.hh", "content": "#ifndef UTIL_PROBING_HASH_TABLE__\n#define UTIL_PROBING_HASH_TABLE__\n\n#include \"util/exception.hh\"\n\n#include \n#include \n#include \n#include \n\n#include \n#include \n\nnamespace util {\n\n/* Thrown when table grows too large */\nclass ProbingSizeException : public Exception {\n public:\n ProbingSizeException() throw() {}\n ~ProbingSizeException() throw() {}\n};\n\n// std::identity is an SGI extension :-(\nstruct IdentityHash {\n template T operator()(T arg) const { return arg; }\n};\n\n/* Non-standard hash table\n * Buckets must be set at the beginning and must be greater than maximum number\n * of elements, else it throws ProbingSizeException.\n * Memory management and initialization is externalized to make it easier to\n * serialize these to disk and load them quickly.\n * Uses linear probing to find value.\n * Only insert and lookup operations. \n */\n\ntemplate > class ProbingHashTable {\n public:\n typedef EntryT Entry;\n typedef typename Entry::Key Key;\n typedef const Entry *ConstIterator;\n typedef Entry *MutableIterator;\n typedef HashT Hash;\n typedef EqualT Equal;\n\n public:\n static uint64_t Size(uint64_t entries, float multiplier) {\n uint64_t buckets = std::max(entries + 1, static_cast(multiplier * static_cast(entries)));\n return buckets * sizeof(Entry);\n }\n\n // Must be assigned to later. \n ProbingHashTable() : entries_(0)\n#ifdef DEBUG\n , initialized_(false)\n#endif\n {}\n\n ProbingHashTable(void *start, std::size_t allocated, const Key &invalid = Key(), const Hash &hash_func = Hash(), const Equal &equal_func = Equal())\n : begin_(reinterpret_cast(start)),\n buckets_(allocated / sizeof(Entry)),\n end_(begin_ + buckets_),\n invalid_(invalid),\n hash_(hash_func),\n equal_(equal_func),\n entries_(0)\n#ifdef DEBUG\n , initialized_(true)\n#endif\n {}\n\n template MutableIterator Insert(const T &t) {\n#ifdef DEBUG\n assert(initialized_);\n#endif\n UTIL_THROW_IF(++entries_ >= buckets_, ProbingSizeException, \"Hash table with \" << buckets_ << \" buckets is full.\");\n return UncheckedInsert(t);\n }\n\n // Return true if the value was found (and not inserted). This is consistent with Find but the opposite if hash_map!\n template bool FindOrInsert(const T &t, MutableIterator &out) {\n#ifdef DEBUG\n assert(initialized_);\n#endif\n for (MutableIterator i(begin_ + (hash_(t.GetKey()) % buckets_));;) {\n Key got(i->GetKey());\n if (equal_(got, t.GetKey())) { out = i; return true; }\n if (equal_(got, invalid_)) {\n UTIL_THROW_IF(++entries_ >= buckets_, ProbingSizeException, \"Hash table with \" << buckets_ << \" buckets is full.\");\n *i = t;\n out = i;\n return false;\n }\n if (++i == end_) i = begin_;\n } \n }\n\n void FinishedInserting() {}\n\n void LoadedBinary() {}\n\n // Don't change anything related to GetKey, \n template bool UnsafeMutableFind(const Key key, MutableIterator &out) {\n#ifdef DEBUG\n assert(initialized_);\n#endif\n for (MutableIterator i(begin_ + (hash_(key) % buckets_));;) {\n Key got(i->GetKey());\n if (equal_(got, key)) { out = i; return true; }\n if (equal_(got, invalid_)) return false;\n if (++i == end_) i = begin_;\n } \n }\n\n template bool Find(const Key key, ConstIterator &out) const {\n#ifdef DEBUG\n assert(initialized_);\n#endif\n for (ConstIterator i(begin_ + (hash_(key) % buckets_));;) {\n Key got(i->GetKey());\n if (equal_(got, key)) { out = i; return true; }\n if (equal_(got, invalid_)) return false;\n if (++i == end_) i = begin_;\n } \n }\n\n void Clear() {\n Entry invalid;\n invalid.SetKey(invalid_);\n std::fill(begin_, end_, invalid);\n entries_ = 0;\n }\n\n // Return number of entries assuming no serialization went on.\n std::size_t SizeNoSerialization() const {\n return entries_;\n }\n\n // Return memory size expected by Double.\n std::size_t DoubleTo() const {\n return buckets_ * 2 * sizeof(Entry);\n }\n\n // Inform the table that it has double the amount of memory.\n // Pass clear_new = false if you are sure the new memory is initialized\n // properly (to invalid_) i.e. by mremap.\n void Double(void *new_base, bool clear_new = true) {\n begin_ = static_cast(new_base);\n MutableIterator old_end = begin_ + buckets_;\n buckets_ *= 2;\n end_ = begin_ + buckets_;\n if (clear_new) {\n Entry invalid;\n invalid.SetKey(invalid_);\n std::fill(old_end, end_, invalid);\n }\n std::vector rolled_over;\n // Move roll-over entries to a buffer because they might not roll over anymore. This should be small.\n for (MutableIterator i = begin_; i != old_end && !equal_(i->GetKey(), invalid_); ++i) {\n rolled_over.push_back(*i);\n i->SetKey(invalid_);\n }\n /* Re-insert everything. Entries might go backwards to take over a\n * recently opened gap, stay, move to new territory, or wrap around. If\n * an entry wraps around, it might go to a pointer greater than i (which\n * can happen at the beginning) and it will be revisited to possibly fill\n * in a gap created later.\n */\n Entry temp;\n for (MutableIterator i = begin_; i != old_end; ++i) {\n if (!equal_(i->GetKey(), invalid_)) {\n temp = *i;\n i->SetKey(invalid_);\n UncheckedInsert(temp);\n }\n }\n // Put the roll-over entries back in.\n for (typename std::vector::const_iterator i(rolled_over.begin()); i != rolled_over.end(); ++i) {\n UncheckedInsert(*i);\n }\n }\n\n // Mostly for tests, check consistency of every entry.\n void CheckConsistency() {\n MutableIterator last;\n for (last = end_ - 1; last >= begin_ && !equal_(last->GetKey(), invalid_); --last) {}\n UTIL_THROW_IF(last == begin_, ProbingSizeException, \"Completely full\");\n MutableIterator i;\n // Beginning can be wrap-arounds.\n for (i = begin_; !equal_(i->GetKey(), invalid_); ++i) {\n MutableIterator ideal = Ideal(*i);\n UTIL_THROW_IF(ideal > i && ideal <= last, Exception, \"Inconsistency at position \" << (i - begin_) << \" should be at \" << (ideal - begin_));\n }\n MutableIterator pre_gap = i;\n for (; i != end_; ++i) {\n if (equal_(i->GetKey(), invalid_)) {\n pre_gap = i;\n continue;\n }\n MutableIterator ideal = Ideal(*i);\n UTIL_THROW_IF(ideal > i || ideal <= pre_gap, Exception, \"Inconsistency at position \" << (i - begin_) << \" with ideal \" << (ideal - begin_));\n }\n }\n\n private:\n template MutableIterator Ideal(const T &t) {\n return begin_ + (hash_(t.GetKey()) % buckets_);\n }\n\n template MutableIterator UncheckedInsert(const T &t) {\n for (MutableIterator i(Ideal(t));;) {\n if (equal_(i->GetKey(), invalid_)) { *i = t; return i; }\n if (++i == end_) { i = begin_; }\n }\n }\n\n MutableIterator begin_;\n std::size_t buckets_;\n MutableIterator end_;\n Key invalid_;\n Hash hash_;\n Equal equal_;\n std::size_t entries_;\n#ifdef DEBUG\n bool initialized_;\n#endif\n};\n\n} // namespace util\n\n#endif // UTIL_PROBING_HASH_TABLE__\n" }, { "path": "src/kenlm/util/probing_hash_table_test.cc", "content": "#include \"util/probing_hash_table.hh\"\n\n#include \"util/murmur_hash.hh\"\n#include \"util/scoped.hh\"\n\n#define BOOST_TEST_MODULE ProbingHashTableTest\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\nnamespace util {\nnamespace {\n\nstruct Entry {\n unsigned char key;\n typedef unsigned char Key;\n\n unsigned char GetKey() const {\n return key;\n }\n\n void SetKey(unsigned char to) {\n key = to;\n }\n\n uint64_t GetValue() const {\n return value;\n }\n\n uint64_t value;\n};\n\ntypedef ProbingHashTable > Table;\n\nBOOST_AUTO_TEST_CASE(simple) {\n size_t size = Table::Size(10, 1.2);\n boost::scoped_array mem(new char[size]);\n memset(mem.get(), 0, size);\n\n Table table(mem.get(), size);\n const Entry *i = NULL;\n BOOST_CHECK(!table.Find(2, i));\n Entry to_ins;\n to_ins.key = 3;\n to_ins.value = 328920;\n table.Insert(to_ins);\n BOOST_REQUIRE(table.Find(3, i));\n BOOST_CHECK_EQUAL(3, i->GetKey());\n BOOST_CHECK_EQUAL(static_cast(328920), i->GetValue());\n BOOST_CHECK(!table.Find(2, i));\n}\n\nstruct Entry64 {\n uint64_t key;\n typedef uint64_t Key;\n\n Entry64() {}\n\n explicit Entry64(uint64_t key_in) {\n key = key_in;\n }\n\n Key GetKey() const { return key; }\n void SetKey(uint64_t to) { key = to; }\n};\n\nstruct MurmurHashEntry64 {\n std::size_t operator()(uint64_t value) const {\n return util::MurmurHash64A(&value, 8);\n }\n};\n\ntypedef ProbingHashTable Table64;\n\nBOOST_AUTO_TEST_CASE(Double) {\n for (std::size_t initial = 19; initial < 30; ++initial) {\n size_t size = Table64::Size(initial, 1.2);\n scoped_malloc mem(MallocOrThrow(size));\n Table64 table(mem.get(), size, std::numeric_limits::max());\n table.Clear();\n for (uint64_t i = 0; i < 19; ++i) {\n table.Insert(Entry64(i));\n }\n table.CheckConsistency();\n mem.call_realloc(table.DoubleTo());\n table.Double(mem.get());\n table.CheckConsistency();\n for (uint64_t i = 20; i < 40 ; ++i) {\n table.Insert(Entry64(i));\n }\n mem.call_realloc(table.DoubleTo());\n table.Double(mem.get());\n table.CheckConsistency();\n }\n}\n\n} // namespace\n} // namespace util\n" }, { "path": "src/kenlm/util/proxy_iterator.hh", "content": "#ifndef UTIL_PROXY_ITERATOR__\n#define UTIL_PROXY_ITERATOR__\n\n#include \n#include \n\n/* This is a RandomAccessIterator that uses a proxy to access the underlying\n * data. Useful for packing data at bit offsets but still using STL\n * algorithms. \n *\n * Normally I would use boost::iterator_facade but some people are too lazy to\n * install boost and still want to use my language model. It's amazing how\n * many operators an iterator has. \n *\n * The Proxy needs to provide:\n * class InnerIterator;\n * InnerIterator &Inner();\n * const InnerIterator &Inner() const;\n *\n * InnerIterator has to implement:\n * operator==(InnerIterator)\n * operator<(InnerIterator)\n * operator+=(std::ptrdiff_t)\n * operator-(InnerIterator)\n * and of course whatever Proxy needs to dereference it. \n *\n * It's also a good idea to specialize std::swap for Proxy. \n */\n\nnamespace util {\ntemplate class ProxyIterator {\n private:\n // Self. \n typedef ProxyIterator S;\n typedef typename Proxy::InnerIterator InnerIterator;\n\n public:\n typedef std::random_access_iterator_tag iterator_category;\n typedef typename Proxy::value_type value_type;\n typedef std::ptrdiff_t difference_type;\n typedef Proxy reference;\n typedef Proxy * pointer;\n\n ProxyIterator() {}\n\n // For cast from non const to const. \n template ProxyIterator(const ProxyIterator &in) : p_(*in) {}\n explicit ProxyIterator(const Proxy &p) : p_(p) {}\n\n // p_'s operator= does value copying, but here we want iterator copying. \n S &operator=(const S &other) {\n I() = other.I();\n return *this;\n }\n\n bool operator==(const S &other) const { return I() == other.I(); }\n bool operator!=(const S &other) const { return !(*this == other); }\n bool operator<(const S &other) const { return I() < other.I(); }\n bool operator>(const S &other) const { return other < *this; }\n bool operator<=(const S &other) const { return !(*this > other); }\n bool operator>=(const S &other) const { return !(*this < other); }\n\n S &operator++() { return *this += 1; }\n S operator++(int) { S ret(*this); ++*this; return ret; }\n S &operator+=(std::ptrdiff_t amount) { I() += amount; return *this; }\n S operator+(std::ptrdiff_t amount) const { S ret(*this); ret += amount; return ret; }\n\n S &operator--() { return *this -= 1; }\n S operator--(int) { S ret(*this); --*this; return ret; }\n S &operator-=(std::ptrdiff_t amount) { I() += (-amount); return *this; }\n S operator-(std::ptrdiff_t amount) const { S ret(*this); ret -= amount; return ret; }\n\n std::ptrdiff_t operator-(const S &other) const { return I() - other.I(); }\n\n Proxy operator*() { return p_; }\n const Proxy operator*() const { return p_; }\n Proxy *operator->() { return &p_; }\n const Proxy *operator->() const { return &p_; }\n Proxy operator[](std::ptrdiff_t amount) const { return *(*this + amount); }\n\n const InnerIterator &Inner() { return p_.Inner(); }\n\n private:\n InnerIterator &I() { return p_.Inner(); }\n const InnerIterator &I() const { return p_.Inner(); }\n\n Proxy p_;\n};\n\ntemplate ProxyIterator operator+(std::ptrdiff_t amount, const ProxyIterator &it) {\n return it + amount;\n}\n\n} // namespace util\n\n#endif // UTIL_PROXY_ITERATOR__\n" }, { "path": "src/kenlm/util/read_compressed.cc", "content": "#include \"util/read_compressed.hh\"\n\n#include \"util/file.hh\"\n#include \"util/have.hh\"\n#include \"util/scoped.hh\"\n\n#include \n#include \n\n#include \n#include \n#include \n#include \n\n#ifdef HAVE_ZLIB\n#include \n#endif\n\n#ifdef HAVE_BZLIB\n#include \n#endif\n\n#ifdef HAVE_XZLIB\n#include \n#endif\n\nnamespace util {\n\nCompressedException::CompressedException() throw() {}\nCompressedException::~CompressedException() throw() {}\n\nGZException::GZException() throw() {}\nGZException::~GZException() throw() {}\n\nBZException::BZException() throw() {}\nBZException::~BZException() throw() {}\n\nXZException::XZException() throw() {}\nXZException::~XZException() throw() {}\n\nclass ReadBase {\n public:\n virtual ~ReadBase() {}\n\n virtual std::size_t Read(void *to, std::size_t amount, ReadCompressed &thunk) = 0;\n\n protected:\n static void ReplaceThis(ReadBase *with, ReadCompressed &thunk) {\n thunk.internal_.reset(with);\n }\n\n static uint64_t &ReadCount(ReadCompressed &thunk) {\n return thunk.raw_amount_;\n }\n};\n\nnamespace {\n\n// Completed file that other classes can thunk to. \nclass Complete : public ReadBase {\n public:\n std::size_t Read(void *, std::size_t, ReadCompressed &) {\n return 0;\n }\n};\n\nclass Uncompressed : public ReadBase {\n public:\n explicit Uncompressed(int fd) : fd_(fd) {}\n\n std::size_t Read(void *to, std::size_t amount, ReadCompressed &thunk) {\n std::size_t got = PartialRead(fd_.get(), to, amount);\n ReadCount(thunk) += got;\n return got;\n }\n\n private:\n scoped_fd fd_;\n};\n\nclass UncompressedWithHeader : public ReadBase {\n public:\n UncompressedWithHeader(int fd, void *already_data, std::size_t already_size) : fd_(fd) {\n assert(already_size);\n buf_.reset(malloc(already_size));\n if (!buf_.get()) throw std::bad_alloc();\n memcpy(buf_.get(), already_data, already_size);\n remain_ = static_cast(buf_.get());\n end_ = remain_ + already_size;\n }\n\n std::size_t Read(void *to, std::size_t amount, ReadCompressed &thunk) {\n assert(buf_.get());\n std::size_t sending = std::min(amount, end_ - remain_);\n memcpy(to, remain_, sending);\n remain_ += sending;\n if (remain_ == end_) {\n ReplaceThis(new Uncompressed(fd_.release()), thunk);\n }\n return sending;\n }\n\n private:\n scoped_malloc buf_;\n uint8_t *remain_;\n uint8_t *end_;\n\n scoped_fd fd_;\n};\n\n#ifdef HAVE_ZLIB\nclass GZip : public ReadBase {\n private:\n static const std::size_t kInputBuffer = 16384;\n public:\n GZip(int fd, void *already_data, std::size_t already_size) \n : file_(fd), in_buffer_(malloc(kInputBuffer)) {\n if (!in_buffer_.get()) throw std::bad_alloc();\n assert(already_size < kInputBuffer);\n if (already_size) {\n memcpy(in_buffer_.get(), already_data, already_size);\n stream_.next_in = static_cast(in_buffer_.get());\n stream_.avail_in = already_size;\n stream_.avail_in += ReadOrEOF(file_.get(), static_cast(in_buffer_.get()) + already_size, kInputBuffer - already_size);\n } else {\n stream_.avail_in = 0;\n }\n stream_.zalloc = Z_NULL;\n stream_.zfree = Z_NULL;\n stream_.opaque = Z_NULL;\n stream_.msg = NULL;\n // 32 for zlib and gzip decoding with automatic header detection. \n // 15 for maximum window size. \n UTIL_THROW_IF(Z_OK != inflateInit2(&stream_, 32 + 15), GZException, \"Failed to initialize zlib.\");\n }\n\n ~GZip() {\n if (Z_OK != inflateEnd(&stream_)) {\n std::cerr << \"zlib could not close properly.\" << std::endl;\n abort();\n }\n }\n\n std::size_t Read(void *to, std::size_t amount, ReadCompressed &thunk) {\n if (amount == 0) return 0;\n stream_.next_out = static_cast(to);\n stream_.avail_out = std::min(std::numeric_limits::max(), amount);\n do {\n if (!stream_.avail_in) ReadInput(thunk);\n int result = inflate(&stream_, 0);\n switch (result) {\n case Z_OK:\n break;\n case Z_STREAM_END:\n {\n std::size_t ret = static_cast(stream_.next_out) - static_cast(to);\n ReplaceThis(new Complete(), thunk);\n return ret;\n }\n case Z_ERRNO:\n UTIL_THROW(ErrnoException, \"zlib error\");\n default:\n UTIL_THROW(GZException, \"zlib encountered \" << (stream_.msg ? stream_.msg : \"an error \") << \" code \" << result);\n }\n } while (stream_.next_out == to);\n return static_cast(stream_.next_out) - static_cast(to);\n }\n\n private:\n void ReadInput(ReadCompressed &thunk) {\n assert(!stream_.avail_in);\n stream_.next_in = static_cast(in_buffer_.get());\n stream_.avail_in = ReadOrEOF(file_.get(), in_buffer_.get(), kInputBuffer);\n ReadCount(thunk) += stream_.avail_in;\n }\n\n scoped_fd file_;\n scoped_malloc in_buffer_;\n z_stream stream_;\n};\n#endif // HAVE_ZLIB\n\n#ifdef HAVE_BZLIB\nclass BZip : public ReadBase {\n public:\n explicit BZip(int fd, void *already_data, std::size_t already_size) {\n scoped_fd hold(fd);\n closer_.reset(FDOpenReadOrThrow(hold));\n int bzerror = BZ_OK;\n file_ = BZ2_bzReadOpen(&bzerror, closer_.get(), 0, 0, already_data, already_size);\n switch (bzerror) {\n case BZ_OK:\n return;\n case BZ_CONFIG_ERROR:\n UTIL_THROW(BZException, \"Looks like bzip2 was miscompiled.\");\n case BZ_PARAM_ERROR:\n UTIL_THROW(BZException, \"Parameter error\");\n case BZ_IO_ERROR:\n UTIL_THROW(BZException, \"IO error reading file\");\n case BZ_MEM_ERROR:\n throw std::bad_alloc();\n }\n }\n\n ~BZip() {\n int bzerror = BZ_OK;\n BZ2_bzReadClose(&bzerror, file_);\n if (bzerror != BZ_OK) {\n std::cerr << \"bz2 readclose error\" << std::endl;\n abort();\n }\n }\n\n std::size_t Read(void *to, std::size_t amount, ReadCompressed &thunk) {\n int bzerror = BZ_OK;\n int ret = BZ2_bzRead(&bzerror, file_, to, std::min(static_cast(INT_MAX), amount));\n long pos;\n switch (bzerror) {\n case BZ_STREAM_END:\n pos = ftell(closer_.get());\n if (pos != -1) ReadCount(thunk) = pos;\n ReplaceThis(new Complete(), thunk);\n return ret;\n case BZ_OK:\n pos = ftell(closer_.get());\n if (pos != -1) ReadCount(thunk) = pos;\n return ret;\n default:\n UTIL_THROW(BZException, \"bzip2 error \" << BZ2_bzerror(file_, &bzerror) << \" code \" << bzerror);\n }\n }\n\n private:\n scoped_FILE closer_;\n BZFILE *file_;\n};\n#endif // HAVE_BZLIB\n\n#ifdef HAVE_XZLIB\nclass XZip : public ReadBase {\n private:\n static const std::size_t kInputBuffer = 16384;\n public:\n XZip(int fd, void *already_data, std::size_t already_size) \n : file_(fd), in_buffer_(malloc(kInputBuffer)), stream_(), action_(LZMA_RUN) {\n if (!in_buffer_.get()) throw std::bad_alloc();\n assert(already_size < kInputBuffer);\n if (already_size) {\n memcpy(in_buffer_.get(), already_data, already_size);\n stream_.next_in = static_cast(in_buffer_.get());\n stream_.avail_in = already_size;\n stream_.avail_in += ReadOrEOF(file_.get(), static_cast(in_buffer_.get()) + already_size, kInputBuffer - already_size);\n } else {\n stream_.avail_in = 0;\n }\n stream_.allocator = NULL;\n lzma_ret ret = lzma_stream_decoder(&stream_, UINT64_MAX, LZMA_CONCATENATED);\n switch (ret) {\n case LZMA_OK:\n break;\n case LZMA_MEM_ERROR:\n UTIL_THROW(ErrnoException, \"xz open error\");\n default:\n UTIL_THROW(XZException, \"xz error code \" << ret);\n }\n }\n\n ~XZip() {\n lzma_end(&stream_);\n }\n\n std::size_t Read(void *to, std::size_t amount, ReadCompressed &thunk) {\n if (amount == 0) return 0;\n stream_.next_out = static_cast(to);\n stream_.avail_out = amount;\n do {\n if (!stream_.avail_in) ReadInput(thunk);\n lzma_ret status = lzma_code(&stream_, action_);\n switch (status) {\n case LZMA_OK:\n break;\n case LZMA_STREAM_END:\n UTIL_THROW_IF(action_ != LZMA_FINISH, XZException, \"Input not finished yet.\");\n {\n std::size_t ret = static_cast(stream_.next_out) - static_cast(to);\n ReplaceThis(new Complete(), thunk);\n return ret;\n }\n case LZMA_MEM_ERROR:\n throw std::bad_alloc();\n case LZMA_FORMAT_ERROR:\n UTIL_THROW(XZException, \"xzlib says file format not recognized\");\n case LZMA_OPTIONS_ERROR:\n UTIL_THROW(XZException, \"xzlib says unsupported compression options\");\n case LZMA_DATA_ERROR:\n UTIL_THROW(XZException, \"xzlib says this file is corrupt\");\n case LZMA_BUF_ERROR:\n UTIL_THROW(XZException, \"xzlib says unexpected end of input\");\n default:\n UTIL_THROW(XZException, \"unrecognized xzlib error \" << status);\n }\n } while (stream_.next_out == to);\n return static_cast(stream_.next_out) - static_cast(to);\n }\n\n private:\n void ReadInput(ReadCompressed &thunk) {\n assert(!stream_.avail_in);\n stream_.next_in = static_cast(in_buffer_.get());\n stream_.avail_in = ReadOrEOF(file_.get(), in_buffer_.get(), kInputBuffer);\n if (!stream_.avail_in) action_ = LZMA_FINISH;\n ReadCount(thunk) += stream_.avail_in;\n }\n\n scoped_fd file_;\n scoped_malloc in_buffer_;\n lzma_stream stream_;\n\n lzma_action action_;\n};\n#endif // HAVE_XZLIB\n\nclass IStreamReader : public ReadBase {\n public:\n explicit IStreamReader(std::istream &stream) : stream_(stream) {}\n\n std::size_t Read(void *to, std::size_t amount, ReadCompressed &thunk) {\n if (!stream_.read(static_cast(to), amount)) {\n UTIL_THROW_IF(!stream_.eof(), ErrnoException, \"istream error\");\n amount = stream_.gcount();\n }\n ReadCount(thunk) += amount;\n return amount;\n }\n\n private:\n std::istream &stream_;\n};\n\nenum MagicResult {\n UNKNOWN, GZIP, BZIP, XZIP\n};\n\nMagicResult DetectMagic(const void *from_void) {\n const uint8_t *header = static_cast(from_void);\n if (header[0] == 0x1f && header[1] == 0x8b) {\n return GZIP;\n }\n if (header[0] == 'B' && header[1] == 'Z' && header[2] == 'h') {\n return BZIP;\n }\n const uint8_t xzmagic[6] = { 0xFD, '7', 'z', 'X', 'Z', 0x00 };\n if (!memcmp(header, xzmagic, 6)) {\n return XZIP;\n }\n return UNKNOWN;\n}\n\nReadBase *ReadFactory(int fd, uint64_t &raw_amount) {\n scoped_fd hold(fd);\n unsigned char header[ReadCompressed::kMagicSize];\n raw_amount = ReadOrEOF(fd, header, ReadCompressed::kMagicSize);\n if (!raw_amount)\n return new Uncompressed(hold.release());\n if (raw_amount != ReadCompressed::kMagicSize)\n return new UncompressedWithHeader(hold.release(), header, raw_amount);\n switch (DetectMagic(header)) {\n case GZIP:\n#ifdef HAVE_ZLIB\n return new GZip(hold.release(), header, ReadCompressed::kMagicSize);\n#else\n UTIL_THROW(CompressedException, \"This looks like a gzip file but gzip support was not compiled in.\");\n#endif\n case BZIP:\n#ifdef HAVE_BZLIB\n return new BZip(hold.release(), header, ReadCompressed::kMagicSize);\n#else\n UTIL_THROW(CompressedException, \"This looks like a bzip file (it begins with BZ), but bzip support was not compiled in.\");\n#endif\n case XZIP:\n#ifdef HAVE_XZLIB\n return new XZip(hold.release(), header, ReadCompressed::kMagicSize);\n#else\n UTIL_THROW(CompressedException, \"This looks like an xz file, but xz support was not compiled in.\");\n#endif\n case UNKNOWN:\n break;\n }\n try {\n SeekOrThrow(fd, 0);\n } catch (const util::ErrnoException &e) {\n return new UncompressedWithHeader(hold.release(), header, ReadCompressed::kMagicSize);\n }\n return new Uncompressed(hold.release());\n}\n\n} // namespace\n\nbool ReadCompressed::DetectCompressedMagic(const void *from_void) {\n return DetectMagic(from_void) != UNKNOWN;\n}\n\nReadCompressed::ReadCompressed(int fd) {\n Reset(fd);\n}\n\nReadCompressed::ReadCompressed(std::istream &in) {\n Reset(in);\n}\n\nReadCompressed::ReadCompressed() {}\n\nReadCompressed::~ReadCompressed() {}\n\nvoid ReadCompressed::Reset(int fd) {\n internal_.reset();\n internal_.reset(ReadFactory(fd, raw_amount_));\n}\n\nvoid ReadCompressed::Reset(std::istream &in) {\n internal_.reset();\n internal_.reset(new IStreamReader(in));\n}\n\nstd::size_t ReadCompressed::Read(void *to, std::size_t amount) {\n return internal_->Read(to, amount, *this);\n}\n\n} // namespace util\n" }, { "path": "src/kenlm/util/read_compressed.hh", "content": "#ifndef UTIL_READ_COMPRESSED__\n#define UTIL_READ_COMPRESSED__\n\n#include \"util/exception.hh\"\n#include \"util/scoped.hh\"\n\n#include \n\n#include \n\nnamespace util {\n\nclass CompressedException : public Exception {\n public:\n CompressedException() throw();\n virtual ~CompressedException() throw();\n};\n\nclass GZException : public CompressedException {\n public:\n GZException() throw();\n ~GZException() throw();\n};\n\nclass BZException : public CompressedException {\n public:\n BZException() throw();\n ~BZException() throw();\n};\n\nclass XZException : public CompressedException {\n public:\n XZException() throw();\n ~XZException() throw();\n};\n\nclass ReadBase;\n\nclass ReadCompressed {\n public:\n static const std::size_t kMagicSize = 6;\n // Must have at least kMagicSize bytes. \n static bool DetectCompressedMagic(const void *from);\n\n // Takes ownership of fd. \n explicit ReadCompressed(int fd);\n\n // Try to avoid using this. Use the fd instead.\n // There is no decompression support for istreams.\n explicit ReadCompressed(std::istream &in);\n\n // Must call Reset later.\n ReadCompressed();\n\n ~ReadCompressed();\n\n // Takes ownership of fd. \n void Reset(int fd);\n\n // Same advice as the constructor.\n void Reset(std::istream &in);\n\n std::size_t Read(void *to, std::size_t amount);\n\n uint64_t RawAmount() const { return raw_amount_; }\n\n private:\n friend class ReadBase;\n\n scoped_ptr internal_;\n\n uint64_t raw_amount_;\n\n // No copying. \n ReadCompressed(const ReadCompressed &);\n void operator=(const ReadCompressed &);\n};\n\n} // namespace util\n\n#endif // UTIL_READ_COMPRESSED__\n" }, { "path": "src/kenlm/util/read_compressed_test.cc", "content": "#include \"util/read_compressed.hh\"\n\n#include \"util/file.hh\"\n#include \"util/have.hh\"\n\n#define BOOST_TEST_MODULE ReadCompressedTest\n#include \n#include \n\n#include \n#include \n\n#include \n\nnamespace util {\nnamespace {\n\nvoid ReadLoop(ReadCompressed &reader, void *to_void, std::size_t amount) {\n uint8_t *to = static_cast(to_void);\n while (amount) {\n std::size_t ret = reader.Read(to, amount);\n BOOST_REQUIRE(ret);\n to += ret;\n amount -= ret;\n }\n}\n\nconst uint32_t kSize4 = 100000 / 4;\n\nstd::string WriteRandom() {\n char name[] = \"tempXXXXXX\";\n scoped_fd original(mkstemp(name));\n BOOST_REQUIRE(original.get() > 0);\n for (uint32_t i = 0; i < kSize4; ++i) {\n WriteOrThrow(original.get(), &i, sizeof(uint32_t));\n }\n return name;\n}\n\nvoid VerifyRead(ReadCompressed &reader) {\n for (uint32_t i = 0; i < kSize4; ++i) {\n uint32_t got;\n ReadLoop(reader, &got, sizeof(uint32_t));\n BOOST_CHECK_EQUAL(i, got);\n }\n\n char ignored;\n BOOST_CHECK_EQUAL((std::size_t)0, reader.Read(&ignored, 1));\n // Test double EOF call.\n BOOST_CHECK_EQUAL((std::size_t)0, reader.Read(&ignored, 1));\n}\n\nvoid TestRandom(const char *compressor) {\n std::string name(WriteRandom());\n\n char gzname[] = \"tempXXXXXX\";\n scoped_fd gzipped(mkstemp(gzname));\n\n std::string command(compressor);\n#ifdef __CYGWIN__\n command += \".exe\";\n#endif\n command += \" <\\\"\";\n command += name;\n command += \"\\\" >\\\"\";\n command += gzname;\n command += \"\\\"\";\n BOOST_REQUIRE_EQUAL(0, system(command.c_str()));\n\n BOOST_CHECK_EQUAL(0, unlink(name.c_str()));\n BOOST_CHECK_EQUAL(0, unlink(gzname));\n\n ReadCompressed reader(gzipped.release());\n VerifyRead(reader);\n}\n\nBOOST_AUTO_TEST_CASE(Uncompressed) {\n TestRandom(\"cat\");\n}\n\n#ifdef HAVE_ZLIB\nBOOST_AUTO_TEST_CASE(ReadGZ) {\n TestRandom(\"gzip\");\n}\n#endif // HAVE_ZLIB\n\n#ifdef HAVE_BZLIB\nBOOST_AUTO_TEST_CASE(ReadBZ) {\n TestRandom(\"bzip2\");\n}\n#endif // HAVE_BZLIB\n\n#ifdef HAVE_XZLIB\nBOOST_AUTO_TEST_CASE(ReadXZ) {\n TestRandom(\"xz\");\n}\n#endif\n\nBOOST_AUTO_TEST_CASE(IStream) {\n std::string name(WriteRandom());\n std::fstream stream(name.c_str(), std::ios::in);\n BOOST_CHECK_EQUAL(0, unlink(name.c_str()));\n ReadCompressed reader;\n reader.Reset(stream);\n VerifyRead(reader);\n}\n\n} // namespace\n} // namespace util\n" }, { "path": "src/kenlm/util/scoped.cc", "content": "#include \"util/scoped.hh\"\n\n#include \n#if !defined(_WIN32) && !defined(_WIN64)\n#include \n#endif\n\nnamespace util {\n\nMallocException::MallocException(std::size_t requested) throw() {\n *this << \"for \" << requested << \" bytes \";\n}\n\nMallocException::~MallocException() throw() {}\n\nnamespace {\nvoid *InspectAddr(void *addr, std::size_t requested, const char *func_name) {\n UTIL_THROW_IF_ARG(!addr && requested, MallocException, (requested), \"in \" << func_name);\n // These routines are often used for large chunks of memory where huge pages help.\n#if MADV_HUGEPAGE\n madvise(addr, requested, MADV_HUGEPAGE);\n#endif\n return addr;\n}\n} // namespace\n\nvoid *MallocOrThrow(std::size_t requested) {\n return InspectAddr(std::malloc(requested), requested, \"malloc\");\n}\n\nvoid *CallocOrThrow(std::size_t requested) {\n return InspectAddr(std::calloc(1, requested), requested, \"calloc\");\n}\n\nscoped_malloc::~scoped_malloc() {\n std::free(p_);\n}\n\nvoid scoped_malloc::call_realloc(std::size_t requested) {\n p_ = InspectAddr(std::realloc(p_, requested), requested, \"realloc\");\n}\n\n} // namespace util\n" }, { "path": "src/kenlm/util/scoped.hh", "content": "#ifndef UTIL_SCOPED__\n#define UTIL_SCOPED__\n/* Other scoped objects in the style of scoped_ptr. */\n\n#include \"util/exception.hh\"\n#include \n\nnamespace util {\n\nclass MallocException : public ErrnoException {\n public:\n explicit MallocException(std::size_t requested) throw();\n ~MallocException() throw();\n};\n\nvoid *MallocOrThrow(std::size_t requested);\nvoid *CallocOrThrow(std::size_t requested);\n\nclass scoped_malloc {\n public:\n scoped_malloc() : p_(NULL) {}\n\n scoped_malloc(void *p) : p_(p) {}\n\n ~scoped_malloc();\n\n void reset(void *p = NULL) {\n scoped_malloc other(p_);\n p_ = p;\n }\n\n void call_realloc(std::size_t to);\n\n void *get() { return p_; }\n const void *get() const { return p_; }\n\n private:\n void *p_;\n\n scoped_malloc(const scoped_malloc &);\n scoped_malloc &operator=(const scoped_malloc &);\n};\n\n// Hat tip to boost. \ntemplate class scoped_array {\n public:\n explicit scoped_array(T *content = NULL) : c_(content) {}\n\n ~scoped_array() { delete [] c_; }\n\n T *get() { return c_; }\n const T* get() const { return c_; }\n\n T &operator*() { return *c_; }\n const T&operator*() const { return *c_; }\n\n T &operator[](std::size_t idx) { return c_[idx]; }\n const T &operator[](std::size_t idx) const { return c_[idx]; }\n\n void reset(T *to = NULL) {\n scoped_array other(c_);\n c_ = to;\n }\n\n private:\n T *c_;\n\n scoped_array(const scoped_array &);\n void operator=(const scoped_array &);\n};\n\ntemplate class scoped_ptr {\n public:\n explicit scoped_ptr(T *content = NULL) : c_(content) {}\n\n ~scoped_ptr() { delete c_; }\n\n T *get() { return c_; }\n const T* get() const { return c_; }\n\n T &operator*() { return *c_; }\n const T&operator*() const { return *c_; }\n\n T *operator->() { return c_; }\n const T*operator->() const { return c_; }\n\n T &operator[](std::size_t idx) { return c_[idx]; }\n const T &operator[](std::size_t idx) const { return c_[idx]; }\n\n void reset(T *to = NULL) {\n scoped_ptr other(c_);\n c_ = to;\n }\n\n private:\n T *c_;\n\n scoped_ptr(const scoped_ptr &);\n void operator=(const scoped_ptr &);\n};\n\n} // namespace util\n\n#endif // UTIL_SCOPED__\n" }, { "path": "src/kenlm/util/sized_iterator.hh", "content": "#ifndef UTIL_SIZED_ITERATOR__\n#define UTIL_SIZED_ITERATOR__\n\n#include \"util/proxy_iterator.hh\"\n\n#include \n#include \n\n#include \n#include \n\nnamespace util {\n\nclass SizedInnerIterator {\n public:\n SizedInnerIterator() {}\n\n SizedInnerIterator(void *ptr, std::size_t size) : ptr_(static_cast(ptr)), size_(size) {}\n\n bool operator==(const SizedInnerIterator &other) const {\n return ptr_ == other.ptr_;\n }\n bool operator<(const SizedInnerIterator &other) const {\n return ptr_ < other.ptr_;\n }\n SizedInnerIterator &operator+=(std::ptrdiff_t amount) {\n ptr_ += amount * size_;\n return *this;\n }\n std::ptrdiff_t operator-(const SizedInnerIterator &other) const {\n return (ptr_ - other.ptr_) / size_;\n }\n\n const void *Data() const { return ptr_; }\n void *Data() { return ptr_; }\n std::size_t EntrySize() const { return size_; }\n\n private:\n uint8_t *ptr_;\n std::size_t size_;\n};\n\nclass SizedProxy {\n public:\n SizedProxy() {}\n\n SizedProxy(void *ptr, std::size_t size) : inner_(ptr, size) {}\n\n operator std::string() const {\n return std::string(reinterpret_cast(inner_.Data()), inner_.EntrySize());\n }\n\n SizedProxy &operator=(const SizedProxy &from) {\n memcpy(inner_.Data(), from.inner_.Data(), inner_.EntrySize());\n return *this;\n }\n\n SizedProxy &operator=(const std::string &from) {\n memcpy(inner_.Data(), from.data(), inner_.EntrySize());\n return *this;\n }\n\n const void *Data() const { return inner_.Data(); }\n void *Data() { return inner_.Data(); }\n\n private:\n friend class util::ProxyIterator;\n\n typedef std::string value_type;\n\n typedef SizedInnerIterator InnerIterator;\n\n InnerIterator &Inner() { return inner_; }\n const InnerIterator &Inner() const { return inner_; }\n InnerIterator inner_;\n};\n\ntypedef ProxyIterator SizedIterator;\n\ninline SizedIterator SizedIt(void *ptr, std::size_t size) { return SizedIterator(SizedProxy(ptr, size)); }\n\n// Useful wrapper for a comparison function i.e. sort. \ntemplate class SizedCompare : public std::binary_function {\n public:\n explicit SizedCompare(const Delegate &delegate = Delegate()) : delegate_(delegate) {}\n\n bool operator()(const Proxy &first, const Proxy &second) const {\n return delegate_(first.Data(), second.Data());\n }\n bool operator()(const Proxy &first, const std::string &second) const {\n return delegate_(first.Data(), second.data());\n }\n bool operator()(const std::string &first, const Proxy &second) const {\n return delegate_(first.data(), second.Data());\n }\n bool operator()(const std::string &first, const std::string &second) const {\n return delegate_(first.data(), second.data());\n }\n\n const Delegate &GetDelegate() const { return delegate_; }\n \n private:\n const Delegate delegate_;\n};\n\n} // namespace util\n#endif // UTIL_SIZED_ITERATOR__\n" }, { "path": "src/kenlm/util/sorted_uniform.hh", "content": "#ifndef UTIL_SORTED_UNIFORM__\n#define UTIL_SORTED_UNIFORM__\n\n#include \n#include \n\n#include \n#include \n\nnamespace util {\n\ntemplate class IdentityAccessor {\n public:\n typedef T Key;\n T operator()(const T *in) const { return *in; }\n};\n\nstruct Pivot64 {\n static inline std::size_t Calc(uint64_t off, uint64_t range, std::size_t width) {\n std::size_t ret = static_cast(static_cast(off) / static_cast(range) * static_cast(width));\n // Cap for floating point rounding\n return (ret < width) ? ret : width - 1;\n }\n};\n\n// Use when off * width is <2^64. This is guaranteed when each of them is actually a 32-bit value. \nstruct Pivot32 {\n static inline std::size_t Calc(uint64_t off, uint64_t range, uint64_t width) {\n return static_cast((off * width) / (range + 1));\n }\n};\n\n// Usage: PivotSelect::T\ntemplate struct PivotSelect;\ntemplate <> struct PivotSelect<8> { typedef Pivot64 T; };\ntemplate <> struct PivotSelect<4> { typedef Pivot32 T; };\ntemplate <> struct PivotSelect<2> { typedef Pivot32 T; };\n\n/* Binary search. */\ntemplate bool BinaryFind(\n const Accessor &accessor,\n Iterator begin,\n Iterator end,\n const typename Accessor::Key key, Iterator &out) {\n while (end > begin) {\n Iterator pivot(begin + (end - begin) / 2);\n typename Accessor::Key mid(accessor(pivot));\n if (mid < key) {\n begin = pivot + 1;\n } else if (mid > key) {\n end = pivot;\n } else {\n out = pivot;\n return true;\n }\n }\n return false;\n}\n\n// Search the range [before_it + 1, after_it - 1] for key. \n// Preconditions:\n// before_v <= key <= after_v\n// before_v <= all values in the range [before_it + 1, after_it - 1] <= after_v\n// range is sorted.\ntemplate bool BoundedSortedUniformFind(\n const Accessor &accessor,\n Iterator before_it, typename Accessor::Key before_v,\n Iterator after_it, typename Accessor::Key after_v,\n const typename Accessor::Key key, Iterator &out) {\n while (after_it - before_it > 1) {\n Iterator pivot(before_it + (1 + Pivot::Calc(key - before_v, after_v - before_v, after_it - before_it - 1)));\n typename Accessor::Key mid(accessor(pivot));\n if (mid < key) {\n before_it = pivot;\n before_v = mid;\n } else if (mid > key) {\n after_it = pivot;\n after_v = mid;\n } else {\n out = pivot;\n return true;\n }\n }\n return false;\n}\n\ntemplate bool SortedUniformFind(const Accessor &accessor, Iterator begin, Iterator end, const typename Accessor::Key key, Iterator &out) {\n if (begin == end) return false;\n typename Accessor::Key below(accessor(begin));\n if (key <= below) {\n if (key == below) { out = begin; return true; }\n return false;\n }\n // Make the range [begin, end]. \n --end;\n typename Accessor::Key above(accessor(end));\n if (key >= above) {\n if (key == above) { out = end; return true; }\n return false;\n }\n return BoundedSortedUniformFind(accessor, begin, below, end, above, key, out);\n}\n\n// May return begin - 1.\ntemplate Iterator BinaryBelow(\n const Accessor &accessor,\n Iterator begin,\n Iterator end,\n const typename Accessor::Key key) {\n while (end > begin) {\n Iterator pivot(begin + (end - begin) / 2);\n typename Accessor::Key mid(accessor(pivot));\n if (mid < key) {\n begin = pivot + 1;\n } else if (mid > key) {\n end = pivot;\n } else {\n for (++pivot; (pivot < end) && accessor(pivot) == mid; ++pivot) {}\n return pivot - 1;\n }\n }\n return begin - 1;\n}\n\n} // namespace util\n\n#endif // UTIL_SORTED_UNIFORM__\n" }, { "path": "src/kenlm/util/sorted_uniform_test.cc", "content": "#include \"util/sorted_uniform.hh\"\n\n#include \n#include \n#include \n#include \n#include \n\n#define BOOST_TEST_MODULE SortedUniformTest\n#include \n\n#include \n#include \n#include \n\nnamespace util {\nnamespace {\n\ntemplate struct Entry {\n typedef KeyT Key;\n typedef ValueT Value;\n\n Key key;\n Value value;\n\n Key GetKey() const {\n return key;\n }\n\n Value GetValue() const {\n return value;\n }\n\n bool operator<(const Entry &other) const {\n return key < other.key;\n }\n};\n\ntemplate struct Accessor {\n typedef KeyT Key;\n template Key operator()(const Entry *entry) const {\n return entry->GetKey();\n }\n};\n\ntemplate void Check(const Entry *begin, const Entry *end, const boost::unordered_map &reference, const Key key) {\n typename boost::unordered_map::const_iterator ref = reference.find(key);\n typedef const Entry *It;\n // g++ can't tell that require will crash and burn.\n It i = NULL;\n bool ret = SortedUniformFind, Pivot64>(Accessor(), begin, end, key, i);\n if (ref == reference.end()) {\n BOOST_CHECK(!ret);\n } else {\n BOOST_REQUIRE(ret);\n BOOST_CHECK_EQUAL(ref->second, i->GetValue());\n }\n}\n\nBOOST_AUTO_TEST_CASE(empty) {\n typedef const Entry T;\n const T *i;\n bool ret = SortedUniformFind, Pivot64>(Accessor(), (const T*)NULL, (const T*)NULL, (uint64_t)10, i);\n BOOST_CHECK(!ret);\n}\n\ntemplate void RandomTest(Key upper, size_t entries, size_t queries) {\n typedef unsigned char Value;\n boost::mt19937 rng;\n boost::uniform_int range_key(0, upper);\n boost::uniform_int range_value(0, 255);\n boost::variate_generator > gen_key(rng, range_key);\n boost::variate_generator > gen_value(rng, range_value);\n\n typedef Entry Ent;\n std::vector backing;\n boost::unordered_map reference;\n Ent ent;\n for (size_t i = 0; i < entries; ++i) {\n Key key = gen_key();\n unsigned char value = gen_value();\n if (reference.insert(std::make_pair(key, value)).second) {\n ent.key = key;\n ent.value = value;\n backing.push_back(ent);\n }\n }\n std::sort(backing.begin(), backing.end());\n\n // Random queries. \n for (size_t i = 0; i < queries; ++i) {\n const Key key = gen_key();\n Check(&*backing.begin(), &*backing.end(), reference, key);\n }\n\n typename boost::unordered_map::const_iterator it = reference.begin();\n for (size_t i = 0; (i < queries) && (it != reference.end()); ++i, ++it) {\n Check(&*backing.begin(), &*backing.end(), reference, it->second);\n }\n}\n\nBOOST_AUTO_TEST_CASE(basic) {\n RandomTest(11, 10, 200);\n}\n\nBOOST_AUTO_TEST_CASE(tiny_dense_random) {\n RandomTest(11, 50, 200);\n}\n\nBOOST_AUTO_TEST_CASE(small_dense_random) {\n RandomTest(100, 100, 200);\n}\n\nBOOST_AUTO_TEST_CASE(small_sparse_random) {\n RandomTest(200, 15, 200);\n}\n\nBOOST_AUTO_TEST_CASE(medium_sparse_random) {\n RandomTest(32000, 1000, 2000);\n}\n\nBOOST_AUTO_TEST_CASE(sparse_random) {\n RandomTest(std::numeric_limits::max(), 100000, 2000);\n}\n\n} // namespace\n} // namespace util\n" }, { "path": "src/kenlm/util/string_piece.cc", "content": "// Copyright 2004 The RE2 Authors. All Rights Reserved.\n// Use of this source code is governed by a BSD-style\n// license that can be found in string_piece.hh.\n\n#include \"util/string_piece.hh\"\n\n#include \n\n#include \n\n#ifndef HAVE_ICU\n\ntypedef StringPiece::size_type size_type;\n\nvoid StringPiece::CopyToString(std::string* target) const {\n target->assign(ptr_, length_);\n}\n\nsize_type StringPiece::find(const StringPiece& s, size_type pos) const {\n // Not sure why length_ < 0 was here since it's std::size_t.\n if (/*length_ < 0 || */pos > static_cast(length_))\n return npos;\n\n const char* result = std::search(ptr_ + pos, ptr_ + length_,\n s.ptr_, s.ptr_ + s.length_);\n const size_type xpos = result - ptr_;\n return xpos + s.length_ <= length_ ? xpos : npos;\n}\n\nsize_type StringPiece::find(char c, size_type pos) const {\n if (length_ <= 0 || pos >= static_cast(length_)) {\n return npos;\n }\n const char* result = std::find(ptr_ + pos, ptr_ + length_, c);\n return result != ptr_ + length_ ? result - ptr_ : npos;\n}\n\nsize_type StringPiece::rfind(const StringPiece& s, size_type pos) const {\n if (length_ < s.length_) return npos;\n const size_t ulen = length_;\n if (s.length_ == 0) return std::min(ulen, pos);\n\n const char* last = ptr_ + std::min(ulen - s.length_, pos) + s.length_;\n const char* result = std::find_end(ptr_, last, s.ptr_, s.ptr_ + s.length_);\n return result != last ? result - ptr_ : npos;\n}\n\nsize_type StringPiece::rfind(char c, size_type pos) const {\n if (length_ <= 0) return npos;\n for (int i = std::min(pos, static_cast(length_ - 1));\n i >= 0; --i) {\n if (ptr_[i] == c) {\n return i;\n }\n }\n return npos;\n}\n\n// For each character in characters_wanted, sets the index corresponding\n// to the ASCII code of that character to 1 in table. This is used by\n// the find_.*_of methods below to tell whether or not a character is in\n// the lookup table in constant time.\n// The argument `table' must be an array that is large enough to hold all\n// the possible values of an unsigned char. Thus it should be be declared\n// as follows:\n// bool table[UCHAR_MAX + 1]\nstatic inline void BuildLookupTable(const StringPiece& characters_wanted,\n bool* table) {\n const size_type length = characters_wanted.length();\n const char* const data = characters_wanted.data();\n for (size_type i = 0; i < length; ++i) {\n table[static_cast(data[i])] = true;\n }\n}\n\nsize_type StringPiece::find_first_of(const StringPiece& s,\n size_type pos) const {\n if (length_ == 0 || s.length_ == 0)\n return npos;\n\n // Avoid the cost of BuildLookupTable() for a single-character search.\n if (s.length_ == 1)\n return find_first_of(s.ptr_[0], pos);\n\n bool lookup[UCHAR_MAX + 1] = { false };\n BuildLookupTable(s, lookup);\n for (size_type i = pos; i < length_; ++i) {\n if (lookup[static_cast(ptr_[i])]) {\n return i;\n }\n }\n return npos;\n}\n\nsize_type StringPiece::find_first_not_of(const StringPiece& s,\n size_type pos) const {\n if (length_ == 0)\n return npos;\n\n if (s.length_ == 0)\n return 0;\n\n // Avoid the cost of BuildLookupTable() for a single-character search.\n if (s.length_ == 1)\n return find_first_not_of(s.ptr_[0], pos);\n\n bool lookup[UCHAR_MAX + 1] = { false };\n BuildLookupTable(s, lookup);\n for (size_type i = pos; i < length_; ++i) {\n if (!lookup[static_cast(ptr_[i])]) {\n return i;\n }\n }\n return npos;\n}\n\nsize_type StringPiece::find_first_not_of(char c, size_type pos) const {\n if (length_ == 0)\n return npos;\n\n for (; pos < length_; ++pos) {\n if (ptr_[pos] != c) {\n return pos;\n }\n }\n return npos;\n}\n\nsize_type StringPiece::find_last_of(const StringPiece& s, size_type pos) const {\n if (length_ == 0 || s.length_ == 0)\n return npos;\n\n // Avoid the cost of BuildLookupTable() for a single-character search.\n if (s.length_ == 1)\n return find_last_of(s.ptr_[0], pos);\n\n bool lookup[UCHAR_MAX + 1] = { false };\n BuildLookupTable(s, lookup);\n for (size_type i = std::min(pos, length_ - 1); ; --i) {\n if (lookup[static_cast(ptr_[i])])\n return i;\n if (i == 0)\n break;\n }\n return npos;\n}\n\nsize_type StringPiece::find_last_not_of(const StringPiece& s,\n size_type pos) const {\n if (length_ == 0)\n return npos;\n\n size_type i = std::min(pos, length_ - 1);\n if (s.length_ == 0)\n return i;\n\n // Avoid the cost of BuildLookupTable() for a single-character search.\n if (s.length_ == 1)\n return find_last_not_of(s.ptr_[0], pos);\n\n bool lookup[UCHAR_MAX + 1] = { false };\n BuildLookupTable(s, lookup);\n for (; ; --i) {\n if (!lookup[static_cast(ptr_[i])])\n return i;\n if (i == 0)\n break;\n }\n return npos;\n}\n\nsize_type StringPiece::find_last_not_of(char c, size_type pos) const {\n if (length_ == 0)\n return npos;\n\n for (size_type i = std::min(pos, length_ - 1); ; --i) {\n if (ptr_[i] != c)\n return i;\n if (i == 0)\n break;\n }\n return npos;\n}\n\nStringPiece StringPiece::substr(size_type pos, size_type n) const {\n if (pos > length_) pos = length_;\n if (n > length_ - pos) n = length_ - pos;\n return StringPiece(ptr_ + pos, n);\n}\n\nconst size_type StringPiece::npos = size_type(-1);\n\n#endif // !HAVE_ICU\n" }, { "path": "src/kenlm/util/string_piece.hh", "content": "/* If you use ICU in your program, then compile with -DHAVE_ICU -licui18n. If\n * you don't use ICU, then this will use the Google implementation from Chrome.\n * This has been modified from the original version to let you choose.\n */\n\n// Copyright 2008, Google Inc.\n// All rights reserved.\n//\n// Redistribution and use in source and binary forms, with or without\n// modification, are permitted provided that the following conditions are\n// met:\n//\n// * Redistributions of source code must retain the above copyright\n// notice, this list of conditions and the following disclaimer.\n// * Redistributions in binary form must reproduce the above\n// copyright notice, this list of conditions and the following disclaimer\n// in the documentation and/or other materials provided with the\n// distribution.\n// * Neither the name of Google Inc. nor the names of its\n// contributors may be used to endorse or promote products derived from\n// this software without specific prior written permission.\n//\n// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n// \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n// Copied from strings/stringpiece.h with modifications\n//\n// A string-like object that points to a sized piece of memory.\n//\n// Functions or methods may use const StringPiece& parameters to accept either\n// a \"const char*\" or a \"string\" value that will be implicitly converted to\n// a StringPiece. The implicit conversion means that it is often appropriate\n// to include this .h file in other files rather than forward-declaring\n// StringPiece as would be appropriate for most other Google classes.\n//\n// Systematic usage of StringPiece is encouraged as it will reduce unnecessary\n// conversions from \"const char*\" to \"string\" and back again.\n//\n\n#ifndef BASE_STRING_PIECE_H__\n#define BASE_STRING_PIECE_H__\n\n#include \"util/have.hh\"\n\n#include \n#include \n#include \n\n#ifdef HAVE_ICU\n#include \n#include \n\n// Old versions of ICU don't define operator== and operator!=.\n#if (U_ICU_VERSION_MAJOR_NUM < 4) || ((U_ICU_VERSION_MAJOR_NUM == 4) && (U_ICU_VERSION_MINOR_NUM < 4))\n#warning You are using an old version of ICU. Consider upgrading to ICU >= 4.6.\ninline bool operator==(const StringPiece& x, const StringPiece& y) {\n if (x.size() != y.size())\n return false;\n\n return std::memcmp(x.data(), y.data(), x.size()) == 0;\n}\n\ninline bool operator!=(const StringPiece& x, const StringPiece& y) {\n return !(x == y);\n}\n#endif // old version of ICU\n\nU_NAMESPACE_BEGIN\n#else\n\n#include \n#include \n#include \n#include \n\n#ifdef WIN32\n#undef max\n#undef min\n#endif\n\nclass StringPiece {\n public:\n typedef size_t size_type;\n\n private:\n const char* ptr_;\n size_type length_;\n\n public:\n // We provide non-explicit singleton constructors so users can pass\n // in a \"const char*\" or a \"string\" wherever a \"StringPiece\" is\n // expected.\n StringPiece() : ptr_(NULL), length_(0) { }\n StringPiece(const char* str)\n : ptr_(str), length_((str == NULL) ? 0 : strlen(str)) { }\n StringPiece(const std::string& str)\n : ptr_(str.data()), length_(str.size()) { }\n StringPiece(const char* offset, size_type len)\n : ptr_(offset), length_(len) { }\n\n // data() may return a pointer to a buffer with embedded NULs, and the\n // returned buffer may or may not be null terminated. Therefore it is\n // typically a mistake to pass data() to a routine that expects a NUL\n // terminated string.\n const char* data() const { return ptr_; }\n size_type size() const { return length_; }\n size_type length() const { return length_; }\n bool empty() const { return length_ == 0; }\n\n void clear() { ptr_ = NULL; length_ = 0; }\n void set(const char* data, size_type len) { ptr_ = data; length_ = len; }\n void set(const char* str) {\n ptr_ = str;\n length_ = str ? strlen(str) : 0;\n }\n void set(const void* data, size_type len) {\n ptr_ = reinterpret_cast(data);\n length_ = len;\n }\n\n char operator[](size_type i) const { return ptr_[i]; }\n\n void remove_prefix(size_type n) {\n ptr_ += n;\n length_ -= n;\n }\n\n void remove_suffix(size_type n) {\n length_ -= n;\n }\n\n int compare(const StringPiece& x) const {\n int r = wordmemcmp(ptr_, x.ptr_, std::min(length_, x.length_));\n if (r == 0) {\n if (length_ < x.length_) r = -1;\n else if (length_ > x.length_) r = +1;\n }\n return r;\n }\n\n std::string as_string() const {\n // std::string doesn't like to take a NULL pointer even with a 0 size.\n return std::string(!empty() ? data() : \"\", size());\n }\n\n void CopyToString(std::string* target) const;\n void AppendToString(std::string* target) const;\n\n // Does \"this\" start with \"x\"\n bool starts_with(const StringPiece& x) const {\n return ((length_ >= x.length_) &&\n (wordmemcmp(ptr_, x.ptr_, x.length_) == 0));\n }\n\n // Does \"this\" end with \"x\"\n bool ends_with(const StringPiece& x) const {\n return ((length_ >= x.length_) &&\n (wordmemcmp(ptr_ + (length_-x.length_), x.ptr_, x.length_) == 0));\n }\n\n // standard STL container boilerplate\n typedef char value_type;\n typedef const char* pointer;\n typedef const char& reference;\n typedef const char& const_reference;\n typedef ptrdiff_t difference_type;\n static const size_type npos;\n typedef const char* const_iterator;\n typedef const char* iterator;\n typedef std::reverse_iterator const_reverse_iterator;\n typedef std::reverse_iterator reverse_iterator;\n iterator begin() const { return ptr_; }\n iterator end() const { return ptr_ + length_; }\n const_reverse_iterator rbegin() const {\n return const_reverse_iterator(ptr_ + length_);\n }\n const_reverse_iterator rend() const {\n return const_reverse_iterator(ptr_);\n }\n\n size_type max_size() const { return length_; }\n size_type capacity() const { return length_; }\n\n size_type copy(char* buf, size_type n, size_type pos = 0) const;\n\n size_type find(const StringPiece& s, size_type pos = 0) const;\n size_type find(char c, size_type pos = 0) const;\n size_type rfind(const StringPiece& s, size_type pos = npos) const;\n size_type rfind(char c, size_type pos = npos) const;\n\n size_type find_first_of(const StringPiece& s, size_type pos = 0) const;\n size_type find_first_of(char c, size_type pos = 0) const {\n return find(c, pos);\n }\n size_type find_first_not_of(const StringPiece& s, size_type pos = 0) const;\n size_type find_first_not_of(char c, size_type pos = 0) const;\n size_type find_last_of(const StringPiece& s, size_type pos = npos) const;\n size_type find_last_of(char c, size_type pos = npos) const {\n return rfind(c, pos);\n }\n size_type find_last_not_of(const StringPiece& s, size_type pos = npos) const;\n size_type find_last_not_of(char c, size_type pos = npos) const;\n\n StringPiece substr(size_type pos, size_type n = npos) const;\n\n static int wordmemcmp(const char* p, const char* p2, size_type N) {\n return memcmp(p, p2, N);\n }\n};\n\ninline bool operator==(const StringPiece& x, const StringPiece& y) {\n if (x.size() != y.size())\n return false;\n\n return std::memcmp(x.data(), y.data(), x.size()) == 0;\n}\n\ninline bool operator!=(const StringPiece& x, const StringPiece& y) {\n return !(x == y);\n}\n\n#endif // HAVE_ICU undefined\n\ninline bool operator<(const StringPiece& x, const StringPiece& y) {\n const int r = std::memcmp(x.data(), y.data(),\n std::min(x.size(), y.size()));\n return ((r < 0) || ((r == 0) && (x.size() < y.size())));\n}\n\ninline bool operator>(const StringPiece& x, const StringPiece& y) {\n return y < x;\n}\n\ninline bool operator<=(const StringPiece& x, const StringPiece& y) {\n return !(x > y);\n}\n\ninline bool operator>=(const StringPiece& x, const StringPiece& y) {\n return !(x < y);\n}\n\n// allow StringPiece to be logged (needed for unit testing).\ninline std::ostream& operator<<(std::ostream& o, const StringPiece& piece) {\n return o.write(piece.data(), static_cast(piece.size()));\n}\n\n#ifdef HAVE_ICU\nU_NAMESPACE_END\nusing U_NAMESPACE_QUALIFIER StringPiece;\n#endif\n\n#endif // BASE_STRING_PIECE_H__\n" }, { "path": "src/kenlm/util/string_piece_hash.hh", "content": "#ifndef UTIL_STRING_PIECE_HASH__\n#define UTIL_STRING_PIECE_HASH__\n\n#include \"util/string_piece.hh\"\n\n#include \n#include \n\ninline size_t hash_value(const StringPiece &str) {\n return boost::hash_range(str.data(), str.data() + str.length());\n}\n\n/* Support for lookup of StringPiece in boost::unordered_map */\nstruct StringPieceCompatibleHash : public std::unary_function {\n size_t operator()(const StringPiece &str) const {\n return hash_value(str);\n }\n};\n\nstruct StringPieceCompatibleEquals : public std::binary_function {\n bool operator()(const StringPiece &first, const StringPiece &second) const {\n return first == second;\n }\n};\ntemplate typename T::const_iterator FindStringPiece(const T &t, const StringPiece &key) {\n#if BOOST_VERSION < 104200\n std::string temp(key.data(), key.size());\n return t.find(temp);\n#else\n return t.find(key, StringPieceCompatibleHash(), StringPieceCompatibleEquals());\n#endif\n}\n\ntemplate typename T::iterator FindStringPiece(T &t, const StringPiece &key) {\n#if BOOST_VERSION < 104200\n std::string temp(key.data(), key.size());\n return t.find(temp);\n#else\n return t.find(key, StringPieceCompatibleHash(), StringPieceCompatibleEquals());\n#endif\n}\n\n#endif // UTIL_STRING_PIECE_HASH__\n" }, { "path": "src/kenlm/util/thread_pool.hh", "content": "#ifndef UTIL_THREAD_POOL__\n#define UTIL_THREAD_POOL__\n\n#include \"util/pcqueue.hh\"\n\n#include \n#include \n#include \n\n#include \n\n#include \n\nnamespace util {\n\ntemplate class Worker : boost::noncopyable {\n public:\n typedef HandlerT Handler;\n typedef typename Handler::Request Request;\n\n template Worker(PCQueue &in, Construct &construct, Request &poison)\n : in_(in), handler_(construct), thread_(boost::ref(*this)), poison_(poison) {}\n\n // Only call from thread.\n void operator()() {\n Request request;\n while (1) {\n in_.Consume(request);\n if (request == poison_) return;\n try {\n (*handler_)(request);\n }\n catch(std::exception &e) {\n std::cerr << \"Handler threw \" << e.what() << std::endl;\n abort();\n }\n catch(...) {\n std::cerr << \"Handler threw an exception, dropping request\" << std::endl;\n abort();\n }\n }\n }\n\n void Join() {\n thread_.join();\n }\n\n private:\n PCQueue &in_;\n\n boost::optional handler_;\n\n boost::thread thread_;\n\n Request poison_;\n};\n\ntemplate class ThreadPool : boost::noncopyable {\n public:\n typedef HandlerT Handler;\n typedef typename Handler::Request Request;\n\n template ThreadPool(size_t queue_length, size_t workers, Construct handler_construct, Request poison) : in_(queue_length), poison_(poison) {\n for (size_t i = 0; i < workers; ++i) {\n workers_.push_back(new Worker(in_, handler_construct, poison));\n }\n }\n\n ~ThreadPool() {\n for (size_t i = 0; i < workers_.size(); ++i) {\n Produce(poison_);\n }\n for (typename boost::ptr_vector >::iterator i = workers_.begin(); i != workers_.end(); ++i) {\n i->Join();\n }\n }\n\n void Produce(const Request &request) {\n in_.Produce(request);\n }\n\n // For adding to the queue.\n PCQueue &In() { return in_; }\n\n private:\n PCQueue in_;\n\n boost::ptr_vector > workers_;\n\n Request poison_;\n};\n\n} // namespace util\n\n#endif // UTIL_THREAD_POOL__\n" }, { "path": "src/kenlm/util/tokenize_piece.hh", "content": "#ifndef UTIL_TOKENIZE_PIECE__\n#define UTIL_TOKENIZE_PIECE__\n\n#include \"util/exception.hh\"\n#include \"util/string_piece.hh\"\n\n#include \n\n#include \n#include \n\nnamespace util {\n\n// Thrown on dereference when out of tokens to parse\nclass OutOfTokens : public Exception {\n public:\n OutOfTokens() throw() {}\n ~OutOfTokens() throw() {}\n};\n\nclass SingleCharacter {\n public:\n SingleCharacter() {}\n explicit SingleCharacter(char delim) : delim_(delim) {}\n\n StringPiece Find(const StringPiece &in) const {\n return StringPiece(std::find(in.data(), in.data() + in.size(), delim_), 1);\n }\n\n private:\n char delim_;\n};\n\nclass MultiCharacter {\n public:\n MultiCharacter() {}\n\n explicit MultiCharacter(const StringPiece &delimiter) : delimiter_(delimiter) {}\n\n StringPiece Find(const StringPiece &in) const {\n return StringPiece(std::search(in.data(), in.data() + in.size(), delimiter_.data(), delimiter_.data() + delimiter_.size()), delimiter_.size());\n }\n\n private:\n StringPiece delimiter_;\n};\n\nclass AnyCharacter {\n public:\n AnyCharacter() {}\n explicit AnyCharacter(const StringPiece &chars) : chars_(chars) {}\n\n StringPiece Find(const StringPiece &in) const {\n return StringPiece(std::find_first_of(in.data(), in.data() + in.size(), chars_.data(), chars_.data() + chars_.size()), 1);\n }\n\n private:\n StringPiece chars_;\n};\n\nclass AnyCharacterLast {\n public:\n AnyCharacterLast() {}\n\n explicit AnyCharacterLast(const StringPiece &chars) : chars_(chars) {}\n\n StringPiece Find(const StringPiece &in) const {\n return StringPiece(std::find_end(in.data(), in.data() + in.size(), chars_.data(), chars_.data() + chars_.size()), 1);\n }\n\n private:\n StringPiece chars_;\n};\n\ntemplate class TokenIter : public boost::iterator_facade, const StringPiece, boost::forward_traversal_tag> {\n public:\n TokenIter() {}\n\n template TokenIter(const StringPiece &str, const Construct &construct) : after_(str), finder_(construct) {\n increment();\n }\n\n bool operator!() const {\n return current_.data() == 0;\n }\n operator bool() const {\n return current_.data() != 0;\n }\n\n static TokenIter end() {\n return TokenIter();\n }\n\n private:\n friend class boost::iterator_core_access;\n\n void increment() {\n do {\n StringPiece found(finder_.Find(after_));\n current_ = StringPiece(after_.data(), found.data() - after_.data());\n if (found.data() == after_.data() + after_.size()) {\n after_ = StringPiece(NULL, 0);\n } else {\n after_ = StringPiece(found.data() + found.size(), after_.data() - found.data() + after_.size() - found.size());\n }\n } while (SkipEmpty && current_.data() && current_.empty()); // Compiler should optimize this away if SkipEmpty is false. \n }\n\n bool equal(const TokenIter &other) const {\n return current_.data() == other.current_.data();\n }\n\n const StringPiece &dereference() const {\n UTIL_THROW_IF(!current_.data(), OutOfTokens, \"Ran out of tokens\");\n return current_;\n }\n\n StringPiece current_;\n StringPiece after_;\n\n Find finder_;\n};\n\n} // namespace util\n\n#endif // UTIL_TOKENIZE_PIECE__\n" }, { "path": "src/kenlm/util/tokenize_piece_test.cc", "content": "#include \"util/tokenize_piece.hh\"\n#include \"util/string_piece.hh\"\n\n#define BOOST_TEST_MODULE TokenIteratorTest\n#include \n\n#include \n\nnamespace util {\nnamespace {\n\nBOOST_AUTO_TEST_CASE(pipe_pipe_none) {\n const char str[] = \"nodelimit at all\";\n TokenIter it(str, MultiCharacter(\"|||\"));\n BOOST_REQUIRE(it);\n BOOST_CHECK_EQUAL(StringPiece(str), *it);\n ++it;\n BOOST_CHECK(!it);\n}\nBOOST_AUTO_TEST_CASE(pipe_pipe_two) {\n const char str[] = \"|||\";\n TokenIter it(str, MultiCharacter(\"|||\"));\n BOOST_REQUIRE(it);\n BOOST_CHECK_EQUAL(StringPiece(), *it);\n ++it;\n BOOST_REQUIRE(it);\n BOOST_CHECK_EQUAL(StringPiece(), *it);\n ++it;\n BOOST_CHECK(!it);\n}\n\nBOOST_AUTO_TEST_CASE(remove_empty) {\n const char str[] = \"|||\";\n TokenIter it(str, MultiCharacter(\"|||\"));\n BOOST_CHECK(!it);\n}\n\nBOOST_AUTO_TEST_CASE(remove_empty_keep) {\n const char str[] = \" |||\";\n TokenIter it(str, MultiCharacter(\"|||\"));\n BOOST_REQUIRE(it);\n BOOST_CHECK_EQUAL(StringPiece(\" \"), *it);\n ++it;\n BOOST_CHECK(!it);\n}\n\n} // namespace\n} // namespace util\n" }, { "path": "src/kenlm/util/usage.cc", "content": "#include \"util/usage.hh\"\n\n#include \"util/exception.hh\"\n\n#include \n#include \n#include \n\n#include \n#include \n#if !defined(_WIN32) && !defined(_WIN64)\n#include \n#include \n#include \n#endif\n\nnamespace util {\n\nnamespace {\n#if !defined(_WIN32) && !defined(_WIN64)\nfloat FloatSec(const struct timeval &tv) {\n return static_cast(tv.tv_sec) + (static_cast(tv.tv_usec) / 1000000.0);\n}\n#endif\n} // namespace\n\nvoid PrintUsage(std::ostream &out) {\n#if !defined(_WIN32) && !defined(_WIN64)\n struct rusage usage;\n if (getrusage(RUSAGE_SELF, &usage)) {\n perror(\"getrusage\");\n return;\n }\n out << \"user\\t\" << FloatSec(usage.ru_utime) << \"\\nsys\\t\" << FloatSec(usage.ru_stime) << '\\n';\n\n // Linux doesn't set memory usage :-(. \n std::ifstream status(\"/proc/self/status\", std::ios::in);\n std::string line;\n while (getline(status, line)) {\n if (!strncmp(line.c_str(), \"VmRSS:\\t\", 7)) {\n out << \"VmRSS: \" << (line.c_str() + 7) << '\\n';\n break;\n } else if (!strncmp(line.c_str(), \"VmPeak:\\t\", 8)) {\n out << \"VmPeak: \" << (line.c_str() + 8) << '\\n';\n }\n }\n#endif\n}\n\nuint64_t GuessPhysicalMemory() {\n#if defined(_WIN32) || defined(_WIN64)\n return 0;\n#elif defined(_SC_PHYS_PAGES) && defined(_SC_PAGESIZE)\n long pages = sysconf(_SC_PHYS_PAGES);\n if (pages == -1) return 0;\n long page_size = sysconf(_SC_PAGESIZE);\n if (page_size == -1) return 0;\n return static_cast(pages) * static_cast(page_size);\n#else\n return 0;\n#endif\n}\n\nnamespace {\nclass SizeParseError : public Exception {\n public:\n explicit SizeParseError(const std::string &str) throw() {\n *this << \"Failed to parse \" << str << \" into a memory size \";\n }\n};\n\ntemplate uint64_t ParseNum(const std::string &arg) {\n std::stringstream stream(arg);\n Num value;\n stream >> value;\n UTIL_THROW_IF_ARG(!stream, SizeParseError, (arg), \"for the leading number.\");\n std::string after;\n stream >> after;\n UTIL_THROW_IF_ARG(after.size() > 1, SizeParseError, (arg), \"because there are more than two characters after the number.\");\n std::string throwaway;\n UTIL_THROW_IF_ARG(stream >> throwaway, SizeParseError, (arg), \"because there was more cruft \" << throwaway << \" after the number.\");\n\n // Silly sort, using kilobytes as your default unit. \n if (after.empty()) after = \"K\";\n if (after == \"%\") {\n uint64_t mem = GuessPhysicalMemory();\n UTIL_THROW_IF_ARG(!mem, SizeParseError, (arg), \"because % was specified but the physical memory size could not be determined.\");\n return static_cast(value) * static_cast(mem) / 100.0;\n }\n \n std::string units(\"bKMGTPEZY\");\n std::string::size_type index = units.find(after[0]);\n UTIL_THROW_IF_ARG(index == std::string::npos, SizeParseError, (arg), \"the allowed suffixes are \" << units << \"%.\");\n for (std::string::size_type i = 0; i < index; ++i) {\n value *= 1024;\n }\n return value;\n}\n\n} // namespace\n\nuint64_t ParseSize(const std::string &arg) {\n return arg.find('.') == std::string::npos ? ParseNum(arg) : ParseNum(arg);\n}\n\n} // namespace util\n" }, { "path": "src/kenlm/util/usage.hh", "content": "#ifndef UTIL_USAGE__\n#define UTIL_USAGE__\n#include \n#include \n#include \n\n#include \n\nnamespace util {\nvoid PrintUsage(std::ostream &to);\n\n// Determine how much physical memory there is. Return 0 on failure.\nuint64_t GuessPhysicalMemory();\n\n// Parse a size like unix sort. Sadly, this means the default multiplier is K.\nuint64_t ParseSize(const std::string &arg);\n} // namespace util\n#endif // UTIL_USAGE__\n" }, { "path": "src/opennlp/ccg/Parse.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2010 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg;\r\n\r\nimport java.io.*;\r\nimport java.net.URL;\r\nimport java.util.HashMap;\r\nimport java.util.Map;\r\n\r\nimport org.jdom.Document;\r\nimport org.jdom.Element;\r\nimport org.jdom.output.Format;\r\nimport org.jdom.output.XMLOutputter;\r\n\r\nimport opennlp.ccg.grammar.Grammar;\r\nimport opennlp.ccg.hylo.HyloHelper;\r\nimport opennlp.ccg.hylo.Nominal;\r\nimport opennlp.ccg.lexicon.Tokenizer;\r\nimport opennlp.ccg.parse.ParseException;\r\nimport opennlp.ccg.parse.Parser;\r\nimport opennlp.ccg.parse.Supertagger;\r\nimport opennlp.ccg.parse.supertagger.WordAndPOSDictionaryLabellingStrategy;\r\nimport opennlp.ccg.synsem.Category;\r\nimport opennlp.ccg.synsem.LF;\r\nimport opennlp.ccg.synsem.Sign;\r\nimport opennlp.ccg.synsem.SignScorer;\r\nimport opennlp.ccg.test.RegressionInfo;\r\nimport opennlp.ccg.test.DerivMaker;\r\nimport opennlp.ccgbank.extract.Testbed;\r\n\r\n/**\r\n * Creates a testbed file by parsing a text file.\r\n * Text is assumed to be tokenized, with one sentence per line.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.2 $, $Date: 2010/10/28 02:46:32 $\r\n */\r\npublic class Parse {\r\n\r\n\tpublic static void main(String[] args) throws IOException {\r\n\t\t\r\n String usage = \"Usage: java opennlp.ccg.Parse \\n\" + \r\n \t\" (-g ) \\n\" + \r\n \t\" -parsescorer \\n\" +\r\n \t\" -supertagger | -stconfig \\n\" +\r\n\t \" (-nbestListSize ) \\n\" +\r\n\t \" (-includederivs) \\n\" +\r\n\t \" (-includescores) \\n\" +\r\n \t\" \";\r\n \r\n if (args.length == 0 || args[0].equals(\"-h\")) {\r\n System.out.println(usage);\r\n System.exit(0);\r\n }\r\n \r\n // args\r\n String grammarfile = \"grammar.xml\";\r\n String inputfile = null;\r\n String outputfile = null;\r\n String parseScorerClass = null;\r\n String supertaggerClass = null, stconfig = null;\r\n boolean includederivs = false;\r\n boolean includescores = false;\r\n int nbestListSize = 1;\r\n \r\n for (int i = 0; i < args.length; i++) {\r\n \tif (args[i].equals(\"-g\")) { grammarfile = args[++i]; continue; }\r\n if (args[i].equals(\"-parsescorer\")) { parseScorerClass = args[++i]; continue; }\r\n if (args[i].equals(\"-supertagger\")) { supertaggerClass = args[++i]; continue; }\r\n if (args[i].equals(\"-stconfig\")) { stconfig = args[++i]; continue; }\r\n if (args[i].equals(\"-nbestListSize\")) { nbestListSize = Integer.parseInt(args[++i]); continue; }\r\n if (args[i].equals(\"-includederivs\")) { includederivs = true; continue; }\r\n if (args[i].equals(\"-includescores\")) { includescores = true; continue; }\r\n if (inputfile == null) { inputfile = args[i]; continue; }\r\n outputfile = args[i];\r\n }\r\n if (nbestListSize < 1) nbestListSize = 1;\r\n\r\n if (inputfile == null || outputfile == null || \r\n \tparseScorerClass == null || (supertaggerClass == null && stconfig == null)) \r\n {\r\n System.out.println(usage);\r\n System.exit(0);\r\n }\r\n \r\n\t\t// make test doc, sign map\r\n\t\tDocument outDoc = new Document();\r\n\t\tElement outRoot = new Element(\"regression\");\r\n\t\toutDoc.setRootElement(outRoot);\r\n\t\tMap signMap = new HashMap();\r\n\r\n // load grammar\r\n URL grammarURL = new File(grammarfile).toURI().toURL();\r\n System.out.println(\"Loading grammar from URL: \" + grammarURL);\r\n Grammar grammar = new Grammar(grammarURL);\r\n Tokenizer tokenizer = grammar.lexicon.tokenizer;\r\n System.out.println();\r\n \r\n // set up parser\r\n Parser parser = new Parser(grammar);\r\n // instantiate scorer\r\n try {\r\n System.out.println(\"Instantiating parsing sign scorer from class: \" + parseScorerClass);\r\n SignScorer parseScorer = (SignScorer) Class.forName(parseScorerClass).newInstance();\r\n parser.setSignScorer(parseScorer);\r\n System.out.println();\r\n } catch (Exception exc) {\r\n throw (RuntimeException) new RuntimeException().initCause(exc);\r\n }\r\n // instantiate supertagger\r\n try {\r\n \tSupertagger supertagger;\r\n \tif (supertaggerClass != null) {\r\n System.out.println(\"Instantiating supertagger from class: \" + supertaggerClass);\r\n supertagger = (Supertagger) Class.forName(supertaggerClass).newInstance();\r\n \t}\r\n \telse {\r\n \t\tSystem.out.println(\"Instantiating supertagger from config file: \" + stconfig);\r\n \t\tsupertagger = WordAndPOSDictionaryLabellingStrategy.supertaggerFactory(stconfig);\r\n \t}\r\n parser.setSupertagger(supertagger);\r\n System.out.println();\r\n } catch (Exception exc) {\r\n throw (RuntimeException) new RuntimeException().initCause(exc);\r\n }\r\n \r\n // loop through input\r\n BufferedReader in = new BufferedReader(new FileReader(inputfile));\r\n String line;\r\n Map predInfoMap = new HashMap();\r\n System.out.println(\"Parsing \" + inputfile);\r\n System.out.println();\r\n int count = 1;\r\n while ((line = in.readLine()) != null) {\r\n \tString id = \"s\" + count;\r\n \ttry {\r\n \t\t// parse it\r\n \t\tSystem.out.println(line);\r\n \t\tparser.parse(line);\r\n \t\tint numParses = Math.min(nbestListSize, parser.getResult().size());\r\n \t\tfor (int i=0; i < numParses; i++) {\r\n \t\t\tSign thisParse = parser.getResult().get(i);\r\n \t\t\t// convert lf\r\n \t\t\tCategory cat = thisParse.getCategory();\r\n \t\t\tLF convertedLF = null;\r\n \t\t\tString predInfo = null;\r\n \t\t\tif (cat.getLF() != null) {\r\n \t\t\t\t// convert LF\r\n \t\t\t\tLF flatLF = cat.getLF();\r\n \t\t\t\tcat = cat.copy();\r\n \t\t\t\tNominal index = cat.getIndexNominal(); \r\n \t\t\t\tconvertedLF = HyloHelper.compactAndConvertNominals(flatLF, index, thisParse);\r\n \t\t\t\t// get pred info\r\n \t\t\t\tpredInfoMap.clear();\r\n \t\t\t\tTestbed.extractPredInfo(flatLF, predInfoMap);\r\n \t\t\t\tpredInfo = Testbed.getPredInfo(predInfoMap);\r\n \t\t\t}\r\n \t\t\t// add test item, sign\r\n \t\t\tElement item = RegressionInfo.makeTestItem(grammar, line, 1, convertedLF);\r\n \t\t\tString actualID = (nbestListSize == 1) ? id : id + \"-\" + (i+1);\r\n \t\t\titem.setAttribute(\"info\", actualID);\r\n \t\t\titem.setAttribute(\"test\",\"true\");\r\n \t\t\toutRoot.addContent(item);\r\n \t\t\tsignMap.put(actualID, thisParse);\r\n \t\t\t// Add parsed words as a separate LF element\r\n \t\t\tElement fullWordsElt = new Element(\"full-words\");\r\n \t\t\tfullWordsElt.addContent(tokenizer.format(thisParse.getWords()));\r\n \t\t\titem.addContent(fullWordsElt);\r\n \t\t\tif (predInfo != null) {\r\n \t\t\t\tElement predInfoElt = new Element(\"pred-info\");\r\n \t\t\t\tpredInfoElt.setAttribute(\"data\", predInfo);\r\n \t\t\t\titem.addContent(predInfoElt);\r\n \t\t\t}\r\n \t\t\tif (includederivs) {\r\n \t\t\t\tElement derivElt = new Element(\"deriv\");\r\n \t\t\t\tderivElt.addContent(DerivMaker.makeDeriv(thisParse));\r\n \t\t\t\titem.addContent(derivElt);\r\n \t\t\t}\r\n \t\t\tif (includescores) {\r\n \t\t\t\tString score = parser.getScores().get(i).toString();\r\n \t\t\t\titem.setAttribute(\"score\", score);\r\n \t\t\t}\r\n \t\t}\r\n \t} catch (ParseException e) {\r\n \t\tSystem.out.println(\"Unable to parse!\");\r\n \t\t// add test item with zero parses\r\n \t\tElement item = RegressionInfo.makeTestItem(grammar, line, 0, null);\r\n \t\titem.setAttribute(\"info\", id);\r\n \t\toutRoot.addContent(item);\r\n \t}\r\n \tcount++;\r\n }\r\n System.out.println();\r\n\r\n\t\t// write test doc, saved signs\r\n System.out.println(\"Writing parses to \" + outputfile);\r\n\t\tXMLOutputter outputter = new XMLOutputter(Format.getPrettyFormat());\r\n\t\tFile regressionFile = new File(outputfile);\r\n\t\toutputter.output(outDoc, new FileOutputStream(regressionFile));\r\n\t\tRegressionInfo.writeSerFile(signMap, regressionFile);\r\n System.out.println();\r\n\t\t\r\n // done\r\n in.close();\r\n System.out.println(\"Done.\");\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccg/Realize.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2003-7 University of Edinburgh, Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg;\r\n\r\nimport opennlp.ccg.realize.*;\r\nimport opennlp.ccg.realize.hypertagger.ZLMaxentHypertagger;\r\nimport opennlp.ccg.grammar.*;\r\nimport opennlp.ccg.synsem.*;\r\nimport opennlp.ccg.ngrams.*;\r\n\r\nimport org.jdom.*;\r\nimport org.jdom.output.*;\r\n\r\nimport java.io.*;\r\nimport java.net.*;\r\nimport java.util.*;\r\nimport java.util.prefs.*;\r\n\r\n/**\r\n * Sample front-end to the realizer, showing the intermediate steps of realization.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.38 $, $Date: 2011/08/10 17:58:45 $\r\n */\r\npublic class Realize\r\n{\r\n private static PrintWriter out;\r\n \r\n @SuppressWarnings(\"unchecked\")\r\n\tpublic static void main(String[] args) throws Exception {\r\n \r\n String usage = \"Usage: java opennlp.ccg.Realize (-g ) (-exactmatches) (-ngramorder N) ()\";\r\n \r\n if (args.length > 0 && args[0].equals(\"-h\")) {\r\n System.out.println(usage);\r\n System.exit(0);\r\n }\r\n \r\n // args\r\n String grammarfile = \"grammar.xml\";\r\n String inputfile = null;\r\n String outputfile = null;\r\n boolean exactMatches = false;\r\n int ngramOrder = 0;\r\n for (int i = 0; i < args.length; i++) {\r\n \tif (args[i].startsWith(\"-D\")) {\r\n \t\tString prop = args[i].substring(2); int equalpos = prop.indexOf(\"=\");\r\n \t\tString key = prop.substring(0, equalpos); String val = prop.substring(equalpos+1);\r\n \t\tSystem.setProperty(key, val); continue;\r\n \t}\r\n if (args[i].equals(\"-g\")) { grammarfile = args[++i]; continue; }\r\n if (args[i].equals(\"-exactmatches\")) { exactMatches = true; continue; }\r\n if (args[i].equals(\"-ngramorder\")) { ngramOrder = Integer.parseInt(args[++i]); continue; }\r\n if (inputfile == null) { inputfile = args[i]; continue; }\r\n outputfile = args[i];\r\n }\r\n if (inputfile == null) {\r\n System.out.println(usage);\r\n System.exit(0);\r\n }\r\n \r\n // set out accordingly\r\n if (outputfile != null) {\r\n out = new PrintWriter(new BufferedWriter(new FileWriter(outputfile)));\r\n }\r\n else {\r\n out = new PrintWriter(System.out); \r\n }\r\n \r\n // remember, modify prefs\r\n Preferences prefs = Preferences.userNodeForPackage(TextCCG.class);\r\n boolean oldShowCompleteness = prefs.getBoolean(Edge.SHOW_COMPLETENESS, false);\r\n boolean oldShowBitset = prefs.getBoolean(Edge.SHOW_BITSET, false);\r\n prefs.putBoolean(Edge.SHOW_COMPLETENESS, true);\r\n prefs.putBoolean(Edge.SHOW_BITSET, true);\r\n \r\n // load grammar\r\n URL grammarURL = new File(grammarfile).toURI().toURL();\r\n out.println(\"Loading grammar from URL: \" + grammarURL);\r\n Grammar grammar = new Grammar(grammarURL);\r\n\r\n // instantiate realizer \r\n Realizer realizer = new Realizer(grammar);\r\n \r\n // get request\r\n out.println();\r\n out.println(\"Request:\");\r\n out.println();\r\n Document doc = grammar.loadFromXml(inputfile);\r\n org.jdom.output.XMLOutputter outputter = new org.jdom.output.XMLOutputter(Format.getPrettyFormat()); \r\n out.flush();\r\n outputter.output(doc, out);\r\n out.flush();\r\n \r\n LF lf = Realizer.getLfFromDoc(doc);\r\n out.println();\r\n out.println(\"** Initial run\");\r\n out.println();\r\n out.println(\"Input LF: \" + lf);\r\n \r\n // set up n-gram scorer\r\n SignScorer ngramScorer;\r\n Element root = doc.getRootElement();\r\n Element ngramModelElt = root.getChild(\"ngram-model\");\r\n if (ngramModelElt == null) {\r\n // just use targets\r\n List targetElts = root.getChildren(\"target\");\r\n String[] targets = new String[targetElts.size()];\r\n out.println();\r\n out.println(\"Targets:\");\r\n for (int i=0; i < targetElts.size(); i++) {\r\n Element ex = (Element) targetElts.get(i);\r\n String target = ex.getText();\r\n out.println(target);\r\n targets[i] = target;\r\n }\r\n if (ngramOrder > 0) {\r\n out.println();\r\n out.println(\"Using order \" + ngramOrder + \" in n-gram precision.\");\r\n \tngramScorer = new NgramPrecisionModel(targets, ngramOrder); \r\n }\r\n else ngramScorer = new NgramPrecisionModel(targets);\r\n if (exactMatches) {\r\n out.println();\r\n out.println(\"Only counting exact matches in n-gram precision.\");\r\n ((NgramPrecisionModel)ngramScorer).setExactMatches(exactMatches);\r\n }\r\n }\r\n else if (ngramModelElt.getAttributeValue(\"class\") != null) {\r\n // load scorer from class\r\n String scorerClass = ngramModelElt.getAttributeValue(\"class\");\r\n out.println();\r\n out.println(\"Instantiating scorer from class: \" + scorerClass);\r\n ngramScorer = (SignScorer) Class.forName(scorerClass).newInstance();\r\n }\r\n else {\r\n // load n-gram model\r\n String filename = ngramModelElt.getAttributeValue(\"file\");\r\n String reverseStr = ngramModelElt.getAttributeValue(\"reverse\");\r\n boolean reverse = (reverseStr != null) ? reverseStr.equals(\"true\") : false; \r\n String factoredStr = ngramModelElt.getAttributeValue(\"factored\");\r\n boolean factored = (factoredStr != null) ? factoredStr.equals(\"true\") : false; \r\n String semClassesStr = ngramModelElt.getAttributeValue(\"sem-classes\");\r\n boolean useSemClasses = (semClassesStr != null) ? semClassesStr.equals(\"true\") : true;\r\n int order = 3; // order can only be changed for standard n-gram models\r\n String orderStr = ngramModelElt.getAttributeValue(\"order\"); \r\n if (orderStr != null) { order = Integer.parseInt(orderStr); }\r\n if (ngramOrder > 0) order = ngramOrder; // preference given to command-line value\r\n out.println();\r\n String msg = \"Loading \";\r\n if (reverse) msg += \"reversed \";\r\n if (factored) msg += \"factored \";\r\n msg += \"n-gram model \";\r\n if (!factored) msg += \"of order \" + order + \" \";\r\n if (useSemClasses) msg += \"with semantic class replacement \";\r\n msg += \"from: \" + filename;\r\n out.println(msg);\r\n if (factored)\r\n ngramScorer = new FactoredNgramModelFamily(filename, useSemClasses);\r\n else\r\n ngramScorer = new StandardNgramModel(order, filename, useSemClasses);\r\n if (reverse) ((NgramScorer)ngramScorer).setReverse(true);\r\n }\r\n \r\n // set pruning strategy (if any)\r\n Element pruningStrategyElt = root.getChild(\"pruning-strategy\");\r\n if (pruningStrategyElt != null) {\r\n // load pruning strategy from class\r\n String pruningStrategyClass = pruningStrategyElt.getAttributeValue(\"class\");\r\n out.println();\r\n out.println(\"Instantiating pruning strategy from class: \" + pruningStrategyClass);\r\n realizer.pruningStrategy = (PruningStrategy) \r\n Class.forName(pruningStrategyClass).newInstance();\r\n }\r\n \r\n // set hypertagger (if any)\r\n Element htModelElt = root.getChild(\"ht-model\");\r\n if (htModelElt != null) {\r\n String htconfig = htModelElt.getAttributeValue(\"config\");\r\n if (htconfig != null) {\r\n out.println();\r\n out.println(\"Instantiating hypertagger from: \" + htconfig);\r\n \trealizer.hypertagger = ZLMaxentHypertagger.ZLMaxentHypertaggerFactory(htconfig);\r\n }\r\n else {\r\n\t String htModelClass = htModelElt.getAttributeValue(\"class\");\r\n\t out.println();\r\n\t out.println(\"Instantiating hypertagger from class: \" + htModelClass);\r\n\t realizer.hypertagger = (Hypertagger) Class.forName(htModelClass).newInstance();\r\n }\r\n }\r\n\r\n // run request\r\n realizer.realize(lf, ngramScorer);\r\n Chart chart = realizer.getChart();\r\n chart.out = out;\r\n\r\n out.println();\r\n out.println(\"Preds:\");\r\n chart.printEPs();\r\n \r\n out.println();\r\n out.println(\"LF chunks:\");\r\n chart.printLfChunks();\r\n\r\n out.println();\r\n out.println(\"LF alts:\");\r\n chart.printLfAlts();\r\n\r\n out.println();\r\n out.println(\"LF optional parts:\");\r\n chart.printLfOpts();\r\n\r\n out.println();\r\n out.println(\"Initial Edges:\");\r\n chart.printInitialEdges();\r\n\r\n out.println();\r\n out.println(\"Marked Edges:\");\r\n chart.printMarkedEdges(); \r\n \r\n out.println();\r\n out.println(\"Instantiated Semantically Null Edges:\");\r\n chart.printInstantiatedNoSemEdges();\r\n\r\n out.println();\r\n out.println(\"Uninstantiated Semantically Null Edges:\");\r\n chart.printNoSemEdges();\r\n\r\n out.println();\r\n out.println(\"Rule Instances:\");\r\n chart.printRuleInstances();\r\n\r\n out.println();\r\n out.println(\"All Edges:\");\r\n chart.printEdges();\r\n\r\n out.println();\r\n out.println(\"Complete Edges (unsorted):\");\r\n chart.printEdges(true);\r\n\r\n out.println();\r\n out.println(\"Complete Edges (sorted):\");\r\n chart.printEdges(true, true);\r\n\r\n out.println();\r\n out.println(\"Best Edge:\");\r\n chart.printBestEdge();\r\n \r\n out.println();\r\n out.println(\"Best Edge Derivation:\");\r\n out.println(chart.bestEdge.getSign().getDerivationHistory());\r\n out.flush();\r\n \r\n if (chart.bestJoinedEdge != null) {\r\n out.println();\r\n out.println(\"Best Joined Edge:\");\r\n chart.printBestJoinedEdge();\r\n \r\n out.println();\r\n out.println(\"Best Joined Edge Derivation:\");\r\n out.println(chart.bestJoinedEdge.getSign().getDerivationHistory());\r\n out.flush();\r\n }\r\n\r\n\r\n // reset prefs\r\n prefs.putBoolean(Edge.SHOW_COMPLETENESS, oldShowCompleteness);\r\n prefs.putBoolean(Edge.SHOW_BITSET, oldShowBitset);\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/TextCCG.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-4 Jason Baldridge, Gann Bierner, \n// University of Edinburgh (Michael White), \n// Alexandros Triantafyllidis and David Reitter\n// Copyright (C) 2006 Ben Wing\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg;\n\nimport opennlp.ccg.lexicon.*;\nimport opennlp.ccg.grammar.*;\nimport opennlp.ccg.parse.*;\nimport opennlp.ccg.util.*;\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.realize.*;\nimport opennlp.ccg.hylo.*;\nimport opennlp.ccg.ngrams.*;\nimport opennlp.ccg.test.*;\nimport opennlp.ccg.realize.Edge; // only realization edges referenced (for preferences)\n\nimport org.jdom.*;\n\nimport java.io.*;\nimport java.net.*;\nimport java.util.*;\nimport java.util.prefs.*;\n\n/**\n * A text interface for testing grammars.\n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @author Alexandros Triantafyllidis\n * @author David Reitter\n * @version $Revision: 1.67 $, $Date: 2011/12/13 04:00:54 $\n */\npublic class TextCCG {\n \n /** Preference key for showing all results. */\n public static final String SHOW_ALL_RESULTS = \"Show All Results\";\n \n /** Preference key for showing derivations. */\n public static final String SHOW_DERIVATIONS = \"Show Derivations\";\n \n /** Preference key for showing features. */\n public static final String SHOW_FEATURES = \"Show Features\";\n \n /** Preference key for showing semantics. */\n public static final String SHOW_SEMANTICS = \"Show Semantics\";\n \n /** Preference key for showing features. */\n public static final String FEATURES_TO_SHOW = \"Features to Show\";\n \n /** Preference key for showing realizer timing. */\n public static final String SHOW_TIMING = \"Show Timing\";\n \n /** Preference key for showing incomplete edges during realization. */\n public static final String SHOW_INCOMPLETE_EDGES = \"Show Incomplete Edges\";\n \n /** Preference key for visualizing a derivation. */\n public static final String VISUALIZE = \"Visualize\";\n\n /** Preference key for command line history. */\n public static final String HISTORY = \"Command Line History\";\t\n\t\n\t\n /** Main method for tccg. */\n @SuppressWarnings(\"unchecked\")\n\tpublic static void main(String[] args) throws IOException, LexException { \n\n String usage = \"java opennlp.ccg.TextCCG \" + \n \"() | (-exportprefs ) | (-importprefs )\";\n\n if (args.length > 0 && args[0].equals(\"-h\")) {\n System.out.println(\"Usage: \" + usage);\n System.exit(0);\n }\n\n // args \n String grammarfile = \"grammar.xml\";\n String prefsfile = null;\n boolean exportPrefs = false;\n boolean importPrefs = false;\n for (int i = 0; i < args.length; i++) {\n if (args[i].equals(\"-exportprefs\")) {\n exportPrefs = true; prefsfile = args[++i]; continue; \n }\n if (args[i].equals(\"-importprefs\")) {\n importPrefs = true; prefsfile = args[++i]; continue; \n }\n grammarfile = args[i];\n }\n\n // prefs\n Preferences prefs = Preferences.userNodeForPackage(TextCCG.class);\n try {\n if (exportPrefs) {\n System.out.println(\"Exporting preferences to prefsfile: \" + prefsfile);\n prefs.exportNode(new FileOutputStream(prefsfile));\n return;\n }\n if (importPrefs) {\n System.out.println(\"Importing preferences from prefsfile: \" + prefsfile);\n Preferences.importPreferences(new FileInputStream(prefsfile));\n return;\n }\n } catch (Exception exc) {\n throw (IOException) new IOException().initCause(exc);\n }\n\n // load grammar\n URL grammarURL = new File(grammarfile).toURI().toURL();\n System.out.println(\"Loading grammar from URL: \" + grammarURL);\n Grammar grammar = new Grammar(grammarURL);\n\t\t\n\t\tif (grammar.getName() != null)\n\t\t\tSystem.out.println(\"Grammar '\" + grammar.getName() + \"' loaded.\");\n System.out.println();\n\n // create parser and realizer\n Parser parser = new Parser(grammar);\n Realizer realizer = new Realizer(grammar);\n\n // stuff to remember during loop\n Sign[] lastResults = null;\n LF[] lastLFs = null;\n String lastSentence = \"\";\n int lastReading = 0;\n \n\t\t// prepare to accept input from user \n String[] completions = \n { \":sh\", \":v\", \":reset\", \":feats\", \":nofeats\", \":foff\", \":sem\", \":nosem\", \":all\", \":notall\", \n \":derivs\", \":noderivs\", \":doff\", \":vison\", \":visoff\", \n \":wordpos\", \":nowordpos\", \":eisner\", \":noeisner\",\n \":ptl\", \":noptl\", \":pel\", \":nopel\", \":ppv\", \":noppv\", \":pcpv\", \":nopcpv\", \":plazy\", \":noplazy\", \n \":r\", \":sel\", \":2xml\", \":2tb\", \":2apml\", \":tl\", \":notl\", \":el\", \":noel\",\n \":nbtl\", \":nonbtl\", \":pv\", \":nopv\", \":cpv\", \":nocpv\", \":upon\", \":upoff\", \n \":t\", \":toff\", \":inc\", \":noinc\",\n \":ion\", \":ioff\", \":mion\", \":mioff\", \":con\", \":coff\", \":flon\", \":floff\", \":ccon\", \":ccoff\", \":pon\", \":poff\",\n \":q\", \":h\"};\n\t\tLineReader lineReader = LineReader.createLineReader(completions);\n\t\t\n\t\t// initialize history, per grammar, from prefs\n\t\tString historyKey = HISTORY + \"_\" + grammar.getName();\n\t\tString histStr = prefs.get(historyKey, \"\");\n lineReader.setCommandHistory(histStr);\n\n // welcome msg\n System.out.println(\"Enter strings to parse.\");\n System.out.println(\"Type ':r' to realize selected reading of previous parse.\");\n System.out.println(\"Type ':h' for help on display options and ':q' to quit.\");\n System.out.println(\"You can use the tab key for command completion, \");\n System.out.println(\"Ctrl-P (prev) and Ctrl-N (next) to access the command history, \");\n System.out.println(\"and emacs-style control keys to edit the line.\");\n System.out.println();\n \n while (true) {\n \n\t\t\tString input = lineReader.readLine(\"tccg> \");\n\t\t\tif (input == null) break; // control-D or the like\n input = input.trim();\t\t\t\n\t\t\tif (input.equals(\":show settings\") || input.equals(\":sh\")) {\n showSettings(prefs);\n } else if (input.equals(\":v\")) {\n prefs.putBoolean(SHOW_ALL_RESULTS, true);\n prefs.putBoolean(SHOW_DERIVATIONS, true);\n prefs.putBoolean(SHOW_FEATURES, true);\n prefs.putBoolean(SHOW_SEMANTICS, true);\n prefs.put(FEATURES_TO_SHOW, \"\");\n } else if (input.equals(\":q\")) {\n break; // end of while loop\n } else if (input.equals(\":h\")) {\n showHelp();\n } else if (input.equals(\":reset\")) {\n prefs.putBoolean(SHOW_ALL_RESULTS, false);\n prefs.putBoolean(SHOW_DERIVATIONS, false);\n prefs.putBoolean(SHOW_TIMING, false);\n prefs.putBoolean(SHOW_INCOMPLETE_EDGES, false);\n prefs.putBoolean(Edge.SHOW_COMPLETENESS, false);\n prefs.putBoolean(Edge.SHOW_BITSET, false);\n prefs.putBoolean(SHOW_FEATURES, false);\n prefs.putBoolean(SHOW_SEMANTICS, false);\n prefs.put(FEATURES_TO_SHOW, \"\");\n prefs.putBoolean(VISUALIZE, false); \n prefs.put(\"VISFNAME\", \"\");\n prefs.putBoolean(Converter.USE_WORD_POSITIONS_FOR_ATOM_CONVERSION, true);\n prefs.putBoolean(AbstractCompositionRule.EISNER_CONSTRAINTS, true);\n AbstractCompositionRule.useEisnerConstraints = true; \n prefs.putInt(Parser.PARSE_TIME_LIMIT, Parser.NO_TIME_LIMIT);\n prefs.putInt(Parser.PARSE_EDGE_LIMIT, Parser.NO_EDGE_LIMIT);\n prefs.putInt(Parser.PARSE_PRUNING_VALUE, Parser.NO_PRUNING);\n prefs.putInt(Parser.PARSE_CELL_PRUNING_VALUE, Parser.NO_PRUNING);\n prefs.putBoolean(Parser.PARSE_LAZY_UNPACKING, true);\n prefs.putBoolean(EdgeFactory.USE_INDEXING, true);\n prefs.putBoolean(EdgeFactory.ALLOW_MISSING_INDEX_COMBOS, false);\t\t\n prefs.putBoolean(EdgeFactory.USE_CHUNKS, true);\n prefs.putBoolean(EdgeFactory.USE_FEATURE_LICENSING, true);\n prefs.putBoolean(opennlp.ccg.realize.Chart.USE_COMBOS, true);\n prefs.putBoolean(opennlp.ccg.realize.Chart.USE_PACKING, false);\n prefs.putInt(opennlp.ccg.realize.Chart.TIME_LIMIT, opennlp.ccg.realize.Chart.NO_TIME_LIMIT);\n prefs.putDouble(opennlp.ccg.realize.Chart.NEW_BEST_TIME_LIMIT, opennlp.ccg.realize.Chart.NO_TIME_LIMIT);\n prefs.putInt(opennlp.ccg.realize.Chart.EDGE_LIMIT, opennlp.ccg.realize.Chart.NO_EDGE_LIMIT);\n prefs.putInt(opennlp.ccg.realize.Chart.PRUNING_VALUE, opennlp.ccg.realize.Chart.NO_PRUNING);\n prefs.putInt(opennlp.ccg.realize.Chart.CELL_PRUNING_VALUE, opennlp.ccg.realize.Chart.NO_PRUNING);\n prefs.putBoolean(opennlp.ccg.realize.Chart.DO_UNPACKING, true);\n } else if (input.equals(\":show feats\") || input.equals(\":feats\") || input.equals(\":f\")) {\n prefs.putBoolean(SHOW_FEATURES, true);\n prefs.put(FEATURES_TO_SHOW, \"\");\n Grammar.theGrammar.prefs.showFeats = true;\n Grammar.theGrammar.prefs.featsToShow = \"\";\n } else if (input.startsWith(\":show feats \") || input.startsWith(\":feats \") || input.startsWith(\":f \")) {\n prefs.putBoolean(SHOW_FEATURES, true);\n String s = input.substring(input.indexOf(' ') + 1);\n if (s.startsWith(\"feats \")) { s = s.substring(6); }\n prefs.put(FEATURES_TO_SHOW, s);\n Grammar.theGrammar.prefs.showFeats = true;\n Grammar.theGrammar.prefs.featsToShow = s;\n } else if (input.equals(\":nofeats\") || input.equals(\":foff\")) {\n prefs.putBoolean(SHOW_FEATURES, false);\n prefs.put(FEATURES_TO_SHOW, \"\");\n Grammar.theGrammar.prefs.showFeats = false;\n Grammar.theGrammar.prefs.featsToShow = \"\";\n } else if (input.equals(\":show semantics\") || input.equals(\":sem\") || input.equals(\":s\")) {\n prefs.putBoolean(SHOW_SEMANTICS, true);\n Grammar.theGrammar.prefs.showSem = true;\n } else if (input.equals(\":nosem\") || input.equals(\":soff\")) {\n prefs.putBoolean(SHOW_SEMANTICS, false);\n Grammar.theGrammar.prefs.showSem = false;\n } else if (input.equals(\":show all\") || input.equals(\":all\") || input.equals(\":a\")) {\n prefs.putBoolean(SHOW_ALL_RESULTS, true);\n } else if (input.equals(\":notall\") || input.equals(\":aoff\")) {\n prefs.putBoolean(SHOW_ALL_RESULTS, false);\n } else if (input.equals(\":show derivs\") || input.equals(\":derivs\") || input.equals(\":d\")) {\n prefs.putBoolean(SHOW_DERIVATIONS, true);\n } else if (input.equals(\":noderivs\") || input.equals(\":doff\")) {\n prefs.putBoolean(SHOW_DERIVATIONS, false);\n } else if (input.startsWith(\":time limit\") || input.startsWith(\":tl\")) {\n String[] tokens = input.split(\"\\\\s+\");\n String last = tokens[tokens.length-1];\n try {\n int limit = Integer.parseInt(last);\n prefs.putInt(opennlp.ccg.realize.Chart.TIME_LIMIT, limit);\n } catch (NumberFormatException exc) {\n System.out.println(\"Expecting a time limit in ms, rather than: \" + last);\n }\n } else if (input.startsWith(\":no time limit\") || input.startsWith(\":notl\")) {\n prefs.putInt(opennlp.ccg.realize.Chart.TIME_LIMIT, opennlp.ccg.realize.Chart.NO_TIME_LIMIT);\n } else if (input.startsWith(\":nbtl\")) {\n String[] tokens = input.split(\"\\\\s+\");\n String last = tokens[tokens.length-1];\n try {\n double limit = Double.parseDouble(last);\n prefs.putDouble(opennlp.ccg.realize.Chart.NEW_BEST_TIME_LIMIT, limit);\n } catch (NumberFormatException exc) {\n System.out.println(\"Expecting a time limit in ms, rather than: \" + last);\n }\n } else if (input.startsWith(\":nonbtl\")) {\n prefs.putDouble(opennlp.ccg.realize.Chart.NEW_BEST_TIME_LIMIT, opennlp.ccg.realize.Chart.NO_TIME_LIMIT);\n } else if (input.startsWith(\":edge limit\") || input.startsWith(\":el\")) {\n String[] tokens = input.split(\"\\\\s+\");\n String last = tokens[tokens.length-1];\n try {\n int limit = Integer.parseInt(last);\n prefs.putInt(opennlp.ccg.realize.Chart.EDGE_LIMIT, limit);\n } catch (NumberFormatException exc) {\n System.out.println(\"Expecting an edge limit, rather than: \" + last);\n }\n } else if (input.startsWith(\":no edge limit\") || input.startsWith(\":noel\")) {\n prefs.putInt(opennlp.ccg.realize.Chart.EDGE_LIMIT, opennlp.ccg.realize.Chart.NO_EDGE_LIMIT);\n } else if (input.startsWith(\":pruning value\") || input.startsWith(\":pv\")) {\n String[] tokens = input.split(\"\\\\s+\");\n String last = tokens[tokens.length-1];\n try {\n int val = Integer.parseInt(last);\n prefs.putInt(opennlp.ccg.realize.Chart.PRUNING_VALUE, val);\n } catch (NumberFormatException exc) {\n System.out.println(\"Expecting an integer pruning value, rather than: \" + last);\n }\n } else if (input.startsWith(\":no pruning value\") || input.startsWith(\":nopv\")) {\n prefs.putInt(opennlp.ccg.realize.Chart.PRUNING_VALUE, opennlp.ccg.realize.Chart.NO_PRUNING);\n } else if (input.startsWith(\":cell pruning value\") || input.startsWith(\":cpv\")) {\n String[] tokens = input.split(\"\\\\s+\");\n String last = tokens[tokens.length-1];\n try {\n int val = Integer.parseInt(last);\n prefs.putInt(opennlp.ccg.realize.Chart.CELL_PRUNING_VALUE, val);\n } catch (NumberFormatException exc) {\n System.out.println(\"Expecting an integer cell pruning value, rather than: \" + last);\n }\n } else if (input.startsWith(\":no cell pruning value\") || input.startsWith(\":nocpv\")) {\n prefs.putInt(opennlp.ccg.realize.Chart.CELL_PRUNING_VALUE, opennlp.ccg.realize.Chart.NO_PRUNING);\n } else if (input.equals(\":upon\")) {\n prefs.putBoolean(opennlp.ccg.realize.Chart.DO_UNPACKING, true);\n } else if (input.equals(\":upoff\")) {\n prefs.putBoolean(opennlp.ccg.realize.Chart.DO_UNPACKING, false);\n } else if (input.startsWith(\":select reading\") || input.startsWith(\":sel\")) {\n String[] tokens = input.split(\"\\\\s+\");\n String last = tokens[tokens.length-1];\n try {\n int reading = Integer.parseInt(last);\n if (reading > lastResults.length) {\n System.out.println(\"Only \" + lastResults.length + \" parses found.\");\n } else if (lastLFs[reading-1] == null) {\n System.out.println(\"LF not available, use :all command and reparse.\");\n } else {\n lastReading = reading-1;\n }\n } catch (NumberFormatException exc) {\n System.out.println(\"Expecting a reading number, rather than: \" + last);\n }\n } else if (input.equals(\":timing on\") || input.equals(\":ton\") || input.equals(\":t\")) {\n prefs.putBoolean(SHOW_TIMING, true);\n } else if (input.equals(\":timing off\") || input.equals(\":toff\")) {\n prefs.putBoolean(SHOW_TIMING, false);\n } else if (input.equals(\":show incomplete\") || input.equals(\":inc\")) {\n prefs.putBoolean(SHOW_INCOMPLETE_EDGES, true);\n prefs.putBoolean(Edge.SHOW_COMPLETENESS, true);\n } else if (input.equals(\":noinc\")) {\n prefs.putBoolean(SHOW_INCOMPLETE_EDGES, false);\n prefs.putBoolean(Edge.SHOW_COMPLETENESS, false);\n } else if (input.equals(\":indexing on\") || input.equals(\":ion\")) {\n prefs.putBoolean(EdgeFactory.USE_INDEXING, true);\n } else if (input.equals(\":indexing off\") || input.equals(\":ioff\")) {\n prefs.putBoolean(EdgeFactory.USE_INDEXING, false);\n } else if (input.equals(\":missing index combos on\") || input.equals(\":mion\")) {\n prefs.putBoolean(EdgeFactory.ALLOW_MISSING_INDEX_COMBOS, true);\t\t\n } else if (input.equals(\":missing index combos off\") || input.equals(\":mioff\")) {\n prefs.putBoolean(EdgeFactory.ALLOW_MISSING_INDEX_COMBOS, false);\t\t\n } else if (input.equals(\":chunks on\") || input.equals(\":con\")) {\n prefs.putBoolean(EdgeFactory.USE_CHUNKS, true);\n } else if (input.equals(\":chunks off\") || input.equals(\":coff\")) {\n prefs.putBoolean(EdgeFactory.USE_CHUNKS, false);\n } else if (input.equals(\":feature licensing on\") || input.equals(\":flon\")) {\n prefs.putBoolean(EdgeFactory.USE_FEATURE_LICENSING, true);\n } else if (input.equals(\":feature licensing off\") || input.equals(\":floff\")) {\n prefs.putBoolean(EdgeFactory.USE_FEATURE_LICENSING, false);\n } else if (input.equals(\":combos on\") || input.equals(\":ccon\")) {\n prefs.putBoolean(opennlp.ccg.realize.Chart.USE_COMBOS, true);\n } else if (input.equals(\":combos off\") || input.equals(\":ccoff\")) {\n prefs.putBoolean(opennlp.ccg.realize.Chart.USE_COMBOS, false);\n } else if (input.equals(\":pon\")) {\n prefs.putBoolean(opennlp.ccg.realize.Chart.USE_PACKING, true);\n } else if (input.equals(\":poff\")) {\n prefs.putBoolean(opennlp.ccg.realize.Chart.USE_PACKING, false);\n } else if (input.startsWith(\":realize\") || input.startsWith(\":r\")) {\n LF lf;\n NgramScorer ngramScorer;\n // nb: need to upgrade, consolidate :r FN option with Realize.java ...\n int space = input.indexOf(\" \");\n if (space != -1) { // check for filename\n String filename = readFilename(input.substring(space));\n if (filename == null) {\n System.out.println(\"Expecting a filename to read from.\");\n continue;\n }\n try {\n Document doc = grammar.loadFromXml(filename);\n lf = Realizer.getLfFromDoc(doc);\n // nb: just handling explicit targets for now ...\n List targetElts = doc.getRootElement().getChildren(\"target\");\n String[] targets = new String[targetElts.size()];\n for (int i=0; i < targetElts.size(); i++) {\n Element ex = (Element) targetElts.get(i);\n String target = ex.getText();\n targets[i] = target;\n }\n ngramScorer = new NgramPrecisionModel(targets);\n }\n catch (IOException exc) {\n System.out.println(\"Unable to read: \" + filename);\n System.out.println(exc.toString());\n continue;\n }\n }\n else { // otherwise use last reading of last LF\n if (lastLFs == null || lastLFs[lastReading] == null) {\n System.out.println(\"Nothing to realize!\");\n continue;\n }\n lf = grammar.transformLF(lastLFs[lastReading]);\n String[] targets = new String[1];\n targets[0] = lastSentence;\n ngramScorer = new NgramPrecisionModel(targets);\n }\n realizer.realize(lf, ngramScorer);\n opennlp.ccg.realize.Chart chart = realizer.getChart();\n boolean showIncompleteEdges = prefs.getBoolean(SHOW_INCOMPLETE_EDGES, false);\n boolean showTiming = prefs.getBoolean(SHOW_TIMING, false);\n if (showIncompleteEdges) chart.printEdges();\n else chart.printEdges(true, true);\n if (showTiming) { chart.printTiming(); }\n } else if (input.startsWith(\":2xml\")) {\n if (lastLFs == null || lastLFs[lastReading] == null) {\n System.out.println(\"Nothing to save!\");\n continue;\n }\n String filename = readFilename(input.substring(5));\n if (filename == null) {\n System.out.println(\"Expecting a filename to save to.\");\n continue;\n }\n grammar.saveToXml(lastLFs[lastReading], lastSentence, filename);\n System.out.println(\"Wrote LF to \\\"\" + filename + \"\\\"\");\n } else if (input.startsWith(\":2tb\")) {\n if (lastLFs == null || lastLFs[lastReading] == null) {\n System.out.println(\"Nothing to save!\");\n continue;\n }\n String filename = readFilename(input.substring(4));\n if (filename == null) { filename = \"testbed.xml\"; }\n RegressionInfo.addToTestbed(grammar, lastResults[lastReading], lastResults.length, lastLFs[lastReading], filename);\n System.out.println(\"Added test item to \\\"\" + filename + \"\\\"\");\n } else if (input.startsWith(\":2apml\")) {\n if (lastSentence.length() == 0) {\n System.out.println(\"Nothing to save!\");\n continue;\n }\n String filename = readFilename(input.substring(6));\n if (filename == null) {\n System.out.println(\"Expecting a filename to save to.\");\n continue;\n }\n grammar.saveToApml(lastResults[lastReading], filename);\n System.out.println(\"Wrote \\\"\" + lastSentence + \"\\\" to \\\"\" + filename + \"\\\" as APML\");\n } else if (input.startsWith(\":vison\")) {\n prefs.putBoolean(VISUALIZE, true);\n if ((input.startsWith(\":vison \")) && (input.length( )>= 8)) { \n String fname = input.substring(7);\n if (fname.lastIndexOf('.')!=-1) { \n System.out.println(\"Filename should not contain a suffix. Suffixes .tex and .dvi are assumed.\");\n prefs.put(\"VISFNAME\", \"\");\n } \n else\n prefs.put(\"VISFNAME\", fname);\n } \n else\n prefs.put(\"VISFNAME\", \"\");\n } else if (input.equals(\":visoff\")) {\n prefs.putBoolean(VISUALIZE, false); \n prefs.put(\"VISFNAME\", \"\");\n } else if (input.equals(\":wordpos\")) {\n prefs.putBoolean(Converter.USE_WORD_POSITIONS_FOR_ATOM_CONVERSION, true); \n } else if (input.equals(\":nowordpos\")) {\n prefs.putBoolean(Converter.USE_WORD_POSITIONS_FOR_ATOM_CONVERSION, false); \n } else if (input.equals(\":eisner\")) {\n prefs.putBoolean(AbstractCompositionRule.EISNER_CONSTRAINTS, true);\n AbstractCompositionRule.useEisnerConstraints = true; \n } else if (input.equals(\":noeisner\")) {\n prefs.putBoolean(AbstractCompositionRule.EISNER_CONSTRAINTS, false);\n AbstractCompositionRule.useEisnerConstraints = false; \n } else if (input.startsWith(\":parse time limit\") || input.startsWith(\":ptl\")) {\n String[] tokens = input.split(\"\\\\s+\");\n String last = tokens[tokens.length-1];\n try {\n int limit = Integer.parseInt(last);\n prefs.putInt(Parser.PARSE_TIME_LIMIT, limit);\n } catch (NumberFormatException exc) {\n System.out.println(\"Expecting a time limit in ms, rather than: \" + last);\n }\n } else if (input.startsWith(\":no parse time limit\") || input.startsWith(\":noptl\")) {\n prefs.putInt(Parser.PARSE_TIME_LIMIT, Parser.NO_TIME_LIMIT);\n } else if (input.startsWith(\":parse edge limit\") || input.startsWith(\":pel\")) {\n String[] tokens = input.split(\"\\\\s+\");\n String last = tokens[tokens.length-1];\n try {\n int limit = Integer.parseInt(last);\n prefs.putInt(Parser.PARSE_EDGE_LIMIT, limit);\n } catch (NumberFormatException exc) {\n System.out.println(\"Expecting an edge limit, rather than: \" + last);\n }\n } else if (input.startsWith(\":no parse edge limit\") || input.startsWith(\":nopel\")) {\n prefs.putInt(Parser.PARSE_EDGE_LIMIT, Parser.NO_EDGE_LIMIT);\n } else if (input.startsWith(\":parse pruning value\") || input.startsWith(\":ppv\")) {\n String[] tokens = input.split(\"\\\\s+\");\n String last = tokens[tokens.length-1];\n try {\n int val = Integer.parseInt(last);\n prefs.putInt(Parser.PARSE_PRUNING_VALUE, val);\n } catch (NumberFormatException exc) {\n System.out.println(\"Expecting an integer pruning value, rather than: \" + last);\n }\n } else if (input.startsWith(\":no parse pruning value\") || input.startsWith(\":noppv\")) {\n prefs.putInt(Parser.PARSE_PRUNING_VALUE, Parser.NO_PRUNING);\n } else if (input.startsWith(\":parse cell pruning value\") || input.startsWith(\":pcpv\")) {\n String[] tokens = input.split(\"\\\\s+\");\n String last = tokens[tokens.length-1];\n try {\n int val = Integer.parseInt(last);\n prefs.putInt(Parser.PARSE_CELL_PRUNING_VALUE, val);\n } catch (NumberFormatException exc) {\n System.out.println(\"Expecting an integer cell pruning value, rather than: \" + last);\n }\n } else if (input.startsWith(\":no parse cell pruning value\") || input.startsWith(\":nopcpv\")) {\n prefs.putInt(Parser.PARSE_CELL_PRUNING_VALUE, Parser.NO_PRUNING);\n } else if (input.equals(\":plazy\")) {\n prefs.putBoolean(Parser.PARSE_LAZY_UNPACKING, true); \n } else if (input.equals(\":noplazy\")) {\n prefs.putBoolean(Parser.PARSE_LAZY_UNPACKING, false); \n } else { \n try {\n if (input.length() == 0) {\n if (lastSentence.length() > 0) { \n input = lastSentence; \n } else {\n System.out.println(\"Nothing to parse!\");\n continue;\n }\n }\n parser.parse(input);\n List parses = parser.getResult();\n Sign[] results = new Sign[parses.size()];\n parses.toArray(results);\n int resLength = results.length;\n switch (resLength) {\n case 0: break;\n case 1: \n System.out.println(resLength + \" parse found.\\n\"); \n break;\n default: System.out.println(resLength + \" parses found.\\n\"); \n }\n \n lastResults = results;\n lastLFs = new LF[resLength];\n if (input.length() > 0) { lastSentence = input; }\n lastReading = 0;\n \n boolean showall = prefs.getBoolean(SHOW_ALL_RESULTS, false);\n boolean showderivs = prefs.getBoolean(SHOW_DERIVATIONS, false);\n boolean showsem = prefs.getBoolean(SHOW_SEMANTICS, true);\n boolean visualize = prefs.getBoolean(VISUALIZE, false); \n boolean showfeats = prefs.getBoolean(SHOW_FEATURES, false);\n String feats_to_show = prefs.get(FEATURES_TO_SHOW, \"\");\n Visualizer vis = null;\n String baseFileName = null;\n\t\t\t\t grammar.prefs.showSem = showsem;\n\t\t\t\t grammar.prefs.showFeats = showfeats;\n\t\t\t\t grammar.prefs.featsToShow = feats_to_show;\n if (visualize) { \n vis = new Visualizer(); \n if (prefs.get(\"VISFNAME\", \"\").equals(\"\"))\n baseFileName = vis.getTempFileName();\n else\n baseFileName = prefs.get(\"VISFNAME\", \"\");\n vis.writeHeader(baseFileName+\".tex\"); \n }\n int numToShow = (showall) ? resLength : 1;\n for (int i=0; i < numToShow; i++) {\n Category cat = results[i].getCategory();\n LF convertedLF = null;\n if (cat.getLF() != null) {\n cat = cat.copy();\n Nominal index = cat.getIndexNominal(); \n Sign rootSign = results[i]; // could add a switch here for naming convention\n convertedLF = HyloHelper.compactAndConvertNominals(cat.getLF(), index, rootSign);\n lastLFs[i] = convertedLF; \n cat.setLF(null);\n }\n String parseNum = (resLength == 1) ? \"Parse: \" : (\"Parse \"+(i+1)+\": \"); \n System.out.print(parseNum + cat.toString());\n if (showsem && convertedLF != null) {\n System.out.println(\" : \");\n System.out.println(\" \" + convertedLF.prettyPrint(\" \"));\n }\n else System.out.println();\n if (showderivs) {\n System.out.println(\"------------------------------\");\n System.out.println(results[i].getDerivationHistory());\n }\n if (visualize)\n vis.saveTeXFile(results[i], baseFileName + \".tex\" );\n }\n if (visualize) {\n vis.writeFooter(baseFileName + \".tex\"); \n vis.show(baseFileName); \n if (prefs.get(\"VISFNAME\",\"\").equals(\"\")) // If temporary file,\n vis.cleanFiles(baseFileName); // clean it\n else {\n vis.cleanAuxFiles(baseFileName);\n System.out.println(\"Saved to files \" + baseFileName + \".tex and \" + baseFileName + \".dvi\");\n }\n vis = null; \n }\n } catch(ParseException pe) {\n System.out.println(pe);\n }\n }\n }\n\t\t\n\t\t// store command input history in preferences\n\t\tprefs.put(historyKey, lineReader.getCommandHistory());\n\t\t\n // done\n\t\tSystem.out.println(\"Exiting tccg.\");\n\t\tSystem.exit(0);\n }\n\n\n // reads the next token in the string as a filename\n private static String readFilename(String s) throws IOException {\n StreamTokenizer st = new StreamTokenizer(new StringReader(s));\n st.wordChars('/','/'); st.wordChars('\\\\','\\\\'); st.wordChars(':',':'); \n st.nextToken();\n return st.sval;\n }\n \n \n /** Shows help for the command-line tool. */\n public static void showHelp() {\n System.out.println();\n System.out.println(\"Commands for tccg (otherwise input is parsed):\");\n System.out.println();\n System.out.println(\" :sh\\t\\t\\tshow current preference settings\");\n System.out.println(\" :v\\t\\t\\tverbose output\");\n System.out.println(\" :reset\\t\\treset options to defaults\");\n System.out.println(\" :feats (L)\\t\\tshow features (or just show features in list L)\");\n System.out.println(\" :nofeats\\t\\tdon't show features\");\n System.out.println(\" :sem\\t\\t\\tshow semantics\");\n System.out.println(\" :nosem\\t\\tdon't show semantics\");\n System.out.println(\" :all\\t\\t\\tshow all parse results\");\n System.out.println(\" :notall\\t\\tdon't show all parse results\");\n System.out.println(\" :derivs\\t\\tshow derivations\");\n System.out.println(\" :noderivs\\t\\tdon't show derivations\");\n System.out.println(\" :vison (FN)\\t\\tturn visualization on (saving to file with name FN)\");\n System.out.println(\" :visoff\\t\\tturn visualization off\");\n System.out.println(\" :wordpos\\t\\tuse word positions to name converted nominals\");\n System.out.println(\" :nowordpos\\t\\tdon't use word positions to name converted nominals\");\n System.out.println(\" :eisner\\t\\tuse Eisner constraints on composition\");\n System.out.println(\" :noeisner\\t\\tturn off Eisner constraints on composition\");\n System.out.println();\n System.out.println(\" :ptl N\\t\\tset parse time limit to N ms\");\n System.out.println(\" :noptl\\t\\tset parse time limit to none\");\n System.out.println(\" :pel N\\t\\tset parse edge limit to N\");\n System.out.println(\" :nopel\\t\\tset parse edge limit to none\");\n System.out.println(\" :ppv N\\t\\tset parse pruning value to N\");\n System.out.println(\" :noppv\\t\\tset parse pruning value to none\");\n System.out.println(\" :pcpv N\\t\\tset parse cell pruning value to N\");\n System.out.println(\" :nopcpv\\t\\tset parse cell pruning value to none\");\n System.out.println(\" :plazy\\t\\tturn lazy unpacking on in parser\");\n System.out.println(\" :noplazy\\t\\tturn lazy unpacking off in parser\");\n System.out.println();\n System.out.println(\" :r (FN)\\t\\trealize selected reading (or from XML file with name FN)\");\n System.out.println(\" :sel N\\t\\tselect reading N for realization or saving\");\n System.out.println(\" :2xml FN\\t\\tsave last input and LF to XML file with name FN\");\n System.out.println(\" :2tb (FN)\\t\\tadd last input and LF as a test item (to file with name FN)\");\n System.out.println(\" :2apml FN\\t\\tsave last input to APML file with name FN\");\n System.out.println();\n System.out.println(\" :tl N\\t\\t\\tset realization time limit to N ms\");\n System.out.println(\" :notl\\t\\t\\tset realization time limit to none\");\n System.out.println(\" :nbtl N\\t\\tset realization new best time limit to N ms | N < 1 of first\");\n System.out.println(\" :nonbtl\\t\\tset realization new best time limit to none\");\n System.out.println(\" :el N\\t\\t\\tset realization edge limit to N\");\n System.out.println(\" :noel\\t\\t\\tset realization edge limit to none\");\n System.out.println(\" :pv N\\t\\t\\tset realization pruning value to N\");\n System.out.println(\" :nopv\\t\\t\\tset realization pruning value to none\");\n System.out.println(\" :cpv N\\t\\tset realization cell pruning value to N\");\n System.out.println(\" :nocpv\\t\\tset realization cell pruning value to none\");\n System.out.println(\" :upon\\t\\t\\tturn unpacking on\");\n System.out.println(\" :upoff\\t\\tturn unpacking off\");\n System.out.println(\" :t\\t\\t\\tturn realization timing on\");\n System.out.println(\" :toff\\t\\t\\tturn realization timing off\");\n System.out.println(\" :inc\\t\\t\\tshow incomplete realization edges\");\n System.out.println(\" :noinc\\t\\tdon't show incomplete realization edges\");\n System.out.println();\n System.out.println(\" :ion\\t\\t\\tturn index filtering on\");\n System.out.println(\" :ioff\\t\\t\\tturn index filtering off\");\n System.out.println(\" :mion\\t\\t\\tturn missing index combos on\");\n System.out.println(\" :mioff\\t\\tturn missing index combos off\");\n System.out.println(\" :con\\t\\t\\tturn LF chunks on\");\n System.out.println(\" :coff\\t\\t\\tturn LF chunks off\");\n System.out.println(\" :flon\\t\\t\\tturn feature licensing on\");\n System.out.println(\" :floff\\t\\tturn feature licensing off\");\n System.out.println(\" :ccon\\t\\t\\tturn collected combos on\");\n System.out.println(\" :ccoff\\t\\tturn collected combos off\");\n System.out.println(\" :pon\\t\\t\\tturn packing on\");\n System.out.println(\" :poff\\t\\t\\tturn packing off\");\n \n System.out.println();\n System.out.println(\" :q\\t\\t\\tquit tccg\");\n System.out.println(\" :h\\t\\t\\tshow this message\");\n System.out.println();\n }\n\n /** Shows current settings. */\n public static void showSettings(Preferences prefs) {\n System.out.println();\n System.out.println(\"Current preference settings:\"); \n System.out.println();\n boolean showfeats = prefs.getBoolean(SHOW_FEATURES, false);\n boolean showsem = prefs.getBoolean(SHOW_SEMANTICS, true);\n String feats = prefs.get(FEATURES_TO_SHOW, \"\");\n System.out.println(\" show feats:\\t\\t\" + showfeats); \n System.out.println(\" show semantics:\\t\" + showsem);\n if (showfeats) {\n System.out.println(\" feats to show:\\t\" + ((feats.length() > 0) ? feats : \"all\")); \n }\n boolean showall = prefs.getBoolean(SHOW_ALL_RESULTS, false);\n boolean showderivs = prefs.getBoolean(SHOW_DERIVATIONS, false);\n System.out.println(\" show all:\\t\\t\" + showall); \n System.out.println(\" show derivs:\\t\\t\" + showderivs);\n boolean visualize = prefs.getBoolean(VISUALIZE, false); \n String visfname = prefs.get(\"VISFNAME\", \"\");\n System.out.println(\" visualize:\\t\\t\" + ((visualize) ? \"on\" : \"off\"));\n if (visfname.length() > 0) {\n System.out.println(\" vis file name:\\t\" + visfname);\n }\n boolean wordpos = prefs.getBoolean(Converter.USE_WORD_POSITIONS_FOR_ATOM_CONVERSION, true); \n System.out.println(\" word pos:\\t\\t\" + ((wordpos) ? \"on\" : \"off\"));\n boolean eisner = prefs.getBoolean(AbstractCompositionRule.EISNER_CONSTRAINTS, true); \n System.out.println(\" Eisner constraints:\\t\" + ((eisner) ? \"on\" : \"off\"));\n System.out.println();\n int ptl = prefs.getInt(Parser.PARSE_TIME_LIMIT, Parser.NO_TIME_LIMIT);\n System.out.println(\" parse time limit:\\t\" + ((ptl == Parser.NO_TIME_LIMIT) ? \"none\" : \"\" + ptl + \" ms\"));\n int pel = prefs.getInt(Parser.PARSE_EDGE_LIMIT, Parser.NO_EDGE_LIMIT);\n System.out.println(\" parse edge limit:\\t\" + ((pel == Parser.NO_EDGE_LIMIT) ? \"none\" : \"\" + pel));\n int ppv = prefs.getInt(Parser.PARSE_PRUNING_VALUE, Parser.NO_PRUNING);\n System.out.println(\" parse pruning value:\\t\" + ((ppv == Parser.NO_PRUNING) ? \"none\" : \"\" + ppv));\n int pcpv = prefs.getInt(Parser.PARSE_CELL_PRUNING_VALUE, Parser.NO_PRUNING);\n System.out.println(\" parse cell prune val:\\t\" + ((pcpv == Parser.NO_PRUNING) ? \"none\" : \"\" + pcpv));\n boolean plazy = prefs.getBoolean(Parser.PARSE_LAZY_UNPACKING, true);\n System.out.println(\" lazy unpacking:\\t\" + ((plazy) ? \"on\" : \"off\")); \n System.out.println();\n int tl = prefs.getInt(opennlp.ccg.realize.Chart.TIME_LIMIT, opennlp.ccg.realize.Chart.NO_TIME_LIMIT);\n System.out.println(\" time limit:\\t\\t\" + ((tl == opennlp.ccg.realize.Chart.NO_TIME_LIMIT) ? \"none\" : \"\" + tl + \" ms\"));\n double nbtl = prefs.getDouble(opennlp.ccg.realize.Chart.NEW_BEST_TIME_LIMIT, opennlp.ccg.realize.Chart.NO_TIME_LIMIT);\n String nbtlStr = (nbtl >= 1) ? (((int)nbtl) + \" ms\") : (nbtl + \" of first\");\n System.out.println(\" new best time limit:\\t\" + ((nbtl == opennlp.ccg.realize.Chart.NO_TIME_LIMIT) ? \"none\" : nbtlStr));\n int el = prefs.getInt(opennlp.ccg.realize.Chart.EDGE_LIMIT, opennlp.ccg.realize.Chart.NO_EDGE_LIMIT);\n System.out.println(\" edge limit:\\t\\t\" + ((el == opennlp.ccg.realize.Chart.NO_EDGE_LIMIT) ? \"none\" : \"\" + el));\n int pv = prefs.getInt(opennlp.ccg.realize.Chart.PRUNING_VALUE, opennlp.ccg.realize.Chart.NO_PRUNING);\n System.out.println(\" pruning value:\\t\" + ((pv == opennlp.ccg.realize.Chart.NO_PRUNING) ? \"none\" : \"\" + pv));\n int cpv = prefs.getInt(opennlp.ccg.realize.Chart.CELL_PRUNING_VALUE, opennlp.ccg.realize.Chart.NO_PRUNING);\n System.out.println(\" cell pruning value:\\t\" + ((cpv == opennlp.ccg.realize.Chart.NO_PRUNING) ? \"none\" : \"\" + cpv));\n boolean unpacking = prefs.getBoolean(opennlp.ccg.realize.Chart.DO_UNPACKING, true);\n System.out.println(\" unpacking:\\t\\t\" + ((unpacking) ? \"on\" : \"off\")); \n boolean showtiming = prefs.getBoolean(SHOW_TIMING, false);\n System.out.println(\" timing:\\t\\t\" + ((showtiming) ? \"on\" : \"off\")); \n boolean showinc = prefs.getBoolean(SHOW_INCOMPLETE_EDGES, false);\n System.out.println(\" show incomplete:\\t\" + ((showinc) ? \"on\" : \"off\")); \n System.out.println();\n boolean indexing = prefs.getBoolean(EdgeFactory.USE_INDEXING, true);\n\tboolean missingIndexCombos = prefs.getBoolean(EdgeFactory.ALLOW_MISSING_INDEX_COMBOS, false);\t\t\n boolean chunks = prefs.getBoolean(EdgeFactory.USE_CHUNKS, true);\n boolean licensing = prefs.getBoolean(EdgeFactory.USE_FEATURE_LICENSING, true);\n boolean combos = prefs.getBoolean(opennlp.ccg.realize.Chart.USE_COMBOS, true);\n boolean packing = prefs.getBoolean(opennlp.ccg.realize.Chart.USE_PACKING, false);\n System.out.println(\" index filtering:\\t\" + ((indexing) ? \"on\" : \"off\"));\n System.out.println(\" missing index combos:\\t\" + ((missingIndexCombos) ? \"on\" : \"off\"));\n System.out.println(\" chunks:\\t\\t\" + ((chunks) ? \"on\" : \"off\"));\n System.out.println(\" licensing:\\t\\t\" + ((licensing) ? \"on\" : \"off\"));\n System.out.println(\" combos:\\t\\t\" + ((combos) ? \"on\" : \"off\"));\n System.out.println(\" packing:\\t\\t\" + ((packing) ? \"on\" : \"off\"));\n System.out.println();\n }\n}\n" }, { "path": "src/opennlp/ccg/WebCCG.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2006 Ben Wing.\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg;\r\n\r\nimport opennlp.ccg.lexicon.*;\r\nimport opennlp.ccg.grammar.*;\r\nimport opennlp.ccg.parse.*;\r\nimport opennlp.ccg.util.*;\r\nimport opennlp.ccg.synsem.*;\r\n//import opennlp.ccg.realize.*;\r\nimport opennlp.ccg.hylo.*;\r\n//import opennlp.ccg.ngrams.*;\r\n//import opennlp.ccg.test.*;\r\n\r\n//import org.jdom.*;\r\n\r\nimport java.io.*;\r\nimport java.net.*;\r\nimport java.util.*;\r\n\r\n/**\r\n * An interface for use with a higher-level web interface. This should\r\n * provide as simple an interface onto parsing as possible, with its output\r\n * in a format that can be easily handled by a CGI program or similar.\r\n * Called as\r\n *\r\n * webccg [-showall] [-showderivs] [-showsem] [-visualize FILE] GRAMMARDIR\r\n\r\n-showall shows all parses rather than just the first one.\r\n-showderivs shows the derivation history of each parse.\r\n-showsem shows the logical form of each parse.\r\n-visualize output a visualization of the parses into FILE (in PNG format).\r\n *\r\n * @author Ben Wing\r\n * @version $Revision: 1.4 $, $Date: 2009/12/21 03:27:18 $\r\n */\r\npublic class WebCCG {\r\n /** Main method for tccg. */\r\n public static void main(String[] args) throws IOException, LexException { \r\n\tString usage = \"java opennlp.ccg.WebCCG \" + \r\n\"[-showall] [-showderivs] [-showsem] [-showfeats] [-visualize FILE] GRAMMARDIR\\n\" +\r\n\"\\n\" +\r\n\"-showall shows all parses rather than just the first one.\\n\" +\r\n\"-showderivs shows the derivation history of each parse.\\n\" +\r\n\"-showsem shows the logical form of each parse.\\n\" +\r\n\"-showfeats shows the features associated with each nonterminal.\\n\" +\r\n\"-visualize output a visualization of the parses into FILE (in TEX format).\\n\"+\r\n\" to convert to an image, try this:\\n\" +\r\n\" latex foo.tex; dvips foo.dvi | pstopnm | pnmtopng > foo.png\\n\" +\r\n\"\\n\" +\r\n\"Sentences to parse are read from standard input.\\n\";\r\n\r\n\tif (args.length > 0 && (args[0].equals(\"-h\") || args[0].equals(\"-help\"))) {\r\n\t System.out.println(\"Usage: \" + usage);\r\n\t System.exit(0);\r\n\t}\r\n\r\n\t// args \r\n\t//String prefsfile = null;\r\n\tboolean showall = false;\r\n\tboolean showderivs = false;\r\n\tboolean showsem = false;\r\n\tboolean showfeats = false;\r\n\tString visfile = null;\r\n\tint i;\r\n\tfor (i = 0; i < args.length; i++) {\r\n\t if (args[i].equals(\"-showall\"))\r\n\t\tshowall = true;\r\n\t else if (args[i].equals(\"-showderivs\"))\r\n\t\tshowderivs = true;\r\n\t else if (args[i].equals(\"-showsem\"))\r\n\t\tshowsem = true;\r\n\t else if (args[i].equals(\"-showfeats\"))\r\n\t\tshowfeats = true;\r\n\t else if (args[i].equals(\"-visualize\"))\r\n\t\tvisfile = args[++i];\r\n\t else\r\n\t\tbreak;\r\n\t}\r\n\tif (i != args.length - 1) {\r\n\t System.out.println(\"Usage: \" + usage);\r\n\t System.exit(0);\r\n\t}\r\n\tString grammarfile = args[i] + \"/grammar.xml\";\r\n\r\n\t// load grammar\r\n\tURL grammarURL = new File(grammarfile).toURI().toURL();\r\n\t//System.out.println(\"Loading grammar from URL: \" + grammarURL);\r\n\tGrammar grammar = new Grammar(grammarURL);\r\n\r\n\t//if (grammar.getName() != null)\r\n\t// System.out.println(\"Grammar '\" + grammar.getName() + \"' loaded.\");\r\n\r\n\t// create parser and realizer\r\n\tParser parser = new Parser(grammar);\r\n\t//Realizer realizer = new Realizer(grammar);\r\n\r\n\tBufferedReader br = new BufferedReader(new InputStreamReader(System.in));\r\n\twhile (true) {\r\n\t String input = br.readLine();\r\n\t if (input == null) break; // EOF\r\n\t input = input.trim();\r\n\t if (input.equals(\"\"))\r\n\t\tcontinue;\r\n\t try {\r\n\t\tparser.parse(input);\r\n\t\tList parses = parser.getResult();\r\n\t\tSign[] results = new Sign[parses.size()];\r\n\t\tparses.toArray(results);\r\n\t\tint resLength = results.length;\r\n\t\tSystem.out.print(\"\\\"\" + input + \"\\\": \");\r\n\t\tswitch (resLength) {\r\n\t\tcase 0: break;\r\n\t\tcase 1: \r\n\t\t System.out.println(resLength + \" parse found.\\n\"); \r\n\t\t break;\r\n\t\tdefault: System.out.println(resLength + \" parses found.\\n\"); \r\n\t\t}\r\n \r\n\t\tVisualizer vis = null; \r\n\t\tgrammar.prefs.showSem = showsem;\r\n\t\tgrammar.prefs.showFeats = showfeats;\r\n\t\tgrammar.prefs.featsToShow = \"\";\r\n\t\tif (visfile != null) { \r\n\t\t vis = new Visualizer(); \r\n\t\t vis.writeHeader(visfile);\r\n\t\t}\r\n\t\tint numToShow = (showall) ? resLength : 1;\r\n\t\tfor (i=0; i < numToShow; i++) {\r\n\t\t Category cat = results[i].getCategory();\r\n\t\t LF convertedLF = null;\r\n\t\t if (cat.getLF() != null) {\r\n\t\t\tcat = cat.copy();\r\n\t\t\tNominal index = cat.getIndexNominal(); \r\n\t\t\tconvertedLF = HyloHelper.compactAndConvertNominals(cat.getLF(), index, results[i]);\r\n\t\t\tcat.setLF(null);\r\n\t\t }\r\n\t\t String parseNum = (resLength == 1) ? \"Parse: \" : (\"Parse \"+(i+1)+\": \"); \r\n\t\t System.out.print(parseNum + cat.toString());\r\n\t\t if (showsem && convertedLF != null) {\r\n\t\t\tSystem.out.println(\" : \");\r\n\t\t\tSystem.out.println(\" \" + convertedLF.prettyPrint(\" \"));\r\n\t\t }\r\n\t\t else System.out.println();\r\n\t\t if (showderivs) {\r\n\t\t\tSystem.out.println(\"------------------------------\");\r\n\t\t\tSystem.out.println(results[i].getDerivationHistory());\r\n\t\t }\r\n\t\t if (visfile != null)\r\n\t\t\tvis.saveTeXFile(results[i], visfile);\r\n\t\t}\r\n\t\tif (visfile != null) {\r\n\t\t vis.writeFooter(visfile);\r\n\t\t}\r\n\t } catch(ParseException pe) {\r\n\t\tSystem.out.print(\"\\\"\" + input + \"\\\": \");\r\n\t\tSystem.out.println(pe + \".\\n\");\r\n\t }\r\n\t}\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/alignment/AbstractEncodingScheme.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\nimport java.util.Arrays;\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Set;\n\n/**\n * Abstract class for implementing encoding schemes. This base class provides a constructor taking all the\n * necessary arguments for implementing {@link EncodingScheme}, and gives getter methods for all of them,\n * as required.\n * @author Scott Martin\n * @see EncodingScheme\n */\npublic abstract class AbstractEncodingScheme implements EncodingScheme {\n\t\n\t/**\n\t * Delimits a field within a mapping.\n\t */\n\tprotected Character fieldDelimiter;\n\t\n\t/**\n\t * Delimits a mapping.\n\t */\n\tprotected Character mappingDelimiter;\n\t\n\t/**\n\t * Delimits a group of mappings.\n\t */\n\tprotected Character groupDelimiter;\n\t\n\t/**\n\t * The index base for IDs.\n\t */\n\tprotected IndexBase phraseNumberBase;\n\t\n\t/**\n\t * The index base for indices.\n\t */\n\tprotected IndexBase indexBase;\n\t\n\t/**\n\t * The order of the fields in mappings corresponding to this encoding scheme.\n\t */\n\tprotected List order;\n\t\n\t/**\n\t * The set of required fields in this encoding scheme.\n\t */\n\tprotected Set required;\n\t\n\t/**\n\t * The fields that this encoding scheme uses by default.\n\t */\n\tprotected Set defaults;\n\t\n\tfinal boolean fieldDelimSep, mappingDelimSep, groupDelimSep;\t\n\t\n\tprotected AbstractEncodingScheme(Character fieldDelimiter, Character mappingDelimiter,\n\t\t\tCharacter groupDelimiter, IndexBase phraseNumberBase, IndexBase indexBase,\n\t\t\tSet defaults, Set required,\n\t\t\tMappingFormat.Field... order) {\n\t\tthis.fieldDelimiter = fieldDelimiter;\n\t\tthis.mappingDelimiter = mappingDelimiter;\n\t\tthis.groupDelimiter = groupDelimiter;\n\t\tthis.phraseNumberBase = phraseNumberBase;\n\t\tthis.indexBase = indexBase;\n\t\t\n\t\tfieldDelimSep = isLineSeparator(fieldDelimiter);\n\t\tmappingDelimSep = isLineSeparator(mappingDelimiter);\n\t\tgroupDelimSep = isLineSeparator(groupDelimiter);\n\t\t\n\t\tthis.defaults = Collections.unmodifiableSet(defaults);\n\t\tthis.required = Collections.unmodifiableSet(required);\n\t\tthis.order = Collections.unmodifiableList(Arrays.asList(order));\n\t}\n\t\n\tstatic boolean isLineSeparator(Character c) {\n\t\t// TODO why doesn't Character.getType(c) == Character.LINE_SEPARATOR work?\n\t\treturn c == '\\r' || c == '\\n';\n\t}\n\t\n\t/**\n\t * Tests whether the supplied character counts as a field delimiter according to this encoding scheme.\n\t * @return true if c is equal to {@link #fieldDelimiter} or both c and \n\t * {@link #fieldDelimiter} are line separators.\n\t */\n\tpublic boolean isFieldDelimiter(Character c) {\n\t\treturn fieldDelimiter.equals(c) || (fieldDelimSep && isLineSeparator(c)); \n\t}\n\t\n\t/**\n\t * Tests whether the supplied character counts as a mapping delimiter according to this encoding scheme.\n\t * @return true if c is equal to {@link #mappingDelimiter} or both c and \n\t * {@link #mappingDelimiter} are line separators.\n\t */\n\tpublic boolean isMappingDelimiter(Character c) {\n\t\treturn mappingDelimiter.equals(c) || (mappingDelimSep && isLineSeparator(c)); \n\t}\n\t\n\t/**\n\t * Tests whether the supplied character counts as a group delimiter according to this encoding scheme.\n\t * @return true if c is equal to {@link #groupDelimiter} or both c and \n\t * {@link #groupDelimiter} are line separators.\n\t */\n\tpublic boolean isGroupDelimiter(Character c) {\n\t\treturn groupDelimiter.equals(c) || (groupDelimSep && isLineSeparator(c)); \n\t}\n\n\t/**\n\t * Gets the delimiter for fields.\n\t */\n\tpublic Character getFieldDelimiter() {\n\t\treturn fieldDelimiter;\n\t}\n\n\t/**\n\t * Gets the delimiter for mappings.\n\t */\n\tpublic Character getMappingDelimiter() {\n\t\treturn mappingDelimiter;\n\t}\n\n\t/**\n\t * Gets the delimiter for groups.\n\t */\n\tpublic Character getGroupDelimiter() {\n\t\treturn groupDelimiter;\n\t}\n\n\t/**\n\t * Gets the numbering base used for phrases.\n\t */\n\tpublic IndexBase getPhraseNumberBase() {\n\t\treturn phraseNumberBase;\n\t}\n\n\t/**\n\t * Gets the numbering base used for mapping indices.\n\t */\n\tpublic IndexBase getIndexBase() {\n\t\treturn indexBase;\n\t}\n\n\t/**\n\t * Gets the order in which fields occur in this encoding scheme.\n\t */\n\tpublic List getOrder() {\n\t\treturn order;\n\t}\n\n\t/**\n\t * Gets the required (non-optional) fields in this scheme.\n\t */\n\tpublic Set getRequired() {\n\t\treturn required;\n\t}\n\n\t/**\n\t * @return Gets the default fields used by this scheme.\n\t */\n\tpublic Set getDefaults() {\n\t\treturn defaults;\n\t}\n\n\t/**\n\t * Gets a hash code for this encoding scheme based on its delimiters, index bases, and fields.\n\t */\n\t@Override\n\tpublic int hashCode() {\n\t\treturn 37 * fieldDelimiter.hashCode() + groupDelimiter.hashCode() + mappingDelimiter.hashCode()\n\t\t\t\t+ indexBase.hashCode() + phraseNumberBase.hashCode() + order.hashCode()\n\t\t\t\t+ required.hashCode() + defaults.hashCode();\n\t}\n\n\t/**\n\t * Tests whether this encoding scheme is equal to another based on its delimiters, index bases, and fields.\n\t */\n\t@Override\n\tpublic boolean equals(Object obj) {\n\t\tif(obj instanceof AbstractEncodingScheme) {\n\t\t\tAbstractEncodingScheme e = (AbstractEncodingScheme)obj;\n\t\t\t\n\t\t\treturn fieldDelimiter.equals(e.fieldDelimiter) && groupDelimiter.equals(e.groupDelimiter)\n\t\t\t\t\t&& mappingDelimiter.equals(e.mappingDelimiter) && indexBase.equals(e.indexBase)\n\t\t\t\t\t&& phraseNumberBase.equals(e.phraseNumberBase) && order.equals(e.order)\n\t\t\t\t\t&& required.equals(e.required) && defaults.equals(e.defaults);\n\t\t}\n\t\t\n\t\treturn false;\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/alignment/Alignment.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\nimport static opennlp.ccg.alignment.PhrasePosition.A;\nimport static opennlp.ccg.alignment.PhrasePosition.B;\n\nimport java.util.AbstractMap;\nimport java.util.AbstractSet;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.Iterator;\nimport java.util.LinkedHashMap;\nimport java.util.LinkedHashSet;\nimport java.util.Map;\nimport java.util.Set;\n\nimport opennlp.ccg.util.DelegatedFilter;\nimport opennlp.ccg.util.Filter;\nimport opennlp.ccg.util.FilteredSet;\nimport opennlp.ccg.util.VisitedFilter;\n\n/**\n * An alignment consisting of a pair of phrases and a set of mappings between them.\n *

\n * This class is a flat representation of the mappings between indices in its \n * {@linkplain #getA() A-position} phrase and its {@linkplain #getB() B-position} phrase in that it is\n * simply a set of mappings. More granularity is available by calling\n * {@link #getTargets(Integer, PhrasePosition)}, which returns all the indices a certain index is mapped to\n * from a specified position.\n * Alignments also allow their indices to be accessed when the phrase position is not necessarily known, \n * via {@link #get(PhrasePosition)}, {@link #getIndices(PhrasePosition)}, and {@link #asMap(PhrasePosition)}.\n *

\n * A detached view of this alignment as a map whose keys are the indices in a\n * specified position and whose values are the sets of indices that index maps to can be obtained by calling\n * {@link #asMap(PhrasePosition)}. If only the set of indices mapped to by a certain index is required,\n * {@link #getTargets(Integer, PhrasePosition)} provides similar functionality. The static method\n * {@link #fromMap(Phrase, Phrase, Map)} allows an alignment to be reconstructed from a map of indices to\n * sets of indices. \n *

\n * A version of this alignment with the phrase positions reversed and all the mappings\n * {@linkplain Mapping#reverse() reversed} can be obtained by calling {@link #reverse()}.\n * \n * @author Scott Martin\n * @see PhrasePosition\n * @see Phrase\n * @see Mapping\n */\npublic class Alignment extends AbstractSet implements Comparable {\n\n\tfinal Phrase a, b;\n\tfinal Set mappings;\n\t\n\t/**\n\t * Creates a new alignment with the specified phrases and mappings between them. The specified set of\n\t * mappings is copied in a way that preserves whatever ordering is present in the original set,\n\t * via {@link LinkedHashSet}.\n\t * \n\t * @param a The phrase to use for {@linkplain PhrasePosition#A the "A" position}.\n\t * @param b The phrase to use for {@linkplain PhrasePosition#B the "B" position}.\n\t * @param mappings The mappings between a and b, where the\n\t * {@linkplain Mapping#getA() first index} is understood to belong to a and the\n\t * {@linkplain Mapping#getB() second index} is understood to belong to b.\n\t * \n\t * @throws IllegalArgumentException If either phrase is null, or if phrases a and b do not\n\t * have matching {@linkplain Phrase#getNumber() numbers}, if mappings is null,\n\t * or if any of the mappings have a non-null phrase number that is not equal to the phrases' numbers. \n\t * @throws IndexOutOfBoundsException If any of the mappings contains an index that does not\n\t * exist in the phrase in the corresponding position.\n\t * \n\t * @see LinkedHashSet\n\t */\n\tpublic Alignment(Phrase a, Phrase b, Collection mappings) {\n\t\tcheckPhrases(a, b);\n\t\t\n\t\tif(mappings == null) {\n\t\t\tthrow new IllegalArgumentException(\"mappings is null\");\n\t\t}\n\t\t\n\t\t// have to set these first or checkMapping() throws exception\n\t\tthis.a = a;\n\t\tthis.b = b;\n\t\t\n\t\tfor(Mapping m : mappings) {\n\t\t\tcheckMapping(m);\n\t\t}\n\t\t\n\t\t\n\t\tthis.mappings = new LinkedHashSet(mappings);\n\t}\n\t\n\t/**\n\t * Creates a new alignment based on the specified phrases and map view of their mappings.\n\t * @param a The {@linkplain PhrasePosition#A A-position} phrase.\n\t * @param b The {@linkplain PhrasePosition#B B-position} phrase.\n\t * @param map A map whose keys are the A-position indices and whose values are the B-position indices \n\t * that the corresponding key is mapped to.\n\t * @return A new alignment with mappings created based on the specified map.\n\t * \n\t * @see #asMap()\n\t */\n\tpublic static Alignment fromMap(Phrase a, Phrase b, Map> map) {\n\t\t@SuppressWarnings(\"unchecked\")\n\t\tSet ms = map.isEmpty() ? Collections.EMPTY_SET : new LinkedHashSet();\n\t\t\n\t\tfor(Integer k : map.keySet()) {\n\t\t\tfor(Integer v : map.get(k)) {\n\t\t\t\tms.add(new Mapping(a.getNumber(), k, v));\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn new Alignment(a, b, ms);\n\t}\n\t\n\t/**\n\t * Creates an alignment based on this one except that the phrases have\n\t * switched positions and all of the mappings are reversed.\n\t * \n\t * @return A new alignment with the phrases swapped and all the mappings'\n\t * indices swapped.\n\t * \n\t * @see Mapping#reverse()\n\t */\n\tpublic Alignment reverse() {\n\t\t@SuppressWarnings(\"unchecked\")\n\t\tAlignment r = new Alignment(getB(), getA(), Collections.EMPTY_SET);\n\t\t\n\t\tfor(Mapping m : mappings) {\n\t\t\tr.add(m.reverse());\n\t\t}\n\t\t\n\t\treturn r;\n\t}\n\n\t/**\n\t * Gets this alignment's number.\n\t * @return The value of the {@linkplain Phrase#getNumber() number} of the phrase in\n\t * both {@linkplain PhrasePosition positions}.\n\t */\n\tpublic Integer getNumber() {\n\t\treturn a.number;\n\t}\n\t\n\t/**\n\t * Gets the phrase in {@linkplain PhrasePosition#A A-position}.\n\t */\n\tpublic Phrase getA() {\n\t\treturn get(A);\n\t}\n\n\t/**\n\t * Gets the phrase in {@linkplain PhrasePosition#B B-position}.\n\t */\n\tpublic Phrase getB() {\n\t\treturn get(B);\n\t}\n\n\t/**\n\t * Gets the phrase in the specified position.\n\t * @param pos The position in which to find the phrase.\n\t * @return If pos is {@link PhrasePosition#notifyAll()}, the A-phrase; otherwise the B-phrase.\n\t */\n\tpublic Phrase get(PhrasePosition pos) {\n\t\treturn (pos == A) ? a : b;\n\t}\n\t\n\t/**\n\t * Adds a new mapping to this alignment.\n\t * @throws IndexOutOfBoundsException If either of the indices in m are out of bounds for the\n\t * phrase in the corresponding {@linkplain PhrasePosition position}.\n\t */\n\t@Override\n\tpublic boolean add(Mapping m) {\n\t\tcheckMapping(m);\n\t\treturn mappings.add(m);\n\t}\n\n\t/**\n\t * Gets an iterator over the mappings in this alignment.\n\t */\n\t@Override\n\tpublic Iterator iterator() {\n\t\treturn mappings.iterator();\n\t}\n\n\t/**\n\t * Gets the number of mappings in this alignment.\n\t */\n\t@Override\n\tpublic int size() {\n\t\treturn mappings.size();\n\t}\n\t\n\t/**\n\t * Compares this alignment to another by comparing their {@linkplain #getNumber() numbers}.\n\t * @param o The alignment to compare to.\n\t * @return The value of getNumber().compareTo(o.getNumber()).\n\t * @see Integer#compareTo(Integer)\n\t */\n\t@Override\n\tpublic int compareTo(Alignment o) {\n\t\treturn getNumber().compareTo(o.getNumber());\n\t}\n\n\t/**\n\t * Tests whether this alignment is equal to another by comparing their mappings and their phrases.\n\t * @see Phrase#equals(Object)\n\t */\n\t@Override\n\tpublic boolean equals(Object o) {\n\t\tif(o instanceof Alignment) {\n\t\t\tAlignment al = (Alignment)o;\n\t\t\treturn super.equals(o) && a.equals(al.a) && b.equals(al.b); \n\t\t}\n\t\t\n\t\treturn false;\n\t}\n\n\t/**\n\t * Generates a hash code for this alignment based on its mappings and phrases.\n\t */\n\t@Override\n\tpublic int hashCode() {\n\t\treturn 37 * super.hashCode() + a.hashCode() + b.hashCode();\n\t}\n\n\t/**\n\t * Gets a string representation of this alignment with both phrases and the mappings between them.\n\t */\n\t@Override\n\tpublic String toString() {\n\t\tStringBuilder sb = new StringBuilder(A.name());\n\t\tsb.append(\": \");\n\t\tsb.append(a.toString());\n\t\tsb.append(\", \");\n\t\tsb.append(B.name());\n\t\tsb.append(\": \");\n\t\tsb.append(b.toString());\n\t\tsb.append(\", mappings: \");\n\t\tsb.append(super.toString());\n\t\t\n\t\treturn sb.toString();\n\t}\n\t\n\t// Views and conveniences\n\t\n\t/**\n\t * Gets the indices mapped to from the specified source, assuming that the source is in the\n\t * {@linkplain PhrasePosition#A A-position}.\n\t * \n\t * @return the value of getTargets(source, PhrasePosition.A)\n\t * @see #getTargets(Integer, PhrasePosition)\n\t */\n\tpublic Set getTargets(Integer source) {\n\t\treturn getTargets(source, A);\n\t}\n\t\n\t/**\n\t * Gets the indices mapped to by a specified index starting from the specified position. For example, if\n\t * an alignment contains the following mappings:\n\t * 

\n\t * 7 <-> 4\n\t * 3 <-> 4\n\t * 4 <-> 4\n\t * ...
\n\t * Then calling getTargets(4, {@link PhrasePosition#B}) returns a set containing 7, 3,\n\t * and 4.\n\t *

\n\t * Calling this method is equivalent to calling\n\t * {@link #asMap(PhrasePosition) asMap}(sourcePosition).get(source), with the exception that if\n\t * no mappings have source in the source position, the empty set is returned rather than\n\t * null.\n\t * \n\t * @param source The index to look for targets of.\n\t * @param sourcePosition The phrase position to assume the source index belongs to.\n\t * \n\t * @return A set of indices in the {@linkplain PhrasePosition#opposite() opposite position} that the\n\t * specified source index maps to (the same as asMap(sourcePosition).get(source)),\n\t * or the empty set if no such indices are present.\n\t * \n\t * @see #add(Mapping)\n\t * @see #asMap(PhrasePosition)\n\t */\n\tpublic Set getTargets(Integer source, PhrasePosition sourcePosition) {\n\t\treturn new LinkedHashSet(new ValueView(source, sourcePosition));\n\t}\n\t\n\t/**\n\t * Gets the indices in a specified phrase position. Specifically, returns a set containing every\n\t * integer i such that there exists a mapping in this alignment that returns i\n\t * for the call {@link Mapping#get(PhrasePosition)} with the specified position as argument.\n\t * @param position The position to get indices for.\n\t * @return The same value as asMap(position).keySet().\n\t * @see #asMap(PhrasePosition)\n\t */\n\tpublic Set getIndices(PhrasePosition position) {\n\t\treturn new LinkedHashSet(new KeyView(position));\n\t}\n\n\t/**\n\t * Gets a map view of this alignment from the {@linkplain Alignments#DEFAULT_PHRASE_POSITION \n\t * default phrase position}. \n\t * @see #asMap(PhrasePosition)\n\t */\n\tpublic Map> asMap() {\n\t\treturn asMap(Alignments.DEFAULT_PHRASE_POSITION);\n\t}\n\t\n\t/**\n\t * Gets a map view of this alignment from the specified key position. The returned map's keys are drawn\n\t * from the mappings by accessing the specified key position, while the values are aggregated together\n\t * into sets from the indices at keyPosition's {@linkplain PhrasePosition#opposite() opposite\n\t * position}. The returned map contains key/value pairs that can be used to reconstruct the alignment\n\t * it is based on via the {@link #fromMap(Phrase, Phrase, Map)} method. That is, calling\n\t * 

Alignment.fromMap(a.getA(), a.getB(), a.asMap(PhrasePosition.A))
\n\t * for any alignment a always returns an alignment that is equivalent to a according\n\t * to the {@link #equals(Object)} method.\n\t *

\n\t * For example, if this alignment contains the following mappings\n\t * 

\n\t * 0 <-> 0\n\t * 0 <-> 1\n\t * 1 <-> 2\n\t * 3 <-> 2
\n\t *
\n\t * then calling asMap(PhrasePosition.A) returns a map with the key/value pairings\n\t * 
\n\t * 0=[0, 1]\n\t * 1=[2]\n\t * 3=[2]
\n\t *
\n\t * while calling asMap(PhrasePosition.B) gives the map view from the "opposite\n\t * direction", i.e.\n\t * 
\n\t * 0=[0]\n\t * 1=[0]\n\t * 2=[1, 3]
\n\t *
\n\t * Note that the order of the keys and values reflects the ordering of the alignment's mappings via\n\t * {@link LinkedHashMap} and {@link LinkedHashSet}, and is dependent\n\t * on its {@linkplain #iterator() iterator}. Also, the behavior of\n\t * the returned map is not specified if mappings are added are removed to this alignment after a call to\n\t * asMap().\n\t *

\n\t * The returned map is detached from (not backed by) this set of mappings, so keys can be removed from and \n\t * added to it without any effect on this alignment. Similarly, the sets of indices that are the values\n\t * of its entry set can be modified without affecting this alignment. The\n\t * {@link #fromMap(Phrase, Phrase, Map)} provides the ability to create an alignment based on a map of\n\t * A indices to sets of B indices.\n\t * \n\t * @param keyPosition The phrase position that the resulting maps keys should be taken from. \n\t * @return A map whose {@linkplain Map#keySet() keys} are from the phrase in the specified position, and\n\t * whose values are sets of indices from the phrase in the {@linkplain PhrasePosition#opposite() opposite}\n\t * position.\n\t * \n\t * @see #fromMap(Phrase, Phrase, Map)\n\t * @see LinkedHashMap\n\t * @see LinkedHashSet\n\t */\n\tpublic Map> asMap(PhrasePosition keyPosition) {\n\t\treturn new LinkedHashMap>(new MapView(keyPosition));\n\t}\n\t\n\tvoid checkPhrases(Phrase ap, Phrase bp) {\n\t\tif(ap == null) {\n\t\t\tthrow new IllegalArgumentException(A.name() + \" phrase is null\");\n\t\t}\n\t\t\n\t\tif(bp == null) {\n\t\t\tthrow new IllegalArgumentException(B.name() + \" phrase is null\");\n\t\t}\n\t\t\n\t\tif(!ap.number.equals(bp.number)) {\n\t\t\tthrow new IllegalArgumentException(\"phrases have different numbers\");\n\t\t}\n\t}\n\t\n\tvoid checkMapping(Mapping m) {\n\t\tif(m == null) {\n\t\t\tthrow new IllegalArgumentException(\"attempt to add null mapping\");\n\t\t}\n\t\t\n\t\tif(m.phraseNumber != null && !m.phraseNumber.equals(a.number)) {\n\t\t\tthrow new IllegalArgumentException(\"mapping's phrase number does not match: expected \"\n\t\t\t\t\t+ a.number + \", but was \" + m.phraseNumber);\n\t\t}\n\t\t\n\t\tfor(PhrasePosition pos : PhrasePosition.values()) {\n\t\t\tcheckIndex(m.get(pos), pos);\n\t\t}\n\t}\n\t\n\tvoid checkIndex(Integer index, PhrasePosition intendedPosition) {\n\t\tif(index == null) {\n\t\t\tthrow new IllegalArgumentException(\"attempt to add null index in position \"\n\t\t\t\t+ intendedPosition.name()); \n\t\t}\n\t\t\n\t\tif(index < -1 || get(intendedPosition).size() <= index) {\n\t\t\tthrow new IndexOutOfBoundsException(intendedPosition.name() + \" index out of bounds: \" + index);\n\t\t}\n\t}\n\t\n\tclass MapView extends AbstractMap> {\n\t\tPhrasePosition keyPosition;\n\t\t\n\t\tMapView(PhrasePosition keyPosition) {\n\t\t\tthis.keyPosition = keyPosition;\n\t\t}\n\n\t\t@Override\n\t\tpublic Set>> entrySet() {\n\t\t\treturn new AbstractSet>>() {\n\t\t\t\tprivate Set keys = new KeyView(keyPosition);\n\t\t\t\t\n\t\t\t\t@Override\n\t\t\t\tpublic int size() {\n\t\t\t\t\treturn keys.size();\n\t\t\t\t}\n\n\t\t\t\t@Override\n\t\t\t\tpublic Iterator>> iterator() {\n\t\t\t\t\treturn new Iterator>>() {\n\t\t\t\t\t\tprivate Iterator i = keys.iterator();\n\t\t\t\t\t\t\n\t\t\t\t\t\t@Override\n\t\t\t\t\t\tpublic boolean hasNext() {\n\t\t\t\t\t\t\treturn i.hasNext();\n\t\t\t\t\t\t}\n\n\t\t\t\t\t\t@Override\n\t\t\t\t\t\tpublic Entry> next() {\n\t\t\t\t\t\t\tfinal Integer key = i.next();\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t// copy values because HashMap's constructor doesn't\n\t\t\t\t\t\t\treturn new SimpleImmutableEntry>(\n\t\t\t\t\t\t\t\t\tkey, new LinkedHashSet(new ValueView(key, keyPosition)));\n\t\t\t\t\t\t}\n\t\t\t\t\t\t\n\t\t\t\t\t\t@Override\n\t\t\t\t\t\tpublic void remove() {\n\t\t\t\t\t\t\ti.remove(); // throws UnsupportedOperationException\n\t\t\t\t\t\t}\n\t\t\t\t\t};\n\t\t\t\t}\n\t\t\t};\n\t\t}\n\t}\n\t\n\tabstract class IndexView extends AbstractSet {\n\n\t\tPhrasePosition indexPosition;\n\t\tFilter indexFilter;\n\t\n\t\tprivate Set indices;\n\t\t\n\t\tIndexView(PhrasePosition indexPosition, Filter indexFilter) {\n\t\t\tthis.indexPosition = indexPosition;\n\t\t\tthis.indexFilter = indexFilter;\n\t\t}\n\t\t\n\t\tSet indices() {\n\t\t\treturn (indices == null)\n\t\t\t\t\t? (indices = new FilteredSet(Alignment.this.mappings, indexFilter))\n\t\t\t\t\t: indices;\n\t\t}\n\t\t\n\t\t@Override\n\t\tpublic int size() {\n\t\t\treturn indices().size();\n\t\t}\n\n\t\t@Override\n\t\tpublic Iterator iterator() {\n\t\t\treturn new Iterator() {\n\t\t\t\tprivate Iterator i = indices().iterator();\n\n\t\t\t\t@Override\n\t\t\t\tpublic boolean hasNext() {\n\t\t\t\t\treturn i.hasNext();\n\t\t\t\t}\n\n\t\t\t\t@Override\n\t\t\t\tpublic Integer next() {\n\t\t\t\t\treturn i.next().get(indexPosition);\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\t@Override\n\t\t\t\tpublic void remove() {\n\t\t\t\t\tthrow new UnsupportedOperationException(); // just in case\n\t\t\t\t}\n\t\t\t};\n\t\t}\t\t\n\t}\n\t\n\tclass KeyView extends IndexView {\n\t\tKeyView(final PhrasePosition keyPosition) {\n\t\t\tsuper(keyPosition, new DelegatedFilter(new VisitedFilter()) {\n\t\t\t\t@Override\n\t\t\t\tpublic Integer delegateValueFor(Mapping e) {\n\t\t\t\t\treturn e.get(keyPosition);\n\t\t\t\t}\n\t\t\t});\n\t\t}\n\t}\n\t\n\tclass ValueView extends IndexView {\n\t\tValueView(final Integer key, final PhrasePosition keyPosition) {\n\t\t\tsuper(keyPosition.opposite(), new Filter() {\n\t\t\t\t@Override\n\t\t\t\tpublic boolean allows(Mapping m) {\n\t\t\t\t\treturn key.equals(m.get(keyPosition));\n\t\t\t\t}\n\t\t\t});\n\t\t}\t\t\n\t}\n}\n" }, { "path": "src/opennlp/ccg/alignment/Alignments.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\nimport static opennlp.ccg.alignment.MappingFormat.Field.A_INDEX_FIELD;\nimport static opennlp.ccg.alignment.MappingFormat.Field.B_INDEX_FIELD;\nimport static opennlp.ccg.alignment.MappingFormat.Field.PHRASE_NUMBER_FIELD;\nimport static opennlp.ccg.alignment.MappingFormat.Field.STATUS_FIELD;\n\nimport java.io.BufferedReader;\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileReader;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.io.Reader;\nimport java.io.Writer;\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.HashSet;\nimport java.util.LinkedHashMap;\nimport java.util.LinkedHashSet;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.SortedMap;\nimport java.util.SortedSet;\nimport java.util.TreeMap;\nimport java.util.TreeSet;\nimport java.util.regex.Pattern;\n\nimport opennlp.ccg.alignment.MappingFormat.Field;\n\n/**\n * Static non-instantiable class that provides convenience methods for reading\n * and writing phrases, mappings, and alignments.\n *

\n * The convenience methods tokenize split a string into an array of strings, and the\n * untokenize methods provide their inverses. \n *

\n * The readXxxPhrases() methods all call {@link #readPhrases(PhraseReader)} to read\n * {@linkplain Phrase phrases} from an underlying reader. Depending on the method, phrases are assumed to be\n * identified or to merely occur in sequence. Similarly, the writeXxxPhrases all call\n * {@link #writePhrases(List, PhraseWriter)}.\n *

\n * {@linkplain Mapping Mappings} can be read using readMappings(...) methods, which will read\n * mappings as formatted by the specified\n * format. The readSortedMappings(...) methods are variants of these that return a map with\n * sorted keys that map to {@linkplain SortedSet sorted sets} of mappings. \n * The methods {@link #writeMappings(Map, File, MappingFormat)} and\n * {@link #writeMappings(Map, Writer, MappingFormat)} perform the inverse of the methods for reading. \n *

\n * Finally, the readXxxAlignments() methods combine the methods for reading phrases and mappings to\n * allow {@linkplain Alignments alignments} to be read. The methods\n * {@link #writeAlignments(List, File, File, File, String, MappingFormat)} and\n * {@link #writeAlignments(List, Writer, Writer, Writer, String, MappingFormat)} write alignments according to\n * a specified word separator and mapping format.\n * \n * @see PhraseReader\n * @see PhraseWriter\n * @see IdentifiedPhraseReader\n * @see IdentifiedPhraseWriter\n * @see MappingReader\n * @see MappingWriter \n * @see MappingFormat\n * @author Scott Martin\n */\npublic final class Alignments {\n\n\t/**\n\t * The default status: {@link Status#SURE}.\n\t */\n\tpublic static final Status DEFAULT_STATUS = Status.SURE;\n\t\n\t/**\n\t * The default mapping confidence: 1.0.\n\t */\n\tpublic static final Double DEFAULT_CONFIDENCE = Double.valueOf(1.0d);\n\t\n\t/**\n\t * The default phrase numbering base: {@link IndexBase#ZERO}.\n\t */\n\tpublic static final IndexBase DEFAULT_PHRASE_NUMBER_BASE = IndexBase.ZERO;\n\t\n\t/**\n\t * The default index base: {@link IndexBase#ZERO}.\n\t */\n\tpublic static final IndexBase DEFAULT_INDEX_BASE = IndexBase.ZERO;\n\t\n\t/**\n\t * The default phrase position: {@link PhrasePosition#A}.\n\t */\n\tpublic static final PhrasePosition DEFAULT_PHRASE_POSITION = PhrasePosition.A;\n\t\n\t/**\n\t * The default word delimiter pattern, which matches multiple whitespace characters.\n\t */\n\tpublic static final Pattern DEFAULT_WORD_DELIMITER = Pattern.compile(\"\\\\s+\");\n\t\n\t/**\n\t * The default word separator, a single space.\n\t */\n\tpublic static final String DEFAULT_WORD_SEPARATOR = \" \";\n\t\n\t/**\n\t * The default phrase identifier attribute name, "snum".\n\t */\n\tpublic static final String DEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE = \"snum\";\n\t\n\t/**\n\t * The default phrase tag, "s".\n\t */\n\tpublic static final String DEFAULT_PHRASE_TAG = \"s\";\n\t\n\t/**\n\t * The Moses encoding scheme.\n\t */\n\tpublic static final EncodingScheme MOSES_ENCODING_SCHEME;\n\t\n\t/**\n\t * The NAACL encoding scheme.\n\t */\n\tpublic static final EncodingScheme NAACL_ENCODING_SCHEME;\n\t\n\t/**\n\t * The default fields for the Moses format.\n\t */\n\tpublic static final Set MOSES_DEFAULT_FIELDS;\n\t\n\t/**\n\t * The default fields for the NAACL format.\n\t */\n\tpublic static final Set NAACL_DEFAULT_FIELDS;\n\t\n\t/**\n\t * The fields used in the abbreviated Moses format: {@link Field#A_INDEX_FIELD},\n\t * {@link Field#B_INDEX_FIELD}.\n\t */\n\tpublic static final Set MOSES_SHORT_FIELDS;\n\t\n\t/**\n\t * The fields used in the abbreviated NAACL format: {@link Field#PHRASE_NUMBER_FIELD},\n\t * {@link Field#A_INDEX_FIELD}, {@link Field#B_INDEX_FIELD}, {@link Field#STATUS_FIELD}.\n\t */\n\tpublic static final Set NAACL_SHORT_FIELDS;\n\t\n\t/**\n\t * The fields used in the extremely abbreviated NAACL format: {@link Field#PHRASE_NUMBER_FIELD},\n\t * {@link Field#A_INDEX_FIELD}, {@link Field#B_INDEX_FIELD}.\n\t */\n\tpublic static final Set NAACL_VERY_SHORT_FIELDS;\n\t\n\t/**\n\t * The default {@linkplain MappingFormat#isStrict() mapping format strictness}: false.\n\t */\n\tpublic static final boolean DEFAULT_STRICTNESS = false;\n\t\n\t/**\n\t * The default {@linkplain IdentifiedPhraseWriter#isPadding() identified phrase writer padding}:\n\t * false.\n\t */\n\tpublic static final boolean DEFAULT_PHRASE_PADDING = false;\n\t\n\t\n\tstatic {\n\t\tSet msf = new HashSet(),\n\t\t\t\tnsf = new HashSet(),\n\t\t\t\tnvsf = new HashSet();\n\t\t\n\t\tmsf.add(A_INDEX_FIELD);\n\t\tmsf.add(B_INDEX_FIELD);\n\t\t\n\t\tnsf.add(PHRASE_NUMBER_FIELD);\n\t\tnsf.add(A_INDEX_FIELD);\n\t\tnsf.add(B_INDEX_FIELD);\n\t\tnsf.add(STATUS_FIELD);\n\t\t\n\t\tnvsf.add(PHRASE_NUMBER_FIELD);\n\t\tnvsf.add(A_INDEX_FIELD);\n\t\tnvsf.add(B_INDEX_FIELD);\n\t\t\n\t\tMOSES_SHORT_FIELDS = Collections.unmodifiableSet(msf);\n\t\tNAACL_SHORT_FIELDS = Collections.unmodifiableSet(nsf);\n\t\tNAACL_VERY_SHORT_FIELDS = Collections.unmodifiableSet(nvsf);\n\t\t\n\t\tMOSES_DEFAULT_FIELDS = MOSES_SHORT_FIELDS;\n\t\tNAACL_DEFAULT_FIELDS = NAACL_SHORT_FIELDS;\n\t\t\n\t\t// these have to come last, they depend on some of the others\n\t\tMOSES_ENCODING_SCHEME = new MosesEncodingScheme(); \n\t\tNAACL_ENCODING_SCHEME = new NAACLEncodingScheme();\n\t}\n\t\n\tprivate Alignments() {\n\t\t// this class should not be instantiated\n\t}\n\t\n\t/**\n\t * Tokenizes a string according to the {@linkplain #DEFAULT_WORD_DELIMITER default word delimiter}.\n\t */\n\tpublic static String[] tokenize(String s) {\n\t\treturn tokenize(s, DEFAULT_WORD_DELIMITER);\n\t}\n\t\n\t/**\n\t * Tokenizes an array of strings according to the {@linkplain #DEFAULT_WORD_DELIMITER default word\n\t * delimiter pattern}.\n\t */\n\tpublic static String[] tokenize(String s, Pattern wordDelimiter) {\n\t\treturn wordDelimiter.split(s);\n\t}\n\t\n\t/**\n\t * Untokenizes a list of tokens into a single string, with former tokens separated by the\n\t * {@linkplain #DEFAULT_WORD_SEPARATOR}.\n\t * \n\t * @see #untokenize(List, String)\n\t */\n\tpublic static String untokenize(List tokens) {\n\t\treturn untokenize(tokens, DEFAULT_WORD_SEPARATOR);\n\t}\n\t\n\t/**\n\t * Untokenizes a list of tokens into a single string, with former tokens separated by the\n\t * specified delimiter string.\n\t */\n\tpublic static String untokenize(List tokens, String delimiter) {\n\t\treturn untokenize(tokens.toArray(new String[tokens.size()]));\n\t}\n\t\n\t/**\n\t * Untokenizes an array of tokens into a single string using the\n\t * {@linkplain #DEFAULT_WORD_SEPARATOR default word separator}.\n\t * @param tokens\n\t * @return An untokenized string from the given tokens, with individual\n\t * tokens separated by the default word separator.\n\t */\n\tpublic static String untokenize(String[] tokens) {\n\t\treturn untokenize(tokens, DEFAULT_WORD_SEPARATOR);\n\t}\n\t\n\t/**\n\t * Untokenizes an array of tokens into a single string, with former tokens separated by the\n\t * specified delimiter string.\n\t */\n\tpublic static String untokenize(String[] tokens, String delimiter) {\n\t\tStringBuilder sb = new StringBuilder();\n\t\t\n\t\tfor(String w : tokens) {\n\t\t\tif(sb.length() > 0) {\n\t\t\t\tsb.append(delimiter);\n\t\t\t}\n\t\t\t\n\t\t\tsb.append(w.toString());\n\t\t}\n\t\t\n\t\treturn sb.toString();\n\t}\n\t\n\t/**\n\t * Reads phrases sequentially from the specified file.\n\t * @see #readPhrases(Reader)\n\t */\n\tpublic static List readPhrases(File f) throws IOException {\n\t\treturn readPhrases(new BufferedReader(new FileReader(f)));\n\t}\n\t\n\t/**\n\t * Reads phrases sequentially from the specified reader.\n\t * @see #readPhrases(Reader, IndexBase)\n\t */\n\tpublic static List readPhrases(Reader r) throws IOException {\n\t\treturn readPhrases(r, DEFAULT_PHRASE_NUMBER_BASE);\n\t}\n\t\n\t/**\n\t * Reads phrases from the specified reader. Phrases will have their line numbers translated into the\n\t * specified index base.\n\t * \n\t * @see #readPhrases(PhraseReader)\n\t */\n\tpublic static List readPhrases(Reader r, IndexBase phraseNumberBase)\n\t\t\tthrows IOException {\n\t\tPhraseReader reader = new PhraseReader(r, phraseNumberBase);\n\t\t\n\t\ttry {\n\t\t\treturn readPhrases(reader);\n\t\t}\n\t\tfinally {\n\t\t\treader.close();\n\t\t}\n\t}\n\t\n\t/**\n\t * Reads phrases from the specified file.\n\t * \n\t * @see #readIdentifiedPhrases(Reader)\n\t */\n\tpublic static List readIdentifiedPhrases(File f) throws IOException {\n\t\treturn readIdentifiedPhrases(new BufferedReader(new FileReader(f)));\n\t}\n\t\n\t/**\n\t * Reads phrases from the specified reader, using the default\n\t * {@linkplain #DEFAULT_PHRASE_NUMBER_BASE phrase number base},\n\t * {@linkplain #DEFAULT_PHRASE_TAG phrase tag}, and\n\t * {@linkplain #DEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE phrase identifier attribute}.\n\t * \n\t * @see #readIdentifiedPhrases(Reader, IndexBase, String, String)\n\t */\n\tpublic static List readIdentifiedPhrases(Reader r) throws IOException {\n\t\treturn readIdentifiedPhrases(r, DEFAULT_PHRASE_NUMBER_BASE, DEFAULT_PHRASE_TAG,\n\t\t\t\tDEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE);\n\t}\n\t\n\t/**\n\t * Reads phrases from the specified reader. The input is assumed to have markup accompanying the phrase data\n\t * that indicates the {@linkplain Phrase#getId() phrase ID} for each phrase.\n\t * \n\t * @param r The underlying reader.\n\t * @param phraseNumberBase The index base to translate line numbers into.\n\t * @param phraseTag The tag name for markup signaling a phrase.\n\t * @param phraseIdentifierAttribute The name of the attribute that contains the phrase's ID in the markup.\n\t * \n\t * @see #readPhrases(PhraseReader)\n\t */\n\tpublic static List readIdentifiedPhrases(Reader r, IndexBase phraseNumberBase, String phraseTag,\n\t\t\tString phraseIdentifierAttribute) throws IOException {\n\t\tPhraseReader reader = new IdentifiedPhraseReader(r, phraseNumberBase, phraseTag,\n\t\t\t\tphraseIdentifierAttribute);\n\t\t\n\t\ttry {\n\t\t\treturn readPhrases(reader);\n\t\t}\n\t\tfinally {\n\t\t\treader.close();\n\t\t}\n\t}\n\t\n\t/**\n\t * Reads phrases sequentially from the specified phrase reader.\n\t * \n\t * @return A list of phrases in the order they are encountered by calling {@link PhraseReader#readPhrase()}\n\t * on the specified reader.\n\t * @throws IOException if one is thrown by the specified phrase reader.\n\t * @see PhraseReader\n\t */\n\tpublic static List readPhrases(PhraseReader reader) throws IOException {\n\t\tList l = new ArrayList();\n\t\t\n\t\tPhrase p;\n\t\twhile((p = reader.readPhrase()) != null) {\n\t\t\tl.add(p);\n\t\t}\n\t\t\n\t\treturn l;\n\t}\n\t\n\t/**\n\t * Writes a list of phrases with IDs to the specified file.\n\t * \n\t * @see #writeIdentifiedPhrases(List, Writer)\n\t */\n\tpublic static void writeIdentifiedPhrases(List phrases, File f) throws IOException {\n\t\twriteIdentifiedPhrases(phrases, new BufferedWriter(new FileWriter(f)));\n\t}\n\t\n\t/**\n\t * Writes a list of phrases with IDs to the specified writer, using the default\n\t * {@linkplain #DEFAULT_WORD_SEPARATOR word separator},\n\t * {@linkplain #DEFAULT_PHRASE_TAG phrase tag}, \n\t * {@linkplain #DEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE phrase identifier attribute}, and\n\t * {@linkplain #DEFAULT_PHRASE_PADDING padding flag}.\n\t * \n\t * @see #writeIdentifiedPhrases(List, Writer, String, String, String, boolean)\n\t */\n\tpublic static void writeIdentifiedPhrases(List phrases, Writer w) throws IOException {\n\t\twriteIdentifiedPhrases(phrases, w, DEFAULT_WORD_SEPARATOR, DEFAULT_PHRASE_TAG,\n\t\t\t\tDEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE, DEFAULT_PHRASE_PADDING);\n\t}\n\t\n\t/**\n\t * Writes a list of phrases to the specified writer.\n\t * \n\t * @param phrases The phrases to write.\n\t * @param w The underlying writer.\n\t * @param wordSeparator The string to use to {@linkplain #untokenize(String[], String) untokenize} with.\n\t * @param phraseTag The name of the tag used to signal a phrase in the output markup.\n\t * @param phraseIdentifierAttribute The name of the attribute bearing the phrase's ID in the output markup.\n\t * @param padding Whether or not to include padding between the markup and the phrase data.\n\t * \n\t * @see #writePhrases(List, PhraseWriter)\n\t */\n\tpublic static void writeIdentifiedPhrases(List phrases, Writer w, String wordSeparator,\n\t\t\tString phraseTag, String phraseIdentifierAttribute, boolean padding) throws IOException {\n\t\tPhraseWriter pw = new IdentifiedPhraseWriter(w, wordSeparator, phraseTag, phraseIdentifierAttribute,\n\t\t\t\tpadding);\n\t\t\n\t\ttry {\n\t\t\twritePhrases(phrases, pw);\n\t\t}\n\t\tfinally {\n\t\t\tpw.close();\n\t\t}\n\t}\t\n\t\n\t/**\n\t * Writes a list of phrases to the specified file.\n\t * \n\t * @see #writePhrases(List, Writer)\n\t */\n\tpublic static void writePhrases(List phrases, File f) throws IOException {\n\t\twritePhrases(phrases, new BufferedWriter(new FileWriter(f)));\n\t}\n\t\n\t/**\n\t * Writes a list of phrases to the specified writer using the \n\t * {@linkplain #DEFAULT_WORD_SEPARATOR default word separator}.\n\t * \n\t * @see #writePhrases(List, Writer, String)\n\t */\n\tpublic static void writePhrases(List phrases, Writer w) throws IOException {\n\t\twritePhrases(phrases, w, DEFAULT_WORD_SEPARATOR);\n\t}\n\t\n\t/**\n\t * Writes a list of phrases to the specified writer using the specified word separator.\n\t * \n\t * @see #writePhrases(List, PhraseWriter)\n\t */\n\tpublic static void writePhrases(List phrases, Writer w, String wordSeparator)\n\t\t\tthrows IOException {\n\t\tPhraseWriter pw = new PhraseWriter(w, wordSeparator);\n\t\t\n\t\ttry {\n\t\t\twritePhrases(phrases, pw);\n\t\t}\n\t\tfinally {\n\t\t\tpw.close();\n\t\t}\n\t}\n\t\n\t/**\n\t * Writes a list of phrases to the specified phrase writer.\n\t * \n\t * @param phrases The phrases to write.\n\t * @param writer The underlying phrase writer.\n\t * @throws IOException if a call to {@link PhraseWriter#writePhrase(Phrase)} throws one for one of the\n\t * phrases.\n\t */\n\tpublic static void writePhrases(List phrases, PhraseWriter writer) throws IOException {\n\t\tfor(Phrase p : phrases) {\n\t\t\twriter.writePhrase(p);\n\t\t}\n\t}\n\t\n\t/**\n\t * Reads mappings from the specified file using the specified format.\n\t * \n\t * @see #readMappings(Reader, MappingFormat)\n\t */\n\tpublic static Map> readMappings(File f, MappingFormat format)\n\t\t\tthrows IOException {\n\t\treturn readMappings(new BufferedReader(new FileReader(f)), format);\n\t}\n\t\n\t/**\n\t * Reads mappings from the specified reader using the specified format.\n\t * \n\t * @see #readMappings(MappingReader)\n\t */\n\tpublic static Map> readMappings(Reader r, MappingFormat format)\n\t\t\tthrows IOException {\n\t\tMappingReader mr = new MappingReader(r, format);\n\t\t\n\t\ttry {\n\t\t\treturn readMappings(mr);\n\t\t}\n\t\tfinally {\n\t\t\tmr.close();\n\t\t}\n\t}\n\t\n\t/**\n\t * Reads mappings from the specified mapping reader. Once all available mappings have been read, the\n\t * specified reader is {@linkplain MappingReader#close() closed}.\n\t * \n\t * @param reader The mapping reader to read mappings from.\n\t * @return A map whose keys are the {@linkplain Phrase#getNumber() phrase numbers} of the corresponding\n\t * phrases and whose values are sets containing the mappings for the key phrase. Both the keys and the \n\t * sets of mappings are maintained in the order in which they occur in the input. \n\t * \n\t * @throws IOException if the underlying reader throws an exception, or if one of the \n\t * {@linkplain MappingGroup mapping groups} contains a duplicate mapping.\n\t * \n\t * @see MappingReader\n\t */\n\tpublic static Map> readMappings(MappingReader reader) throws IOException {\n\t\tMap> am = new LinkedHashMap>();\n\t\t\n\t\ttry {\n\t\t\tMappingGroup ag = null;\n\t\t\twhile((ag = reader.nextGroup()) != null) {\n\t\t\t\tSet as = am.get(ag.phraseNumber);\n\t\t\t\tif(as == null) {\n\t\t\t\t\tas = new LinkedHashSet();\n\t\t\t\t\tam.put(ag.phraseNumber, as);\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\twhile(reader.canRead()) {\n\t\t\t\t\tMapping m = reader.readMapping();\n\t\t\t\t\tif(!as.add(m)) {\n\t\t\t\t\t\tthrow new IOException(\"duplicate mapping in group \" + ag + \": \" + m);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tfinally {\n\t\t\treader.close();\n\t\t}\n\t\t\n\t\treturn am;\n\t}\n\t\n\t/**\n\t * Reads mappings into a sorted map from the specified file, based on the specified format.\n\t * \n\t * @see #readSortedMappings(Reader, MappingFormat)\n\t */\n\tpublic static SortedMap> readSortedMappings(File f, MappingFormat format) \n\t\t\tthrows IOException {\n\t\treturn readSortedMappings(new BufferedReader(new FileReader(f)), format);\n\t}\n\t\n\t/**\n\t * Reads mappings into a sorted map from the specified reader, using the specified format to parse\n\t * mappings. \n\t * \n\t * @see #readSortedMappings(MappingReader)\n\t */\n\tpublic static SortedMap> readSortedMappings(Reader r, MappingFormat format)\n\t\t\tthrows IOException {\n\t\tMappingReader mr = new MappingReader(r, format);\n\t\t\n\t\ttry {\n\t\t\treturn readSortedMappings(mr);\n\t\t}\n\t\tfinally {\n\t\t\tmr.close();\n\t\t}\n\t}\n\t\n\t/**\n\t * Reads mappings into a sorted map from the specified reader, using the specified format to parse\n\t * mappings.\n\t * \n\t * @param mr The mapping reader to use.\n\t * @return A sorted map whose keys and values are also sorted according to their natural order.\n\t * \n\t * @throws IOException if the underlying reader throws an exception.\n\t * \n\t * @see Mapping#compareTo(Mapping)\n\t */\n\tpublic static SortedMap> readSortedMappings(MappingReader mr)\n\t\t\tthrows IOException {\n\t\tSortedMap> sm = new TreeMap>();\n\t\t\n\t\tMap> m = readMappings(mr);\n\t\t\n\t\tfor(Integer k : m.keySet()) {\n\t\t\tsm.put(k, new TreeSet(m.get(k)));\n\t\t}\n\t\t\n\t\treturn sm;\n\t}\n\t\n\t/**\n\t * Writes the specified map to the specified file using the format provided.\n\t * \n\t * @see #writeMappings(Map, Writer, MappingFormat)\n\t */\n\tpublic static void writeMappings(Map> map, File f, MappingFormat format)\n\t\t\tthrows IOException {\n\t\twriteMappings(map, new BufferedWriter(new FileWriter(f)), format);\n\t}\n\t\n\t/**\n\t * Writes the specified map to the specified writer using the format provided.\n\t * \n\t * @see #writeMappings(Map, MappingWriter)\n\t */\n\tpublic static void writeMappings(Map> map, Writer w, MappingFormat format)\n\t\t\tthrows IOException {\n\t\twriteMappings(map, new MappingWriter(w, format));\n\t}\n\t\n\t/**\n\t * Writes the specified map to the specified writer, starting {@linkplain MappingGroup mapping groups}\n\t * as needed based on the key and the {@linkplain Set#size() size} of the value set. After all the sets\n\t * of mappings in the map have been written, {@link MappingWriter#close()} is called.\n\t * \n\t * @param map The mappings to write.\n\t * @param writer The underlying writer to write to.\n\t * @see MappingGroup\n\t * @throws IOException if one occurs in the underlying writer.\n\t */\n\tpublic static void writeMappings(Map> map, MappingWriter writer) throws IOException {\n\t\ttry {\n\t\t\tfor(Integer k : map.keySet()) {\n\t\t\t\tSet as = map.get(k);\n\t\t\t\t\n\t\t\t\twriter.startGroup(new MappingGroup(k, as.size()));\t\t\t\n\t\t\t\t\n\t\t\t\tfor(Mapping a : as) {\n\t\t\t\t\twriter.writeMapping(a);\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tfinally {\n\t\t\twriter.close();\n\t\t}\n\t}\n\t\n\t/**\n\t * Reads alignments from the specified files, using the provided mapping format.\n\t * \n\t * @see #readAlignments(Reader, Reader, Reader, MappingFormat)\n\t */\n\tpublic static List readAlignments(File phraseA, File phraseB,\n\t\t\tFile mappings, Pattern wordDelimiter, MappingFormat format) throws IOException {\n\t\treturn readAlignments(new BufferedReader(new FileReader(phraseA)),\n\t\t\tnew BufferedReader(new FileReader(phraseB)), new BufferedReader(new FileReader(mappings)), format);\n\t}\n\t\n\t/**\n\t * Reads alignments from the specified readers, using the provided mapping format.\n\t * \n\t * @see #readAlignments(PhraseReader, PhraseReader, Reader, MappingFormat)\n\t */\n\tpublic static List readAlignments(Reader phraseA, Reader phraseB,\n\t\t\tReader mappings, MappingFormat format) throws IOException {\n\t\tIndexBase idBase = format.encodingScheme.getPhraseNumberBase();\n\t\treturn readAlignments(new PhraseReader(phraseA, idBase),\n\t\t\t\tnew PhraseReader(phraseB, idBase), mappings, format);\n\t}\n\t\n\t/**\n\t * Reads alignments from the specified files, using the provided word delimiter pattern and mapping format.\n\t * The files containing phrases are assumed to contain markup indicating the phrase IDs.\n\t * \n\t * @see #readIdentifiedAlignments(Reader, Reader, Reader, MappingFormat)\n\t */\n\tpublic static List readIdentifiedAlignments(File phraseA, File phraseB,\n\t\t\tFile mappings, Pattern wordDelimiter, MappingFormat format) throws IOException {\n\t\treturn readIdentifiedAlignments(new BufferedReader(new FileReader(phraseA)),\n\t\t\t\tnew BufferedReader(new FileReader(phraseB)), new BufferedReader(new FileReader(mappings)), \n\t\t\t\tformat);\n\t}\n\t\n\t/**\n\t * Reads alignments from the specified readers, using the provided word delimiter pattern and mapping format.\n\t * The readers for phrases are assumed to have input with markup indicating the phrase IDs.\n\t * \n\t * @see #readAlignments(PhraseReader, PhraseReader, Reader, MappingFormat)\n\t */\n\tpublic static List readIdentifiedAlignments(Reader phraseA,\n\t\t\tReader phraseB, Reader mappings, MappingFormat format) throws IOException {\n\t\treturn readAlignments(new IdentifiedPhraseReader(phraseA), new IdentifiedPhraseReader(phraseB),\n\t\t\t\tmappings, format);\n\t}\n\t\n\t/**\n\t * Reads alignments from the specified readers, using the provided format to parse mappings. The\n\t * line numbers of the phrases are translated into the\n\t * {@linkplain EncodingScheme#getPhraseNumberBase() phrase number base} of the\n\t * {@linkplain MappingFormat#getEncodingScheme() format's encoding scheme}.\n\t * \n\t * @param phraseA The reader from which the {@linkplain PhrasePosition#A A-position} phrases are read.\n\t * @param phraseB The reader from which the {@linkplain PhrasePosition#B B-position} phrases are read.\n\t * @param mappings The reader whose input contains the mappings from A-phrases to B-phrases, where the\n\t * A-position indices are assumed to correspond to the phrases in phraseA and the B-position\n\t * indices are assumed to correspond to the phrases in phraseB.\n\t * @param format The mapping format used to parse mappings read from mapping.\n\t * @return A list of alignments where the {@linkplain Alignment#getA() A-phrases} are from phraseA,\n\t * the {@linkplain Alignment#getB() B-phrases} are from phraseB, and the mappings are the ones\n\t * read from mappings with the corresponding {@linkplain Phrase#getNumber() phrase number}. \n\t * @throws IOException if phraseA has a different number of phrases than phraseB, or if\n\t * one is thrown by any of the underlying readers.\n\t * \n\t * @see PhraseReader\n\t * @see MappingReader\n\t * @see #readPhrases(PhraseReader)\n\t * @see #readMappings(MappingReader)\n\t */\n\tpublic static List readAlignments(PhraseReader phraseA, PhraseReader phraseB,\n\t\t\tReader mappings, MappingFormat format) throws IOException {\n\t\tList m = new ArrayList();\n\t\t\t\t\n\t\ttry {\n\t\t\tList ps1 = readPhrases(phraseA), ps2 = readPhrases(phraseB);\n\t\t\tMap> mm = readMappings(mappings, format);\n\t\t\t\n\t\t\t// sanity check\n\t\t\tif(ps1.size() != ps2.size()) {\n\t\t\t\tthrow new IOException(\"number of phrases different between first and second\");\n\t\t\t}\n\n\t\t\tfor(int i = 0; i < ps1.size(); i++) {\t\t\t\t\n\t\t\t\tm.add(new Alignment(ps1.get(i), ps2.get(i), mm.get(i)));\n\t\t\t}\n\t\t}\n\t\tfinally {\n\t\t\tphraseA.close();\n\t\t\tphraseA.close();\n\t\t\tmappings.close();\n\t\t}\n\t\t\n\t\treturn m;\n\t}\n\t\n\t/**\n\t * Writes a list of alignments to the specified files, using the word separator and format provided.\n\t * \n\t * @see #writeAlignments(List, Writer, Writer, Writer, String, MappingFormat)\n\t */\n\tpublic static void writeAlignments(List alignments,\n\t\t\tFile phraseA, File phraseB, File mappings, String wordSeparator, MappingFormat format)\n\t\t\t\tthrows IOException {\n\t\twriteAlignments(alignments, new BufferedWriter(new FileWriter(phraseA)),\n\t\t\tnew BufferedWriter(new FileWriter(phraseB)), new BufferedWriter(new FileWriter(mappings)),\n\t\t\t\twordSeparator, format);\n\t}\n\t\n\t/**\n\t * Writes a list of alignments to the specified readers, using the word separator and format provided.\n\t * After all the alignments are written, all the associated writers are closed.\n\t * \n\t * @param alignments The alignments to write.\n\t * @param phraseA The writer to which {@linkplain Alignment#getA() A-position phrases} are written.\n\t * @param phraseB The writer to which {@linkplain Alignment#getB() B-position phrases} are written.\n\t * @param mappings The mappings to write, where the {@linkplain Mapping#getA() A-position indices} are\n\t * assumed to correspond to the phrases in phraseA and the \n\t * {@linkplain Mapping#getB() B-position indices} are assumed to correspond to the ones in phraseB.\n\t * @param wordSeparator The word separator to use for\n\t * {@linkplain #untokenize(List, String) untokenization}.\n\t * @param format The format to use for formatting mappings in the alignments.\n\t * @throws IOException if one is thrown by any of the underlying writers.\n\t */\n\tpublic static void writeAlignments(List alignments,\n\t\t\tWriter phraseA, Writer phraseB, Writer mappings, String wordSeparator, MappingFormat format)\n\t\t\t\tthrows IOException {\n\t\tPhraseWriter pw1 = new PhraseWriter(phraseA, wordSeparator),\n\t\t\tpw2 = new PhraseWriter(phraseB, wordSeparator);\n\t\tMappingWriter mw = new MappingWriter(mappings, format);\n\t\t\n\t\ttry {\n\t\t\tfor(int i = 0; i < alignments.size(); i++) {\n\t\t\t\tAlignment a = alignments.get(i);\n\t\t\t\t\n\t\t\t\tpw1.writePhrase(a.a);\n\t\t\t\tpw2.writePhrase(a.b);\n\t\n\t\t\t\tmw.startGroup(new MappingGroup(i, a.size()));\n\t\t\t\t\n\t\t\t\tfor(Mapping m : a) {\n\t\t\t\t\tmw.writeMapping(m);\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tfinally {\n\t\t\tpw1.close();\n\t\t\tpw2.close();\n\t\t\tmw.close();\n\t\t}\n\t}\n}\n" }, { "path": "src/opennlp/ccg/alignment/EncodingScheme.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\nimport java.util.List;\nimport java.util.Set;\n\n/**\n * An encoding scheme for mappings. Some examples are\n * the {@linkplain MosesEncodingScheme Moses scheme} and the {@linkplain NAACLEncodingScheme NAACL scheme}.\n *

\n * Implementers keep track of the {@link IndexBase}s corresponding to an encoding scheme (for both phrase \n * numbers and indices), which characters it uses to delimit mappings, groups of mappings, and fields within\n * mappings. It also captures which fields occur\n * in which {@linkplain #getOrder() order} in an encoding scheme, along with which ones are\n * {@linkplain #getRequired() required}, and which are used by {@linkplain #getDefaults() default}.\n *

\n * Some convenience methods are provided for determining whether a given\n * character is the field, mapping, or group delimiter for this encoding scheme. These methods mainly allow\n * comparison when one character is a line separator that may be different from the one on the current platform.\n *

\n * Encoding schemes are used in the {@link MappingFormat} class, as well as in\n * the readers and writers for {@link Mapping}s and {@link Alignment}s. \n * \n * @author Scott Martin\n * @see AbstractEncodingScheme\n * @see MappingFormat\n * @see Moses alignment format\n * @see NAACL shared task alignment format\n */\npublic interface EncodingScheme {\n\t\n\t/**\n\t * Tests whether the supplied character counts as a field delimiter according to this encoding scheme.\n\t * @return true if c is equal to {@link #getFieldDelimiter()} or both c and \n\t * field delimiters are line separators.\n\t */\n\tpublic boolean isFieldDelimiter(Character c);\n\t\n\t/**\n\t * Tests whether the supplied character counts as a mapping delimiter according to this encoding scheme.\n\t * @return true if c is equal to {@link #getMappingDelimiter()} or both c and \n\t * mapping delimiters are line separators.\n\t */\n\tpublic boolean isMappingDelimiter(Character c);\n\t\n\t/**\n\t * Tests whether the supplied character counts as a group delimiter according to this encoding scheme.\n\t * @return true if c is equal to {@link #getGroupDelimiter()} or both c and \n\t * group delimiters are line separators.\n\t */\n\tpublic boolean isGroupDelimiter(Character c);\n\n\t/**\n\t * Gets the delimiter for fields.\n\t */\n\tpublic Character getFieldDelimiter();\n\n\t/**\n\t * Gets the delimiter for mappings.\n\t */\n\tpublic Character getMappingDelimiter();\n\n\t/**\n\t * Gets the delimiter for groups.\n\t */\n\tpublic Character getGroupDelimiter();\n\n\t/**\n\t * Gets the numbering base used for phrases.\n\t */\n\tpublic IndexBase getPhraseNumberBase();\n\n\t/**\n\t * Gets the numbering base used for mapping indices.\n\t */\n\tpublic IndexBase getIndexBase();\n\n\t/**\n\t * Gets the order in which fields occur in this encoding scheme.\n\t */\n\tpublic List getOrder();\n\n\t/**\n\t * Gets the required (non-optional) fields in this scheme.\n\t */\n\tpublic Set getRequired();\n\t\n\t/**\n\t * Gets the default fields in this scheme.\n\t */\n\tpublic Set getDefaults();\n}\n" }, { "path": "src/opennlp/ccg/alignment/IdentifiedPhraseReader.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\nimport static opennlp.ccg.alignment.Alignments.DEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE;\nimport static opennlp.ccg.alignment.Alignments.DEFAULT_PHRASE_NUMBER_BASE;\nimport static opennlp.ccg.alignment.Alignments.DEFAULT_PHRASE_TAG;\nimport static opennlp.ccg.alignment.Alignments.DEFAULT_WORD_DELIMITER;\n\nimport java.io.IOException;\nimport java.io.Reader;\nimport java.util.ArrayList;\nimport java.util.List;\n\n/**\n * Phrase reader for phrases that have an identifier in addition to a line number, usually formatted by\n * markup like\n * 

\n * <s snum="157"> ... </s>\n *
\n * Since the value of the identifier is not necessarily an {@link Integer}, this class gives IDs as strings.\n * The {@linkplain #getPhraseIdentifierAttribute() phrase identifier attribute} (here snum) and\n * {@linkplain #getPhraseTag() phrase tag} (here s) are configurable when instances are\n * constructed.\n * \n * @author Scott Martin\n * @see Alignments#DEFAULT_PHRASE_TAG\n * @see Alignments#DEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE\n */\npublic class IdentifiedPhraseReader extends PhraseReader {\n\n\tstatic final String TAG_START = \"<\", TAG_END = \">\";\n\t\n\tfinal String phraseTag, phraseIdentifierAttribute;\n\t\n\tprivate String lastId = null;\n\t\n\t/**\n\t * Creates an identified phrase reader with the {@linkplain Alignments#DEFAULT_PHRASE_NUMBER_BASE default\n\t * number base}.\n\t * \n\t * @see #IdentifiedPhraseReader(Reader, IndexBase)\n\t * @see Alignments#DEFAULT_PHRASE_TAG\n\t * @see Alignments#DEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE\n\t */\n\tpublic IdentifiedPhraseReader(Reader in) {\n\t\tthis(in, DEFAULT_PHRASE_NUMBER_BASE);\t\t\n\t}\n\t\n\t/**\n\t * Creates an identified phrase reader with the {@linkplain Alignments#DEFAULT_PHRASE_TAG default phrase tag}\n\t * and {@link Alignments#DEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE default phrase identifier attribute}.\n\t * \n\t * @see #IdentifiedPhraseReader(Reader, IndexBase, String, String)\n\t * @see Alignments#DEFAULT_PHRASE_TAG\n\t * @see Alignments#DEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE\n\t */\n\tpublic IdentifiedPhraseReader(Reader in, IndexBase numberBase) {\n\t\tthis(in, numberBase, DEFAULT_PHRASE_TAG, DEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE);\t\t\n\t}\n\n\t/**\n\t * Creates an identified phrase reader for the specified phrase tag and identifier attribute.\n\t * @param phraseTag The phrase tag that will be used to parse identifiers.\n\t * @param phraseIdentifierAttribute The attribute that will denote identifiers when parsing phrases.\n\t * @throws IllegalArgumentException if phraseIdentifierAttribute or phraseTag \n\t * is null. \n\t */\n\tpublic IdentifiedPhraseReader(Reader in, IndexBase numberBase, String phraseTag,\n\t\t\tString phraseIdentifierAttribute) {\n\t\tsuper(in, numberBase);\n\t\t\n\t\tif(phraseIdentifierAttribute == null) {\n\t\t\tthrow new IllegalArgumentException(\"phraseIdentifierAttribute is null\");\n\t\t}\n\t\tif(phraseTag == null) {\n\t\t\tthrow new IllegalArgumentException(\"phraseTag is null\");\n\t\t}\n\t\t\n\t\tthis.phraseIdentifierAttribute = phraseIdentifierAttribute;\n\t\tthis.phraseTag = phraseTag;\n\t}\n\n\t/**\n\t * Gets the last ID encountered.\n\t */\n\tpublic String getLastId() {\n\t\treturn lastId;\n\t}\n\n\t/**\n\t * Gets the tag that denotes a phrase.\n\t */\n\tpublic String getPhraseTag() {\n\t\treturn phraseTag;\n\t}\n\n\t/**\n\t * Gets the attribute name that denotes a phrase identifier.\n\t */\n\tpublic String getPhraseIdentifierAttribute() {\n\t\treturn phraseIdentifierAttribute;\n\t}\n\n\t/**\n\t * Reads a phrase from the underlying input stream, first parsing the {@linkplain Phrase#getNumber()\n\t * phrase's number} based on the {@linkplain #getPhraseTag() phrase tag} and\n\t * {@linkplain #getPhraseIdentifierAttribute() identifier attribute} being used.\n\t * @return A phrase with the ID signaled in the input.\n\t * @throws IOException If the underlying input contains ill-formated phrase markup, or if no ID cannot be\n\t * determined after parsing the phrase markup.\n\t */\n\t@Override\n\tpublic Phrase readPhrase() throws IOException {\n\t\tString ln = readLine();\n\t\tif(ln == null) {\n\t\t\treturn null;\n\t\t}\n\t\t\n\t\tString[] chunks = DEFAULT_WORD_DELIMITER.split(ln);\n\t\tString c = chunks[0];\n\t\t\n\t\tif(!c.trim().startsWith(TAG_START)) {\n\t\t\tthrow new IOException(\"unable to parse: \" + ln + \"; expected <, but was \" + c.trim());\n\t\t}\n\t\t\n\t\tint clen = c.length();\n\t\tint pos = (clen > 1) ? 1 : 2;\n\t\tString t = (pos == 1) ? c.substring(1).trim() : chunks[1];\n\t\tif(!t.equals(phraseTag)) {\n\t\t\tthrow new IOException(\"expected sequence tag \" + phraseTag + \", but was \" + t);\n\t\t}\n\t\t\n\t\tboolean foundIndex = false;\n\t\tint start = -1;\n\t\t\n\t\tfor(int i = pos; i < chunks.length; i++) {\n\t\t\tif(!foundIndex) {\n\t\t\t\tString[] subchunks = chunks[i].trim().split(\"=\");\n\t\t\t\tif(subchunks.length > 1) {\n\t\t\t\t\tif(subchunks[0].equals(phraseIdentifierAttribute)) {\n\t\t\t\t\t\tchar[] idVal = subchunks[1].toCharArray();\n\t\t\t\t\t\t\n\t\t\t\t\t\tint idStart = 0, idEnd = idVal.length - 1;\n\t\t\t\t\t\tboolean foundStart = false;\n\t\t\t\t\t\tfor(int j = 0; j < idVal.length; j++) {\n\t\t\t\t\t\t\tif(idVal[j] == '\\'' || idVal[j] == '\\\"') {\n\t\t\t\t\t\t\t\tif(foundStart) {\n\t\t\t\t\t\t\t\t\tidEnd = j;\n\t\t\t\t\t\t\t\t\tbreak;\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\telse {\n\t\t\t\t\t\t\t\t\tfoundStart = true;\n\t\t\t\t\t\t\t\t\tidStart = j + 1;\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t\t\n\t\t\t\t\t\tlastId = new String(idVal).substring(idStart, idEnd);\n\t\t\t\t\t\tfoundIndex = true;\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tif(chunks[i].contains(TAG_END)) {\n\t\t\t\tstart = i;\n\t\t\t\tbreak;\n\t\t\t}\n\t\t}\n\t\t\n\t\tif(!foundIndex) {\n\t\t\tthrow new IOException(\"no ID found on line \" + getLineNumber());\n\t\t}\n\t\t\n\t\tList l = new ArrayList(chunks.length);\n\t\tfor(int j = start; j < chunks.length; j++) {\n\t\t\tString cj = chunks[j];\n\t\t\t\n\t\t\tif(j == start) {\n\t\t\t\tint te = cj.indexOf(TAG_END);\n\t\t\t\tif(te != -1) {\n\t\t\t\t\tcj = cj.substring(te + 1);\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tif(j + 1 == chunks.length) {\n\t\t\t\tint ts = cj.indexOf(TAG_START);\n\t\t\t\tif(ts != -1) {\n\t\t\t\t\tcj = cj.substring(0, ts);\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tif(cj.length() > 0) {\n\t\t\t\tl.add(cj);\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn new Phrase(lastId, getPhraseNumber(), l);\n\t}\n}\n" }, { "path": "src/opennlp/ccg/alignment/IdentifiedPhraseWriter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\nimport java.io.IOException;\nimport java.io.Writer;\n\nimport static opennlp.ccg.alignment.Alignments.*;\n\n/**\n * A writer for phrases with {@linkplain Phrase#getId() ids} in addition to\n * {@linkplain Phrase#getNumber() numbers}. Identified phrases are ones read from\n * markup that signals an ID for each phrase.\n *

\n * In addition to allowing the phrase tag and identifier attribute to be configured, this class can optionally\n * output {@linkplain #isPadding() padding} of a single space between the markup and the phrase.\n * \n * @author Scott Martin\n * @see IdentifiedPhraseReader\n */\npublic class IdentifiedPhraseWriter extends PhraseWriter {\n\n\tfinal String phraseTag, phraseIdentifierAttribute;\n\tfinal boolean padding;\n\t\n\t/**\n\t * Creates a new identified phrase writer. The word separator used is\n\t * {@link Alignments#DEFAULT_WORD_SEPARATOR}.\n\t * @see #IdentifiedPhraseWriter(Writer, String)\n\t */\n\tpublic IdentifiedPhraseWriter(Writer out) {\n\t\tthis(out, Alignments.DEFAULT_WORD_SEPARATOR);\n\t}\n\n\t/**\n\t * Creates a new identified phrase writer with the given word separator. The phrase tag used is the default,\n\t * {@link Alignments#DEFAULT_PHRASE_TAG}, as is the phrase ID attribute\n\t * ({@link Alignments#DEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE}).\n\t * @see #IdentifiedPhraseWriter(Writer, String, String, String)\n\t */\n\tpublic IdentifiedPhraseWriter(Writer out, String wordSeparator) {\n\t\tthis(out, wordSeparator, DEFAULT_PHRASE_TAG, DEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE);\n\t}\n\t\n\t/**\n\t * Creates a new identified phrase writer with the given word separator. The phrase tag used is the default,\n\t * {@link Alignments#DEFAULT_PHRASE_TAG}, as is the phrase ID attribute\n\t * ({@link Alignments#DEFAULT_PHRASE_IDENTIFIER_ATTRIBUTE}).\n\t * @see #IdentifiedPhraseWriter(Writer, String, String, String, boolean)\n\t */\n\tpublic IdentifiedPhraseWriter(Writer out, String wordSeparator, String phraseTag,\n\t\t\tString phraseIdentifierAttribute) {\n\t\tthis(out, wordSeparator, phraseTag, phraseIdentifierAttribute, DEFAULT_PHRASE_PADDING);\n\t}\n\t\n\t/**\n\t * Creates a new identified phrase writer for writing phrases to the underlying writer.\n\t * @param phraseTag The name of the tag that holds the phrase identifier.\n\t * @param phraseIdentifierAttribute The phrase identifier attribute.\n\t * @param padding Whether this writer should write a space between the pre-markup and the phrase, and\n\t * between the phrase and the post-markup.\n\t * @throws IllegalArgumentException if phraseTag or phraseIdentifierAttribute\n\t * is null.\n\t */\n\tpublic IdentifiedPhraseWriter(Writer out, String wordSeparator, String phraseTag,\n\t\t\tString phraseIdentifierAttribute, boolean padding) {\n\t\tsuper(out, wordSeparator);\n\t\t\n\t\tif(phraseTag == null) {\n\t\t\tthrow new IllegalArgumentException(\"phraseTag is null\");\n\t\t}\n\t\tif(phraseIdentifierAttribute == null) {\n\t\t\tthrow new IllegalArgumentException(\"phraseIdentifierAttribute is null\");\n\t\t}\n\t\t\n\t\tthis.phraseTag = phraseTag;\n\t\tthis.phraseIdentifierAttribute = phraseIdentifierAttribute;\n\t\tthis.padding = padding;\n\t}\n\n\t/**\n\t * Gets the phrase tag this writer generates with each phrase.\n\t */\n\tpublic String getPhraseTag() {\n\t\treturn phraseTag;\n\t}\n\n\t/**\n\t * Gets the phrase identifier attribute used to signal the phrase ID for each phrase.\n\t */\n\tpublic String getPhraseIdentifierAttribute() {\n\t\treturn phraseIdentifierAttribute;\n\t}\n\n\t/**\n\t * Tests whether this writer writes space padding between its pre- and post-markup and the phrase itself.\n\t * @return true if this writer uses padding.\n\t */\n\tpublic boolean isPadding() {\n\t\treturn padding;\n\t}\n\n\t/**\n\t * Writes the start markup indicating a phrase's ID. The tag and ID attribute are the ones used to\n\t * create this writer.\n\t * \n\t * @see #getPhraseTag()\n\t * @see #getPhraseIdentifierAttribute()\n\t * @throws IOException If a problem occurs in the underlying writer.\n\t */\n\t@Override\n\tpublic void preWritePhrase(Phrase phrase) throws IOException {\n\t\tString id = phrase.getId();\n\t\tif(id == null) {\n\t\t\tthrow new IOException(\"attempt to write phrase with null ID: \" + phrase);\n\t\t}\n\t\t\n\t\tprintWriter.print('<');\n\t\tprintWriter.print(phraseTag);\n\t\tprintWriter.print(' ');\n\t\tprintWriter.print(phraseIdentifierAttribute);\n\t\tprintWriter.print(\"=\\\"\");\n\t\tprintWriter.print(id);\n\t\tprintWriter.print(\"\\\">\");\n\t\t\n\t\tif(padding) {\n\t\t\tprintWriter.print(' ');\n\t\t}\n\t}\n\t\n\t/**\n\t * Writes the end markup signaling the end of a phrase, plus a line separator.\n\t * \n\t * @throws IOException If a problem occurs in the underlying writer.\n\t */\n\t@Override\n\tpublic void postWritePhrase(Phrase phrase) throws IOException {\n\t\tif(padding) {\n\t\t\tprintWriter.print(' ');\n\t\t}\n\t\t\n\t\tprintWriter.print(\"\");\n\t}\n}\n" }, { "path": "src/opennlp/ccg/alignment/IndexBase.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\n/**\n * A set of {@link Enum} constants representing the two common indexing bases used in representing alignments.\n * The constants are ordered so that their {@link Enum#ordinal()} method returns a number corresponding to \n * their {@linkplain #getStart() start index}.\n *

\n * An index base is characterized by its {@linkplain IndexBase#getStart() starting point} and its \n * {@linkplain #getNullValue() null value}, which is derived from its starting point by decrementing by one.\n * One index base can translate an integer in that base into another base via the\n * {@link #translate(Integer, IndexBase)} method.\n * \n * @author Scott Martin\n * @see EncodingScheme\n */\npublic enum IndexBase {\n\t\n\t/**\n\t * The index base that starts with 0.\n\t */\n\tZERO,\n\t\n\t/**\n\t * The index base starting with 1.\n\t */\n\tONE;\n\t\n\tfinal Integer start, nullValue;\n\t\n\tprivate IndexBase() {\n\t\tthis.start = Integer.valueOf(ordinal());\n\t\tthis.nullValue = start - 1;\n\t}\n\t\n\t/**\n\t * Gets the starting point of this index base.\n\t */\n\tpublic Integer getStart() {\n\t\treturn start;\n\t}\n\n\t/**\n\t * This index base's special null value, the value of {@link #getStart()} - 1.\n\t */\n\tpublic Integer getNullValue() {\n\t\treturn nullValue;\n\t}\n\n\t/**\n\t * Tests whether the specified index is valid in this index base.\n\t * @param index The index to test.\n\t * @return true If index is non-null and not less than {@link #getNullValue()}.\n\t */\n\tpublic boolean isValidIndex(Integer index) {\n\t\treturn index != null && nullValue <= index;\n\t}\n\t\n\t/**\n\t * Translates an index in this base to another base. For example, ZERO.translate(2, ONE) yields\n\t * 3 and ONE.translate(1, ZERO) yields 0. Note that supplying the same\n\t * index base as the target has no effect, so that if b is an index base constant, then\n\t * b.translate(n, b) returns n for every {@link Integer} n as long\n\t * as n is {@linkplain #isValidIndex(Integer) valid} (throwing an exception otherwise).\n\t * @param index The index to translate.\n\t * @param target The target index base to translate index into.\n\t * @return The value of index as it is represented in the index base target.\n\t * @throws IllegalArgumentException If index is invalid for this index base.\n\t * @see #isValidIndex(Integer) \n\t */\n\tpublic Integer translate(Integer index, IndexBase target) {\n\t\tif(!isValidIndex(index)) {\n\t\t\tthrow new IllegalArgumentException(\"invalid index for index base \" + name() + \": \" + index);\n\t\t}\n\t\t\n\t\tif(target == this) {\n\t\t\treturn index;\n\t\t}\n\t\t\n\t\treturn Integer.valueOf(index + (target.start - start));\n\t}\n}" }, { "path": "src/opennlp/ccg/alignment/Mapping.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\nimport static opennlp.ccg.alignment.PhrasePosition.*;\n\n/**\n * A mapping from an index in the {@linkplain PhrasePosition#A A} phrase position to an index in the\n * {@linkplain PhrasePosition#B B} phrase position. Mappings are interpreted as the individual pairs\n * that make up an {@link Alignment} from one {@link Phrase} to another. This class implements\n * {@link Comparable} so that mappings can be easily sorted (see the {@link #compareTo(Mapping)} method).\n *

\n * Although different\n * {@linkplain EncodingScheme encoding schemes} may use different {@linkplain IndexBase index bases}, all\n * mappings share a common one, namely {@link Alignments#DEFAULT_INDEX_BASE}. As such, no index can be \n * specified that is less than the {@linkplain IndexBase#nullValue null value} of that index base. Similarly,\n * the phrase number specified (if any) must be in the range of the\n * {@linkplain Alignments#DEFAULT_PHRASE_NUMBER_BASE default phrase number base},\n * even though different encoding schemes may have different phrase number bases.\n *

\n * Since some {@linkplain EncodingScheme encoding schemes} do not require an\n * {@linkplain #getPhraseNumber() phrase number}\n * to be specified, the phrase number field may be null. However, none of the other fields may be\n * null. To specify that the mapping is to the special null word value, the\n * {@linkplain Alignments#DEFAULT_INDEX_BASE default index base}'s\n * {@linkplain IndexBase#nullValue null value} is used.\n * \n * @author Scott Martin\n * @see PhrasePosition\n * @see EncodingScheme\n * @see IndexBase\n * @see Alignments#DEFAULT_PHRASE_NUMBER_BASE\n * @see Alignments#DEFAULT_INDEX_BASE\n * @see Alignments#DEFAULT_STATUS\n * @see Alignments#DEFAULT_CONFIDENCE\n */\npublic class Mapping implements Comparable {\n\n\tfinal Integer phraseNumber, a, b;\n\tStatus status;\n\tDouble confidence;\n\t\n\t/**\n\t * Creates a mapping between the two specified indices with a null phrase number.\n\t * @see #Mapping(Integer, Integer, Integer)\n\t */\n\tpublic Mapping(Integer a, Integer b) {\n\t\tthis(null, a, b);\n\t}\n\t\n\t/**\n\t * Creates a mapping between the two specified indices with a null phrase number and the\n\t * {@linkplain Alignments#DEFAULT_STATUS default status}.\n\t * @see #Mapping(Integer, Integer, Integer, Status)\n\t */\n\tpublic Mapping(Integer phraseNumber, Integer a, Integer b) {\n\t\tthis(phraseNumber, a, b, Alignments.DEFAULT_STATUS);\n\t}\n\t\n\t/**\n\t * Creates a mapping between the two specified indices with the specified phrase number and status, with the\n\t * {@linkplain Alignments#DEFAULT_CONFIDENCE default confidence}.\n\t * @see #Mapping(Integer, Integer, Integer, Status, Double)\n\t */\n\tpublic Mapping(Integer phraseNumber, Integer a, Integer b, Status status) {\n\t\tthis(phraseNumber, a, b, status, Alignments.DEFAULT_CONFIDENCE);\n\t}\n\t\n\t/**\n\t * Creates a mapping between the two specified indices.\n\t * @param phraseNumber The phrase number of the corresponding mapping.\n\t * @param a The A index, corresponding to {@link PhrasePosition#A}.\n\t * @param b The B index, corresponding to {@link PhrasePosition#B}.\n\t * @param status The status of this mapping.\n\t * @param confidence This mapping's confidence value.\n\t * \n\t * @throws IndexOutOfBoundsException If any of phraseNumber, a, or\n\t * b is out of bounds according to the corresponding default.\n\t * @throws IllegalArgumentException If either index is null, or if status or\n\t * confidence is null.\n\t * \n\t * @see Alignments#DEFAULT_PHRASE_NUMBER_BASE\n\t * @see Alignments#DEFAULT_INDEX_BASE\n\t */\n\tpublic Mapping(Integer phraseNumber, Integer a, Integer b, Status status, Double confidence) {\n\t\tcheckPhraseNumber(phraseNumber);\n\t\tcheckIndex(a);\n\t\tcheckIndex(b);\n\t\tcheckField(status, \"status\");\n\t\tcheckField(confidence, \"confidence\");\n\t\t\n\t\tthis.phraseNumber = phraseNumber;\n\t\tthis.a = a;\n\t\tthis.b = b;\n\t\tthis.status = status;\n\t\tthis.confidence = confidence;\n\t}\n\t\n\t/**\n\t * Gets a copy of this mapping with its {@linkplain #getPhraseNumber() phrase number}\n\t * set to the specified phrase number. This method is a convenience for the {@link MappingReader} class,\n\t * which, for certain encoding scheme like {@link MosesEncodingScheme}, may not be able to\n\t * parse the phrase number from the input string.\n\t * @param phraseNumber The phrase number the returned mapping should have.\n\t * @return This mapping if its {@linkplain #getPhraseNumber() phrase number} is null and the\n\t * specified phrase number is null, or if this mapping's phrase number is\n\t * {@linkplain Integer#equals(Object) equivalent to} the specified phrase number. Otherwise, a new \n\t * mapping is returned with all the same field values as this mapping, but with its phrase number set\n\t * to phraseNumber.\n\t * \n\t * @see #Mapping(Integer, Integer, Integer, Status, Double)\n\t */\n\tpublic Mapping copyWithPhraseNumber(Integer phraseNumber) {\n\t\tif((this.phraseNumber == null && phraseNumber == null)\n\t\t\t\t|| (this.phraseNumber != null && this.phraseNumber.equals(phraseNumber))) {\n\t\t\treturn this;\n\t\t}\n\t\t\n\t\treturn new Mapping(phraseNumber, a, b, status, confidence);\n\t}\n\t\n\t/**\n\t * Convenience method for creating mappings when the phrase position may possibly be \n\t * {@link PhrasePosition#B}.\n\t * @see #mappingByPosition(Integer, Integer, Integer, Status, Double, PhrasePosition)\n\t */\n\tpublic static Mapping mappingByPosition(Integer phraseNumber, Integer a, Integer b,\n\t\t\tPhrasePosition firstPosition) {\n\t\treturn mappingByPosition(phraseNumber, a, b, Alignments.DEFAULT_STATUS,\n\t\t\t\tAlignments.DEFAULT_CONFIDENCE, firstPosition);\n\t}\n\n\t/**\n\t * Convenience method for creating mappings in case which index should be {@linkplain #getA() A} and\n\t * which should be {@linkplain #getB() B} depends on the phrase position.\n\t * \n\t * @param phraseNumber The phrase number to use.\n\t * @param a The index to use either for the A (if the first position is\n\t * {@link PhrasePosition#A}) or B index (if the first position is {@link PhrasePosition#B}).\n\t * @param b The index to use either for the B (if the first position is\n\t * {@link PhrasePosition#A}) or A index (if the first position is {@link PhrasePosition#B}).\n\t * @param status The status to use.\n\t * @param confidence The confidence to use.\n\t * @param firstPosition Which phrase position the new mapping should reflect. If this argument is\n\t * {@link PhrasePosition#A}, the a argument will be the A index and the\n\t * b argument the B. If it is {@link PhrasePosition#B}, these are reversed.\n\t * @return A new mapping with its indices configured per the specified firstPosition.\n\t * @see Mapping#Mapping(Integer, Integer, Integer, Status, Double)\n\t */\n\tpublic static Mapping mappingByPosition(Integer phraseNumber, Integer a, Integer b, Status status,\n\t\t\tDouble confidence, PhrasePosition firstPosition) {\n\t\treturn new Mapping(phraseNumber,\n\t\t\t\t(firstPosition == A) ? a : b, (firstPosition == B) ? a : b, status, confidence);\n\t}\n\t\n\t/**\n\t * Gets a new mapping just like this one except that the indices in the {@link PhrasePosition#A}\n\t * and {@link PhrasePosition#B} positions are swapped. The original status and confidence are\n\t * unchanged.\n\t * @return A mapping with indices reversed.\n\t * @see Mapping#mappingByPosition(Integer, Integer, Integer, Status, Double, PhrasePosition)\n\t */\n\tpublic Mapping reverse() {\n\t\treturn mappingByPosition(phraseNumber, a, b, status, confidence, B);\n\t}\n\t\n\t/**\n\t * Gets this mapping's phrase number.\n\t */\n\tpublic Integer getPhraseNumber() {\n\t\treturn phraseNumber;\n\t}\n\n\t/**\n\t * Gets this mapping's index in the {@linkplain PhrasePosition#A A-position}.\n\t * @return The value of {@link #get(PhrasePosition)} for {@link PhrasePosition#A}.\n\t */\n\tpublic Integer getA() {\n\t\treturn get(A);\n\t}\n\n\t/**\n\t * Gets this mapping's index in the {@linkplain PhrasePosition#B B-position}.\n\t * @return The value of {@link #get(PhrasePosition)} for {@link PhrasePosition#B}.\n\t */\n\tpublic Integer getB() {\n\t\treturn get(B);\n\t}\n\n\t/**\n\t * Gets this mapping's index at the specified phrase position.\n\t * @param pos The phrase position at which to return the corresponding index.\n\t * @return If pos is {@link PhrasePosition#A}, the A index; otherwise the B index.\n\t */\n\tpublic Integer get(PhrasePosition pos) {\n\t\treturn (pos == A) ? a : b;\n\t}\n\t\n\t/**\n\t * Gets this mapping's status.\n\t */\n\tpublic Status getStatus() {\n\t\treturn status;\n\t}\n\n\t/**\n\t * Sets this mapping's status to the specified value.\n\t */\n\tpublic void setStatus(Status status) {\n\t\tcheckField(status, \"status\");\n\t\tthis.status = status;\n\t}\n\n\t/**\n\t * Gets this mapping's confidence.\n\t */\n\tpublic Double getConfidence() {\n\t\treturn confidence;\n\t}\n\n\t/**\n\t * Sets this mapping's confidence to the supplied value.\n\t * @param confidence May be null. No bounds checking is performed on this value even if it\n\t * is non-null.\n\t */\n\tpublic void setConfidence(Double confidence) {\n\t\tcheckField(confidence, \"confidence\");\n\t\tthis.confidence = confidence;\n\t}\n\n\t/**\n\t * Compares this mapping to another according to their natural ordering. The natural ordering of\n\t * mappings is that first their IDs are compared, then their A indices, then their B indices,\n\t * and finally their status and confidence value (in that order).\n\t *

\n\t * For the ID field, which may be null, the comparison is performed as follows. If both this mapping's \n\t * ID and the other's are null, they are considered equivalent. If this mapping's ID is non-null, it is\n\t * compared to the (possibly null) other mapping's ID via {@link Integer#compareTo(Integer)}.\n\t */\n\t@Override\n\tpublic int compareTo(Mapping o) {\n\t\tint i = (phraseNumber == null && o.phraseNumber == null) ? 0 : phraseNumber.compareTo(o.phraseNumber);\n\t\t\n\t\tif(i == 0) {\n\t\t\ti = a.compareTo(o.a);\n\t\t}\n\t\tif(i == 0) {\n\t\t\ti = b.compareTo(o.b);\n\t\t}\n\t\tif(i == 0) {\n\t\t\ti = status.compareTo(o.status);\n\t\t}\n\t\tif(i == 0) {\n\t\t\ti = confidence.compareTo(o.confidence);\n\t\t}\n\t\t\n\t\treturn i;\n\t}\n\n\t/**\n\t * Computes a hash code based on the ID, and A and B indices. The status and confidence fields are not\n\t * used for hash code computation because they are mutable. Because of this, two mappings may have\n\t * identical hash codes but not be equivalent according to {@link #equals(Object)}.\n\t */\n\t@Override\n\tpublic int hashCode() {\n\t\tint h = 37 * 1 + a + b;\n\t\t\n\t\treturn (phraseNumber == null) ? h : h + phraseNumber;\n\t}\n\n\t/**\n\t * Tests whether this mapping is equal to another.\n\t * @return true If this mapping's fields match the other according to the corresponding equals\n\t * methods. For the phrase number field, they are considered equal if both null\n\t * or if their corresponding equals method returns true, unequal otherwise.\n\t */\n\t@Override\n\tpublic boolean equals(Object obj) {\n\t\tif(obj instanceof Mapping) {\n\t\t\tMapping m = (Mapping)obj;\n\t\t\treturn ((phraseNumber == null && m.phraseNumber == null) || phraseNumber.equals(m.phraseNumber))\n\t\t\t\t&& a.equals(m.a) && b.equals(m.b) && status.equals(m.status)\n\t\t\t\t&& confidence.equals(m.confidence);\n\t\t}\n\t\t\n\t\treturn false;\n\t}\n\n\tvoid checkPhraseNumber(Integer phraseNumber) throws IndexOutOfBoundsException {\n\t\tif(phraseNumber != null && !Alignments.DEFAULT_PHRASE_NUMBER_BASE.isValidIndex(phraseNumber)) {\n\t\t\tthrow new IndexOutOfBoundsException(\"invalid phrase number: \" + phraseNumber);\n\t\t}\n\t}\n\t\n\tvoid checkIndex(Integer index) throws IndexOutOfBoundsException {\n\t\tif(index == null) {\n\t\t\tthrow new IllegalArgumentException(\"null index\");\n\t\t}\n\t\t\n\t\tif(!Alignments.DEFAULT_INDEX_BASE.isValidIndex(index)) {\n\t\t\tthrow new IndexOutOfBoundsException(\"invalid index: \" + index);\n\t\t}\n\t}\n\t\n\tvoid checkField(Object obj, String name) throws IllegalArgumentException {\n\t\tif(obj == null) {\n\t\t\tthrow new IllegalArgumentException(\"null \" + name);\n\t\t}\n\t}\n\n\t/**\n\t * A string representation of this mapping's indices.\n\t * @return For a mapping with a {@linkplain #getA() A} index of 3 and a \n\t * {@linkplain #getB() B} index of 6, this method gives the string\n\t * "3 <-> 6".\n\t */\n\t@Override\n\tpublic String toString() {\n\t\tStringBuilder sb = new StringBuilder(a.toString());\n\t\tsb.append(\" <-> \");\n\t\tsb.append(b.toString());\n\t\t\n\t\treturn sb.toString();\n\t}\n}\n" }, { "path": "src/opennlp/ccg/alignment/MappingFormat.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\nimport static opennlp.ccg.alignment.Alignments.DEFAULT_CONFIDENCE;\nimport static opennlp.ccg.alignment.Alignments.DEFAULT_INDEX_BASE;\nimport static opennlp.ccg.alignment.Alignments.DEFAULT_PHRASE_NUMBER_BASE;\nimport static opennlp.ccg.alignment.Alignments.DEFAULT_STATUS;\nimport static opennlp.ccg.alignment.Alignments.DEFAULT_STRICTNESS;\nimport static opennlp.ccg.alignment.MappingFormat.Field.A_INDEX_FIELD;\nimport static opennlp.ccg.alignment.MappingFormat.Field.B_INDEX_FIELD;\nimport static opennlp.ccg.alignment.MappingFormat.Field.CONFIDENCE_FIELD;\nimport static opennlp.ccg.alignment.MappingFormat.Field.PHRASE_NUMBER_FIELD;\nimport static opennlp.ccg.alignment.MappingFormat.Field.STATUS_FIELD;\n\nimport java.text.FieldPosition;\nimport java.text.Format;\nimport java.text.ParseException;\nimport java.text.ParsePosition;\nimport java.util.HashMap;\nimport java.util.HashSet;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.regex.Matcher;\nimport java.util.regex.Pattern;\n\n/**\n * A class for formatting mappings according to an {@linkplain EncodingScheme encoding scheme} and a specified\n * set of {@linkplain Field formatting fields}. This class\n * extends {@link Format} so that it fits in with the Java text parsing API.\n *

\n * Care is always taken to translate a mapping's phrase number and indices to the target\n * encoding scheme's {@linkplain EncodingScheme#getIndexBase() index base}.\n * If a mapping format is not {@linkplain #isStrict() strict}, parsing is handled robustly in that it tolerates\n * fields that are optional and may not always be specified. On output, a non-strict mapping format will only\n * include fields that either (1) do not have a {@linkplain Field#hasDefaultValue() default value}, or (2)\n * have a value different from {@linkplain Field#getDefaultValue() the default}. Strict mapping\n * formats, on the other hand, always expect and generate all and only the {@linkplain #getFields() fields\n * specified}.\n *

\n * Instances of mapping formats can be obtained by calling one of the\n * getInstance(...) methods.\n * \n * @see #getFields()\n * @see #isStrict()\n * @see EncodingScheme\n * @see IndexBase\n * @author Scott Martin\n */\npublic class MappingFormat extends Format {\n\tprivate static final long serialVersionUID = 1L;\n\t\n\tfinal EncodingScheme encodingScheme;\n\tfinal Set fields;\n\tfinal Pattern mappingPattern, fieldPattern;\n\tfinal boolean strict;\n\t\n\tprivate static Map> formatCache; \n\t\n\t/**\n\t * Marks mapping fields like ID, first index, second index, etc. Fields can have a \n\t * {@linkplain #getDefaultValue() default value}, which may be used depending on whether this format is\n\t * {@linkplain MappingFormat#isStrict() strict}.\n\t * @author Scott Martin\n\t */\n\tpublic static class Field extends java.text.Format.Field {\n\t\tprivate static final long serialVersionUID = 1L;\n\t\t\n\t\tObject defaultValue = null;\n\t\t\n\t\t/**\n\t\t * Field representing a mapping's ID. \n\t\t */\n\t\tpublic static final Field PHRASE_NUMBER_FIELD = new Field(\"PHRASE_NUMBER\", null);\n\t\t\n\t\t/**\n\t\t * Field representing a mapping's A index. \n\t\t */\n\t\tpublic static final Field A_INDEX_FIELD = new Field(\"A_INDEX\", null);\n\t\t\t\n\t\t/**\n\t\t * Field representing a mapping's B index. \n\t\t */\n\t\tpublic static final Field B_INDEX_FIELD = new Field(\"B_INDEX\", null);\n\t\t\t\n\t\t/**\n\t\t * Field representing a mapping's status field. \n\t\t */\n\t\tpublic static final Field STATUS_FIELD = new Field(\"STATUS\", DEFAULT_STATUS);\n\t\t\t\n\t\t/**\n\t\t * Field representing a mapping's confidence field. \n\t\t */\n\t\tpublic static final Field CONFIDENCE_FIELD = new Field(\"CONFIDENCE\", DEFAULT_CONFIDENCE);\n\t\t\n\t\t/**\n\t\t * Creates a new mapping format field.\n\t\t * @param name The new field's name.\n\t\t * @param defaultValue The default value for this field. If this field does not have a default value,\n\t\t * null is specified.\n\t\t * @throws IllegalArgumentException if name is null.\n\t\t */\n\t\tprotected Field(String name, Object defaultValue) {\n\t\t\tsuper(name);\n\t\t\t\n\t\t\tif(name == null) {\n\t\t\t\tthrow new IllegalArgumentException(\"name is null\");\n\t\t\t}\n\t\t\t\n\t\t\tthis.defaultValue = defaultValue;\n\t\t}\n\n\t\t/**\n\t\t * Tests whether this field has a default value.\n\t\t * @return true If the {@linkplain #getDefaultValue() default value} is non-null.\n\t\t */\n\t\tpublic boolean hasDefaultValue() {\n\t\t\treturn defaultValue != null;\n\t\t}\n\t\t\n\t\t/**\n\t\t * Gets the default value for this field, if any.\n\t\t * @see #hasDefaultValue()\n\t\t */\n\t\tpublic Object getDefaultValue() {\n\t\t\treturn defaultValue;\n\t\t}\n\n\t\t/**\n\t\t * Overrides the superclass method to return the value of {@link java.text.Format.Field#getName()}.\n\t\t */\n\t\t@Override\n\t\tpublic String toString() {\n\t\t\treturn getName();\n\t\t}\n\t}\n\t\n\t/**\n\t * Creates a mapping format based on the given encoding scheme and fields. The boolean flag tells instances\n\t * whether or not to output values when they are the default, or expect them during parsing.\n\t * \n\t * @param scheme The encoding scheme to create a formatter/parser for.\n\t * @param fields The fields to include.\n\t * @param strict Whether or not to use/expect default values in formatting and parsing.\n\t * @throws IllegalArgumentException If scheme or fields is null,\n\t * or if the specified set of fields does not contain all of the\n\t * {@linkplain EncodingScheme#getRequired() required fields} of the specified scheme.\n\t * @see Field#hasDefaultValue()\n\t */\n\tprotected MappingFormat(EncodingScheme scheme, Set fields, boolean strict) {\n\t\tif(scheme == null) {\n\t\t\tthrow new IllegalArgumentException(\"encoding scheme is null\");\n\t\t}\n\t\tif(fields == null) {\n\t\t\tthrow new IllegalArgumentException(\"fields is null\");\n\t\t}\n\t\t\n\t\tthis.encodingScheme = scheme;\n\t\tthis.fields = fields;\n\t\tthis.strict = strict;\n\t\t\n\t\tif(!fields.containsAll(encodingScheme.getRequired())) {\n\t\t\tthrow new IllegalArgumentException(\"specified fields does not contain all required fields\");\n\t\t}\n\t\tif(!encodingScheme.getOrder().containsAll(fields)) {\n\t\t\tthrow new IllegalArgumentException(\"encoding scheme does not use all the specified fields\");\n\t\t}\n\t\t\n\t\tfieldPattern = Pattern.compile(scheme.getFieldDelimiter().toString());\n\t\tmappingPattern = Pattern.compile(\"([\\\\w\\\\.]+\" + scheme.getFieldDelimiter() + \")+[\\\\w\\\\.]+\");\n\t}\n\t\n\t/**\n\t * Gets an instance of a mapping formatter/parser for a given encoding scheme with the scheme's\n\t * {@linkplain EncodingScheme#getDefaults() default fields} as the specified fields.\n\t * @see #getInstance(EncodingScheme, Set)\n\t */\n\tpublic static MappingFormat getInstance(EncodingScheme scheme) {\n\t\treturn getInstance(scheme, scheme.getDefaults());\n\t}\n\t\n\t/**\n\t * Gets an instance of a mapping formatter/parser for a given encoding scheme and field set.\n\t * @see #getInstance(EncodingScheme, Set, boolean)\n\t * @see Alignments#DEFAULT_STRICTNESS\n\t */\n\tpublic static MappingFormat getInstance(EncodingScheme scheme, Set fields) {\n\t\treturn getInstance(scheme, fields, DEFAULT_STRICTNESS);\n\t}\n\t\n\t/**\n\t * Gets an instance of a mapping formatter/parser for a given encoding scheme and field set. The\n\t * returned instances are cached to avoid creating multiple copies with the same scheme, fields, and\n\t * strictness flag. Cache access is synchronized to avoid threading issues.\n\t * \n\t * @param scheme The scheme to create a formatter/parser for.\n\t * @param fields The fields to include in the parser/formatter.\n\t * @param strict Whether the returned format should be {@linkplain #isStrict() strict}.\n\t * @return A new formatter/parser for mappings that will expect mappings encoded per the specified\n\t * scheme and will format mappings to strings of that encoding scheme.\n\t */\n\tpublic static MappingFormat getInstance(EncodingScheme scheme, Set fields, boolean strict) {\n\t\tif(scheme == null) { // test for this here so null isn't added to cache as a key\n\t\t\tthrow new IllegalArgumentException(\"encoding scheme is null\");\n\t\t}\n\t\t\n\t\tsynchronized(MappingFormat.class) {\n\t\t\tMappingFormat mf = null;\n\t\t\tSet fs = null;\n\t\t\t\n\t\t\tif(formatCache == null) {\n\t\t\t\tformatCache = new HashMap>();\n\t\t\t}\n\t\t\telse {\n\t\t\t\tfs = formatCache.get(scheme);\n\t\t\t}\t\t \n\t\t\t\n\t\t\tif(fs == null) {\n\t\t\t\tfs = new HashSet();\n\t\t\t\tformatCache.put(scheme, fs);\n\t\t\t}\n\t\t\t\n\t\t\tfor(MappingFormat f : fs) {\n\t\t\t\tif(f.strict == strict && f.fields.equals(fields)) {\n\t\t\t\t\tmf = f;\n\t\t\t\t\tbreak;\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tif(mf == null) {\n\t\t\t\tmf = new MappingFormat(scheme, fields, strict);\n\t\t\t\tfs.add(mf);\n\t\t\t}\n\t\t\t\n\t\t\treturn mf;\n\t\t}\n\t}\n\t\n\t/**\n\t * Gets the encoding scheme used by this mapping formatter/parser.\n\t * @return The encoding scheme used to create this instance.\n\t * @see #getInstance(EncodingScheme)\n\t */\n\tpublic EncodingScheme getEncodingScheme() {\n\t\treturn encodingScheme;\n\t}\n\n\t/**\n\t * Gets the field set used by this mapping formatter/parser.\n\t * @see #getInstance(EncodingScheme)\n\t */\n\tpublic Set getFields() {\n\t\treturn fields;\n\t}\n\t\n\t/**\n\t * Tests whether this mapping format will output default values, or expect them during parsing. If\n\t * true, this format will output and parse for every specified field. Otherwise, this format\n\t * will only write/expect fields that have a default value if their value differs from the default.\n\t * @see Field#hasDefaultValue()\n\t */\n\tpublic boolean isStrict() {\n\t\treturn strict;\n\t}\n\n\t/**\n\t * Formats a mapping according to the {@linkplain #getEncodingScheme() encoding scheme in effect}.\n\t * @param mapping The mapping to format.\n\t * @return A string in the format required by this formatter/parser's {@link #getEncodingScheme()}.\n\t * @see #format(Mapping, StringBuffer, FieldPosition) \n\t */\n\tpublic String formatMapping(Mapping mapping) {\n\t\tList order = encodingScheme.getOrder(); \n\t\tField field = null;\n\t\tint index = -1;\n\t\t\n\t\tfor(int i = 0; i < order.size(); i++) {\n\t\t\tField f = order.get(i);\n\t\t\tif(fields.contains(f)) {\n\t\t\t\tfield = f;\n\t\t\t\tindex = i;\n\t\t\t\tbreak;\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn format(mapping, new StringBuffer(), new FieldPosition(field, index)).toString();\n\t}\n\t\n\t/**\n\t * Overrides the {@link Format#format(Object, StringBuffer, FieldPosition)} method to make sure the\n\t * specified Object obj is an instance of {@link Mapping}.\n\t * @see #format(Mapping, StringBuffer, FieldPosition)\n\t */\n\t@Override\n\tpublic StringBuffer format(Object obj, StringBuffer toAppendTo, FieldPosition pos) {\n\t\tif(!(obj instanceof Mapping)) {\n\t\t\tthrow new IllegalArgumentException(\"not a mapping: \" + obj);\n\t\t}\n\t\tif(pos.getFieldAttribute() == null) {\n\t\t\tint i = pos.getField();\n\t\t\tpos = new FieldPosition(fieldAtIndex(i), i);\n\t\t}\n\t\t\n\t\treturn format((Mapping)obj, toAppendTo, pos);\n\t}\n\t\n\t/**\n\t * Formats a given {@link Mapping} based on the specified field position, appending the\n\t * output to the string buffer provided. \n\t * @param mapping The mapping to format.\n\t * @param toAppendTo The string buffer to append to.\n\t * @param pos The field position to use.\n\t * @return A string buffer with the information from mapping corresponding to the\n\t * field position pos appended to the one provided.\n\t * @throws IllegalArgumentException For the following reasons:\n\t *

    \n\t * \t
  • An attempt is made to format an alignment with a {@link IndexBase#nullValue null index}.
    • \n\t * \t
  • The value of pos.getFieldAttribute() is not an instance of {@link MappingFormat.Field}.\n\t * \t
  • The encoding scheme does not contain the field specified by pos.
    • \n\t * \t
  • The specified {@linkplain #getFields() list of fields} does not contain the field specified by\n\t * \t\tpos.\n\t * \t
  • Either a field required by the encoding scheme or the field specified by pos\n\t * \t\thas a null value.\n\t *
\n\t */\n\tpublic StringBuffer format(Mapping mapping, StringBuffer toAppendTo, FieldPosition pos) {\n\t\tif(mapping.a.equals(Alignments.DEFAULT_INDEX_BASE.nullValue)\n\t\t\t\t|| mapping.b.equals(Alignments.DEFAULT_INDEX_BASE.nullValue)) {\n\t\t\tthrow new IllegalArgumentException(\"cannot format null mapping: \" + mapping);\n\t\t}\n\t\t\n\t\tjava.text.Format.Field f = pos.getFieldAttribute();\n\t\tif(!(f instanceof MappingFormat.Field)) {\n\t\t\tint i = pos.getField();\n\t\t\tpos = new FieldPosition(fieldAtIndex(i), i);\n\t\t}\n\t\t\n\t\tField field = (MappingFormat.Field)f;\n\t\t\n\t\tif(!encodingScheme.getOrder().contains(field)) {\n\t\t\tthrow new IllegalArgumentException(\"no such field \\'\" + field + \"\\' in scheme \" + encodingScheme);\n\t\t}\n\t\tif(!fields.contains(field)) {\n\t\t\tthrow new IllegalArgumentException(\"field \\'\" + field + \"\\' not specified for this format\");\n\t\t}\n\t\t\n\t\tint start = toAppendTo.length();\n\t\tfor(Field af : encodingScheme.getOrder()) {\n\t\t\tif(!fields.contains(af)) {\n\t\t\t\tcontinue;\n\t\t\t}\n\t\t\t\n\t\t\tObject val = null;\n\t\t\t\n\t\t\tif(af.equals(PHRASE_NUMBER_FIELD)) {\n\t\t\t\tval = mapping.phraseNumber;\t\t\t\t\n\t\t\t}\n\t\t\telse if(af.equals(A_INDEX_FIELD)) {\n\t\t\t\tval = mapping.a;\n\t\t\t}\n\t\t\telse if(af.equals(B_INDEX_FIELD)) {\n\t\t\t\tval = mapping.b;\n\t\t\t}\n\t\t\telse if(af.equals(STATUS_FIELD)) {\n\t\t\t\tval = (strict || encodingScheme.getRequired().contains(STATUS_FIELD)) ? mapping.status\n\t\t\t\t\t\t: (mapping.status == DEFAULT_STATUS) ? null : mapping.status;\n\t\t\t}\n\t\t\telse if(af.equals(CONFIDENCE_FIELD)) {\n\t\t\t\tval = (strict || encodingScheme.getRequired().contains(CONFIDENCE_FIELD)) ? mapping.confidence\n\t\t\t\t\t\t: (mapping.confidence.equals(DEFAULT_CONFIDENCE)) ? null : mapping.confidence;\n\t\t\t}\n\t\t\t\n\t\t\tif(val == null) { // skip nulls, but check\n\t\t\t\tif(encodingScheme.getRequired().contains(af)) {\n\t\t\t\t\tthrow new IllegalArgumentException(\"required field \" + af + \" contains null value\");\n\t\t\t\t}\n\t\t\t\tif(field.equals(af)) {\n\t\t\t\t\tthrow new IllegalArgumentException(\"specified field \" + field + \" contains null value\");\n\t\t\t\t} \n\t\t\t}\n\t\t\telse {\n\t\t\t\tif(af.equals(PHRASE_NUMBER_FIELD)\n\t\t\t\t\t\t|| af.equals(A_INDEX_FIELD) || af.equals(B_INDEX_FIELD)) { // translate indices?\n\t\t\t\t\tboolean pn = af.equals(PHRASE_NUMBER_FIELD);\n\t\t\t\t\tIndexBase mappingBase = pn ? encodingScheme.getPhraseNumberBase()\n\t\t\t\t\t\t\t\t: encodingScheme.getIndexBase(),\n\t\t\t\t\t\t\tdefaultBase = pn ? DEFAULT_PHRASE_NUMBER_BASE : DEFAULT_INDEX_BASE;\n\t\t\t\t\t\n\t\t\t\t\tval = defaultBase.translate((Integer)val, mappingBase);\n\t\t\t\t}\n\t\t\t\telse if(af.equals(STATUS_FIELD)) {\n\t\t\t\t\tval = ((Status)val).abbreviation;\n\t\t\t\t}\n\t\t\t\n\t\t\t\tif(start < toAppendTo.length()) {\n\t\t\t\t\ttoAppendTo.append(encodingScheme.getFieldDelimiter());\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tif(field.equals(af)) {\n\t\t\t\t\tpos.setBeginIndex(toAppendTo.length());\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\ttoAppendTo.append(val);\n\t\t\t\t\n\t\t\t\tif(field.equals(af)) {\n\t\t\t\t\tpos.setEndIndex(toAppendTo.length());\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn toAppendTo;\n\t}\n\t\n\tField fieldAtIndex(int i) throws IndexOutOfBoundsException {\n\t\tField f = encodingScheme.getOrder().get(i);\n\t\tif(f == null) {\n\t\t\tthrow new IndexOutOfBoundsException(\"no field at position \" + i);\n\t\t}\n\t\t\n\t\treturn f;\n\t}\n\t\n\t/**\n\t * Parses a {@link Mapping} from a given string, based on the {@linkplain #getEncodingScheme() encoding\n\t * scheme} in effect. \n\t * @param source The string to parse.\n\t * @return A mapping object representing the specified string.\n\t * @throws ParseException If the string is ill-formed according to this formatter/parser's\n\t * {@link #getEncodingScheme() encoding scheme}. The exception thrown will contain an\n\t * {@linkplain ParseException#getErrorOffset() error offset} reflecting the position in the string where\n\t * the parse error occurred, if possible. \n\t */\n\tpublic Mapping parseMapping(String source) throws ParseException {\n\t\tParsePosition pos = new ParsePosition(0);\n\t\tMapping m = (Mapping)parseObject(source, pos);\n\t\tif(pos.getErrorIndex() != -1) {\n\t\t\tthrow new ParseException(\"problem parsing input \\\"\" + source + \"\\\"\", pos.getErrorIndex());\n\t\t}\n\t\t\n\t\treturn m;\n\t}\n\t\n\t/**\n\t * Overrides the {@link Format#parseObject(String, ParsePosition)} method to return a mapping, parsing\n\t * from the specified {@link ParsePosition}.\n\t * @see #parseMapping(String) \n\t */\n\t@Override\n\tpublic Object parseObject(String source, ParsePosition pos) {\n\t\tif(pos == null) {\n\t\t\tthrow new NullPointerException(\"parse position is null\");\n\t\t}\n\t\t\n\t\tint index = pos.getIndex();\n\t\t\n\t\tMatcher matcher = mappingPattern.matcher(source);\n\t\tif(!matcher.matches()) {\n\t\t\tpos.setErrorIndex(index);\n\t\t\treturn null;\n\t\t}\t\t\n\t\tif(matcher.start() != index) {\n\t\t\tpos.setErrorIndex(index);\n\t\t\treturn null;\n\t\t}\n\t\t\n\t\tString[] chunks = fieldPattern.split(source);\n\t\tIterator oi = encodingScheme.getOrder().iterator();\n\t\tMap values\n\t\t\t= new HashMap(encodingScheme.getOrder().size());\n\t\t\n\t\tfor(int i = 0; i < chunks.length; i++) {\n\t\t\tString c = chunks[i];\n\t\t\t\n\t\t\tif(c.length() == 0 || !oi.hasNext()) {\n\t\t\t\tpos.setErrorIndex(index);\n\t\t\t\treturn null;\n\t\t\t}\n\t\t\t\n\t\t\tObject val;\n\t\t\tMappingFormat.Field af = null;\n\t\t\twhile(oi.hasNext()) {\n\t\t\t\taf = oi.next();\n\t\t\t\ttry {\n\t\t\t\t\tif(af.equals(PHRASE_NUMBER_FIELD)\n\t\t\t\t\t\t\t|| af.equals(A_INDEX_FIELD) || af.equals(B_INDEX_FIELD)) {\n\t\t\t\t\t\tboolean pn = af.equals(PHRASE_NUMBER_FIELD);\n\t\t\t\t\t\tIndexBase mappingBase = pn ? encodingScheme.getPhraseNumberBase()\n\t\t\t\t\t\t\t\t\t: encodingScheme.getIndexBase(),\n\t\t\t\t\t\t\t\tdefaultBase = pn ? DEFAULT_PHRASE_NUMBER_BASE : DEFAULT_INDEX_BASE;\n\t\t\t\t\t\t\n\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\tval = mappingBase.translate(Integer.valueOf(c), defaultBase);\n\t\t\t\t\t\t}\n\t\t\t\t\t\tcatch(IllegalArgumentException iie) { // thrown by IndexBase.translate()\n\t\t\t\t\t\t\tpos.setErrorIndex(index);\n\t\t\t\t\t\t\treturn null;\n\t\t\t\t\t\t}\n\t\t\t\t\t\t\n\t\t\t\t\t\t// can't have null value\n\t\t\t\t\t\tif(((Integer)val).equals(defaultBase.nullValue)) {\n\t\t\t\t\t\t\tpos.setErrorIndex(index);\n\t\t\t\t\t\t\treturn null;\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\telse if(af.equals(STATUS_FIELD)) {\n\t\t\t\t\t\tval = Status.forAbbreviation(c);\n\t\t\t\t\t}\n\t\t\t\t\telse if(af.equals(CONFIDENCE_FIELD)) {\n\t\t\t\t\t\tval = Double.valueOf(c);\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tval = null;\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tcatch(NumberFormatException e) {\n\t\t\t\t\tpos.setErrorIndex(index);\n\t\t\t\t\treturn null;\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tif(val == null) {\n\t\t\t\t\tif(encodingScheme.getRequired().contains(af) || (strict && fields.contains(af))) {\n\t\t\t\t\t\tpos.setErrorIndex(index);\n\t\t\t\t\t\treturn null;\n\t\t\t\t\t}\n\t\t\t\t\t\n\t\t\t\t\tcontinue; // keep going if not required\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tvalues.put(af, val);\n\t\t\t\tbreak;\n\t\t\t}\n\t\t\t\n\t\t\t// update parse index\n\t\t\tindex += c.length();\n\t\t\tif(i < chunks.length - 1) {\n\t\t\t\tindex++; // add one for delimiter\n\t\t\t}\n\t\t}\n\t\t\n\t\tpos.setIndex(matcher.end());\n\t\t\n\t\tSet keys = values.keySet();\n\t\tif(!keys.containsAll(encodingScheme.getRequired()) || (strict && !keys.containsAll(fields))) {\n\t\t\tpos.setErrorIndex(index);\n\t\t\treturn null;\n\t\t}\n\t\t\n\t\tInteger id = values.containsKey(PHRASE_NUMBER_FIELD) ? (Integer)values.get(PHRASE_NUMBER_FIELD) : null;\n\t\t\n\t\tInteger first = values.containsKey(A_INDEX_FIELD) ? (Integer)values.get(A_INDEX_FIELD) : null,\n\t\t\tsecond = values.containsKey(B_INDEX_FIELD) ? (Integer)values.get(B_INDEX_FIELD) : null;\n\t\t\t\t\n\t\tStatus status = (Status)values.get(STATUS_FIELD);\n\t\tif(status == null && STATUS_FIELD.hasDefaultValue()) {\n\t\t\tstatus = (Status)STATUS_FIELD.defaultValue;\n\t\t}\n\t\t\n\t\tDouble confidence = (Double)values.get(CONFIDENCE_FIELD);\n\t\tif(confidence == null && CONFIDENCE_FIELD.hasDefaultValue()) {\n\t\t\tconfidence = (Double)CONFIDENCE_FIELD.defaultValue;\n\t\t}\n\t\t\n\t\treturn new Mapping(id, first, second, status, confidence);\n\t}\n}\n" }, { "path": "src/opennlp/ccg/alignment/MappingGroup.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\n\n/**\n * Class for representing groups of mappings during reading or writing.\n * Mapping groups encapsulate a phrase number (usually associated with a\n * phrase's line number) and a number of mappings in the group (its\n * {@linkplain #getLength() length}).\n * \n * @author Scott Martin\n */\npublic class MappingGroup implements Comparable {\n\n\tInteger phraseNumber;\n\tint length;\n\t\n\t/**\n\t * Creates a new mapping group with the given phrase number and length.\n\t * @throws IllegalArgumentException if phraseNumber is null, or if\n\t * length < 0.\n\t */\n\tpublic MappingGroup(Integer phraseNumber, int length) {\n\t\tif(phraseNumber == null) {\n\t\t\tthrow new IllegalArgumentException(\"phraseNumber is null\");\n\t\t}\n\t\tif(length < 0) {\n\t\t\tthrow new IllegalArgumentException(\"length < 0: \" + length);\n\t\t}\n\t\t\n\t\tthis.phraseNumber = phraseNumber;\n\t\tthis.length = length;\n\t}\n\t\n\t/**\n\t * Gets this group's phrase number.\n\t */\n\tpublic Integer getPhraseNumber() {\n\t\treturn phraseNumber;\n\t}\n\n\t/**\n\t * Gets the length of this mapping group.\n\t */\n\tpublic int getLength() {\n\t\treturn length;\n\t}\n\n\t/**\n\t * Tests whether this group is equal to another by comparing the two groups'\n\t * phrase numbers and lengths.\n\t */\n\t@Override\n\tpublic boolean equals(Object obj) {\n\t\tif(obj instanceof MappingGroup) {\n\t\t\tMappingGroup ag = (MappingGroup)obj;\n\t\t\treturn phraseNumber.equals(ag.phraseNumber) && length == ag.length;\n\t\t}\n\t\t\n\t\treturn false;\n\t}\n\n\t/**\n\t * Compares this mapping group to another by comparing their IDs and lengths.\n\t */\n\tpublic int compareTo(MappingGroup ag) {\n\t\tint i = phraseNumber.compareTo(ag.phraseNumber);\n\t\t\n\t\tif(i == 0) {\n\t\t\ti = (length == ag.length) ? 0 : length < ag.length ? -1 : 1;\n\t\t}\n\t\t\n\t\treturn i;\n\t}\n\n\t/**\n\t * Computes a hash code for this mapping group based on its ID and length.\n\t */\n\t@Override\n\tpublic int hashCode() {\n\t\t// brackets are to guard against 0\n\t\treturn 37 * (1 + phraseNumber.intValue() + length);\n\t}\n\n\t/**\n\t * Gets a string representation of this group.\n\t * @return For a group with phrase number 37 and length \n\t * 12, the string "Group 37 (12 mappings)". \n\t */\n\t@Override\n\tpublic String toString() {\n\t\tStringBuilder sb = new StringBuilder(\"Group \");\n\t\tsb.append(phraseNumber);\n\t\tsb.append(\" (\");\n\t\tsb.append(length);\n\t\tsb.append(\" mappings)\");\n\t\t\n\t\treturn sb.toString();\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/alignment/MappingReader.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\nimport java.io.FilterReader;\nimport java.io.IOException;\nimport java.io.Reader;\nimport java.text.ParseException;\nimport java.util.LinkedList;\nimport java.util.Queue;\n\n\n/**\n * Class for reading {@link Mapping}s from an underlying reader based on a specified {@link MappingFormat}.\n *

\n * Instances of this class read mappings group by group, with {@link #nextGroup()} called after reading the\n * correct number of mappings for the current group, as signaled by the {@link #canRead()} method.\n *

\n * The following code fragment illustrates the usage of this class:\n * 

\n * MappingReader reader = ...;\n * \n * MappingGroup g;\n * while((g = reader.getNextGroup()) != null) {\n * \twhile(reader.canRead()) {\n * \t\tMapping m = reader.readMapping();\n * \t\t...\n * \t}\n * }
\n * @author Scott Martin\n * @see MappingFormat\n */\npublic class MappingReader extends FilterReader {\n\n\tfinal MappingFormat format;\n\t\n\tprivate MappingGroup currentGroup;\t\n\tprivate Queue mappingQueue;\n\tprivate int mappingCount = 0;\n\tprivate boolean skipLF = false;\n\t\n\t/**\n\t * Creates a mapping reader.\n\t * @param r The underlying reader.\n\t * @param format The mapping format to use for reading {@link Mapping}s.\n\t * @throws IllegalArgumentException if format is null.\n\t */\n\tpublic MappingReader(Reader r, MappingFormat format) {\n\t\tsuper(r);\n\t\t\n\t\tif(format == null) {\n\t\t\tthrow new IllegalArgumentException(\"format is null\");\n\t\t}\n\t\t\n\t\tthis.format = format;\n\t\tmappingQueue = new LinkedList();\t\t\n\t}\n\t\n\t/**\n\t * @return The format used to read mappings.\n\t */\n\tpublic MappingFormat getFormat() {\n\t\treturn format;\n\t}\n\t\n\t/**\n\t * Starts reading from the next mapping group.\n\t * @return The next {@link MappingGroup} found by reading from the underlying reader.\n\t * @throws IOException If a {@link ParseException} is encountered when calling\n\t * {@link MappingFormat#parseMapping(String)} based on the underlying input, or if one is thrown by the\n\t * underlying reader. An IOException is also thrown if the number of mappings in the\n\t * {@linkplain MappingGroup#getLength() current group} could not be read. \n\t */\n\tpublic MappingGroup nextGroup() throws IOException {\n\t\tcheckMappingCount();\n\t\tmappingCount = 0;\n\t\t\n\t\tMappingGroup previous = (currentGroup == null) ? null : currentGroup;\n\t\tint newCount = mappingQueue.size();\n\t\t\n\t\tcurrentGroup = (newCount == 0)\n\t\t\t? null : new MappingGroup(mappingQueue.peek().phraseNumber, newCount);\n\t\t\n\t\tboolean eog = false;\n\t\t\n\t\twhile(!eog) {\n\t\t\tStringBuilder sb = new StringBuilder();\n\t\t\t\n\t\t\tint i;\n\t\t\twhile((i = in.read()) != -1) {\n\t\t\t\tchar c = (char)i;\n\t\t\t\t\n\t\t\t\tif(skipLF) {\n\t\t\t\t\tskipLF = false;\n\t\t\t\t\tif(c == '\\n') {\n\t\t\t\t\t\tcontinue;\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tif(c == '\\r') {\n\t\t\t\t\tskipLF = true;\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tif(format.encodingScheme.isMappingDelimiter(c)) {\n\t\t\t\t\tbreak;\n\t\t\t\t}\n\t\t\t\telse if(format.encodingScheme.isGroupDelimiter(c)) {\n\t\t\t\t\teog = true;\n\t\t\t\t\tbreak;\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tsb.append(c); \n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tif(sb.length() == 0) {\n\t\t\t\tbreak; // for EOF and end of group\n\t\t\t}\n\t\t\t\n\t\t\tMapping a = null;\n\t\t\ttry {\n\t\t\t\ta = format.parseMapping(sb.toString());\n\t\t\t}\n\t\t\tcatch(ParseException pe) {\n\t\t\t\tthrow new IOException(((currentGroup == null) ? \"\"\n\t\t\t\t\t\t: \"group \" + currentGroup.phraseNumber + \": \") + \"problem formatting mapping \"\n\t\t\t\t\t\t+ sb.toString() + \" at offset \" + pe.getErrorOffset() + \": \" + pe.getMessage(), pe);\n\t\t\t}\n\t\t\t\n\t\t\t// if the format allows null IDs, use previous's running counter\n\t\t\tif(currentGroup == null) {\n\t\t\t\tInteger I = (a.phraseNumber == null)\n\t\t\t\t\t? (previous == null) ? format.encodingScheme.getPhraseNumberBase().start\n\t\t\t\t\t\t\t: previous.phraseNumber + 1\n\t\t\t\t\t: a.phraseNumber;\n\t\t\t\t\n\t\t\t\tcurrentGroup = new MappingGroup(I, 0);\n\t\t\t}\n\t\t\t\n\t\t\tif(a.phraseNumber == null) {\n\t\t\t\t// have to copy because phraseNumber is immutable (and final)\n\t\t\t\ta = a.copyWithPhraseNumber(currentGroup.phraseNumber);\n\t\t\t}\n\t\t\t\n\t\t\tif(!currentGroup.phraseNumber.equals(a.phraseNumber)) {\n\t\t\t\teog = true;\n\t\t\t}\n\t\t\telse {\n\t\t\t\tnewCount++; // only increment if should be read\n\t\t\t}\t\t\t\n\t\t\t\n\t\t\tif(!mappingQueue.offer(a)) { // save for next read\n\t\t\t\tthrow new IOException(\"unable to read mapping\");\n\t\t\t}\n\t\t}\n\t\t\n\t\tif(currentGroup != null) {\n\t\t\tcurrentGroup.length = newCount;\n\t\t}\n\t\t\n\t\treturn (currentGroup == null || currentGroup.length == 0) ? null : currentGroup;\n\t}\n\t\n\t/**\n\t * Tests whether mappings can be read from this reader without throwing an {@link IOException}.\n\t * @return true If there is a current mapping group and mappings remain to be read from it.\n\t * @see #nextGroup()\n\t * @see #readMapping()\n\t */\n\tpublic boolean canRead() {\n\t\treturn currentGroup != null && mappingCount < currentGroup.length;\n\t}\n\t\n\t/**\n\t * Overrides the superclass method to check first if any mappings are available.\n\t * @throws IOException If no mappings are available.\n\t * @see #canRead()\n\t */\n\t@Override\n\tpublic int read() throws IOException {\n\t\tcheckRead();\n\t\tint c = super.read();\n\t\tif(skipLF) {\n\t\t\tskipLF = false;\n\t\t\tif(c == '\\n') {\n\t\t\t\tc = super.read();\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn c;\n\t}\n\n\t/**\n\t * Overrides the superclass method to check first if any mappings are available.\n\t * @throws IOException If no mappings are available.\n\t * @see #canRead()\n\t */\n\t@Override\n\tpublic int read(char[] cbuf, int off, int len) throws IOException {\n\t\tcheckRead();\n\t\t\n\t\tif(len < 1) {\n\t\t\treturn 0;\n\t\t}\n\t\t\n\t\tif(skipLF) {\n\t\t\tint c = read();\n\t\t\tskipLF = false;\n\t\t\t\n\t\t\tif(c == -1) {\n\t\t\t\treturn c;\n\t\t\t}\n\t\t\telse if(c != '\\n') {\n\t\t\t\tcbuf[off++] = (char)c;\n\t\t\t\tlen--;\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn super.read(cbuf, off, len);\n\t}\n\n\t/**\n\t * Tests whether mappings can be read without blocking.\n\t * @return true If it is guaranteed that a call to {@link #readMapping()} will not block for input.\n\t */\n\t@Override\n\tpublic boolean ready() throws IOException {\n\t\treturn canRead();\n\t}\n\n\t/**\n\t * Reads a mapping from the underlying reader, if one is {@linkplain #canRead() available}.\n\t * @return A mapping formatted by the {@linkplain #getFormat() format in effect}.\n\t * @throws IOException If no mappings are available in the current group.\n\t * @see #canRead()\n\t */\n\tpublic Mapping readMapping() throws IOException {\n\t\tcheckRead();\n\t\t\n\t\tMapping a = mappingQueue.poll();\n\t\t\n\t\tif(a != null) {\n\t\t\tmappingCount++;\n\t\t}\n\t\t\n\t\treturn a;\n\t}\n\t\n\t/**\n\t * Closes this reader, checking first if the correct number of mappings were read.\n\t * @throws IOException If mappings remain to be read from the current group.\n\t */\n\t@Override\n\tpublic void close() throws IOException {\n\t\ttry {\n\t\t\tcheckMappingCount();\n\t\t}\n\t\tfinally {\n\t\t\tsuper.close();\n\t\t}\n\t}\n\t\n\tvoid checkRead() throws IOException {\n\t\tif(!canRead()) {\n\t\t\tthrow new IOException(\"no mappings available\");\n\t\t}\n\t}\n\n\tvoid checkMappingCount() throws IOException {\n\t\tif((currentGroup == null && mappingCount > 0) \n\t\t\t\t|| (currentGroup != null && mappingCount != currentGroup.length)) {\n\t\t\tthrow new IOException(\n\t\t\t\tcurrentGroup == null ? \"\" : \"group \" + currentGroup.phraseNumber + \": \"\n\t\t\t\t\t+ \"mapping count does not match: expected \"\n\t\t\t\t\t+ ((currentGroup == null) ? 0 : currentGroup.length)\n\t\t\t\t\t+ \", but was \" + mappingCount);\n\t\t}\n\t}\n}\n" }, { "path": "src/opennlp/ccg/alignment/MappingWriter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\nimport java.io.FilterWriter;\nimport java.io.IOException;\nimport java.io.Writer;\n\n/**\n * A writer for outputting {@link Mapping}s to a specified underlying {@link Writer}.\n *

\n * Mapping writers write mappings by group, so that before any mappings are written,\n * {@link #startGroup(MappingGroup)} must always be called\n * to signal to the writer that a new group is starting (and what its length is). The outgoing mappings are\n * formatted according to a {@link MappingFormat} specified at creation.\n *

\n * After a new group is started, exactly the {@linkplain MappingGroup#getLength() number of mappings} in that\n * group must be written. Otherwise, an {@link IOException} is thrown. An {@link IOException} is also thrown\n * if an attempt is made to write a mapping whose {@linkplain Mapping#getPhraseNumber() id} is different from the \n * current group's {@linkplain MappingGroup#getPhraseNumber() id}, or if {@link #writeMapping(Mapping)} is called without\n * first calling {@link #startGroup(MappingGroup)}.\n *

\n * Example usage:\n * 

\n * MappingWriter mw = ...;\n * \n * // while there are more groups\n * mw.startGroup(new MappingGroup(...));\n * while(mw.canWrite()) {\n * \tmw.writeMapping(...);\n * }\n * \n * mw.close();
\n * @author Scott Martin\n */\npublic class MappingWriter extends FilterWriter {\n\n\tfinal MappingFormat format;\n\t\n\tprivate MappingGroup currentGroup;\n\tprivate int mappingCount = 0;\n\t\n\tprivate String mappingDelimiter, groupDelimiter;\n\t\n\t/**\n\t * Creates a new mapping writer.\n\t * @param out The underlying writer.\n\t * @param format The mapping format to use.\n\t * @throws IllegalArgumentException if format is null.\n\t */\n\tpublic MappingWriter(Writer out, MappingFormat format) {\n\t\tsuper(out);\n\t\t\n\t\tif(format == null) {\n\t\t\tthrow new IllegalArgumentException(\"format is null\");\n\t\t}\n\t\t\n\t\tthis.format = format;\n\t}\n\t\t\n\t/**\n\t * Gets the mapping format used by this writer.\n\t */\n\tpublic MappingFormat getFormat() {\n\t\treturn format;\n\t}\n\t\n\t/**\n\t * Gets the current mapping group being written.\n\t */\n\tpublic MappingGroup getCurrentGroup() {\n\t\treturn currentGroup;\n\t}\n\t\n\tvoid checkWrite() throws IOException {\n\t\tif(!canWrite()) {\n\t\t\tthrow new IOException(\"unable to write\");\n\t\t}\n\t}\n\t\n\tvoid checkMappingCount() throws IOException {\n\t\tif(currentGroup != null && mappingCount != currentGroup.length) {\n\t\t\tthrow new IOException(\"incorrect mapping count for group \" +\n\t\t\t\t\t+ currentGroup.phraseNumber + \"; expected \"\n\t\t\t\t\t+ currentGroup.length + \", but was \" + mappingCount);\n\t\t}\n\t}\n\t\n\t/**\n\t * Starts a new mapping group for writing mappings. If {@link #endGroup()} was not called explicitly,\n\t * it is first called to end the current group.\n\t * @param mappingGroup The group to start.\n\t * @throws IOException If the number of mappings written since the last call to\n\t * {@link #startGroup(MappingGroup)}does not exactly equal the length of the\n\t * {@linkplain #getCurrentGroup() current group}.\n\t * @see #writeMapping(Mapping)\n\t * @see #endGroup()\n\t */\n\tpublic void startGroup(MappingGroup mappingGroup) throws IOException {\n\t\tif(currentGroup != null) {\n\t\t\tendGroup();\t\n\t\t}\n\t\t\n\t\tcurrentGroup = mappingGroup;\n\t}\n\t\n\t/**\n\t * Ends the current group, writing the {@linkplain EncodingScheme#getGroupDelimiter() proper group\n\t * delimiter} for the {@linkplain #getFormat() mapping format in effect}.\n\t * @throws IOException If {@link #startGroup(MappingGroup)} was not first called, or if the correct number\n\t * of mappings for the current group was not written.\n\t * @see #startGroup(MappingGroup)\n\t */\n\tpublic void endGroup() throws IOException {\n\t\tif(currentGroup == null) {\n\t\t\tthrow new IOException(\"no current group\");\n\t\t}\n\t\t\n\t\tcheckMappingCount();\n\t\t\n\t\tif(currentGroup.length > 0) {\n\t\t\tif(groupDelimiter == null) {\n\t\t\t\tCharacter gd = format.encodingScheme.getGroupDelimiter();\n\t\t\t\t\n\t\t\t\tgroupDelimiter = AbstractEncodingScheme.isLineSeparator(gd)\n\t\t\t\t\t\t? System.getProperty(\"line.separator\") : String.valueOf(gd);\n\t\t\t}\n\t\t\t\n\t\t\tout.write(groupDelimiter); // no empty lines\n\t\t}\n\t\t\n\t\tcurrentGroup = null;\n\t\tmappingCount = 0;\n\t}\n\t\n\t/**\n\t * Writes to the underlying writer, first checking if mappings can be written.\n\t * @see #canWrite()\n\t */\n\t@Override\n\tpublic void write(char[] cbuf, int off, int len) throws IOException {\n\t\tcheckWrite();\n\t\tsuper.write(cbuf, off, len);\n\t}\n\n\t/**\n\t * Writes to the underlying writer, first checking if mappings can be written.\n\t * @see #canWrite()\n\t */\n\t@Override\n\tpublic void write(int c) throws IOException {\n\t\tcheckWrite();\n\t\tsuper.write(c);\n\t}\n\n\t/**\n\t * Writes to the underlying writer, first checking if mappings can be written.\n\t * @see #canWrite()\n\t */\n\t@Override\n\tpublic void write(String str, int off, int len) throws IOException {\n\t\tcheckWrite();\n\t\tsuper.write(str, off, len);\n\t}\n\n\t/**\n\t * Writes to the underlying writer, first checking if mappings can be written.\n\t * @see #canWrite()\n\t */\n\t@Override\n\tpublic void write(char[] cbuf) throws IOException {\n\t\tcheckWrite();\n\t\tsuper.write(cbuf);\n\t}\n\n\t/**\n\t * Writes to the underlying writer, first checking if mappings can be written.\n\t * @see #canWrite()\n\t */\n\t@Override\n\tpublic void write(String str) throws IOException {\n\t\tcheckWrite();\n\t\tsuper.write(str);\n\t}\n\t\n\t/**\n\t * Tests whether the mapping writer is currently in a state in which mappings can be written without\n\t * throwing an {@link IOException}. Mappings\n\t * can only be written when a current group has been {@linkplain #startGroup(MappingGroup) started} and \n\t * the number of mappings written since the last group started is less than the total\n\t * {@linkplain MappingGroup#getLength() length} of the current group.\n\t * @return true If the {@linkplain #getCurrentGroup() current group} is non-null and the number of mappings\n\t * written to the current group is less than its length.\n\t */\n\tpublic boolean canWrite() {\n\t\treturn currentGroup != null && mappingCount < currentGroup.length; \n\t}\n\n\t/**\n\t * Writes a mapping to the underlying writer, formatted by {@linkplain #getFormat() the mapping format}. If\n\t * other mappings have been written since the last call to {@link #startGroup(MappingGroup)}, the\n\t * {@linkplain EncodingScheme#getMappingDelimiter() mapping delimiter} used by the current format is first\n\t * written.\n\t * @param mapping The mapping to write.\n\t * @throws IOException If {@link #canWrite()} returns false, if\n\t * {@link #startGroup(MappingGroup)} was not first called, or if an attempt is made\n\t * to write a mapping with an {@linkplain Mapping#getPhraseNumber() id} that does not equal the current group's\n\t * {@linkplain MappingGroup#getPhraseNumber() id}.\n\t */\n\tpublic void writeMapping(Mapping mapping) throws IOException {\n\t\tcheckWrite();\n\t\tif(mapping == null) {\n\t\t\tthrow new NullPointerException(\"null mapping\");\n\t\t}\t\t\n\t\tif(mapping.phraseNumber != null && !mapping.phraseNumber.equals(currentGroup.phraseNumber)) {\n\t\t\tthrow new IOException(\"mapping from group \" + mapping.phraseNumber\n\t\t\t\t+ \", but current group is \" + currentGroup.phraseNumber);\n\t\t}\n\t\t\n\t\tif(mappingCount > 0) {\n\t\t\tif(mappingDelimiter == null) {\n\t\t\t\tCharacter md = format.encodingScheme.getMappingDelimiter();\n\t\t\t\tmappingDelimiter = AbstractEncodingScheme.isLineSeparator(md)\n\t\t\t\t\t\t? System.getProperty(\"line.separator\") : String.valueOf(md);\n\t\t\t}\n\t\t\t\n\t\t\tout.write(mappingDelimiter);\n\t\t}\t\t\n\t\t\n\t\tout.write(format.formatMapping(mapping));\n\t\tmappingCount++;\n\t}\n\n\t/**\n\t * Overrides the superclass method to first check that the correct number of mappings were written.\n\t * @throws IOException If a number of mappings have been written that does not exactly equal the length \n\t * of the {@linkplain #getCurrentGroup() current group}.\n\t */\n\t@Override\n\tpublic void close() throws IOException {\n\t\ttry {\n\t\t\tcheckMappingCount();\n\t\t}\n\t\tfinally {\n\t\t\tsuper.close();\n\t\t}\n\t}\n}\n" }, { "path": "src/opennlp/ccg/alignment/MosesEncodingScheme.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.alignment;\n\nimport static opennlp.ccg.alignment.MappingFormat.Field.A_INDEX_FIELD;\nimport static opennlp.ccg.alignment.MappingFormat.Field.B_INDEX_FIELD;\nimport static opennlp.ccg.alignment.MappingFormat.Field.STATUS_FIELD;\n\nimport java.util.Arrays;\nimport java.util.HashSet;\n\n/**\n * Represents the Moses encoding.\n * \n * \t\n * \t\n * \t\n * \t\n * \t\n * \t\n *
Field separator-
Mapping separator(space)
Group separator(newline)
ID base{@link IndexBase#ZERO}
Index base{@link IndexBase#ZERO}
Example group0-1 2-3 2-4-P 3-0
\n * @see Moses Word Alignment Tutorial\n * \n * @author Scott Martin\n */\npublic class MosesEncodingScheme extends AbstractEncodingScheme {\n\t\n\t/**\n\t * Creates a new instance of the Moses encoding scheme.\n\t * @see Alignments#MOSES_ENCODING_SCHEME\n\t */\n\tpublic MosesEncodingScheme() {\n\t\tsuper('-', ' ', '\\n', Alignments.DEFAULT_INDEX_BASE, IndexBase.ZERO,\n\t\t\t\tnew HashSet(Arrays.asList(A_INDEX_FIELD, B_INDEX_FIELD, STATUS_FIELD)),\n\t\t\t\tnew HashSet(Arrays.asList(A_INDEX_FIELD, B_INDEX_FIELD)),\n\t\t\t\tA_INDEX_FIELD, B_INDEX_FIELD, STATUS_FIELD);\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/alignment/NAACLEncodingScheme.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.alignment;\n\nimport static opennlp.ccg.alignment.MappingFormat.Field.*;\n\nimport java.util.Arrays;\nimport java.util.HashSet;\n\n/**\n * Represents the NAACL shared task encoding.\n * \n * \t\n * \t\n * \t\n * \t\n * \t\n * \t\n *
Field separator(space)
Mapping separator(newline)
Group separator(newline)
ID base{@link IndexBase#ZERO}
Index base{@link IndexBase#ONE}
Example group\n * 
\n * 37 1 2 S\n * 37 3 4 S\n * 37 3 5 P\n * 37 4 1 S\n *
\n *
\n * @see NAACL shared task word alignment guidelines\n * @author Scott Martin\n */\npublic class NAACLEncodingScheme extends AbstractEncodingScheme {\n\t\n\t/**\n\t * Creates a new instance of the NAACL encoding scheme.\n\t * @see Alignments#NAACL_ENCODING_SCHEME\n\t */\n\tpublic NAACLEncodingScheme() {\n\t\tsuper(' ', '\\n', '\\n', IndexBase.ZERO, IndexBase.ONE,\n\t\t\tAlignments.NAACL_DEFAULT_FIELDS,\t\t\t\t\n\t\t\tnew HashSet(Arrays.asList(PHRASE_NUMBER_FIELD, A_INDEX_FIELD, B_INDEX_FIELD)),\t\t\t\t\n\t\t\tPHRASE_NUMBER_FIELD, A_INDEX_FIELD, B_INDEX_FIELD, STATUS_FIELD, CONFIDENCE_FIELD);\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/alignment/Phrase.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\nimport java.util.AbstractList;\nimport java.util.List;\n\n/**\n * A phrase, i.e., a sequence of words with an associated {@linkplain #getNumber() phrase number}.\n * Phrases may additionally have a {@linkplain #getId() string ID}.\n * To save access time and space, this class is backed by an array of\n * {@link String}s, but extends {@link AbstractList} so that phrases can be iterated over and have \n * all of the usual convenience methods.\n *

\n * Phrases are immutable once created, so calling any of the {@link List#add(Object)},\n * {@link List#remove(int)}, or {@link List#set(int, Object)}\n * methods will throw an {@link UnsupportedOperationException}. Similarly, the\n * {@linkplain #iterator() iterator's} remove() method also throws an\n * {@link UnsupportedOperationException}. \n *

\n * For convenience, this class implements the {@link Comparable} interface, comparing\n * phrases by their {@linkplain #getNumber() numbers}.\n * \n * @author Scott Martin\n */\npublic class Phrase extends AbstractList implements Comparable {\n\n\tfinal Integer number;\n\tfinal String id;\n\tfinal String[] words;\n\t\n\t/**\n\t * Creates a new phrase with the given number and list of words.\n\t * @see Phrase#Phrase(String, Integer, String...)\n\t */\n\tpublic Phrase(Integer number, List words) {\n\t\tthis(number, words.toArray(new String[words.size()]));\n\t}\n\t\n\t/**\n\t * Creates a new phrase with the specified number, made up of the given words.\n\t * @see Phrase#Phrase(String, Integer, String...)\n\t */\n\tpublic Phrase(Integer number, String... words) {\n\t\tthis(null, number, words);\n\t}\n\t\n\t/**\n\t * Creates a new phrase with the given number, ID, and list of words.\n\t * @see Phrase#Phrase(String, Integer, String...)\n\t */\n\tpublic Phrase(String id, Integer number, List words) {\n\t\tthis(id, number, words.toArray(new String[words.size()]));\n\t}\n\t\n\t/**\n\t * Creates a new phrase with the specified number and id, made up of the given words.\n\t * The ID may be null, but the cannot be null. The list of words cannot be\n\t * null or contain null members, although it can be empty.\n\t * @throws IllegalArgumentException If number, words is null,\n\t * or one of the strings in words is null.\n\t */\n\tpublic Phrase(String id, Integer number, String... words) {\n\t\tcheckObject(number, \"number\");\n\t\tcheckObject(words, \"words\");\n\t\t\n\t\tfor(int i = 0; i < words.length; i++) {\n\t\t\tcheckObject(words[i], \"word \" + i);\n\t\t}\n\t\t\n\t\tthis.number = number;\n\t\tthis.id = id;\n\t\tthis.words = words;\n\t}\n\n\tvoid checkObject(Object obj, String name) {\n\t\tif(obj == null) {\n\t\t\tthrow new IllegalArgumentException(name + \" is null\");\n\t\t}\n\t}\n\t\n\t/**\n\t * Gets this phrase's ID, if any was specified.\n\t * @return The ID of this phrase, possibly null.\n\t */\n\tpublic String getId() {\n\t\treturn id;\n\t}\n\n\t/**\n\t * @return This phrase's number.\n\t */\n\tpublic Integer getNumber() {\n\t\treturn number;\n\t}\n\n\t/**\n\t * Returns the word in this phrase at the supplied index.\n\t */\n\t@Override\n\tpublic String get(int index) {\n\t\treturn words[index];\n\t}\n\n\t/**\n\t * @return The number of words in this phrase.\n\t */\n\t@Override\n\tpublic int size() {\n\t\treturn words.length;\n\t}\n\n\t/**\n\t * Compares this phrase to another by comparing their {@linkplain #getNumber() numbers}.\n\t * @return The value of getNumber().compareTo(o.getNumber()).\n\t * @param o The phrase to compare to.\n\t * @see Integer#compareTo(Integer)\n\t */\n\t@Override\n\tpublic int compareTo(Phrase o) {\n\t\treturn getNumber().compareTo(o.getNumber());\n\t}\n\n\t/**\n\t * Tests whether this phrase is equal to another by first calling the superclass method \n\t * {@link AbstractList#equals(Object)}, then comparing this phrase's number and id to the other.\n\t */\n\t@Override\n\tpublic boolean equals(Object o) {\n\t\tif(o instanceof Phrase && super.equals(o)) {\n\t\t\tPhrase p = (Phrase)o;\n\t\t\treturn number.equals(p.number) && ((id == null && p.id == null) || id.equals(p.id));\n\t\t}\n\t\t\n\t\treturn false;\n\t}\n\n\t/**\n\t * Generates a hash code for this phrase based on the superclass hash code, its number, and its ID\n\t * (if any).\n\t */\n\t@Override\n\tpublic int hashCode() {\n\t\tint h = 31 * super.hashCode() + number.hashCode();\n\t\t\n\t\treturn (id == null) ? h : h + id.hashCode();\n\t}\n\n\t/**\n\t * Gets a string representation of this phrase.\n\t * @return For a phrase with number 37, ID phrase 3, and words\n\t * "Test phrase", prepends 37 (phrase 3): to the result of calling the\n\t * superclass method {@link AbstractList#toString()}; \n\t */\n\t@Override\n\tpublic String toString() {\n\t\tStringBuilder sb = new StringBuilder(number.toString());\n\t\t\n\t\tif(id != null) {\n\t\t\tsb.append(\" (\");\n\t\t\tsb.append(id);\n\t\t\tsb.append(')');\n\t\t}\n\t\t\n\t\tsb.append(\": \");\n\t\tsb.append(super.toString());\n\t\t\n\t\treturn sb.toString();\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/alignment/PhrasePosition.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\n/**\n * In an alignment, a phrase position indicates which of the two aligned phrases\n * is first (the "A" phrase) and which is second (the "B" phrase)\n * in terms of the alignment indices.\n * \n * @author Scott Martin\n */\npublic enum PhrasePosition {\n\t\n\t/**\n\t * The "A" position.\n\t */\n\tA,\n\t\n\t/**\n\t * The "B" position.\n\t */\n\tB;\n\t\n\t/**\n\t * Gives the opposite of this phrase position.\n\t * @return {@link #B} if this position is {@link #A}, otherwise \n\t * {@link #A}.\n\t */\n\tpublic PhrasePosition opposite() {\n\t\treturn (this == A) ? B : A;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/alignment/PhraseReader.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\nimport java.io.IOException;\nimport java.io.LineNumberReader;\nimport java.io.Reader;\n\n/**\n * A phrase reader just reads line numbers, but does not search for phrase IDs. The line number of each phrase\n * read is {@linkplain IndexBase#translate(Integer, IndexBase) translated} into the {@link IndexBase} provided\n * at creation.\n * @author Scott Martin\n */\npublic class PhraseReader extends LineNumberReader {\n\n\tfinal IndexBase numberBase;\n\t\n\t/**\n\t * The index base that starts numbering lines at 1.\n\t * @see IndexBase#ONE\n\t */\n\tpublic static final IndexBase LINE_NUMBER_BASE = IndexBase.ONE;\n\t\n\t/**\n\t * Creates a phrase reader from the specified underlying reader and word index base for phrase numbers.\n\t * @see #PhraseReader(Reader, IndexBase)\n\t * @see Alignments#DEFAULT_PHRASE_NUMBER_BASE\n\t */\n\tpublic PhraseReader(Reader in) {\n\t\tthis(in, Alignments.DEFAULT_PHRASE_NUMBER_BASE);\n\t}\n\t\n\t/**\n\t * Creates a phrase reader.\n\t * @param numberBase The target phrase numbering base. Phrases read from the underlying reader will have\n\t * their numbers translated from {@link #LINE_NUMBER_BASE the default} to numberBase.\n\t * @throws IllegalArgumentException if numberBase is null.\n\t * @see IndexBase#translate(Integer, IndexBase)\n\t * @see #readPhrase()\n\t */\n\tpublic PhraseReader(Reader in, IndexBase numberBase) {\n\t\tsuper(in);\n\t\t\n\t\tif(numberBase == null) {\n\t\t\tthrow new IllegalArgumentException(\"numberBase is null\");\n\t\t}\n\t\t\n\t\tthis.numberBase = numberBase;\n\t}\n\n\t/**\n\t * The target number base that new phrase IDs will have their line numbers translated into.\n\t * @see #readPhrase()\n\t */\n\tpublic IndexBase getNumberBase() {\n\t\treturn numberBase;\n\t}\n\t\n\t/**\n\t * Gets the number last assigned to a phrase, translated into the specified\n\t * {@linkplain #getNumberBase() number base}. Note that this method may return a different result than\n\t * {@link #getLineNumber()} due to the base translation.\n\t * @see IndexBase#translate(Integer, IndexBase)\n\t */\n\tpublic Integer getPhraseNumber() {\n\t\treturn LINE_NUMBER_BASE.translate(getLineNumber(), numberBase);\n\t}\n\n\t/**\n\t * Reads the next phrase from the underlying reader. The number is determined by translating the line number\n\t * of the phrase into the {@linkplain #getNumberBase() target number base}. The words in the phrase are\n\t * tokenized by the {@link Alignments#tokenize(String)} method.\n\t * @return null if no phrases can be read from the underlying reader.\n\t */\n\tpublic Phrase readPhrase() throws IOException {\n\t\tString ln = readLine();\n\t\treturn (ln == null) ? null : new Phrase(getPhraseNumber(), Alignments.tokenize(ln));\n\t}\n}\n" }, { "path": "src/opennlp/ccg/alignment/PhraseWriter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\nimport java.io.FilterWriter;\nimport java.io.IOException;\nimport java.io.PrintWriter;\nimport java.io.Writer;\n\n/**\n * A writer for phrases. This writer writes {@linkplain Phrase phrase} instances to the underlying writer with\n * the specified string as a word separator. No markup is placed around phrases that are written; only a line\n * separator is written after the phrase.\n * \n * @author Scott Martin\n * @see Alignments#untokenize(String[], String)\n */\npublic class PhraseWriter extends FilterWriter {\n\n\tfinal String wordSeparator;\n\t\n\tprotected PrintWriter printWriter;\n\t\n\t/**\n\t * Creates a new phrase writer with the default word separator.\n\t * @see #PhraseWriter(Writer, String)\n\t * @see Alignments#DEFAULT_WORD_SEPARATOR\n\t */\n\tpublic PhraseWriter(Writer out) {\n\t\tthis(out, Alignments.DEFAULT_WORD_SEPARATOR);\n\t}\n\n\t/**\n\t * Creates a new phrase writer for the underlying input stream that will use\n\t * the specified word separator when untokenizing phrases.\n\t * \n\t * @param wordSeparator The word separator to use when translating phrases into strings.\n\t * @throws IllegalArgumentException if wordSeparator is null.\n\t * @see Alignments#untokenize(String[], String) \n\t */\n\tpublic PhraseWriter(Writer out, String wordSeparator) {\n\t\tsuper(new PrintWriter(out));\n\t\t\n\t\tif(wordSeparator == null) {\n\t\t\tthrow new IllegalArgumentException(\"wordSeparator is null\");\n\t\t}\n\t\t\n\t\tthis.wordSeparator = wordSeparator;\n\t\tprintWriter = (PrintWriter)this.out;\n\t}\n\t\n\t/**\n\t * Gets the word separator that this phrase writer uses when writing phrases.\n\t */\n\tpublic String getWordSeparator() {\n\t\treturn wordSeparator;\n\t}\n\t\n\t/**\n\t * Writes a phrase by {@linkplain Alignments#untokenize(List, String) untokenizing} its words\n\t * according to the {@linkplain #getWordSeparator() word separator being used}. Before writing the\n\t * untokenized phrase, {@link #preWritePhrase(Phrase)} is called exactly once. After writing the phrase,\n\t * {@link #postWritePhrase(Phrase)} is called exactly once.\n\t * \n\t * @param phrase The phrase to write.\n\t * @throws IOException If the underlying writer throws an {@link IOException}, or if one is thrown by\n\t * either {@link #preWritePhrase(Phrase)} or {@link #postWritePhrase(Phrase)}.\n\t */\n\tpublic void writePhrase(Phrase phrase) throws IOException {\n\t\tpreWritePhrase(phrase);\n\t\tprintWriter.print(Alignments.untokenize(phrase, wordSeparator));\n\t\tpostWritePhrase(phrase);\n\t}\n\t\n\t/**\n\t * Called before {@link #writePhrase(Phrase)} (to be overridden by implementing classes).\n\t * @param phrase The phrase about to be written.\n\t */\n\tprotected void preWritePhrase(Phrase phrase) throws IOException {\n\t\t// default is to do nothing\n\t}\n\t\n\t/**\n\t * Called after {@link #writePhrase(Phrase)} (to be overridden by implementing classes). This implementation\n\t * just writes a line separator.\n\t * @param phrase The phrase that was just written.\n\t */\n\tprotected void postWritePhrase(Phrase phrase) throws IOException {\n\t\tprintWriter.println();\n\t}\n}\n" }, { "path": "src/opennlp/ccg/alignment/Status.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.alignment;\n\n/**\n * A set of {@link Enum} constants for describing the status of a {@linkplain Mapping mapping}, either as\n * {@link #POSSIBLE} or {@link #SURE}.\n *

\n * Statuses have a corresponding {@linkplain #getAbbreviation() abbreviated form} for use when mappings are\n * formatted and parsed as strings. The enum constants are arranged in order of strength of surety, so that \n * {@link Enum#ordinal()} returns numbers in order of increasing surety.\n * \n * @author Scott Martin\n * @see MappingFormat\n */\npublic enum Status {\n\t\n\t/**\n\t * The status of a mapping that is only possible, not sure.\n\t */\n\tPOSSIBLE,\n\t\n\t/**\n\t * The status of a sure mapping (not just possible).\n\t */\n\tSURE;\n\t\n\t/**\n\t * An abbreviated form for this status, for use in parsing and formatting.\n\t * @see MappingFormat\n\t */\n\tfinal String abbreviation;\n\t\n\tprivate Status() {\n\t\tthis.abbreviation = name().substring(0, 1);\n\t}\n\t\n\t/**\n\t * Gets the abbreviated form of this status, "S" for {@link #SURE}\n\t * and "P" for {@link #POSSIBLE}.\n\t */\n\tpublic String getAbbreviation() {\n\t\treturn abbreviation;\n\t}\n\n\t/**\n\t * Gives the status constant corresponding to the given abbreviation.\n\t * @param abbreviation The abbreviated form to find a status constant for.\n\t * @return A status constant if one is found whose {@link #getAbbreviation()} is equal to the specified \n\t * abbreviation, otherwise null.\n\t */\n\tpublic static Status forAbbreviation(String abbreviation) {\n\t\tfor(Status s : values()) {\n\t\t\tif(s.abbreviation.equals(abbreviation)) {\n\t\t\t\treturn s;\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn null;\n\t}\n}" }, { "path": "src/opennlp/ccg/alignment/package.html", "content": "\n\t\n

This package provides classes for modeling alignments between two\nphrases.

\n\n

\nThe {@link opennlp.ccg.alignment.Phrase} class models a phrase as simply\na list of words. A single alignment between an index in one phrase and an index\nin another is captured by the {@link opennlp.ccg.alignment.Mapping} class. The\n{@link opennlp.ccg.alignment.Alignment} class is build on top of Phrase and\nMapping, encapsulating an "A" phrase and a "B" phrase\nalong with a set of mappings between them (see\n{@link opennlp.ccg.alignment.PhrasePosition}).\n

\n\n

\nSince there are multiple different encoding schemes for representing alignments,\na goal of this package is to make the conceptual representation of aligned phrases\nas independent of encoding as possible. The interface\n{@link opennlp.ccg.alignment.EncodingScheme} gives a way to represent\nencoding schemes, and two popular encoding schemes (Moses and NAACL) are provided by this\npackage. Different encoding schemes use different\nnumbering bases: some start numbering indices at 0 and some at 1. To reconcile this\ndifference, the class {@link opennlp.ccg.alignment.IndexBase} provides a way to translate\ndifferent numbering bases into a common index base. The {@link opennlp.ccg.alignment.MappingFormat}\nclass extends {@link java.text.Format} to give encoding-independent formatting and parsing\nfor mappings.\n

\n\n

\nThe classes {@link opennlp.ccg.alignment.PhraseReader} and {@link opennlp.ccg.alignment.PhraseWriter}\n(and their descendants), {@link opennlp.ccg.alignment.MappingReader} and\n{@link opennlp.ccg.alignment.MappingWriter} provide convenient ways to read and write phrases\nand mappings. Lastly, the non-instantiable class {@link opennlp.ccg.alignment.Alignments} provides some\nconvenience methods for reading phrases, mappings and alignments.\n

\n\n\n" }, { "path": "src/opennlp/ccg/disjunctivizer/AlignedEdgeFilter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.disjunctivizer;\n\nimport static opennlp.ccg.disjunctivizer.MatchType.SOURCE_ALIGNED;\nimport static opennlp.ccg.disjunctivizer.MatchType.SOURCE_UNALIGNED;\nimport static opennlp.ccg.disjunctivizer.MatchType.TARGET_ALIGNED;\nimport static opennlp.ccg.disjunctivizer.MatchType.TARGET_UNALIGNED;\n\nimport java.util.Collection;\nimport java.util.Set;\n\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.hylo.graph.LFVertex;\n\n\n/**\n * A filter for edges that tests whether they are aligned based on a specified set of \n * {@linkplain #getAlignmentIndices() alignment indices}. Whether the source or target vertices\n * (or both) is considered depends on the match type criteria in effect. For example, if the match\n * type criteria contains {@link MatchType#SOURCE_ALIGNED}, this filter's {@link #allows(LFEdge)} method\n * will check whether argument edges have a {@linkplain LFVertex#getIndex() source index} that is \n * contained in the set of alignment indices. \n *

\n * Instances of this class use the following match type criteria: {@link MatchType#SOURCE_ALIGNED},\n * {@link MatchType#SOURCE_UNALIGNED}, {@link MatchType#TARGET_ALIGNED}, \n * and {@link MatchType#TARGET_UNALIGNED}. If the set of alignment indices is modified after an instance\n * of this class is created, the filter will reflect the changes because the set is not copied at\n * creation.\n * \n * @author Scott Martin\n */\npublic class AlignedEdgeFilter extends MatchTypeFilter {\n\n\tSet alignmentIndices;\n\t\t\n\t/**\n\t * Creates a new aligned edge filter based on the specified alignment indices for the specified\n\t * match type criteria.\n\t * @param alignmentIndices The set of indices to check for alignment.\n\t * @param matchTypes The match type criteria to use.\n\t * @throws IllegalArgumentException If alignmentIndices is null.\n\t */\n\tpublic AlignedEdgeFilter(Set alignmentIndices, MatchType... matchTypes) {\n\t\tsuper(matchTypes);\n\t\t\n\t\tcheckAlignmentIndices(alignmentIndices);\n\t\tthis.alignmentIndices = alignmentIndices;\n\t}\n\n\t/**\n\t * Creates a new aligned edge filter based on the specified alignment indices for the specified\n\t * match type criteria.\n\t * @param alignmentIndices The set of indices to check for alignment.\n\t * @param matchTypes The match type criteria to use.\n\t * @throws IllegalArgumentException If alignmentIndices is null.\n\t */\n\tpublic AlignedEdgeFilter(Set alignmentIndices, Collection matchTypes) {\n\t\tsuper(matchTypes);\n\t\t\n\t\tcheckAlignmentIndices(alignmentIndices);\n\t\tthis.alignmentIndices = alignmentIndices;\n\t}\n\t\n\tprivate void checkAlignmentIndices(Set alignmentIndices) {\n\t\tif(alignmentIndices == null) {\n\t\t\tthrow new IllegalArgumentException(\"alignmentIndices is null\");\n\t\t}\n\t}\n\n\t/**\n\t * Gets the alignment indices used by this filter.\n\t */\n\tpublic Set getAlignmentIndices() {\n\t\treturn alignmentIndices;\n\t}\n\n\t/**\n\t * Sets the alignment indices used by this filter.\n\t * @throws IllegalArgumentException If alignmentIndices is null.\n\t */\n\tpublic void setAlignmentIndices(Set alignmentIndices) {\n\t\tcheckAlignmentIndices(alignmentIndices);\t\n\t\tthis.alignmentIndices = alignmentIndices;\n\t}\n\n\t/**\n\t * Tests whether this filter allows the specified LF edge.\n\t * @return false if {@link #getMatchTypes()} contains\n\t *

    \n\t * \t
  • {@link MatchType#SOURCE_ALIGNED}, but the alignment indices does not contain the edge's\n\t * \t\tsource vertex's index,
    • \n\t * \t
  • {@link MatchType#SOURCE_UNALIGNED}, but the alignment indices contains the edge's\n\t * \t\tsource vertex's index,
    • \n\t * \t
  • {@link MatchType#TARGET_ALIGNED}, but the alignment indices does not contain the edge's\n\t * \t\ttarget vertex's index,
    • \n\t * \t
  • {@link MatchType#TARGET_UNALIGNED}, but the alignment indices contains the edge's\n\t * \t\ttarget vertex's index,
    • \n\t *
\n\t * and true otherwise.\n\t * \n\t * @see #getAlignmentIndices()\n\t * @see LFEdge#getSource()\n\t * @see LFEdge#getTarget()\n\t * @see LFVertex#getIndex()\n\t */\n\t@Override\n\tpublic boolean allows(LFEdge edge) {\n\t\tfor(MatchType t : matchTypes) {\n\t\t\tif(t == SOURCE_ALIGNED && !alignmentIndices.contains(edge.getSource().getIndex())) {\n\t\t\t\treturn false;\n\t\t\t}\n\t\t\telse if(t == SOURCE_UNALIGNED && alignmentIndices.contains(edge.getSource().getIndex())) {\n\t\t\t\treturn false;\n\t\t\t}\n\t\t\telse if(t == TARGET_ALIGNED && !alignmentIndices.contains(edge.getTarget().getIndex())) {\n\t\t\t\treturn false;\n\t\t\t}\n\t\t\telse if(t == TARGET_UNALIGNED && alignmentIndices.contains(edge.getTarget().getIndex())) {\n\t\t\t\treturn false;\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn true;\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/disjunctivizer/Disjunctivizer.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.disjunctivizer;\n\nimport static opennlp.ccg.alignment.PhrasePosition.A;\nimport static opennlp.ccg.alignment.PhrasePosition.B;\nimport static opennlp.ccg.disjunctivizer.Disjunctivizer.VertexType.LOCAL_ANCESTOR;\nimport static opennlp.ccg.disjunctivizer.Disjunctivizer.VertexType.OPTIONAL;\nimport static opennlp.ccg.disjunctivizer.Disjunctivizer.VertexType.PREDICATES;\nimport static opennlp.ccg.disjunctivizer.Disjunctivizer.VertexType.SHARED;\nimport static opennlp.ccg.disjunctivizer.Disjunctivizer.VertexType.VISITED;\nimport static opennlp.ccg.disjunctivizer.MatchType.TARGET_PREDICATE_MISMATCH;\n\nimport java.util.Arrays;\nimport java.util.Collections;\nimport java.util.EnumMap;\nimport java.util.HashMap;\nimport java.util.HashSet;\nimport java.util.Iterator;\nimport java.util.Map;\nimport java.util.Set;\n\nimport javax.xml.parsers.DocumentBuilderFactory;\nimport javax.xml.parsers.ParserConfigurationException;\n\nimport opennlp.ccg.alignment.PhrasePosition;\nimport opennlp.ccg.hylo.Mode;\nimport opennlp.ccg.hylo.Proposition;\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.hylo.graph.LFEdgeLabel;\nimport opennlp.ccg.hylo.graph.LFGraph;\nimport opennlp.ccg.hylo.graph.LFVertex;\nimport opennlp.ccg.util.DelegatedFilter;\nimport opennlp.ccg.util.Filter;\nimport opennlp.ccg.util.FilteredSet;\nimport opennlp.ccg.util.MembershipFilter;\nimport opennlp.ccg.util.VisitedFilter;\n\nimport org.jgrapht.traverse.DepthFirstIterator;\nimport org.w3c.dom.Document;\nimport org.w3c.dom.Element;\nimport org.w3c.dom.Node;\nimport org.w3c.dom.NodeList;\n\n/**\n * Creates a disjunctive logical form from a difference between two graphs.\n * Instances can be configured to switch handling of {@link LFGraphDifference#inserts()},\n * {@link LFGraphDifference#deletes()} and {@link LFGraphDifference#substitutions()}\n * on or off. By default, all three are performed.\n *

\n * Disjunctivizers can be re-used, so that all of the boolean parameters\n * {@link #setProcessingDeletes(boolean)}, {@link #setProcessingInserts(boolean)},\n * and {@link #setProcessingSubstitutions(boolean)} can be modified in between calls to\n * {@link #buildDisjunctiveLFFor(LFGraphDifference)}.\n * When these parameters are changed, the disjunctive LF built will change as well.\n * The {@link Document} used to create disjunctive LF elements (and the elements within them)\n * can be configured as well, either at creation or via {@link #setDocument(Document)}. \n * \n * @author Scott Martin\n */\npublic class Disjunctivizer {\n\n\t/**\n\t * Attribute set tag name: atts.\n\t */\n\tpublic static final String ATTS_TAG = \"atts\";\n\t\n\t/**\n\t * Choice disjunction tag name: one-of.\n\t */\n\tpublic static final String CHOICE_TAG = \"one-of\";\n\t\n\t/**\n\t * Disjunctive LF tag name: dlf.\n\t */\n\tpublic static final String DLF_TAG = \"dlf\";\n\t\n\t/**\n\t * Node tag name: node.\n\t */\n\tpublic static final String NODE_TAG = \"node\";\n\t\n\t/**\n\t * Optional disjunction tag name: opt.\n\t */\n\tpublic static final String OPTIONAL_TAG = \"opt\";\n\t\n\t/**\n\t * Relation tag name: rel.\n\t */\n\tpublic static final String RELATION_TAG = \"rel\";\n\t\n\t/**\n\t * ID attribute name: id.\n\t */\n\tpublic static final String ID_ATTR = \"id\";\n\t\n\t/**\n\t * ID reference attribute name: idref.\n\t */\n\tpublic static final String IDREF_ATTR = \"idref\";\n\t\n\t/**\n\t * Name attribute name: name.\n\t */\n\tpublic static final String NAME_ATTR = \"name\";\n\t\n\t/**\n\t * Predicate attribute name: pred.\n\t */\n\tpublic static final String PRED_ATTR = \"pred\";\n\t\n\t/**\n\t * Attribute name for node sharedness: shared.\n\t */\n\tpublic static final String SHARED_ATTR = \"shared\";\n\t\n\t/**\n\t * The suffix appended to foreign nodes: f.\n\t */\n\tpublic static final String FOREIGN_SUFFIX = \"f\";\n\t\n\tDocument document;\n\tboolean processingInserts, processingDeletes, processingSubstitutions;\n\t\n\tprivate Element disjunctiveLF;\n\t\n\tprivate LFGraphDifference graphDifference;\n\tprivate Set importedVertices = null;\n\tprivate Map vertexAliases = null;\n\tprivate Map foreignAlignedSubgraphRoots = null;\n\t\n\t/**\n\t * Creates a new disjunctivizer using a new document.\n\t * @see #Disjunctivizer(Document)\n\t */\n\tpublic Disjunctivizer() throws ParserConfigurationException {\n\t\tthis(DocumentBuilderFactory.newInstance().newDocumentBuilder().newDocument());\n\t}\n\t\n\t/**\n\t * Creates a new disjunctivizer that will use the specified document\n\t * to create the elements in its generated disjunctive LFs.\n\t * @see #Disjunctivizer(Document, boolean, boolean, boolean)\n\t */\n\tpublic Disjunctivizer(Document document) {\n\t\tthis(document, true, true, true);\n\t}\n\t\n\t/**\n\t * Creates a new disjunctivizer that will use the specified document\n\t * to create the elements in its generated disjunctive LFs, with the specified parameters dictating\n\t * whether to process inserts, deletes, and substitutions.\n\t * \n\t * @param document The document to use for creating elements, attributes, nodes, etc.\n\t * @param processingInserts Whether to process {@link LFGraphDifference#inserts()}.\n\t * @param processingDeletes Whether to process {@link LFGraphDifference#deletes()}.\n\t * @param processingSubstitutions Whether to process {@link LFGraphDifference#substitutions()}.\n\t */\n\tpublic Disjunctivizer(Document document,\n\t\t\tboolean processingInserts, boolean processingDeletes, boolean processingSubstitutions) {\n\t\tif(document == null) {\n\t\t\tthrow new IllegalArgumentException(\"document is null\");\n\t\t}\n\t\t\n\t\tthis.document = document;\n\t\tthis.processingInserts = processingInserts;\n\t\tthis.processingDeletes = processingDeletes;\n\t\tthis.processingSubstitutions = processingSubstitutions;\n\t}\n\n\t/**\n\t * Gets the document used to create elements, nodes, attributes, etc.\n\t * @see #Disjunctivizer(Document, boolean, boolean, boolean)\n\t */\n\tpublic Document getDocument() {\n\t\treturn document;\n\t}\n\n\t/**\n\t * Sets the document used to create elements.\t\n\t * @param document The document that will be used while building disjunctive LF elements.\n\t */\n\tpublic void setDocument(Document document) {\n\t\tthis.document = document;\n\t}\n\t\n\t/**\n\t * Returns whether this disjunctivizer processes {@link LFGraphDifference#inserts()}.\n\t */\n\tpublic boolean isProcessingInserts() {\n\t\treturn processingInserts;\n\t}\n\n\t/**\n\t * Sets whether this disjunctivizer processes {@link LFGraphDifference#inserts()}.\n\t */\n\tpublic void setProcessingInserts(boolean processingInserts) {\n\t\tif(this.processingInserts != processingInserts) {\n\t\t\tthis.processingInserts = processingInserts;\n\t\t\tresetDisjunctiveLF();\n\t\t}\n\t}\n\n\t/**\n\t * Returns whether this disjunctivizer processes {@link LFGraphDifference#deletes()}.\n\t */\n\tpublic boolean isProcessingDeletes() {\n\t\treturn processingDeletes;\n\t}\n\n\t/**\n\t * Sets whether this disjunctivizer processes {@link LFGraphDifference#deletes()}.\n\t */\n\tpublic void setProcessingDeletes(boolean processingDeletes) {\n\t\tif(this.processingDeletes != processingDeletes) {\n\t\t\tthis.processingDeletes = processingDeletes;\n\t\t\tresetDisjunctiveLF();\n\t\t}\n\t}\n\n\t/**\n\t * Returns whether this disjunctivizer processes {@link LFGraphDifference#substitutions()}.\n\t */\n\tpublic boolean isProcessingSubstitutions() {\n\t\treturn processingSubstitutions;\n\t}\n\n\t/**\n\t * Sets whether this disjunctivizer processes {@link LFGraphDifference#substitutions()}.\n\t */\n\tpublic void setProcessingSubstitutions(boolean processingSubstitutions) {\n\t\tif(this.processingSubstitutions != processingSubstitutions) {\n\t\t\tthis.processingSubstitutions = processingSubstitutions;\n\t\t\tresetDisjunctiveLF();\n\t\t}\n\t}\n\n\tprivate void resetDisjunctiveLF() {\n\t\tthis.disjunctiveLF = null;\n\t}\n\t\n\t/**\n\t * Builds a disjunctive LF based on the specified graph difference.\n\t * The shape of the returned element will change depending on whether inserts, deletes, or substitutions\n\t * are being processed.\n\t * @param graphDifference The graph difference to use for building the disjunctive LF.\n\t * @return A recursively build disjunctive LF based on this disjunctivizer's graph difference.\n\t * @throws IllegalArgumentException If graphDifference is null.\n\t */\n\tpublic Element buildDisjunctiveLFFor(LFGraphDifference graphDifference) {\n\t\tif(graphDifference == null) {\n\t\t\tthrow new IllegalArgumentException(\"graph difference is null\");\n\t\t}\n\t\t\n\t\tif(disjunctiveLF == null || !this.graphDifference.equals(graphDifference)) {\n\t\t\tthis.graphDifference = graphDifference;\n\t\t\t\t\t\t\n\t\t\t// reset in case this has been previously called\n\t\t\tif(foreignAlignedSubgraphRoots != null) {\n\t\t\t\tforeignAlignedSubgraphRoots = null;\n\t\t\t}\n\t\t\t\n\t\t\tif(importedVertices == null) {\n\t\t\t\timportedVertices = new HashSet();\n\t\t\t}\n\t\t\telse {\n\t\t\t\timportedVertices.clear();\n\t\t\t}\n\t\t\t\n\t\t\tif(vertexAliases == null) {\n\t\t\t\tvertexAliases = new HashMap();\n\t\t\t}\n\t\t\telse {\n\t\t\t\tvertexAliases.clear();\n\t\t\t}\t\t\n\t\t\t\n\t\t\tdisjunctiveLF = document.createElement(DLF_TAG);\n\t\t\t\n\t\t\tfor(LFVertex p : graphDifference.a.highestLFAncestors()) {\n\t\t\t\tdisjunctiveLF.appendChild(createDisjunctiveElement(new DLFContext(A, p, disjunctiveLF)));\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn disjunctiveLF;\n\t}\n\t\n\tprivate void findForeignAlignedSubgraphRoots() {\n\t\tforeignAlignedSubgraphRoots = new HashMap();\n\t\t\n\t\tLFGraphIterator rootIterator = new LFGraphIterator(graphDifference.b);\n\t\tMap> mappings = graphDifference.alignment.asMap(B);\n\t\t\n\t\twhile(rootIterator.hasNext()) {\n\t\t\tLFVertex vertex = rootIterator.next();\n\t\t\t\n\t\t\tif(!foreignAlignedSubgraphRoots.containsKey(vertex)) { // already encountered?\n\t\t\t\tif(mappings.containsKey(vertex.getIndex())) { // aligned?\n\t\t\t\t\tLFGraphIterator subgraphIterator = new LFGraphIterator(graphDifference.b, vertex);\n\t\t\t\t\t\n\t\t\t\t\twhile(subgraphIterator.hasNext()) { // map whole subgraph to the aligned vertex\n\t\t\t\t\t\tforeignAlignedSubgraphRoots.put(subgraphIterator.next(), vertex);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\t\n\tprivate Element createDisjunctiveElement(DLFContext context) {\n\t\tLFVertex alias = vertexAliases.get(context.vertex);\n\t\t\n\t\tString vertexName = nameFor(context.vertex);\n\t\tboolean imported = importedVertices.contains(context.vertex);\n\t\t\n\t\tElement newNode = document.createElement(NODE_TAG);\n\t\tcontext.parent.appendChild(newNode);\n\t\t\n\t\tDLFContext localContext = context.copy();\n\t\tlocalContext.parent = newNode;\n\t\tSet locals = localContext.getVertices(LOCAL_ANCESTOR),\n\t\t\t\tvisited = localContext.getVertices(VISITED);\n\t\t\n\t\tif(locals.contains(localContext.vertex)\n\t\t\t\t|| (alias != null && visited.contains(localContext.vertex))) {\n\t\t\tLFVertex v = (alias == null) ? localContext.vertex : alias;\n\t\t\tlocalContext.parent.setAttribute(IDREF_ATTR, nameFor(v));\n\t\t\t\n\t\t\tif(visited.contains(localContext.vertex)\n\t\t\t\t\t&& localContext.getVertices(SHARED).contains(v) && !locals.contains(v)) {\n\t\t\t\tlocalContext.parent.setAttribute(SHARED_ATTR, \"true\");\n\t\t\t}\n\t\t}\n\t\telse {\n\t\t\tlocalContext.parent.setAttribute(ID_ATTR, alias == null ? vertexName : nameFor(alias));\n\t\t\t\n\t\t\tvisited.add(localContext.vertex);\n\t\t\tlocals.add(localContext.vertex);\n\t\t\t\n\t\t\taddNonPredAttributes(localContext);\n\t\t\t\n\t\t\tif(!imported && processingInserts) { // inserts\n\t\t\t\tprocessInserts(localContext);\n\t\t\t}\n\t\t\tif(!imported && processingDeletes) { // deletes\n\t\t\t\tprocessDeletes(localContext);\n\t\t\t}\n\t\t\t\n\t\t\tLFGraph graph = localContext.getGraph();\n\t\t\t\n\t\t\t@SuppressWarnings(\"unchecked\")\n\t\t\tSet outgoingEdges = graph.containsVertex(localContext.vertex)\n\t\t\t\t\t? graph.outgoingEdgesOf(localContext.vertex) : Collections.EMPTY_SET;\n\t\t\t\n\t\t\tif(outgoingEdges.isEmpty()) { // leaf?\n\t\t\t\tsetPredicateName(localContext);\n\t\t\t}\n\t\t\telse {\n\t\t\t\tif(!imported && processingSubstitutions) { // do substitutions, if applicable\n\t\t\t\t\tprocessSubstitutions(localContext);\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tfor(LFEdge out : outgoingEdges) {\n\t\t\t\t\t\tif(imported) {\n\t\t\t\t\t\t\timportedVertices.add(out.getTarget());\n\t\t\t\t\t\t}\n\t\t\t\t\t\t\n\t\t\t\t\t\t// context.graph should be the B graph if imported\n\t\t\t\t\t\tprocessNonsubstitutedEdge(localContext, out);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\t\n\t\tfixLabelReferences(newNode);\n\t\t\n\t\treturn newNode;\n\t}\n\t\n\tprivate String nameFor(LFVertex vertex) {\n\t\tString vn = vertex.getName();\n\t\treturn importedVertices.contains(vertex) ? vn + FOREIGN_SUFFIX : vn;\n\t}\n\t\n\tprivate void processInserts(DLFContext context) {\n\t\tElement optional = null;\n\t\t\n\t\tfor(LFEdge ins : graphDifference.insertsFor(context.vertex)) {\n\t\t\t// check if subgraph is aligned somewhere\n\t\t\tif(foreignAlignedSubgraphRoots == null) {\n\t\t\t\tfindForeignAlignedSubgraphRoots();\n\t\t\t}\n\t\t\t\n\t\t\tif(!foreignAlignedSubgraphRoots.containsKey(ins.getTarget())) {\n\t\t\t\tDLFContext ctxt = context.copy();\n\t\t\t\tctxt.graphPosition = B; // use foreign graph\n\t\t\t\t\n\t\t\t\tif(optional == null) {\n\t\t\t\t\tctxt.parent = addOptional(context);\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\timportedVertices.add(ins.getTarget()); // remember that inserted vertex is foreign\n\t\t\t\t\n\t\t\t\tdoInsertDelete(ctxt, ins);\n\t\t\t}\n\t\t}\n\t}\n\t\n\tprivate void processDeletes(DLFContext context) {\n\t\tElement optional = null;\n\t\t\n\t\tfor(LFEdge del : graphDifference.deletesFor(context.vertex)) {\n\t\t\tDLFContext ctxt = context.copy();\n\t\t\t\n\t\t\tif(optional == null) {\n\t\t\t\tctxt.parent = addOptional(context);\n\t\t\t}\n\t\t\t\n\t\t\tdoInsertDelete(ctxt.copy(), del);\n\t\t}\n\t}\n\t\n\tprivate void doInsertDelete(DLFContext context, LFEdge edge) {\n\t\tLFVertex trg = edge.getTarget();\n\t\tDLFContext ctxt = context.copy(); \n\t\tctxt.vertex = trg;\n\t\tctxt.parent = addRelation(ctxt, edge.getLabel());\n\t\tctxt.addVertex(trg, OPTIONAL);\n\t\t\n\t\tctxt.parent.appendChild(createDisjunctiveElement(ctxt));\n\t}\n\t\n\tprivate void processSubstitutions(DLFContext context) {\n\t\tfor(LFEdge outgoing : context.getGraph().outgoingEdgesOf(context.vertex)) {\n\t\t\tif(graphDifference.substitutionsFor(outgoing).isEmpty()) { // no substitution(s) for this edge?\n\t\t\t\tprocessNonsubstitutedEdge(context.copy(), outgoing);\n\t\t\t}\n\t\t\telse {\n\t\t\t\tprocessSubstitutedEdge(context.copy(), outgoing);\n\t\t\t}\n\t\t}\n\t}\n\t\n\tprivate void processNonsubstitutedEdge(DLFContext context, LFEdge outgoing) {\n\t\tif(!context.getVertices(PREDICATES).contains(context.vertex)) {\n\t\t\tsetPredicateName(context);\n\t\t}\n\t\t\n\t\tLFVertex trg = outgoing.getTarget();\n\t\tSet similarTargets = new FilteredSet(context.getVertices(OPTIONAL), \n\t\t\t\tnew SimilarTargetVertexFilter(trg));\n\t\t\n\t\tif(similarTargets.isEmpty()) {\n\t\t\tDLFContext ctxt = context.copy();\n\t\t\tctxt.vertex = trg;\n\t\t\tctxt.parent = addRelation(context, outgoing.getLabel());\n\t\t\t\n\t\t\tctxt.parent.appendChild(createDisjunctiveElement(ctxt));\n\t\t}\n\t\telse { // target already present as an option\n\t\t\tfor(LFVertex similar : similarTargets) {\n\t\t\t\tassimilateAttributes(context.copy(), trg, similar);\n\t\t\t}\n\t\t}\n\t}\n\t\n\tprivate void processSubstitutedEdge(DLFContext context, LFEdge outgoing) {\n\t\tprocessSubstitutedSimilarTarget(context.copy(), outgoing);\n\t\t\n\t\tif(!context.getVertices(PREDICATES).contains(context.vertex)) {\n\t\t\tprocessSubstitutedPredicates(context.copy(), outgoing);\n\t\t}\n\t\t\n\t\t// get the substitutions for the outgoing edge\n\t\tMap> subsBySource = graphDifference.substitutionsBySourceFor(outgoing);\n\t\tEdgeMatchFilter predicateFilter = null;\n\t\t\n\t\tfor(LFVertex subSource : subsBySource.keySet()) {\n\t\t\tSet subEdges = subsBySource.get(subSource);\n\t\t\t\n\t\t\tif(predicateFilter == null) {\n\t\t\t\tpredicateFilter = new EdgeMatchFilter(outgoing, TARGET_PREDICATE_MISMATCH);\n\t\t\t}\n\t\t\telse {\n\t\t\t\tpredicateFilter.setBasis(outgoing);\n\t\t\t}\n\t\t\t\n\t\t\t// find the edges matching the outgoing edge's label, and the vertices with different predicates\n\t\t\t// from the outgoing edge's target vertex\n\t\t\tFilteredLFEdgeSet identicals = new FilteredLFEdgeSet(subEdges, new MembershipFilter(\n\t\t\t\t\t\t\t\t\t\tcontext.getGraph().outgoingEdgesOf(context.vertex)));\n\t\t\tSet matchingLabels\n\t\t\t\t= new FilteredLFEdgeSet(subEdges, new LabelMatchFilter(outgoing.getLabel()));\n\t\t\tSet differentPredicates\n\t\t\t\t= new FilteredSet(\n\t\t\t\t\t\tnew FilteredLFEdgeSet(matchingLabels, predicateFilter).targetView(),\n\t\t\t\t\t\tnew VisitedFilter());\n\t\t\t\n\t\t\t// deal with the edges with matching labels separately from other substitutions\n\t\t\tsubEdges.removeAll(matchingLabels);\n\t\t\tdifferentPredicates.removeAll(identicals.targetView());\n\t\t\tsubEdges.removeAll(identicals);\n\t\t\t\n\t\t\tif(subEdges.isEmpty() && differentPredicates.isEmpty()) { // no substitutions to make\n\t\t\t\tDLFContext ctxt = context.copyWithVertexMask(LOCAL_ANCESTOR, PREDICATES);\n\t\t\t\tfixOptions(ctxt, outgoing.getLabel());\n\t\t\t\t\n\t\t\t\tctxt.parent = addRelation(context, outgoing.getLabel());\n\t\t\t\tctxt.vertex = outgoing.getTarget();\n\t\t\t\tctxt.parent.appendChild(createDisjunctiveElement(ctxt));\t\t\t\t\n\t\t\t}\n\t\t\telse {\n\t\t\t\tif(!differentPredicates.isEmpty()) { // handle matching labels but different predicates\n\t\t\t\t\tprocessDifferentPredicates(context.copy(), outgoing, differentPredicates);\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tif(!subEdges.isEmpty()) { // handle others\n\t\t\t\t\tprocessSubstitutedEdges(context.copy(), outgoing, subEdges);\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\t\n\tprivate void processDifferentPredicates(DLFContext context, LFEdge outgoing,\n\t\t\tSet differentPredicates) {\n\t\tLFEdgeLabel label = outgoing.getLabel();\n\t\t\n\t\tboolean terminal = context.getGraph().outDegreeOf(context.vertex) == 0;\n\t\tif(!terminal) {\n\t\t\tterminal = !new FilteredSet(\n\t\t\t\t\t\t\tdifferentPredicates, new TerminalFilter(graphDifference.b))\n\t\t\t\t\t\t.isEmpty();\n\t\t}\n\t\t\n\t\tif(terminal) {\n\t\t\tDLFContext ctxt = context.copy();\n\t\t\tctxt.vertex = outgoing.getTarget();\n\t\t\t\t\t\t\n\t\t\tif(differentPredicates.size() == 1) { // if we're here at all, it's at least non-empty\n\t\t\t\tprocessSingletonDifferentPredicate(ctxt, outgoing, differentPredicates.iterator().next());\n\t\t\t}\n\t\t\telse {\n\t\t\t\tprocessMultipleDifferentPredicates(ctxt, outgoing, differentPredicates);\n\t\t\t}\n\t\t}\n\t\telse { // non-terminal, continue recursing through the graph\n\t\t\tDLFContext ctxt = context.copyWithVertexMask(LOCAL_ANCESTOR, PREDICATES);\n\t\t\t\n\t\t\tctxt.vertex = outgoing.getTarget();\n\t\t\tctxt.parent = addRelation(ctxt, label);\n\t\t\t\n\t\t\tctxt.parent.appendChild(createDisjunctiveElement(ctxt));\n\t\t}\n\t}\n\t\n\tprivate void processSingletonDifferentPredicate(DLFContext context, LFEdge outgoing,\n\t\t\tLFVertex differentPredicate) {\n\t\tLFEdgeLabel label = outgoing.getLabel();\n\t\t\n\t\t// add relation, then choice point\n\t\tElement newRel = addRelation(context, label);\n\t\tcontext.parent = newRel;\n\t\tElement choiceElement = addChoice(context);\n\t\tcontext.parent = choiceElement;\n\t\t\n\t\t// generate the target element, but do not propagate changes to tracked vertices\n\t\tElement targetElement = createDisjunctiveElement(context.copy(true));\n\t\t\n\t\tif(!vertexAliases.containsKey(differentPredicate)) {\n\t\t\tvertexAliases.put(differentPredicate, outgoing.getTarget());\n\t\t}\n\t\t\n\t\tcontext.vertex = differentPredicate;\n\t\tcontext.parent.appendChild(createDisjunctiveElement(context.copy(true)));\n\t\t\n\t\t// cleanup: how many new nodes were aliased?\n\t\tNodeList newNodes = newRel.getElementsByTagName(NODE_TAG);\n\t\tfor(int j = 0; j < newNodes.getLength(); j++) {\n\t\t\tif(newNodes.item(j).getAttributes().getNamedItem(IDREF_ATTR) == null) {\n\t\t\t\treturn; // one wasn't aliased\n\t\t\t}\n\t\t}\n\t\t\n\t\t// if we get here, they all were aliased: use generated target element instead\n\t\tnewRel.replaceChild(targetElement, choiceElement);\n\t}\n\t\n\tprivate void processMultipleDifferentPredicates(DLFContext context, LFEdge outgoing,\n\t\t\tSet differentPredicates) {\n\t\tLFEdgeLabel label = outgoing.getLabel();\n\t\t\n\t\t// generate the choice point\n\t\tElement choiceElement = addChoice(context);\n\t\tcontext.parent = choiceElement;\n\t\t\n\t\t// and the relation, but do not propagate changes to tracked vertices\n\t\tcontext.parent = addRelation(context, label);\n\t\tcontext.parent.appendChild(createDisjunctiveElement(context.copy(true)));\n\t\t\n\t\t// add attributes tag, after resetting parent to choice point\n\t\tcontext.parent = choiceElement;\n\t\tElement atts = addElement(context, ATTS_TAG);\n\t\t\n\t\t// then go through the different predicates, checking for aliases\n\t\tboolean aliased = false;\n\t\tfor(LFVertex d : differentPredicates) {\n\t\t\tcontext.parent = atts;\n\t\t\t\n\t\t\tif(!aliased && !vertexAliases.containsKey(d)) {\n\t\t\t\tvertexAliases.put(d, outgoing.getTarget());\n\t\t\t\taliased = true;\n\t\t\t}\n\t\t\t\n\t\t\t// add new relation for each different pred.\n\t\t\tcontext.parent = addRelation(context, label);\n\t\t\tcontext.vertex = d;\n\t\t\tcontext.parent.appendChild(createDisjunctiveElement(context.copy(true)));\n\t\t}\n\t}\n\t\n\tprivate void processSubstitutedEdges(DLFContext context, LFEdge outgoing, Set substituedEdges) {\n\t\tLFEdgeLabel label = outgoing.getLabel();\n\t\tboolean singleton = substituedEdges.size() == 1; // can't be empty if we get here\n\t\t\n\t\tElement choiceElement = addChoice(context);\n\t\tcontext.parent = choiceElement;\n\t\t\n\t\tElement toAppendTo = singleton ? choiceElement : addElement(context, ATTS_TAG);\n\t\t\n\t\tcontext.parent = addRelation(context, label);\n\t\t\n\t\tDLFContext ctxt = context.copy(true);\n\t\tctxt.vertex = outgoing.getTarget();\n\t\tctxt.parent.appendChild(createDisjunctiveElement(ctxt));\n\t\t\n\t\tboolean aliased = false;\n\t\tcontext.parent = toAppendTo;\n\t\t\n\t\tfor(LFEdge s : substituedEdges) {\n\t\t\tLFVertex t = s.getTarget();\n\t\t\tString vPred = context.vertex.getPredicate(), tPred = t.getPredicate();\n\t\t\tLFEdgeLabel l = s.getLabel();\n\t\t\t\n\t\t\tcontext.parent = addRelation(context, l);\n\t\t\t\n\t\t\t// shared?\n\t\t\tif(vPred != null && vPred.equals(tPred) && !label.equals(l)) { \n\t\t\t\tElement subNode = addElement(context, NODE_TAG);\n\t\t\t\tsubNode.setAttribute(IDREF_ATTR, nameFor(context.vertex));\n\t\t\t\t\n\t\t\t\tLFVertex sAlias = vertexAliases.get(context.vertex);\n\t\t\t\tif(context.getVertices(VISITED).contains(sAlias)\n\t\t\t\t\t\t&& context.getVertices(SHARED).contains(sAlias)\n\t\t\t\t\t\t&& !context.getVertices(LOCAL_ANCESTOR).contains(sAlias)) {\n\t\t\t\t\tsubNode.setAttribute(SHARED_ATTR, \"true\");\n\t\t\t\t}\n\t\t\t}\n\t\t\telse {\n\t\t\t\tif((singleton || !aliased) && !vertexAliases.containsKey(t)) {\n\t\t\t\t\tvertexAliases.put(t, outgoing.getTarget());\n\t\t\t\t\taliased = true;\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tDLFContext c = context.copy(true);\n\t\t\t\t\n\t\t\t\tc.vertex = t;\n\t\t\t\timportedVertices.add(t);\n\t\t\t\t\n\t\t\t\tc.graphPosition = B; // use foreign graph for substitution\n\t\t\t\tc.parent.appendChild(createDisjunctiveElement(c));\n\t\t\t}\n\t\t}\n\t}\n\t\n\tprivate void processSubstitutedSimilarTarget(DLFContext context, LFEdge outgoing) {\n\t\tLFVertex target = outgoing.getTarget();\n\t\tMap> subsBySource = graphDifference.substitutionsBySourceFor(outgoing);\n\t\t\n\t\tDLFContext ctxt = context.copy();\n\t\t\n\t\t// for each substituted edge, look for similar target\n\t\tfor(LFVertex subSource : subsBySource.keySet()) {\n\t\t\tSet similarTargetEdges = new FilteredLFEdgeSet(subsBySource.get(subSource),\n\t\t\t\t\tnew SimilarTargetEdgeFilter(ctxt.vertex, outgoing.getLabel()));\n\t\t\t\n\t\t\tif(!similarTargetEdges.isEmpty()) {\n\t\t\t\tif(similarTargetEdges.size() > 1) { // more than one similar target?\n\t\t\t\t\tSystem.err.println(\"more than one similar target edge for \" + ctxt.vertex\n\t\t\t\t\t\t\t+ \": \" + similarTargetEdges); // TODO figure out what to do about this\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tassimilateAttributes(ctxt, target, similarTargetEdges.iterator().next().getTarget());\n\t\t\t\tLFVertex hp = ctxt.getGraph().highestLFAncestorOf(target);\n\t\t\t\t\n\t\t\t\tif(hp == null || hp.equals(outgoing.getSource())) {\n\t\t\t\t\tcontext.getVertices(SHARED).add(target);\n\t\t\t\t\t\n\t\t\t\t\tctxt.vertex = target;\n\t\t\t\t\tctxt.vertices = context.copyVertices(LOCAL_ANCESTOR, PREDICATES);\n\t\t\t\t\tctxt.parent.appendChild(createDisjunctiveElement(ctxt));\n\t\t\t\t\t\n\t\t\t\t\treturn; // stop after similar target found\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\t\n\tprivate void processSubstitutedPredicates(DLFContext context, LFEdge outgoing) {\n\t\tfinal String predicate = context.vertex.getPredicate();\n\t\tif(predicate != null) {\n\t\t\tSet alternates = new FilteredSet(\n\t\t\t\t\tgraphDifference.substitutionsBySourceFor(outgoing).keySet(),\n\t\t\t\t\tnew DelegatedFilter(new Filter(){\n\t\t\t\t\t\t@Override\n\t\t\t\t\t\tpublic boolean allows(String s) {\n\t\t\t\t\t\t\treturn !predicate.equals(s);\n\t\t\t\t\t\t}\n\t\t\t\t\t}) {\n\t\t\t\t\t\t@Override\n\t\t\t\t\t\tpublic String delegateValueFor(LFVertex e) {\n\t\t\t\t\t\t\treturn e.getPredicate();\n\t\t\t\t\t\t}\n\t\t\t\t\t});\n\t\t\t\n\t\t\tif(alternates.isEmpty()) { // the simple case, no other predicates involved\n\t\t\t\tsetPredicateName(context);\n\t\t\t}\n\t\t\telse { // add alternates as choice, with predicate an option\n\t\t\t\tDLFContext ctxt = context.copy();\n\t\t\t\tctxt.getVertices(PREDICATES).add(ctxt.vertex);\n\t\t\t\t\n\t\t\t\tctxt.parent = addChoice(ctxt);\n\t\t\t\taddAttributes(ctxt, PRED_ATTR, predicate);\n\t\t\t\t\n\t\t\t\tfor(LFVertex ap : alternates) {\n\t\t\t\t\taddAttributes(ctxt, PRED_ATTR, ap.getPredicate());\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\t\n\tprivate Element addRelation(DLFContext context, LFEdgeLabel label) {\n\t\tElement newRel = addElement(context, RELATION_TAG);\n\t\tnewRel.setAttribute(NAME_ATTR, label.getName());\n\t\t\n\t\treturn newRel;\n\t}\n\t\n\tprivate Element addOptional(DLFContext context) {\n\t\treturn addElement(context, OPTIONAL_TAG);\n\t}\n\t\n\tprivate Element addChoice(DLFContext context) {\n\t\treturn addElement(context, CHOICE_TAG);\n\t}\n\t\n\tprivate Element addElement(DLFContext context, String elementName) {\n\t\tElement newEl = document.createElement(elementName);\n\t\tcontext.parent.appendChild(newEl);\n\t\t\n\t\treturn newEl;\n\t}\n\t\n\tprivate Element addAttributes(DLFContext context, String name, String value) {\n\t\tElement newAtts = document.createElement(ATTS_TAG);\n\t\tcontext.parent.appendChild(newAtts);\n\t\t\n\t\tnewAtts.setAttribute(name, value);\n\t\t\n\t\treturn newAtts;\n\t}\n\t\n\tprivate Element addAttributes(DLFContext context, Map attributes) {\n\t\tElement newAtts = document.createElement(ATTS_TAG);\n\t\tcontext.parent.appendChild(newAtts);\n\t\t\n\t\tfor(Mode m : attributes.keySet()) {\n\t\t\tString n = m.getName();\n\t\t\tif(!n.equals(PRED_ATTR)) { // TODO does this ever happen?\n\t\t\t\tnewAtts.setAttribute(n, attributes.get(m).getName());\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn newAtts;\n\t}\n\t\n\tprivate void fixLabelReferences(Element newNode) {\n\t\tNodeList rels = newNode.getChildNodes();\n\t\tint rlen = rels.getLength();\n\t\tMap refRels = new HashMap(rlen);\n\t\t\n\t\tfor(int k = 0; k < rlen; k++) {\n\t\t\tNode n = rels.item(k);\n\t\t\tif(n != null && n.getNodeType() == Node.ELEMENT_NODE && n.getNodeName().equals(RELATION_TAG)) {\n\t\t\t\tElement ne = (Element)n;\n\t\t\t\tNode m = ne.getFirstChild();\n\t\t\t\tif(m != null && m.getNodeType() == Node.ELEMENT_NODE && m.getNodeName().equals(NODE_TAG)) {\n\t\t\t\t\tElement me = (Element)m;\n\t\t\t\t\tString l = ne.getAttribute(NAME_ATTR);\n\t\t\t\t\tString idref = me.getAttribute(IDREF_ATTR);\n\t\t\t\t\t\n\t\t\t\t\tif(idref == null || idref.length() == 0) {\n\t\t\t\t\t\tString id = me.getAttribute(ID_ATTR);\n\t\t\t\t\t\tif(id != null && id.length() > 0) {\n\t\t\t\t\t\t\trefRels.put(l, id);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tif(idref.equals(refRels.get(l))) {\n\t\t\t\t\t\t\tnewNode.removeChild(n);\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse {\n\t\t\t\t\t\t\trefRels.put(l, idref);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\t\n\tprivate void fixOptions(DLFContext context, LFEdgeLabel label) {\n\t\tString cPred = context.vertex.getPredicate();\n\t\tif(cPred == null) {\n\t\t\treturn;\n\t\t}\n\t\t\n\t\tNodeList ncs = context.parent.getChildNodes();\n\t\tfor(int j = 0; j < ncs.getLength(); j++) {\n\t\t\tNode c = ncs.item(j);\n\t\t\tif(c != null && c.getNodeType() == Node.ELEMENT_NODE && c.getNodeName().equals(OPTIONAL_TAG)) {\n\t\t\t\tNodeList rs = c.getChildNodes();\n\t\t\t\tfor(int k = 0; k < rs.getLength(); k++) {\n\t\t\t\t\tNode r = rs.item(k);\n\t\t\t\t\tif(r != null && r.getNodeType() == Node.ELEMENT_NODE\n\t\t\t\t\t\t\t&& r.getNodeName().equals(RELATION_TAG)) {\n\t\t\t\t\t\tElement re = (Element)r;\n\t\t\t\t\t\tif(label.getName().equals(re.getAttribute(NAME_ATTR))) {\n\t\t\t\t\t\t\tNode d = re.getFirstChild();\n\t\t\t\t\t\t\tif(d != null && d.getNodeType() == Node.ELEMENT_NODE\n\t\t\t\t\t\t\t\t\t&& d.getNodeName().equals(NODE_TAG)\n\t\t\t\t\t\t\t\t\t&& cPred.equals(((Element)d).getAttribute(PRED_ATTR))) {\n\t\t\t\t\t\t\t\tcontext.parent.removeChild(c); \n\t\t\t\t\t\t\t\tbreak; // don't try to remove more than once, throws DOMException\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\t\n\tprivate void setPredicateName(DLFContext context) {\n\t\tString p = context.vertex.getPredicate(); \n\t\tif(p != null) {\n\t\t\tcontext.parent.setAttribute(PRED_ATTR, p);\n\t\t\tcontext.getVertices(PREDICATES).add(context.vertex);\n\t\t}\n\t}\n\t\t\n\tprivate void addNonPredAttributes(DLFContext context) {\n\t\tfor(Mode m : context.vertex.attributeNames()) {\n\t\t\tString n = m.getName();\n\t\t\tif(!n.equals(PRED_ATTR)) { // TODO is this attribute ever present??\n\t\t\t\tcontext.parent.setAttribute(n, context.vertex.getAttributeValue(m).getName());\n\t\t\t}\n\t\t}\n\t}\n\t\n\tprivate void assimilateAttributes(DLFContext context, LFVertex one, LFVertex two) {\n\t\t// copy attribute maps for both vertices\n\t\tMap oneAttrs = new HashMap(one.getAttributeMap()),\n\t\t\t\ttwoAttrs = new HashMap(two.getAttributeMap());\n\t\t\n\t\t// add all attributes common to both vertices, remove from both maps\n\t\tIterator> i = oneAttrs.entrySet().iterator();\n\t\twhile(i.hasNext()) {\n\t\t\tMap.Entry e = i.next();\n\t\t\tSet> tes = twoAttrs.entrySet();\t\t\t\n\t\t\t\n\t\t\tif(tes.contains(e)) {\n\t\t\t\tcontext.parent.setAttribute(e.getKey().getName(), e.getValue().getName());\n\t\t\t\t\n\t\t\t\ti.remove();\n\t\t\t\ttes.remove(e);\n\t\t\t}\n\t\t}\n\t\t\n\t\tif(oneAttrs.isEmpty()) { // if first is empty, add second as optional\n\t\t\tif(!twoAttrs.isEmpty()) {\n\t\t\t\tDLFContext ctxt = context.copy(true);\n\t\t\t\tctxt.parent = addOptional(context);\n\t\t\t\t\n\t\t\t\taddAttributes(ctxt, twoAttrs);\n\t\t\t}\n\t\t}\n\t\telse if(twoAttrs.isEmpty()) { // some attributes remain for first vertex\n\t\t\taddAttributes(context.copy(), oneAttrs);\n\t\t}\n\t\telse { // both are non-empty, make choice\n\t\t\tDLFContext ctxt = context.copy(true);\n\t\t\tctxt.parent = addChoice(context);\n\t\t\t\n\t\t\taddAttributes(ctxt, oneAttrs);\n\t\t\taddAttributes(ctxt, twoAttrs);\n\t\t}\n\t}\n\n\tstatic class SimilarTargetVertexFilter implements Filter {\n\t\tLFVertex vertex;\n\t\t\n\t\tSimilarTargetVertexFilter(LFVertex vertex) {\n\t\t\tthis.vertex = vertex;\n\t\t}\n\t\t\n\t\t@Override\n\t\tpublic boolean allows(LFVertex v) {\n\t\t\tString p = vertex.getPredicate();\n\t\t\treturn p != null && p.equals(v.getPredicate());\n\t\t}\n\t}\n\t\n\tstatic class SimilarTargetEdgeFilter extends DelegatedFilter {\n\n\t\tLFEdgeLabel label;\n\t\t\n\t\tSimilarTargetEdgeFilter(LFVertex vertex, LFEdgeLabel label) {\n\t\t\tsuper(new SimilarTargetVertexFilter(vertex));\n\t\t\tthis.label = label;\n\t\t}\n\t\t\n\t\t@Override\n\t\tpublic boolean allows(LFEdge e) {\n\t\t\treturn super.allows(e) && label.equals(e.getLabel());\n\t\t}\n\n\t\t@Override\n\t\tpublic LFVertex delegateValueFor(LFEdge e) {\n\t\t\treturn e.getTarget();\n\t\t}\n\t}\n\t\n\tclass TerminalFilter implements Filter {\n\t\tLFGraph graph;\n\t\t\n\t\tTerminalFilter(LFGraph graph) {\n\t\t\tthis.graph = graph;\n\t\t}\n\n\t\t@Override\n\t\tpublic boolean allows(LFVertex e) {\n\t\t\treturn graph.outDegreeOf(e) == 0;\n\t\t}\n\t}\n\t\n\tstatic enum VertexType {\n\t\tLOCAL_ANCESTOR, OPTIONAL, PREDICATES, SHARED, VISITED;\n\t}\n\t\n\tclass DLFContext {\n\t\tPhrasePosition graphPosition;\n\t\tLFVertex vertex;\n\t\tElement parent;\n\t\t\n\t\tprivate Map> vertices;\n\t\t\n\t\tDLFContext(PhrasePosition graphPosition, LFVertex vertex, Element parent) {\n\t\t\tthis(graphPosition, vertex, parent, new EnumMap>(VertexType.class));\n\t\t}\n\t\t\n\t\tDLFContext(PhrasePosition graphPosition, LFVertex vertex, Element parent, Map> vertices) {\n\t\t\tthis.graphPosition = graphPosition;\n\t\t\tthis.vertex = vertex;\n\t\t\tthis.parent = parent;\n\t\t\tthis.vertices = vertices;\n\t\t}\n\t\t\n\t\tLFGraph getGraph() {\n\t\t\treturn graphDifference.get(graphPosition);\n\t\t}\n\t\t\n\t\tDLFContext copy() {\n\t\t\treturn copy(false);\n\t\t}\n\t\t\n\t\tDLFContext copy(boolean copyVertices) {\n\t\t\treturn copyVertices ? copyWithVertexMask(VertexType.values())\n\t\t\t\t\t: new DLFContext(graphPosition, vertex, parent, vertices);\n\t\t}\n\t\t\n\t\tDLFContext copyWithVertexMask(VertexType... vertexType) {\n\t\t\treturn new DLFContext(graphPosition, vertex, parent, copyVertices(vertexType));\n\t\t}\n\t\t\n\t\tMap> copyVertices() {\n\t\t\treturn copyVertices(VertexType.values());\n\t\t}\n\t\t\n\t\tMap> copyVertices(VertexType... vertexType) {\n\t\t\tMap> m = new EnumMap>(vertices);\n\t\t\tm.keySet().retainAll(Arrays.asList(vertexType));\n\t\t\t\n\t\t\treturn m;\n\t\t}\n\t\t\n\t\tSet getVertices(VertexType vertexType) {\n\t\t\tSet vs = vertices.get(vertexType);\n\t\t\t\n\t\t\tif(vs == null) {\n\t\t\t\tvs = new HashSet();\n\t\t\t\tvertices.put(vertexType, vs);\n\t\t\t}\n\t\t\t\n\t\t\treturn vs;\n\t\t}\n\t\t\n\t\tboolean addVertex(LFVertex vertex, VertexType vertexType) {\n\t\t\treturn getVertices(vertexType).add(vertex);\n\t\t}\n\t}\n\t\n\tstatic class LFGraphIterator extends DepthFirstIterator {\n\t\tLFGraphIterator(LFGraph graph) {\n\t\t\tsuper(graph);\n\t\t}\n\t\t\n\t\tLFGraphIterator(LFGraph graph, LFVertex startVertex) {\n\t\t\tsuper(graph, startVertex);\n\t\t}\n\t}\n}\n" }, { "path": "src/opennlp/ccg/disjunctivizer/EdgeMatchFilter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.disjunctivizer;\n\nimport static opennlp.ccg.disjunctivizer.MatchType.LABEL_MATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.LABEL_MISMATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.SOURCE_MATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.SOURCE_MISMATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.SOURCE_PREDICATE_MATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.SOURCE_PREDICATE_MISMATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.TARGET_MATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.TARGET_MISMATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.TARGET_PREDICATE_MATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.TARGET_PREDICATE_MISMATCH;\n\nimport java.util.Arrays;\nimport java.util.Collection;\nimport java.util.EnumSet;\n\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.util.CompositeFilter;\nimport opennlp.ccg.util.Filter;\nimport opennlp.ccg.util.InverseFilter;\n\n\n/**\n * A filter for LF edges based on a set of {@linkplain #getMatchTypes() match type criteria}. Instances of\n * this class compare a specified {@linkplain #getBasis() basis edge} based on their match type criteria.\n * This class extends {@link CompositeFilter}, and the constructors add various filters as members depending on \n * the criteria in effect. \n *

\n * Edge match filters use the following match types as criteria in addition to the ones used by\n * {@link VertexMatchFilter}: {@link MatchType#LABEL_MATCH}, and {@link MatchType#LABEL_MISMATCH}. \n * \n * @see VertexMatchFilter\n * @see LabelMatchFilter\n * \n * @author Scott Martin\n */\npublic class EdgeMatchFilter extends CompositeFilter {\n\t\n\tLFEdge basis;\n\tfinal EnumSet matchTypes;\n\t\n\t/**\n\t * Creates a new edge match filter based on the specified edge and criteria.\n\t * @see #EdgeMatchFilter(LFEdge, Collection)\n\t */\n\tpublic EdgeMatchFilter(LFEdge basis, MatchType... matchTypes) {\n\t\tthis(basis, Arrays.asList(matchTypes));\n\t}\n\t\n\t/**\n\t * Creates a new edge match filter based on the specified edge, using the specified\n\t * match type criteria. Depending on the criteria, this constructor adds instances of\n\t * {@link VertexMatchFilter} and {@link LabelMatchFilter} (or their inverses) to the\n\t * set of filters composing it.\n\t * \n\t * @param basis The LF edge to use for comparison.\n\t * @param matchTypes The comparison criteria, used to populate this composite filter.\n\t * \n\t * @throws IllegalArgumentException if basis is null.\n\t */\n\tpublic EdgeMatchFilter(LFEdge basis, Collection matchTypes) {\n\t\tsuper();\n\t\t\n\t\tcheckBasis(basis);\n\t\tthis.basis = basis;\n\t\tthis.matchTypes = EnumSet.copyOf(matchTypes);\n\t\t\t\t\n\t\tfor(MatchType t : matchTypes) {\n\t\t\tFilter f = null;\n\t\t\t\n\t\t\tif(t == LABEL_MATCH || t == LABEL_MISMATCH) {\n\t\t\t\tf = new LabelMatchFilter(basis.getLabel());\n\t\t\t\tif(t == LABEL_MISMATCH) {\n\t\t\t\t\tf = new InverseFilter(f);\n\t\t\t\t}\n\t\t\t}\n\t\t\telse if(t == SOURCE_MATCH || t == SOURCE_MISMATCH\n\t\t\t\t\t|| t == SOURCE_PREDICATE_MATCH || t == SOURCE_PREDICATE_MISMATCH) {\n\t\t\t\tf = new VertexMatchFilter(basis.getSource(), t);\n\t\t\t}\n\t\t\telse if(t == TARGET_MATCH || t == TARGET_MISMATCH\n\t\t\t\t\t|| t == TARGET_PREDICATE_MATCH || t == TARGET_PREDICATE_MISMATCH) {\n\t\t\t\tf = new VertexMatchFilter(basis.getTarget(), t);\n\t\t\t}\n\t\t\t\n\t\t\tif(f != null) {\n\t\t\t\taddFilter(f);\n\t\t\t}\n\t\t}\n\t}\n\t\n\tprivate void checkBasis(LFEdge basis) {\n\t\tif(basis == null) {\n\t\t\tthrow new IllegalArgumentException(\"basis is null\");\n\t\t}\n\t}\n\t\n\t/**\n\t * Gets the edge that comparisons are based on.\n\t */\n\tpublic LFEdge getBasis() {\n\t\treturn basis;\n\t}\n\n\t/**\n\t * Sets the edge used for comparisons.\n\t * @throws IllegalArgumentException if basis is null.\n\t */\n\tpublic void setBasis(LFEdge basis) {\n\t\tcheckBasis(basis);\n\t\tthis.basis = basis;\n\t}\n\n\t/**\n\t * Gets the match type criteria used by this edge match filter.\n\t */\n\tpublic EnumSet getMatchTypes() {\n\t\treturn matchTypes;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/disjunctivizer/FilteredLFEdgeSet.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.disjunctivizer;\n\nimport java.util.AbstractCollection;\nimport java.util.Collection;\nimport java.util.Iterator;\nimport java.util.Set;\n\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.hylo.graph.LFEdgeLabel;\nimport opennlp.ccg.hylo.graph.LFVertex;\nimport opennlp.ccg.util.Filter;\nimport opennlp.ccg.util.FilteredSet;\n\n\n/**\n * A filtered set of LF edges. This class extends {@link FilteredSet} to provide the additional functionality\n * of edge-based views of its contents: {@link #sourceView()}, {@link #targetView()}, and {@link #labelView()},\n * which respectively get the set of source vertices, set of target vertices, and set of labels contained\n * in the edges in this set. All of these view sets are read-only, so that attempting to add or remove\n * elements from them (including via the iterator) throws {@link UnsupportedOperationException}.\n * \n * @author Scott Martin\n */\npublic class FilteredLFEdgeSet extends FilteredSet {\n\n\tprivate VertexView sourceView = null, targetView = null;\n\tprivate LabelView labelView = null;\n\t\n\t/**\n\t * Creates a new filtered edge set based on the specified underlying edge set and edge filter.\n\t */\n\tpublic FilteredLFEdgeSet(Set edges, Filter edgeFilter) {\n\t\tsuper(edges, edgeFilter);\n\t}\n\t\n\t/**\n\t * Gets a view of this filtered edge set as a set of LF vertices that are the \n\t * {@linkplain LFEdge#getSource() source vertices} for each edge in this set.\n\t * @return A set containing every LF vertex that is the source vertex of some edge in this set.\n\t * Note that the returned collection is immutable, and may contain duplicate vertices.\n\t */\n\tpublic Collection sourceView() {\n\t\treturn (sourceView == null) ? (sourceView = new VertexView(true)) : sourceView;\n\t}\n\t\n\t/**\n\t * Gets a view of this filtered edge set as a set of LF vertices that are the \n\t * {@linkplain LFEdge#getTarget() target vertices} for each edge in this set.\n\t * @return A set containing every LF vertex that is the target vertex of some edge in this set.\n\t * Note that the returned collection is immutable, and may contain duplicate vertices.\n\t */\n\tpublic Collection targetView() {\n\t\treturn (targetView == null) ? (targetView = new VertexView(false)) : targetView;\n\t}\n\t\n\t/**\n\t * Gets a view of this filtered edge set as a set of LF vertices that are the \n\t * {@linkplain LFEdge#getLabel() labels} for each edge in this set.\n\t * @return A set containing every LF edge label that is the label of some edge in this set.\n\t * Note that the returned collection is immutable, and may contain duplicate labels.\n\t */\n\tpublic Collection labelView() {\n\t\treturn (labelView == null) ? (labelView = new LabelView()) : labelView;\n\t}\n\t\n\tabstract class ComponentView extends AbstractCollection {\n\t\t\n\t\tabstract T componentOf(LFEdge edge);\n\t\t\n\t\t@Override\n\t\tpublic Iterator iterator() {\n\t\t\treturn new Iterator() {\n\t\t\t\tprivate Iterator i = FilteredLFEdgeSet.this.iterator();\n\t\t\t\t\n\t\t\t\t@Override\n\t\t\t\tpublic boolean hasNext() {\n\t\t\t\t\treturn i.hasNext();\n\t\t\t\t}\n\n\t\t\t\t@Override\n\t\t\t\tpublic T next() {\n\t\t\t\t\t// don't have to worry whether hasNext() is true, iterator should throw exception if not\n\t\t\t\t\treturn componentOf(i.next());\n\t\t\t\t}\n\n\t\t\t\t@Override\n\t\t\t\tpublic void remove() {\n\t\t\t\t\tthrow new UnsupportedOperationException();\n\t\t\t\t}\n\t\t\t};\n\t\t}\n\n\t\t@Override\n\t\tpublic int size() {\n\t\t\treturn FilteredLFEdgeSet.this.size();\n\t\t}\n\t}\n\t\n\tclass VertexView extends ComponentView {\n\n\t\tboolean source;\n\t\t\n\t\tVertexView(boolean source) {\n\t\t\tthis.source = source;\n\t\t}\n\t\t\n\t\t@Override\n\t\tLFVertex componentOf(LFEdge edge) {\n\t\t\treturn source ? edge.getSource() : edge.getTarget();\n\t\t}\t\t\n\t}\n\t\n\tclass LabelView extends ComponentView {\n\n\t\t@Override\n\t\tLFEdgeLabel componentOf(LFEdge edge) {\n\t\t\treturn edge.getLabel();\n\t\t}\n\t}\n}\n" }, { "path": "src/opennlp/ccg/disjunctivizer/LFGraphDifference.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.disjunctivizer;\n\nimport static opennlp.ccg.alignment.PhrasePosition.A;\nimport static opennlp.ccg.alignment.PhrasePosition.B;\nimport static opennlp.ccg.disjunctivizer.MatchType.SOURCE_ALIGNED;\nimport static opennlp.ccg.disjunctivizer.MatchType.SOURCE_MATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.SOURCE_UNALIGNED;\nimport static opennlp.ccg.disjunctivizer.MatchType.TARGET_ALIGNED;\nimport static opennlp.ccg.disjunctivizer.MatchType.TARGET_UNALIGNED;\n\nimport java.util.AbstractMap;\nimport java.util.AbstractSet;\nimport java.util.Collections;\nimport java.util.Iterator;\nimport java.util.LinkedHashSet;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.Map.Entry;\n\nimport opennlp.ccg.alignment.Alignment;\nimport opennlp.ccg.alignment.PhrasePosition;\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.hylo.graph.LFGraph;\nimport opennlp.ccg.hylo.graph.LFVertex;\nimport opennlp.ccg.util.CompositeFilter;\nimport opennlp.ccg.util.Filter;\nimport opennlp.ccg.util.FilteredMap;\nimport opennlp.ccg.util.FilteredSet;\nimport opennlp.ccg.util.VisitedFilter;\n\n/**\n * Represents the difference between two {@link LFGraph}s that characterizes their difference as a set \n * of edits: either {@link #inserts()}, {@link #deletes()}, or {@link #substitutions()}. These edits are\n * determined by a specified {@linkplain #getAlignment() alignment} between the phrases the graphs\n * are supposed to represent. All of the sets of edges returned by this class, and the convenience maps\n * build on top of them, are read-only. Attempting to add or remove elements or keys from any of these\n * (including via any of the iterators) throws an {@link UnsupportedOperationException}.\n *

\n * This class also provides the convenience methods {@link #insertsFor(LFVertex)} and \n * {@link #deletesFor(LFVertex)}, for getting just the inserts or deletes for a specified vertex.\n * The convenience methods {@link #substitutionsFor(LFEdge)} gets the set of substitutions correspoding\n * to a given edge, and {@link #substitutionsBySource()} and {@link #substitutionsBySourceFor(LFEdge)}\n * are similar methods that provide maps whose keys are the source vertices of the substituted edges. \n * \n * @author Scott Martin\n */\npublic class LFGraphDifference {\n\n\tfinal LFGraph a, b;\n\tfinal Alignment alignment;\n\t\n\tprivate Set deletes, inserts, substitutions;\n\t\n\t/**\n\t * Creates a new graph difference between a and b, as determined by the specified\n\t * alignment.\n\t * @param a The {@linkplain PhrasePosition#A A-position} graph.\n\t * @param b The {@linkplain PhrasePosition#B B-position} graph.\n\t * @param alignment An alignment between a and b where the\n\t * {@linkplain PhrasePosition#A A-position} indices are understood to correspond to a and\n\t * {@linkplain PhrasePosition#B B-position} indices are understood to correspond to b.\n\t * @throws IllegalArgumentException If either graph is null, or if the alignment is\n\t * null.\n\t */\n\tpublic LFGraphDifference(LFGraph a, LFGraph b, Alignment alignment) {\n\t\tcheckGraph(a, A);\n\t\tcheckGraph(b, B);\n\t\t\n\t\tif(alignment == null) {\n\t\t\tthrow new IllegalArgumentException(\"alignment is null\");\n\t\t}\n\t\t\n\t\tthis.a = a;\n\t\tthis.b = b;\n\t\tthis.alignment = alignment;\n\t}\n\t\n\tprivate void checkGraph(LFGraph g, PhrasePosition pos) {\n\t\tif(g == null) {\n\t\t\tthrow new IllegalArgumentException(pos.name() + \" graph is null\");\n\t\t}\n\t}\n\n\t/**\n\t * Gets the {@linkplain PhrasePosition#A A-position} graph.\n\t */\n\tpublic LFGraph getA() {\n\t\treturn get(A);\n\t}\n\n\t/**\n\t * Gets the {@linkplain PhrasePosition#B B-position} graph.\n\t */\n\tpublic LFGraph getB() {\n\t\treturn get(B);\n\t}\n\t\n\t/**\n\t * Gets the graph in the specified position.\n\t * @param position The position to retrieve a graph for.\n\t * @return The value of {@link #getA()} if position is {@link PhrasePosition#A}, and\n\t * the value of {@link #getB()} otherwise.\n\t */\n\tpublic LFGraph get(PhrasePosition position) {\n\t\treturn (position == A) ? a : b;\n\t}\n\n\t/**\n\t * Gets the alignment used to determine the edits between the two graphs.\n\t */\n\tpublic Alignment getAlignment() {\n\t\treturn alignment;\n\t}\n\t\n\t/**\n\t * Computes a hash code for this graph difference based on its graphs and the\n\t * alignment between them.\n\t */\n\t@Override\n\tpublic int hashCode() {\n\t\treturn 31 * a.hashCode() + b.hashCode() + alignment.hashCode(); \n\t}\n\n\t/**\n\t * Tests whether this LF graph difference is equivalent to another by comparing their\n\t * graphs and the alignment between them.\n\t */\n\t@Override\n\tpublic boolean equals(Object obj) {\n\t\tif(obj instanceof LFGraphDifference) {\n\t\t\tLFGraphDifference diff = (LFGraphDifference)obj;\n\t\t\treturn a.equals(diff.a) && b.equals(diff.b) && alignment.equals(diff.alignment);\n\t\t}\n\t\t\n\t\treturn false;\n\t}\n\t\n\t/**\n\t * Gets a string representation of this graph difference.\n\t */\n\t@Override\n\tpublic String toString() {\n\t\tStringBuilder sb = new StringBuilder(\"difference for graphs: \");\n\t\t\n\t\tfor(PhrasePosition pos : PhrasePosition.values()) {\n\t\t\tsb.append(pos);\n\t\t\tsb.append(\": \");\n\t\t\tsb.append(get(pos));\n\t\t\tsb.append(\", \");\n\t\t}\n\t\t\n\t\tsb.append(\"alignment: \");\n\t\tsb.append(alignment.toString());\n\t\t\n\t\treturn sb.toString();\n\t}\n\n\t/**\n\t * Gets an LF graph difference that is the reverse of the present one.\n\t * @return An LF graph difference whose {@linkplain PhrasePosition#A A-position} graph is the value of this\n\t * difference's {@link #getB()}, whose {@linkplain PhrasePosition#B B-position} graph is the value of this\n\t * difference's {@link #getA()}, and whose alignments are the {@linkplain Alignment#reverse() reverse} of\n\t * this difference's {@link #getAlignment()}. \n\t */\n\tpublic LFGraphDifference reverse() {\n\t\treturn new LFGraphDifference(b, a, alignment.reverse());\n\t}\n\t\n\t/**\n\t * Gets the deletes for this graph difference.\n\t * @return The set of edges in the {@linkplain PhrasePosition#A A-position} graph that have an aligned \n\t * {@linkplain LFEdge#getSource() source vertex} and an unaligned \n\t * {@linkplain LFEdge#getTarget() target vertex}.\n\t * \n\t * @see AlignedEdgeFilter\n\t */\n\tpublic Set deletes() {\n\t\treturn (deletes == null) ? (deletes = doDeletes(A)) : deletes;\n\t}\n\t\n\t/**\n\t * Gets the inserts for this graph difference.\n\t * @return The set of edges in the {@linkplain PhrasePosition#B B-position} graph that have an aligned \n\t * {@linkplain LFEdge#getSource() source vertex} and an unaligned \n\t * {@linkplain LFEdge#getTarget() target vertex}.\n\t * \n\t * @see AlignedEdgeFilter\n\t */\n\tpublic Set inserts() {\n\t\treturn (inserts == null) ? (inserts = doDeletes(B)) : inserts;\n\t}\n\t\n\tSet doDeletes(PhrasePosition keyPosition) {\n\t\treturn Collections.unmodifiableSet(new FilteredLFEdgeSet(get(keyPosition).edgeSet(),\n\t\t\t\tnew AlignedEdgeFilter(alignment.asMap(keyPosition).keySet(),\n\t\t\t\t\t\tSOURCE_ALIGNED, TARGET_UNALIGNED)));\n\t}\n\t\n\t/**\n\t * Gets the inserts for a specified vertex.\t\n\t * @param vertex The vertex to return the inserts for.\n\t * @return The subset of {@link #inserts()} whose {@linkplain LFEdge#getSource() source} index is among the\n\t * {@linkplain Alignment#getTargets(Integer) targets} for the specified vertex, or {@link Collections#EMPTY_SET}\n\t * if none exist.\n\t * \n\t * @see AlignedEdgeFilter\n\t */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic Set insertsFor(LFVertex vertex) {\n\t\tSet indices = alignment.getTargets(vertex.getIndex());\n\t\t\n\t\treturn (indices.isEmpty()) ? Collections.EMPTY_SET\n\t\t\t\t: new FilteredLFEdgeSet(inserts(), new AlignedEdgeFilter(indices, SOURCE_ALIGNED));\n\t}\n\t\n\t/**\n\t * Gets the deletes for a specified vertex.\n\t * @param vertex The vertex to get the deletes for.\n\t * @return The subset of {@link #deletes()} whose {@linkplain LFEdge#getSource() source vertex} is\n\t * the specified vertex.\n\t * \n\t * @see VertexMatchFilter\n\t */\n\tpublic Set deletesFor(LFVertex vertex) {\n\t\treturn new FilteredLFEdgeSet(deletes(), new VertexMatchFilter(vertex, SOURCE_MATCH));\n\t}\n\t\n\t/**\n\t * Gets the substitutions for this graph difference.\n\t * @return The subset of the {@linkplain PhrasePosition#B B-position} graph's edges for which there\n\t * exists an edge in the {@linkplain PhrasePosition#A A-position} graph that meets the following\n\t * conditions:\n\t *

    \n\t * \t
  1. The B edge's source is aligned to the A edge's source, but the B edge's target is not.
    2. \n\t * \t
  2. The B edge's target is aligned to the A edge's target, but the B edge's source is not.
    3. \n\t *
\n\t * \n\t * @see CompositeFilter\n\t */\n\tpublic Set substitutions() {\n\t\tif(substitutions == null) {\n\t\t\tsubstitutions = new LinkedHashSet();\n\t\t\tSet bEdges = b.edgeSet();\n\t\t\t\n\t\t\tAlignedEdgeFilter sourceFilter = null, targetFilter = null;\n\t\t\tCompositeFilter filter = new CompositeFilter();\n\t\t\t\n\t\t\tfor(LFEdge aEdge : a.edgeSet()) {\n\t\t\t\tSet sMaps = alignment.getTargets(aEdge.getSource().getIndex()),\n\t\t\t\t\t\t\t\ttMaps = alignment.getTargets(aEdge.getTarget().getIndex());\n\t\t\t\t\n\t\t\t\tif(!sMaps.isEmpty() && !tMaps.isEmpty()) {\n\t\t\t\t\tif(sourceFilter == null) {\n\t\t\t\t\t\tsourceFilter = new AlignedEdgeFilter(sMaps, SOURCE_ALIGNED, TARGET_UNALIGNED);\n\t\t\t\t\t\ttargetFilter = new AlignedEdgeFilter(tMaps, TARGET_ALIGNED, SOURCE_UNALIGNED);\n\t\t\t\t\t\t\n\t\t\t\t\t\tfilter.addFilter(sourceFilter);\n\t\t\t\t\t\tfilter.addFilter(targetFilter);\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tsourceFilter.setAlignmentIndices(sMaps);\n\t\t\t\t\t\ttargetFilter.setAlignmentIndices(tMaps);\n\t\t\t\t\t}\n\t\t\t\t\t\n\t\t\t\t\tsubstitutions.addAll(new FilteredLFEdgeSet(bEdges, filter));\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn Collections.unmodifiableSet(substitutions);\n\t}\n\t\n\t/**\n\t * Gets the substitutions for the specified edge.\n\t * @param edge The edge to get substitutions for.\n\t * @return The subset of {@link #substitutions()} whose source is aligned to the edge's source and\n\t * whose target is aligned to the edge's target, or {@link Collections#EMPTY_SET} if none exist. \n\t */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic Set substitutionsFor(LFEdge edge) {\n\t\tSet srcMapsTo = alignment.getTargets(edge.getSource().getIndex()),\n\t\t\ttrgMapsTo = alignment.getTargets(edge.getTarget().getIndex());\n\t\t\n\t\treturn (srcMapsTo.isEmpty() || trgMapsTo.isEmpty()) ? Collections.EMPTY_SET\n\t\t\t\t: Collections.unmodifiableSet(new FilteredLFEdgeSet(substitutions(),\n\t\t\t\t\t\tnew CompositeFilter(new AlignedEdgeFilter(srcMapsTo, SOURCE_ALIGNED),\n\t\t\t\t\t\t\t\tnew AlignedEdgeFilter(trgMapsTo, TARGET_ALIGNED))));\n\t}\n\t\n\t/**\n\t * Gets a map view of the substitutions in this graph difference.\n\t * @return A map whose keys are the source vertices in the set of {@link #substitutions()} and whose values\n\t * are the edges whose {@linkplain LFEdge#getSource() source vertex} is the same as the corresponding key.\n\t * If there are no substitutions, {@link Collections#EMPTY_MAP} is returned.\n\t *

\n\t * Note that the returned map is\n\t * read-only, that is, both its {@link Map#put(Object, Object)} method and its \n\t * {@linkplain Map#entrySet() entry set}'s iterator's {@link Iterator#remove()} method throw an\n\t * {@link UnsupportedOperationException}. Also, the members of the returned map's entry set are immutable,\n\t * so that their {@link Entry#setValue(Object)} methods also throw an {@link UnsupportedOperationException}.\n\t */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic Map> substitutionsBySource() {\n\t\tSet subs = substitutions();\n\t\t\n\t\treturn subs.isEmpty() ? Collections.EMPTY_MAP\n\t\t\t\t: Collections.unmodifiableMap(\n\t\t\t\t\t\tnew FilteredMap>(new SourceView(),\n\t\t\t\t\t\t\t\tnew VisitedFilter()));\n\t}\n\t\n\t/**\n\t * Gets a map view of the substitutions for the specified edge.\n\t * @param edge The edge to get substitutions for.\n\t * @return The subset of {@link #substitutionsBySource()} in which the keys are aligned to the specified edge's\n\t * source and the associated values' targets are aligned to the specified edge's target. Since this map is\n\t * based on the one returned by {@link #substitutionsBySource()}, it is also read-only, and the same stipulations\n\t * apply to it.\n\t * \n\t * @see #substitutionsBySource()\n\t */\n\tpublic Map> substitutionsBySourceFor(LFEdge edge) {\n\t\tMap> subsBySource = substitutionsBySource();\t\t\n\t\treturn subsBySource.isEmpty() ? subsBySource : new SubstitutedSourceView(subsBySource, edge);\n\t}\n\t\n\tclass SourceView extends AbstractMap {\n\n\t\t@Override\n\t\tpublic Set> entrySet() {\n\t\t\treturn new AbstractSet>() {\n\t\t\t\tSet subs = substitutions();\n\t\t\t\t\n\t\t\t\t@Override\n\t\t\t\tpublic int size() {\n\t\t\t\t\treturn subs.size();\n\t\t\t\t}\n\n\t\t\t\t@Override\n\t\t\t\tpublic Iterator> iterator() {\n\t\t\t\t\treturn new Iterator>() {\n\t\t\t\t\t\tprivate Iterator edgeIterator = null;\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t@Override\n\t\t\t\t\t\tpublic boolean hasNext() {\n\t\t\t\t\t\t\tif(edgeIterator == null) {\n\t\t\t\t\t\t\t\tedgeIterator = subs.iterator();\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\treturn edgeIterator.hasNext();\n\t\t\t\t\t\t}\n\n\t\t\t\t\t\t@Override\n\t\t\t\t\t\tpublic Entry next() {\n\t\t\t\t\t\t\tif(edgeIterator == null) {\n\t\t\t\t\t\t\t\tedgeIterator = subs.iterator();\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\tLFVertex src = edgeIterator.next().getSource();\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\treturn new SimpleImmutableEntry(src,\n\t\t\t\t\t\t\t\t\tnew FilteredLFEdgeSet(subs, new VertexMatchFilter(src, SOURCE_MATCH)));\n\t\t\t\t\t\t}\n\t\t\t\t\t\t\n\t\t\t\t\t\t@Override\n\t\t\t\t\t\tpublic void remove() { // subs.iterator() should be read-only, but just in case\n\t\t\t\t\t\t\tthrow new UnsupportedOperationException();\n\t\t\t\t\t\t}\n\t\t\t\t\t};\n\t\t\t\t}\n\t\t\t};\n\t\t}\n\t}\n\t\n\tclass SubstitutedSourceView extends AbstractMap> {\n\n\t\tMap> sourceView;\n\t\tLFEdge edge;\n\t\t\n\t\tprivate Set>> entrySet;\n\t\t\n\t\tSubstitutedSourceView(Map> sourceView, LFEdge edge) {\n\t\t\tthis.sourceView = sourceView;\n\t\t\tthis.edge = edge;\n\t\t}\n\t\t\n\t\t@Override\n\t\tpublic Set>> entrySet() {\n\t\t\treturn (entrySet == null) ? (entrySet = new EntrySet()) : entrySet;\n\t\t}\n\t\t\n\t\tclass EntrySet extends AbstractSet>> {\n\t\t\tprivate Set>> entries;\n\t\t\tSet srcMapsTo = alignment.getTargets(edge.getSource().getIndex()),\n\t\t\t\t\t\t\t\ttrgMapsTo = alignment.getTargets(edge.getTarget().getIndex());\n\t\t\t\n\t\t\tSet>> entries() {\n\t\t\t\tif(entries == null) {\n\t\t\t\t\tentries = new FilteredSet>>(\n\t\t\t\t\t\tsourceView.entrySet(),\n\t\t\t\t\t\tnew Filter>>() {\n\t\t\t\t\t\t\t@Override\n\t\t\t\t\t\t\tpublic boolean allows(Entry> e) {\n\t\t\t\t\t\t\t\tif(srcMapsTo.contains(e.getKey().getIndex())) {\n\t\t\t\t\t\t\t\t\tfor(LFEdge t : e.getValue()) {\n\t\t\t\t\t\t\t\t\t\tif(trgMapsTo.contains(t.getTarget().getIndex())) {\n\t\t\t\t\t\t\t\t\t\t\treturn true;\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\treturn false;\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t);\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\treturn entries;\n\t\t\t}\n\t\t\t\n\t\t\t@Override\n\t\t\tpublic int size() {\n\t\t\t\treturn entries().size();\n\t\t\t}\n\t\t\t\n\t\t\t@Override\n\t\t\tpublic Iterator>> iterator() {\n\t\t\t\treturn new Iterator>>() {\n\t\t\t\t\tprivate Iterator>> i = entries().iterator();\n\t\t\t\t\t\n\t\t\t\t\t@Override\n\t\t\t\t\tpublic boolean hasNext() {\n\t\t\t\t\t\treturn i.hasNext();\n\t\t\t\t\t}\n\n\t\t\t\t\t@Override\n\t\t\t\t\tpublic Entry> next() {\n\t\t\t\t\t\tEntry> e = i.next();\n\t\t\t\t\t\treturn new SimpleImmutableEntry>(\n\t\t\t\t\t\t\t\te.getKey(),\n\t\t\t\t\t\t\t\tnew FilteredLFEdgeSet(e.getValue(), new Filter() {\n\t\t\t\t\t\t\t\t\t@Override\n\t\t\t\t\t\t\t\t\tpublic boolean allows(LFEdge e) {\n\t\t\t\t\t\t\t\t\t\treturn trgMapsTo.contains(e.getTarget().getIndex());\n\t\t\t\t\t\t\t\t\t}\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t)\n\t\t\t\t\t\t);\n\t\t\t\t\t}\n\n\t\t\t\t\t@Override\n\t\t\t\t\tpublic void remove() { // source view is already read-only, so just in case\n\t\t\t\t\t\tthrow new UnsupportedOperationException();\t\t\t\t\t\t\n\t\t\t\t\t}\t\t\t\t\t\n\t\t\t\t};\n\t\t\t}\n\t\t}\n\t}\n}\n" }, { "path": "src/opennlp/ccg/disjunctivizer/LabelMatchFilter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.disjunctivizer;\n\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.hylo.graph.LFEdgeLabel;\nimport opennlp.ccg.util.Filter;\n\n\n/**\n * A filter for edges based on a comparison of their {@linkplain LFEdge#getLabel() labels}. Instances of\n * this class compare a specified {@linkplain #getBasis() basis edge label}, so that the \n * {@link #allows(LFEdge)} method returns true if it is\n * {@linkplain LFEdgeLabel#equals(Object) equivalent to} the specified edge's label. \n * \n * @author Scott Martin\n */\npublic class LabelMatchFilter implements Filter {\n\t\n\tLFEdgeLabel basis;\n\t\n\t/**\n\t * Creates a new label match filter that will compare edge labels to the specified label.\n\t * @param basis The label to be used for comparison.\n\t * @throws IllegalArgumentException if basis is null.\n\t */\n\tpublic LabelMatchFilter(LFEdgeLabel basis) {\n\t\tcheckBasis(basis);\n\t\tthis.basis = basis;\n\t}\n\n\tprivate void checkBasis(LFEdgeLabel basis) {\n\t\tif(basis == null) {\n\t\t\tthrow new IllegalArgumentException(\"basis is null\");\n\t\t}\n\t}\n\t\n\t/**\n\t * Gets the label used as the basis for comparison in the {@link #allows(LFEdge)} method.\n\t * @return The edge label specified at creation.\n\t * \n\t * @see #LabelMatchFilter(LFEdgeLabel)\n\t */\n\tpublic LFEdgeLabel getBasis() {\n\t\treturn basis;\n\t}\n\n\t/**\n\t * Sets the edge label used as the basis for comparison.\n\t * @throws IllegalArgumentException if basis is null.\n\t */\n\tpublic void setBasis(LFEdgeLabel basis) {\n\t\tcheckBasis(basis);\n\t\tthis.basis = basis;\n\t}\n\n\t/**\n\t * Tests whether the specified edge's label is equivalent to this filter's {@linkplain #getBasis() basis\n\t * edge}.\n\t * @param edge The edge to test.\n\t * @return true if the basis edge label is equivalent to edge.getLabel() based on a\n\t * comparison via their {@link LFEdgeLabel#equals(Object)} method.\n\t */\n\t@Override\n\tpublic boolean allows(LFEdge edge) {\n\t\treturn basis.equals(edge.getLabel());\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/disjunctivizer/MatchType.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.disjunctivizer;\n\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.hylo.graph.LFVertex;\n\n/**\n * A set of enum constants for identifying the matching criteria used by a {@link MatchTypeFilter}.\n * Not all match type filters will use all of the match type criteria contained in this enum.\n * \n * @see LFVertex\n * @see LFEdge\n * \n * @author Scott Martin\n */\npublic enum MatchType {\n\t/**\n\t * Matching source vertices.\n\t */\n\tSOURCE_MATCH,\n\t\n\t/**\n\t * Matching target vertices.\n\t */\n\tTARGET_MATCH,\n\t\n\t/**\n\t * Matching edge labels.\n\t */\n\tLABEL_MATCH,\n\t\n\t/**\n\t * Matching predicates for source vertices.\n\t */\n\tSOURCE_PREDICATE_MATCH,\n\t\n\t/**\n\t * Matching predicates for target vertices.\n\t */\n\tTARGET_PREDICATE_MATCH,\n\t\n\t/**\n\t * Mismatching source vertices.\n\t */\n\tSOURCE_MISMATCH,\n\t\n\t/**\n\t * Mismatching target vertices.\n\t */\n\tTARGET_MISMATCH,\n\t\n\t/**\n\t * Mismatching edge labels.\n\t */\n\tLABEL_MISMATCH,\n\t\n\t/**\n\t * Mismatching source vertex predicates.\n\t */\n\tSOURCE_PREDICATE_MISMATCH,\n\t\n\t/**\n\t * Mismatching target vertex predicates.\n\t */\n\tTARGET_PREDICATE_MISMATCH,\n\t\n\t/**\n\t * Source vertex is aligned.\n\t */\n\tSOURCE_ALIGNED,\n\t\n\t/**\n\t * Source vertex is not aligned.\n\t */\n\tSOURCE_UNALIGNED,\n\t\n\t/**\n\t * Target vertex is aligned.\n\t */\n\tTARGET_ALIGNED,\n\t\n\t/**\n\t * Target vertex is unaligned.\n\t */\n\tTARGET_UNALIGNED\n}\n" }, { "path": "src/opennlp/ccg/disjunctivizer/MatchTypeFilter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.disjunctivizer;\n\nimport java.util.Arrays;\nimport java.util.Collection;\nimport java.util.EnumSet;\n\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.util.Filter;\n\n/**\n * Abstract class for filters that allow {@link LFEdge}s based on criteria indicated by the set of\n * {@linkplain #getMatchTypes() match types} they use. \n * \n * @author Scott Martin\n *\n */\npublic abstract class MatchTypeFilter implements Filter {\n\t\n\t/**\n\t * The set of match types used as criteria by this filter.\n\t */\n\tprotected final EnumSet matchTypes;\n\n\t/**\n\t * Creates a new match type filter based on the specified match types.\n\t * @see #MatchTypeFilter(Collection)\n\t */\n\tprotected MatchTypeFilter(MatchType... matchTypes) {\n\t\tthis(Arrays.asList(matchTypes));\n\t}\n\t\n\t/**\n\t * Creates a new match type filter based on the specified match types.\n\t * @param matchTypes The collection of match types to use. The specified collection is \n\t * copied via {@link EnumSet#copyOf(Collection)}.\n\t */\n\tprotected MatchTypeFilter(Collection matchTypes) {\n\t\tthis.matchTypes = EnumSet.copyOf(matchTypes);\n\t}\n\n\t/**\n\t * Gets the match types used by this match type filter.\n\t */\n\tpublic EnumSet getMatchTypes() {\n\t\treturn matchTypes;\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/disjunctivizer/VertexMatchFilter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.disjunctivizer;\n\nimport static opennlp.ccg.disjunctivizer.MatchType.SOURCE_MATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.SOURCE_MISMATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.SOURCE_PREDICATE_MATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.SOURCE_PREDICATE_MISMATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.TARGET_MATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.TARGET_MISMATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.TARGET_PREDICATE_MATCH;\nimport static opennlp.ccg.disjunctivizer.MatchType.TARGET_PREDICATE_MISMATCH;\n\nimport java.util.Collection;\n\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.hylo.graph.LFVertex;\n\n\n/**\n * A filter that matches vertices based on a basis vertex and a set of match type criteria.\n *

\n * Instances of this class use the following match types: {@link MatchType#SOURCE_MATCH}, \n * {@link MatchType#SOURCE_MISMATCH}, {@link MatchType#TARGET_MATCH}, {@link MatchType#TARGET_MISMATCH},\n * {@link MatchType#SOURCE_PREDICATE_MATCH}, {@link MatchType#SOURCE_PREDICATE_MISMATCH},\n * {@link MatchType#TARGET_PREDICATE_MATCH}, {@link MatchType#TARGET_PREDICATE_MISMATCH}.\n * \n * @author Scott Martin\n */\npublic class VertexMatchFilter extends MatchTypeFilter {\n\n\tLFVertex basis;\n\t\n\t/**\n\t * Creates a new vertex match filter using the specified vertex as a basis for comparison and the\n\t * specified match type criteria.\n\t * @throws IllegalArgumentException if basis is null.\n\t * @see #VertexMatchFilter(LFVertex, Collection)\n\t */\n\tpublic VertexMatchFilter(LFVertex basis, MatchType... matchTypes) {\n\t\tsuper(matchTypes);\n\t\t\n\t\tcheckBasis(basis);\n\t\tthis.basis = basis;\n\t}\n\t\t\n\t/**\n\t * Creates a new vertex match filter using the specified vertex as a basis for comparison and the\n\t * specified match type criteria.\n\t * @param basis The vertex to use as a basis for comparison.\n\t * @param matchTypes The set of match type criteria to use.\n\t * @throws IllegalArgumentException if basis is null.\n\t */\n\tpublic VertexMatchFilter(LFVertex basis, Collection matchTypes) {\n\t\tsuper(matchTypes);\n\t\t\n\t\tcheckBasis(basis);\n\t\tthis.basis = basis;\n\t}\n\n\tprivate void checkBasis(LFVertex basis) {\n\t\tif(basis == null) {\n\t\t\tthrow new IllegalArgumentException(\"basis is null\");\n\t\t}\n\t}\n\t\n\t/**\n\t * Gets the vertex that is the basis for comparison in this filter's {@link #allows(LFEdge)} method.\n\t */\n\tpublic LFVertex getBasis() {\n\t\treturn basis;\n\t}\n\n\t/**\n\t * Sets the vertex used as a basis for comparison.\n\t * @throws IllegalArgumentException if basis is null.\n\t */\n\tpublic void setBasis(LFVertex basis) {\n\t\tcheckBasis(basis);\n\t\tthis.basis = basis;\n\t}\n\n\t/**\n\t * Tests whether a specified edge is allowed based on the match type criteria in effect and the\n\t * vertex used as a basis for comparison.\n\t * @param edge The edge to test.\n\t * @return false if {@link #getMatchTypes()} contains\n\t *

    \n\t * \t
  • {@link MatchType#SOURCE_MATCH}, but the basis edge does not equal the edge's source,
    • \n\t * \t
  • {@link MatchType#SOURCE_MISMATCH}, but the basis edge is equal to the edge's source,
    • \n\t * \t
  • {@link MatchType#TARGET_MATCH}, but the basis edge does not equal the edge's target,
    • \n\t * \t
  • {@link MatchType#TARGET_MISMATCH}, but the basis edge is equal to the edge's target,
    • \n\t * \t
  • {@link MatchType#SOURCE_PREDICATE_MATCH}, but the basis edge's predicate does not equal the \n\t * \t\tedge's source vertex's predicate,
    • \n\t * \t
  • {@link MatchType#SOURCE_PREDICATE_MISMATCH}, but the basis edge's predicate is equal to the \n\t * \t\tedge's source vertex's predicate,
    • \n\t * \t
  • {@link MatchType#TARGET_PREDICATE_MATCH}, but the basis edge's predicate does not equal the \n\t * \t\tedge's target vertex's predicate,
    • \n\t * \t
  • {@link MatchType#TARGET_PREDICATE_MISMATCH}, but the basis edge's predicate is equal to the \n\t * \t\tedge's target vertex's predicate,
    • \n\t *
\n\t * and true otherwise.\n\t */\n\t@Override\n\tpublic boolean allows(LFEdge edge) {\n\t\tfor(MatchType t : matchTypes) {\n\t\t\tif(t == SOURCE_MATCH && !basis.equals(edge.getSource())) {\n\t\t\t\treturn false;\n\t\t\t}\n\t\t\telse if(t == SOURCE_MISMATCH && basis.equals(edge.getSource())) {\n\t\t\t\treturn false;\n\t\t\t}\n\t\t\telse if(t == TARGET_MATCH && !basis.equals(edge.getTarget())) {\n\t\t\t\treturn false;\n\t\t\t}\n\t\t\telse if(t == TARGET_MISMATCH && basis.equals(edge.getTarget())) {\n\t\t\t\treturn false;\n\t\t\t}\n\t\t\telse if(basis.getPredicate() != null) {\n\t\t\t\tif(t == SOURCE_PREDICATE_MATCH \n\t\t\t\t\t\t&& !basis.getPredicate().equals(edge.getSource().getPredicate())) {\n\t\t\t\t\treturn false;\n\t\t\t\t}\n\t\t\t\telse if(t == SOURCE_PREDICATE_MISMATCH \n\t\t\t\t\t\t&& basis.getPredicate().equals(edge.getSource().getPredicate())) {\n\t\t\t\t\treturn false;\n\t\t\t\t}\n\t\t\t\telse if(t == TARGET_PREDICATE_MATCH \n\t\t\t\t\t\t&& !basis.getPredicate().equals(edge.getTarget().getPredicate())) {\n\t\t\t\t\treturn false;\n\t\t\t\t}\n\t\t\t\telse if(t == TARGET_PREDICATE_MISMATCH \n\t\t\t\t\t\t&& basis.getPredicate().equals(edge.getTarget().getPredicate())) {\n\t\t\t\t\treturn false;\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn true;\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/disjunctivizer/package.html", "content": "\n\t\n

Package for the disjunctivizer, which creates a disjunctive LF XML structure based on\nan {@link opennlp.ccg.disjunctivizer.LFGraphDifference}. An LF graph difference is a\ncharacterization of the difference\nbetween two {@link opennlp.ccg.hylo.graph.LFGraph graphs} and an\n{@link opennlp.ccg.alignment.Alignment} between them in terms of\nthe edits needed to make one into the other: inserts, deletes, and substitutions.

\n\n

The {@link opennlp.ccg.disjunctivizer.FilteredLFEdgeSet} class extends\n{@link opennlp.ccg.util.FilteredSet} to also provide\nviews of its edges by {@linkplain opennlp.ccg.hylo.graph.LFEdge#getSource() source},\n{@linkplain opennlp.ccg.hylo.graph.LFEdge#getTarget() target}, or\n{@linkplain opennlp.ccg.hylo.graph.LFEdge#getLabel() label}. The various filters in this\npackage provide a lot of flexibility in terms of how to filter edges: whether their source,\ntarget, or label matches a given edge's (or just their predicates), whether their source or\ntarget is aligned, etc. These filters, and the {@link opennlp.ccg.disjunctivizer.FilteredLFEdgeSet} class,\nare used internally by the {@link opennlp.ccg.disjunctivizer.Disjunctivizer} for generating\ndisjunctive LFs.

\n\n" }, { "path": "src/opennlp/ccg/grammar/AbstractApplicationRule.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.synsem.*;\n\nimport java.util.*;\n\nimport org.jdom.Element;\n\n/**\n * Super class for application rules.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.8 $, $Date: 2009/12/21 03:27:18 $\n */\npublic abstract class AbstractApplicationRule extends AbstractRule {\n\t\n\tprivate static final long serialVersionUID = 1L;\n\t\n\tprotected Slash _functorSlash;\n\n /** Returns an XML element representing the rule. */\n public Element toXml(String dir) {\n \tElement retval = new Element(\"application\");\n \tretval.setAttribute(\"dir\", dir);\n \treturn retval;\n }\n\n public int arity() {\n return 2;\n }\n\n protected List apply(Category xyCat, Category yCat) throws UnifyFailure {\n\n if (xyCat instanceof ComplexCat) {\n\n ComplexCat xyCurCat = (ComplexCat)xyCat;\n Arg xyOuter = xyCurCat.getOuterArg();\n\n List results;\n _headCats.clear();\n\n if (xyOuter instanceof BasicArg) {\n xyOuter.unifySlash(_functorSlash);\n Category xyOuterCat = ((BasicArg)xyOuter).getCat();\n Substitution sub = new GSubstitution();\n GUnifier.unify(xyOuterCat, yCat, sub);\n results = new ArrayList(1);\n ((GSubstitution)sub).condense();\n Category result = (Category) xyCurCat.getResult().fill(sub);\n appendLFs(xyCat, yCat, result, sub);\n results.add(result);\n Slash xyOuterSlash = ((BasicArg)xyOuter).getSlash();\n _headCats.add(xyOuterSlash.isModifier() ? yCat : xyCat); \n } else if (xyOuter instanceof SetArg) {\n SetArg xyOuterSet = (SetArg)xyOuter;\n results = new ArrayList(xyOuterSet.size());\n for (int i=0; i apply(Category xyCat, Category yzCat) throws UnifyFailure {\n\n\t\tif (xyCat instanceof ComplexCat && yzCat instanceof ComplexCat) {\n\t\t\tList results = new ArrayList(1);\n _headCats.clear();\n\t\t\tComplexCat xyCC = (ComplexCat) xyCat;\n\t\t\tComplexCat yzCC = (ComplexCat) yzCat;\n\n\t\t\tArg xyOuter = xyCC.getOuterArg();\n\t\t\tif (xyOuter instanceof BasicArg) {\n\t\t\t\tSlash xySlash = ((BasicArg) xyOuter).getSlash();\n\t\t\t\txySlash.unifyCheck(_functorSlash);\n\t\t\t\tif (eisner() && xySlash.isHarmonicCompositionResult()) throw new UnifyFailure();\n\t\t\t\tCategory xyOuterCat = ((BasicArg) xyOuter).getCat();\n\n\t\t\t\tif (xyOuterCat instanceof AtomCat) {\n\t\t\t\t\t// e.g. s/s Y/Z\n\t\t\t\t\tArgStack zStack = yzCC.getArgStack();\n\t\t\t\t\tzStack.slashesUnify(_argSlash);\n\t\t\t\t\tSubstitution sub = new GSubstitution();\n\t\t\t\t\tGUnifier.unify(xyOuterCat, yzCC.getTarget(), sub);\n\t\t\t\t\txySlash = (Slash) xySlash.fill(sub);\n\t\t\t\t\txySlash.unifyCheck(_functorSlash);\n\t\t\t\t\tCategory outcome = createResult(xyCC.getResult(), zStack, xySlash, sub);\n\t\t\t\t\tappendLFs(xyCat, yzCat, outcome, sub);\n\t\t\t\t\tresults.add(outcome);\n\t _headCats.add(xySlash.isModifier() ? yzCat : xyCat); \n\t\t\t\t} else if (xyOuterCat instanceof ComplexCat) {\n\t\t\t\t\t// e.g. s/(s/n) Y/Z\n\t\t\t\t\tSubstitution sub = new GSubstitution();\n\t\t\t\t\tArgStack zStack = composeComplexY((ComplexCat) xyOuterCat, xySlash, yzCC, sub);\n\t\t\t\t\txySlash = (Slash) xySlash.fill(sub);\n\t\t\t\t\txySlash.unifyCheck(_functorSlash);\n\t\t\t\t\tCategory outcome = createResult(xyCC.getResult(), zStack, xySlash, sub);\n\t\t\t\t\tappendLFs(xyCat, yzCat, outcome, sub);\n\t\t\t\t\tresults.add(outcome);\n\t _headCats.add(xySlash.isModifier() ? yzCat : xyCat); \n\t\t\t\t}\n\t\t\t} else if (xyOuter instanceof SetArg) {\n\t\t\t\t// e.g. s/{s,n} Y/Z\n\t\t\t\tCategory yzTarget = yzCC.getTarget();\n\t\t\t\tSetArg xyOuterSet = (SetArg) xyOuter;\n\t\t\t\tint targetIndex = xyOuterSet.indexOf(yzTarget);\n\t\t\t\tif (targetIndex > -1) {\n\t\t\t\t\tSlash xySlash = xyOuterSet.get(targetIndex).getSlash();\n\t\t\t\t\txySlash.unifyCheck(_functorSlash);\n\t\t\t\t\tif (eisner() && xySlash.isHarmonicCompositionResult()) throw new UnifyFailure();\n\t\t\t\t\tSubstitution sub = new GSubstitution();\n\t\t\t\t\tGUnifier.unify(xyOuterSet.getCat(targetIndex), yzTarget, sub);\n\t\t\t\t\tCategory result = xyCC.copy();\n\t\t\t\t\t((ComplexCat) result).setOuterArgument(xyOuterSet.copyWithout(targetIndex));\n\t\t\t\t\tArgStack zStack = yzCC.getArgStack();\n\t\t\t\t\tzStack.slashesUnify(_argSlash);\n\t\t\t\t\tCategory outcome = createResult(result, zStack, xySlash, sub);\n\t\t\t\t\tappendLFs(xyCat, yzCat, outcome, sub);\n\t\t\t\t\tresults.add(outcome);\n\t _headCats.add(xySlash.isModifier() ? yzCat : xyCat); \n\t\t\t\t} else {\n\t\t\t\t\tboolean success = false;\n\t\t\t\t\tfor (int i = 0; i < xyOuterSet.size(); i++) {\n\t\t\t\t\t\tBasicArg yInSet = xyOuterSet.get(i);\n\t\t\t\t\t\tif (yInSet.getCat() instanceof ComplexCat) {\n\t\t\t\t\t\t\tSlash xySlash = yInSet.getSlash();\n\t\t\t\t\t\t\txySlash.unifyCheck(_functorSlash);\n\t\t\t\t\t\t\tif (eisner() && xySlash.isHarmonicCompositionResult()) throw new UnifyFailure();\n\t\t\t\t\t\t\tComplexCat yCat = (ComplexCat) yInSet.getCat();\n\t\t\t\t\t\t\tSubstitution sub = new GSubstitution();\n\t\t\t\t\t\t\tArgStack zStack = composeComplexY((ComplexCat) yCat, xySlash, yzCC, sub);\n\t\t\t\t\t\t\txySlash = (Slash) xySlash.fill(sub);\n\t\t\t\t\t\t\txySlash.unifyCheck(_functorSlash);\n\t\t\t\t\t\t\tCategory result = xyCC.copy();\n\t\t\t\t\t\t\t((ComplexCat) result).setOuterArgument(xyOuterSet.copyWithout(i));\n\t\t\t\t\t\t\tCategory outcome = createResult(result, zStack, xySlash, sub);\n\t\t\t\t\t\t\tappendLFs(xyCat, yzCat, outcome, sub);\n\t\t\t\t\t\t\tresults.add(outcome);\n\t\t\t _headCats.add(xySlash.isModifier() ? yzCat : xyCat); \n\t\t\t\t\t\t\tsuccess = true;\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\tif (!success) {\n\t\t\t\t\t\tthrow new UnifyFailure();\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t} else {\n\t\t\t\tthrow new UnifyFailure();\n\t\t\t}\n\n\t\t\treturn results;\n\t\t} else {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\t}\n\n\tprivate Category createResult(Category result, ArgStack zStack,\n\t\t\tSlash xySlash, Substitution sub) throws UnifyFailure {\n\t\t((GSubstitution) sub).condense();\n\t\tresult = (Category) result.fill(sub);\n\t\tArgStack newStack = zStack.fill(sub);\n\t\tif (!_isHarmonic\n\t\t\t\t&& (!xySlash.sameDirAsModality() || zStack\n\t\t\t\t\t\t.containsContrarySlash())) {\n\t\t\tnewStack.deepMap(INERTIZER_FCN);\n\t\t}\n\t\tnewStack.get(0).setSlashModifier(false);\n\t\tif (_isHarmonic && useEisnerConstraints) \n\t\t\tnewStack.setSlashHarmonicCompositionResult(true);\n\t\tif (result instanceof ComplexCat) {\n\t\t\t((ComplexCat) result).add(newStack);\n\t\t} else {\n\t\t\tresult = new ComplexCat((TargetCat) result, newStack);\n\t\t}\n\t\treturn result;\n\t}\n\n\t/**\n\t * A function that tries to unify the value ant=+ into feature structures.\n\t */\n\tprivate static ModFcn INERTIZER_FCN = new ModFcn() {\n\t\tpublic void modify(Mutable m) {\n\t\t\tif (m instanceof Slash) {\n\t\t\t\t((Slash) m).setAbility(\"inert\");\n\t\t\t}\n\t\t}\n\t};\n\n\tprivate ArgStack composeComplexY(ComplexCat xyOuterCC, Slash xySlash,\n\t\t\tComplexCat yzCC, Substitution sub) throws UnifyFailure {\n\n\t\tGUnifier.unify(xyOuterCC.getTarget(), yzCC.getTarget(), sub);\n\t\tArgStack zStack = yzCC.getArgStack();\n\t\tif (xyOuterCC.containsDollarArg()) {\n\t\t\t// e.g. s$/(s$\\n) s\\n/n\n\t\t\txyOuterCC.getArgStack().unifyPrefix(zStack, zStack.size() - 1, sub);\n\t\t\tzStack = zStack.subList(zStack.size() - 1);\n\t\t\tzStack.slashesUnify(_argSlash);\n\t\t\txySlash = (Slash) xySlash.fill(sub);\n\t\t\txySlash.unifyCheck(_functorSlash);\n\t\t\treturn zStack;\n\t\t} else if (xyOuterCC.arity() == 1) {\n\t\t\tArgStack yzStack = yzCC.getArgStack();\n\t\t\tif (!(xyOuterCC.getArg(0) instanceof BasicArg)) {\n\t\t\t\tthrow new UnifyFailure();\n\t\t\t}\n\t\t\tBasicArg xyOuterOuter = (BasicArg) xyOuterCC.getArg(0);\n\t\t\tArg yzStackInner = yzStack.get(0);\n\t\t\tif (yzStackInner instanceof SetArg) {\n\t\t\t\t// e.g. s/(s/n) s/{s,n}\n\t\t\t\tSetArg yzSetArg = (SetArg) yzStackInner;\n\t\t\t\tint iaIndex = yzSetArg.indexOf(xyOuterOuter);\n\t\t\t\tif (iaIndex == -1)\n\t\t\t\t\tthrow new UnifyFailure();\n\t\t\t\txyOuterOuter.unify(yzSetArg.get(iaIndex), sub);\n\t\t\t\txySlash = (Slash) xySlash.fill(sub);\n\t\t\t\txySlash.unifyCheck(_functorSlash);\n\t\t\t\tzStack = yzStack.copy();\n\t\t\t\tzStack.set(0, yzSetArg.copyWithout(iaIndex));\n\t\t\t\tzStack.slashesUnify(_argSlash);\n\t\t\t\treturn zStack;\n\t\t\t} else {\n\t\t\t\t// e.g. s/(s/n) s/n/s\n\t\t\t\tif (yzStack.size() < 2) {\n\t\t\t\t\tthrow new UnifyFailure();\n\t\t\t\t}\n\t\t\t\txyOuterOuter.unify(yzStackInner, sub);\n\t\t\t\tzStack = yzStack.subList(1).copy();\n\t\t\t\tzStack.slashesUnify(_argSlash);\n\t\t\t\txySlash = (Slash) xySlash.fill(sub);\n\t\t\t\txySlash.unifyCheck(_functorSlash);\n\t\t\t\treturn zStack;\n\t\t\t}\n\t\t} else if (xyOuterCC.arity() == 2) {\n\t\t\t// e.g. s\\np/(s\\np) s\\np/(s\\np)/n\n\t\t\t// nb: not dealing with set args\n\t\t\tArgStack yzStack = yzCC.getArgStack();\n\t\t\tif (!(xyOuterCC.getArg(0) instanceof BasicArg)\n\t\t\t\t\t|| !(xyOuterCC.getArg(1) instanceof BasicArg)\n\t\t\t\t\t|| yzStack.size() < 3) {\n\t\t\t\tthrow new UnifyFailure();\n\t\t\t}\n\t\t\tBasicArg xyOuterOuter1 = (BasicArg) xyOuterCC.getArg(0);\n\t\t\tBasicArg xyOuterOuter2 = (BasicArg) xyOuterCC.getArg(1);\n\t\t\tArg yzStackInner1 = yzStack.get(0);\n\t\t\tArg yzStackInner2 = yzStack.get(1);\n\t\t\txyOuterOuter1.unify(yzStackInner1, sub);\n\t\t\txyOuterOuter2.unify(yzStackInner2, sub);\n\t\t\tzStack = yzStack.subList(2).copy();\n\t\t\tzStack.slashesUnify(_argSlash);\n\t\t\txySlash = (Slash) xySlash.fill(sub);\n\t\t\txySlash.unifyCheck(_functorSlash);\n\t\t\treturn zStack;\n\t\t} else {\n\t\t\t// nb: not dealing with xyOuterCC arity > 2\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\t}\n}\n" }, { "path": "src/opennlp/ccg/grammar/AbstractRule.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.hylo.*;\n\nimport java.io.Serializable;\nimport java.util.*;\n\nimport org.jdom.Element;\n\n/**\n * Implements some default behavior for Rule objects.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.18 $, $Date: 2009/12/21 03:27:18 $\n */\npublic abstract class AbstractRule implements Rule, Serializable {\n\n\tprivate static final long serialVersionUID = 1L;\n\n\t/** The interned name of this rule. */\n protected String _name;\n \n /** The rule group which contains this rule. */\n protected RuleGroup _ruleGroup;\n \n /** Reusable list of head cats, one for each result. */\n protected List _headCats = new ArrayList(); \n\n /** Returns an XML element representing the rule. */\n abstract public Element toXml();\n\n /** Applies the rule to the given input signs, adding to the given list of results. */\n public void applyRule(Sign[] inputs, List results) {\n\n if (inputs.length != arity()) { // shouldn't happen\n throw new RuntimeException(\"Inputs must have length \" + arity());\n }\n\n Category[] cats = new Category[inputs.length];\n for (int i=0; i < cats.length; i++) {\n cats[i] = inputs[i].getCategory();\n }\n\n try {\n List resultCats = applyRule(cats);\n if (resultCats.isEmpty()) return;\n \n for (int i=0; i < resultCats.size(); i++) {\n \tCategory catResult = resultCats.get(i);\n distributeTargetFeatures(catResult);\n Category headCat = _headCats.get(i);\n Sign lexHead = inputs[0].getLexHead();\n for (int j=0; j < inputs.length; j++) {\n \tif (inputs[j].getCategory() == headCat) lexHead = inputs[j].getLexHead();\n }\n Sign sign = Sign.createDerivedSign(catResult, inputs, this, lexHead);\n results.add(sign);\n }\n } catch (UnifyFailure uf) {}\n }\n \n /** Propagates distributive features from target cat to the rest. */\n // nb: it would be nicer to combine inheritsFrom with $, but \n // this would be complicated, as inheritsFrom is compiled out\n protected void distributeTargetFeatures(Category cat) {\n \tif (_ruleGroup == null) return;\n if (_ruleGroup.grammar.lexicon.getDistributiveAttrs() == null) return;\n if (!(cat instanceof ComplexCat)) return;\n ComplexCat complexCat = (ComplexCat) cat;\n Category targetCat = (Category) complexCat.getTarget();\n targetFS = (GFeatStruc) targetCat.getFeatureStructure();\n if (targetFS == null) return;\n cat.forall(distributeTargetFeaturesFcn);\n }\n \n // target cat's feature structure\n private GFeatStruc targetFS = null;\n\n // copies ground distributive features from _targetFS to the rest\n private CategoryFcn distributeTargetFeaturesFcn = new DistributeTargetFeaturesFcn();\n \n private class DistributeTargetFeaturesFcn extends CategoryFcnAdapter implements Serializable {\n\t\tprivate static final long serialVersionUID = 5247861522003485434L;\n\t\tpublic void forall(Category c) {\n if (!(c instanceof AtomCat)) return;\n FeatureStructure fs = c.getFeatureStructure();\n if (fs == null) return;\n if (fs == targetFS) return;\n String[] distrAttrs = _ruleGroup.grammar.lexicon.getDistributiveAttrs();\n for (int i = 0; i < distrAttrs.length; i++) {\n Object targetVal = targetFS.getValue(distrAttrs[i]);\n if (targetVal != null && !(targetVal instanceof Variable)) {\n fs.setFeature(distrAttrs[i], UnifyControl.copy(targetVal));\n }\n }\n }\n }\n \n \n /**\n * The number of arguments this rule takes. For example, the arity of the\n * forward application rule of categorial grammar (X/Y Y => Y) is 2.\n *\n * @return the number of arguments this rule takes\n **/\n public abstract int arity();\n\n /**\n * Apply this rule to some input categories.\n *\n * @param inputs the input categories to try to combine\n * @return the categories resulting from using this rule to combine the\n * inputs\n * @exception UnifyFailure if the inputs cannot be combined by this rule\n **/\n public abstract List applyRule(Category[] inputs) throws UnifyFailure;\n\n \n /** Prints an apply instance for the given categories to System.out. */\n protected void showApplyInstance(Category[] inputs) {\n StringBuffer sb = new StringBuffer(); \n sb.append(_name).append(\": \");\n \n for (int i=0; i < inputs.length; i++) {\n sb.append(inputs[i]).append(' ');\n }\n\n System.out.println(sb);\n }\n\n /** Prints an apply instance for the given categories to System.out. */\n protected void showApplyInstance(Category first, Category second) {\n Category[] ca = {first,second};\n showApplyInstance(ca);\n }\n\n \n /**\n * Returns the interned name of this rule.\n */\n public String name() {\n return _name;\n }\n \n /**\n * Returns the rule group which contains this rule.\n */\n public RuleGroup getRuleGroup() { return _ruleGroup; }\n \n /**\n * Sets this rule's rule group.\n */\n public void setRuleGroup(RuleGroup ruleGroup) { _ruleGroup = ruleGroup; }\n\n \n /** Appends, fills, sorts and checks the LFs from cats 1 and 2 into the result cat. */\n protected void appendLFs(Category cat1, Category cat2, Category result, Substitution sub) \n throws UnifyFailure\n {\n LF lf = HyloHelper.append(cat1.getLF(), cat2.getLF());\n if (lf != null) {\n lf = (LF) lf.fill(sub);\n HyloHelper.sort(lf);\n HyloHelper.check(lf);\n }\n result.setLF(lf);\n }\n}\n\n" }, { "path": "src/opennlp/ccg/grammar/AbstractSubstitutionRule.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-3 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.synsem.*;\n\nimport java.util.*;\n\nimport org.jdom.Element;\n\n/**\n * Super class for substitution rules.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2009/12/21 03:27:18 $\n */\npublic abstract class AbstractSubstitutionRule extends AbstractApplicationRule {\n\n\tprivate static final long serialVersionUID = 1L;\n\t\n\tprotected boolean _isHarmonic;\n protected Slash _argSlash;\n\n /** Returns an XML element representing the rule. */\n public Element toXml(String dir) {\n \tElement retval = new Element(\"substitution\");\n \tretval.setAttribute(\"dir\", dir);\n \tretval.setAttribute(\"harmonic\", Boolean.toString(_isHarmonic));\n \treturn retval;\n }\n\n protected List apply (Category xyzCat, Category yzCat)\n throws UnifyFailure {\n \n if (xyzCat instanceof ComplexCat && yzCat instanceof ComplexCat) {\n ComplexCat xyzCC = (ComplexCat)xyzCat;\n ComplexCat yzCC = (ComplexCat)yzCat;\n\n if (xyzCC.arity() < 2\n || xyzCC.containsDollarArg()\n || xyzCC.containsSetArg()\n || yzCC.containsSetArg()\n || yzCC.containsDollarArg()) {\n throw new UnifyFailure();\n }\n\n ArgStack primaryStack = xyzCC.getArgStack();\n int size = primaryStack.size();\n \n BasicArg primaryArgY = (BasicArg)primaryStack.get(size-2);\n primaryArgY.unifySlash(_functorSlash);\n BasicArg primaryArgZ = (BasicArg)primaryStack.get(size-1);\n primaryArgZ.unifySlash(_argSlash);\n\n BasicArg secondaryArgZ = (BasicArg)yzCC.getOuterArg();\n secondaryArgZ.unifySlash(_argSlash);\n Category secondaryY = yzCC.getResult();\n\n GSubstitution sub = new GSubstitution();\n\n GUnifier.unify(primaryArgZ.getCat(), secondaryArgZ.getCat(), sub);\n GUnifier.unify(primaryArgY.getCat(), secondaryY, sub); \n \n Category result =\n new ComplexCat(xyzCC.getTarget(), primaryStack.copyWithout(size-2));\n ((GSubstitution)sub).condense();\n result = (Category)result.fill(sub);\n ((ComplexCat)result).getOuterArg().setSlashModifier(false);\n \n appendLFs(xyzCat, yzCat, result, sub);\n \n List results = new ArrayList(1);\n _headCats.clear();\n results.add(result);\n _headCats.add(primaryArgY.getSlash().isModifier() ? yzCat : xyzCat); \n return results;\n } else {\n throw new UnifyFailure();\n }\n\n }\n}\n\n" }, { "path": "src/opennlp/ccg/grammar/AbstractTypeRaisingRule.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.synsem.*;\n\nimport java.util.*;\n\nimport org.jdom.Element;\n\n/**\n * Type-raising, e.g. np => s/(s\\np).\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.9 $, $Date: 2009/12/21 03:27:18 $\n */\n// NB: It might make sense to eventually make this a subclass of TypeChangingRule, \n// but currently it's structured a bit differently.\npublic abstract class AbstractTypeRaisingRule extends AbstractRule {\n\n\tprivate static final long serialVersionUID = 1L;\n\n\t/** The upper slash, eg the first slash in s/(s\\np). */\n protected Slash _upperSlash; \n\n /** The embedded slash, eg the second slash in s/(s\\np). */\n protected Slash _embeddedSlash;\n\n /** \n * The result of the type raised category, eg the \"s\" in s/(s\\np). Defaults to \"s\". \n */\n protected Category _result;\n\n /** \n * The argument of the ComplexCat argument of the type raised category,\n * eg the \"np\" in s/(s\\np). Defaults to \"np\".\n */\n protected Category _arg;\n\n /**\n * Creates a new type raising rule with the given name; upper and lower slashes; \n * use dollar switch; arg category; and result category. Defaults are used \n * for the arg and result categories if null.\n */\n protected AbstractTypeRaisingRule(\n String name, Slash uslash, Slash eslash, \n boolean useDollar, Category arg, Category result\n ) {\n _name = name;\n _upperSlash = uslash;\n _upperSlash.setAbility(\"active\");\n _upperSlash.setModifier(true);\n _embeddedSlash = eslash;\n _embeddedSlash.setAbility(\"active\");\n\n if (arg != null) { _arg = arg; }\n else { _arg = new AtomCat(\"np\", new GFeatStruc()); }\n\n if (result != null) { \n _result = result;\n result.getFeatureStructure().setIndex(1);\n }\n else {\n GFeatStruc resfs = new GFeatStruc();\n resfs.setIndex(1);\n _result = new AtomCat(\"s\", resfs);\n }\n \n if (useDollar) {\n Dollar dol = new Dollar(\"1\");\n dol.setIndex(1);\n _result = new ComplexCat((AtomCat)_result, dol);\n }\n \n }\n\n /** Returns an XML element representing the rule. */\n public Element toXml(String dir) {\n \tElement retval = new Element(\"typeraising\");\n \tretval.setAttribute(\"dir\", dir);\n \tboolean usesDollar = (_result instanceof ComplexCat) && ((ComplexCat)_result).containsDollarArg();\n \tretval.setAttribute(\"useDollar\", Boolean.toString(usesDollar));\n \tif (!(_arg instanceof AtomCat) || !((AtomCat)_arg).getType().equals(\"np\")) {\n \tElement argElt = new Element(\"arg\");\n \tretval.addContent(argElt);\n \targElt.addContent(_arg.toXml());\n \t}\n \tif (!((AtomCat)_result.getTarget()).getType().equals(\"s\")) {\n \tElement resultElt = new Element(\"result\");\n \tretval.addContent(resultElt);\n \tresultElt.addContent(_result.getTarget().toXml());\n \t}\n \treturn retval;\n }\n\n /** Returns 1. */\n public int arity() {\n return 1;\n }\n\n /** Applies this rule to the given inputs. */\n public List applyRule(Category[] inputs) throws UnifyFailure {\n if (inputs.length != 1) {\n throw new UnifyFailure();\n }\n return apply(inputs[0]);\n }\n\n /** Applies this rule to the given input. */\n protected List apply(Category input) throws UnifyFailure {\n Substitution sub = new GSubstitution();\n Category arg = (Category)_arg.unify(input, sub);\n ((GSubstitution)sub).condense();\n \n Category result = _result.copy();\n ComplexCat range;\n UnifyControl.reindex(result);\n if (result instanceof ComplexCat) {\n range = (ComplexCat)result.copy();\n range.add(new BasicArg(_embeddedSlash, arg));\n ((ComplexCat)result).add(new BasicArg(_upperSlash, range));\n } else {\n range = new ComplexCat((TargetCat)result.copy(),\n new BasicArg(_embeddedSlash, arg));\n result = new ComplexCat((TargetCat)result.copy(),\n new BasicArg(_upperSlash, range));\n }\n \n // nb: with defined type changing rules, this step is done when the \n // rule is created; with type raising, it is done here, so that \n // the arg need not have its distributive features yet, and since \n // the full result category doesn't exist beforehand\n _ruleGroup.grammar.lexicon.propagateDistributiveAttrs(result);\n \n LF inputLF = input.getLF();\n if (inputLF != null) {\n result.setLF((LF)inputLF.copy());\n }\n \n List results = new ArrayList(1);\n _headCats.clear();\n results.add(result);\n _headCats.add(input);\n return results;\n }\n}\n\n" }, { "path": "src/opennlp/ccg/grammar/BackwardApplication.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.synsem.*;\n\nimport java.util.*;\n\nimport org.jdom.Element;\n\n/**\n * Forward application: X/Y Y => X\n *\n * @author Jason Baldridge\n * @version $Revision: 1.3 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class BackwardApplication extends AbstractApplicationRule {\n\n\tprivate static final long serialVersionUID = 6981288425455531650L;\n\n\tpublic BackwardApplication () {\n\t\t_name = \"<\";\n\t\t_functorSlash = new Slash('\\\\');\n\t\t_functorSlash.setAbility(\"active\");\n }\n \n /** Returns an XML element representing the rule. */\n public Element toXml() { return super.toXml(\"backward\"); }\n \n public List applyRule(Category[] inputs) throws UnifyFailure {\n\t\tif (inputs.length != 2) {\n\t\t throw new UnifyFailure();\n\t\t}\n\t\treturn apply(inputs[1], inputs[0]);\n }\n\n public String toString() {\n\t\treturn \"Y X\\\\Y => X\";\n }\n}\n\n" }, { "path": "src/opennlp/ccg/grammar/BackwardComposition.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.synsem.*;\n\nimport java.util.*;\n\nimport org.jdom.Element;\n\n/**\n * Backward composition, e.g. Y\\Z X\\Y => X\\Z\n * \n * @author Jason Baldridge\n * @version $Revision: 1.3 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class BackwardComposition extends AbstractCompositionRule {\n\n\tprivate static final long serialVersionUID = -937944882697380690L;\n\n\tpublic BackwardComposition() {\n\t\tthis(true);\n\t}\n\n\tpublic BackwardComposition(boolean isHarmonic) {\n\t\t_isHarmonic = isHarmonic;\n\t\tif (isHarmonic) {\n\t\t\t_name = \" applyRule(Category[] inputs) throws UnifyFailure {\n\t\tif (inputs.length != 2) {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\n\t\treturn apply(inputs[1], inputs[0]);\n\t}\n\n\tpublic String toString() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tsb.append(\"Y\").append(_argSlash.toString()).append(\"Z \").append(\n\t\t\t\t\"X\\\\Y => X\").append(_argSlash.toString()).append(\"Z\");\n\t\treturn sb.toString();\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/grammar/BackwardSubstitution.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.synsem.*;\n\nimport java.util.*;\n\nimport org.jdom.Element;\n\n/**\n * Backward substitution, e.g. Y\\Z X\\Y\\Z => X\\Z\n * \n * @author Jason Baldridge\n * @version $Revision: 1.3 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class BackwardSubstitution extends AbstractSubstitutionRule {\n\n\tprivate static final long serialVersionUID = -4597839433754132265L;\n\n\tpublic BackwardSubstitution() {\n\t\tthis(true);\n\t}\n\n\tpublic BackwardSubstitution(boolean isHarmonic) {\n\t\t_isHarmonic = isHarmonic;\n\t\tif (isHarmonic) {\n\t\t\t_name = \" applyRule(Category[] inputs) throws UnifyFailure {\n\t\tif (inputs.length != 2) {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\n\t\treturn apply(inputs[1], inputs[0]);\n\t}\n\n\tpublic String toString() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tsb.append(\"Y\").append(_argSlash.toString()).append(\"Z \").append(\n\t\t\t\t\"X\\\\Y => X\").append(_argSlash.toString()).append(\"Z\");\n\t\treturn sb.toString();\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/grammar/BackwardTypeRaising.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport org.jdom.Element;\n\nimport opennlp.ccg.synsem.*;\n\n/**\n * Backward type-raising: X => Y\\(Y/X).\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class BackwardTypeRaising extends AbstractTypeRaisingRule {\n\n\tprivate static final long serialVersionUID = 4334502669369098203L;\n\n\t/** Creates a backward type raising rule with the given parameters. */\n public BackwardTypeRaising(boolean useDollar, Category arg, Category result) {\n super(\" Y\\\\(Y/X)\";\n }\n}\n\n" }, { "path": "src/opennlp/ccg/grammar/ForwardApplication.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.synsem.*;\n\nimport java.util.*;\n\nimport org.jdom.Element;\n\n/**\n * Forward application: X/Y Y => X\n * \n * @author Jason Baldridge\n * @version $Revision: 1.3 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class ForwardApplication extends AbstractApplicationRule {\n\n\tprivate static final long serialVersionUID = 1336124476870410093L;\n\n\tpublic ForwardApplication() {\n\t\t_name = \">\";\n\t\t_functorSlash = new Slash('/');\n\t\t_functorSlash.setAbility(\"active\");\n\t}\n\t\n /** Returns an XML element representing the rule. */\n public Element toXml() { return super.toXml(\"forward\"); }\n\n\tpublic List applyRule(Category[] inputs) throws UnifyFailure {\n\t\tif (inputs.length != 2) {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\n\t\treturn apply(inputs[0], inputs[1]);\n\t}\n\n\tpublic String toString() {\n\t\treturn \"X/Y Y => X\";\n\t}\n}\n" }, { "path": "src/opennlp/ccg/grammar/ForwardComposition.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.synsem.*;\n\nimport java.util.*;\n\nimport org.jdom.Element;\n\n/**\n * Forward composition, e.g. X/Y Y/Z => X/Z\n * \n * @author Jason Baldridge\n * @version $Revision: 1.3 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class ForwardComposition extends AbstractCompositionRule {\n\n\tprivate static final long serialVersionUID = -5029901211362928251L;\n\n\tpublic ForwardComposition() {\n\t\tthis(true);\n\t}\n\n\tpublic ForwardComposition(boolean isHarmonic) {\n\t\t_isHarmonic = isHarmonic;\n\t\tif (isHarmonic) {\n\t\t\t_name = \">B\";\n\t\t\t_functorSlash = new Slash('/', \"^\");\n\t\t\t_argSlash = new Slash('/', \"^\");\n\t\t} else {\n\t\t\t_name = \">Bx\";\n\t\t\t_functorSlash = new Slash('/', \"x\");\n\t\t\t_argSlash = new Slash('\\\\', \"x\");\n\t\t}\n\t\t_functorSlash.setAbility(\"active\");\n\t}\n\n /** Returns an XML element representing the rule. */\n public Element toXml() { return super.toXml(\"forward\"); }\n\n\tpublic List applyRule(Category[] inputs) throws UnifyFailure {\n\t\tif (inputs.length != 2) {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\n\t\treturn apply(inputs[0], inputs[1]);\n\t}\n\n\tpublic String toString() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tsb.append(\"X\").append(_functorSlash.toString()).append(\"Y Y\").append(\n\t\t\t\t_argSlash.toString()).append(\"Z => X\").append(\n\t\t\t\t_argSlash.toString()).append(\"Z\");\n\t\treturn sb.toString();\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/grammar/ForwardSubstitution.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.synsem.*;\n\nimport java.util.*;\n\nimport org.jdom.Element;\n\n/**\n * Forward substitution, e.g. X/Y/Z Y/Z => X/Z\n * \n * @author Jason Baldridge\n * @version $Revision: 1.3 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class ForwardSubstitution extends AbstractSubstitutionRule {\n\n\tprivate static final long serialVersionUID = 7324585108055853456L;\n\n\tpublic ForwardSubstitution() {\n\t\tthis(true);\n\t}\n\n\tpublic ForwardSubstitution(boolean isHarmonic) {\n\t\t_isHarmonic = isHarmonic;\n\t\tif (isHarmonic) {\n\t\t\t_name = \">S\";\n\t\t\t_functorSlash = new Slash('/', \"^\");\n\t\t\t_argSlash = new Slash('/', \"^\");\n\t\t} else {\n\t\t\t_name = \">Sx\";\n\t\t\t_functorSlash = new Slash('/', \"x\");\n\t\t\t_argSlash = new Slash('\\\\', \"x\");\n\t\t}\n\t\t_functorSlash.setAbility(\"active\");\n\t}\n\n /** Returns an XML element representing the rule. */\n public Element toXml() { return super.toXml(\"forward\"); }\n\n\tpublic List applyRule(Category[] inputs) throws UnifyFailure {\n\t\tif (inputs.length != 2) {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\n\t\treturn apply(inputs[0], inputs[1]);\n\t}\n\n\tpublic String toString() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tsb.append(\"X\").append(_functorSlash.toString()).append(\"Y Y\").append(\n\t\t\t\t_argSlash.toString()).append(\"Z => X\").append(\n\t\t\t\t_argSlash.toString()).append(\"Z\");\n\t\treturn sb.toString();\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/grammar/ForwardTypeRaising.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport org.jdom.Element;\n\nimport opennlp.ccg.synsem.*;\n\n/**\n * Forward type-raising: X => Y/(Y\\X).\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class ForwardTypeRaising extends AbstractTypeRaisingRule {\n \n\tprivate static final long serialVersionUID = 1417585756957436141L;\n\n\t/** Creates a forward type raising rule with the given parameters. */\n public ForwardTypeRaising (boolean useDollar, Category arg, Category result) {\n super(\">T\", new Slash('/', new VarModality(\"i\")), new Slash('\\\\', new VarModality(\"i\")),\n useDollar, arg, result);\n }\n \n /** Returns an XML element representing the rule. */\n public Element toXml() {\n \treturn super.toXml(\"forward\");\n }\n\n public String toString() {\n return \"X => Y/(Y\\\\X)\";\n }\n}\n\n" }, { "path": "src/opennlp/ccg/grammar/FragmentJoining.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2007 Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\n\nimport java.util.*;\n\nimport org.jdom.Element;\n\n\n/**\n * Rule for joining fragments that don't fit together, when all else fails.\n *\n * @author Michael White\n * @version $Revision: 1.4 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class FragmentJoining extends AbstractRule {\n\n\tprivate static final long serialVersionUID = 7451163798607652012L;\n\n\t/** Constructor. */\n\tpublic FragmentJoining() { _name = \"*\"; }\n\t\n /** Returns an XML element representing the rule (not supported). */\n public Element toXml() { throw new RuntimeException(\"toXml not supported for FragmentJoining rules\"); }\n\n\t/**\n\t * Returns the result of applying this rule to two input signs.\n\t */\n\tpublic Sign applyRule(Sign sign1, Sign sign2) {\n\t\tList results = new ArrayList(1);\n\t\tSign[] inputs = new Sign[] { sign1, sign2 };\n\t\tapplyRule(inputs, results);\n\t\treturn results.get(0);\n\t}\n\t\n /**\n * Apply this rule to two input categories. \n * Returns a copy of the first cat with the LFs appended.\n **/\n public List applyRule(Category[] inputs) throws UnifyFailure {\n\t\tif (inputs.length != 2) { throw new UnifyFailure(); }\n\t\tList results = new ArrayList(1);\n _headCats.clear();\n\t\tCategory result = inputs[0].shallowCopy();\n\t\ttry {\n\t\t\tappendLFs(inputs[0], inputs[1], result, new EmptySubstitution());\n\t\t}\n\t\tcatch (UnifyFailure uf) { // not expected\n\t\t\t// System.err.println(\"Unexpected unify failure in appending LFs when joining fragments:\");\n\t\t\t// System.err.println(\"cat0: \" + inputs[0] + \" lf: \" + inputs[0].getLF());\n\t\t\t// System.err.println(\"cat1: \" + inputs[1] + \" lf: \" + inputs[1].getLF());\n\t\t}\n\t\tresults.add(result);\n\t\t_headCats.add(inputs[0]);\n\t\treturn results;\n }\n\n /**\n * The number of arguments this rule takes.\n **/\n public int arity() { return 2; }\n\n /** Returns a string for this rule. */\n\tpublic String toString() {\n\t\treturn \"X Y *=> X\";\n\t}\n}\n\n" }, { "path": "src/opennlp/ccg/grammar/GlueRule.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2011 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.grammar;\r\n\r\nimport java.util.*;\r\n\r\nimport org.jdom.Element;\r\n\r\nimport opennlp.ccg.synsem.*;\r\nimport opennlp.ccg.unify.*;\r\n\r\n/**\r\n * Implements a glue rule for combining a sequence of fragments.\r\n * The rule is frag|cat cat => frag, allowing only the first input \r\n * to itself be a fragment, unless the second input has the \r\n * frag completion flag set, meaning that it completes a chunk/alt.\r\n * \r\n * @author Michael White\r\n * @version $Revision: 1.3 $, $Date: 2011/06/07 05:12:01 $\r\n */\r\npublic class GlueRule extends AbstractRule {\r\n\r\n\tprivate static final long serialVersionUID = 4867141181941895272L;\r\n\r\n\t// empty subst for combining LFs\r\n\tprivate static final Substitution emptySubst = new SimpleSubstitution();\r\n\t\r\n\t/** Fragment result type. */\r\n\tpublic static final String resultType = \"frag\";\r\n\t\r\n\t/** Constructor. */\r\n\tpublic GlueRule() { _name = \"glue\"; }\r\n\t\r\n /** Returns an XML element representing the rule (not supported). */\r\n public Element toXml() { throw new RuntimeException(\"toXml not supported for GlueRule rules\"); }\r\n\r\n\t/** Arity. */\r\n\tpublic int arity() { return 2; }\r\n\r\n\t/** Glues cats into fragments. */\r\n\tpublic List applyRule(Category[] inputs) throws UnifyFailure {\r\n\t\t// check num inputs\r\n\t\tif (inputs.length != 2) {\r\n\t\t\tthrow new UnifyFailure();\r\n\t\t}\r\n\t\t// check for frag as second input with completion false\r\n\t\tif (inputs[1] instanceof AtomCat) {\r\n\t\t\tAtomCat ac2 = (AtomCat) inputs[1];\r\n\t\t\tif (ac2.isFragment() && !ac2.fragCompletion) \r\n\t\t\t\tthrow new UnifyFailure();\r\n\t\t}\r\n\t\t// make result cat\r\n List results = new ArrayList(1);\r\n _headCats.clear();\r\n AtomCat ac = new AtomCat(resultType);\r\n appendLFs(inputs[0], inputs[1], ac, emptySubst);\r\n results.add(ac);\r\n // guess head, with left as default\r\n boolean leftHead = true;\r\n boolean leftMod = isModifier(inputs[0]);\r\n boolean rightMod = isModifier(inputs[1]);\r\n if ((inputs[0] instanceof AtomCat && inputs[1] instanceof ComplexCat && !rightMod) ||\r\n \t(leftMod && !rightMod)) \r\n {\r\n \tleftHead = false;\r\n }\r\n\t\t// return result cat with guessed head\r\n _headCats.add(leftHead ? inputs[0] : inputs[1]);\r\n\t\treturn results;\r\n\t}\r\n\t\r\n\t// modifier check\r\n\tprivate static boolean isModifier(Category cat) {\r\n\t\tif (cat instanceof ComplexCat) {\r\n\t\t\tComplexCat xyCat = (ComplexCat) cat;\r\n\t\t\tArg arg = xyCat.getOuterArg();\r\n\t\t\tif (arg instanceof BasicArg) {\r\n\t\t\t\treturn ((BasicArg)arg).getSlash().isModifier();\r\n\t\t\t}\r\n\t\t}\r\n\t\treturn false;\r\n\t}\r\n\r\n\t/** toString. */\r\n\tpublic String toString() {\r\n\t\treturn \"frag|cat cat => frag\";\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccg/grammar/Grammar.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-5 University of Edinburgh (Michael White) and Gunes Erkan\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.lexicon.*;\nimport opennlp.ccg.util.*;\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.hylo.*;\nimport opennlp.ccg.parse.Parser;\nimport opennlp.ccg.parse.ParseException;\nimport opennlp.ccg.realize.Realizer;\n\nimport org.jdom.*;\nimport org.jdom.input.*;\nimport org.jdom.output.*;\nimport org.jdom.transform.*;\nimport org.xml.sax.*;\n\nimport javax.xml.parsers.*;\nimport javax.xml.transform.*;\nimport javax.xml.transform.stream.*;\nimport javax.xml.transform.sax.*;\nimport java.io.*;\nimport java.net.URL;\nimport java.util.*;\n\n/**\n * A CCG grammar is essentially a lexicon plus a rule group.\n * A grammar may also have sequences of transformations to use in \n * loading/saving LFs from/to XML.\n *\n * @author Michael White\n * @author Gunes Erkan\n * @version $Revision: 1.45 $, $Date: 2010/12/06 02:39:35 $ \n */\npublic class Grammar {\n\n /** The lexicon. */\n public final Lexicon lexicon;\n \n /** The rule group. */\n public final RuleGroup rules;\n\n /** The type hierarchy. */\n public final Types types;\n \n /** The features to include in supertags. */\n public final Set supertagFeatures = new HashSet();\n \n /** The sequence of transformations to use when loading LFs from XML. */\n public final URL[] fromXmlTransforms;\n \n /** The sequence of transformations to use when saving LFs to XML. */\n public final URL[] toXmlTransforms;\n\n /** Preferences for displaying elements in this grammar. */\n public DisplayPrefs prefs = new DisplayPrefs();\n \n /** For access to the current grammar; should be generalized eventually. */\n public static Grammar theGrammar;\n\t\n // name of the grammar\n private String grammarName = null;\n\t\n // parser, for getting parsed words\n private Parser parser = null; \n\n // XML factories\n private SAXParserFactory spf = null; \n private static SAXTransformerFactory stf = null; \n \n // transformer for loading/saving LFs from/to XML\n private Transformer transformer = null;\n \n // transformations for loading/saving LFs from/to XML\n private Templates[] fromXmlTemplates = null;\n private Templates[] toXmlTemplates = null;\n \n // transformer for saving strings to APML\n private Transformer apmlTransformer = null;\n \n /** The pitch accents recognized as underscored suffixes for translation to APML. */\n public static final String[] pitchAccents = { \n \"H*\", \"L*\", \"L+H*\", \"L*+H\", \"H*+L\", \"H+L*\"\n };\n\n // set of pitch accents\n private static Set pitchAccentsSet = null; \n \n /** The boundary tones recognized as separate tokens for translation to APML. */\n public static final String[] boundaryTones = { \n \"L\", \"H\", \"LL%\", \"HH%\", \"LH%\", \"HL%\"\n };\n \n // set of boundary tones\n private static Set boundaryTonesSet = null; \n\n \n /** Loads a grammar from the given filename. */\n public Grammar(String filename) throws IOException {\n this(new File(filename).toURI().toURL());\n }\n \n /** Loads a grammar from the given URL. */\n\tpublic Grammar(URL url) throws IOException {\n \tthis(url, false);\n }\n \n /** Loads a grammar from the given URL, with the given flag for whether to ignore rule combos. */\n @SuppressWarnings(\"unchecked\")\n\tpublic Grammar(URL url, boolean ignoreCombos) throws IOException {\n theGrammar = this;\n // read XML\n SAXBuilder builder = new SAXBuilder();\n Document doc;\n try {\n doc = builder.build(url);\n } catch (JDOMException jde) {\n throw (IOException) new IOException().initCause(jde);\n }\n Element root = doc.getRootElement();\t// root corresponds to \n\t\t grammarName = root.getAttributeValue(\"name\");\n\t\t\n Element supertagsElt = root.getChild(\"supertags\");\n if (supertagsElt != null) {\n String feats = supertagsElt.getAttributeValue(\"feats\");\n if (feats != null) {\n String[] names = feats.split(\"\\\\s+\");\n for (int i = 0; i < names.length; i++) {\n supertagFeatures.add(names[i]);\n }\n }\n }\n if (supertagFeatures.isEmpty()) {\n // default is \"form\" and \"lex\"\n supertagFeatures.add(\"form\"); supertagFeatures.add(\"lex\"); \n }\n \n Tokenizer tokenizer = null;\n Element tokenizerElt = root.getChild(\"tokenizer\");\n if (tokenizerElt != null) {\n String tokenizerClass = tokenizerElt.getAttributeValue(\"classname\");\n if (tokenizerClass != null) {\n try {\n tokenizer = (Tokenizer) Class.forName(tokenizerClass).newInstance();\n } catch (Exception exc) {\n throw (IOException) new IOException().initCause(exc);\n }\n }\n else tokenizer = new DefaultTokenizer();\n String replacementSemClasses = tokenizerElt.getAttributeValue(\"replacement-sem-classes\");\n if (replacementSemClasses != null) {\n String[] semClasses = replacementSemClasses.split(\"\\\\s+\");\n for (int i = 0; i < semClasses.length; i++) {\n tokenizer.addReplacementSemClass(semClasses[i]);\n }\n }\n }\n \n Element typesElt = root.getChild(\"types\");\n URL typesUrl;\n if (typesElt != null) {\n typesUrl = new URL(url, typesElt.getAttributeValue(\"file\"));\n }\n else typesUrl = null;\n Element lexiconElt = root.getChild(\"lexicon\");\n boolean openlex = \"true\".equals(lexiconElt.getAttributeValue(\"openlex\"));\n URL lexiconUrl = new URL(url, lexiconElt.getAttributeValue(\"file\")); \n Element morphElt = root.getChild(\"morphology\");\n URL morphUrl = new URL(url, morphElt.getAttributeValue(\"file\"));\n Element rulesElt = root.getChild(\"rules\");\n URL rulesUrl = new URL(url, rulesElt.getAttributeValue(\"file\"));\n Element fromXmlElt = root.getChild(\"LF-from-XML\");\n if (fromXmlElt != null) {\n List children = fromXmlElt.getChildren();\n fromXmlTransforms = new URL[children.size()];\n for (int i = 0; i < children.size(); i++) {\n Element transformElt = (Element) children.get(i);\n fromXmlTransforms[i] = new URL(url, transformElt.getAttributeValue(\"file\"));\n }\n } else {\n fromXmlTransforms = new URL[0];\n }\n Element toXmlElt = root.getChild(\"LF-to-XML\");\n if (toXmlElt != null) {\n List children = toXmlElt.getChildren();\n toXmlTransforms = new URL[children.size()];\n for (int i = 0; i < children.size(); i++) {\n Element transformElt = (Element) children.get(i);\n toXmlTransforms[i] = new URL(url, transformElt.getAttributeValue(\"file\"));\n }\n } else {\n toXmlTransforms = new URL[0];\n }\n \n // load type hierarchy, lexicon and rules\n if (typesUrl != null) types = new Types(typesUrl, this);\n else types = new Types(this);\n if (tokenizer != null) lexicon = new Lexicon(this, tokenizer);\n else lexicon = new Lexicon(this);\n lexicon.openlex = openlex;\n lexicon.init(lexiconUrl, morphUrl); \n rules = new RuleGroup(rulesUrl, this);\n \n // add observed supertag-rule combos for filtering, if any, unless ignoring combos\n if (!ignoreCombos) {\n\t String combosfile = rulesElt.getAttributeValue(\"combosfile\");\n\t if (combosfile != null) {\n\t \tURL combosUrl = new URL(url, combosfile);\n\t \trules.loadSupercatRuleCombos(combosUrl);\n\t }\n\t // set dynamic combos: defaults to true with a combosfile, otherwise defaults to false\n\t boolean dynamic = (combosfile != null);\n\t String dynamicCombos = rulesElt.getAttributeValue(\"dynamic-combos\");\n\t if (dynamicCombos != null) dynamic = Boolean.parseBoolean(dynamicCombos);\n\t rules.setDynamicCombos(dynamic);\n }\n }\n\n \n /**\n * Returns a file url string relative to the user's current directory \n * for the given filename.\n */\n public static String convertToFileUrl(String filename) {\n try {\n return new File(filename).toURI().toURL().toString();\n }\n catch (java.net.MalformedURLException exc) {\n throw (RuntimeException) new RuntimeException().initCause(exc);\n }\n // return \"file:\"+System.getProperty(\"user.dir\")+\"/\"+filename;\n }\n \n \n // initializes factories and transformers\n private void initializeTransformers() throws TransformerConfigurationException {\n // init factories\n if (spf == null) {\n spf = SAXParserFactory.newInstance(); \n spf.setNamespaceAware(true);\n }\n if (stf == null) {\n stf = (SAXTransformerFactory) TransformerFactory.newInstance();\n try { // try setting indent at factory level\n stf.setAttribute(\"indent-number\", new Integer(2));\n } catch (IllegalArgumentException exc) {} // ignore\n }\n // set up transformer with indenting\n // nb: with some JVMs (eg JDK 1.4.1 on Windows), \n // the transformer needs to be reinitialized each time, in order to \n // run multiple :r FN commands in tccg \n if (transformer == null) {\n transformer = stf.newTransformer();\n transformer.setOutputProperty(OutputKeys.INDENT, \"yes\");\n try { // also try setting indent as a xalan property \n transformer.setOutputProperty(\"{http://xml.apache.org/xalan}indent-amount\", \"2\");\n } catch (IllegalArgumentException exc) {} // ignore\n }\n // set up apml transformer \n if (apmlTransformer == null) {\n InputStream toApmlStr = ClassLoader.getSystemResourceAsStream(\"opennlp/ccg/grammar/to-apml.xsl\");\n apmlTransformer = stf.newTransformer(new StreamSource(toApmlStr));\n // nb: DOCTYPE SYSTEM also specified in to-apml.xsl; including \n // redundant specification here to workaround omission of DOCTYPE with Linux 1.5 JVM\n apmlTransformer.setOutputProperty(OutputKeys.DOCTYPE_SYSTEM, \"apml.dtd\");\n }\n }\n \n \n // does setup for LF from XML transformation, and returns a SAXSource for the given input stream\n // nb: need a new filter chain one for each use (perhaps due to an underyling bug)\n private SAXSource fromXmlSetup(InputStream istream) throws IOException {\n try {\n // initialize transformer\n initializeTransformers();\n // load transformations\n if (fromXmlTemplates == null) {\n fromXmlTemplates = new Templates[fromXmlTransforms.length];\n for (int i = 0; i < fromXmlTemplates.length; i++) {\n String url = fromXmlTransforms[i].toString();\n fromXmlTemplates[i] = stf.newTemplates(new StreamSource(url));\n }\n }\n // set up initial reader\n SAXParser parser = spf.newSAXParser();\n XMLReader reader = parser.getXMLReader();\n // set up chain of filters\n XMLFilter[] filters = new XMLFilter[fromXmlTransforms.length];\n for (int i = 0; i < filters.length; i++) {\n // create filter\n filters[i] = stf.newXMLFilter(fromXmlTemplates[i]);\n // set parent\n if (i == 0) { filters[0].setParent(reader); }\n else { filters[i].setParent(filters[i-1]); }\n }\n // set final reader/filter\n XMLReader finalReader = (filters.length == 0) ? reader : filters[filters.length-1];\n // set up and return LF from XML SAX source with final reader/filter\n return new SAXSource(finalReader, new InputSource(istream));\n } catch (ParserConfigurationException pce) {\n throw (IOException) new IOException().initCause(pce);\n } catch (SAXException se) {\n throw (IOException) new IOException().initCause(se);\n } catch (TransformerConfigurationException tce) {\n throw (IOException) new IOException().initCause(tce);\n }\n }\n \n /**\n * Loads a document from the XML in the given input stream, \n * applying the configured from-XML transformations.\n */\n public synchronized Document loadFromXml(InputStream istream) throws IOException {\n try {\n // do setup and get source\n Source source = fromXmlSetup(istream);\n // do transformation\n JDOMResult result = new JDOMResult();\n transformer.transform(source, result);\n // return result doc\n return result.getDocument();\n } catch (TransformerException exc) { \n throw (IOException) new IOException().initCause(exc);\n }\n }\n \n /**\n * Loads a document from the XML file with the given filename, \n * applying the configured from-XML transformations.\n */\n public synchronized Document loadFromXml(String filename) throws IOException {\n BufferedInputStream bis = new BufferedInputStream(new FileInputStream(filename));\n Document retval = loadFromXml(bis);\n bis.close();\n return retval;\n }\n \n\n // does setup for LF to XML transformation, and returns a SAXSource for the given source\n // nb: need a new filter chain one for each use (perhaps due to an underyling bug)\n private SAXSource toXmlSetup(Source source) throws IOException {\n try {\n // initialize transformer\n initializeTransformers();\n // load transformations\n if (toXmlTemplates == null) {\n toXmlTemplates = new Templates[toXmlTransforms.length];\n for (int i = 0; i < toXmlTemplates.length; i++) {\n // File file = new File(toXmlTransforms[i]);\n // toXmlTemplates[i] = stf.newTemplates(new StreamSource(file));\n String url = toXmlTransforms[i].toString();\n toXmlTemplates[i] = stf.newTemplates(new StreamSource(url));\n }\n }\n // set up initial reader\n SAXParser parser = spf.newSAXParser();\n XMLReader reader = parser.getXMLReader();\n // set up chain of filters\n XMLFilter[] filters = new XMLFilter[toXmlTransforms.length];\n for (int i = 0; i < filters.length; i++) {\n // create filter\n filters[i] = stf.newXMLFilter(toXmlTemplates[i]);\n // set parent\n if (i == 0) { filters[0].setParent(reader); }\n else { filters[i].setParent(filters[i-1]); }\n }\n // set final reader/filter\n XMLReader finalReader = (filters.length == 0) ? reader : filters[filters.length-1];\n // set up and return LF to XML SAX source with final reader/filter\n return new SAXSource(finalReader, SAXSource.sourceToInputSource(source));\n } catch (ParserConfigurationException pce) {\n throw (IOException) new IOException().initCause(pce);\n } catch (SAXException se) {\n throw (IOException) new IOException().initCause(se);\n } catch (TransformerConfigurationException tce) {\n throw (IOException) new IOException().initCause(tce);\n }\n }\n\n /**\n * Saves the given LF with the given target string to an XML file \n * with the given filename, applying the configured to-XML\n * transformations.\n */\n public synchronized void saveToXml(LF lf, String target, String filename) throws IOException { \n // ensure dirs exist for filename\n File file = new File(filename);\n File parent = file.getParentFile();\n if (parent != null && !parent.exists()) { parent.mkdirs(); }\n FileOutputStream out = new FileOutputStream(file); \n saveToXml(lf, target, out);\n out.close();\n }\n\n /**\n * Saves the given LF with the given target string as XML to the \n * given output stream, applying the configured to-XML\n * transformations.\n */\n public synchronized void saveToXml(LF lf, String target, OutputStream out) throws IOException { \n // make doc with XML for LF and target\n Document doc = new Document();\n Element root = new Element(\"xml\");\n doc.setRootElement(root);\n root.addContent(HyloHelper.toXml(lf));\n Element targetElt = new Element(\"target\");\n targetElt.addContent(target);\n root.addContent(targetElt);\n\n // write transformed doc to file\n try {\n // do setup and get source\n Source source = toXmlSetup(new JDOMSource(doc));\n // do transformation\n transformer.transform(source, new StreamResult(new OutputStreamWriter(out)));\n } catch (TransformerException exc) { \n throw (IOException) new IOException().initCause(exc);\n }\n }\n\n \n /**\n * Transforms an LF by applying the configured to-XML and from-XML transformations, \n * then loading the LF from the resulting doc.\n */\n public synchronized LF transformLF(LF lf) throws IOException {\n ByteArrayOutputStream out = new ByteArrayOutputStream();\n saveToXml(lf, \"\", out);\n ByteArrayInputStream in = new ByteArrayInputStream(out.toByteArray());\n Document doc = loadFromXml(in);\n return Realizer.getLfFromDoc(doc);\n }\n \n /**\n * Loads an LF by applying the configured from-XML transformations, \n * then loading the LF from the resulting doc.\n */\n public synchronized LF loadLF(Document doc) throws IOException {\n ByteArrayOutputStream out = new ByteArrayOutputStream();\n serializeXml(doc, out);\n ByteArrayInputStream in = new ByteArrayInputStream(out.toByteArray());\n Document doc2 = loadFromXml(in);\n return Realizer.getLfFromDoc(doc2);\n }\n \n \n /**\n * Convenience method to serialize XML.\n */\n public synchronized void serializeXml(Document doc, OutputStream out) throws IOException {\n try {\n initializeTransformers();\n JDOMResult result = new JDOMResult(); // as suggested by Amy Isard, for better java/xml version compatibility\n transformer.transform(new JDOMSource(doc), result);\n XMLOutputter outputter = new XMLOutputter();\n outputter.setFormat(Format.getPrettyFormat());\n outputter.output(result.getDocument(), new OutputStreamWriter(out)); // end of A.I. suggestion\n } catch (TransformerException exc) { \n throw (IOException) new IOException().initCause(exc);\n }\n }\n\n \n /** \n * Makes an element for the given LF, applying the configured to-XML transformations.\n */\n public synchronized Element makeLfElt(LF lf) throws IOException { \n // make doc with LF in it\n Document lfDoc = new Document();\n lfDoc.setRootElement(HyloHelper.toXml(lf));\n // apply to-XML transformations\n try {\n // do setup and get source\n Source source = toXmlSetup(new JDOMSource(lfDoc));\n // do transformation and get resulting doc\n JDOMResult result = new JDOMResult();\n transformer.transform(source, result);\n lfDoc = result.getDocument();\n } catch (TransformerException exc) { \n throw (IOException) new IOException().initCause(exc);\n }\n return lfDoc.detachRootElement();\n }\n\n \n /** \n * Returns whether the given string is a recognized pitch accent.\n */\n public static boolean isPitchAccent(String s) {\n if (pitchAccentsSet == null) {\n pitchAccentsSet = new HashSet();\n for (int i = 0; i < pitchAccents.length; i++) {\n pitchAccentsSet.add(pitchAccents[i]);\n }\n }\n return pitchAccentsSet.contains(s);\n }\n \n /** \n * Returns whether the given string is a recognized boundary tone. \n */\n public static boolean isBoundaryTone(String s) {\n if (boundaryTonesSet == null) {\n boundaryTonesSet = new HashSet();\n for (int i = 0; i < boundaryTones.length; i++) {\n boundaryTonesSet.add(boundaryTones[i]);\n }\n }\n return boundaryTonesSet.contains(s);\n }\n \n \n /**\n * Saves the given sign's words, pitch accents and boundary tones \n * to an APML file with the given filename.\n */\n public synchronized void saveToApml(Sign sign, String filename) throws IOException {\n // ensure dirs exist for filename\n File file = new File(filename);\n File parent = file.getParentFile();\n if (parent != null && !parent.exists()) { parent.mkdirs(); }\n // do transformation\n FileWriter fw = new FileWriter(file);\n saveToApml(sign, fw);\n fw.close();\n }\n \n /**\n * Saves the given sign's words, pitch accents and boundary tones \n * as APML to the given writer.\n * The orthography is first converted to XML using Sign.getWordsInXml, \n * and then converted to APML using opennlp/ccg/grammar/to-apml.xsl.\n * The string is assumed to be a single performative.\n */\n public synchronized void saveToApml(Sign sign, Writer writer) throws IOException { \n // convert words\n Document doc = sign.getWordsInXml();\n // write transformed doc to file\n try {\n // do setup and get source\n initializeTransformers();\n Source source = new JDOMSource(doc);\n // do transformation\n apmlTransformer.transform(source, new StreamResult(writer));\n } catch (TransformerException exc) { \n throw (IOException) new IOException().initCause(exc);\n }\n }\n \n \n /** \n * Returns the words for the given string, as determined by its \n * first parse, or an empty list, if it cannot be parsed.\n */\n // NB: Could try to extend this to find the parse with the intended LF.\n public List getParsedWords(String s) {\n // ensure parser instantiated\n if (parser == null) parser = new Parser(this);\n // get parses\n try {\n parser.parse(s);\n }\n catch (ParseException pe) {\n return new ArrayList(0);\n }\n List parses = parser.getResult();\n // return words of first parse\n Sign sign = parses.get(0);\n return sign.getWords();\n }\n\n\n\t/**\n\t* Returns the name of the loaded grammar. Null if no name given.\n\t*/\n\tpublic final String getName() {\n\t\treturn grammarName;\n\t}\n\t\n\t\n /**\n * Writes the list of words to a basic morph file.\n * @throws IOException \n */\n public void toMorphXml(List words, String filename) throws IOException {\n \tCollections.sort(words);\n \tXMLOutputter xout = new XMLOutputter();\n \txout.setFormat(Format.getPrettyFormat());\n \tPrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(filename)));\n \tout.println(\"\");\n \tout.println(\"\");\n \tfor (Word w : words) {\n \t\tElement e = new Element(\"entry\");\n \t\te.setAttribute(\"word\", w.getForm());\n \t\tif (w.getForm() != w.getStem() && w.getStem() != null) e.setAttribute(\"stem\", w.getStem());\n \t\tif (w.getPOS() != null) e.setAttribute(\"pos\", w.getPOS());\n \t\tif (w.getSemClass() != null) e.setAttribute(\"class\", w.getSemClass());\n \t\txout.output(e, out); out.println();\n \t}\n \tout.println(\"\");\n \tout.flush(); out.close();\n }\n\t\n /**\n * Writes the list of categories and associated POS tags to a basic lexicon file.\n * Note that the LFs are expected to have a [*DEFAULT*] proposition in the \n * desired location for predicate insertion.\n * @throws IOException \n */\n public void toLexiconXml(List cats, List POSs, String filename) throws IOException {\n \t// create map from supertags with unique suffixes to cat/pos pairs\n \tMap> stagMap = new HashMap>();\n \tfor (int i=0; i < cats.size(); i++) {\n \t\tCategory cat = cats.get(i); String pos = POSs.get(i);\n \t\tString stag = cat.getSupertag();\n \t\tif (stagMap.containsKey(stag)) {\n \t\tint j = 1;\n \t\t\twhile (stagMap.containsKey(stag+\"-\"+j)) j++;\n \t\t\tstag += \"-\"+j;\n \t\t}\n \t\tstagMap.put(stag, new Pair(cat, pos));\n \t}\n \tList keys = new ArrayList(stagMap.keySet());\n \tCollections.sort(keys);\n \tXMLOutputter xout = new XMLOutputter();\n \txout.setFormat(Format.getPrettyFormat());\n \tPrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(filename)));\n \tout.println(\"\");\n \tout.println(\"\");\n \tfor (String key : keys) {\n \t\tPair p = stagMap.get(key);\n \t\tCategory cat = p.a; String pos = p.b;\n \t\tElement fam = new Element(\"family\");\n \t\tfam.setAttribute(\"name\", key);\n \t\tfam.setAttribute(\"pos\", pos);\n \t\tElement ent = new Element(\"entry\");\n \t\tent.setAttribute(\"name\", \"1\");\n \t\tfam.addContent(ent);\n \t\tent.addContent(cat.toXml());\n \t\txout.output(fam, out); out.println();\n \t}\n \tout.println(\"\");\n \tout.flush(); out.close();\n }\n}\n\n" }, { "path": "src/opennlp/ccg/grammar/Rule.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\nimport java.util.*;\n\nimport org.jdom.Element;\n\n\n/**\n * Interface for categorial rules.\n *\n * @author Gann Bierner\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2009/11/30 20:36:15 $\n */\npublic interface Rule {\n\n /**\n * Apply this rule to some input categories.\n *\n * @param inputs the input categories to try to combine\n * @return the Category (or categories) resulting from using this Rule to combine the\n * inputs\n * @exception UnifyFailure if the inputs cannot be combined by this Rule\n **/\n public List applyRule(Category[] inputs) throws UnifyFailure;\n\n /**\n * The number of arguments this rule takes. For example, the arity of the\n * forward application rule of categorial grammar (X/Y Y => Y) is 2.\n *\n * @return the number of arguments this rule takes\n **/\n public int arity();\n\n /**\n * Returns the interned name of this rule.\n */\n public String name();\n \n /**\n * Returns the rule group which contains this rule.\n */\n public RuleGroup getRuleGroup();\n \n /**\n * Sets this rule's rule group.\n */\n public void setRuleGroup(RuleGroup ruleGroup);\n \n /** Returns an XML element representing the rule. */\n public Element toXml();\n}\n\n" }, { "path": "src/opennlp/ccg/grammar/RuleGroup.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-6 Jason Baldridge, Gann Bierner and \n// Michael White (University of Edinburgh, The Ohio State University)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.hylo.*;\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.util.*;\n\nimport org.jdom.*;\nimport org.jdom.output.*;\nimport gnu.trove.*;\n\nimport java.io.*;\nimport java.net.*;\nimport java.util.*;\n\n/**\n * A set of rules for combining categories.\n * Observed rule combos can be cached, either statically or dynamically.\n *\n * During deserialization, the grammar is set to the current grammar, \n * and supercat rule combos are borrowed from the current grammar's rule group.\n * \n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.32 $, $Date: 2011/06/07 05:12:01 $\n */\npublic class RuleGroup implements Serializable {\n\n\tprivate static final long serialVersionUID = -6240266013357142289L;\n\n\t/** The grammar that this rule group is part of. */\n public transient Grammar grammar;\n \n // rules\n private List unaryRules = new ArrayList();\n private List binaryRules = new ArrayList();\n\n // maps of type changing rules by their semantics\n private GroupMap predsToRules = new GroupMap();\n private GroupMap relsToRules = new GroupMap();\n \n // rule for use in applying coarticulations\n private BackwardApplication bapp = new BackwardApplication();\n\n // glue rule\n private GlueRule glueRule = new GlueRule();\n \n // supercat-rule combos, to support filtering on observed ones\n private class SupercatRuleCombo {\n \t// NB: strings must be interned\n\t\tprivate String supercat; \n \tprivate String supercat2;\n \tprivate String rule;\n \t// unary rule constructor\n \tpublic SupercatRuleCombo(String supercat, String rule) {\n \t\tsetCombo(supercat.intern(), (rule != null) ? rule.intern() : null);\n \t}\n \t// binary rule constructor\n \tpublic SupercatRuleCombo(String supercat, String supercat2, String rule) {\n \t\tsetCombo(supercat.intern(), supercat2.intern(), (rule != null) ? rule.intern() : null);\n \t}\n \t// setters\n \t// NB: assume interned strings!\n \tpublic void setCombo(String supercat, String rule) {\n \t\tthis.supercat = supercat; this.supercat2 = null; this.rule = rule;\n \t}\n \tpublic void setCombo(String supercat, String supercat2, String rule) {\n \t\tthis.supercat = supercat; this.supercat2 = supercat2; this.rule = rule;\n \t}\n \t// hashcode\n \tpublic int hashCode() {\n \t\treturn 31*System.identityHashCode(supercat) + 17*System.identityHashCode(rule) + System.identityHashCode(supercat2);\n \t}\n \t// equals\n \tpublic boolean equals(Object obj) {\n \t\tif (!(obj instanceof SupercatRuleCombo)) return false;\n \t\tSupercatRuleCombo combo = (SupercatRuleCombo) obj;\n \t\treturn supercat == combo.supercat && supercat2 == combo.supercat2 && rule == combo.rule;\n \t}\n \t// supercat hashcode, excluding rule\n \tpublic int supercatHashCode() {\n \t\treturn 31*System.identityHashCode(supercat) + System.identityHashCode(supercat2);\n \t}\n \t// supercat equals\n \tpublic boolean supercatEquals(Object obj) {\n \t\tif (!(obj instanceof SupercatRuleCombo)) return false;\n \t\tSupercatRuleCombo combo = (SupercatRuleCombo) obj;\n \t\treturn supercat == combo.supercat && supercat2 == combo.supercat2;\n \t}\n \t// toString\n \tpublic String toString() {\n \t\tStringBuffer sb = new StringBuffer(supercat);\n \t\tif (supercat2 != null) sb.append(' ').append(supercat2);\n \t\tsb.append(' ').append(rule);\n \t\treturn sb.toString();\n \t}\n }\n \n // class for seen combos when determined dynamically\n // nb: for space efficiency, allows representative to be retrieved from set\n private static class SupercatComboSet extends THashSet {\n\t\tprivate static final long serialVersionUID = 1L;\n\t\tSupercatComboSet() {\n \t\tsuper(\n \t new TObjectHashingStrategy() {\n\t\t\t\t\tprivate static final long serialVersionUID = 1L;\n\t\t\t\t\tpublic int computeHashCode(Object o) {\n \t\t\t\t\treturn (o instanceof SupercatRuleCombo) ? ((SupercatRuleCombo)o).supercatHashCode() : 0;\n \t }\n \t public boolean equals(Object o1, Object o2) {\n \t\t\t\t\treturn (o1 instanceof SupercatRuleCombo) ? ((SupercatRuleCombo)o1).supercatEquals(o2) : false;\n \t }\n \t }\n \t);\n \t}\n \t// return the seen combo, or null if none\n \tSupercatRuleCombo get(SupercatRuleCombo combo) {\n \t\tint index = index(combo);\n \t\tif (index < 0) return null;\n \t\treturn (SupercatRuleCombo) this._set[index];\n \t}\n }\n \n // observed supercat-rule combos\n private transient Set supercatRuleCombos = null;\n \n // observed supercat combos (for which complete rule combos are known)\n private transient SupercatComboSet supercatCombosSeen = null;\n \n // reusable combo for checking presence\n private transient SupercatRuleCombo combo = new SupercatRuleCombo(\"dummy\", \"dummy\");\n \n // flag for whether observed supercat combos is determined dynamically\n private boolean dynamicCombos = false;\n \n /**\n * Constructs an empty rule group for the given grammar.\n */\n public RuleGroup(Grammar grammar) {\n this.grammar = grammar;\n bapp.setRuleGroup(this);\n }\n \n /**\n * Constructs a rule group from the given URL, for \n * the given grammar.\n */\n public RuleGroup(URL url, Grammar grammar) throws IOException {\n\n this.grammar = grammar;\n bapp.setRuleGroup(this);\n \n XmlScanner ruleScanner = new XmlScanner() {\n \tpublic void handleElement(Element ruleEl) {\n String active = ruleEl.getAttributeValue(\"active\");\n if (active == null || active.equals(\"true\")) {\n try { addRule(readRule(ruleEl)); }\n catch (RuntimeException exc) {\n System.err.println(\"Skipping rule: \" + ruleEl.getAttributeValue(\"name\"));\n System.err.println(exc.toString());\n }\n }\n \t}\n };\n ruleScanner.parse(url);\n }\n\n \n // during deserialization, sets grammar to the current grammar\n private void readObject(java.io.ObjectInputStream in) throws IOException, ClassNotFoundException {\n \tin.defaultReadObject();\n \tgrammar = Grammar.theGrammar;\n \tborrowSupercatRuleCombos(grammar.rules);\n }\n \n \n // reads in a rule\n private Rule readRule(Element ruleEl) {\n Rule r;\n String type = ruleEl.getName();\n if (type.equals(\"application\")) {\n String dir = ruleEl.getAttributeValue(\"dir\");\n if (dir.equals(\"forward\")) {\n r = new ForwardApplication();\n } else {\n r = new BackwardApplication();\n }\n } else if (type.equals(\"composition\")) {\n String dir = ruleEl.getAttributeValue(\"dir\");\n String harmonic = ruleEl.getAttributeValue(\"harmonic\");\n boolean isHarmonic = new Boolean(harmonic).booleanValue();\n if (dir.equals(\"forward\")) {\n r = new ForwardComposition(isHarmonic);\n } else {\n r = new BackwardComposition(isHarmonic);\n }\n } else if (type.equals(\"substitution\")) {\n String dir = ruleEl.getAttributeValue(\"dir\");\n String harmonic = ruleEl.getAttributeValue(\"harmonic\");\n boolean isHarmonic = new Boolean(harmonic).booleanValue();\n if (dir.equals(\"forward\")) {\n r = new ForwardSubstitution(isHarmonic);\n } else {\n r = new BackwardSubstitution(isHarmonic);\n }\n } else if (type.equals(\"typeraising\")) {\n String dir = ruleEl.getAttributeValue(\"dir\");\n String useDollar = ruleEl.getAttributeValue(\"useDollar\");\n boolean addDollar = new Boolean(useDollar).booleanValue();\n Category arg = null;\n Element argElt = ruleEl.getChild(\"arg\");\n if (argElt != null) {\n arg = CatReader.getCat((Element)argElt.getChildren().get(0));\n }\n Category result = null;\n Element resultElt = ruleEl.getChild(\"result\");\n if (resultElt != null) {\n result = CatReader.getCat((Element)resultElt.getChildren().get(0));\n }\n if (dir.equals(\"forward\")) {\n r = new ForwardTypeRaising(addDollar, arg, result);\n } else {\n r = new BackwardTypeRaising(addDollar, arg, result);\n }\n } else if (type.equals(\"typechanging\")) {\n r = readTypeChangingRule(ruleEl);\n } else {\n throw new RuntimeException(\"Invalid element in rules: \" + type);\n }\n return r;\n }\n \n // reads in a type changing rule\n private Rule readTypeChangingRule(Element ruleEl) {\n \n String rname = ruleEl.getAttributeValue(\"name\");\n Element argCatElt = (Element)ruleEl.getChild(\"arg\").getChildren().get(0);\n Category arg = CatReader.getCat(argCatElt);\n Element resultCatElt = (Element)ruleEl.getChild(\"result\").getChildren().get(0);\n Element lfElt = resultCatElt.getChild(\"lf\");\n Category result = CatReader.getCat(resultCatElt);\n LF firstEP = null;\n if (lfElt != null) {\n firstEP = HyloHelper.firstEP(HyloHelper.getLF(lfElt));\n }\n \n grammar.lexicon.propagateTypes(result, arg);\n grammar.lexicon.propagateDistributiveAttrs(result, arg);\n grammar.lexicon.expandInheritsFrom(result, arg);\n\n return new TypeChangingRule(arg, result, rname, firstEP);\n }\n\n /**\n * Writes the rules to an XML file with the given name.\n * @throws IOException \n */\n public void toXml(String filename) throws IOException {\n \tXMLOutputter xout = new XMLOutputter();\n \txout.setFormat(Format.getPrettyFormat());\n \tPrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(filename)));\n \tout.println(\"\");\n \tout.println(\"\");\n \tfor (Rule r : binaryRules) { \n \t\txout.output(r.toXml(), out); out.println();\n \t}\n \tfor (Rule r : unaryRules) {\n \t\txout.output(r.toXml(), out); out.println();\n \t}\n \tout.println(\"\");\n \tout.flush(); out.close();\n }\n \n /**\n * Sets the dynamic combos flag to the given value, controlling whether the \n * observed supercat combos is determined dynamically.\n */\n public void setDynamicCombos(boolean dynamic) {\n \tthis.dynamicCombos = dynamic;\n \tif (!dynamicCombos) supercatCombosSeen = null;\n \telse if (dynamicCombos) {\n \t\tif (supercatCombosSeen == null) supercatCombosSeen = new SupercatComboSet();\n \t\tif (supercatRuleCombos == null) supercatRuleCombos = new HashSet();\n \t}\n }\n \n /**\n * Returns the dynamic combos flag.\n */\n public boolean getDynamicCombos() { return dynamicCombos; }\n \n /** \n * Loads the observed supercat-rule combos, for filtering. \n * Only file URLs are supported at present.\n * Missing files are ignored. \n **/\n public void loadSupercatRuleCombos(URL url) throws IOException {\n \tsupercatRuleCombos = new HashSet();\n \tFile combosFile = new File(url.getFile());\n \tif (!combosFile.exists()) return;\n \tSystem.out.println(\"Loading supercat combos from \" + url.getFile());\n \tBufferedReader in = new BufferedReader(new FileReader(combosFile));\n \tString line;\n \twhile ((line = in.readLine()) != null) {\n \t\tString[] tokens = line.split(\"\\\\s\");\n \t\tif (tokens.length < 2) {\n \t\t\tSystem.err.println(\"Warning: skipping supercat-rule combo with fewer than two tokens: \" + line);\n \t\t\tcontinue;\n \t\t}\n \t\tif (tokens.length == 2) {\n \t\t\tsupercatRuleCombos.add(new SupercatRuleCombo(tokens[0], tokens[1]));\n \t\t}\n \t\telse {\n\t \t\tif (tokens.length > 3) {\n\t \t\t\tSystem.err.println(\"Warning: ignoring extra tokens (beyond 3rd) in supercat-rule combo: \" + line);\n\t \t\t}\n \t\t\tsupercatRuleCombos.add(new SupercatRuleCombo(tokens[0], tokens[1], tokens[2]));\n \t\t}\n \t}\n \tin.close();\n }\n \n \n /** Borrows the observed supercat-rule combos from the given rule group. */\n public void borrowSupercatRuleCombos(RuleGroup ruleGroup) {\n \tsupercatRuleCombos = ruleGroup.supercatRuleCombos;\n \tsupercatCombosSeen = ruleGroup.supercatCombosSeen;\n }\n \n \n /** Adds the given rule. */\n public void addRule(Rule r) {\n r.setRuleGroup(this);\n if (r instanceof TypeChangingRule) {\n unaryRules.add(r);\n index((TypeChangingRule)r);\n }\n else if (r.arity() == 1) { unaryRules.add(r); } \n else if (r.arity() == 2) { binaryRules.add(r); } \n else {\n // shouldn't happen\n throw new RuntimeException(\"Can't determine arity of rule: \" + r);\n }\n }\n\n // indexes type changing rules by preds and rels\n private void index(TypeChangingRule rule) {\n LF firstEP = rule.getFirstEP();\n if (firstEP == null) { return; }\n String pred = HyloHelper.getLexPred(firstEP);\n if (pred != null) { \n predsToRules.put(pred, rule); \n return; \n }\n String rel = HyloHelper.getRel(firstEP);\n if (rel != null) { \n relsToRules.put(rel, rule);\n }\n }\n \n \n /** Returns the unary rules. */\n public List getUnaryRules() { return unaryRules; }\n\n /** Returns the binary rules. */\n public List getBinaryRules() { return binaryRules; }\n\n /** Returns the type changing rule with the given name, or null if none. */\n public TypeChangingRule getTypeChangingRule(String name) {\n for (Iterator it = unaryRules.iterator(); it.hasNext(); ) {\n Object rule = it.next();\n if (rule instanceof TypeChangingRule) {\n TypeChangingRule tcr = (TypeChangingRule) rule;\n if (tcr.name().equals(name)) return tcr;\n }\n }\n return null;\n }\n \n /**\n * Returns the type changing rules indexed by the given lexical predicate. \n * The type changing rules are indexed by their first elementary predication.\n */\n public Collection getRulesForPred(String pred) {\n return predsToRules.get(pred);\n }\n \n /**\n * Returns the type changing rules indexed by the given relation.\n * The type changing rules are indexed by their first elementary predication.\n */\n public Collection getRulesForRel(String rel) {\n return relsToRules.get(rel);\n }\n \n \n /** Applies the unary rules to the given input sign, returning the list of results. */\n public List applyUnaryRules(Sign input) {\n \tSign[] inputs = { input };\n List results = new ArrayList(2);\n String supertag = input.getCategory().getSupertag();\n // check whether dynamic combos update required, or whether rules can be skipped\n boolean dynamicCombosUpdate = false;\n boolean skip = false;\n if (dynamicCombos) {\n \t\tcombo.setCombo(supertag, null);\n \t\tSupercatRuleCombo rep = supercatCombosSeen.get(combo);\n \t\tif (rep == null) dynamicCombosUpdate = true;\n \t\telse if (rep.rule == null) skip = true;\n }\n // skip if possible\n if (skip) return results;\n // try each rule \n for (Rule r : unaryRules) {\n \t// filter on observed supercat-rule combos, if any, if not updating\n \tif (!dynamicCombosUpdate && supercatRuleCombos != null) {\n \t\tcombo.setCombo(supertag, r.name());\n \t\tif (!supercatRuleCombos.contains(combo)) { continue; }\n \t}\n \t// if updating combos, apply rule and record results\n \tif (dynamicCombosUpdate) {\n \t\tint prevsize = results.size();\n \t((AbstractRule)r).applyRule(inputs, results);\n \t// update upon success\n \tif (results.size() > prevsize) {\n \t\tSupercatRuleCombo newCombo = null;\n \t\tcombo.setCombo(supertag, r.name());\n \t\tif (!supercatRuleCombos.contains(combo)) { \n \t\t\tnewCombo = new SupercatRuleCombo(supertag, r.name());\n \t\t\tsupercatRuleCombos.add(newCombo);\n \t\t}\n \t\tif (!supercatCombosSeen.contains(combo)) {\n \t\t\tif (newCombo == null) newCombo = new SupercatRuleCombo(supertag, r.name());\n \t\tsupercatCombosSeen.add(newCombo);\n \t\t}\n \t}\n \t}\n \t// otherwise just apply rule\n \telse ((AbstractRule)r).applyRule(inputs, results);\n }\n // if updating combos and none succeeded, add one with null rule\n if (dynamicCombosUpdate) {\n \t\tcombo.setCombo(supertag, null);\n \t\tif (!supercatCombosSeen.contains(combo)) {\n \t\t\tSupercatRuleCombo newCombo = new SupercatRuleCombo(supertag, null);\n \t\t\tsupercatCombosSeen.add(newCombo);\n \t\t}\n }\n // done\n return results;\n }\n \n /** Applies the binary rules to the given input signs, returning the list of results. */\n public List applyBinaryRules(Sign input1, Sign input2) {\n \tSign[] inputs = { input1, input2 };\n List results = new ArrayList(2);\n\t\tString supertag1 = input1.getCategory().getSupertag();\n\t\tString supertag2 = input2.getCategory().getSupertag();\n // check whether dynamic combos update required, or whether rules can be skipped\n boolean dynamicCombosUpdate = false;\n boolean skip = false;\n if (dynamicCombos) {\n \t\tcombo.setCombo(supertag1, supertag2, null);\n \t\tSupercatRuleCombo rep = supercatCombosSeen.get(combo);\n \t\tif (rep == null) dynamicCombosUpdate = true;\n \t\telse if (rep.rule == null) skip = true;\n }\n // skip if possible\n if (skip) return results;\n // try each rule\n for (Rule r : binaryRules) {\n \t// filter on observed supercat-rule combos, if any, if not updating\n \tif (!dynamicCombosUpdate && supercatRuleCombos != null) {\n \t\tcombo.setCombo(supertag1, supertag2, r.name());\n \t\tif (!supercatRuleCombos.contains(combo)) { continue; }\n \t}\n \t// if updating combos, apply rule and record results\n \tif (dynamicCombosUpdate) {\n \t\tint prevsize = results.size();\n \t((AbstractRule)r).applyRule(inputs, results);\n \t// update upon success\n \tif (results.size() > prevsize) {\n \t\tSupercatRuleCombo newCombo = null;\n \t\tcombo.setCombo(supertag1, supertag2, r.name());\n \t\tif (!supercatRuleCombos.contains(combo)) { \n \t\t\tnewCombo = new SupercatRuleCombo(supertag1, supertag2, r.name());\n \t\t\tsupercatRuleCombos.add(newCombo);\n \t\t}\n \t\tif (!supercatCombosSeen.contains(combo)) {\n \t\t\tif (newCombo == null) newCombo = new SupercatRuleCombo(supertag1, supertag2, r.name());\n \t\tsupercatCombosSeen.add(newCombo);\n \t\t}\n \t}\n \t}\n \t// otherwise just apply rule\n \telse ((AbstractRule)r).applyRule(inputs, results);\n }\n // if updating combos and none succeeded, add one with null rule\n if (dynamicCombosUpdate) {\n \t\tcombo.setCombo(supertag1, supertag2, null);\n \t\tif (!supercatCombosSeen.contains(combo)) {\n \t\tSupercatRuleCombo newCombo = new SupercatRuleCombo(supertag1, supertag2, null);\n \t\t\tsupercatCombosSeen.add(newCombo);\n \t\t}\n }\n // done\n return results;\n }\n \n \n /** Applies the glue rule to the given input signs, returning the list of results. */\n public List applyGlueRule(Sign input1, Sign input2) {\n \tSign[] inputs = { input1, input2 };\n List results = new ArrayList(1);\n \tglueRule.applyRule(inputs, results);\n return results;\n }\n\n \n /** Applies the coarticulation to the given sign, adding the result (if any) to the given ones. */\n public void applyCoart(Sign lexSign, Sign coartSign, List results) {\n\n Category[] cats = new Category[] { lexSign.getCategory(), coartSign.getCategory() }; \n\n try {\n List resultCats = bapp.applyRule(cats);\n if (resultCats.isEmpty()) return;\n \n for (Iterator it = resultCats.iterator(); it.hasNext();) {\n Category catResult = it.next();\n bapp.distributeTargetFeatures(catResult);\n Sign sign = Sign.createCoartSign(catResult, lexSign, coartSign);\n results.add(sign);\n }\n } catch (UnifyFailure uf) {}\n }\n}\n" }, { "path": "src/opennlp/ccg/grammar/TypeChangingRule.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-4 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.grammar;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.hylo.HyloHelper;\nimport opennlp.ccg.synsem.*;\n\nimport java.util.*;\n\nimport org.jdom.Element;\n\n/**\n * A CCG unary type changing rule.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.12 $, $Date: 2009/11/30 20:36:16 $\n **/\npublic class TypeChangingRule extends AbstractRule implements LexSemOrigin {\n\t\n\tprivate static final long serialVersionUID = -2654945192870162776L;\n\n\t/**\n\t * String used as the POS for all type changing rules, \n\t * to satisfy the LexSemOrigin interface.\n\t * Defaults to \"URULE\".\n\t */\n\tpublic static String POS_STRING = \"URULE\";\n\t\n\n /** The argument category. */ \n protected Category _arg; \n\n /** The result category. */\n protected Category _result; \n \n /** The first elementary predication in the result LF (or null), before sorting. */\n protected LF _firstEP;\n\n\n /** Constructor. */\n public TypeChangingRule(Category arg, Category result, String name, LF firstEP) {\n _arg = arg; _result = result; _name = name.intern(); _firstEP = firstEP;\n setOrigin();\n }\n\n /** Returns an XML element representing the rule. */\n public Element toXml() {\n \tElement retval = new Element(\"typechanging\");\n \tretval.setAttribute(\"name\", _name);\n \tElement argElt = new Element(\"arg\");\n \tretval.addContent(argElt);\n \targElt.addContent(_arg.toXml());\n \tElement resultElt = new Element(\"result\");\n \tretval.addContent(resultElt);\n \tresultElt.addContent(_result.toXml());\n \treturn retval;\n }\n\n /** Returns 1. */\n public int arity() { return 1; }\n \n /** Returns the arg. */\n public Category getArg() { return _arg; }\n \n /** Returns the result. */\n public Category getResult() { return _result; }\n\n /** Returns the first elementary predication in the result LF (or null), before sorting. */\n public LF getFirstEP() { return _firstEP; }\n\n\n /** Applies this rule to the given inputs. */\n public List applyRule(Category[] inputs) throws UnifyFailure {\n // check arity\n if (inputs.length != 1) {\n throw new UnifyFailure();\n }\n return apply(inputs[0]);\n }\n\n /** Applies this rule to the given input. */\n protected List apply(Category input) throws UnifyFailure {\n\n // unify quick check\n _arg.unifyCheck(input);\n \n // copy arg and result\n Category arg = _arg.copy();\n Category result = _result.copy();\n \n // make variables unique\n UnifyControl.reindex(result, arg);\n\n // unify\n Substitution sub = new GSubstitution();\n GUnifier.unify(input, arg, sub);\n ((GSubstitution)sub).condense();\n\n // fill in result\n Category $result = (Category)result.fill(sub);\n appendLFs(input, result, $result, sub);\n\n // return\n List results = new ArrayList(1);\n _headCats.clear();\n results.add($result);\n _headCats.add(input);\n return results;\n }\n \n \n /** Returns 'name: arg => result'. */\n public String toString() {\n StringBuffer sb = new StringBuffer();\n sb.append(_name).append(\": \");\n sb.append(_arg).append(' ');\n sb.append(\"=> \").append(_result);\n return sb.toString();\n }\n \n /** Returns 'arg_=>_result' as the supertag. */\n public String getSupertag() {\n StringBuffer sb = new StringBuffer();\n sb.append(_arg.getSupertag()).append(\"_=>_\").append(_result.getSupertag());\n return sb.toString();\n }\n \n /**\n * Always returns POS_STRING. \n */\n public String getPOS() { return POS_STRING; }\n \n /**\n * Sets the origin of the elementary predications.\n */\n public void setOrigin() { HyloHelper.setOrigin(_result.getLF(), this); }\n}\n\n" }, { "path": "src/opennlp/ccg/grammar/Types.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n//// Copyright (C) 2003-4 Gunes Erkan and University of Edinburgh (Michael White)\r\n//// \r\n//// This library is free software; you can redistribute it and/or\r\n//// modify it under the terms of the GNU Lesser General Public\r\n//// License as published by the Free Software Foundation; either\r\n//// version 2.1 of the License, or (at your option) any later version.\r\n//// \r\n//// This library is distributed in the hope that it will be useful,\r\n//// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n//// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n//// GNU Lesser General Public License for more details.\r\n//// \r\n//// You should have received a copy of the GNU Lesser General Public\r\n//// License along with this program; if not, write to the Free Software\r\n//// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n////////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.grammar;\r\n\r\nimport opennlp.ccg.util.*;\r\nimport opennlp.ccg.unify.*;\r\n\r\nimport org.jdom.*;\r\nimport org.jdom.input.*;\r\n\r\nimport java.io.*;\r\nimport java.net.*;\r\nimport java.util.*;\r\n\r\nimport gnu.trove.*;\r\n\r\n/**\r\n * Class for constructing and holding the hierarchical simple type maps.\r\n *\r\n * @author Gunes Erkan\r\n * @author Michael White\r\n * @version $Revision: 1.13 $, $Date: 2009/12/21 03:27:18 $\r\n */\r\npublic class Types {\r\n\r\n public final Grammar grammar;\r\n private final HashMap nameToType = new HashMap();\r\n private final ArrayList indexToType = new ArrayList();\r\n private int maxTypeIndex = 0;\r\n public static final String TOP_TYPE = \"top\";\r\n public static final String BOT_TYPE = \"bottom\";\r\n\t\r\n /** Constructor for an empty hierarchy (with just the top type). */\r\n public Types(Grammar grammar) {\r\n getSimpleType(TOP_TYPE);\r\n this.grammar = grammar;\r\n }\r\n\r\n /**\r\n * Constructs the type hierarchy from the given URL, for \r\n * the given grammar.\r\n */\r\n @SuppressWarnings(\"unchecked\")\r\n\tpublic Types(URL url, Grammar grammar) throws IOException {\r\n this.grammar = grammar;\r\n SAXBuilder builder = new SAXBuilder();\r\n Document doc;\r\n try {\r\n doc = builder.build(url);\r\n }\r\n catch (JDOMException exc) {\r\n getSimpleType(TOP_TYPE);\r\n throw (IOException) new IOException().initCause(exc);\r\n }\r\n List entries = doc.getRootElement().getChildren();\r\n readTypes(entries);\r\n // for debugging: print the indexToType list\r\n //printTypes();\r\n }\r\n \r\n /** Returns the simple type with the given name, or a new one if none yet exists. */\r\n public SimpleType getSimpleType(String typeName) {\r\n SimpleType type = nameToType.get(typeName);\r\n if (type == null) {\r\n BitSet bs = new BitSet();\r\n bs.set(maxTypeIndex);\r\n SimpleType newtype = new SimpleType(maxTypeIndex, typeName, bs, this);\r\n nameToType.put(typeName, newtype);\r\n indexToType.add(newtype);\r\n nameToType.get(TOP_TYPE).getBitSet().set(maxTypeIndex++);\r\n return newtype;\r\n }\r\n else return type;\r\n }\r\n\r\n /** Returns whether there is a simple type with the given name. */\r\n public boolean containsSimpleType(String typeName) {\r\n return nameToType.containsKey(typeName);\r\n }\r\n \r\n /** Returns the list of types, with parents preceding children in the hierarchy. */\r\n public ArrayList getIndexMap() {\r\n return indexToType;\r\n }\r\n\r\n\r\n /** Reads the rules and constructs the nameToType and indexToType maps. */\r\n private void readTypes(List _types) {\r\n \r\n GroupMap hierarchy = new GroupMap(); // map from types to all subtypes\r\n GroupMap parents = new GroupMap(); // map from types to parents\r\n TObjectIntHashMap depthMap = new TObjectIntHashMap(); // map from types to max depth\r\n\r\n // Construct the initial hierarchy of types without \r\n // taking transitive closure.\r\n // Also store parents.\r\n for (int i=0; i < _types.size(); i++) {\r\n Element typeEl = _types.get(i);\r\n String typeName = typeEl.getAttributeValue(\"name\");\r\n String _parents = typeEl.getAttributeValue(\"parents\");\r\n hierarchy.put(typeName, BOT_TYPE);\r\n if (_parents == null) {\r\n hierarchy.put(TOP_TYPE, typeName);\r\n parents.put(typeName, TOP_TYPE);\r\n }\r\n \t\telse {\r\n \t\t\tString[] parentsArray = _parents.split(\"\\\\s+\");\r\n \t\t\tfor (int j = 0; j < parentsArray.length; j++) {\r\n \t\t\t\thierarchy.put(parentsArray[j], typeName);\r\n parents.put(typeName, parentsArray[j]);\r\n \t\t\t}\r\n \t\t}\r\n \t}\r\n\r\n // Compute depth from parents.\r\n for (String type : parents.keySet()) {\r\n int depth = computeDepth(type, parents, type);\r\n depthMap.put(type, depth);\r\n }\r\n\r\n \t// Compute ALL subtypes of each type and insert into the hierarchy.\r\n \tfor (String type : hierarchy.keySet()) { \r\n \t\t hierarchy.putAll(type, findAllSubtypes(hierarchy, type));\r\n \t\t}\r\n \r\n \t// Assign a unique int to each type in breadth-first order.\r\n \t// Then create the string -> SimpleType map.\r\n \tcreateSimpleTypes(hierarchy, depthMap);\r\n }\r\n\r\n /** Returns the max depth of the given type, checking for cycles. */\r\n private static int computeDepth(String type, GroupMap parents, String startType) {\r\n if (type.equals(TOP_TYPE)) return 0;\r\n int maxParentDepth = 0;\r\n Set parentSet = parents.get(type);\r\n\tif (parentSet != null) {\r\n\t for (String parent : parentSet) {\r\n\t\tif (parent.equals(startType)) {\r\n\t\t throw new RuntimeException(\"Error, type hierarchy contains cycle from/to: \" + startType);\r\n\t\t}\r\n\t\tint parentDepth = computeDepth(parent, parents, startType);\r\n\t\tmaxParentDepth = Math.max(maxParentDepth, parentDepth);\r\n\t }\r\n\t}\r\n return maxParentDepth + 1;\r\n }\r\n \r\n /** \r\n * Computes the list of all sub-types of a given type (key) \r\n * in depth-first order. \r\n */\r\n private Collection findAllSubtypes(GroupMap hierarchy, String key) {\r\n ArrayList subs = new ArrayList();\r\n if (hierarchy.get(key) != null) {\r\n \t Stack look = new Stack();\r\n\t for (String type : hierarchy.get(key)) {\r\n\t\tlook.push(type);\r\n\t }\r\n for (; !look.empty() ; ) {\r\n String new_sub = look.pop();\r\n subs.add(new_sub);\r\n if (hierarchy.get(new_sub) != null) {\r\n for (String type : hierarchy.get(new_sub)) {\r\n look.push(type);\r\n }\r\n }\r\n }\r\n }\r\n return subs;\r\n }\r\n\r\n /** \r\n * Creates the SimpleType objects and constructs the nameToType and indexToType maps. \r\n */\r\n private void createSimpleTypes(GroupMap hierarchy, TObjectIntHashMap depthMap) {\r\n \r\n // find max depth\r\n int maxDepth = 0;\r\n int[] depths = depthMap.getValues();\r\n for (int i = 0; i < depths.length; i++) {\r\n maxDepth = Math.max(maxDepth, depths[i]);\r\n }\r\n\r\n // add types in order of increasing depth\r\n ArrayList typesVisited = new ArrayList();\r\n typesVisited.add(TOP_TYPE);\r\n Object[] types = depthMap.keys();\r\n ArrayList typesAtSameDepth = new ArrayList();\r\n for (int i = 1; i <= maxDepth; i++) {\r\n typesAtSameDepth.clear();\r\n for (int j = 0; j < types.length; j++) {\r\n if (depthMap.get(types[j]) == i)\r\n typesAtSameDepth.add((String)types[j]);\r\n }\r\n Collections.sort(typesAtSameDepth);\r\n typesVisited.addAll(typesAtSameDepth);\r\n }\r\n\r\n // construct the maps\r\n for (int i=0; i < typesVisited.size(); i++) {\r\n String typeName = typesVisited.get(i);\r\n BitSet bitset = new BitSet();\r\n bitset.set(i);\r\n\t if (hierarchy.get(typeName) != null) {\r\n\t\tfor (String type : hierarchy.get(typeName)) {\r\n\t\t int indexToSet = typesVisited.indexOf(type); \r\n\t\t if (indexToSet != -1) bitset.set(indexToSet);\r\n\t\t}\r\n\t }\r\n SimpleType st = new SimpleType(i, typeName, bitset, this);\r\n nameToType.put(typeName, st);\r\n indexToType.add(st);\r\n }\r\n maxTypeIndex = typesVisited.size();\r\n }\r\n \r\n /**\r\n * Prints the types and their subtypes to System.out.\r\n */\r\n public void printTypes() {\r\n System.out.println(\"types:\");\r\n for (int i=0; i < indexToType.size(); i++) {\r\n SimpleType st = indexToType.get(i); \r\n System.out.println(i + \": \" + st.getName() + \" subtypes: \" + st.getBitSet());\r\n }\r\n System.out.println();\r\n }\r\n \r\n /** Tests serialization of simple types, including resolution. */\r\n public void debugSerialization() throws IOException, ClassNotFoundException {\r\n // test serialization\r\n SimpleType st = indexToType.get(1);\r\n \tString filename = \"tmp.ser\";\r\n \tObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(filename));\r\n \tSystem.out.println(\"Writing st: \" + st.getIndex() + \": \" + st + \" \" + st.getBitSet());\r\n \tout.writeObject(st);\r\n \tout.close();\r\n \tObjectInputStream in = new ObjectInputStream(new FileInputStream(filename));\r\n \tSystem.out.print(\"Reading st2: \");\r\n \tSimpleType st2 = (SimpleType) in.readObject();\r\n \tSystem.out.println(st2.getIndex() + \": \" + st2 + \" \" + st2.getBitSet());\r\n \tin.close();\r\n \t// test identity (and thus readResolve)\r\n \tSystem.out.println(\"st == st2?: \" + (st == st2));\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/grammar/to-apml.xsl", "content": "\n\n\n \n \n \n \n\n '\n n't\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n Hstar\n Lstar\n LplusHstar\n LstarplusH\n HstarplusL\n HplusLstar\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n aamm\n \n \n ppmm\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "src/opennlp/ccg/grammardoc/AbstractDocumenter.java", "content": "/*\n * $Id: AbstractDocumenter.java,v 1.2 2006/11/01 02:53:20 coffeeblack Exp $\n */\npackage opennlp.ccg.grammardoc;\n\n/**\n * Convenience base class for documenters to extend\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.2 $\n */\npublic abstract class AbstractDocumenter implements Documenter {\n\n\tprotected DocumenterContext documenterContext;\n\tprotected String name;\n\n\t/**\n\t * Default constructor. All documenters must have a no-argument constructor.\n\t */\n\tprotected AbstractDocumenter() {}\n\n\t/**\n\t * Creates a new documenter with the specified name.\n\t */\n\tprotected AbstractDocumenter(String name) {\n\t\tthis.name = name;\n\t}\n\n\t/**\n\t * Sets this documenter's context.\n\t */\n\tpublic void setDocumenterContext(DocumenterContext documenterContext) {\n\t\tthis.documenterContext = documenterContext;\n\t}\n\n\t/**\n\t * Gets this documenter's name.\n\t */\n\tpublic String getName() {\n\t\treturn name;\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/Documenter.java", "content": "/*\n * $Id: Documenter.java,v 1.2 2006/11/01 02:53:20 coffeeblack Exp $\n */\npackage opennlp.ccg.grammardoc;\n\n/**\n * A documenter that produces documentation for an OpenCCG grammar.\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.2 $\n */\npublic interface Documenter {\n\n\t/**\n\t * Gets the name of this documenter.\n\t * \n\t * @return A string like SGML Documenter.\n\t */\n\tpublic String getName();\n\n\t/**\n\t * Sets this documenter's context.\n\t * \n\t * @param documenterContext The context this documenter should use.\n\t */\n\tpublic void setDocumenterContext(DocumenterContext documenterContext);\n\n\t/**\n\t * Causes a documenter to produce documentation for the provided grammar.\n\t * Before any calls to this method are made, the documenter will first be\n\t * configured with a (single) call to\n\t * {@link #setDocumenterContext(DocumenterContext)}.\n\t * \n\t * @param grammar The grammar to document.\n\t * @throws DocumenterException If any problems occur during the process of\n\t * generating documentation.\n\t */\n\tpublic void document(SourceGrammar grammar) throws DocumenterException;\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/DocumenterContext.java", "content": "/*\n * $Id: DocumenterContext.java,v 1.2 2006/11/01 02:53:20 coffeeblack Exp $\n */\npackage opennlp.ccg.grammardoc;\n\nimport java.io.File;\n\n/**\n * Provides a context inside which a {@link Documenter} will execute. This class\n * gives the documenter access to objects it will need to produce its\n * documentation.\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.2 $\n */\npublic interface DocumenterContext {\n\n\t/**\n\t * Logs a message from a documenter.\n\t */\n\tpublic void log(String message);\n\n\t/**\n\t * Gets the target location where the documenter should generate its\n\t * documentation files.\n\t * \n\t * @return A directory that exists in a filesystem.\n\t */\n\tpublic File getDestinationDirectory();\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/DocumenterException.java", "content": "/*\n * $Id: DocumenterException.java,v 1.2 2006/11/01 02:53:20 coffeeblack Exp $\n */\npackage opennlp.ccg.grammardoc;\n\n/**\n * An exception thrown by a {@link Documenter}.\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.2 $\n */\npublic class DocumenterException extends GrammarDocException {\n\n\tprivate static final long serialVersionUID = 1L;\n\n\t/**\n\t * Creates a new documenter exception.\n\t */\n\tpublic DocumenterException() {\n\t\tsuper();\n\t}\n\n\t/**\n\t * Creates a new exception with the specified message.\n\t */\n\tpublic DocumenterException(String message) {\n\t\tsuper(message);\n\t}\n\n\t/**\n\t * Creates a new exception with the specified message and underlying cause.\n\t */\n\tpublic DocumenterException(String message, Throwable cause) {\n\t\tsuper(message, cause);\n\t}\n\n\t/**\n\t * Creates a new exception with the specified underlying cause.\n\t */\n\tpublic DocumenterException(Throwable cause) {\n\t\tsuper(cause);\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/DocumenterFactory.java", "content": "/*\n * $Id: DocumenterFactory.java,v 1.4 2006/12/11 18:19:24 coffeeblack Exp $\n */\npackage opennlp.ccg.grammardoc;\n\nimport java.util.EnumMap;\nimport java.util.Map;\n\n/**\n * Factory class for creating documenters based on a predefined name.\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.4 $\n */\npublic class DocumenterFactory {\n\n\tprivate static final Map documenters\n\t\t= new EnumMap(DocumenterName.class);\n\tstatic final DocumenterFactory documenterFactory = new DocumenterFactory();\n\n\tprivate DocumenterFactory() {}\n\n\t/**\n\t * Gets a new documenter factory.\n\t */\n\tpublic static DocumenterFactory newInstance() {\n\t\treturn documenterFactory;\n\t}\n\n\t/**\n\t * Gets a new instance of the default documenter.\n\t * \n\t * @return Calls {@link #newDocumenter(DocumenterName)} with\n\t * {@link DocumenterName#DEFAULT} as its argument.\n\t * @throws DocumenterNotFoundException Does not throw this exception. This\n\t * is included for binary compatibility with\n\t * {@link #newDocumenter(DocumenterName)}.\n\t */\n\tpublic Documenter newDocumenter() throws DocumenterNotFoundException {\n\t\treturn newDocumenter(DocumenterName.DEFAULT);\n\t}\n\n\t/**\n\t * Gets a new instance of the named documenter.\n\t * \n\t * @param name Used to look up the documenter {@link DocumenterName name}.\n\t * @return A documenter that corresponds to the specified name.\n\t * @throws DocumenterNotFoundException If no documenter can be created for\n\t * the specified name.\n\t */\n\tpublic Documenter newDocumenter(String name)\n\t\t\tthrows DocumenterNotFoundException {\n\t\ttry {\n\t\t\tDocumenterName nm = DocumenterName.valueOf(name);\n\t\t\treturn newDocumenter(nm);\n\t\t}\n\t\tcatch(IllegalArgumentException iae) {\n\t\t\tthrow new DocumenterNotFoundException(name);\n\t\t}\n\t}\n\n\t/**\n\t * Gets a new instance of the named documenter.\n\t * \n\t * @param name The {@link DocumenterName name} of the documenter to create.\n\t * @return A documenter that corresponds to the specified name.\n\t * @throws DocumenterNotFoundException If no documenter can be created for\n\t * the specified name.\n\t */\n\tpublic synchronized Documenter newDocumenter(DocumenterName name)\n\t\t\tthrows DocumenterNotFoundException {\n\t\tDocumenter d = documenters.get(name);\n\n\t\tif(d == null) {\n\t\t\ttry {\n\t\t\t\td = name.documenterClass.newInstance();\n\t\t\t}\n\t\t\tcatch(InstantiationException ie) {\n\t\t\t\tthrow new DocumenterNotFoundException(name, ie);\n\t\t\t}\n\t\t\tcatch(IllegalAccessException iae) {\n\t\t\t\tthrow new DocumenterNotFoundException(name, iae);\n\t\t\t}\n\t\t\t\n\t\t\tdocumenters.put(name, d);\n\t\t}\n\n\t\treturn d;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/DocumenterName.java", "content": "/*\n * $Id: DocumenterName.java,v 1.3 2006/12/11 18:19:24 coffeeblack Exp $\n */\npackage opennlp.ccg.grammardoc;\n\nimport opennlp.ccg.grammardoc.html.HTMLDocumenter;\n\n/**\n * The known documenters that the {@link DocumenterFactory documenter factory}\n * is aware of. Calling {@link DocumenterFactory#newDocumenter(DocumenterName)}\n * with any of these values will return a valid documenter without throwing an\n * exception.\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.3 $\n */\npublic enum DocumenterName {\n\t/**\n\t * The default documenter.\n\t */\n\tDEFAULT(HTMLDocumenter.class),\n\n\t/**\n\t * A documenter that produces HTML output.\n\t */\n\tHTML(HTMLDocumenter.class);\n\t\n\tClass documenterClass;\n\t\n\tprivate DocumenterName(Class documenterClass) {\n\t\tthis.documenterClass = documenterClass;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/DocumenterNotFoundException.java", "content": "/*\n * $Id: DocumenterNotFoundException.java,v 1.3 2006/12/11 18:19:24 coffeeblack Exp $\n */\npackage opennlp.ccg.grammardoc;\n\n/**\n * Signals that no documenter could be found for a given name.\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.3 $\n */\npublic class DocumenterNotFoundException extends DocumenterException {\n\n\tprivate static final long serialVersionUID = 1L;\n\n\t/**\n\t * Creates a new exception indicating that a documenter with the specified\n\t * name was not found.\n\t */\n\tpublic DocumenterNotFoundException(DocumenterName name) {\n\t\tthis(name.name());\n\t}\n\n\t/**\n\t * Creates a new exception indicating that a documenter with the specified\n\t * name was not found.\n\t */\n\tpublic DocumenterNotFoundException(String name) {\n\t\tsuper(name);\n\t}\n\n\t/**\n\t * Creates a new exception indicating that a documenter with the specified\n\t * name was not found for the specified underlying reason.\n\t */\n\tDocumenterNotFoundException(DocumenterName name, Throwable cause) {\n\t\tsuper(name.name(), cause);\n\t}\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/DocumenterSourceException.java", "content": "/*\n * $Id: DocumenterSourceException.java,v 1.2 2006/11/01 02:53:20 coffeeblack Exp $\n */\npackage opennlp.ccg.grammardoc;\n\n/**\n * An exception thrown by a {@link Documenter} because of a problem in the\n * source grammar. This exception tracks the\n * {@link SourceGrammarFile source grammar file} where the problem occurred.\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.2 $\n */\npublic class DocumenterSourceException extends DocumenterException {\n\n\tprivate static final long serialVersionUID = 1L;\n\n\tprotected SourceGrammarFile sourceGrammarFile;\n\n\t/**\n\t * Creates a new exception signaling a problem in the specified source\n\t * grammar file.\n\t */\n\tpublic DocumenterSourceException(SourceGrammarFile sourceGrammarFile) {\n\t\tthis.sourceGrammarFile = sourceGrammarFile;\n\t}\n\n\t/**\n\t * Creates a new exception with the specified message, signaling a problem\n\t * in the specified source grammar file.\n\t */\n\tpublic DocumenterSourceException(String message,\n\t\t\tSourceGrammarFile sourceGrammarFile) {\n\t\tsuper(message);\n\t\tthis.sourceGrammarFile = sourceGrammarFile;\n\t}\n\n\t/**\n\t * Creates a new exception with the specified message and underlying cause,\n\t * signaling a problem in the specified source grammar file.\n\t */\n\tpublic DocumenterSourceException(String message, Throwable cause,\n\t\t\tSourceGrammarFile sourceGrammarFile) {\n\t\tsuper(message, cause);\n\t\tthis.sourceGrammarFile = sourceGrammarFile;\n\t}\n\n\t/**\n\t * Creates a new exception with the specified underlying cause, signaling a\n\t * problem in the specified source grammar file.\n\t */\n\tpublic DocumenterSourceException(Throwable cause,\n\t\t\tSourceGrammarFile sourceGrammarFile) {\n\t\tsuper(cause);\n\t\tthis.sourceGrammarFile = sourceGrammarFile;\n\t}\n\n\t/**\n\t * Gets the source grammar file where this problem occurred.\n\t */\n\tpublic SourceGrammarFile getSourceGrammarFile() {\n\t\treturn sourceGrammarFile;\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/GrammarDoc.java", "content": "/*\n * $Id: GrammarDoc.java,v 1.7 2007/05/30 22:53:17 coffeeblack Exp $\n */\npackage opennlp.ccg.grammardoc;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.PrintStream;\nimport java.util.Arrays;\nimport java.util.Iterator;\nimport java.util.List;\n\nimport javax.xml.parsers.DocumentBuilder;\nimport javax.xml.parsers.DocumentBuilderFactory;\nimport javax.xml.parsers.ParserConfigurationException;\n\nimport org.apache.tools.ant.BuildException;\nimport org.apache.tools.ant.Project;\nimport org.apache.tools.ant.Target;\nimport org.apache.tools.ant.Task;\nimport org.w3c.dom.Document;\nimport org.w3c.dom.NodeList;\nimport org.xml.sax.SAXException;\n\n/**\n * Class that implements the grammardoc ant task.\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.7 $\n */\npublic class GrammarDoc extends Task implements DocumenterContext {\n\n\tprivate File srcDir, destDir;\n\n\t/**\n\t * Executes the grammardoc task. This method finds the specified\n\t * srcDir and destDir, then invokes a\n\t * {@link Documenter documenter} with those parameters.\n\t * \n\t * @throws BuildException If a source directory or destination directory is\n\t * not specified, or the source directory does not exist.\n\t */\n\t@Override\n\tpublic void execute() throws BuildException {\n\t\tif(srcDir == null) {\n\t\t\tsrcDir = new File(System.getProperty(\"user.dir\"));\n\t\t}\n\t\t\n\t\tif(!srcDir.exists()) {\n\t\t\tthrow new BuildException(\"Source directory does not exist\");\n\t\t}\n\t\t\n\t\tif(destDir == null) {\n\t\t\tdestDir = new File(srcDir, \"docs\");\n\t\t}\n\n\t\tif(!destDir.exists()) {\n\t\t\tlog(\"Creating directory \" + destDir);\n\t\t\tdestDir.mkdirs();\n\t\t}\n\n\t\ttry {\n\t\t\tDocumenter documenter = DocumenterFactory.newInstance()\n\t\t\t\t\t.newDocumenter();\n\n\t\t\tlog(\"Using \" + documenter.getName());\n\t\t\tlog(\"Documenting \" + srcDir.getAbsolutePath());\n\t\t\tlog(\"Generating documentation to \" + destDir);\n\n\t\t\tdocumenter.setDocumenterContext(this);\n\t\t\tdocumenter.document(loadSourceGrammar());\n\t\t\tlog(\"Done\");\n\t\t}\n\t\tcatch(DocumenterNotFoundException dnfe) {\n\t\t\tthrow new BuildException(\"documenter not found: \"\n\t\t\t\t\t+ dnfe.getMessage(), getLocation());\n\t\t}\n\t\tcatch(DocumenterSourceException dse) {\n\t\t\tthrow new BuildException(\"problem in source file \"\n\t\t\t\t\t+ dse.getSourceGrammarFile() + \": \" + dse.getMessage(),\n\t\t\t\t\tgetLocation());\n\t\t}\n\t\tcatch(DocumenterException de) {\n\t\t\tthrow new BuildException(\"problem documenting: \" + de.getMessage(),\n\t\t\t\t\tde);\n\t\t}\n\t\tcatch(GrammarDocException gde) {\n\t\t\tthrow new BuildException(gde.getMessage(), getLocation());\n\t\t}\n\t}\n\n\tSourceGrammar loadSourceGrammar() throws GrammarDocException {\n\t\tSourceGrammar sourceGrammar = new SourceGrammar(srcDir);\n\n\t\ttry {\n\t\t\tSourceGrammarFile grammar = loadGrammarFile(\n\t\t\t\t\tSourceGrammarFileType.GRAMMAR,\n\t\t\t\t\tnew File(srcDir, SourceGrammarFileType.GRAMMAR.fileName\n\t\t\t\t\t\t\t+ \".xml\"));\n\t\t\tsourceGrammar.addSourceGrammarFile(SourceGrammarFileType.GRAMMAR,\n\t\t\t\t\tgrammar);\n\t\t\t//TODO make these StreamSource instead\n\t\t\tFile gd = grammar.sourceFile;\n\t\t\t\n\t\t\tfor(SourceGrammarFileType fileType\n\t\t\t\t\t: SourceGrammarFileType.values()) {\n\t\t\t\tif(!fileType.equals(SourceGrammarFileType.GRAMMAR)) { // already\n\t\t\t\t\tDocumentBuilder db = DocumentBuilderFactory.newInstance()\n\t\t\t\t\t\t.newDocumentBuilder();\n\t\t\t\t\tDocument doc = db.parse(gd);\n\t\t\t\t\t\n\t\t\t\t\tNodeList fileEls = doc.getElementsByTagName(\n\t\t\t\t\t\t\tfileType.name().toLowerCase());\n\t\t\t\t\tif(fileEls.getLength() == 0) {\n\t\t\t\t\t\tif(fileType.isRequired()) {\n\t\t\t\t\t\t\tthrow new GrammarDocException(\n\t\t\t\t\t\t\t\t\"file type required but missing: \" + fileType);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tsourceGrammar.addSourceGrammarFile(fileType,\n\t\t\t\t\t\t\t\tloadGrammarFile(fileType, \n\t\t\t\t\t\t\t\t\tnew File(srcDir,\n\t\t\t\t\t\t\t\t\t\t\tfileType.fileName + \".xml\")));\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tcatch(ParserConfigurationException pce) {\n\t\t\tthrow new GrammarDocException(\"parser configuration problem: \"\n\t\t\t\t\t+ pce.getMessage(), pce);\n\t\t}\n\t\tcatch(SAXException se) {\n\t\t\tthrow new GrammarDocException(\"problem parsing source files: \"\n\t\t\t\t\t+ se.getMessage(), se);\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tthrow new GrammarDocException(\"io problem with source files: \"\n\t\t\t\t\t+ io.getMessage(), io);\n\t\t}\n\n\t\treturn sourceGrammar;\n\t}\n\t\n\tSourceGrammarFile loadGrammarFile(SourceGrammarFileType fileType, File file) \n\t\t\tthrows GrammarDocException {\n\t\tif(!file.exists()) {\t\t\t\n\t\t\tthrow new GrammarDocException(\"file \" + file.getName()\n\t\t\t\t\t+ \" does not exist in \" + srcDir);\n\t\t}\n\t\telse if(file.isDirectory()) {\n\t\t\tthrow new GrammarDocException(file.getName()\n\t\t\t\t\t+ \" refers to a directory in \" + srcDir);\n\t\t}\n\t\telse {\n\t\t\tlog(\"Loading \" + file.getName());\n\n\t\t\ttry {\n\t\t\t\treturn new SourceGrammarFile(fileType, file);\n\t\t\t}\n\t\t\tcatch(Exception e) {\n\t\t\t\tthrow new GrammarDocException(\"problem parsing \"\n\t\t\t\t\t\t+ file + \": \" + e.getMessage(), e);\n\t\t\t}\n\t\t}\n\t}\n\n\t/**\n\t * For conformance with {@link DocumenterContext}.\n\t */\n\tpublic File getDestinationDirectory() {\n\t\treturn getDestDir();\n\t}\n\n\t/**\n\t * @return Returns the destDir.\n\t */\n\tpublic File getDestDir() {\n\t\treturn destDir;\n\t}\n\n\t/**\n\t * @param destDir The destDir to set.\n\t */\n\tpublic void setDestDir(File destDir) {\n\t\tthis.destDir = destDir.getAbsoluteFile();\n\t}\n\n\t/**\n\t * @return Returns the sourceDirectory.\n\t */\n\tpublic File getSrcDir() {\n\t\treturn srcDir;\n\t}\n\n\t/**\n\t * @param srcDir The sourceDirectory to set.\n\t */\n\tpublic void setSrcDir(File srcDir) {\n\t\tthis.srcDir = srcDir.getAbsoluteFile();\n\t}\n\t\n\tpublic static void main(String[] args) {\n\t\tList arguments = Arrays.asList(args);\n\t\tPrintStream out = System.out;\n\t\tGrammarDoc gd = new CommandGrammarDoc(out);\n\t\t\n\t\tif(arguments.contains(\"-h\") || arguments.contains(\"--help\")) {\n\t\t\tout.println(\"usage: ccg-grammardoc [-s|--source sourceDir] \"\n\t\t\t\t+ \"[-d|--dest destDir]\");\n\t\t}\n\t\telse {\n\t\t\tIterator i = arguments.iterator();\n\t\t\t\n\t\t\ttry {\n\t\t\t\twhile(i.hasNext()) {\n\t\t\t\t\tString s = i.next();\n\t\t\t\t\t\n\t\t\t\t\tif(s.equals(\"-s\") || s.equals(\"--source\")) {\n\t\t\t\t\t\tif(gd.srcDir != null) {\n\t\t\t\t\t\t\tthrow new IllegalArgumentException(\n\t\t\t\t\t\t\t\t\t\"source directory already specified\");\n\t\t\t\t\t\t}\n\t\t\t\t\t\tif(!i.hasNext()) {\n\t\t\t\t\t\t\tthrow new IllegalArgumentException(\n\t\t\t\t\t\t\t\t\t\"encountered flag \" + s \n\t\t\t\t\t\t\t\t\t\t+ \", but no directory specified\");\n\t\t\t\t\t\t}\n\t\t\t\t\t\t\n\t\t\t\t\t\tgd.setSrcDir(new File(i.next()));\n\t\t\t\t\t}\n\t\t\t\t\telse if(s.equals(\"-d\") || s.equals(\"--dest\")) {\n\t\t\t\t\t\tif(gd.destDir != null) {\n\t\t\t\t\t\t\tthrow new IllegalArgumentException(\n\t\t\t\t\t\t\t\t\t\"destination directory already specified\");\n\t\t\t\t\t\t}\n\t\t\t\t\t\tif(!i.hasNext()) {\n\t\t\t\t\t\t\tthrow new IllegalArgumentException(\n\t\t\t\t\t\t\t\t\t\"encountered flag \" + s\n\t\t\t\t\t\t\t\t\t\t+ \", but no directory specified\");\n\t\t\t\t\t\t}\n\t\t\t\t\t\t\n\t\t\t\t\t\tgd.setDestDir(new File(i.next()));\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tgd.execute();\n\t\t\t}\n\t\t\tcatch(Exception e) {\n\t\t\t\tgd.log(\"Error: \" + e.getMessage());\n\t\t\t}\t\t\t\n\t\t}\n\t}\n\t\n\tstatic final class CommandGrammarDoc extends GrammarDoc {\n\t\tPrintStream out;\n\t\tstatic final String logPrefix = \"[grammardoc] \";\n\t\t\n\t\tCommandGrammarDoc(PrintStream out) {\n\t\t\tthis.out = out;\n\t\t\t\n\t\t\tsetProject(new Project());\n\t\t\tsetOwningTarget(new Target());\n\t\t}\n\n\t\t@Override\n\t\tpublic void log(String s) {\n\t\t\tout.print(logPrefix);\t\t\t\n\t\t\tout.println(s);\n\t\t}\n\t}\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/GrammarDocException.java", "content": "/*\n * $Id: GrammarDocException.java,v 1.2 2006/11/01 02:53:20 coffeeblack Exp $\n */\npackage opennlp.ccg.grammardoc;\n\n/**\n * An exception that occurs during the execution of {@link GrammarDoc}.\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.2 $\n */\npublic class GrammarDocException extends Exception {\n\n\tprivate static final long serialVersionUID = 1L;\n\n\t/**\n\t * Creates a new exception.\n\t */\n\tpublic GrammarDocException() {\n\t\tsuper();\n\t}\n\n\t/**\n\t * Creates a new exception with the specified message.\n\t */\n\tpublic GrammarDocException(String message) {\n\t\tsuper(message);\n\t}\n\n\t/**\n\t * Creates a new exception with the specified message and underlying cause.\n\t */\n\tpublic GrammarDocException(String message, Throwable cause) {\n\t\tsuper(message, cause);\n\t}\n\n\t/**\n\t * Creates a new exception with the specified underlying cause.\n\t */\n\tpublic GrammarDocException(Throwable cause) {\n\t\tsuper(cause);\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/SourceGrammar.java", "content": "/*\n * $Id: SourceGrammar.java,v 1.4 2007/03/19 17:45:35 coffeeblack Exp $\n */\npackage opennlp.ccg.grammardoc;\n\nimport java.io.File;\nimport java.util.EnumMap;\nimport java.util.Map;\nimport java.util.Set;\n\n/**\n * Represents an OpenCCG grammar that is specified in a series of XML files.\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.4 $\n */\npublic class SourceGrammar {\n\n\tFile sourceDirectory;\n\tMap sourceFiles;\n\n\tSourceGrammar(File sourceDirectory) {\n\t\tthis.sourceDirectory = sourceDirectory;\n\t\tsourceFiles = new EnumMap\n\t\t\t(SourceGrammarFileType.class);\n\t}\n\n\t/**\n\t * @return Returns the sourceDirectory.\n\t */\n\tpublic File getSourceDirectory() {\n\t\treturn sourceDirectory;\n\t}\n\n\t/**\n\t * Gets the {@link SourceGrammarFileType file names} found in this source\n\t * grammar.\n\t */\n\tpublic Set getSourceGrammarFileTypes() {\n\t\treturn sourceFiles.keySet();\n\t}\n\n\t/**\n\t * Gets a source grammar file based on a specified\n\t * {@link SourceGrammarFileType file name}.\n\t * \n\t * @param fileType The file name identifying the desired\n\t * {@link SourceGrammarFile source grammar file}.\n\t */\n\tpublic SourceGrammarFile getSourceGrammarFile(\n\t\t\tSourceGrammarFileType fileType) {\n\t\treturn sourceFiles.get(fileType);\n\t}\n\n\tvoid addSourceGrammarFile(SourceGrammarFileType fileType,\n\t\t\tSourceGrammarFile sourceGrammarFile) {\n\t\tsourceFiles.put(fileType, sourceGrammarFile);\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/SourceGrammarFile.java", "content": "/*\n * $Id: SourceGrammarFile.java,v 1.4 2007/05/30 22:53:17 coffeeblack Exp $\n */\npackage opennlp.ccg.grammardoc;\n\nimport java.io.File;\n\nimport org.w3c.dom.Document;\n\n/**\n * A file found in a source grammar. This class encapsulates the predefined\n * {@link SourceGrammarFileType file name} and {@link Document DOM document}.\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.4 $\n */\npublic class SourceGrammarFile {\n\n\tSourceGrammarFileType fileType;\n\tFile sourceFile;\n\n\t/**\n\t * Creates a new source grammar file.\n\t */\n\tSourceGrammarFile(SourceGrammarFileType fileName, File sourceFile) {\n\t\tthis.fileType = fileName;\n\t\tthis.sourceFile = sourceFile;\n\t}\n\n\t/**\n\t * @return Returns the fileType.\n\t */\n\tpublic SourceGrammarFileType getFileType() {\n\t\treturn fileType;\n\t}\n\n\t/**\n\t * @return Returns the source file.\n\t */\n\tpublic File getSourceFile() {\n\t\treturn sourceFile;\n\t}\n\n\t/**\n\t * Gets a string version of this source grammar file.\n\t * \n\t * @return The value of this grammar file's\n\t * {@link SourceGrammarFileType#toString() file type}.\n\t */\n\t@Override\n\tpublic String toString() {\n\t\treturn fileType.toString();\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/SourceGrammarFileType.java", "content": "/*\n * $Id: SourceGrammarFileType.java,v 1.3 2007/05/02 21:51:35 coffeeblack Exp $\n */\npackage opennlp.ccg.grammardoc;\n\n/**\n * File types for source grammars.\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.3 $\n */\npublic enum SourceGrammarFileType {\n\tGRAMMAR(\"grammar\"),\n\tLEXICON(\"lexicon\"),\n\tMORPHOLOGY(\"morph\"),\n\tRULES(\"rules\"),\n\tTYPES(\"types\", false),\n\tDOCUMENTATION(\"documentation\", false);\n\t\n\tfinal String fileName;\n\tfinal boolean required;\n\t\n\tprivate SourceGrammarFileType(String fileName) {\n\t\tthis(fileName, true);\n\t}\n\t\n\tprivate SourceGrammarFileType(String fileName, boolean required) {\n\t\tthis.fileName = fileName;\n\t\tthis.required = required;\n\t}\n\t\n\t/**\n\t * Gets the file name associated with this file type.\n\t */\n\tpublic String getFileName() {\n\t\treturn fileName;\n\t}\n\t\n\t/**\n\t * Tests whether or not this grammar file name is required.\n\t * @return true iff this grammar file type is required to be present in a\n\t * grammar.\n\t */\n\tpublic boolean isRequired() {\n\t\treturn required;\n\t}\n}" }, { "path": "src/opennlp/ccg/grammardoc/html/HTMLDocumenter.java", "content": "/*\n * $Id: HTMLDocumenter.java,v 1.9 2009/12/21 04:18:31 mwhite14850 Exp $\n */\npackage opennlp.ccg.grammardoc.html;\n\nimport java.io.BufferedOutputStream;\nimport java.io.File;\nimport java.io.FileOutputStream;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.util.HashMap;\nimport java.util.Map;\n\nimport javax.xml.transform.Source;\nimport javax.xml.transform.Templates;\nimport javax.xml.transform.Transformer;\nimport javax.xml.transform.TransformerConfigurationException;\nimport javax.xml.transform.TransformerException;\nimport javax.xml.transform.TransformerFactory;\nimport javax.xml.transform.URIResolver;\nimport javax.xml.transform.stream.StreamResult;\nimport javax.xml.transform.stream.StreamSource;\n\nimport opennlp.ccg.grammardoc.AbstractDocumenter;\nimport opennlp.ccg.grammardoc.DocumenterException;\nimport opennlp.ccg.grammardoc.DocumenterSourceException;\nimport opennlp.ccg.grammardoc.SourceGrammar;\nimport opennlp.ccg.grammardoc.SourceGrammarFile;\nimport opennlp.ccg.grammardoc.SourceGrammarFileType;\n\n/**\n * A grammardoc documenter that produces HTML documentation.\n * \n * @author Scott Martin (http://www.ling.osu.edu/~scott/)\n * @version $Revision: 1.9 $\n */\npublic class HTMLDocumenter extends AbstractDocumenter implements URIResolver {\n\n\tprivate static final int FILE_BUFFER_SIZE = 256;\n\tprivate SourceGrammar sourceGrammar;\n\tprivate Map templateCache;\n\t\n\tfinal TransformerFactory factory = TransformerFactory.newInstance();\n\t\n\tstatic enum FileName {\n\t\tSTYLESHEET(\"grammardoc.css\"), LEXICON_SCRIPT(\"lexicon.js\");\n\n\t\tfinal String name;\n\n\t\tprivate FileName(String name) {\n\t\t\tthis.name = name;\n\t\t}\n\t}\n\n\t/**\n\t * Creates a new HTML Documenter.\n\t */\n\tpublic HTMLDocumenter() {\n\t\tsuper(\"GrammarDoc HTML Documenter\");\n\t\ttemplateCache = new HashMap();\n\t\tfactory.setURIResolver(this);\n\t}\n\n\t/**\n\t * Documents a source grammar, producing linked HTML files for its source\n\t * files.\n\t */\n\tpublic void document(SourceGrammar grammar) throws DocumenterException {\n\t\tthis.sourceGrammar = grammar;\n\t\tFile destDir = documenterContext.getDestinationDirectory();\n\t\tcopyFiles(destDir);\n\t\t\n\t\tString sections;\n\t\tStringBuilder sb = new StringBuilder();\n\t\tfor(SourceGrammarFileType fileType\n\t\t\t\t: grammar.getSourceGrammarFileTypes()) {\n\t\t\tif(sb.length() > 0) {\n\t\t\t\tsb.append('|');\n\t\t\t}\n\t\t\tsb.append(fileType.getFileName());\n\t\t}\n\t\tsections = sb.toString();\n\n\t\tfor(SourceGrammarFileType fileType : grammar\n\t\t\t\t.getSourceGrammarFileTypes()) {\n\t\t\tString baseName = fileType.getFileName();\n\n\t\t\tStringBuilder fb = new StringBuilder();\n\t\t\tfb.append(baseName.equals(\n\t\t\t\t\tSourceGrammarFileType.GRAMMAR.getFileName())\n\t\t\t\t\t? \"index\" : baseName);\n\t\t\tfb.append(\".html\");\n\t\t\tString targetName = fb.toString();\n\n\t\t\tSourceGrammarFile sourceFile\n\t\t\t\t= grammar.getSourceGrammarFile(fileType);\n\t\t\tTemplates templates = loadTemplates(baseName);\n\t\t\t\n\t\t\tif(templates != null) {\n\t\t\t\tdocumenterContext.log(\"Generating \" + targetName);\n\n\t\t\t\ttry {\n\t\t\t\t\tFile f = new File(destDir, targetName);\n\t\t\t\t\tStreamResult res = new StreamResult(\n\t\t\t\t\t\t\tnew BufferedOutputStream(new FileOutputStream(f)));\n\t\t\t\t\tres.setSystemId(f);\n\t\t\t\t\t\n\t\t\t\t\tTransformer transformer = templates.newTransformer();\n\t\t\t\t\ttransformer.setURIResolver(this);\n\t\t\t\t\ttransformer.setParameter(\"sections\", sections);\t\t\n\t\t\t\t\ttransformer.transform(\n\t\t\t\t\t\t\tnew StreamSource(sourceFile.getSourceFile()), res);\n\t\t\t\t}\n\t\t\t\tcatch(TransformerException te) {\n\t\t\t\t\tthrow new DocumenterSourceException(\n\t\t\t\t\t\t\t\"problem transforming output: \"\n\t\t\t\t\t\t\t\t\t+ te.getMessageAndLocation(), te,\n\t\t\t\t\t\t\t\t\tsourceFile);\n\t\t\t\t}\n\t\t\t\tcatch(IOException ioe) {\n\t\t\t\t\tthrow new DocumenterException(ioe);\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\t/**\n\t * Resolves URIs to sources. Used by the XSLT files in this package to\n\t * resolve xsl:import and document() URIs.\n\t */\n\tpublic Source resolve(String href, String base) throws TransformerException {\n\n\t\tStreamSource ss = null;\n\t\t\n\t\tif(href != null && href.length() > 0) {\n\t\t\tif(href.endsWith(\".xsl\")) {\n\t\t\t\tss = new StreamSource(getResource(href));\n\t\t\t}\n\t\t\telse {\n\t\t\t\tFile f = new File(sourceGrammar.getSourceDirectory(), href);\n\t\t\t\tif(!f.exists()) {\n\t\t\t\t\tthrow new TransformerException(\"file does not exist: \" + f);\n\t\t\t\t}\n\t\t\t\tif(f.isDirectory()) {\n\t\t\t\t\tthrow new TransformerException(\"file is a directory: \" + f);\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tss = new StreamSource(f);\n\t\t\t\tss.setSystemId(f);\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn ss;\n\t}\n\n\tprivate Templates loadTemplates(String baseName)\n\t\t\tthrows DocumenterException {\n\t\tStringBuilder tb = new StringBuilder(baseName);\n\t\ttb.append(\".xsl\");\n\t\tString templateName = tb.toString();\n\n\t\tif(!templateCache.containsKey(templateName)) {\n\t\t\tInputStream is = getResource(templateName.toString());\n\t\t\tif(is == null) {\n\t\t\t\treturn null;\n\t\t\t}\n\n\t\t\ttry { // cache for later\n\t\t\t\ttemplateCache.put(templateName, \n\t\t\t\t\t\tfactory.newTemplates(new StreamSource(is)));\n\t\t\t}\n\t\t\tcatch(TransformerConfigurationException tce) {\n\t\t\t\tthrow new DocumenterException(\"problem loading template \"\n\t\t\t\t\t\t+ templateName.toString() + \": \"\n\t\t\t\t\t\t\t+ tce.getMessageAndLocation(), tce);\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn templateCache.get(templateName);\n\t}\n\n\tprivate void copyFiles(File destDir) throws DocumenterException {\n\t\tfor(FileName fileName : FileName.values()) {\n\t\t\tdoCopyFile(fileName, destDir);\n\t\t}\n\t}\n\n\tprivate void doCopyFile(FileName fileName, File destDir)\n\t\t\tthrows DocumenterException {\n\t\tInputStream in = getResource(fileName.name);\n\n\t\tif(in == null) {\n\t\t\tthrow new DocumenterException(\"Could not find \" + fileName.name);\n\t\t}\n\n\t\tFile f = new File(destDir, fileName.name);\n\t\tdocumenterContext.log(\"Writing \" + f.getAbsolutePath());\n\n\t\ttry {\n\t\t\tFileOutputStream fileOut = new FileOutputStream(f);\n\t\t\tbyte[] buffer = new byte[HTMLDocumenter.FILE_BUFFER_SIZE];\n\n\t\t\tint i;\n\t\t\twhile((i = in.read(buffer)) != -1) {\n\t\t\t\tfileOut.write(buffer, 0, i);\n\t\t\t}\n\n\t\t\tin.close();\n\t\t\tfileOut.close();\n\t\t}\n\t\tcatch(IOException ioe) {\n\t\t\tthrow new DocumenterException(\"problem copying file: \"\n\t\t\t\t\t+ ioe.getMessage(), ioe);\n\t\t}\n\t}\n\n\tprivate InputStream getResource(String resourceName) {\n\t\tClass cl = getClass();\n\t\tString cn = cl.getName();\n\t\tString pkg = cn.substring(0, cn.lastIndexOf('.'));\n\n\t\tStringBuilder sb = new StringBuilder();\n\t\tsb.append(pkg.replace('.', '/'));\n\t\tsb.append('/');\n\t\tsb.append(resourceName);\n\n\t\treturn cl.getClassLoader().getResourceAsStream(sb.toString());\n\t}\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/html/base.xsl", "content": "\n\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\tgrammar\n\t\t\t\tlexicon\n\t\t\t\tmorph\n\t\t\t\ttypes\n\t\t\t\trules\n\t\t\t\n\t\t\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t<xsl:text>Documentation for CCG grammar </xsl:text>\n\t\t\t\t\t<xsl:value-of\n\t\t\t\t\t\tselect=\"$grammar-name\"/>\n\t\t\t\t\t<xsl:if test=\"not($page-name='grammar')\">\n\t\t\t\t\t\t<xsl:text> : </xsl:text>\n\t\t\t\t\t\t<xsl:call-template name=\"capitalize\">\n\t\t\t\t\t\t\t<xsl:with-param name=\"word\" select=\"$page-name\"/>\n\t\t\t\t\t\t</xsl:call-template>\n\t\t\t\t\t</xsl:if>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\tGenerated documentation for \n\t\t\t\t\t\tthe CCG grammar \n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t

\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\n\t\t\t\t

\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t
\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t
\n\t\t\t\n\t\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n\n" }, { "path": "src/opennlp/ccg/grammardoc/html/categories.xsl", "content": "\n\n\n\t\n\t\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\t
\n\t\t\t\t\n\t\t\t
\n\t\t\n\t\t\n\t\n\t\n\t\n\t\t(\n\t\t
\n\t\t\t
\n\t\t\t\t\n\t\t\t
\n\t\t
\n\t\t)\n\t\n\t\n\t\n\t\t
\n\t\t\t
\n\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\t\t<\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t~\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t>\n\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t
\n\t\t
\n\t\n\t\n\t\n\t\t
\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t,\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t[\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\n\t\t\t\n\t\t\t\t]\n\t\t\t\n\t\t
\n\t\n\t\n\t\n\t\t\n\t\n\t\n\t\n\t\t\n\t\n\t\n\t\n\t\t\n\t\n\t\n\t\n\t\t{\n\t\t\n\t\t}\n\t\n\t\n\t\n\t\t
\n\t\t\t\n\t\t\t\tslash\n\t\t\t\t\n\t\t\t\t\t redundant\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t.\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t|\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\tapplication only\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\tassociative\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\tpermutative\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\tpermutative right\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\tpermutative left\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\tassociative permutative right\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\tassociative permutative left\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\tall rules\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t
\n\t\n\t\n\t\n\t\t
\n\t\t\t
\n\t\t\t\t : \n\t\t\t\t\n\t\t\t
\n\t\t
\n\t\n\t\n\t\n\t\t\n\t\t\t ^ \n\t\t\n\t\t
@
\n\t\t
\n\t\t\t\n\t\t
\n\t\t
\n\t\t\t(\n\t\t\t\n\t\t\t)\n\t\t
\n\t\n\t\n\t\n\t\t\n\t\t\t\n\t\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\n\t\t\n\t\t\t ^ \n\t\t\t\n\t\t\t\t(\n\t\t\t\n\t\t\n\t\t\n\t\t<\n\t\t\n\t\t>\n\t\t\n\t\t\n\t\t\t(\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t)\n\t\t\n\t\t\n\t\t\n\t\t\t)\n\t\t\n\t\n\t\n\t\n\t\t\n\t\t\t ^ \n\t\t\n\t\n\t\t\n\t\t\t\n\t\t\n\t\n\n\t\n\t\t
$
\n\t\t
\n\t\t\t\n\t\t
\n\t\n\n" }, { "path": "src/opennlp/ccg/grammardoc/html/comments.xsl", "content": "\n\n\n\t\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\t
\n\t\t\t\t\n\t\t\t
\n\t\t\n\t\n\t\n" }, { "path": "src/opennlp/ccg/grammardoc/html/grammar.xsl", "content": "\n\n\n\t\n\t\n\t\n\t\n\t\t\n\t\n\t\n\t\n" }, { "path": "src/opennlp/ccg/grammardoc/html/grammardoc.css", "content": "/*\n * $Id: grammardoc.css,v 1.8 2007/04/18 22:54:13 coffeeblack Exp $\n * Author: Scott Martin (http://www.ling.osu.edu/~scott/)\n */\nbody\n{\n\tmargin: 0 0 2em 0;\n\tpadding: 0;\n\tbackground-color: rgb(250,250,250);\n\tcolor: rgb(25,25,25);\n\tfont-family: Verdana, Arial, Helvetica, sans-serif;\n\tfont-size: small;\n}\n\na:hover\n{\n\ttext-decoration: none;\n}\n\nacronym\n{\n\tborder-bottom: 1px dotted;\n\tcursor: help;\t\n}\n\ndiv.back\n{\n\ttext-align: right;\n}\n\ndiv#container {\n\tmargin: 0 20px;\n\tclear: left;\n}\n\nh1, h2, h3, h4, h5\n{\n\tfont-family: Trebuchet MS, Arial, Helvetica, sans-serif;\n}\n\nh1\n{\n\tmargin: 0 0 1em 0;\n\tborder-bottom: 1px dotted rgb(150,150,150);\n\tpadding: 2em 0 .5em 20px;\n\tbackground-color: rgb(153,0,0);\n\tcolor: rgb(250,250,250);\n\tfont-size: 1.8em;\n}\n\nh2\n{\n\tmargin: 2em 0 1em 0;\n\tborder-bottom: 1px dotted rgb(160,160,160);\n\tpadding-bottom: .25em;\n\tcolor: rgb(153,0,0);\n\tfont-size: 1.6em;\n}\n\nh3\n{\n\tcolor: rgb(80,0,0);\n\tfont-size: 1.4em;\n}\n\nh4\n{\n\tfont-size: 1.2em;\n}\n\nh5\n{\n\tfont-size: 1.1em;\n}\n\nul\n{\n\tpadding-left: 40px;\n}\n\ndl.box\n{\n\tborder: 1px solid rgb(150,150,150);\n\tbackground-color: rgb(235,235,235);\n\tpadding: 10px;\n}\n\ndl.box dt\n{\n\tcolor: rgb(153,51,0);\n\tfont-family: Trebuchet MS, Arial, Helvetica, sans-serif;\n\tfont-size: 1.1em;\n\tfont-weight: bold;\n}\n\ndl.box dd, dl.attributes dd\n{\n\tmargin-left: 0;\n}\n\ndl.box dd li, ul.members li\n{\n\tcolor: rgb(153,51,0);\n}\n\nol.categories li\n{\n\t/*font-size: 1.2em;*/\n}\n\ndl.category dt {\n\tfont-size: 1.1em;\n}\n\ndl.category, dl.category dt, dl.category dd, dl.category dd dl\n{\n\tdisplay: inline;\n}\n\ndl.attributes dt\n{\n\tfloat: left;\n\tmargin-right: .25em;\n\tcolor: rgb(153,51,0);\n\tfont-weight: bold;\n}\n\ndl.attributes dd\n{\n\tcolor: rgb(153,85,50);\n}\n\ndl.category dt\n{\n\tcolor: rgb(80,0,0);\t\n}\n\ndl.feature-structure {\n\tfont-size: .85em;\n}\n\ndl.category dd, dl.feature-structure dt, span.ruleLabel\n{ \n\tcolor: rgb(153,85,50);\n\tfont-family: Georgia, \"Times New Roman\", serif;\n}\n\ndl.feature-structure dt\n{\n\tfont-family: Verdana, Arial, Helvetica, sans-serif;\n\tfont-weight: normal;\n}\n\ndl.feature-structure dt, dl.feature-structure dd\n{\n\tdisplay: none;\n}\n\nspan.enclosure, span.divider {\n\tfont-size: 1.1em;\n\tfont-weight: bold;\n}\n\ndl.feature-structure dd span.enclosure\n{\n\tfont-style: normal;\n}\n\nol.categories li a\n{\n\tmargin-right: 10px;\n\t/*font-size: .85em;*/\n\ttext-decoration: none;\n}\n\n.expanded dl.feature-structure dt, .expanded dl.feature-structure dd\n{\n\tdisplay: inline;\n}\n\ndl.category dd, span.ruleLabel {\n\tvertical-align: sub;\n}\n\ndl.category dd.category-container\n{\n\tvertical-align: baseline;\t\n}\n\ndd.category-container dl.category\n{\n\t/*font-size: 1.1em;*/\n}\n\ndl.category dd.feat\n{\n\tfont-style: italic;\n}\n\ndt.slash span.mode\n{\n\t/*font-size: .7em;*/\n}\n\ndl.category dt.redundant {\n\tdisplay: none;\n}\n\ndl.lf {\n\tdisplay: inline;\n}\n\ndl.lf dd.nomvar, dl.lf span.prop {\n\tfont-weight: normal;\n}\n\ndl.lf dd.nomvar {\n\tfont-style: italic;\n}\n\ndl.category span.prop {\n\tfont-weight: bold;\n}\n\ndl.lf span.diamond, dl.lf span.prop {\n\tcolor: rgb(153,85,50);\n}\n\ndl.entries dl dt {\n\tmargin-top: 1em;\n}\n\ndl.entries dl dd {\n\tmargin-left: 20px;\t\n}\n\nul.rules {\n\tpadding-left: 20px;\n\tlist-style-type: none;\n}\n\nul.rules li {\n\tpadding-top: .5em;\t\n}\n\nul#navigation {\n\tmargin-left: 20px;\n\tpadding-bottom: 2em;\n\tpadding-left: 0;\n\tlist-style-type: none;\n}\n\nul#navigation li a {\n\tfloat: left;\n\tmargin-right: 10px;\n\tborder: 1px solid rgb(150,150,150);\n\tbackground-color: rgb(235,235,235);\n\tcolor: rgb(153,0,0);\n\tpadding: 5px;\n\ttext-decoration: none;\n\tfont-weight: bold;\n}\n\nul#navigation li a.current, ul#navigation li a:hover,\n\tul#navigation li a:active {\n\tbackground-color: rgb(153,0,0);\n\tborder: 1px dotted rgb(150,150,150);\n\tcolor: rgb(255,255,255);\n}\n\na.unaryRuleExpander {\n\tfloat: left;\n\tmargin-right: 5px;\n\ttext-decoration: none;\n}\n\ndd.unaryResult {\n\tdisplay: inline;\n}\n\ndl.unaryRule {\n\t/*font-size: 1.1em !important;*/\n\tcolor: rgb(153,51,0);\n}\n\ndl.unaryRule span.arrow {\n\tcolor: rgb(153,85,50);\n\tfont-weight: bold;\n}\n\nul.short {\n\tlist-style-type: none;\n}\n\nul.short li {\n\tdisplay: inline;\n}\n\ndl.shortcuts ul.short li {\n\tcolor: rgb(25,25,25);\n}\n\nul.short li + li:before {\n\tcontent: \",\";\n}\n" }, { "path": "src/opennlp/ccg/grammardoc/html/lexicon.js", "content": "/*\n * $Id: lexicon.js,v 1.2 2006/12/13 19:25:22 coffeeblack Exp $\n * Author: Scott Martin (http://www.ling.osu.edu/~scott/)\n */\nfunction toggleFeatures(elem) {\n\telem.className = (elem.className == \"expanded\")\n\t\t? \"\" : \"expanded\";\n\t\n\tvar anchors = elem.getElementsByTagName(\"a\");\n\tanchors[0].innerHTML = (elem.className == \"expanded\")\n\t\t? \"[-]\" : \"[+]\";\n\t\n\tanchors[0].setAttribute(\"title\", \n\t\t((elem.className == \"expanded\")\n\t\t\t? \"collapse\" : \"expand\") + \" feature structures\");\n}" }, { "path": "src/opennlp/ccg/grammardoc/html/lexicon.xsl", "content": "\n\n\n\t\n\t\n\t\n\n\t\n\t\t
\n\t\t\t
Lexical Families
\n\t\t\t
\n\t\t\t\t\n\t\t\t
\n\t\t
\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\n\n\t\n\t\t
\n\t\t\t\n\t\t\t\t

\n\t\t\t\t\t\n\t\t\t\t

\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t
\n\t\t\t\t
Closed:
\n\t\t\t\t
\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\tfalse\n\t\t\t\t\t\n\t\t\t\t
\n\t\t\t\t
Part of Speech:
\n\t\t\t\t
\n\t\t\t\t\t\n\t\t\t\t
\n\t\t\t
\n\t\t\t\n\t\t\t\n\t\t\t\tLexical Items\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t

Members

\n\t\t\t\t
    \n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t
\n\t\t\t\n\t\t\t\n\t\t\t
\n\t\t\t\tBack to top\n\t\t\t
\n\t\t
\n\t\n\t\n\t\n\t\t\n\t\t\t\n\t\t\t-\n\t\t\t\n\t\t\n\t\t
  • \n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t
    • \n\t    \n\t\n\t\n\t\t

    \n\t\t\t\n\t\t

    \n\t\t\n\t\t
    \n\t\t\t
    Categories
    \n\t\t\t
    \n\t\t\t\t
      \n\t\t\t\t\t\n\t\t\t\t\t\t
    1. \n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t[+]\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t
      \n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t
      \n\t\t\t\t\t\t\t\t      \n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t
      \n\t\t\t\t\t\t
      2. \n\t\t\t\t\t      \n\t\t\t\t
    \t\n\t\t\t
    \n\t\t
    \n\t    \n\n    \n" }, { "path": "src/opennlp/ccg/grammardoc/html/morph.xsl", "content": "\n\n\n\n\t\n\t\n\t\n\t\n\t\t
    \n\t\t\t
    Entries
    \n\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t
    \n\t\t\t\t\t    \n\t\t\t\t
    \n\t\t\t
    \n\t\t
    \n\t\t\t\t\t\n\t\t
    \n\t\t\t
    Macros
    \n\t\t\t
    \n\t\t\t\t\n\t\t\t
    \n\t\t
    \n\t\t\n\t\t

    Entries

    \n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t

    \n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t

    \n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t    \n\t\t\n\t\t

    Macros

    \n\t\t\n\t\t\t\n\t\t\n\t    \n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t

    \n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t

    \n\t\t\t\n\t\t\t\n\t\t\t\tMember of: \n\t\t\t\t\n\t\t\t\n\t\n\t\t\t
    \n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t
    \n\t\t\t\n\t\t\t
    \n\t\t\t\tBack to top\n\t\t\t
    \n\t\t    \n\t    \n\t\n\t\n\t\t
    \n\t\t\t\n\t\t
    \n\t\t
    \n\t\t\t\n\t\t
    \n\t    \n\t\n\t\n\t\t\n\t\t\t
      \n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t
    \n\t\t    \n\t    \n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t
  • \n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t
    • \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t    \n\t    \n\t\n\t\n\t\t

    \n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t

    \n\t\t\n\t\t\t
    \n\t\t\t\t
    Feature Structures
    \n\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t\t\t\t\n\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t
    \n\t\t    \n\t\t\n\t\t\t
    \n\t\t\t\t
    Logical Form
    \n\t\t\t\t\n\t\t\t
    \n\t\t    \n\t\t\n\t\t
    \n\t\t\tBack to top\n\t\t
    \n\t    \n\t\n\t\n\t\t
    \n\t\t\t\n\t\t
    \n\t\t
    \n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t=\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t
    \n\t    \n\t\n\t\n\t\t, \n\t\t\n\t\t\n\t\t\t=\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\n\t\n\t\n\t\t
    \n\t\t\t
    \n\t\t\t\t\n\t\t\t
    \n\t\t
    \n\t    \t\n\t\n    \n" }, { "path": "src/opennlp/ccg/grammardoc/html/navigation.xsl", "content": "\n\n\n\n\t\n\tgrammar|lexicon|morph|rules|types\n\t\n\n\t\n\t\t\n\t\n\t\t
      \n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t
    \n\t    \n\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t
  • \n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\tcurrent\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\tindex\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t.html\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t
    • \n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t    \n\n    " }, { "path": "src/opennlp/ccg/grammardoc/html/rules.xsl", "content": "\n\n\n\n\t\n\t\n\t\n\t\n\t\t
    \n\t\t\t
    List of Rules
    \n\t\t\t
    \n\t\t\t\t
      \n\t\t\t\t\t\n\t\t\t\t\t\t
    • \n\t\t\t\t\t\t\tApplication\n\t\t\t\t\t\t
      • \n\t\t\t\t\t      \n\t\t\t\t\t\n\t\t\t\t\t\t
    • \n\t\t\t\t\t\t\tComposition\n\t\t\t\t\t\t
      • \n\t\t\t\t\t      \n\t\t\t\t\t\n\t\t\t\t\t\t
    • \n\t\t\t\t\t\t\tSubstitution\n\t\t\t\t\t\t
      • \n\t\t\t\t\t      \n\t\t\t\t\t\n\t\t\t\t\t\t
    • \n\t\t\t\t\t\t\tTyperaising\n\t\t\t\t\t\t
      • \n\t\t\t\t\t      \n\t\t\t\t\t\n\t\t\t\t\t\t
    • \n\t\t\t\t\t\t\tTypechanging\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t
      • \n\t\t\t\t\t      \n\t\t\t\t
    \n\t\t\t
    \n\t\t
    \n\n\t\t\n\t\t\t\n\t\t\t\t

    Application

    \n\t\t\t    \n\n\t\t\t
    \n\t\t\t\t
    Rules
    \n\t\t\t\t
    \n\t\t\t\t\t
      \n\t\t\t\t\t\t\n\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t
    \n\t\t\t\n\t\t\t
    \n\t\t\t\tBack to top\n\t\t\t
    \n\t\t    \n\t\t\n\t\t\t\n\t\t\t\t

    Composition

    \n\t\t\t    \n\n\t\t\t
    \n\t\t\t\t
    Rules
    \n\t\t\t\t
    \n\t\t\t\t\t
      \n\t\t\t\t\t\t\n\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t
    \n\t\t\t\n\t\t\t
    \n\t\t\t\tBack to top\n\t\t\t
    \n\t\t    \n\t\t\n\t\t\t\n\t\t\t\t

    Substitution

    \n\t\t\t    \n\n\t\t\t
    \n\t\t\t\t
    Rules
    \n\t\t\t\t
    \n\t\t\t\t\t
      \n\t\t\t\t\t\t\n\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t
    \n\t\t\t\n\t\t\t
    \n\t\t\t\tBack to top\n\t\t\t
    \n\t\t    \n\t\t\n\t\t\t\n\t\t\t\t

    Typeraising

    \n\t\t\t    \n\n\t\t\t
    \n\t\t\t\t
    Rules
    \n\t\t\t\t
    \n\t\t\t\t\t
      \n\t\t\t\t\t\t\n\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t
    \n\t\t\t\n\t\t\t
    \n\t\t\t\tBack to top\n\t\t\t
    \n\t\t    \n\t\t\n\t\t\t\n\t\t\t\t

    Typechanging

    \n\t\t\t    \n\n\t\t\t\n\t\t    \n\t    \n\n\t\n\t\t\n\t\n\t\t\n\t\t\t

    \n\t\t\t\t\n\t\t\t

    \n\t\t    \n\t\t\n\t\t
    \n\t\t\t
    \n\t\t\t\t[+]\n\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t\t\t\t\n\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t
    \n\t\t
    \n\t\t\n\t\t
    \n\t\t\tBack to top\n\t\t
    \n\t    \n\t\n\t\n\t\t\n\t\t\t
    \n\t\t\t\t\n\t\t\t\t\t
    \n\t\t\t\t\t\t\n\t\t\t\t\t
    \n\t\t\t\t    \n\t\n\t\t\t\t\n\t\t\t
    \n\t\t    \n\t    \n\t\n\t\n\t\t
  • \n\t\t\t\n\t\t\t\n\t\t
    • \n\t    \n\t\n\t\n\t\t
  • \n\t\t\t\t\n\t\t\tB\n\t\t\t\n\t\t
    • \n\t    \n\t\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\tnp\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\ts\n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\\\n\t\t\t\t/\n\t\t\t\n\t\t\n\t\t\n\t\t
  • \n\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t
    \n\t\t\t\t\n\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\n\t\t\t\t\t\t
    \n\t\t\t\t\t\t\n\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\n\t\t\t\t\t\t
    \n\t\t\t\t\t\t(\n\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\\\n\t\t\t\t\t\t\t\t\t\t/\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t\t)\n\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t
    \n\t\t
    • \n\t    \n\t\n\t\n\t\t\n\t\t\t
    \n\t\t\t\t$\n\t\t\t\t1\n\t\t\t
    \n\t\t    \n\t    \n\t\n\t\n\t\t
  • \n\t\t\t\t\n\t\t\tS\n\t\t\t\n\t\t
    • \n\t    \n\t\n\t\n\t\t\n\t\t\t\n\t\t\t\t>\n\t\t\t\n\t\t\t\n\t\t\t\t<\n\t\t\t\n\t\t\n\t\n\n\t\n\t\t\n\t\t\tx\n\t\t\n\t\n\t\n\t\n\t\t\n\t\t\t&nbsp;&mdash;>&nbsp;\n\t\t\n\t\n\n    \n" }, { "path": "src/opennlp/ccg/grammardoc/html/types.xsl", "content": "\n\n\n\n\t\n\t\n\t\n\t\t
    \n\t\t\t
    Type Hierarchy
    \n\t\t\t
    \n\t\t\t\t
      \n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t
    \n\t\t\t
    \n\t\t
    \n\t    \n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t
  • \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t
      \n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t
    \n\t\t\t    \n\t\t
    • \n\t    \n\t\n    " }, { "path": "src/opennlp/ccg/hylo/Alt.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-6 Michael White (University of Edinburgh, The Ohio State University)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport java.io.Serializable;\nimport java.util.BitSet;\n\n/**\n * Class for representing alts.\n * LF alts are used during realization to represent \n * exclusive disjunctions in the input.\n * The alts are represented as pairs of ints, \n * one for the alt set and one for the alt within the set.\n * The alts are numbered starting with 0.\n * An alt has a bitset for the elementary predications \n * within the alt.\n *\n * @author Michael White\n * @version $Revision: 1.6 $, $Date: 2009/07/17 04:23:30 $\n */\npublic final class Alt implements Comparable, Serializable {\n \n\tprivate static final long serialVersionUID = 7241395629445814238L;\n\t\n\t/** The alt set number. */\n public final int altSet;\n /** The alt within the set. */\n public final int numInSet;\n /** The bitset. */\n public final BitSet bitset = new BitSet();\n \n /** Constructor. */\n public Alt(int altSet, int numInSet) { \n this.altSet = altSet; this.numInSet = numInSet;\n }\n \n /** Equals. */\n public boolean equals(Object o) {\n if (!(o instanceof Alt)) return false;\n Alt a = (Alt) o;\n return altSet == a.altSet && numInSet == a.numInSet;\n }\n /** Comparison. */\n public int compareTo(Alt a) {\n if (altSet < a.altSet) return -1;\n if (altSet == a.altSet && numInSet < a.numInSet) return -1;\n if (altSet == a.altSet && numInSet == a.numInSet) return 0;\n return 1;\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/Box.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\nimport org.jdom.*;\n\n\n/**\n * A modal box operator, such as [F]q.\n *\n * @author Jason Baldridge\n * @version $Revision: 1.5 $, $Date: 2009/07/17 04:23:30 $\n **/\npublic final class Box extends ModalOp {\n\n\tprivate static final long serialVersionUID = 1575311851235814524L;\n\n\tpublic Box(Element e) {\n super(e);\n }\n\n private Box(Mode mode, LF arg) {\n super(mode, arg);\n }\n\n public LF copy() {\n return new Box ((Mode)_mode.copy(), _arg.copy());\n }\n \n public boolean equals(Object o) {\n if (o instanceof Box) {\n return super.equals((Box)o);\n } else {\n return false;\n }\n }\n\n public void unifyCheck(Object u) throws UnifyFailure {\n if (u instanceof Box) {\n super.unifyCheck((Box)u);\n } else {\n throw new UnifyFailure();\n }\n }\n\n public Object fill(Substitution sub) throws UnifyFailure {\n return new Box((Mode)_mode.fill(sub), (LF)_arg.fill(sub));\n }\n \n /** Returns the string form of this modal op, without the arg. */\n public String modalOpString() {\n return new StringBuffer().append('[').append(_mode.toString()).append(']').toString();\n }\n \n\n /**\n * Returns an XML representation of this LF (not currently supported).\n * Throws a runtime exception.\n */\n public Element toXml() {\n throw new RuntimeException(\"toXml() not currently supported for Box.\");\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/Compacter.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-6 Michael White (University of Edinburgh, The Ohio State University)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.util.*;\n\nimport java.util.*;\n\n/**\n * A class implementing compaction of flattened LFs.\n *\n * @author Michael White\n * @version $Revision: 1.1 $, $Date: 2006/09/04 14:09:10 $\n **/\npublic class Compacter {\n\n /**\n * Returns a compacted LF from the given flattened one. \n * A root nominal may also be given (otherwise null). \n * Nominals with multiple parents are kept separate.\n * If there are any duplicate predications, an attempt \n * is made to attach them in different locations.\n */\n static LF compact(LF lf, Nominal root) {\n // get preds, make copies\n List preds = HyloHelper.getPreds(lf);\n for (int i=0; i < preds.size(); i++) {\n SatOp pred = preds.get(i);\n preds.set(i, (SatOp) pred.copy());\n }\n \n // check for single pred\n if (preds.size() == 1) return preds.get(0);\n \n // find unique parents and multiple parents\n Map parents = new HashMap();\n GroupMap multipleParents = new GroupMap();\n for (int i = 0; i < preds.size(); i++) {\n SatOp pred = preds.get(i);\n // get principal nominal as nom1\n Nominal nom1 = HyloHelper.getPrincipalNominal(pred);\n // get secondary nominal\n Nominal nom2 = HyloHelper.getSecondaryNominal(pred);\n // skip if none or nom2 equal to root\n if (nom2 == null) continue;\n if (root != null && nom2.equals(root)) continue;\n // if nom2 already in group map, add nom1 as another parent\n if (multipleParents.containsKey(nom2)) {\n multipleParents.put(nom2, nom1);\n }\n // if nom2 already in parent map, add existing parent and nom1 to group map, \n // record pred, then remove nom2 from parent map\n else if (parents.containsKey(nom2)) {\n multipleParents.put(nom2, parents.get(nom2));\n multipleParents.put(nom2, nom1);\n parents.remove(nom2);\n }\n // otherwise put in nom1 as parent\n else {\n parents.put(nom2, nom1);\n }\n }\n\n // check multiple parent nominals for cycles\n int prevSize = -1;\n List history = new ArrayList();\n while (multipleParents.size() != prevSize) {\n prevSize = multipleParents.size();\n for (Iterator it = multipleParents.keySet().iterator(); it.hasNext(); ) {\n \tNominal nom = it.next();\n Set nomParents = multipleParents.get(nom);\n for (Iterator it2 = nomParents.iterator(); it2.hasNext(); ) {\n \tNominal parent = it2.next();\n history.clear();\n history.add(nom);\n while (parent != null && !history.contains(parent)) { \n history.add(parent);\n parent = parents.get(parent);\n }\n // remove if cycle found\n if (parent != null) it2.remove();\n }\n // switch to single parent if others removed\n if (nomParents.size() == 1) {\n \tNominal parent = nomParents.iterator().next();\n parents.put(nom, parent);\n it.remove();\n }\n }\n }\n \n // break any remaining cycles in parent relationships\n for (Iterator it = parents.keySet().iterator(); it.hasNext(); ) {\n \tNominal nom = it.next();\n \tNominal parent = parents.get(nom);\n history.clear();\n history.add(nom);\n while (parent != null && !history.contains(parent)) {\n history.add(parent);\n parent = parents.get(parent);\n }\n if (parent != null) { it.remove(); } \n }\n \n // ensure sorted\n HyloHelper.sort(preds);\n \n // combine preds on same nominal\n // also: gather any duplicate preds \n List combinedPreds = new ArrayList(preds.size());\n List dupPreds = new ArrayList(preds.size());\n SatOp currentSatOp = preds.get(0);\n Nominal currentNominal = currentSatOp.getNominal();\n combinedPreds.add(currentSatOp);\n for (int i = 1; i < preds.size(); i++) {\n SatOp satOp = preds.get(i);\n // skip if equal to previous, saving in dupPreds\n if (satOp.equals(preds.get(i-1))) {\n \tdupPreds.add(satOp); continue; \n }\n // check for different nominal\n Nominal nominal = satOp.getNominal();\n if (!nominal.equals(currentNominal)) {\n // add to combined preds, update current refs\n currentSatOp = satOp;\n currentNominal = nominal;\n combinedPreds.add(currentSatOp);\n }\n // otherwise combine\n else {\n combine(currentSatOp, satOp);\n }\n }\n \n // compact preds with unique parent\n for (int i = 0; i < combinedPreds.size(); i++) {\n SatOp satOp1 = combinedPreds.get(i);\n Nominal nom1 = satOp1.getNominal();\n if (!parents.containsValue(nom1)) continue;\n for (int j = 0; j < combinedPreds.size(); j++) {\n SatOp satOp2 = combinedPreds.get(j);\n Nominal nom2 = satOp2.getNominal();\n if (nom1.equals(nom2)) continue;\n if (!parents.containsKey(nom2)) continue;\n if (nom1.equals(parents.get(nom2))) {\n subst(satOp1, satOp2, nom2, null);\n }\n }\n }\n \n // get root nominals, root preds, and multiple parent preds\n List roots = new ArrayList();\n List rootPreds = new ArrayList();\n List multipleParentPreds = new ArrayList();\n for (int i = 0; i < combinedPreds.size(); i++) {\n SatOp pred = combinedPreds.get(i);\n Nominal nom = pred.getNominal();\n if (!parents.containsKey(nom) && !multipleParents.containsKey(nom)) {\n roots.add(nom);\n rootPreds.add(pred);\n }\n if (multipleParents.containsKey(nom)) {\n multipleParentPreds.add(pred);\n }\n }\n \n // compact preds with multiple parents, using parent that is closest to a root\n prevSize = -1;\n while (multipleParentPreds.size() != prevSize) {\n prevSize = multipleParentPreds.size();\n // for each nominal with multiple parents\n for (Iterator it = multipleParentPreds.iterator(); it.hasNext(); ) {\n SatOp pred = it.next();\n Nominal nom = pred.getNominal();\n // find parent closest to root, but checking for a parent not below a root\n Set nomParents = multipleParents.get(nom);\n Nominal parentClosestToRoot = null;\n int closestDist = 0;\n int closestRootIndex = -1;\n for (Iterator it2 = nomParents.iterator(); it2.hasNext(); ) {\n Nominal parent = it2.next();\n int dist = 0;\n // trace parents to top ancestor\n Nominal topAncestor = parent;\n while (parents.containsKey(topAncestor)) {\n topAncestor = parents.get(topAncestor);\n dist++;\n }\n // if top ancestor a root, update closest parent\n if (roots.contains(topAncestor)) {\n if (parentClosestToRoot == null || dist < closestDist) {\n parentClosestToRoot = parent; \n closestDist = dist;\n closestRootIndex = roots.indexOf(topAncestor);\n }\n }\n // otherwise set closest dist to -1, to indicate that not all ancestors are roots\n else { closestDist = -1; }\n }\n // check for a parent not below a root, or no closest root, and skip this nom if so\n if (closestDist == -1 || closestRootIndex == -1) { continue; }\n // otherwise compact under root pred of parent closest to root\n SatOp closestRootPred = rootPreds.get(closestRootIndex);\n subst(closestRootPred, pred, nom, parentClosestToRoot);\n // update parents map\n parents.put(nom, parentClosestToRoot);\n // and remove from iterator\n it.remove();\n }\n }\n \n // set retval to single remaining pred or conjunction of remaining ones\n LF retval;\n List retPreds = new ArrayList();\n retPreds.addAll(rootPreds);\n retPreds.addAll(multipleParentPreds);\n if (retPreds.size() == 1) { retval = retPreds.get(0); }\n else { retval = new Op(Op.CONJ, retPreds); }\n \n // tmp\n for (SatOp dup : dupPreds) {\n \tNominal nom = dup.getNominal();\n \tNominal dupParent = findDupParent(retval, dup, nom);\n \tsubst(retval, dup, nom, dupParent); \n }\n\n // return\n return retval;\n }\n \n \n // combines two preds for the same nominal into the first pred, \n // where either both preds are elementary, \n // or the first is the result of an earlier combination\n private static void combine(SatOp satOp1, SatOp satOp2) {\n // get args\n LF arg1 = satOp1.getArg();\n LF arg2 = satOp2.getArg();\n // check if arg1 already conj op\n if (arg1 instanceof Op && ((Op)arg1).getName().equals(Op.CONJ)) {\n List args = ((Op)arg1).getArguments();\n args.add(arg2);\n }\n // or make it one\n else {\n List args = new ArrayList(2);\n args.add(arg1); args.add(arg2);\n satOp1.setArg(new Op(Op.CONJ, args));\n }\n }\n \n \n // substitutes the second satop into the first lf at nom2, optionally \n // respecting the given parent constraint (if non-null)\n // returns whether the substitution has been made\n private static boolean subst(LF lf, SatOp satOp2, Nominal nom2, Nominal requiredParent) {\n return subst(lf, null, satOp2, nom2, requiredParent);\n }\n \n // recursive implementation that tracks the current parent and \n // returns whether the substitution has been made\n private static boolean subst(LF lf, Nominal currentParent, SatOp satOp2, Nominal nom2, Nominal requiredParent) {\n // recurse to nom2, then append if requiredParent constraint met\n if (lf instanceof SatOp) {\n SatOp satOp = (SatOp) lf;\n return subst(satOp.getArg(), satOp.getNominal(), satOp2, nom2, requiredParent);\n }\n else if (lf instanceof Diamond) {\n Diamond d = (Diamond) lf;\n LF arg = d.getArg();\n // check for nom2, and that requiredParent constraint met\n if (arg.equals(nom2) && (requiredParent == null || requiredParent.equals(currentParent))) {\n // make substitution\n d.setArg(HyloHelper.append(arg, satOp2.getArg()));\n return true;\n }\n else {\n return subst(arg, currentParent, satOp2, nom2, requiredParent);\n }\n }\n else if (lf instanceof Op) {\n \tOp op = (Op) lf;\n List args = op.getArguments();\n for (int i = 0; i < args.size(); i++) {\n LF arg = args.get(i);\n if (arg instanceof Nominal) {\n // check for nom2, and that requiredParent constraint met\n if (arg.equals(nom2) && (requiredParent == null || requiredParent.equals(currentParent))) {\n // make substitution\n \t// nb: this (rarely used) operation doesn't nec. preserve the sort order, unfortunately\n \top.appendArgs(satOp2.getArg());\n return true;\n }\n // otherwise, set current parent and continue\n else {\n\t currentParent = (Nominal) arg;\n\t continue;\n }\n }\n boolean madeSubst = subst(arg, currentParent, satOp2, nom2, requiredParent);\n if (madeSubst) return true;\n }\n }\n return false;\n }\n \n \n // returns a parent nominal where the duplicate pred can be substituted \n // if there is no equivalent pred there already; otherwise returns null\n private static Nominal findDupParent(LF lf, SatOp dup, Nominal dupNom) {\n return findDupParent(lf, null, dup, dupNom);\n }\n \n // recursive implementation that tracks the current parent\n private static Nominal findDupParent(LF lf, Nominal currentParent, SatOp dup, Nominal dupNom) {\n // recurse to dupNom, then return parent if apropos\n if (lf instanceof SatOp) {\n SatOp satOp = (SatOp) lf;\n return findDupParent(satOp.getArg(), satOp.getNominal(), dup, dupNom);\n }\n else if (lf instanceof Diamond) {\n Diamond d = (Diamond) lf;\n LF arg = d.getArg();\n // check for dupNom by itself, and return parent\n if (arg.equals(dupNom)) return currentParent;\n else return findDupParent(arg, currentParent, dup, dupNom);\n }\n else if (lf instanceof Op) {\n \tOp op = (Op) lf;\n List args = op.getArguments();\n for (int i = 0; i < args.size(); i++) {\n LF arg = args.get(i);\n if (arg instanceof Nominal) {\n // check for dupNom, and that no equiv pred constraint met\n if (arg.equals(dupNom) && !args.contains(dup.getArg()))\n \t// return parent\n \treturn currentParent;\n // otherwise, set current parent and continue\n else {\n\t currentParent = (Nominal) arg;\n\t continue;\n }\n }\n Nominal retval = findDupParent(arg, currentParent, dup, dupNom);\n if (retval != null) return retval;\n }\n }\n return null;\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/Converter.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-9 Michael White (University of Edinburgh, The Ohio State University)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.TextCCG;\nimport opennlp.ccg.synsem.*;\nimport java.util.*;\nimport java.util.prefs.Preferences;\n\n/**\n * A class implementing conversion of nominal variables to nominal atoms.\n *\n * @author Michael White\n * @version $Revision: 1.7 $, $Date: 2010/08/31 16:20:43 $\n **/\npublic class Converter {\n \n /** Preference key for using word positions to name atoms. */\n public static final String USE_WORD_POSITIONS_FOR_ATOM_CONVERSION = \"Use Word Positions To Convert Atoms\";\n\n\t// map to already converted nominals \n private Map nominalMap = new HashMap();\n \n // map to int for names\n private Map nameMap = new HashMap();\n\t\n // flag for whether to skip absent props\n private boolean skipAbsentProp = true;\n \t\n /** Converts nominal vars to atoms, renaming them based on lexical propositions. */\n\tstatic void convertNominals(LF lf) {\n\t\tconvertNominals(lf, null, null);\n }\n\t\n\t/**\n\t * Converts nominal vars to atoms, renaming them based on word position, if \n\t * a root sign is given, otherwise using lexical propositions; \n\t * returns the converted nominal root. \n\t */\n\tstatic Nominal convertNominals(LF lf, Sign root, Nominal nominalRoot) {\n\t\t// check preference for naming with word positions; set root to null if false\n Preferences prefs = Preferences.userNodeForPackage(TextCCG.class);\n boolean useWordPositions = prefs.getBoolean(USE_WORD_POSITIONS_FOR_ATOM_CONVERSION, true);\n\t\tif (!useWordPositions) root = null;\n // traverse twice, skipping absent props the first time\n \tConverter converter = new Converter();\n \tconverter.convertNoms(lf, root);\n converter.skipAbsentProp = false;\n \tconverter.convertNoms(lf, root);\n \t// return converted nominal root, if any\n \tNominal retval = null;\n \tif (nominalRoot != null) {\n \t\tretval = converter.nominalMap.get(nominalRoot);\n \t}\n \treturn retval;\n }\n\n // recurse through lf, converting nominals\n private void convertNoms(LF lf, Sign root) {\n if (lf instanceof SatOp) {\n SatOp satOp = (SatOp) lf;\n // try finding word index of lex origin in root sign\n int wordIndex = -1;\n if (root != null) {\n\t LexSemOrigin origin = satOp.getOrigin();\n\t if (origin instanceof Sign) {\n\t \tSign lexSign = (Sign) origin;\n\t \t// make sure it's not dominated by another lex pred\n\t \t// nb: also need to check for special pred 'elem', which isn't \n\t \t// dominated in sample flights grammar\n\t \tString lexPred = HyloHelper.getLexPred(satOp);\n\t \tif (lexPred != null && !lexPred.equals(\"elem\")) {\n\t \t\tif (!lexDominated(lexPred, lexSign)) \n\t\t \t\twordIndex = root.wordIndex(lexSign);\n\t \t}\n\t }\n }\n Nominal oldNom = satOp.getNominal();\n Proposition prop = null;\n LF arg = satOp.getArg();\n if (arg instanceof Proposition) { prop = (Proposition) arg; }\n else if (arg instanceof Op) {\n Op op = (Op) arg;\n LF first = (LF) op.getArguments().get(0);\n if (first instanceof Proposition) { prop = (Proposition) first; }\n }\n Nominal convertedNom = convertNominal(oldNom, prop, wordIndex);\n satOp.setNominal(convertedNom);\n convertNoms(arg, root);\n }\n else if (lf instanceof Diamond) {\n Diamond d = (Diamond) lf;\n LF arg = d.getArg();\n if (arg instanceof Nominal) {\n Nominal oldNom = (Nominal) arg;\n Nominal convertedNom = convertNominal(oldNom, null, -1);\n d.setArg(convertedNom);\n }\n else if (arg instanceof Op) {\n Op op = (Op) arg;\n List args = op.getArguments();\n LF first = args.get(0);\n if (first instanceof Nominal) {\n Nominal oldNom = (Nominal) first;\n LF second = args.get(1);\n Proposition prop = null;\n if (second instanceof Proposition) { prop = (Proposition) second; }\n Nominal convertedNom = convertNominal(oldNom, prop, -1);\n args.set(0, convertedNom);\n }\n convertNoms(arg, root);\n }\n }\n else if (lf instanceof Op) {\n List args = ((Op)lf).getArguments();\n for (int i = 0; i < args.size(); i++) {\n \tconvertNoms(args.get(i), root);\n }\n }\n }\n\n // returns a nominal atom based on the old nominal, prop and maps, \n // which are updated accordingly; \n // wordIndex is used instead if non-negative;\n // the skipAbsentProp flag controls whether to skip a null prop, \n // so that a meaningful name might be created later\n private Nominal convertNominal(Nominal oldNom, Proposition prop, int wordIndex) {\n // check for an atom\n if (oldNom instanceof NominalAtom) return oldNom;\n // handle word index case\n if (wordIndex >= 0) return convertNominal(oldNom, \"w\" + wordIndex);\n // skip absent props according to flag\n if (prop == null && skipAbsentProp) return oldNom;\n // check if already converted, and return copy\n Nominal alreadyConvertedNom = nominalMap.get(oldNom);\n if (alreadyConvertedNom != null) {\n return (Nominal) alreadyConvertedNom.copy();\n }\n // otherwise create new atom, with name based on prop (if possible)\n String nameBase = \"x\";\n if (prop != null) { \n nameBase = prop.toString().toLowerCase().substring(0,1); \n // use \"n\" if not a letter\n if (!Character.isLetter(nameBase.charAt(0))) nameBase = \"n\";\n }\n int ext = 1;\n Integer baseCount = nameMap.get(nameBase);\n if (baseCount != null) { ext = baseCount.intValue() + 1; }\n nameMap.put(nameBase, new Integer(ext));\n String name = nameBase + ext;\n return convertNominal(oldNom, name);\n }\n\n // returns the converted nominal using the given name, updating the map\n private Nominal convertNominal(Nominal oldNom, String name) {\n Nominal retval = new NominalAtom(name, oldNom.getType());\n nominalMap.put(oldNom, retval);\n return retval;\n }\n \n \n //---------------------------------------------------------------------------\n // check for dominating lex pred\n //\n \n // returns true if the EP for the lexPred is dominated by another lex pred\n private static boolean lexDominated(String lexPred, Sign lexSign) {\n \tCategory cat = lexSign.getCategory();\n \tLF lf = cat.getLF();\n \tNominal index = cat.getIndexNominal();\n \tList preds = HyloHelper.getPreds(lf);\n \t// find EP with lexPred, other lex preds\n \tSatOp lexEP = null;\n \tList otherLexPreds = new ArrayList();\n \tfor (SatOp pred : preds) {\n \t\tif (HyloHelper.isLexPred(pred)) {\n \t\tif (lexPred.equals(HyloHelper.getLexPred(pred))) \n \t\t\tlexEP = pred; \n \t\telse otherLexPreds.add(pred);\n \t\t}\n \t}\n \tif (lexEP == null) { \n \t\tthrow new RuntimeException(\"Couldn't find lexPred: \" + lexPred); \n\t\t}\n \t// check domination\n \tNominal lexNom = HyloHelper.getPrincipalNominal(lexEP);\n \tfor (SatOp pred : otherLexPreds) {\n \t\tNominal otherNom = HyloHelper.getPrincipalNominal(pred);\n \t\tStack seen = new Stack();\n \t\tseen.push(index); // don't recurse through index nominal\n \t\tif (dominates(otherNom, lexNom, preds, seen)) return true; \n \t}\n \t// otherwise false\n \treturn false;\n }\n \n // returns true if a dominates b in preds, using seen stack to avoid looping\n private static boolean dominates(Nominal a, Nominal b, List preds, Stack seen) {\n \t// check for identity\n \tif (a.equals(b)) return false;\n \t// push a to seen noms\n \tseen.push(a);\n \t// check relations\n \tfor (SatOp pred : preds) {\n \t\tif (a.equals(HyloHelper.getPrincipalNominal(pred))) {\n \t\t\tNominal c = HyloHelper.getSecondaryNominal(pred);\n \t\t\tif (c == null) continue;\n \t\t\t// check immed dominance\n \t\t\tif (b.equals(c)) return true; // found dominance!\n \t\t\t// check seen\n \t\t\tif (seen.contains(c)) continue;\n \t\t\t// recurse\n \t\t\tif (dominates(c, b, preds, seen)) return true;\n \t\t}\n \t}\n \t// otherwise not; pop a and return\n \tseen.pop();\n \treturn false;\n }\n \n \n //---------------------------------------------------------------------------\n // convert nominal atoms back to vars\n //\n \n /** Converts nominal atoms back to vars. */\n\tstatic void convertNominalsToVars(List preds) {\n\t\tconvertNominalsToVars(preds, null);\n }\n\t\n\t/**\n\t * Converts nominal atoms back to vars, returning the converted nominal root. \n\t */\n\tstatic Nominal convertNominalsToVars(List preds, Nominal nominalRoot) {\n\t\tNominal retval = null;\n\t\tfor (SatOp pred : preds) {\n\t\t\tNominal nom = pred._nominal;\n\t\t\tNominal nv = convertNominalToVar(nom);\n\t\t\tif (nom.equals(nominalRoot)) retval = nv;\n\t\t\tpred.setNominal(nv);\n\t\t\tLF arg = pred.getArg();\n\t\t\tif (arg instanceof Diamond) {\n\t\t\t\tDiamond dArg = (Diamond) arg;\n\t\t\t\tLF arg2 = dArg.getArg();\n\t\t\t\tif (arg2 instanceof Nominal) {\n\t\t\t\t\tNominal nv2 = convertNominalToVar((Nominal)arg2);\n\t\t\t\t\tdArg.setArg(nv2);\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\treturn retval;\n\t}\n\t\n\t// returns a nominal var with the same name as the given nominal\n\tstatic Nominal convertNominalToVar(Nominal nom) {\n\t\treturn new NominalVar(nom.getName().toUpperCase(), nom.getType());\n\t}\n}\n" }, { "path": "src/opennlp/ccg/hylo/Diamond.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\nimport org.jdom.*;\n\n/**\n * A modal diamond operator, such as <P>p.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.6 $, $Date: 2009/07/17 04:23:30 $\n **/\npublic final class Diamond extends ModalOp {\n\n\tprivate static final long serialVersionUID = 543211908001651361L;\n\n\tpublic Diamond(Element e) {\n super(e);\n }\n\n public Diamond(Mode mode, LF arg) {\n super(mode, arg);\n }\n\n public LF copy() {\n return new Diamond ((Mode)_mode.copy(), _arg.copy());\n }\n \n public boolean equals(Object o) {\n if (o instanceof Diamond) {\n return super.equals((Diamond)o);\n } else {\n return false;\n }\n }\n\n public void unifyCheck(Object u) throws UnifyFailure {\n if (u instanceof Diamond) {\n super.unifyCheck((Diamond)u);\n } else {\n throw new UnifyFailure();\n }\n }\n\n public Object unify(Object u, Substitution sub) throws UnifyFailure {\n if (u instanceof HyloFormula) {\n if (u instanceof Diamond) {\n Mode $mode = (Mode) Unifier.unify(_mode, ((Diamond)u)._mode, sub);\n LF $arg = (LF) Unifier.unify(_arg,((Diamond)u)._arg, sub);\n return new Diamond($mode, $arg);\n }\n else return super.unify(u,sub);\n } else {\n throw new UnifyFailure();\n }\n }\n\n public Object fill(Substitution sub) throws UnifyFailure {\n return new Diamond((Mode)_mode.fill(sub), (LF)_arg.fill(sub));\n }\n \n /** Returns the string form of this modal op, without the arg. */\n public String modalOpString() {\n return new StringBuffer().append('<').append(_mode.toString()).append('>').toString();\n }\n \n /**\n * Returns an XML representation of this LF.\n */\n public Element toXml() {\n Element retval = new Element(\"diamond\");\n retval.setAttribute(\"mode\", _mode.toString());\n Element argElt = _arg.toXml();\n retval.addContent(argElt);\n return retval;\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/EPsScorer.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Michael White (The Ohio State University)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.util.Pair;\n\nimport java.text.NumberFormat;\nimport java.util.*;\n\n/**\n * A class implementing LF scoring in terms of recall and precision \n * of elementary predications.\n *\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2010/11/15 03:21:12 $\n **/\npublic class EPsScorer {\n\n\t/**\n\t * Class for scoring results.\n\t */\n\tpublic static class Results {\n\t\t// results\n\t\tpublic double recall = 0.0;\n\t\tpublic double precision = 0.0;\n\t\tpublic double fscore = 0.0;\n\t\tpublic double depsRecall = 0.0;\n\t\tpublic double depsPrecision = 0.0;\n\t\tpublic double depsFscore = 0.0;\n\t\tpublic double unlabeledDepsRecall = 0.0;\n\t\tpublic double unlabeledDepsPrecision = 0.0;\n\t\tpublic double unlabeledDepsFscore = 0.0;\n\t\t// display\n\t\tpublic String toString() {\n\t\t\treturn \"fscore: \" + nf.format(fscore) + \n\t\t\t\t\" recall: \" + nf.format(recall) + \n\t\t\t\t\" precision: \" + nf.format(precision) + \n\t\t\t\t\" deps fscore: \" + nf.format(depsFscore) + \n\t\t\t\t\" deps recall: \" + nf.format(depsRecall) + \n\t\t\t\t\" deps precision: \" + nf.format(depsPrecision) +\n\t\t\t\t\" unlabeled deps fscore: \" + nf.format(unlabeledDepsFscore) + \n\t\t\t\t\" unlabeled deps recall: \" + nf.format(unlabeledDepsRecall) + \n\t\t\t\t\" unlabeled deps precision: \" + nf.format(unlabeledDepsPrecision);\n\t\t}\n\t // formats to four decimal places\n\t private static final NumberFormat nf = initNF();\n\t private static NumberFormat initNF() { \n\t NumberFormat f = NumberFormat.getInstance();\n\t f.setMinimumIntegerDigits(1);\n\t f.setMinimumFractionDigits(1);\n\t f.setMaximumFractionDigits(4);\n\t return f;\n\t }\n\t}\n\n\t/**\n\t * Returns the results of scoring an LF against a gold LF.\n\t */\n\tpublic static Results score(LF lf, LF goldLF) {\n\t\t// get EPs\n\t\tList eps = HyloHelper.flatten(lf);\n\t\tList goldEPs = HyloHelper.flatten(goldLF);\n\t\tSet epsSet = new HashSet(eps);\n\t\tSet goldEPsSet = new HashSet(goldEPs);\n\t\t// get unlabeled deps\n\t\tSet> unlabeledDepsSet = new HashSet>();\n\t\tSet> goldUnlabeledDepsSet = new HashSet>();\n\t\tfor (SatOp ep : eps) {\n\t\t\tPair dep = getDep(ep);\n\t\t\tif (dep != null) unlabeledDepsSet.add(dep);\n\t\t}\n\t\tfor (SatOp ep : goldEPs) {\n\t\t\tPair dep = getDep(ep);\n\t\t\tif (dep != null) goldUnlabeledDepsSet.add(dep);\n\t\t}\n\t\t// calc recall\n\t\tResults retval = new Results();\n\t\tint recalled = 0, depsRecalled = 0, unlabeledDepsRecalled = 0;\n\t\tint goldDeps = goldUnlabeledDepsSet.size();\n\t\tfor (SatOp ep : goldEPs) {\n\t\t\tboolean isdep = HyloHelper.isRelPred(ep);\n\t\t\tif (epsSet.contains(ep)) {\n\t\t\t\trecalled++;\n\t\t\t\tif (isdep) depsRecalled++;\n\t\t\t}\n\t\t\tif (isdep && unlabeledDepsSet.contains(getDep(ep))) unlabeledDepsRecalled++;\n\t\t}\n\t\tretval.recall = 1.0 * recalled / goldEPs.size();\n\t\tretval.depsRecall = (goldDeps > 0) ? 1.0 * depsRecalled / goldDeps : 1.0;\n\t\tretval.unlabeledDepsRecall = (goldDeps > 0) ? 1.0 * unlabeledDepsRecalled / goldDeps : 1.0;\n\t\t// calc precision\n\t\tint precise = 0, depsPrecise = 0, unlabeledDepsPrecise = 0;\n\t\tint lfDeps = unlabeledDepsSet.size();\n\t\tfor (SatOp ep : eps) {\n\t\t\tboolean isdep = HyloHelper.isRelPred(ep);\n\t\t\tif (goldEPsSet.contains(ep)) {\n\t\t\t\tprecise++;\n\t\t\t\tif (isdep) depsPrecise++;\n\t\t\t}\n\t\t\tif (isdep && goldUnlabeledDepsSet.contains(getDep(ep))) unlabeledDepsPrecise++;\n\t\t}\n\t\tretval.precision = 1.0 * precise / eps.size();\n\t\tretval.depsPrecision = (lfDeps > 0) ? 1.0 * depsPrecise / lfDeps : 1.0;\n\t\tretval.unlabeledDepsPrecision = (lfDeps > 0) ? 1.0 * unlabeledDepsPrecise / lfDeps : 1.0;\n\t\t// calc f-score\n\t\tretval.fscore = fscore(retval.recall, retval.precision);\n\t\tretval.depsFscore = fscore(retval.depsRecall, retval.depsPrecision);\n\t\tretval.unlabeledDepsFscore = fscore(retval.unlabeledDepsRecall, retval.unlabeledDepsPrecision);\n\t\t// done\n\t\treturn retval;\n\t}\n\n\t// returns an unlabeled dependency as a pair of nominals, or null if the ep is not relational\n\tprivate static Pair getDep(SatOp ep) {\n\t\tif (HyloHelper.isRelPred(ep)) {\n\t\t\t// put nominals in canonical order, so that direction of dependency doesn't matter\n\t\t\tNominal n1 = HyloHelper.getPrincipalNominal(ep);\n\t\t\tNominal n2 = HyloHelper.getSecondaryNominal(ep);\n\t\t\tif (n1.compareTo(n2) <= 0) return new Pair(n1, n2);\n\t\t\telse return new Pair(n2, n1);\n\t\t}\n\t\telse\n\t\t\treturn null;\n\t}\n\t\n\t/** Calculates f-score as balanced harmonic mean of recall and precision. */\n\tpublic static double fscore(double recall, double precision) {\n\t\tif (recall + precision == 0.0) return 0.0;\n\t\treturn 2.0 * recall * precision / (recall + precision);\n\t}\n}\n" }, { "path": "src/opennlp/ccg/hylo/EnglishAgreementExtractor.java", "content": "package opennlp.ccg.hylo;\n\n///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2011 Rajakrishnan Rajkumar\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.perceptron.Alphabet;\nimport opennlp.ccg.perceptron.FeatureExtractor;\nimport opennlp.ccg.perceptron.FeatureMap;\nimport opennlp.ccg.perceptron.FeatureVector;\nimport opennlp.ccg.synsem.AtomCat;\nimport opennlp.ccg.synsem.Category;\nimport opennlp.ccg.synsem.ComplexCat;\nimport opennlp.ccg.synsem.Sign;\nimport opennlp.ccg.unify.FeatureStructure;\nimport opennlp.ccg.unify.SimpleType;\nimport opennlp.ccg.util.TrieMap;\nimport opennlp.ccg.hylo.LexDependency;\nimport java.util.*;\n\n/**\n * Class which extracts subject verb and relative pronoun agreement features for\n * English (described in):\n * \n * @InProceedings{rajkumar-white:2010:POSTERS, \n * \tauthor = {Rajkumar, Rajakrishnan and White, Michael}, \n * \ttitle = {Designing Agreement Features for Realization Ranking}, \n * \tbooktitle = {Coling 2010: Posters}, \n * \tmonth = {August}, \n * \tyear = {2010}, \n * \taddress = {Beijing, China}, \n * \tpublisher = {Coling 2010 Organizing Committee}, \n * \tpages = {1032--1040}, \n * \turl = {http://www.aclweb.org/anthology/C10-2119} \n * }\n * \n * The class extracts features based on the OpenCCG HLDS specific LF rels: ArgN (subject rel), whApposRel, GenRel, First, Next\n * \n * @author raja\n * @version $Revision: 1.11 $, $Date: 2011/11/25 18:18:33 $\n */\npublic class EnglishAgreementExtractor implements FeatureExtractor{\n\n\t/** Feature map wrapper, for unique retrieval from a sign's data objects. */\n\tpublic static class FeatureMapWrapper {\n\t\tpublic FeatureMap featureMap;\n\t\tpublic FeatureMapWrapper(FeatureMap featureMap) { this.featureMap = featureMap;}\n\t}\n\t\n\t/** Inner class to store specific properties of signs (right now for unbalanced punctuation status). */\n\tprivate class SignProps{\n\n\t\t//Store comma/dash unbalanced punctuation\n\t\tprivate String unbalPunct=null;\n\t\t\n\t\t/** Constructor to specify unbalanced punctuation. */\n\t\tpublic SignProps(String unbalPunct){\n\t\t\tthis.unbalPunct=unbalPunct;\n\t\t}\n\t\t\n\t\tpublic String getUnbalancedPunct(){\n\t\t\treturn unbalPunct;\n\t\t}\n\t}\n\t\n\t/** The alphabet. */\n\tprotected Alphabet alphabet = null;\n\n\t/** Current feature map. */\n\tprotected FeatureMap currentMap = null;\n\t\n\t/** Head and dependent signs (For feature extraction) .*/\n\tprotected Sign headSign=null;\n\tprotected Sign depSign=null;\n\t\n\t/** Error analysis related. */\n\t//Sentence id\n\tString sentId=null;\n\t//Instance num\n\tint INSTANCENUM=0;\n\t\n\t/** Subject-verb agreement feature extractors. */\n\tprotected List>> agrExtractors = new ArrayList>>();\n\tprotected List>> agrConjExtractors = new ArrayList>>();\n\tprotected List>> agrOfComplementExtractors = new ArrayList>>();\n\t\n\t/** WH-pronoun agreement feature extractors. */\n\tprotected List>> whExtractors = new ArrayList>>();\n\tprotected List>> whConjExtractors = new ArrayList>>();\n\t\n\t/** Punctuation agreement feature extractors. */\n\tprotected List>> punctExtractor = new ArrayList>>();\n\t\n\t/** Constructors. */\n\t\n\t//Constructor used during actual perceptron training and testing\n\tpublic EnglishAgreementExtractor() {\n\t\t\n\t\t// init lazy feature extractors\n\t\tthis.init();\n\t}\n\t\n\t//Constructor used during error analysis using serialized signs\n\tpublic EnglishAgreementExtractor(String sentId) {\n\t\t\n\t\t//init lazy feature extractors\n\t\tthis.sentId=sentId;\n\t\tthis.INSTANCENUM=0;\n\t\tthis.init();\n\t}\n\t\n\t/** Sets the alphabet. */\n\tpublic void setAlphabet(Alphabet alphabet) {\n\t\tthis.alphabet = alphabet;\n\t}\n\t\n\t/** Initializes lazy feature extractors .*/\n\tpublic void init() {\n\t\n\t\t//Agreement: Simple subj-verb feature extractors\n\t\tthis.agrExtractors.add(dep_word_head_word(1));\n\t\tthis.agrExtractors.add(dep_word_head_pos(1));\n\t\tthis.agrExtractors.add(dep_pos_head_word(1));\n\t\tthis.agrExtractors.add(dep_pos_head_pos(1));\n\n\t\t//Agreement: Disjunct subj feature extractors\n\t\tthis.agrConjExtractors.add(dep_word_head_word(2));\n\t\tthis.agrConjExtractors.add(dep_word_head_pos(2));\n\t\tthis.agrConjExtractors.add(dep_pos_head_word(2));\n\t\tthis.agrConjExtractors.add(dep_pos_head_pos(2));\n\t\t\n\t\t//Agreement: Of-complement feature extractors\n\t\tthis.agrOfComplementExtractors.add(dep_word_head_word(3));\n\t\tthis.agrOfComplementExtractors.add(dep_word_head_pos(3));\n\t\tthis.agrOfComplementExtractors.add(dep_pos_head_word(3));\n\t\tthis.agrOfComplementExtractors.add(dep_pos_head_pos(3));\n\t\t\n\t\t//WH-pronoun: Simple relative pronoun feature extractors\n\t\tthis.whExtractors.add(dep_word_head_stem(4));\n\t\tthis.whExtractors.add(dep_word_head_pos(4));\n\t\tthis.whExtractors.add(dep_word_head_class(4));\n\t\t\n\t\t//WH-pronoun: Conjunct/Disjunct subj feature extractors\n\t\tthis.whConjExtractors.add(dep_word_head_stem(5));\n\t\tthis.whConjExtractors.add(dep_word_head_pos(5));\n\t\tthis.whConjExtractors.add(dep_word_head_class(5));\n\t\t\n\t\t//Unbalanced punctuation\n\t\tthis.punctExtractor.add(unbal_punct());\n\t\t\n\t}\n\t\n\t/** Returns the features for the given sign and completeness flag. */\n\tpublic FeatureVector extractFeatures(Sign sign, boolean complete) {\n\t\taddFeatures(sign, complete);\n\t\treturn getFeatureMap(sign);\n\t}\n\t\n\t/** Recursively adds features to the feature map for the given sign, if not already present. */\n\t//TODO: Lazier feature extraction involving conditional feature extractors\n\tprotected void addFeatures(Sign sign, boolean complete) {\n\t\t// check for existing map, otherwise make one\n\t\tif (getFeatureMap(sign) != null) return;\n\t\t// lex case\n\t\tif (sign.isLexical()) {\n\t\t\tcurrentMap = new FeatureMap(0);\n\t\t}\n\t\t// non-terminal\n\t\telse {\n\t\t\tSign[] inputs = sign.getDerivationHistory().getInputs();\n\t\t\t// first recurse\n\t\t\tfor (Sign child : inputs) addFeatures(child, false);\n\t\t\t// use input maps in making current map\n\t\t\tif (inputs.length == 1) {\n\t\t\t\tcurrentMap = new FeatureMap(getFeatureMap(inputs[0]));\n\t\t\t}\n\t\t\telse if (inputs.length == 2) {\n\t\t\t\tcurrentMap = new FeatureMap(getFeatureMap(inputs[0]), getFeatureMap(inputs[1]));\n\t\t\t}\n\n\t\t\tString subjArg=null;\n\t\t\t\n\t\t\t//do each newly filled dep\n\t\t\tfor (LexDependency dep : sign.getFilledDeps()) {\n\t\t\t\t\n\t\t\t\tthis.headSign=dep.lexHead;\n\t\t\t\tthis.depSign=dep.lexDep;\n\t\t\t\t//System.out.println(\"DEP: \"+dep);\n\t\t\t\t\n\t\t\t\t//Find value of the subject feature\n\t\t\t\tif(subjArg==null){\n\t\t\t\t\tsubjArg=getSubjectFeature(dep.lexHead.getCategory());\n\t\t\t\t\t//Back-off to Arg0 heuristic if subject feature not available for this verb\n\t\t\t\t\tif(subjArg==null)subjArg=\"Arg0\";\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\t//Subject-verb agr features\n\t\t\t\tif(subjArg.equals(dep.rel) && (dep.lexHead.getOrthography().equals(\"was\") || dep.lexHead.getOrthography().equals(\"were\") || dep.lexHead.getPOS().equals(\"VBZ\") || dep.lexHead.getPOS().equals(\"VBP\"))){\n\t\t\t\t\t\n\t\t\t\t\t//Simple subj-verb feats\n\t\t\t\t\t//Increment instance number if in error analysis mode\n\t\t\t\t\tif(sentId!=null)INSTANCENUM++;\n\t\t\t\t\tinc(agrExtractors);\n\n\t\t\t\t\t//Disjunct features\n\t\t\t\t\tif(dep.lexDep.getOrthography().equals(\"or\")){\n\t\t\t\t\t\tArrayListrels=new ArrayList(2);\n\t\t\t\t\t\trels.add(\"First\");rels.add(\"Next\");\n\t\t\t\t\t\tHashtablecdeps=this.getLowerSiblingDeps(inputs,dep.lexDep,rels,null);\n\t\t\t\t\t\tif(cdeps!=null){\n\t\t\t\t\t\t\tfor(Enumeratione=cdeps.keys();e.hasMoreElements();){\n\t\t\t\t\t\t\t\tLexDependency cdep=e.nextElement();\n\t\t\t\t\t\t\t\tthis.depSign=cdep.lexDep;\n\t\t\t\t\t\t\t\tinc(agrConjExtractors);\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t\n\t\t\t\t\t//Of-complement subjects (for non-numeral, non-%-sign subjs)\n\t\t\t\t\tString subjClass=dep.lexDep.getWords().get(0).getSemClass();\n\t\t\t\t\tString subjPOS=dep.lexDep.getPOS();\n\t\t\t\t\tif(subjClass==null)subjClass=\"NULL\";\n\t\t\t\t\tif(!subjClass.equals(\"PERCENT\") && !subjPOS.startsWith(\"CD\")){\n\t\t\t\t\t\tArrayListrels=new ArrayList(1);\n\t\t\t\t\t\trels.add(\"Mod\");\n\t\t\t\t\t\tArrayListdepPreds=new ArrayList(1);\n\t\t\t\t\t\tdepPreds.add(\"of\");\n\t\t\t\t\t\tHashtableofComplDeps=this.getLowerSiblingDeps(inputs,dep.lexDep,rels,depPreds);\n\t\t\t\t\t\tif(ofComplDeps!=null){\n\t\t\t\t\t\t\tfor(Enumeratione1=ofComplDeps.keys();e1.hasMoreElements();){\n\t\t\t\t\t\t\t\tLexDependency ofComplDep=e1.nextElement();\n\t\t\t\t\t\t\t\tSign[] ofComplSigns=ofComplDeps.get(ofComplDep).getDerivationHistory().getInputs();\n\t\t\t\t\t\t\t\trels=new ArrayList(1);\n\t\t\t\t\t\t\t\trels.add(\"Arg1\");\n\t\t\t\t\t\t\t\tHashtableofDeps=this.getLowerSiblingDeps(ofComplSigns,ofComplDep.lexDep,rels,null);\n\t\t\t\t\t\t\t\tif(ofDeps!=null){\n\t\t\t\t\t\t\t\t\tfor(Enumeratione2=ofDeps.keys();e2.hasMoreElements();){\n\t\t\t\t\t\t\t\t\t\tLexDependency ofDep=e2.nextElement();\n\t\t\t\t\t\t\t\t\t\tthis.depSign=ofDep.lexDep;\n\t\t\t\t\t\t\t\t\t\tinc(agrOfComplementExtractors);\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\t//Relative clause features\n\t\t\t\tString whPrn=dep.lexDep.getOrthography();\n\t\t\t\tif((dep.rel.equals(\"GenRel\")||dep.rel.equals(\"whApposRel\")) && (whPrn.equals(\"that\")||whPrn.equals(\"who\")||whPrn.equals(\"which\")||whPrn.equals(\"whose\"))){\n\t\t\t\t\t\n\t\t\t\t\t//Make sure relative clause is linked to head of the quoted NP \n\t\t\t\t\t//(and not the quotation mark itself)\n\t\t\t\t\tSign sib=this.getSibling(sign.getSiblingFilledDeps(),\"Arg\");\n\t\t\t\t\tif(sib!=null){\n\t\t\t\t\t\tthis.headSign=sib;\n\t\t\t\t\t}\n\t\t\t\t\t\n\t\t\t\t\t//Simple WH-pronoun features\n\t\t\t\t\t//Increment instance number if in error analysis mode\n\t\t\t\t\tif(sentId!=null)INSTANCENUM++;\n\t\t\t\t\tinc(whExtractors);\n\n\t\t\t\t\t//Proximal conjunct features\n\t\t\t\t\tif(dep.lexDep.getPOS().equals(\"CC\") || dep.lexDep.getOrthography().equals(\",\") || dep.lexDep.getOrthography().equals(\";\")|| dep.lexDep.getOrthography().equals(\"or\")|| dep.lexDep.getOrthography().equals(\"and\")){\n\t\t\t\t\t\tArrayListrels=new ArrayList(1);\n\t\t\t\t\t\trels.add(\"Next\");\n\t\t\t\t\t\tHashtablecdeps=this.getLowerSiblingDeps(inputs,dep.lexDep,rels,null);\n\t\t\t\t\t\tif(cdeps!=null){\n\t\t\t\t\t\t\tfor(Enumeratione=cdeps.keys();e.hasMoreElements();){\n\t\t\t\t\t\t\t\tLexDependency cdep=e.nextElement();\n\t\t\t\t\t\t\t\tthis.depSign=cdep.lexDep;\n\t\t\t\t\t\t\t\tinc(whConjExtractors);\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\t//Punctuation feature extraction: Unbalanced sentence medial appositions are flagged\n\t\t\tif (sign!=null && inputs!=null) {\n\t\t\t\t\n\t\t\t\t//Pass up unbalanced punctuation indicator\n\t\t\t\t\n\t\t\t\t//Result cat of current has unbal feature\n\t\t\t\tCategory target = sign.getCategory().getTarget();\n\t\t\t\tFeatureStructure fs = target.getFeatureStructure();\n\t\t\t\tString punctFeatVal=null;\n\t\t\t\tif ( (fs != null && fs.hasAttribute(\"unbal\"))) {\n\t\t\t\t\tObject val = fs.getValue(\"unbal\");\n\t\t\t\t\tpunctFeatVal = (val instanceof SimpleType) ? ((SimpleType)val).getName() : null;\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\t//Right child (binary case) or only child (unary case) has unbalanced punct feature\n\t\t\t\tSignProps childProps=(SignProps)inputs[inputs.length-1].getData(SignProps.class);\n\t\t\t\tif(childProps!=null)punctFeatVal=childProps.getUnbalancedPunct();\n\t\t\t\t\n\t\t\t\tif(punctFeatVal!=null){\n\t\t\t\t\tSignProps currProps=new SignProps(punctFeatVal);\n\t\t\t\t\tsign.addData(currProps);\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\t//Extract unbalanced punctuation feature for unbalanced sentence medial punctuation\n\t\t\t\tif (inputs.length == 2) {\n\t\t\t\t\t//Left child has unbalanced punct feature\n\t\t\t\t\tSignProps lchildProps=(SignProps)inputs[0].getData(SignProps.class);\n\t\t\t\t\tif(lchildProps!=null && lchildProps.getUnbalancedPunct()!=null){\n\t\t\t\t\t\tWord nextWord = inputs[1].getWords().get(0);\n\t\t\t\t\t\t//Check whether right child begins with a punctuation mark; else fire feature\n\t\t\t\t\t\tif (!isPunct(nextWord)){\n\t\t\t\t\t\t\tinc(punctExtractor);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\t// store it\n\t\tstoreFeatureMap(sign);\n\t}\n\t\n\tpublic Sign getOfComplSign(){\n\t\n\t\tSign retval=null;\n\t\t\n\t\treturn retval;\n\t}\n\t\n\t/** Stores the current feature map as a data object in the given sign. */\n\tprotected void storeFeatureMap(Sign sign) {\n\t\tsign.addData(new FeatureMapWrapper(currentMap));\n\t}\n\t\n\t/** Returns the feature map for this extractor from the given sign (null if none). */\n\tpublic FeatureMap getFeatureMap(Sign sign) {\n\t\tFeatureMapWrapper fmw = (FeatureMapWrapper)sign.getData(FeatureMapWrapper.class);\n\t\treturn (fmw != null) ? fmw.featureMap : null;\n\t}\n\t\n\n\t/**\n\t * Increments the count of the given features, if relevant.\n\t */\n\tprotected void inc(List>> extractors) {\n\t\tfor (List> lazyExtractor : extractors) {\n\t\t\tAlphabet.Feature f = alphabet.indexLazy(lazyExtractor);\n\t\t\tif (f != null)currentMap.inc(f);\n\t\t}\n\t}\n\n\t//------------------------------------\n\t// utility functions\n\t\n\t//Get value of subject feature from verb's result cat\n\tpublic String getSubjectFeature(Category cat){\n\t\n\t\tString retval=null;\n\t\tif (cat instanceof ComplexCat) {\n\t\t\tCategory resCat = ((ComplexCat)cat).getResult();\n\t\t\tretval=this.getSubjectFeature(resCat);\n\t\t}\n\t\telse if (cat instanceof AtomCat) {\n\t\t\tAtomCat ac = (AtomCat) cat;\n\t\t\tFeatureStructure fs = ac.getFeatureStructure();\n\t\t\tfor(String attr: fs.getAttributes()){\n\t\t\t\tif(attr.equals(\"sbj\")){\n\t\t\t\t\tretval=fs.getValue(attr).toString();\n\t\t\t\t\tbreak;\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn retval;\n\t}\n\t\n\t//checks for punct\n\tprivate boolean isPunct(Word w) {\n\t\tString pos = w.getPOS();\n\t\tboolean retval = pos.startsWith(\"PUNCT\");\n\t\tretval = retval || pos.equals(\".\") || pos.equals(\",\") || pos.equals(\";\") || pos.equals(\":\") || pos.equals(\"LRB\") || pos.equals(\"RRB\");\n\t\t//if (retval) {\n\t\t//System.out.println(\"isPunct: \" + w.getForm() + \" pos: \" + pos);\n\t\t//}\n\t\treturn retval;\n\t}\n\t\n\t// Get siblings of a given head 1-step down the derivation, given the head-sibling relations and lexical preds of deps .*/\n\tpublic Hashtable getLowerSiblingDeps(Sign[] inputs,Sign headSign,ArrayListrels,ArrayListdepPreds){\n\t\t\n\t\tHashtable retval=new Hashtable();\n\t\tfor(Sign sign: inputs){\n\t\t\tif(retval.size()==rels.size())break;\n\t\t\tListsdeps=sign.getSiblingFilledDeps();\n\t\t\tsdeps.addAll(sign.getFilledDeps());\n\t\t\tfor(LexDependency sdep: sdeps){\n\t\t\t\tif(sdep.lexHead==headSign && rels.contains(sdep.rel) && !retval.containsKey(sdep)){\n\t\t\t\t\tif(depPreds==null || depPreds.contains(sdep.lexDep.getOrthography())){\n\t\t\t\t\t\tretval.put(sdep,sign);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tif(retval.size()==0)retval=null;\n\t\treturn retval;\n\t}\n\t\n\t//returns sibling sign of a given head given a relation label\n\tprivate Sign getSibling(List sdeps,String rel){\n\t\t\n\t\tSign retval=null;\n\t\tif(sdeps!=null){\n\t\t\tfor(LexDependency dep: sdeps){\n\t\t\t\tif(dep.rel.equals(rel)){\n\t\t\t\t\tretval=dep.lexDep;\n\t\t\t\t\tbreak;\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn retval;\n\t}\n\t\n\t// returns acceptable paraphrases for words\n\tprivate String adjustWord(String word) {\n\t\t\n\t\tString retval=word;\n\t\t//Account for acceptable paraphrases\n\t\tif (word.equals(\"'ve\"))\n\t\t\tretval=\"have\";\n\t\telse if (word.equals(\"'s\"))\n\t\t\tretval=\"is\";\n\t\telse if (word.equals(\"'re\"))\n\t\t\tretval=\"are\";\n\t\t\n\t\treturn retval;\n\t}\n\t\n\t//adjusts POS tags\n\tprivate String adjustPOS(String word,String pos,String semClass) {\n\t\t\n\t\tString retval=pos;\n\t\t\n\t\tif(word.equals(\"has\"))\n\t\t\tretval=\"VBZ\";\n\t\telse if (word.equals(\"have\"))\n\t\t\tretval=\"VBP\";\n\t\telse if(word.equals(\"one\") || word.equals(\"1\"))\n\t\t\tpos=\"CD-1\";\n\t\telse if(semClass!=null && semClass.equals(\"PERCENT\"))\n\t\t\tretval=semClass;\n\t\telse if(word.equals(\",\") || word.equals(\";\"))\n\t\t\tretval=\"CC\";\n\t\t\n\t\treturn retval;\n\t}\n\t\n\t//adjust sem class info\n\tprivate String adjustSemClass(String semClass) {\n\t\tString retval=\"UNK\";\n\t\tif(semClass!=null){\n\t\t\tString[]temp=semClass.split(\"\\\\|\");\n\t\t\tretval=temp[0].split(\":\")[0];\n\t\t}\n\t\treturn retval;\n\t}\n\t\n\t//main prefixes (AGR=Agr; CONJ=Conjn/Disjn; WH=wh-pronoun; OF=Of-complement)\n\tprivate void add_prefix_main1(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return \"AGR\"; }});\n\t}\n\tprivate void add_prefix_main2(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return \"AGRCONJ\"; }});\n\t}\n\tprivate void add_prefix_main3(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return \"AGROF\"; }});\n\t}\n\tprivate void add_prefix_main4(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return \"AGRWH\"; }});\n\t}\n\tprivate void add_prefix_main5(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return \"AGRWHCONJ\"; }});\n\t}\n\t\n\t//instance # in error analysis mode\n\tprivate void add_instance_num(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return Integer.toString(INSTANCENUM); }});\n\t}\n\t\n\t//sub-prefixes (W=Word; P=POS tag; S=Stem; C=SemClass)\n\tprivate void add_prefix_sub1(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return \"WW\"; }});\n\t}\n\tprivate void add_prefix_sub2(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return \"WP\"; }});\n\t}\n\tprivate void add_prefix_sub3(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return \"PW\"; }});\n\t}\n\tprivate void add_prefix_sub4(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return \"PP\"; }});\n\t}\n\tprivate void add_prefix_sub5(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return \"WS\"; }});\n\t}\n\tprivate void add_prefix_sub6(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return \"WC\"; }});\n\t}\n\n\t//select required feature prefix\n\tprivate void add_prefix(int prefix,List> retval) {\n switch (prefix) {\n \t\tcase 1:add_prefix_main1(retval);break;\n \t\tcase 2:add_prefix_main2(retval);break;\n \t\tcase 3:add_prefix_main3(retval);break;\n \t\tcase 4:add_prefix_main4(retval);break;\n \t\tcase 5:add_prefix_main5(retval);break;\n }\n }\n\t\n\t//\t-------------------------------\n\t// feature extractors\n\t\n\t// dep-word-head-word\n\tprivate List> dep_word_head_word(int prefix) {\n\t\tList> retval = new ArrayList>(3);\n\t\tif(this.sentId!=null)add_instance_num(retval);\n\t\tadd_prefix(prefix,retval);\n\t\tadd_prefix_sub1(retval);\n\t\tadd_dep_word(retval);\n\t\tadd_head_word(retval);\n\t\treturn retval;\n\t}\n\t\n\t//dep-word head-pos\n\tprivate List> dep_word_head_pos(int prefix) {\n\t\tList> retval = new ArrayList>(3);\n\t\tif(this.sentId!=null)add_instance_num(retval);\n\t\tadd_prefix(prefix,retval);\n\t\tadd_prefix_sub2(retval);\n\t\tadd_dep_word(retval);\n\t\tadd_head_pos(retval);\n\t\treturn retval;\n\t}\n\t\n\t//dep-pos head-word\n\tprivate List> dep_pos_head_word(int prefix) {\n\t\tList> retval = new ArrayList>(3);\n\t\tif(this.sentId!=null)add_instance_num(retval);\n\t\tadd_prefix(prefix,retval);\n\t\tadd_prefix_sub3(retval);\n\t\tadd_dep_pos(retval);\n\t\tadd_head_word(retval);\n\t\treturn retval;\n\t}\n\t\n\t//dep-pos head-pos\n\tprivate List> dep_pos_head_pos(int prefix) {\n\t\tList> retval = new ArrayList>(3);\n\t\tif(this.sentId!=null)add_instance_num(retval);\n\t\tadd_prefix(prefix,retval);\n\t\tadd_prefix_sub4(retval);\n\t\tadd_dep_pos(retval);\n\t\tadd_head_pos(retval);\n\t\treturn retval;\n\t}\n\n\t//dep-word head-stem\n\tprivate List> dep_word_head_stem(int prefix) {\n\t\tList> retval = new ArrayList>(3);\n\t\tif(this.sentId!=null)add_instance_num(retval);\n\t\tadd_prefix(prefix,retval);\n\t\tadd_prefix_sub5(retval);\n\t\tadd_dep_word(retval);\n\t\tadd_head_stem(retval);\n\t\treturn retval;\n\t}\n\t\n\t//dep-word head-class\n\tprivate List> dep_word_head_class(int prefix) {\n\t\tList> retval = new ArrayList>(3);\n\t\tif(this.sentId!=null)add_instance_num(retval);\n\t\tadd_prefix(prefix,retval);\n\t\tadd_prefix_sub6(retval);\n\t\tadd_dep_word(retval);\n\t\tadd_head_class(retval);\n\t\treturn retval;\n\t}\n\t\n\t//unbalanced punctuation\n\tprivate List> unbal_punct() {\n\t\tList> retval = new ArrayList>(1);\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return \"$punct\"; }});\n\t\treturn retval;\n\t}\n\t\n\t//head word\n\tprivate void add_head_word(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ String word=adjustWord(headSign.getWordForm());return word; }});\n\t}\n\t\n\t//head stem\n\tprivate void add_head_stem(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return headSign.getWords().get(0).getStem();}});\n\t}\n\t\n\t//head class\n\tprivate void add_head_class(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ String semClass=adjustSemClass(headSign.getWords().get(0).getSemClass());return semClass;}});\n\t}\n\t\n\t// head pos\n\tprivate void add_head_pos(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ String pos=adjustPOS(headSign.getOrthography(),headSign.getPOS(),headSign.getWords().get(0).getSemClass());return pos; }});\n\t}\n\t\n\t// dep word\n\tprivate void add_dep_word(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ String word=adjustWord(depSign.getWordForm());return word; }});\n\t}\n\t\n\t// dep pos\n\tprivate void add_dep_pos(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ String pos=adjustPOS(depSign.getOrthography(),depSign.getPOS(),depSign.getWords().get(0).getSemClass());return pos; }});\n\t}\n}\n" }, { "path": "src/opennlp/ccg/hylo/Flattener.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-6 Michael White (University of Edinburgh, The Ohio State University)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.util.*;\n\nimport java.util.*;\n\nimport gnu.trove.*;\n\n/**\n * A class for performing flattening operations on LFs.\n *\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2011/06/06 18:52:30 $\n **/\npublic class Flattener {\n\n\t// the resulting preds\n\tprivate List preds = new ArrayList();\n \n\t// counter for alts\n\tprivate int altCount = 0;\n\t\n\t// counter for opts\n\tprivate int optCount = 0;\n\t\n\t// root preds\n\tprivate List roots = new ArrayList();\n\t\n\t// map from preds to children from original expression (identity keys);\n\t// includes dummy parents to preserve structure \n\tprivate ListMap childMap = new ListMap(true);\n \n // map from nominals to highest pred for that nominal from original expression\n\tprivate Map nomMap = new HashMap();\n \n // map from pred to depth in original expression\n\tprivate Map depthMap = new HashMap(); \n\t\n\t// map from nominal to highest parent nominal in original expression, or null if a root\n\tprivate Map parentMap = new HashMap();\n\t\n\t// null nominal for use in dummy parents during flattening\n\tprivate static Nominal nullNom = new NominalAtom(\"null\");\n\n\t// null prop for use in dummy parents during flattening\n\tprivate static Proposition nullProp = new Proposition(\"null\");\n\n\t/**\n\t * Returns a map from a nominal to its highest parent nominal in the original expression, \n\t * after flattening, or null if none.\n\t */\n\tpublic Map getHighestParentMap() {\n\t\treturn parentMap;\n\t}\n\t\n /**\n * Recursively flattens the given LF and returns a list of elementary preds.\n * LF chunks are preserved on satops, as are alts (exclusive disjunctions) \n * and opts (optional parts). \n * Chunks, alts and opts are propagated through shared nominals.\n * A runtime exception is thrown if the LF cannot be flattened.\n */\n public List flatten(LF lf) {\n flatten(lf, null, null, 0, new Stack(), new TIntArrayList());\n if (altCount > 0 || optCount > 0) propAltsOptsChunks();\n return preds;\n }\n\n // recursive flattening, with conversion of alts and opts\n private void flatten(\n LF lf, \n Nominal currentNominal, SatOp parent, int depth, \n Stack alts, TIntArrayList opts\n ) {\n if (lf instanceof SatOp) {\n // flatten arg with new current nominal\n SatOp satOp = (SatOp) lf;\n currentNominal = satOp.getNominal();\n SatOp dummyParent = makeDummySatOp(currentNominal);\n addSatOp(dummyParent, parent, depth, alts, opts, lf);\n flatten(satOp.getArg(), currentNominal, dummyParent, depth, alts, opts);\n }\n else if (lf instanceof Op) {\n Op op = (Op) lf;\n SatOp dummyParent = makeDummySatOp(currentNominal);\n addSatOp(dummyParent, parent, depth, alts, opts, lf);\n if (op._name.equals(Op.XOR)) {\n // introduce new alt set; add alt for each item\n int altSet = altCount++;\n for (int i = 0; i < op._args.size(); i++) {\n alts.push(new Alt(altSet, i));\n LF arg = op._args.get(i);\n flatten(arg, currentNominal, dummyParent, depth+1, alts, opts);\n alts.pop();\n }\n }\n else if (op._name.equals(Op.OPT)) {\n // introduce new opt index for arg\n opts.add(optCount++);\n LF arg = op._args.get(0);\n flatten(arg, currentNominal, dummyParent, depth+1, alts, opts);\n opts.remove(opts.size()-1);\n }\n else {\n // otherwise just flatten each item\n for (Iterator it = op.getArguments().iterator(); it.hasNext(); ) {\n flatten(it.next(), currentNominal, dummyParent, depth+1, alts, opts);\n }\n }\n }\n else if (lf instanceof Proposition) {\n // add SatOp for lf\n if (currentNominal == null) {\n throw new RuntimeException(\"No current nominal in trying to flatten \" + lf);\n }\n SatOp satOp = new SatOp(currentNominal, lf);\n addSatOp(satOp, parent, depth, alts, opts, lf);\n }\n else if (lf instanceof HyloVar) {\n // just skip for now\n }\n else if (lf instanceof Diamond) {\n Diamond diamond = (Diamond) lf;\n LF arg = diamond.getArg();\n if (arg instanceof Proposition || arg instanceof Nominal || arg instanceof HyloVar) {\n // add SatOp for diamond\n SatOp satOp = new SatOp(currentNominal, lf);\n addSatOp(satOp, parent, depth, alts, opts, lf);\n }\n else if (arg instanceof Op && ((Op)arg)._name.equals(Op.CONJ)) {\n // add SatOp for diamond with first nominal arg, \n // and flatten the rest of the args with the first nominal arg as the \n // new current nominal\n Op argOp = (Op) arg;\n Iterator args = argOp._args.iterator();\n LF firstArg = args.next();\n if (!(firstArg instanceof Nominal)) {\n throw new RuntimeException(\"First arg of diamond is not a nominal: \" + firstArg);\n }\n Nominal firstNominalArg = (Nominal) firstArg;\n // add SatOp for diamond\n SatOp satOp = new SatOp(currentNominal, new Diamond(diamond.getMode(), firstNominalArg));\n addSatOp(satOp, parent, depth, alts, opts, lf);\n // flatten rest of list\n for (; args.hasNext(); ) {\n flatten(args.next(), firstNominalArg, satOp, depth+1, alts, opts);\n }\n }\n else if (arg instanceof Op && ((Op)arg)._name.equals(Op.XOR)) {\n Op argOp = (Op) arg;\n SatOp dummyParent = makeDummySatOp(currentNominal);\n addSatOp(dummyParent, parent, depth, alts, opts, lf);\n // as before, process xor by introducing new alt set and adding alt for each disjunct; \n // this time, also assume each disjunct is a conj op or nominal, and add a diamond satop \n // to the disjunct nominal\n int altSet = altCount++;\n for (int i = 0; i < argOp._args.size(); i++) {\n alts.push(new Alt(altSet, i));\n LF disjunct = argOp._args.get(i);\n if (!(disjunct instanceof Op && ((Op)disjunct)._name.equals(Op.CONJ)) && !(disjunct instanceof Nominal)) {\n throw new RuntimeException(\"Disjunct of diamond is not a conj op or nominal: \" + disjunct);\n }\n // conj op case\n if (disjunct instanceof Op) {\n Op disjunctOp = (Op) disjunct;\n Iterator args = disjunctOp._args.iterator();\n LF firstArg = args.next();\n if (!(firstArg instanceof Nominal)) {\n throw new RuntimeException(\"First arg of conj op under xor op is not a nominal: \" + firstArg);\n }\n // add SatOp for diamond\n Nominal disjunctNominal = (Nominal) firstArg;\n SatOp satOp = new SatOp(currentNominal, new Diamond(diamond.getMode(), disjunctNominal));\n addSatOp(satOp, dummyParent, depth+1, alts, opts, lf);\n // flatten rest of list\n for (; args.hasNext(); ) {\n flatten(args.next(), disjunctNominal, satOp, depth+2, alts, opts);\n }\n }\n // nominal case\n else {\n // just add SatOp for diamond\n Nominal disjunctNominal = (Nominal) disjunct;\n SatOp satOp = new SatOp(currentNominal, new Diamond(diamond.getMode(), disjunctNominal));\n addSatOp(satOp, dummyParent, depth+1, alts, opts, lf);\n }\n alts.pop();\n }\n }\n else { \n throw new RuntimeException(\"Arg of diamond is not a proposition, nominal or list: \" + arg);\n }\n }\n else throw new RuntimeException(\"Unable to flatten \" + lf);\n }\n\n // makes a dummy satop for the given nominal, if any; otherwise uses nullNom\n private static SatOp makeDummySatOp(Nominal nom) {\n return new SatOp((nom != null) ? nom : nullNom, nullProp);\n }\n\n // handles new preds\n private void addSatOp(SatOp satOp, SatOp parent, int depth, Stack alts, TIntArrayList opts, LF lf) {\n \t// add non-dummy satops to result\n if (satOp._arg != nullProp) preds.add(satOp);\n // update roots, maps\n if (parent == null) roots.add(satOp);\n else childMap.put(parent, satOp);\n Nominal nom = satOp._nominal;\n if (!nom.isShared()) {\n\t if (!nomMap.containsKey(nom) || depth < depthMap.get(nom)) {\n\t \t\tnomMap.put(nom, satOp);\n\t \t\tdepthMap.put(nom, depth);\n\t \t\tparentMap.put(nom, (parent != null && parent._nominal != nullNom) ? parent._nominal : null);\n\t }\n }\n // set alts, opts, chunks\n if (!alts.empty()) satOp.alts = new ArrayList(alts);\n if (opts.size() > 0) satOp.opts = new TIntArrayList(opts.toNativeArray());\n satOp.setChunks(lf.getChunks());\n }\n \n // propagates alts, opts and chunks down from roots\n private void propAltsOptsChunks() {\n // propagate for each root nom\n List alts = Collections.emptyList(); \n TIntArrayList opts = new TIntArrayList(0);\n TIntArrayList chunks = new TIntArrayList(0);\n for (SatOp root : roots) {\n \tpropAltsOptsChunks(root, alts, opts, chunks);\n }\n }\n \n // prop alts, opts & chunks, recursing through preds in child map and shared nom refs in nomMap\n\tprivate void propAltsOptsChunks(SatOp satOp, List alts, TIntArrayList opts, TIntArrayList chunks) { \n // prop alts and opts\n if (!alts.isEmpty()) {\n if (satOp.alts == null) satOp.alts = new ArrayList(3);\n for (Alt alt : alts) {\n if (!satOp.alts.contains(alt)) satOp.alts.add(alt);\n }\n Collections.sort(satOp.alts); \n }\n if (!opts.isEmpty()) {\n if (satOp.opts == null) satOp.opts = new TIntArrayList(3);\n for (int i=0; i < opts.size(); i++) {\n int opt = opts.get(i);\n if (!satOp.opts.contains(opt)) satOp.opts.add(opt);\n }\n satOp.opts.sort();\n }\n if (!chunks.isEmpty()) {\n if (satOp.chunks == null) satOp.chunks = new TIntArrayList(3);\n for (int i=0; i < chunks.size(); i++) {\n int chunk = chunks.get(i);\n if (!satOp.chunks.contains(chunk)) satOp.chunks.add(chunk);\n }\n satOp.chunks.sort();\n }\n // gather alts, opts & chunks for recursion\n List alts2 = (satOp.alts != null) ? satOp.alts : alts;\n TIntArrayList opts2 = (satOp.opts != null) ? satOp.opts : opts;\n TIntArrayList chunks2 = (satOp.chunks != null) ? satOp.chunks : chunks;\n // recurse through children, if any\n List children = childMap.get(satOp);\n if (children != null) {\n for (SatOp child : children) \n \tpropAltsOptsChunks(child, alts2, opts2, chunks2);\n }\n // recurse through shared nominals, if apropos\n Nominal nom = satOp._nominal;\n if (nom.isShared()) {\n SatOp nomPred = nomMap.get(nom); \n if (nomPred != null) \n \tpropAltsOptsChunks(nomPred, alts2, opts2, chunks2);\n }\n Nominal nom2 = HyloHelper.getSecondaryNominal(satOp);\n if (nom2 != null && nom2.isShared()) {\n SatOp nom2Pred = nomMap.get(nom2); \n if (nom2Pred != null) \n \tpropAltsOptsChunks(nom2Pred, alts2, opts2, chunks2);\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/HyloAtom.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.unify.*;\nimport gnu.trove.*;\n\n/**\n * A logical atomic formula.\n * The type is optional, so by default, it is not considered in determining equality.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.9 $, $Date: 2009/12/21 03:27:19 $\n **/\npublic abstract class HyloAtom extends HyloFormula {\n \n\tprivate static final long serialVersionUID = 1L;\n\t\n\tprotected String _name;\n protected SimpleType type;\n\n protected HyloAtom(String name) {\n this(name, null);\n }\n \n protected HyloAtom(String name, SimpleType st) {\n _name = name; type = st;\n }\n \n public void setAtomName(String name) {\n _name = name;\n }\n\n public String getName() { return _name; } \n \n public SimpleType getType() { return type; }\n\n public boolean occurs(Variable var) {\n return false;\n }\n\n public String toString() { \n return _name;\n }\n\n /**\n * Returns a pretty-printed string of this LF, with the given indent.\n */\n public String prettyPrint(String indent) {\n return toString();\n }\n \n public int compareTo(HyloAtom ha) {\n return _name.compareTo(ha._name);\n }\n\n /** Returns a hash code based on the atom name. */\n public int hashCode() { \n return _name.hashCode();\n }\n\n /**\n * Returns whether this atom equals the given object \n * based on the atom name.\n */\n public boolean equals(Object obj) {\n if (this == obj) return true;\n if (obj == null || obj.getClass() != this.getClass()) { return false; }\n HyloAtom ha = (HyloAtom) obj;\n return _name.equals(ha._name);\n }\n \n /**\n * Returns a hash code using the given map from vars to ints.\n */\n public int hashCode(TObjectIntHashMap varMap) { return hashCode(); }\n \n /**\n * Returns whether this atom equals the given object \n * up to variable names, using the given maps from vars to ints.\n */\n public boolean equals(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2) {\n return equals(obj);\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/HyloFormula.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-3 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport java.io.Serializable;\nimport org.jdom.*;\nimport gnu.trove.*;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\n\n/**\n * A parent class to implement reasonable default behavior for classes\n * representing data structures for hybrid logic.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.13 $, $Date: 2009/12/21 03:27:19 $\n **/\npublic abstract class HyloFormula implements LF, Serializable {\n \n\tprivate static final long serialVersionUID = 1L;\n\t\n\t/**\n * The LF chunks to which this LF belongs.\n */\n protected TIntArrayList chunks = null;\n \n /**\n * Sets the LF chunks to which this LF belongs.\n * LF chunks are used during realization to ensure \n * that certain edges are semantically complete \n * before combination is attempted with edges \n * with semantics outside the chunk.\n * The chunks are numbered starting with 0, \n * and null represents no chunks.\n */\n public void setChunks(TIntArrayList chunks) { this.chunks = chunks; }\n \n /**\n * Gets the LF chunks to which this LF belongs.\n */\n public TIntArrayList getChunks() { return chunks; }\n \n \n /** Returns null as the default type. */\n public SimpleType getType() { return null; }\n\n \n /**\n * Returns a copy of this LF.\n * (LF chunks are not copied.)\n */\n public abstract LF copy();\n \n /**\n * Applies a ModFcn to this LF and then applies it to all fields\n * which are themselves Mutables.\n *\n * @param mf a function to be applied\n */\n public void deepMap(ModFcn mf) {\n mf.modify(this);\n }\n\n \n /**\n * Unify this Unfiable with another Object. \n * This default implementation will reverse the direction of unification \n * for a variable, otherwise it fails.\n * NB: The implementation of unification in the hylo package is not \n * complete; a particular limitation is that no attempt is made to unify lists of terms \n * connected by an Op instance.\n *\n * @param o object to unify with\n * @param s Substitution containing the variable resolutions\n * @exception UnifyFailure if this Unifiable cannot be unified with \n * the Object\n * @return an object which represents the unification of \n * this Unifiable with the Object\n */\n public Object unify(Object u, Substitution s) throws UnifyFailure {\n if (u instanceof Variable) return ((Unifiable)u).unify(this, s);\n else throw new UnifyFailure(this.toString(), u.toString());\n }\n\n \n /**\n * Check if this Unifiable can unify with another Object. This\n * should be implemented as a quick check to allow users of the\n * Unifiable to scan a group of Unifications to rapidly see if the\n * entire group is at least possible before descending into each\n * one with a full unification procedure. Thus, if a call to this\n * method does not result in a UnifyFailure exception being\n * thrown, it doesn't mean that the Object can definitely be\n * unified with this Unifiable -- what is important is that when a\n * call to this method throws a UnifyFailure exception, it permits\n * one to avoid calling the unify() method on other Unifiables in\n * a group because the quick check failed on this one.\n *\n * @param o object to check for unifiability\n * @exception UnifyFailure if this Unifiable cannot be unified with \n * the Object\n * @return the Object o, unmodified \n **/\n public void unifyCheck(Object u) throws UnifyFailure {}\n\n\n /**\n * Replaces any variables in this Unifiable with the values found\n * for them in the Substitution argument.\n *\n * @param s Substitution containing the variable resolutions\n * @return a copy of this Unifiable with all variables from the\n * Substitution replaced by their values. \n */\n public Object fill(Substitution s) throws UnifyFailure {\n return this;\n }\n \n /**\n * Returns a hash code using the given map from vars to ints.\n */\n public abstract int hashCode(TObjectIntHashMap varMap);\n\n /**\n * Returns whether this LF equals the given object \n * up to variable names, using the given maps from vars to ints.\n */\n public abstract boolean equals(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2);\n \n /**\n * Returns an XML representation of this LF.\n */\n public abstract Element toXml();\n \n /**\n * Returns a pretty-printed string of this LF, with the given indent.\n */\n public abstract String prettyPrint(String indent);\n}\n" }, { "path": "src/opennlp/ccg/hylo/HyloHelper.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-5 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.grammar.*;\nimport opennlp.ccg.lexicon.Lexicon;\nimport org.jdom.*;\n\nimport java.util.*;\n\nimport gnu.trove.*;\n\n/**\n * A utility class to help with certain global operations over hybrid logic\n * terms.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.43 $, $Date: 2011/08/26 05:18:39 $\n **/\npublic class HyloHelper {\n\n //-----------------------------------------------------------------\n // XML functions\n \n /** \n * Builds a Hylo term from the given element.\n * An \"lf\" element may be used to wrap one or more (implicitly conj-ed) terms.\n */\n public static LF getLF(Element e) {\n LF retval = null;\n String type = e.getName();\n if (type.equals(\"op\")) {\n retval = new Op(e);\n } else if (type.equals(\"var\")) {\n String name = getName(e);\n retval = new HyloVar(prefix(name), type(name));\n } else if (type.equals(\"nomvar\")) {\n String name = getName(e); \n boolean shared = \"true\".equals(e.getAttributeValue(\"shared\"));\n retval = new NominalVar(prefix(name), type(name), shared);\n } else if (type.equals(\"nom\")) {\n String name = getName(e);\n boolean shared = \"true\".equals(e.getAttributeValue(\"shared\"));\n retval = new NominalAtom(prefix(name), type(name), shared);\n } else if (type.equals(\"prop\")) {\n String name = getName(e);\n retval = new Proposition(name, existingType(name));\n } else if (type.equals(\"satop\")) {\n retval = new SatOp(e);\n } else if (type.equals(\"box\") || type.equals(\"b\")) {\n retval = new Box(e);\n } else if (type.equals(\"diamond\") || type.equals(\"d\")) {\n retval = new Diamond(e);\n } else if (type.equals(\"mode\")) {\n String name = getName(e);\n retval = new ModeLabel(name);\n } else if (type.equals(\"modevar\")) {\n String name = getName(e);\n retval = new ModeVar(name);\n } else if (type.equals(\"lf\")) {\n retval = getLF_FromChildren(e);\n } else {\n System.out.println(\"Invalid hybrid logic LF type: \" + type);\n }\n // assign chunks\n if (retval != null) {\n String chunks = e.getAttributeValue(\"chunks\");\n if (chunks != null) {\n retval.setChunks(convertChunks(chunks));\n }\n }\n // done\n return retval;\n }\n\n // returns the value of the attribute 'name' or 'n'\n private static String getName(Element e) { \n String name = e.getAttributeValue(\"name\");\n if (name == null) name = e.getAttributeValue(\"n\");\n return name;\n }\n \n // returns the simple type with the given name, if it exists, or null if not\n private static SimpleType existingType(String name) {\n Types types = Grammar.theGrammar.types;\n if (types.containsSimpleType(name)) return types.getSimpleType(name);\n else return null;\n }\n \n /** Returns the prefix of the name, up to an optional colon. */\n protected static String prefix(String name) {\n int index = name.indexOf(\":\");\n if (index >= 0) return name.substring(0, index);\n else return name;\n }\n\n /** Returns the simple type given by the suffix of the name after the colon, or null if none. */\n protected static SimpleType type(String name) {\n int index = name.indexOf(\":\");\n String suffix = (index >=0 && index+1 < name.length()) ? name.substring(index+1) : null;\n if (suffix != null) return Grammar.theGrammar.types.getSimpleType(suffix);\n else return null;\n }\n \n \n /**\n * Returns a Hylo term from the children of the given element, \n * adding an implicit CONJ op if necessary.\n */\n @SuppressWarnings(\"unchecked\")\n\tpublic static LF getLF_FromChildren(Element e) {\n List children = e.getChildren();\n if (children.size() > 1) {\n List preds = new ArrayList(children.size());\n for (int i=0; i < children.size(); i++) {\n preds.add(getLF(children.get(i)));\n }\n Op conj = new Op(Op.CONJ, preds);\n return conj;\n }\n else return getLF(children.get(0));\n }\n\n /**\n * Returns an XML representation of the given LF, \n * wrapped with an 'lf' element, \n * removing CONJ ops that may be left implicit.\n */\n public static Element toXml(LF lf) {\n Element retval = new Element(\"lf\");\n retval.addContent(lf.toXml());\n removeConjOps(retval);\n return retval;\n }\n\n \n //-----------------------------------------------------------------\n // process chunks\n \n /** \n * Processes and removes any chunk elements.\n * Each chunk element is numbered, and all contained elements are marked \n * as being contained by this chunk, via a \"chunks\" attribute.\n */\n public static void processChunks(Element e) {\n processChunks(e, null, 0);\n removeChunkElts(e);\n }\n \n // recursively processes chunks, threading count through calls\n @SuppressWarnings(\"unchecked\")\n\tprivate static int processChunks(Element e, String chunks, int count) {\n // check for chunk\n if (e.getName().equals(\"chunk\")) {\n // update chunks string and counter\n if (chunks == null) { chunks = \"\" + count; }\n else { chunks += \" \" + count; }\n count++;\n }\n // otherwise add chunks attr, if val non-null\n else if (chunks != null) {\n e.setAttribute(\"chunks\", chunks);\n }\n // do children\n List children = e.getChildren();\n for (int i=0; i < children.size(); i++) {\n count = processChunks(children.get(i), chunks, count);\n }\n // return current count\n return count;\n }\n \n // converts chunk strings\n private static TIntArrayList convertChunks(String chunks) {\n String[] tokens = chunks.split(\"\\\\s+\");\n TIntArrayList retval = new TIntArrayList(tokens.length);\n for (int i = 0; i < tokens.length; i++) {\n retval.add(Integer.parseInt(tokens[i]));\n }\n return retval;\n }\n \n //-----------------------------------------------------------------\n // recursively remove certain elements\n \n private static abstract class ElementTest {\n abstract boolean test(Element elt);\n }\n\n // recursively removes elements meeting given test\n @SuppressWarnings(\"unchecked\")\n\tprivate static void removeElts(Element elt, ElementTest eltTest) {\n // nb: need to dump children into a new list, in order to get a list iterator \n // that will allow multiple adds\n List children = elt.getChildren();\n List newChildren = new ArrayList(children.size());\n newChildren.addAll(children);\n for (ListIterator li = newChildren.listIterator(); li.hasNext(); ) {\n Element nextElt = li.next();\n removeElts(nextElt, eltTest);\n if (eltTest.test(nextElt)) {\n li.remove();\n for (Iterator it = nextElt.getChildren().iterator(); it.hasNext(); ) {\n Element childElt = it.next();\n it.remove(); // removes childElt from nextElt's children, so it can become a child of elt\n li.add(childElt);\n }\n }\n }\n elt.removeContent(); \n elt.setContent(newChildren); \n }\n \n // recursively removes conj ops\n private static void removeConjOps(Element lfElt) {\n removeElts(\n lfElt, \n new ElementTest() {\n boolean test(Element elt) {\n return elt.getName().equals(\"op\") && \n elt.getAttributeValue(\"name\").equals(Op.CONJ);\n }\n }\n );\n }\n \n // recursively removes chunk elements\n private static void removeChunkElts(Element lfElt) {\n removeElts(\n lfElt, \n new ElementTest() {\n boolean test(Element elt) {\n return elt.getName().equals(\"chunk\");\n }\n }\n );\n }\n \n\n //-----------------------------------------------------------------\n // functions for elementary predications\n\n /**\n * Returns whether the given LF is an elementary predication, \n * ie a lexical predication, relation predication or attribute-value predication.\n */\n public static boolean isElementaryPredication(LF lf) {\n return isLexPred(lf) || isRelPred(lf) || isAttrPred(lf);\n }\n \n /**\n * Returns whether the given elementary predication is a lexical predication, \n * ie one of the form @x(prop).\n */\n public static boolean isLexPred(LF pred) {\n if (!(pred instanceof SatOp)) return false;\n SatOp satOp = (SatOp) pred;\n LF arg = satOp.getArg();\n return (arg instanceof Proposition);\n }\n\n /**\n * Returns whether the given elementary predication is a relation predication, \n * ie one of the form @x(<Rel>y).\n */\n public static boolean isRelPred(LF pred) {\n if (!(pred instanceof SatOp)) return false;\n SatOp satOp = (SatOp) pred;\n LF arg = satOp.getArg();\n if (!(arg instanceof Diamond)) return false;\n Diamond d = (Diamond) arg;\n return (d.getArg() instanceof Nominal);\n }\n\n /**\n * Returns whether the given elementary predication is an attribute-value predication, \n * ie one of the form @x(<Rel>prop). Note that the prop is also allowed to be \n * a HyloVar.\n */\n public static boolean isAttrPred(LF pred) {\n if (!(pred instanceof SatOp)) return false;\n SatOp satOp = (SatOp) pred;\n LF arg = satOp.getArg();\n return isAttr(arg);\n }\n \n /**\n * Returns whether the given arg is an attribute-value pair, \n * ie one of the form <Rel>prop. Note that the prop is also allowed to be \n * a HyloVar.\n */\n public static boolean isAttr(LF arg) {\n if (!(arg instanceof Diamond)) return false;\n Diamond d = (Diamond) arg;\n LF dArg = d.getArg();\n return ( dArg instanceof Proposition || \n (dArg instanceof HyloVar && !(dArg instanceof NominalVar)) );\n }\n \n /**\n * Returns the name of the lexical predicate of the given elementary predication, \n * or null, if the given LF is not a lexical predicate.\n */\n public static String getLexPred(LF lf) {\n if (!isLexPred(lf)) return null;\n LF arg = ((SatOp)lf).getArg();\n return ((Proposition)arg).toString();\n }\n \n /**\n * Returns the name of the relation of the given elementary predication, \n * or null, if the given LF is not a relation or attribute-value predicate.\n */\n public static String getRel(LF lf) {\n if (!isRelPred(lf) && !isAttrPred(lf)) return null;\n LF arg = ((SatOp)lf).getArg();\n return ((Diamond)arg).getMode().toString();\n }\n \n /**\n * Returns the string value of the attribute-value predicate, or \n * null if the given LF is not an attribute-value predicate or has no value. \n */\n public static String getVal(LF lf) {\n \tif (!isAttrPred(lf)) return null;\n LF arg = ((SatOp)lf).getArg();\n LF dArg = ((Diamond)arg).getArg();\n if (dArg instanceof Proposition) return ((Proposition)dArg).getName();\n return null;\n }\n \n /**\n * Returns the principal nominal the given elementary predication, \n * or null, if the given LF is not an elementary predication.\n */\n public static Nominal getPrincipalNominal(LF lf) {\n if (!isElementaryPredication(lf)) return null;\n return ((SatOp)lf).getNominal();\n }\n\n /**\n * Returns the secondary nominal of the given elementary predication, \n * or null, if the given LF is not a relation predication.\n */\n public static Nominal getSecondaryNominal(LF lf) {\n if (!isRelPred(lf)) return null;\n LF arg = ((SatOp)lf).getArg();\n return (Nominal) ((Diamond)arg).getArg(); \n }\n\n \n //-----------------------------------------------------------------\n // flattening \n\n /**\n * Returns a flattened, sorted list of elementary preds from the given LF \n * as a conjunction op, or as a single LF, if there is only one. \n * LF chunks are preserved on satops, as are alts (exclusive disjunctions) \n * and opts (optional parts).\n * A runtime exception is thrown if the LF cannot be flattened.\n */\n @SuppressWarnings(\"unchecked\")\n\tpublic static LF flattenLF(LF lf) {\n List preds = flatten(lf);\n if (preds.size() == 1) {\n return (LF) preds.get(0);\n }\n else {\n \treturn new Op(Op.CONJ, (List)preds);\n }\n }\n \n /**\n * Returns a list of predications from the given LF, which is assumed to be either \n * a conjunction of elementary predications or a single elementary predication.\n */\n public static List getPreds(LF lf) {\n if (lf instanceof Op && ((Op)lf).getName().equals(Op.CONJ)) {\n List args = ((Op)lf).getArguments();\n List retval = new ArrayList(args.size());\n for (LF arg : args) retval.add((SatOp)arg);\n return retval;\n }\n else { \n List retval = new ArrayList(1);\n retval.add((SatOp)lf);\n return retval;\n }\n }\n \n /**\n * Returns the first elementary predication from the given LF, which is assumed to be either \n * a conjunction of elementary predications or a single elementary predication; \n * otherwise returns null.\n */\n public static SatOp getFirstPred(LF lf) {\n \tif (lf instanceof SatOp) return (SatOp) lf;\n if (lf instanceof Op && ((Op)lf).getName().equals(Op.CONJ)) {\n List args = ((Op)lf).getArguments();\n return (SatOp) args.get(0);\n }\n return null;\n }\n \n /**\n * Returns a flattened, sorted list of elementary preds from the given LF \n * as a list.\n * LF chunks are preserved on satops, as are alts (exclusive disjunctions) \n * and opts (optional parts).\n * Chunks, alts and opts are propagated through shared nominals.\n * A runtime exception is thrown if the LF cannot be flattened.\n */\n public static List flatten(LF lf) { \n List retval = new Flattener().flatten(lf);\n sort(retval);\n return retval;\n }\n \n /**\n * Returns the first elementary predication in the flattened LF.\n * A runtime exception is thrown if the LF cannot be flattened.\n */\n public static LF firstEP(LF lf) { \n List preds = new Flattener().flatten(lf);\n return preds.get(0);\n }\n \n /**\n * Sets the origin of the elementary preds in the given LF (if any).\n */\n public static void setOrigin(LF lf, LexSemOrigin origin) {\n \tif (lf == null) return;\n \tif (lf instanceof SatOp) ((SatOp)lf).setOrigin(origin);\n \telse if (lf instanceof Op && ((Op)lf).getName().equals(Op.CONJ)) {\n List args = ((Op)lf).getArguments();\n for (LF arg : args) {\n \tif (arg instanceof SatOp) ((SatOp)arg).setOrigin(origin);\n }\n }\n\n }\n \n /** \n * Returns a map from nominals to index positions for the first EP for\n * that nominal in a sorted list of elementary predications. \n */\n public static Map nomIndex(List preds) {\n \tHashMap retval = new HashMap(preds.size()/2);\n \tfor (int i=0; i < preds.size(); i++) {\n \t\tSatOp pred = preds.get(i);\n \t\tNominal nom = pred._nominal;\n \t\tif (!retval.containsKey(nom)) retval.put(nom, i);\n \t}\n \treturn retval;\n }\n \n /**\n * Returns whether a nominal is a root in the list of EPs using a linear search.\n */\n public static boolean isRoot(Nominal nom, List preds) {\n \tfor (SatOp pred : preds) {\n \t\tNominal child = getSecondaryNominal(pred);\n \t\tif (child != null && child.equals(nom)) return false;\n \t}\n \treturn true;\n }\n \n \n //-----------------------------------------------------------------\n // lexical dependencies \n\n /** Returns the unfilled lexical dependencies for a lexical item's LF. */\n public static List getUnfilledLexDeps(LF lf) {\n \tif (lf == null) return Collections.emptyList();\n \treturn LexDependency.unfilledLexDeps(getPreds(lf));\n }\n \n\t/**\n\t * Returns the filled lexical dependencies from those in the unfilled list \n\t * by checking the sign's LF for ones that have become filled, removing the \n\t * corresponding no longer unfilled deps. \n\t */\n\tpublic static List getFilledLexDeps(List unfilled, LF lf) {\n \tif (lf == null) return Collections.emptyList();\n \treturn LexDependency.filledLexDeps(unfilled, getPreds(lf));\n }\n\n\t/**\n\t * Returns the semantic features (attribute-value preds) for the given nominal \n\t * in the given LF. \n\t */\n\tpublic static List getSemFeatsForHead(Nominal nominal, LF lf) {\n\t\tif (nominal == null || lf == null) return Collections.emptyList();\n\t\tList retval = new ArrayList(3);\n\t\tfor (SatOp pred : getPreds(lf)) {\n\t\t\tif (nominal.equals(pred._nominal) && isAttrPred(pred))\n\t\t\t\tretval.add(pred);\n\t\t}\n\t\treturn retval;\n\t}\n\t\n\t\n //-----------------------------------------------------------------\n // compacting \n \n /** Composes compact and convertNominals. */\n public static LF compactAndConvertNominals(LF lf, Nominal root) {\n LF retval = compact(lf, root);\n convertNominals(retval);\n return retval;\n }\n \n /** Composes compact and convertNominals with a root sign, for conversion using word positions. */\n public static LF compactAndConvertNominals(LF lf, Nominal root, Sign rootSign) {\n root = convertNominals(lf, rootSign, root);\n LF retval = compact(lf, root);\n return retval;\n }\n \n /**\n * Returns a compacted LF from the given flattened one. \n * A root nominal may also be given (otherwise null). \n * Nominals with multiple parents are kept separate.\n * If there are any duplicate predications, an attempt \n * is made to attach them in different locations.\n */\n public static LF compact(LF lf, Nominal root) {\n \treturn Compacter.compact(lf, root);\n }\n \n \n //-----------------------------------------------------------------\n // convert nominals\n \n /** Converts nominal vars to atoms, renaming them based on lexical propositions. */\n public static void convertNominals(LF lf) {\n \tConverter.convertNominals(lf);\n }\n\n\t/**\n\t * Converts nominal vars to atoms, renaming them based on word position, if \n\t * a root sign is given, otherwise using lexical propositions; \n\t * returns the converted nominal root. \n\t */\n\tpublic static Nominal convertNominals(LF lf, Sign root, Nominal nominalRoot) {\n\t\treturn Converter.convertNominals(lf, root, nominalRoot);\n\t}\n\t\n\t/**\n\t * Converts nominal atoms back to vars, returning the converted nominal root. \n\t * The LF is assumed to be flattened to elementary predications.\n\t */\n\tpublic static Nominal convertNominalsToVars(LF lf, Nominal nominalRoot) {\n\t\treturn Converter.convertNominalsToVars(getPreds(lf), nominalRoot);\n\t}\n\t\n\n //-----------------------------------------------------------------\n // append \n\n /**\n * Returns a the conjunction of the two LFs, either \n * as a conjunction op, or as a single LF, if one is null.\n * If either LF is itself a conj op, its elements are appended \n * instead of the conj op itself.\n * If both LFs are null, null is returned.\n */\n public static LF append(LF lf1, LF lf2) {\n \n // set up new list\n int size = 0;\n List args1 = null;\n if (lf1 instanceof Op && ((Op)lf1).getName().equals(Op.CONJ)) {\n args1 = ((Op)lf1).getArguments();\n size += args1.size();\n } else if (lf1 != null) {\n size++;\n } \n List args2 = null;\n if (lf2 instanceof Op && ((Op)lf2).getName().equals(Op.CONJ)) {\n args2 = ((Op)lf2).getArguments();\n size += args2.size();\n } else if (lf2 != null) {\n size++;\n }\n List combined = new ArrayList(size);\n \n // add to new list\n if (args1 != null) { \n combined.addAll(args1);\n } else if (lf1 != null) {\n combined.add(lf1);\n }\n if (args2 != null) { \n combined.addAll(args2);\n } else if (lf2 != null) {\n combined.add(lf2);\n }\n \n // return\n if (combined.isEmpty()) { return null; }\n else if (combined.size() == 1) { return combined.get(0); }\n else { return new Op(Op.CONJ, combined); }\n }\n\n \n //-----------------------------------------------------------------\n // sort \n\n /**\n * Sorts the list of elementary predications in a conj op, \n * or does nothing if the LF is not a conj op.\n */\n public static void sort(LF lf) {\n if (lf instanceof Op && ((Op)lf).getName().equals(Op.CONJ)) {\n sort(((Op)lf).getArguments());\n }\n }\n \n /**\n * Sorts a list of elementary predications.\n */\n public static void sort(List preds) {\n Collections.sort(preds, predComparator);\n }\n\n // compares elementary predications\n private static final Comparator predComparator = new Comparator() {\n public int compare(LF lf1, LF lf2){\n // sort first on principal nominal\n int nomCompare = getPrincipalNominal(lf1).compareTo(getPrincipalNominal(lf2));\n if (nomCompare != 0) return nomCompare;\n // sort next on type of elementary predication\n int typeCompare = epType(lf1).compareTo(epType(lf2));\n if (typeCompare != 0) return typeCompare;\n // then on lex pred\n if (isLexPred(lf1)) {\n return getLexPred(lf1).compareToIgnoreCase(getLexPred(lf2));\n }\n // then rels\n String rel1 = getRel(lf1);\n String rel2 = getRel(lf2);\n Lexicon theLexicon = Grammar.theGrammar.lexicon;\n Integer rel1Index = theLexicon.getRelationSortIndex(rel1);\n Integer rel2Index = theLexicon.getRelationSortIndex(rel2);\n int relIndexCompare = rel1Index.compareTo(rel2Index);\n if (relIndexCompare != 0) return relIndexCompare;\n int relCompare = rel1.compareToIgnoreCase(rel2);\n if (relCompare != 0) return relCompare;\n // then secondary nominal\n if (isRelPred(lf1)) {\n return getSecondaryNominal(lf1).compareTo(getSecondaryNominal(lf2));\n }\n // otherwise 0\n return 0;\n }\n };\n \n // order of elementary predication type\n private static Integer epType(LF lf) {\n if (isLexPred(lf)) return LEX_PRED;\n else if (isAttrPred(lf)) return ATTR_PRED;\n else if (isRelPred(lf)) return REL_PRED;\n // shouldn't happen\n else return null;\n }\n \n private static Integer LEX_PRED = new Integer(1);\n private static Integer ATTR_PRED = new Integer(2);\n private static Integer REL_PRED = new Integer(3);\n\n \n //-----------------------------------------------------------------\n // check\n\n /**\n * Checks the list of elementary predications in a conj op \n * for well-formedness, or does nothing if the LF is not a conj op.\n * A UnifyFailure exception is thrown if the check fails.\n * The only current check is that there is no more than one lexical \n * predication per nominal. \n * The list of predications is assumed to be already sorted.\n */\n public static void check(LF lf) throws UnifyFailure {\n if (lf instanceof Op && ((Op)lf).getName().equals(Op.CONJ)) {\n check(((Op)lf).getArguments());\n }\n }\n \n private static void check(List preds) throws UnifyFailure {\n for (int i = 0; i < preds.size()-1; i++) {\n LF lf1 = preds.get(i);\n LF lf2 = preds.get(i+1);\n if (isLexPred(lf1) && isLexPred(lf2) &&\n getPrincipalNominal(lf1).equals(getPrincipalNominal(lf2))) \n {\n throw new UnifyFailure();\n }\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/HyloVar.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-4 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.grammar.*;\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\nimport org.jdom.*;\nimport gnu.trove.*;\n\n/**\n * A class for objects which can stand for any HyloFormula object.\n * Types are unified with other hylo vars, and with other hylo formulas when present.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.17 $, $Date: 2009/07/17 04:23:30 $\n **/\npublic class HyloVar extends HyloFormula implements Variable, Indexed {\n \n\tprivate static final long serialVersionUID = 3455577234911944031L;\n\t\n\tprotected final String _name;\n protected int _index;\n protected int _hashCode;\n protected SimpleType type;\n \n public HyloVar(String name) {\n this(name, 0, null);\n }\n\n public HyloVar(String name, SimpleType st) {\n this(name, 0, st);\n }\n\n protected HyloVar(String name, int index, SimpleType st) {\n _name = name;\n _index = index;\n type = (st != null) ? st : Grammar.theGrammar.types.getSimpleType(Types.TOP_TYPE);\n _hashCode = _name.hashCode() + _index + type.getIndex();\n }\n \n public String name() {\n return _name;\n }\n\n public LF copy() {\n return new HyloVar(_name, _index, type);\n }\n\n\n public int getIndex() {\n return _index;\n }\n\n public void setIndex(int index) {\n _hashCode += index - _index;\n _index = index;\n }\n\n public SimpleType getType() {\n return type;\n }\n \n public boolean occurs(Variable var) {\n return equals(var);\n }\n\n public boolean equals(Object o) {\n if (this == o) return true;\n if (!(o instanceof HyloVar)) return false;\n HyloVar var = (HyloVar) o;\n return _index == var._index && _name.equals(var._name) && type.equals(var.type);\n }\n \n public int compareTo(HyloVar hv) {\n int retval = _name.compareTo(hv._name);\n if (retval == 0) {\n if (_index < hv._index) { retval = -1; }\n else if (_index > hv._index) { retval = 1; }\n }\n return retval;\n }\n \n public Object unify(Object u, Substitution sub) throws UnifyFailure {\n // with nominal vars, reverse direction of unification\n if (u instanceof NominalVar) return ((NominalVar)u).unify(this, sub);\n // check for equality with u\n if (equals(u)) return this; \n // make sure u is an LF\n if (!(u instanceof LF)) throw new UnifyFailure();\n LF lf = (LF) u;\n // check type compatibility, if present\n SimpleType st = null;\n if (lf.getType() != null) st = (SimpleType) type.unify(lf.getType(), sub);\n // with hylo vars, substitute according to type specificity then comparison order, \n // so that the direction of unification doesn't matter\n if (u instanceof HyloVar) {\n HyloVar u_hv = (HyloVar)u;\n // equal types, use comparison order\n if (type.equals(u_hv.getType())) {\n if (compareTo(u_hv) >= 0) return sub.makeSubstitution(this, u_hv); \n else return sub.makeSubstitution(u_hv, this);\n }\n // unequal types, use most specific one\n if (type.equals(st)) return sub.makeSubstitution(u_hv, this);\n if (u_hv.getType().equals(st)) return sub.makeSubstitution(this, u_hv); \n // otherwise make new hylo var with intersection type, \n // name based on comparison order and index, and new index\n String name = (compareTo(u_hv) >= 0) ? (u_hv._name + u_hv._index) : (_name + this._index);\n HyloVar hv_st = new HyloVar(name, UnifyControl.getUniqueVarIndex(), st);\n // and subst both\n sub.makeSubstitution(u_hv, hv_st);\n return sub.makeSubstitution(this, hv_st); \n }\n // with props, check for more specific type\n if (u instanceof Proposition) {\n Proposition prop = (Proposition) u;\n // if no or same type, just subst\n if (st == null || prop.getType().equals(st)) return sub.makeSubstitution(this, prop);\n // otherwise subst prop with name of type\n Proposition prop_st = new Proposition(st.getName(), st); \n return sub.makeSubstitution(this, prop_st);\n }\n // otherwise, do occurs check ... \n if (((LF)u).occurs(this)) throw new UnifyFailure(); \n // and then go ahead and substitute\n return sub.makeSubstitution(this, u);\n }\n\n public Object fill(Substitution sub) throws UnifyFailure {\n Object val = sub.getValue(this);\n if (val != null) {\n return val;\n } else {\n return this;\n }\n }\n\n public String toString() { \n String retval = _name+\"_\"+_index;\n if (!type.getName().equals(Types.TOP_TYPE)) retval += \":\" + type.getName();\n return retval;\n }\n\n /** Returns the name with the type separated by a colon if the type is not the top type. */\n public String nameWithType() { \n String retval = _name;\n if (!type.getName().equals(Types.TOP_TYPE)) retval += \":\" + type.getName();\n return retval;\n }\n\n /**\n * Returns a pretty-printed string of this LF, with the given indent.\n */\n public String prettyPrint(String indent) {\n return toString();\n }\n \n /** Returns a hash code based on the name, index and type. */\n public int hashCode() {\n return _hashCode;\n }\n \n /**\n\t * Returns a hash code using the given map from vars to ints.\n\t */\n\tpublic int hashCode(TObjectIntHashMap varMap) {\n\t\t// see if this already in map\n\t\tif (varMap.containsKey(this))\n\t\t\treturn varMap.get(this);\n\t\t// otherwise add it\n\t\tint next = varMap.size() + 1;\n\t\tvarMap.put(this, next);\n\t\treturn next;\n\t}\n \n /**\n\t * Returns whether this var equals the given object up to variable names,\n\t * using the given maps from vars to ints.\n\t * (Note that the name and index may differ, but the types must be equal.)\n\t */\n public boolean equals(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2) {\n if (obj.getClass() != this.getClass()) { return false; }\n HyloVar hv = (HyloVar) obj;\n if (varMap.get(this) != varMap2.get(hv)) return false;\n if (!this.type.equals(hv.type)) return false;\n return true;\n }\n \n /**\n * Returns an XML representation of this LF.\n */\n public Element toXml() {\n Element retval = new Element(\"var\");\n retval.setAttribute(\"name\", nameWithType());\n return retval;\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/LexDepFeatureExtractor.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2011-3 Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport java.util.*;\n\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.perceptron.*;\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.util.TrieMap;\n\n/** \n * A class for extracting lexical dependency features. This class\n * implements the features in White and Rajkumar's EMNLP-12 paper \n * \n * Minimal Dependency Length in Realization Ranking.\n * \n * Features are extracted lazily for efficiency, using a prefix of \"ld\" \n * for \"lexical dependency\". \n * \n * Features potentially involve the word, POS tag, word class, definiteness, syntactic \n * complexity, short-long order, and total dependency length, where the latter \n * three can be controlled by the appropriate include* flags; of\n * these, only dependency length is included by default.\n *\n * Word class is based on the semantic class, a check for color terms, common suffixes, \n * and the presence of a hyphen or capitalization.\n * \n * The checks for definite NPs, color terms and common suffixes are done by methods with \n * defaults for the English CCGbank, which can be overridden in subclasses or \n * reconfigured in the case of color terms and suffixes.\n * \n * @author Michael White\n * @version $Revision: 1.4 $, $Date: 2011/11/10 15:12:53 $\n */ \npublic class LexDepFeatureExtractor implements FeatureExtractor {\n\n\t/** Feature prefix constant: \"ld\". */\n\tpublic static final String PREFIX = \"ld\";\n\t\n\t/** Head precedes dep constant: \"hpd\". */\n\tpublic static final String HEAD_PRECEDES_DEP = \"hpd\";\n\t\n\t/** Dep precedes head constant: \"dph\". */\n\tpublic static final String DEP_PRECEDES_HEAD = \"dph\";\n\t\n\t/** Left of head sibs precedence constant: \"lsp\". */\n\tpublic static final String LEFT_SIBS_PRECEDENCE = \"lsp\";\n\t\n\t/** Right of head sibs precedence constant: \"rsp\". */\n\tpublic static final String RIGHT_SIBS_PRECEDENCE = \"rsp\";\n\t\n\t/** Returns the appropriate siblings precedence constant for the head-dep order constant. */\n\tpublic static String sibPrecedenceForDep(String depConst) {\n\t\treturn (depConst == DEP_PRECEDES_HEAD) ? LEFT_SIBS_PRECEDENCE : RIGHT_SIBS_PRECEDENCE;\n\t}\n\t\n\t/** Definiteness constant: \"def1\". */\n\tpublic static final String DEF = \"def1\";\n\t\n\t/** Indefiniteness constant: \"def0\". */\n\tpublic static final String INDEF = \"def0\";\n\t\n\t/** Class constant PRO. */\n\tpublic static final String CLASS_PRO = \"PRO\";\n\t\n\t/** Class constant COLOR. */\n\tpublic static final String CLASS_COLOR = \"COLOR\";\n\t\n\t/** Class constant HYPH. */\n\tpublic static final String CLASS_HYPH = \"HYPH\";\n\t\n\t/** Class constant CAP. */\n\tpublic static final String CLASS_CAP = \"CAP\";\n\t\n\t/** Class constant NIL. */\n\tpublic static final String CLASS_NIL = \"NIL\";\n\t\n\t/** Flag for whether to include syntactic complexity ordering features (defaults to false). */\n\tpublic boolean includeComplexityFeats = false;\n\t\n\t/** Complexity ordering constant for verb presence: \"1v\". */\n\tpublic static final String HAS_V = \"1v\";\n\t\n\t/** Complexity ordering constant for verb presence: \"0v\". */\n\tpublic static final String NO_V = \"0v\";\n\t\n\t/** Complexity ordering constant for punct presence: \"1p\". */\n\tpublic static final String HAS_P = \"1p\";\n\t\n\t/** Complexity ordering constant for punct absence: \"0p\". */\n\tpublic static final String NO_P = \"0p\";\n\t\n\t/** Flag for whether to include short-long features (defaults to false). */\n\tpublic boolean includeShortLong = false;\n\n\t/** Short-long order constant: \"sl\". */\n\tpublic static final String SHORT_LONG_ORDER = \"sl\";\n\t\n\t/** Long-short order constant: \"ls\". */\n\tpublic static final String LONG_SHORT_ORDER = \"ls\";\n\t\n\t/** Flag for whether to include global dependency length features (defaults to true). */\n\tpublic boolean includeDepLen = true;\n\n\t/** Global dependency length feature constant: \"$deplen\". */\n\tpublic static final String DEPLEN = \"$deplen\";\n\t\n\t/** Conditional lazy extractor, for lazily extracting a feature subject to a test. */\n\tpublic static abstract class ConditionalLazyExtractor {\n\t\tabstract boolean test();\n\t\tList> lazyExtractor = new ArrayList>(5);\n\t}\n\t\n\t/** Conditional lazy evaluator, for lazily extracting a feature and its value, subject to a test. */\n\tpublic static abstract class ConditionalLazyEvaluator extends ConditionalLazyExtractor {\n\t\tabstract float eval();\n\t}\n\t\n\t/** Feature map wrapper, for unique retrieval from a sign's data objects. */\n\tpublic static class FeatureMapWrapper {\n\t\tpublic FeatureMap featureMap;\n\t\tpublic FeatureMapWrapper(FeatureMap featureMap) { this.featureMap = featureMap; }\n\t}\n\t\n\t/** The alphabet. */\n\tprotected Alphabet alphabet = null;\n\n\t/** Current feature map. */\n\tprotected FeatureMap currentMap = null;\n\t\n\t/** Current sign (for extracting features). */\n\tprotected Sign currentSign = null;\n\t\n\t/** Current input signs (for extracting features). */\n\tprotected Sign[] currentInputs = null;\n\t\n\t/** Current dependency (for extracting features). */\n\tprotected LexDependency currentDep = null;\n\n\t/** Current head index. */\n\tprotected int currentHeadIndex = -1;\n\t\n\t/** Current dependent index. */\n\tprotected int currentDepIndex = -1;\n\t\n\t/** Current sibling dependency (for extracting features). */\n\tprotected LexDependency currentSib = null;\n\t\n\t/** Current sibling dependent index. */\n\tprotected int currentSibIndex = -1;\n\t\n\t/** Current head broad POS (for extracting features). */\n\tprotected String currentHeadBroadPOS = null;\n\n\t/** Current head-dependent order (for extracting features). */\n\tprotected String currentHeadDepOrder = null;\n\n\t/** Current siblings precedence relation (for extracting features). */\n\tprotected String currentSibsPrecedence = null;\n\n\t/** Current dep prececes sib flag (for extracting features). */\n\tprotected boolean currentDepPrecedesSib = false;\n\n\t/** Current dep sign (for extracting features). */\n\tprotected Sign currentDepSign = null;\n\t\n\t/** Current sib sign (for extracting features). */\n\tprotected Sign currentSibSign = null;\n\t\n\t/** Current dep phrase lengths (for extracting features). */\n\tprotected PhraseLengths currentDepLengths = null;\n\t\n\t/** Current sib phrase lengths (for extracting features). */\n\tprotected PhraseLengths currentSibLengths = null;\n\t\n\t/** Current difference in lengths between second and first siblings (for extracting features). */\n\tprotected PhraseLengths currentLengthsDiff = null;\n\t\n\t///** Lexical feature extractors. */\n\t//protected List>> lexExtractors = new ArrayList>>();\n\t\n\t/** Dependency order feature extractors. */\n\tprotected List>> depOrderExtractors = new ArrayList>>();\n\t\n\t/** Siblings order feature extractors. */\n\tprotected List>> sibsOrderExtractors = new ArrayList>>();\n\t\n\t/** Siblings order conditional feature extractors. */\n\tprotected List sibsOrderCondExtractors = new ArrayList();\n\t\n\t/** Siblings complexity order conditional feature extractors. */\n\tprotected List sibsComplexityOrderCondExtractors = new ArrayList();\n\t\n\t/** Short-long conditional feature evaluators. */\n\tprotected List shortLongCondEvaluators = new ArrayList();\n\t\n\t/** Global dependency length keys. */\n\tprotected List depLenKeys = new ArrayList(1);\n\t\n\t/** Global dependency length feature. */\n\tprotected Alphabet.Feature depLenFeat = null;\n\t\n\t/** Constructor. */\n\tpublic LexDepFeatureExtractor() {\n\t\t// init lazy feature extractors\n\t\tdepOrderExtractors.add(head_dep_order_words());\n\t\tdepOrderExtractors.add(head_dep_order_pos());\n\t\tdepOrderExtractors.add(head_dep_order_word_pos());\n\t\tdepOrderExtractors.add(head_dep_order_pos_word());\n\t\tsibsOrderExtractors.add(sibs_precedence_words());\n\t\tsibsOrderExtractors.add(sibs_precedence_word_pos());\n\t\tsibsOrderExtractors.add(sibs_precedence_pos_word());\n\t\tsibsOrderCondExtractors.add(sibs_precedence_pos());\n\t\tsibsOrderExtractors.add(sibs_precedence_word_class());\n\t\tsibsOrderExtractors.add(sibs_precedence_class_word());\n\t\tsibsOrderCondExtractors.add(sibs_precedence_class());\n\t\tsibsOrderCondExtractors.add(sibs_precedence_rels());\n\t\tsibsOrderCondExtractors.add(sibs_precedence_defs());\n\t\t// init complexity feature extractors\n\t\tsibsComplexityOrderCondExtractors.add(sibs_precedence_verbs());\n\t\tsibsComplexityOrderCondExtractors.add(sibs_precedence_puncts());\n\t\t// init short-long feature evaluators\n\t\tshortLongCondEvaluators.add(short_long_words());\n\t\t// init dep len keys\n\t\tdepLenKeys.add(DEPLEN);\n\t}\n\t\n\t/** Sets the alphabet. */\n\tpublic void setAlphabet(Alphabet alphabet) {\n\t\tthis.alphabet = alphabet;\n\t}\n\t\n\t\n\t/** Returns the features for the given sign and completeness flag. */\n\tpublic FeatureVector extractFeatures(Sign sign, boolean complete) {\n\t\taddFeatures(sign, complete);\n\t\treturn getFeatureMap(sign);\n\t}\n\t\n\t/** Recursively adds features to the feature map for the given sign, if not already present. */\n\tprotected void addFeatures(Sign sign, boolean complete) {\n\t\t// check for existing map, otherwise make one\n\t\tif (getFeatureMap(sign) != null) return;\n\t\t// lex case\n\t\tif (sign.isLexical()) {\n\t\t\tcurrentSign = sign;\n\t\t\tcurrentMap = new FeatureMap(0);\n\t\t\t//inc(lexExtractors);\n\t\t}\n\t\t// non-terminal\n\t\telse {\n\t\t\tSign[] inputs = sign.getDerivationHistory().getInputs();\n\t\t\t// first recurse\n\t\t\tfor (Sign child : inputs) addFeatures(child, false);\n\t\t\t// use input maps in making current map\n\t\t\tcurrentSign = sign;\n\t\t\tcurrentInputs = inputs;\n\t\t\tif (inputs.length == 1) {\n\t\t\t\tcurrentMap = new FeatureMap(getFeatureMap(inputs[0]));\n\t\t\t}\n\t\t\telse if (inputs.length == 2) {\n\t\t\t\tcurrentMap = new FeatureMap(getFeatureMap(inputs[0]), getFeatureMap(inputs[1]));\n\t\t\t}\n\t\t\t// do each newly filled dep\n\t\t\tfor (LexDependency dep : sign.getFilledDeps()) {\n\t\t\t\tcurrentDep = dep;\n\t\t\t\tcurrentHeadBroadPOS = getHeadBroadPOS(dep);\n\t\t\t\tsetDepIndexes(dep);\n\t\t\t\tcurrentHeadDepOrder = getHeadDepOrder();\n\t\t\t\tinc(depOrderExtractors);\n\t\t\t\t// do dep len\n\t\t\t\tif (includeDepLen) {\n\t\t\t\t\tAlphabet.Feature f = getDepLenFeat();\n\t\t\t\t\tif (f != null) currentMap.add(f, (float)depLen());\t\t\t\t\t\n\t\t\t\t}\n\t\t\t\t// do order for each sib on the same side of the same head\n\t\t\t\tfor (LexDependency sib : sign.getSiblingFilledDeps()) {\n\t\t\t\t\tcurrentSib = sib;\n\t\t\t\t\tif (dep.lexHead != sib.lexHead) continue;\n\t\t\t\t\tsetSibIndex(sib);\n\t\t\t\t\tif (currentHeadDepOrder != getHeadSibOrder()) continue;\n\t\t\t\t\tcurrentSibsPrecedence = sibPrecedenceForDep(currentHeadDepOrder);\n\t\t\t\t\tcurrentDepPrecedesSib = depPrecedesSib();\n\t\t\t\t\tinc(sibsOrderExtractors);\n\t\t\t\t\tincCond(sibsOrderCondExtractors);\n\t\t\t\t\t// do complexity, length feats\n\t\t\t\t\tif (includeComplexityFeats || includeShortLong) {\n\t\t\t\t\t\tsetLengthsDiff();\n\t\t\t\t\t\tif (currentLengthsDiff != null) {\n\t\t\t\t\t\t\tif (includeComplexityFeats) incCond(sibsComplexityOrderCondExtractors);\n\t\t\t\t\t\t\tif (includeShortLong) addCond(shortLongCondEvaluators);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\t// store it\n\t\tstoreFeatureMap(sign);\n\t}\n\t\n\t/** Stores the current feature map as a data object in the given sign. */\n\tprotected void storeFeatureMap(Sign sign) {\n\t\tsign.addData(new FeatureMapWrapper(currentMap));\n\t}\n\t\n\t/** Returns the feature map for this extractor from the given sign (null if none). */\n\tprotected FeatureMap getFeatureMap(Sign sign) {\n\t\tFeatureMapWrapper fmw = (FeatureMapWrapper)sign.getData(FeatureMapWrapper.class);\n\t\treturn (fmw != null) ? fmw.featureMap : null;\n\t}\n\t\n\t/**\n\t * Increments the count of the given features, if relevant.\n\t */\n\tprotected void inc(List>> extractors) {\n\t\tfor (List> lazyExtractor : extractors) {\n\t\t\tAlphabet.Feature f = alphabet.indexLazy(lazyExtractor);\n\t\t\tif (f != null) currentMap.inc(f);\n\t\t}\n\t}\n\n\t/**\n\t * Increments the count of the given conditional features, if relevant.\n\t */\n\tprotected void incCond(List condExtractors) {\n\t\tfor (ConditionalLazyExtractor condExtractor : condExtractors) {\n\t\t\tif (condExtractor.test()) {\n\t\t\t\tAlphabet.Feature f = alphabet.indexLazy(condExtractor.lazyExtractor);\n\t\t\t\tif (f != null) currentMap.inc(f);\n\t\t\t}\n\t\t}\n\t}\n\n\t/**\n\t * Adds to the values of the given conditional features, if relevant.\n\t */\n\tprotected void addCond(List condEvaluators) {\n\t\tfor (ConditionalLazyEvaluator condEvaluator : condEvaluators) {\n\t\t\tif (condEvaluator.test()) {\n\t\t\t\tAlphabet.Feature f = alphabet.indexLazy(condEvaluator.lazyExtractor);\n\t\t\t\tif (f != null) currentMap.add(f, condEvaluator.eval());\n\t\t\t}\n\t\t}\n\t}\n\n\t\n\t//------------------------------------\n\t// utility functions\n\t\n\t// returns up to the first two chars of the head POS\n\tprivate String getHeadBroadPOS(LexDependency dep) {\n\t\tString pos = dep.lexHead.getPOS();\n\t\tString retval = pos;\n\t\tif (pos.length() > 2) retval = pos.substring(0, 2).intern();\n\t\treturn retval;\n\t\t\n\t}\n\t\n\t// sets the current head and dep indexes, and the dep sign\n\tprivate void setDepIndexes(LexDependency dep) {\n\t\tcurrentHeadIndex = currentSign.wordIndex(dep.lexHead);\n\t\tcurrentDepIndex = currentSign.wordIndex(dep.lexDep);\n\t\tcurrentDepSign = currentSign.getSignHeadedByDep(dep);\n\t}\n\n\t// returns the head-dependent order\n\tprivate String getHeadDepOrder() {\n\t\treturn (currentHeadIndex < currentDepIndex) ? HEAD_PRECEDES_DEP : DEP_PRECEDES_HEAD;\n\t}\n\t\n\t// sets the current sib index and the sib sign\n\tprivate void setSibIndex(LexDependency sib) {\n\t\tcurrentSibIndex = currentSign.wordIndex(sib.lexDep);\t\n\t\tcurrentSibSign = currentSign.getSignHeadedByDep(sib);\n\t}\n\n\t// returns the head--sibling dependent order\n\tprivate String getHeadSibOrder() {\n\t\treturn (currentHeadIndex < currentSibIndex) ? HEAD_PRECEDES_DEP : DEP_PRECEDES_HEAD;\n\t}\n\t\n\t// returns whether the dep precedes the sib\n\tprivate boolean depPrecedesSib() {\n\t\treturn currentDepIndex < currentSibIndex;\n\t}\n\n\t\n\t//------------------------------------\n\t// definiteness functions\n\t\n\t/** Class for storing whether a sign is or immediately contains a definite NP. */\n\tpublic static class DefiniteNP {\n\t\t/** Definiteness value; null means not an NP (or NP parent). */\n\t\tpublic Boolean def;\n\t\tpublic DefiniteNP(Boolean def) { this.def = def; }\n\t\tpublic String toString() { return \"defNP: \" + def; }\n\t}\n\t\n\t/** Returns the definite NP status for a sign, caching it in the sign. */\n\tpublic DefiniteNP getDefiniteNP(Sign sign) {\n\t\t// check cached\n\t\tDefiniteNP defNP = (DefiniteNP)sign.getData(DefiniteNP.class);\n\t\tif (defNP != null) return defNP;\n\t\t// determine def NP status\n\t\tBoolean def = null;\n\t\t// check for NP\n\t\tSign npSign = getSignOrChildSignAsNP(sign);\n\t\tif (npSign != null) {\n\t\t\t// set status to definite by default; check for indef\n\t\t\tdef = Boolean.TRUE;\n\t\t\t// get sem feats\n\t\t\tNominal npNom = npSign.getCategory().getIndexNominal();\n\t\t\tList semFeats = HyloHelper.getSemFeatsForHead(npNom, npSign.getCategory().getLF());\n\t\t\t// check for nil\n\t\t\tfor (SatOp feat : semFeats) {\n\t\t\t\tif (isIndefFeat(feat)) {\n\t\t\t\t\tdef = Boolean.FALSE; break;\n\t\t\t\t}\n\t\t\t}\n\t\t\t// otherwise check deps\n\t\t\tif (def) {\n\t\t\t\t// get all deps\n\t\t\t\tList allDeps = new ArrayList(5);\n\t\t\t\tallDeps.addAll(npSign.getFilledDeps());\n\t\t\t\tallDeps.addAll(npSign.getSiblingFilledDeps());\n\t\t\t\t// check for a|an|some|any\n\t\t\t\tfor (LexDependency dep : allDeps) {\n\t\t\t\t\tif (isIndefDep(dep)) {\n\t\t\t\t\t\tdef = Boolean.FALSE; break;\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t} \n\t\t// store result and return\n\t\tdefNP = new DefiniteNP(def);\n\t\tsign.addData(defNP);\n\t\treturn defNP;\n\t}\n\t\n\t/**\n\t * Returns the given sign if it's an NP sign;\n\t * otherwise returns the first child sign that's an NP sign;\n\t * otherwise returns null.\n\t */\n\tprotected Sign getSignOrChildSignAsNP(Sign sign) {\n\t\tif (isNP(sign)) return sign;\n\t\tif (sign.isLexical()) return null;\n\t\tSign[] inputs = sign.getDerivationHistory().getInputs();\n for (int i = 0; i < inputs.length; i++) {\n \tif (isNP(inputs[i])) return inputs[i];\n }\n\t\treturn null;\n\t}\n\t\n\t/**\n\t * Returns whether the given sign is an NP.\n\t * The default implementation tests for a category type of \"np\".\n\t */\n\tprotected boolean isNP(Sign sign) {\n\t\tCategory cat = sign.getCategory();\n\t\tif (!(cat instanceof AtomCat)) return false;\n\t\tAtomCat ac = (AtomCat) cat;\n\t\treturn (ac.getType().equals(\"np\"));\n\t}\n\t\n\t/** \n\t * Returns whether the given semantic features signals indefiniteness.\n\t * The default implementation tests for <:det>nil.\n\t */\n\tprotected boolean isIndefFeat(SatOp feat) {\n\t\treturn HyloHelper.getRel(feat).equals(\"det\") && \"nil\".equals(HyloHelper.getVal(feat));\n\t}\n\t\n\t/**\n\t * Returns whether the given lexical dependency signals indefiniteness.\n\t * The default implementation checks for <Det>a|an|any|some.\n\t */\n\tprotected boolean isIndefDep(LexDependency dep) {\n\t\tif (dep.rel.equalsIgnoreCase(\"Det\")) {\n\t\t\tString form = dep.lexDep.getWordForm();\n\t\t\tif (form==\"a\" || form==\"an\" || form==\"any\" || form==\"some\")\n\t\t\t\treturn true;\n\t\t}\n\t\treturn false;\n\t}\n\t\n\t/** Returns whether two signs differ in definiteness. */\n\tpublic boolean defDifference(Sign sign1, Sign sign2) {\n\t\tDefiniteNP defNP1 = getDefiniteNP(sign1);\n\t\tif (defNP1.def == null) return false;\n\t\tDefiniteNP defNP2 = getDefiniteNP(sign2);\n\t\tif (defNP2.def == null) return false;\n\t\treturn defNP1.def != defNP2.def;\n\t}\n\t\n\t/** Returns the appropriate definiteness/indefiniteness constant. */\n\tpublic String defConstant(DefiniteNP defNP) {\n\t\treturn (defNP.def) ? DEF : INDEF;\n\t}\n\t\n\t\n\t//------------------------------------\n\t// phrase length functions\n\n\t/** \n\t * Class for storing length of phrase in words, puncts and (finite) verbs, for unique retrieval from a sign's data objects. \n\t */\n\tpublic static class PhraseLengths {\n\t\tpublic int wordlen, punctlen, verblen;\n\t\tpublic PhraseLengths(int wordlen, int punctlen, int verblen) {\n\t\t\tthis.wordlen = wordlen; this.punctlen = punctlen; this.verblen = verblen;\n\t\t}\n\t\tpublic String toString() { return \" wordlen: \" + wordlen + \" punctlen: \" + punctlen + \" verblen: \" + verblen; }\n\t}\n\t\n\t/** Returns the phrase lengths for a sign, caching them in the sign. */\n\tpublic PhraseLengths getPhraseLengths(Sign sign) {\n\t\t// check cached\n\t\tPhraseLengths lengths = (PhraseLengths)sign.getData(PhraseLengths.class);\n\t\tif (lengths != null) return lengths;\n\t\tint wordlen = 0, punctlen = 0, verblen = 0;\n\t\t// lex case\n\t\tif (sign.isLexical()) {\n\t\t\tfor (Word w: sign.getWords()) {\n\t\t\t\twordlen++;\n\t\t\t\tif (isPunct(w)) punctlen++;\n\t\t\t\tif (isVerb(w)) verblen++;\n\t\t\t}\n\t\t}\n\t\t// non-lex: add child lengths\n\t\telse {\n\t\t\tSign[] inputs = sign.getDerivationHistory().getInputs();\n\t for (int i = 0; i < inputs.length; i++) {\n\t \tPhraseLengths lengthsI = getPhraseLengths(inputs[i]);\n\t \twordlen += lengthsI.wordlen; punctlen += lengthsI.punctlen; verblen += lengthsI.punctlen;\n\t }\n\t\t}\n\t\t// store result and return\n\t\tlengths = new PhraseLengths(wordlen, punctlen, verblen);\n\t\tsign.addData(lengths);\n\t\treturn lengths;\n\t}\n\t\n\t/**\n\t * Returns whether a word is a punctuation mark that typically signals sentence-internal complexity.\n\t * The default implementation tests for commas, dashes (--), semi-colons and colons.\n\t */\n\tprotected boolean isPunct(Word word) {\n\t\t// NB: in principle could use POS, but sometimes punctuation marks seem to end up with IN as the POS tag\n\t\tString form = word.getForm();\n\t\treturn (form == \",\" || form == \"--\" || form == \";\" || form == \":\");\n\t}\n\t\n\t/**\n\t * Returns whether a word is a verb that indicates a substantial clause.\n\t * The default implementation tests for the finite verb POS tags VBD, VBP and VBZ.\n\t */\n\tprotected boolean isVerb(Word word) {\n\t\tString pos = word.getPOS();\n\t\treturn (pos == \"VBD\" || pos == \"VBP\" || pos == \"VBZ\");\n\t}\n\t\n\t/** \n\t * Sets the differences in length between the signs headed by the current dep and sib, or null if none; \n\t * also sets the current dep and sib lengths. The lengths are set to the lengths of the second sign \n\t * minus those of the first sign.\n\t */\n\tprotected void setLengthsDiff() {\n\t\t// reset\n\t\tcurrentLengthsDiff = null; currentDepLengths = null; currentSibLengths = null;\n\t\t// ensure both there\n\t\tif (currentDepSign == null || currentSibSign == null) return;\n\t\t// get phrase lengths\n\t\tcurrentDepLengths = getPhraseLengths(currentDepSign);\n\t\tcurrentSibLengths = getPhraseLengths(currentSibSign);\n\t\t// get 1st and 2nd phrase lengths\n\t\tPhraseLengths pl1 = (currentDepPrecedesSib) ? currentDepLengths : currentSibLengths;\n\t\tPhraseLengths pl2 = (currentDepPrecedesSib) ? currentSibLengths : currentDepLengths;\n\t\t// set diff to 2nd - 1st\n\t\tcurrentLengthsDiff = new PhraseLengths(pl2.wordlen-pl1.wordlen, pl2.punctlen-pl1.punctlen, pl2.verblen-pl1.verblen);\n\t}\n\t\n\n\t//------------------------------------\n\t// dep len functions\n\t\n\t/** Returns the dep len feature if not already set. */\n\tprotected Alphabet.Feature getDepLenFeat() {\n\t\tif (depLenFeat == null) \n\t\t\tdepLenFeat = alphabet.index(depLenKeys);\n\t\treturn depLenFeat;\n\t}\n\t\n\t/**\n\t * Returns the dependency length between the current head and the current dependent.\n\t * The default implementation returns the number of intervening words excluding \n\t * punctuation (as determined by isPunct), and doesn't count each word in a collapsed NE separately. \n\t */ \n\tprotected int depLen() {\n\t\tList words = currentSign.getWords();\n\t\tint min = Math.min(currentHeadIndex, currentDepIndex);\n\t\tint max = Math.max(currentHeadIndex, currentDepIndex);\n\t\tint count = 0;\n\t\tfor (int i=min+1; i < max; i++) {\n\t\t\tWord w = words.get(i); \n\t\t\tif (!isPunct(w)) count++;\n\t\t}\n\t\treturn count;\n\t}\n\t\n\t\n\t//------------------------------------\n\t// word class functions\n\t\n\t/**\n\t * Returns a class for the word, or CLASS_NIL if none.\n\t * The default implementation returns one of the following, in this order:\n\t * the semantic class of the word; \n\t * CLASS_PRO, if a pronoun; \n\t * CLASS_COLOR, if a color word; \n\t * the suffix, if getSuffix returns a value; \n\t * CLASS_HYPH, if the word is hyphenated; \n\t * CLASS_CAP, if capitalized; \n\t * or CLASS_NIL, otherwise. \n\t * The word class is cached using cachedWordClasses. \n\t */\n\tprotected String getWordClass(Word word) {\n\t\tString retval = cachedWordClasses.get(word);\n\t\tif (retval != null) return retval;\n\t\tString wClass = word.getSemClass();\n\t\tif (wClass != null) return updateCachedWordClasses(word, wClass);\n\t\tif (isPro(word)) return updateCachedWordClasses(word, CLASS_PRO);\n\t\tString form = word.getForm();\n\t\tif (colors.contains(form)) return updateCachedWordClasses(word, CLASS_COLOR);\n\t\tString suffix = getSuffix(form);\n\t\tif (suffix != null) return updateCachedWordClasses(word, suffix);\n\t\tif (form.indexOf('-') >= 0) return updateCachedWordClasses(word, CLASS_HYPH); \n\t\tif (Character.isUpperCase(form.charAt(0))) return updateCachedWordClasses(word, CLASS_CAP);\n\t\treturn updateCachedWordClasses(word, CLASS_NIL);\n\t}\n\t\n\t/**\n\t * Returns whether a word is a pronoun.\n\t * The default implementation returns whether the POS tag starts with \"PR\".\n\t */\n\tprotected boolean isPro(Word word) { return word.getPOS().startsWith(\"PR\"); }\n\t\n\t/**\n\t * The set of color words to check for in determining the word class.\n\t */\n\tprotected Set colors = defaultColors();\n\t\n\t/**\n\t * Sets the set of color words.\n\t */\n\tpublic void setColorWords(Set colorWords) { colors = colorWords; }\n\t\n\t/**\n\t * Returns the default set of color words: 11 common English colors, with two spellings of gray/grey.\n\t */\n\tprotected Set defaultColors() {\n\t\tString[] colors = { \n\t\t\t\t\"black\", \"blue\", \"brown\", \"gray\", \"grey\", \"green\", \n\t\t\t\t\"orange\", \"pink\", \"purple\", \"red\", \"white\", \"yellow\"\n\t\t\t};\n\t\treturn new HashSet(Arrays.asList(colors));\n\t}\n\n\t/**\n\t * A sequence of suffixes to check for in determining the word class, ordered by specificity.\n\t */\n\tprotected String[] suffixClasses = defaultSuffixClasses();\n\t\n\t/**\n\t * Returns the default suffix classes: 61 common English suffixes from various lists on the web.\n\t */\n\tprotected String[] defaultSuffixClasses() {\n\t\treturn new String[] { \n\t\t\t\t\"ancy\", \"aphy\", \"arch\", \"crat\", \"gram\", \"less\", \"logy\", \"ness\", \"nomy\", \"ship\", \"some\", \"sque\", \"tude\",\n\t\t\t\t\"ade\", \"age\", \"ant\", \"aph\", \"ary\", \"ast\", \"ate\", \"ble\", \"dom\", \"ent\", \"est\", \"ful\", \n\t\t\t\t\"ian\", \"ile\", \"ion\", \"ing\", \"ish\", \"ism\", \"ist\", \"ise\", \"ite\", \"ium\", \"ive\", \"ize\", \n\t\t\t\t\"nce\", \"oid\", \"ory\", \"ose\", \"ote\", \"ous\", \"sig\", \"ure\",\n\t\t\t\t\"ac\", \"al\", \"an\", \"cy\", \"ed\", \"en\", \"er\", \"fy\", \"ic\", \"le\", \"ly\", \"or\", \"se\", \"sy\", \"ty\", \n\t\t\t\t\"y\"\n\t\t\t};\n\t}\n\t\n\t/**\n\t * Sets the suffix classes, which are assumed to be interned.\n\t */\n\tpublic void setSuffixClasses(String[] suffixes) { suffixClasses = suffixes; }\n\t\n\t/**\n\t * Returns a matching suffix class, or null if none.\n\t */\n\tprotected String getSuffix(String form) {\n\t\tfor (int i=0; i < suffixClasses.length; i++) {\n\t\t\tString suff = suffixClasses[i];\n\t\t\tif (form.length() > suff.length() && form.endsWith(suff)) return suff;\n\t\t}\n\t\treturn null;\n\t}\n\t\n\t/**\n\t * Cache of word classes, using a weak hash map.\n\t */\n\tprotected WeakHashMap cachedWordClasses = new WeakHashMap();\n\t\n\t/**\n\t * Updates the cached word classes with the given word and word class, and returns the word class.\n\t */\n\tprotected String updateCachedWordClasses(Word word, String wordClass) {\n\t\tcachedWordClasses.put(word, wordClass); return wordClass;\n\t}\n\t\n\t\n\t//------------------------------------\n\t// shared feature extractor elements\n\n\t// prefix: \"ld\" + head broad POS\n\tprivate void add_prefix(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return PREFIX; }});\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentHeadBroadPOS; }});\n\t}\n\t\n\t// head-dep order\n\tprivate void add_head_dep_order(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentHeadDepOrder; }});\n\t}\n\t\n\t// rel\n\tprivate void add_rel(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentDep.rel.intern(); }});\n\t}\n\t\n\t// common head-dep order elements\n\tprivate void add_head_dep_order_common(List> retval) {\n\t\tadd_prefix(retval);\n\t\tadd_head_dep_order(retval);\n\t\tadd_rel(retval);\n\t}\n\t\n\t// head word\n\tprivate void add_head_word(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentDep.lexHead.getWordForm(); }});\n\t}\n\t\n\t// head pos\n\tprivate void add_head_pos(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentDep.lexHead.getPOS(); }});\n\t}\n\t\n\t// dep word\n\tprivate void add_dep_word(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentDep.lexDep.getWordForm(); }});\n\t}\n\t\n\t// dep pos\n\tprivate void add_dep_pos(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentDep.lexDep.getPOS(); }});\n\t}\n\t\n\t// sibs precedence\n\tprivate void add_sibs_precedence(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentSibsPrecedence; }});\n\t}\n\t\n\t// common sibs precedence elements\n\tprivate void add_sibs_precedence_common(List> retval) {\n\t\tadd_prefix(retval);\n\t\tadd_sibs_precedence(retval);\n\t}\n\t\n\t// sibs word1\n\tprivate void add_sibs_word1(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ \n\t\t\treturn (currentDepPrecedesSib) ? currentDep.lexDep.getWordForm() : currentSib.lexDep.getWordForm(); \n\t\t}});\n\t}\n\t\n\t// sibs word2\n\tprivate void add_sibs_word2(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ \n\t\t\treturn (currentDepPrecedesSib) ? currentSib.lexDep.getWordForm() : currentDep.lexDep.getWordForm(); \n\t\t}});\n\t}\n\t\n\t// sibs pos1\n\tprivate void add_sibs_pos1(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ \n\t\t\treturn (currentDepPrecedesSib) ? currentDep.lexDep.getPOS() : currentSib.lexDep.getPOS(); \n\t\t}});\n\t}\n\t\n\t// sibs pos2\n\tprivate void add_sibs_pos2(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ \n\t\t\treturn (currentDepPrecedesSib) ? currentSib.lexDep.getPOS() : currentDep.lexDep.getPOS(); \n\t\t}});\n\t}\n\t\n\t// sibs class1\n\tprivate void add_sibs_class1(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){\n\t\t\tSign first = (currentDepPrecedesSib) ? currentDep.lexDep : currentSib.lexDep;\n\t\t\treturn getWordClass(first.getWords().get(0));\n\t\t}});\n\t}\n\t\n\t// sibs class2\n\tprivate void add_sibs_class2(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){\n\t\t\tSign second = (currentDepPrecedesSib) ? currentSib.lexDep : currentDep.lexDep;\n\t\t\treturn getWordClass(second.getWords().get(0));\n\t\t}});\n\t}\n\t\n\t// sibs rel1\n\tprivate void add_sibs_rel1(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ \n\t\t\treturn (currentDepPrecedesSib) ? currentDep.rel.intern() : currentSib.rel.intern(); \n\t\t}});\n\t}\n\t\n\t// sibs rel2\n\tprivate void add_sibs_rel2(List> retval) {\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ \n\t\t\treturn (currentDepPrecedesSib) ? currentSib.rel.intern() : currentDep.rel.intern(); \n\t\t}});\n\t}\n\t\n\n\t//-------------------------------\n\t// feature extractors\n\t\n\t// head-dep order words\n\tprivate List> head_dep_order_words() {\n\t\tList> retval = new ArrayList>(5);\n\t\tadd_head_dep_order_common(retval);\n\t\tadd_head_word(retval);\n\t\tadd_dep_word(retval);\n\t\treturn retval;\n\t}\n\t\n\t// head-dep order pos\n\tprivate List> head_dep_order_pos() {\n\t\tList> retval = new ArrayList>(5);\n\t\tadd_head_dep_order_common(retval);\n\t\tadd_head_pos(retval);\n\t\tadd_dep_pos(retval);\n\t\treturn retval;\n\t}\n\n\t// head-dep order word/pos\n\tprivate List> head_dep_order_word_pos() {\n\t\tList> retval = new ArrayList>(5);\n\t\tadd_head_dep_order_common(retval);\n\t\tadd_head_word(retval);\n\t\tadd_dep_pos(retval);\n\t\treturn retval;\n\t}\n\t\n\t// head-dep order pos/word\n\tprivate List> head_dep_order_pos_word() {\n\t\tList> retval = new ArrayList>(5);\n\t\tadd_head_dep_order_common(retval);\n\t\tadd_head_pos(retval);\n\t\tadd_dep_word(retval);\n\t\treturn retval;\n\t}\n\t\n\t// sibs precedence words\n\tprivate List> sibs_precedence_words() {\n\t\tList> retval = new ArrayList>(5);\n\t\tadd_sibs_precedence_common(retval);\n\t\tadd_sibs_word1(retval);\n\t\tadd_sibs_word2(retval);\n\t\treturn retval;\n\t}\n\t\n\t// sibs precedence word pos\n\tprivate List> sibs_precedence_word_pos() {\n\t\tList> retval = new ArrayList>(5);\n\t\tadd_sibs_precedence_common(retval);\n\t\tadd_sibs_word1(retval);\n\t\tadd_sibs_pos2(retval);\n\t\treturn retval;\n\t}\n\t\n\t// sibs precedence pos word\n\tprivate List> sibs_precedence_pos_word() {\n\t\tList> retval = new ArrayList>(5);\n\t\tadd_sibs_precedence_common(retval);\n\t\tadd_sibs_pos1(retval);\n\t\tadd_sibs_word2(retval);\n\t\treturn retval;\n\t}\n\t\n\t// sibs precedence pos\n\tprivate ConditionalLazyExtractor sibs_precedence_pos() {\n\t\tConditionalLazyExtractor retval = new ConditionalLazyExtractor() {\n\t\t\tboolean test() { return currentDep.lexDep.getPOS() != currentSib.lexDep.getPOS(); }\n\t\t};\n\t\tadd_sibs_precedence_common(retval.lazyExtractor);\n\t\tadd_sibs_pos1(retval.lazyExtractor);\n\t\tadd_sibs_pos2(retval.lazyExtractor);\n\t\treturn retval;\n\t}\n\t\n\t// sibs precedence word / class\n\tprivate List> sibs_precedence_word_class() {\n\t\tList> retval = new ArrayList>(5);\n\t\tadd_sibs_precedence_common(retval);\n\t\tadd_sibs_word1(retval);\n\t\tadd_sibs_class2(retval);\n\t\treturn retval;\n\t}\n\t\n\t// sibs precedence class / word\n\tprivate List> sibs_precedence_class_word() {\n\t\tList> retval = new ArrayList>(5);\n\t\tadd_sibs_precedence_common(retval);\n\t\tadd_sibs_class1(retval);\n\t\tadd_sibs_word2(retval);\n\t\treturn retval;\n\t}\n\t\n\t// sibs precedence class\n\tprivate ConditionalLazyExtractor sibs_precedence_class() {\n\t\tConditionalLazyExtractor retval = new ConditionalLazyExtractor() {\n\t\t\tboolean test() { \n\t\t\t\treturn getWordClass(currentDep.lexDep.getWords().get(0)) != getWordClass(currentSib.lexDep.getWords().get(0)); \n\t\t\t}\n\t\t};\n\t\tadd_sibs_precedence_common(retval.lazyExtractor);\n\t\tadd_sibs_class1(retval.lazyExtractor);\n\t\tadd_sibs_class2(retval.lazyExtractor);\n\t\treturn retval;\n\t}\n\t\n\t// sibs precedence rels\n\tprivate ConditionalLazyExtractor sibs_precedence_rels() {\n\t\tConditionalLazyExtractor retval = new ConditionalLazyExtractor() {\n\t\t\tboolean test() { return !currentDep.rel.equals(currentSib.rel); }\n\t\t};\n\t\tadd_sibs_precedence_common(retval.lazyExtractor);\n\t\tadd_sibs_rel1(retval.lazyExtractor);\n\t\tadd_sibs_rel2(retval.lazyExtractor);\n\t\treturn retval;\n\t}\n\t\n\t// sibs precedence defs\n\tprivate ConditionalLazyExtractor sibs_precedence_defs() {\n\t\tConditionalLazyExtractor retval = new ConditionalLazyExtractor() {\n\t\t\tboolean test() {\n\t\t\t\tif (currentDepSign == null || currentSibSign == null) return false;\n\t\t\t\treturn defDifference(currentDepSign, currentSibSign); \n\t\t\t}\n\t\t};\n\t\tadd_sibs_precedence_common(retval.lazyExtractor);\n\t\tretval.lazyExtractor.add(new TrieMap.KeyExtractor(){public String getKey(){ \n\t\t\treturn (currentDepPrecedesSib) ? defConstant(getDefiniteNP(currentDepSign)) : defConstant(getDefiniteNP(currentSibSign));\n\t\t}});\n\t\tretval.lazyExtractor.add(new TrieMap.KeyExtractor(){public String getKey(){ \n\t\t\treturn (currentDepPrecedesSib) ? defConstant(getDefiniteNP(currentSibSign)) : defConstant(getDefiniteNP(currentDepSign));\n\t\t}});\n\t\treturn retval;\n\t}\n\t\n\t// sibs precedence verbs\n\tprivate ConditionalLazyExtractor sibs_precedence_verbs() {\n\t\tConditionalLazyExtractor retval = new ConditionalLazyExtractor() {\n\t\t\tboolean test() { \n\t\t\t\treturn currentLengthsDiff.verblen != 0 && \n\t\t\t\t\t\t(currentDepLengths.verblen == 0 || currentSibLengths.verblen == 0); \n\t\t\t}\n\t\t};\n\t\tadd_sibs_precedence_common(retval.lazyExtractor);\n\t\tretval.lazyExtractor.add(new TrieMap.KeyExtractor(){public String getKey(){ \n\t\t\treturn (currentLengthsDiff.verblen > 0) ? NO_V : HAS_V;\n\t\t}});\n\t\tretval.lazyExtractor.add(new TrieMap.KeyExtractor(){public String getKey(){ \n\t\t\treturn (currentLengthsDiff.verblen > 0) ? HAS_V : NO_V;\n\t\t}});\n\t\treturn retval;\n\t}\n\t\n\t// sibs precedence puncts\n\tprivate ConditionalLazyExtractor sibs_precedence_puncts() {\n\t\tConditionalLazyExtractor retval = new ConditionalLazyExtractor() {\n\t\t\tboolean test() { \n\t\t\t\treturn currentLengthsDiff.punctlen != 0 && \n\t\t\t\t\t\t(currentDepLengths.punctlen == 0 || currentSibLengths.punctlen == 0); \n\t\t\t}\n\t\t};\n\t\tadd_sibs_precedence_common(retval.lazyExtractor);\n\t\tretval.lazyExtractor.add(new TrieMap.KeyExtractor(){public String getKey(){ \n\t\t\treturn (currentLengthsDiff.punctlen > 0) ? NO_P : HAS_P;\n\t\t}});\n\t\tretval.lazyExtractor.add(new TrieMap.KeyExtractor(){public String getKey(){ \n\t\t\treturn (currentLengthsDiff.punctlen > 0) ? HAS_P : NO_P;\n\t\t}});\n\t\treturn retval;\n\t}\n\n\t// short-long words\n\tprivate ConditionalLazyEvaluator short_long_words() {\n\t\tConditionalLazyEvaluator retval = new ConditionalLazyEvaluator() {\n\t\t\tboolean test() { return currentLengthsDiff.wordlen != 0; }\n\t\t\tfloat eval() { return (float) Math.abs(currentLengthsDiff.wordlen); }\n\t\t};\n\t\tadd_sibs_precedence_common(retval.lazyExtractor);\n\t\tretval.lazyExtractor.add(new TrieMap.KeyExtractor(){public String getKey(){ \n\t\t\treturn (currentLengthsDiff.wordlen > 0) ? SHORT_LONG_ORDER : LONG_SHORT_ORDER;\n\t\t}});\n\t\treturn retval;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/hylo/LexDependency.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2011 Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport java.util.*;\n\nimport opennlp.ccg.synsem.*;\n\n/** \n * A class for tracking semantic dependencies between lexical items.\n * A lex dependency is a triple consisting of a lexical head, a relation \n * and a lexical dependent. Either the head or dependent can be null, in \n * which case the dependency is considered unfilled. To be equal, a lex \n * dependency must have identical heads and dependents and equal relations.\n * Relations can involve chains of individual relations (concatenated with dots) \n * when there are intervening nominals for the same lexical item.\n * \n * @author Michael White\n * @version $Revision: 1.2 $, $Date: 2011/08/26 21:31:52 $\n */ \npublic class LexDependency {\n\n\t/** The lexical head. */\n\tpublic Sign lexHead;\n\t\n\t/** The relation. */\n\tpublic String rel;\n\t\n\t/** The lexical dependent. */\n\tpublic Sign lexDep;\n\t\n\t/** Constructor. */\n\tpublic LexDependency(Sign lexHead, String rel, Sign lexDep) {\n\t\tthis.lexHead = lexHead; this.rel = rel; this.lexDep = lexDep;\n\t}\n\t\n\t/** Hash code. */\n\tpublic int hashCode() {\n\t\tint retval = rel.hashCode();\n\t\tif (lexHead != null) retval += 31 * lexHead.hashCode();\n\t\tif (lexDep != null) retval += 7 * lexDep.hashCode();\n\t\treturn retval;\n\t}\n\t\n\t/** Equals. */\n\tpublic boolean equals(Object obj) {\n if (obj == this) return true;\n if (!(obj instanceof LexDependency)) return false;\n LexDependency dep = (LexDependency) obj;\n\t\treturn lexHead == dep.lexHead && lexDep == dep.lexDep && rel.equals(dep.rel);\n\t}\n\t\n\t/** toString. */\n\tpublic String toString() {\n\t\tStringBuffer retval = new StringBuffer();\n\t\tif (lexHead == null) retval.append(\"null\");\n\t\telse retval.append(lexHead.getOrthography());\n\t\tretval.append('<').append(rel).append('>');\n\t\tif (lexDep == null) retval.append(\"null\");\n\t\telse retval.append(lexDep.getOrthography());\n\t\treturn retval.toString();\n\t}\n\t\n\t/** Filled test: neither head nor dependent null. */\n\tpublic boolean filled() {\n\t\treturn lexHead != null && lexDep != null;\n\t}\n\t\n\t/**\n\t * Returns a list of unfilled dependencies for a list of EPs \n\t * for a lexical item.\n\t */\n\tpublic static List unfilledLexDeps(List preds) {\n\t\tList retval = new ArrayList(4);\n\t\tMap nomIndex = HyloHelper.nomIndex(preds);\n\t\tSet nominals = nomIndex.keySet();\n\t\t// special case for indexRels\n\t\tif (nominals.size() == 1 && !HyloHelper.isLexPred(preds.get(0))) {\n\t\t\tSign lexHead = null;\n\t\t\tLexSemOrigin origin = preds.get(0).getOrigin();\n\t\t\tif (origin instanceof Sign) lexHead = (Sign) origin;\n\t\t\telse return retval;\n\t\t\tfor (SatOp pred : preds) {\n\t\t\t\tString rel = HyloHelper.getRel(pred);\n\t\t\t\tif (rel != null) {\n\t\t\t\t\t// add unfilled dep with lex head as dep\n\t\t\t\t\tretval.add(new LexDependency(null, rel, lexHead));\n\t\t\t\t\tif (HyloHelper.isRelPred(pred)) {\n\t\t\t\t\t\t// also add unfilled dep with lex head as head\n\t\t\t\t\t\tretval.add(new LexDependency(lexHead, rel, null));\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\treturn retval;\n\t\t}\n\t\t// otherwise, starting with each root nominal, enumerate paths to leaf nominals\n\t\tfor (Nominal root : nominals) {\n\t\t\t// check for root nominal\n\t\t\tif (!HyloHelper.isRoot(root, preds)) continue;\n\t\t\t// set lex head\n\t\t\tSign lexHead = null;\n\t\t\tint rootIndex = nomIndex.get(root);\n\t\t\tSatOp rootPred = preds.get(rootIndex);\n\t\t\tif (HyloHelper.isLexPred(rootPred)) {\n\t\t\t\tLexSemOrigin origin = rootPred.getOrigin();\n\t\t\t\tif (origin instanceof Sign) lexHead = (Sign) origin;\n\t\t\t\trootIndex++;\n\t\t\t}\n\t\t\t// start path for each rel for root nom\n\t\t\tfor (int i=rootIndex; i < preds.size() && HyloHelper.getPrincipalNominal(preds.get(i)).equals(root); i++) {\n\t\t\t\trootPred = preds.get(i);\n\t\t\t\tNominal dep = HyloHelper.getSecondaryNominal(rootPred);\n\t\t\t\tif (dep == null) continue;\n\t\t\t\tString rel = HyloHelper.getRel(rootPred);\n\t\t\t\taddUnfilledLexDep(dep, lexHead, rel, preds, nomIndex, retval);\n\t\t\t}\n\t\t}\n\t\treturn retval;\n\t}\n\t\n\t// recursively adds unfilled lex deps to retval for leaf nominals \n\tprivate static void addUnfilledLexDep(Nominal dep, Sign lexHead, String rel, List preds, Map nomIndex, List retval) {\n\t\t// if dep not in nom index, then just add unfilled dep for the current rel\n\t\tif (!nomIndex.containsKey(dep)) {\n\t\t\tretval.add(new LexDependency(lexHead, rel, null));\n\t\t\treturn;\n\t\t}\n\t\t// otherwise continue with the preds for the current dep\n\t\tint depIndex = nomIndex.get(dep);\n\t\tSatOp depPred = preds.get(depIndex);\n\t\t// if lex head null, add unfilled dep for the current rel, \n\t\t// then update lex head and reset rel\n\t\tif (lexHead == null) {\n\t\t\tSign lexDep = null;\n\t\t\tLexSemOrigin origin = depPred.getOrigin();\n\t\t\tif (origin instanceof Sign) {\n\t\t\t\tlexDep = (Sign) origin;\n\t\t\t\tretval.add(new LexDependency(lexHead, rel, lexDep));\n\t\t\t}\n\t\t\tlexHead = lexDep;\n\t\t\trel = null;\n\t\t}\n\t\t// then recurse through further rels, if any\n\t\tfor (int i=depIndex; i < preds.size() && HyloHelper.getPrincipalNominal(preds.get(i)).equals(dep); i++) {\n\t\t\tdepPred = preds.get(i);\n\t\t\tNominal depdep = HyloHelper.getSecondaryNominal(depPred);\n\t\t\tif (depdep == null) continue;\n\t\t\tString relrel = (rel == null) \n\t\t\t\t? HyloHelper.getRel(depPred) \n\t\t\t\t: rel + \".\" + HyloHelper.getRel(depPred);\n\t\t\taddUnfilledLexDep(depdep, lexHead, relrel, preds, nomIndex, retval);\n\t\t}\n\t}\n\t\n\t\n\t/**\n\t * Returns the filled lexical dependencies from those in the unfilled list \n\t * by checking the list of EPs for ones that have become filled, removing the \n\t * corresponding no longer unfilled deps. \n\t */\n\tpublic static List filledLexDeps(List unfilled, List preds) {\n\t\tList retval = new ArrayList(unfilled.size());\n\t\tMap nomIndex = HyloHelper.nomIndex(preds);\n\t\t// check each unfilled dep\n\t\tfor (Iterator it = unfilled.iterator(); it.hasNext(); ) {\n\t\t\tLexDependency udep = it.next();\n\t\t\tString[] rels = udep.rel.split(\"\\\\.\");\n\t\t\t// dependent missing case\n\t\t\tif (udep.lexDep == null) {\n\t\t\t\t// follow rels to descendant pred\n\t\t\t\tSatOp relPred = findPred(udep.lexHead, rels[0], preds);\n\t\t\t\tSatOp descendantPred = findDescendantPred(relPred, 0, rels, preds, nomIndex);\n\t\t\t\t// check if dep filled\n\t\t\t\tNominal depnom = HyloHelper.getSecondaryNominal(descendantPred);\n\t\t\t\tif (!nomIndex.containsKey(depnom)) continue;\n\t\t\t\tSatOp depPred = preds.get(nomIndex.get(depnom));\n\t\t\t\tif (HyloHelper.isLexPred(depPred)) {\n\t\t\t\t\t// remove dep from unfilled\n\t\t\t\t\tit.remove();\n\t\t\t\t\t// add filled dep, if lexical\n\t\t\t\t\tif (depPred.getOrigin() instanceof Sign) {\n\t\t\t\t\t\tSign lexDep = (Sign) depPred.getOrigin();\n\t\t\t\t\t\tretval.add(new LexDependency(udep.lexHead, udep.rel, lexDep));\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\t// head missing case\n\t\t\telse if (udep.lexHead == null) {\n\t\t\t\t// follow rels to ancestor pred\n\t\t\t\tSatOp relPred = findPred(udep.lexDep, rels[rels.length-1], preds);\n\t\t\t\tSatOp ancestorPred = findAncestorPred(relPred, rels.length-1, rels, preds);\n\t\t\t\t// check if head filled\n\t\t\t\tNominal headnom = HyloHelper.getPrincipalNominal(ancestorPred);\n\t\t\t\tif (!nomIndex.containsKey(headnom)) continue;\n\t\t\t\tSatOp headPred = preds.get(nomIndex.get(headnom));\n\t\t\t\tif (HyloHelper.isLexPred(headPred)) {\n\t\t\t\t\t// remove dep from unfilled\n\t\t\t\t\tit.remove();\n\t\t\t\t\t// add filled dep, if lexical\n\t\t\t\t\tif (headPred.getOrigin() instanceof Sign) {\n\t\t\t\t\t\tSign lexHead = (Sign) headPred.getOrigin();\n\t\t\t\t\t\tretval.add(new LexDependency(lexHead, udep.rel, udep.lexDep));\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\treturn retval;\n\t}\n\t\n\t// returns the EP with the given origin and rel, or null if not found\n\tprivate static SatOp findPred(Sign origin, String rel, List preds) {\n\t\tfor (SatOp pred : preds) {\n\t\t\tif (pred.getOrigin() != origin) continue;\n\t\t\tif (rel.equals(HyloHelper.getRel(pred))) return pred;\n\t\t}\n\t\treturn null;\n\t}\n\t\n\t// returns the descendant EP for the given rels, or null if not found\n\tprivate static SatOp findDescendantPred(SatOp current, int index, String[] rels, List preds, Map nomIndex) {\n\t\tif (index == rels.length-1) return current;\n\t\t// find EP for next rel\n\t\tSatOp next = null;\n\t\tString rel = rels[++index];\n\t\tNominal depnom = HyloHelper.getSecondaryNominal(current);\n\t\tfor (int i=nomIndex.get(depnom); i < preds.size() && HyloHelper.getPrincipalNominal(preds.get(i)).equals(depnom); i++) {\n\t\t\tSatOp pred = preds.get(i);\n\t\t\tif (rel.equals(HyloHelper.getRel(pred))) {\n\t\t\t\tnext = pred; break;\n\t\t\t}\n\t\t}\n\t\tif (next == null) return null;\n\t\t// recurse\n\t\treturn findDescendantPred(next, index, rels, preds, nomIndex);\n\t}\n\t\n\t// returns the ancestor EP for the given rels, or null if not found\n\tprivate static SatOp findAncestorPred(SatOp current, int index, String[] rels, List preds) {\n\t\tif (index == 0) return current;\n\t\t// find EP for previous rel\n\t\tSatOp prev = null;\n\t\tString rel = rels[--index];\n\t\tNominal headnom = HyloHelper.getPrincipalNominal(current);\n\t\tfor (int i=0; i < preds.size(); i++) {\n\t\t\tSatOp pred = preds.get(i);\n\t\t\tif (headnom.equals(HyloHelper.getSecondaryNominal(pred)) && rel.equals(HyloHelper.getRel(pred))) {\n\t\t\t\tprev = pred; break;\n\t\t\t}\n\t\t}\n\t\tif (prev == null) return null;\n\t\t// recurse\n\t\treturn findAncestorPred(prev, index, rels, preds);\n\t}\n\t\n\t\n\t/** \n\t * Filters the first list of dependencies to those sharing a head with a dependency in the second list.\n\t */\n\tpublic static List filterSameHead(List deps1, List deps2) {\n\t\tif (deps1.isEmpty() || deps2.isEmpty()) return Collections.emptyList();\n\t\tList retval = new ArrayList(deps1.size());\n\t\tfor (LexDependency dep1 : deps1) {\n\t\t\tfor (LexDependency dep2 : deps2) {\n\t\t\t\tif (dep1.lexHead == dep2.lexHead) {\n\t\t\t\t\tretval.add(dep1); break;\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\treturn retval;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/hylo/ModalOp.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\nimport org.jdom.*;\nimport java.util.*;\nimport gnu.trove.*;\n\n/**\n * A parent class for modal operators, such as <P>p, [F]q, and\n * <>(p ^ q).\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.6 $, $Date: 2009/12/21 03:27:19 $\n **/\npublic abstract class ModalOp extends HyloFormula {\n\n\tprivate static final long serialVersionUID = 1L;\n\t\n\tprotected Mode _mode;\n protected LF _arg;\n\n @SuppressWarnings(\"unchecked\")\n\tprotected ModalOp(Element e) {\n String atomLabel = e.getAttributeValue(\"mode\");\n if (atomLabel == null) atomLabel = e.getAttributeValue(\"m\");\n if (atomLabel != null) {\n _mode = new ModeLabel(atomLabel);\n _arg = HyloHelper.getLF_FromChildren(e);\n } else {\n List children = e.getChildren();\n _mode = (Mode)HyloHelper.getLF((Element)children.get(0));\n _arg = HyloHelper.getLF((Element)children.get(1));\n }\n }\n \n protected ModalOp(Mode mode, LF arg) {\n _mode = mode;\n _arg = arg;\n }\n\n public Mode getMode() { return _mode; }\n public void setMode(Mode mode) { _mode = mode; }\n\n public LF getArg() { return _arg; }\n public void setArg(LF arg) { _arg = arg; }\n \n public void deepMap(ModFcn mf) {\n _arg.deepMap(mf);\n mf.modify(this);\n }\n\n public boolean occurs(Variable var) {\n return _mode.occurs(var) || _arg.occurs(var);\n }\n\n protected boolean equals(ModalOp mo) {\n if (_mode.equals(mo._mode) && _arg.equals(mo._arg)) {\n return true;\n } else {\n return false;\n }\n }\n\n protected void unifyCheck(ModalOp mo) throws UnifyFailure {\n _mode.unifyCheck(mo._mode);\n _arg.unifyCheck(mo._arg);\n }\n\n /** Returns a hash code based on the mode and arg. */\n public int hashCode() { return _mode.hashCode() + _arg.hashCode(); }\n \n /**\n * Returns a hash code using the given map from vars to ints.\n */\n public int hashCode(TObjectIntHashMap varMap) { \n return _mode.hashCode(varMap) + _arg.hashCode(varMap); \n }\n \n /**\n * Returns whether this modal op equals the given object \n * up to variable names, using the given maps from vars to ints.\n */\n public boolean equals(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2) {\n if (obj.getClass() != this.getClass()) { return false; }\n ModalOp mo = (ModalOp) obj;\n return _mode.equals(mo._mode, varMap, varMap2) && \n _arg.equals(mo._arg, varMap, varMap2); \n }\n\n /** Returns the string form of this modal op, without the arg. */\n abstract public String modalOpString();\n \n public String toString() {\n return new StringBuffer().append(modalOpString()).append(_arg.toString()).toString();\n }\n\n /**\n * Returns a pretty-printed string of this LF, with the given indent.\n */\n public String prettyPrint(String indent) {\n // calc new indent\n StringBuffer ibuf = new StringBuffer();\n ibuf.append(indent).append(' ');\n String modalOpString = modalOpString(); \n for (int i = 0; i < modalOpString.length(); i++) {\n ibuf.append(' ');\n }\n String newIndent = ibuf.toString();\n // calc string\n StringBuffer sb = new StringBuffer();\n sb.append('\\n').append(indent).append(modalOpString);\n sb.append(_arg.prettyPrint(newIndent));\n // done\n return sb.toString();\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/Mode.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\n\n/**\n * A interface for hybrid logic nominals, to allow polymorphism for\n * both Modality labels and variables over Modality labels.\n *\n * @author Jason Baldridge\n * @author \t\tScott Martin\n * @version $Revision: 1.2 $, $Date: 2005/10/19 21:27:15 $\n **/\npublic interface Mode extends LF {\n\t\n\t/**\n\t * Gets the name of this mode.\n\t */\n\tString getName();\n}\n\n" }, { "path": "src/opennlp/ccg/hylo/ModeLabel.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-4 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\nimport org.jdom.*;\n\n/**\n * A modality label.\n * Types are not currently supported.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2009/07/17 04:23:30 $\n **/\npublic final class ModeLabel extends HyloAtom implements Mode {\n\n\tprivate static final long serialVersionUID = -4101305505903588678L;\n\n\tpublic ModeLabel(String name) {\n super(name);\n }\n\n public LF copy() {\n return new ModeLabel(_name);\n }\n\n public Object unify(Object u, Substitution sub) throws UnifyFailure {\n if (equals(u)) return this;\n return super.unify(u, sub);\n }\n \n /**\n * Returns an XML representation of this LF (not currently supported).\n * Throws a runtime exception.\n */\n public Element toXml() {\n throw new RuntimeException(\"toXml() not currently supported for ModeLabel.\");\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/ModeVar.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-4 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\nimport org.jdom.*;\n\n/**\n * A class for variables over ModeLabel objects.\n * Mode vars are not really supported at present, and \n * type unification is not implemented.\n *\n * @author Jason Baldridge\n * @author\t\tScott Martin\n * @version $Revision: 1.5 $, $Date: 2009/07/17 04:23:30 $\n **/\npublic class ModeVar extends HyloVar implements Mode {\n \n\tprivate static final long serialVersionUID = -6872985893931836901L;\n\n\tpublic ModeVar(String name) {\n super(name);\n }\n\n protected ModeVar(String name, int index, SimpleType st) {\n super(name, index, st);\n }\n \n /**\n * Gets the name of this mode variable.\n * @return This method just delegates to the {@link #name()} method.\n */\n @Override\n\tpublic String getName() {\n\t\treturn name();\n\t}\n\n\tpublic LF copy() {\n return new ModeVar(_name, _index, type);\n }\n\n \n public boolean equals(Object o) {\n if (!(o instanceof ModeVar)) return false;\n return super.equals(o);\n }\n \n public Object unify(Object u, Substitution sub) throws UnifyFailure {\n if (u instanceof ModeLabel) {\n return sub.makeSubstitution(this, u);\n } else if (u instanceof ModeVar) {\n ModeVar u_nv = (ModeVar)u;\n if (equals(u_nv)) return this;\n // substitute according to comparison order, \n // so that the direction of unification doesn't matter\n if (compareTo(u_nv) >= 0) {\n return sub.makeSubstitution(this, u_nv);\n } else {\n return sub.makeSubstitution(u_nv, this);\n }\n } else {\n throw new UnifyFailure();\n }\n }\n\n public Object fill(Substitution sub) throws UnifyFailure {\n Object val = sub.getValue(this);\n if (val != null) {\n return val;\n } else {\n return this;\n }\n }\n\n \n /**\n * Returns an XML representation of this LF (not currently supported).\n * Throws a runtime exception.\n */\n public Element toXml() {\n throw new RuntimeException(\"toXml() not currently supported for ModeVar.\");\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/Nominal.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-3 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\n\n/**\n * A interface for hybrid logic nominals, to allow polymorphism for\n * both Nominal atoms and Nominal variables.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.6 $, $Date: 2005/11/01 19:39:27 $\n **/\npublic interface Nominal extends LF {\n public String getName();\n public boolean isShared();\n public void setShared(boolean shared);\n public int compareTo(Nominal nom);\n}\n" }, { "path": "src/opennlp/ccg/hylo/NominalAtom.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-4 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport java.io.FileInputStream;\nimport java.io.FileOutputStream;\nimport java.io.IOException;\nimport java.io.ObjectInputStream;\nimport java.io.ObjectOutputStream;\n\nimport opennlp.ccg.grammar.*;\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\n\nimport org.jdom.*;\n\n/**\n * A hybrid logic nominal, an atomic formula which holds true at exactly one\n * point in a model.\n * The type is checked for compatibility during unification with nominal vars, \n * but it is not updated, since nominal atoms are constants.\n * If no type is given, the TOP type is used for backwards compatibility.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.10 $, $Date: 2009/07/17 04:23:30 $\n **/\npublic class NominalAtom extends HyloAtom implements Nominal {\n\n\tprivate static final long serialVersionUID = -6002484920078196411L;\n\t\n\tprotected boolean shared = false;\n \n public NominalAtom(String name) {\n this(name, null);\n }\n \n public NominalAtom(String name, SimpleType st) {\n this(name, st, false);\n }\n \n public NominalAtom(String name, SimpleType st, boolean shared) {\n super(name, st);\n type = (st != null) ? st : Grammar.theGrammar.types.getSimpleType(Types.TOP_TYPE);\n this.shared = shared;\n }\n\n public String getName() { return _name; }\n \n public boolean isShared() { return shared; }\n\n public void setShared(boolean shared) { this.shared = shared; }\n \n public LF copy() {\n return new NominalAtom(_name, type, shared);\n }\n\n /** Returns a hash code based on the atom name and type. */\n public int hashCode() { \n return _name.hashCode() + type.hashCode();\n }\n\n /**\n * Returns whether this atom equals the given object based on the atom name and type.\n */\n public boolean equals(Object obj) {\n if (!super.equals(obj)) return false;\n NominalAtom nom = (NominalAtom) obj;\n return type.equals(nom.type);\n }\n\n public Object unify(Object u, Substitution sub) throws UnifyFailure {\n if (equals(u)) return this;\n return super.unify(u, sub);\n }\n \n public int compareTo(Nominal nom) {\n if (nom instanceof NominalAtom) { \n return super.compareTo((NominalAtom)nom);\n }\n int retval = _name.compareTo(nom.getName());\n if (retval == 0) { retval = -1; } // atom precedes var if names equal\n return retval;\n }\n \n public String toString() {\n String retval = _name;\n if (!type.getName().equals(Types.TOP_TYPE)) retval += \":\" + type.getName();\n return retval;\n }\n \n /**\n * Returns an XML representation of this LF.\n */\n public Element toXml() {\n Element retval = new Element(\"nom\");\n retval.setAttribute(\"name\", toString());\n return retval;\n }\n \n /** Tests serialization. */\n public static void debugSerialization() throws IOException, ClassNotFoundException {\n // test serialization\n NominalAtom n = new NominalAtom(\"w1\");\n \tString filename = \"tmp.ser\";\n \tObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(filename));\n \tSystem.out.println(\"Writing n: \" + n);\n \tout.writeObject(n);\n \tout.close();\n \tObjectInputStream in = new ObjectInputStream(new FileInputStream(filename));\n \tSystem.out.print(\"Reading n2: \");\n \tNominalAtom n2 = (NominalAtom) in.readObject();\n \tSystem.out.println(n2);\n \tin.close();\n \t// test identity and equality\n \tSystem.out.println(\"n == n2?: \" + (n == n2));\n \tSystem.out.println(\"n.equals(n2)?: \" + (n.equals(n2)));\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/NominalVar.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-4 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\nimport org.jdom.*;\n\n/**\n * A class for variables over NominalAtom objects.\n * Nominal vars take precedence over generic hylo vars.\n * Types are unified with other hylo vars and nominal atoms.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.13 $, $Date: 2009/07/17 04:23:30 $\n **/\npublic class NominalVar extends HyloVar implements Nominal {\n \n\tprivate static final long serialVersionUID = -2086362887254623273L;\n\t\n\tprotected boolean shared = false;\n \n public NominalVar(String name) {\n super(name);\n }\n\n public NominalVar(String name, SimpleType st) {\n super(name, st);\n }\n\n public NominalVar(String name, SimpleType st, boolean shared) {\n super(name, st);\n this.shared = shared;\n }\n\n protected NominalVar(String name, int index, SimpleType st) {\n super(name, index, st);\n }\n \n protected NominalVar(String name, int index, SimpleType st, boolean shared) {\n super(name, index, st);\n this.shared = shared;\n }\n \n public String getName() { return _name; }\n \n public boolean isShared() { return shared; }\n\n public void setShared(boolean shared) { this.shared = shared; }\n \n public void setType(SimpleType st) { \n _hashCode += st.getIndex() - type.getIndex();\n type = st; \n }\n \n public LF copy() {\n return new NominalVar(_name, _index, type, shared);\n }\n\n \n public boolean equals(Object o) {\n if (!(o instanceof NominalVar)) return false;\n return super.equals(o);\n }\n \n public int compareTo(Nominal nom) {\n if (nom instanceof NominalVar) { \n return super.compareTo((NominalVar)nom);\n }\n int retval = _name.compareTo(nom.getName());\n if (retval == 0) { retval = 1; } // atom precedes var if names equal\n return retval;\n }\n \n \n public Object unify(Object u, Substitution sub) throws UnifyFailure {\n // check for equality with u\n if (equals(u)) return this; \n // make sure u is an LF\n if (!(u instanceof LF)) throw new UnifyFailure(); \n // check type compatibility\n LF lf = (LF) u;\n if (lf.getType() == null) throw new UnifyFailure();\n SimpleType st = (SimpleType) type.unify(lf.getType(), sub);\n // with nominal atoms, go ahead and substitute\n if (u instanceof NominalAtom) return sub.makeSubstitution(this, u); \n // with nominal vars, substitute according to type specificity then comparison order, \n // so that the direction of unification doesn't matter\n if (u instanceof NominalVar) {\n NominalVar u_nv = (NominalVar) u;\n // equal types, use comparison order\n if (type.equals(u_nv.getType())) {\n if (super.compareTo(u_nv) >= 0) return sub.makeSubstitution(this, u_nv); \n else return sub.makeSubstitution(u_nv, this);\n }\n // unequal types, use most specific one\n if (type.equals(st)) return sub.makeSubstitution(u_nv, this);\n if (u_nv.getType().equals(st)) return sub.makeSubstitution(this, u_nv); \n // otherwise make new nom var with intersection type, \n // name based on comparison order and index, and new index\n String name = (super.compareTo(u_nv) >= 0) ? (u_nv._name + u_nv._index) : (_name + _index);\n NominalVar nv_st = new NominalVar(name, UnifyControl.getUniqueVarIndex(), st);\n // and subst both\n sub.makeSubstitution(u_nv, nv_st);\n return sub.makeSubstitution(this, nv_st); \n }\n // with hylo vars, substitute the hylo var for this \n if (u instanceof HyloVar) { \n HyloVar u_hv = (HyloVar) u;\n // check for same type\n if (type.equals(st)) return sub.makeSubstitution(u_hv, this); \n // otherwise make new nom var with intersection type, \n // same name, and new index\n NominalVar nv_st = new NominalVar(this._name, UnifyControl.getUniqueVarIndex(), st);\n // and subst both\n sub.makeSubstitution(u_hv, nv_st);\n return sub.makeSubstitution(this, nv_st); \n }\n // otherwise give up\n throw new UnifyFailure();\n }\n\n public Object fill(Substitution sub) throws UnifyFailure {\n Object val = sub.getValue(this);\n if (val != null) {\n return val;\n } else {\n return this;\n }\n }\n \n /**\n * Returns an XML representation of this LF.\n */\n public Element toXml() {\n Element retval = new Element(\"nomvar\");\n retval.setAttribute(\"name\", nameWithType());\n return retval;\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/Op.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-5 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.grammar.Grammar;\nimport org.jdom.*;\nimport java.util.*;\nimport gnu.trove.*;\n\n/**\n * A generic operator, such as conjunction, disjunction, exclusive-or, \n * negation or optionality (^, v, v_, ~, ?).\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.17 $, $Date: 2009/12/21 03:27:19 $\n **/\npublic class Op extends HyloFormula {\n \n\tprivate static final long serialVersionUID = -7489598953770901195L;\n\n\t/** Conjunction constant. */\n public static final String CONJ = \"conj\";\n \n /** Disjunction constant. */\n public static final String DISJ = \"disj\";\n \n /** Exclusive-or constant. */\n public static final String XOR = \"xor\";\n \n /** Negation constant. */\n public static final String NEG = \"neg\";\n \n /** Optionality constant. */\n public static final String OPT = \"opt\";\n\n \n /** The name of the operator (ie its kind). */\n protected final String _name;\n \n /** The args. */\n protected List _args;\n \n /** Element constructor. */\n @SuppressWarnings(\"unchecked\")\n\tpublic Op(Element e) {\n String name = e.getAttributeValue(\"name\");\n if (name == null) name = e.getAttributeValue(\"n\");\n _name = name;\n List argElements = e.getChildren();\n int argSize = argElements.size();\n List args = new ArrayList(argSize);\n for (int i=0; i 1 && (name.equals(NEG) || name.equals(OPT))) {\n _args = new ArrayList(1);\n _args.add(new Op(CONJ, args));\n }\n else _args = args;\n }\n\n /** Constructor. */\n public Op(String name, List args) {\n _name = name; _args = args; \n }\n\n /** Two arg convenience constructor. */\n public Op(String name, LF first, LF second) {\n _name = name;\n _args = new ArrayList();\n _args.add(first); _args.add(second);\n }\n \n\tpublic String getName() { \n return _name;\n }\n\n public List getArguments() {\n return _args;\n }\n \n public void addArgument(LF formula) {\n _args.add(formula);\n }\n \n /**\n * Appends the args if the given lf is a CONJ op,\n * otherwise just adds it.\n */\n public void appendArgs(LF lf) {\n if (lf instanceof Op && ((Op)lf).getName().equals(Op.CONJ)) \n _args.addAll(((Op)lf).getArguments());\n else _args.add(lf);\n }\n\n public LF copy() {\n List $args = new ArrayList(_args.size());\n for (LF arg : _args) {\n $args.add(arg.copy());\n }\n return new Op(_name, $args);\n }\n\n public void deepMap(ModFcn mf) {\n for (Iterator argsIt = _args.iterator(); argsIt.hasNext(); ) {\n argsIt.next().deepMap(mf);\n }\n mf.modify(this);\n }\n\n public boolean occurs(Variable var) {\n for (Iterator argsIt = _args.iterator(); argsIt.hasNext(); ) {\n if (argsIt.next().occurs(var)) {\n return true;\n }\n }\n return false;\n }\n\n /** Returns true iff the given object equals this op. */\n public boolean equals(Object o) {\n if (!(o instanceof Op)) return false;\n Op op = (Op) o;\n if (_name != op._name) return false;\n List opArgs = op._args;\n if (_args.size() != opArgs.size()) return false;\n if (!opArgs.containsAll(_args)) return false;\n return true;\n }\n \n /** Unification is not attempted for Ops. */\n public void unifyCheck(Object u) throws UnifyFailure {\n throw new UnifyFailure();\n }\n\n /** Unification is not attempted for Ops. */\n public Object unify(Object u, Substitution s) throws UnifyFailure {\n throw new UnifyFailure();\n }\n \n public Object fill(Substitution sub) throws UnifyFailure {\n List $args = new ArrayList(_args.size());\n for (LF arg : _args) {\n $args.add((LF)arg.fill(sub));\n }\n return new Op(_name, $args);\n }\n \n public String toString() {\n StringBuffer sb = new StringBuffer();\n String opString = printOp(_name);\n if (_args.size() == 1) {\n sb.append(opString);\n sb.append(_args.get(0).toString());\n } else {\n sb.append('(');\n Iterator argsIt = filteredArgs().iterator();\n for (; argsIt.hasNext(); ) {\n sb.append(argsIt.next().toString());\n if (argsIt.hasNext()) sb.append(' ').append(opString).append(' ');\n }\n sb.append(')');\n }\n return sb.toString();\n }\n \n /**\n * Returns a pretty-printed string of this LF, with the given indent.\n */\n public String prettyPrint(String indent) {\n StringBuffer sb = new StringBuffer();\n String opString = printOp(_name);\n if (_args.size() == 1) {\n sb.append(opString);\n sb.append(((LF)_args.get(0)).prettyPrint(indent));\n } else {\n sb.append('(');\n Iterator argsIt = filteredArgs().iterator();\n for (; argsIt.hasNext(); ) {\n sb.append(argsIt.next().prettyPrint(indent));\n if (argsIt.hasNext()) sb.append(' ').append(opString).append(' ');\n }\n sb.append(')');\n }\n return sb.toString();\n }\n \n public static String printOp(String o) {\n if (o.equals(CONJ)) return \"^\";\n else if (o.equals(DISJ)) return \"v\";\n else if (o.equals(XOR)) return \"v_\";\n else if (o.equals(NEG)) return \"~\";\n else if (o.equals(OPT)) return \"?\";\n else return o;\n }\n \n // filters out semantic features if apropos\n private List filteredArgs() {\n String featsToShow = Grammar.theGrammar.prefs.featsToShow;\n if (featsToShow.length() == 0) return _args;\n List retval = new ArrayList(_args.size());\n for (Iterator it = _args.iterator(); it.hasNext(); ) {\n LF arg = it.next();\n String attr = null;\n if (arg instanceof SatOp && HyloHelper.isAttrPred(arg)) \n attr = HyloHelper.getRel(arg);\n else if (arg instanceof Diamond && HyloHelper.isAttr(arg))\n attr = ((Diamond)arg).getMode().toString();\n if (attr == null || featsToShow.indexOf(attr) != -1)\n retval.add(arg);\n }\n return retval;\n }\n\n /** Returns a hash code. */\n public int hashCode() {\n int retval = _name.hashCode();\n for (Iterator it = _args.iterator(); it.hasNext(); ) {\n retval += it.next().hashCode();\n }\n return retval;\n }\n \n /**\n * Returns a hash code using the given map from vars to ints.\n */\n public int hashCode(TObjectIntHashMap varMap) { \n int retval = _name.hashCode();\n for (Iterator it = _args.iterator(); it.hasNext(); ) {\n LF arg = it.next();\n retval += arg.hashCode(varMap);\n }\n return retval;\n }\n \n /**\n * Returns whether this op equals the given object \n * up to variable names, using the given maps from vars to ints\n * (where args must be in the same order).\n */\n public boolean equals(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2) {\n if (obj.getClass() != this.getClass()) { return false; }\n Op op = (Op) obj;\n if (!_name.equals(op._name)) return false;\n if (_args.size() != op._args.size()) return false;\n for (int i = 0; i < _args.size(); i++) {\n LF arg = (LF) _args.get(i);\n LF arg2 = (LF) op._args.get(i);\n if (!arg.equals(arg2, varMap, varMap2)) return false;\n }\n return true;\n }\n \n /**\n * Returns an XML representation of this LF.\n */\n public Element toXml() {\n Element retval = new Element(\"op\");\n retval.setAttribute(\"name\", _name);\n for (int i = 0; i < _args.size(); i++) {\n LF arg = (LF) _args.get(i);\n Element argElt = arg.toXml();\n retval.addContent(argElt);\n }\n return retval;\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/Proposition.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-4 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\nimport org.jdom.*;\n\n/**\n * A propositional value, such as the predicate \"sleep\", \n * or the value of a semantic feature, such as \"past\" for tense.\n * Types are unified if present.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.7 $, $Date: 2009/07/17 04:23:30 $\n **/\npublic class Proposition extends HyloAtom {\n\n\tprivate static final long serialVersionUID = -5392519210634765414L;\n\n\tpublic Proposition(String name) {\n super(name);\n }\n \n public Proposition(String name, SimpleType st) {\n super(name, st);\n }\n \n public LF copy() {\n return new Proposition(_name, type);\n }\n \n public Object unify(Object u, Substitution sub) throws UnifyFailure {\n // check equality\n if (equals(u)) return this;\n // check for prop with compatible type\n if (u instanceof Proposition) {\n Proposition prop = (Proposition) u;\n if (type == null || prop.type == null) throw new UnifyFailure();\n SimpleType st = (SimpleType) type.unify(prop.type, sub);\n // return prop with most specific type\n if (st.equals(type)) return this;\n if (st.equals(prop.type)) return prop;\n // otherwise return prop with name of intersection type\n return new Proposition(st.getName(), st);\n }\n // otherwise defer to default routine\n return super.unify(u, sub);\n }\n \n /**\n * Returns an XML representation of this LF.\n */\n public Element toXml() {\n Element retval = new Element(\"prop\");\n retval.setAttribute(\"name\", toString());\n return retval;\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/SatOp.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-5 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\nimport org.jdom.*;\nimport java.util.*;\nimport gnu.trove.*;\n\n/**\n * A hybrid logic satifaction operator, which tests whether a formula is true\n * a particular point named by a nominal.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.13 $, $Date: 2009/07/17 04:23:30 $\n **/\npublic class SatOp extends HyloFormula {\n \n\tprivate static final long serialVersionUID = -4953978442971984002L;\n\t\n\t/**\n * The LF alts to which this LF belongs.\n * Null represents no alts.\n */\n protected List alts = null;\n \n /**\n * Gets the LF alts to which this LF belongs.\n */\n public List getAlts() { return alts; }\n \n /**\n * The LF opts (optional parts) to which this LF belongs.\n * LF opts are used during realization to represent \n * optional parts of the input.\n * The opts are numbered starting with 0, \n * and null represents no opts.\n */\n protected TIntArrayList opts = null;\n \n /**\n * Gets the LF opts to which this LF belongs.\n */\n public TIntArrayList getOpts() { return opts; }\n \n /**\n * The sign or unary rule which introduced this predication.\n */\n protected LexSemOrigin _origin = null;\n \n /**\n * Gets the sign or unary rule which introduced this predication (or null if none).\n */\n public LexSemOrigin getOrigin() { return _origin; }\n \n /**\n * Sets the sign or unary rule which introduced this predication.\n */\n public void setOrigin(LexSemOrigin origin) { _origin = origin; }\n \n \n // the real contents of the satop\n protected Nominal _nominal;\n protected LF _arg;\n\n public SatOp(Element e) {\n boolean shared = \"true\".equals(e.getAttributeValue(\"shared\"));\n String nom = e.getAttributeValue(\"nom\");\n if (nom != null) {\n _nominal = new NominalAtom(HyloHelper.prefix(nom), HyloHelper.type(nom), shared);\n } else {\n nom = e.getAttributeValue(\"nomvar\");\n if (nom != null) {\n _nominal = new NominalVar(HyloHelper.prefix(nom), HyloHelper.type(nom), shared);\n } else {\n throw new RuntimeException(\"Satop must have a nom or nomvar.\");\n }\n }\n _arg = HyloHelper.getLF_FromChildren(e);\n }\n\n public SatOp(Nominal nom, LF arg) {\n _nominal = nom;\n _arg = arg;\n }\n\n public Nominal getNominal() { return _nominal; }\n public void setNominal(Nominal nominal) { _nominal = nominal; }\n\n public LF getArg() { return _arg; }\n public void setArg(LF arg) { _arg = arg; }\n \n public LF copy() {\n SatOp retval = new SatOp((Nominal)_nominal.copy(), _arg.copy());\n retval._origin = _origin;\n return retval;\n }\n\n public void deepMap(ModFcn mf) {\n _nominal.deepMap(mf);\n _arg.deepMap(mf);\n mf.modify(this);\n }\n\n public boolean occurs(Variable var) {\n return (_nominal.occurs(var) || _arg.occurs(var));\n }\n\n /** Returns true iff the nominal and arg are equal. */\n public boolean equals(Object o) {\n if (o instanceof SatOp\n && _nominal.equals(((SatOp)o)._nominal)\n && _arg.equals(((SatOp)o)._arg)) \n {\n return true;\n } else {\n return false;\n }\n }\n \n public Object unify(Object u, Substitution sub) throws UnifyFailure {\n if (u instanceof HyloFormula) {\n if (u instanceof SatOp) {\n Nominal $nom = (Nominal) Unifier.unify(_nominal, ((SatOp)u)._nominal, sub);\n LF $arg = (LF)Unifier.unify(_arg,((SatOp)u)._arg, sub);\n SatOp retval = new SatOp($nom, $arg);\n retval._origin = _origin;\n return retval;\n }\n else return super.unify(u,sub);\n } else {\n throw new UnifyFailure();\n }\n }\n\n public Object fill(Substitution sub) throws UnifyFailure {\n SatOp retval = new SatOp((Nominal)_nominal.fill(sub), (LF)_arg.fill(sub));\n retval._origin = _origin;\n return retval;\n }\n\n public String toString() { \n boolean includeParens = !(_arg instanceof Op);\n StringBuffer sbuf = new StringBuffer();\n sbuf.append('@').append(_nominal.toString());\n if (includeParens) { sbuf.append('('); }\n sbuf.append(_arg.toString());\n if (includeParens) { sbuf.append(')'); }\n return sbuf.toString();\n }\n\n /**\n * Returns a pretty-printed string of this LF, with the given indent.\n */\n public String prettyPrint(String indent) {\n // calc new indent\n StringBuffer ibuf = new StringBuffer();\n ibuf.append(indent).append(\" \");\n String nomStr = _nominal.toString();\n for (int i = 0; i < nomStr.length(); i++) {\n ibuf.append(' ');\n }\n String newIndent = ibuf.toString();\n // calc string\n boolean includeParens = !(_arg instanceof Op);\n StringBuffer sbuf = new StringBuffer();\n sbuf.append('@').append(nomStr); \n if (includeParens) { sbuf.append('('); }\n sbuf.append(_arg.prettyPrint(newIndent));\n if (includeParens) { sbuf.append(')'); }\n // done\n return sbuf.toString();\n }\n \n \n /** Returns a hash code using the nominal and arg. */\n public int hashCode() { return _nominal.hashCode() + _arg.hashCode(); }\n \n /**\n * Returns a hash code using the given map from vars to ints.\n */\n public int hashCode(TObjectIntHashMap varMap) { \n return _nominal.hashCode(varMap) + _arg.hashCode(varMap); \n }\n \n /**\n * Returns whether this sat op equals the given object \n * up to variable names, using the given maps from vars to ints.\n */\n public boolean equals(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2) {\n if (obj.getClass() != this.getClass()) { return false; }\n SatOp so = (SatOp) obj;\n return _nominal.equals(so._nominal, varMap, varMap2) && \n _arg.equals(so._arg, varMap, varMap2); \n }\n \n /**\n * Returns an XML representation of this LF.\n */\n public Element toXml() {\n Element retval = new Element(\"satop\");\n if (_nominal instanceof NominalAtom) {\n retval.setAttribute(\"nom\", _nominal.toString());\n } else {\n retval.setAttribute(\"nomvar\", ((NominalVar)_nominal).nameWithType());\n }\n Element argElt = _arg.toXml();\n retval.addContent(argElt);\n return retval;\n }\n}\n" }, { "path": "src/opennlp/ccg/hylo/graph/DefaultLFEdgeFactory.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.hylo.graph;\n\n/**\n * A factory for LF edges that creates edges from specified source and target vertices and an edge label.\n * This class provides a default implementation of the {@link LFEdgeFactory} interface for instantiating\n * {@link LFGraph}s.\n * \n * @author Scott Martin\n */\npublic class DefaultLFEdgeFactory implements LFEdgeFactory {\n\t\n\t/**\n\t * Creates an edge from a specified source and target vertex.\n\t * @return A new edge with a null label.\n\t * @see #createLabeledEdge(LFVertex, LFVertex, LFEdgeLabel)\n\t */\n\t@Override\n\tpublic LFEdge createEdge(LFVertex sourceVertex, LFVertex targetVertex) {\n\t\treturn createLabeledEdge(sourceVertex, targetVertex, null);\n\t}\n\n\t/**\n\t * Creates a new labeled, directed edge from a specified vertex pair and edge label.\n\t * @param sourceVertex The source vertex of the new edge.\n\t * @param targetVertex The target vertex of the new edge.\n\t * @param label The label of the new edge.\n\t * @return An instance of {@link LFEdge} with the specfied parameters.\n\t * \n\t * @see LFEdge#LFEdge(LFVertex, LFVertex, LFEdgeLabel)\n\t */\n\t@Override\n\tpublic LFEdge createLabeledEdge(LFVertex sourceVertex, LFVertex targetVertex, LFEdgeLabel label) {\n\t\treturn new LFEdge(sourceVertex, targetVertex, label);\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/hylo/graph/LFEdge.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.hylo.graph;\n\n/**\n * An edge in an {@linkplain LFGraph LF graph}. LF graph edges are directed, containing a \n * {@linkplain #getSource() source} and {@linkplain #getTarget() target} vertex as well as\n * an {@linkplain #getLabel() edge label} representing the type of relation between the two\n * vertices.\n * \n * @author Scott Martin\n */\npublic class LFEdge {\n\n\tfinal LFVertex source, target;\n\tfinal LFEdgeLabel label;\n\t\n\t/**\n\t * Convenience constructor for creating edges with a null edge label. This constructor just\n\t * calls LFEdge(source, target, null).\n\t * @see #LFEdge(LFVertex, LFVertex, LFEdgeLabel)\n\t */\n\tpublic LFEdge(LFVertex source, LFVertex target) {\n\t\tthis(source, target, null);\n\t}\n\t\n\t/**\n\t * Creates a new LF edge with the specified source and target vertices, and edge label.\n\t * @param source The source vertex of the new edge.\n\t * @param target The target vertex of the new edge.\n\t * @param label The label of the new edge, possibly null.\n\t * \n\t * @throws IllegalArgumentException If either source or target is null.\n\t */\n\tpublic LFEdge(LFVertex source, LFVertex target, LFEdgeLabel label) {\n\t\tcheckVertex(source, \"source\");\n\t\tcheckVertex(target, \"target\");\n\t\t\n\t\tthis.source = source;\n\t\tthis.target = target;\n\t\tthis.label = label;\n\t}\n\t\n\tvoid checkVertex(LFVertex v, String name) {\n\t\tif(v == null) {\n\t\t\tthrow new IllegalArgumentException(name + \" is null\");\n\t\t}\n\t}\n\n\t/**\n\t * Gets the edge label, which may be null.\n\t */\n\tpublic LFEdgeLabel getLabel() {\n\t\treturn label;\n\t}\n\n\t/**\n\t * Gets the source vertex.\n\t */\n\tpublic LFVertex getSource() {\n\t\treturn source;\n\t}\n\t\n\t/**\n\t * Gets the target vertex.\n\t */\n\tpublic LFVertex getTarget() {\n\t\treturn target;\n\t}\n\n\t/**\n\t * Tests whether this edge is equal to another by comparing the source and target vertices by using their\n\t * {@link LFVertex#equals(Object)} methods.\n\t * If the label is null, it is considered equivalent to the other edge's label only if\n\t * the other edge's label is also null. Otherwise, the labels are compared using their\n\t * {@link LFEdgeLabel#equals(Object)} method.\n\t * \n\t * @param o The edge to compare this edge to.\n\t */\n\t@Override\n\tpublic boolean equals(Object o) {\n\t\tif(o instanceof LFEdge) {\n\t\t\tLFEdge e = (LFEdge)o;\n\t\t\treturn source.equals(e.source) && target.equals(e.target) \n\t\t\t\t&& (label != null) ? label.equals(e.label) : e.label == null;\n\t\t}\n\t\t\n\t\treturn false;\n\t}\n\n\t/**\n\t * Computes a hash code for this edge based on the hash codes of its vertices and label, assuming the\n\t * label is non-null.\n\t */\n\t@Override\n\tpublic int hashCode() {\n\t\tint h = 37 * source.hashCode() + target.hashCode();\n\t\t\n\t\tif(label != null) {\n\t\t\th += label.hashCode();\n\t\t}\n\t\t\n\t\treturn h;\n\t}\n\t\n\t/**\n\t * Gets a string representation of this edge. For example, if the edge's source is w1@woman,\n\t * its target is w0@the, and its label is Det, this method returns\n\t * w1@woman --Det--> w0@the.\n\t */\n\t@Override\n\tpublic String toString() {\n\t\tStringBuilder sb = new StringBuilder(source.toString());\n\t\tsb.append(\" --\");\n\t\tsb.append((label == null) ? \"(no label)\" : label.toString());\n\t\tsb.append(\"--> \");\n\t\tsb.append(target.toString());\n\t\t\n\t\treturn sb.toString();\n\t}\n}\n\n" }, { "path": "src/opennlp/ccg/hylo/graph/LFEdgeFactory.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.hylo.graph;\n\nimport org.jgrapht.EdgeFactory;\n\n/**\n * A factory for LF edges that creates edges from specified source and target vertices and an edge label.\n * This interface extends the {@link EdgeFactory} interface for the specialized case of\n * directed, labeled LF edges with LF vertices as their nodes. A default implementation\n * is provided in {@link DefaultLFEdgeFactory}.\n * \n * @author Scott Martin\n */\npublic interface LFEdgeFactory extends EdgeFactory {\n\t\n\t/**\n\t * Creates a new labeled, directed edge from a specified vertex pair and edge label.\n\t * @param sourceVertex The source vertex of the new edge.\n\t * @param targetVertex The target vertex of the new edge.\n\t * @param label The label of the new edge.\n\t * @return An instance of {@link LFEdge} with the specified parameters.\n\t * \n\t * @see LFEdge#LFEdge(LFVertex, LFVertex, LFEdgeLabel)\n\t */\n\tpublic LFEdge createLabeledEdge(LFVertex sourceVertex, LFVertex targetVertex, LFEdgeLabel label);\n\n}\n" }, { "path": "src/opennlp/ccg/hylo/graph/LFEdgeLabel.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.hylo.graph;\n\nimport java.util.HashMap;\nimport java.util.Map;\n\nimport opennlp.ccg.hylo.Mode;\n\n/**\n * A label for an {@linkplain LFEdge LF edge}. LF edge labels are drawn from \n * {@link Mode}s, so this class encapsulates one.\n *

    \n * Since certain modes reoccur frequently, this class maintains a cache of\n * modes mapped to edge labels, accessible via {@link #forMode(Mode)}.\n * \n * @author Scott Martin\n */\npublic class LFEdgeLabel {\n\n\tfinal Mode mode;\n\t\n\tprivate static Map labelCache;\n\t\n\t/**\n\t * Creates an LF edge label with the specified mode.\n\t * @param mode The mode representing the type of relation between the vertices.\n\t * @throws IllegalArgumentException if mode is null.\n\t */\n\tpublic LFEdgeLabel(Mode mode) {\n\t\tif(mode == null) {\n\t\t\tthrow new IllegalArgumentException(\"mode is null\");\n\t\t}\n\t\t\n\t\tthis.mode = mode;\n\t}\n\n\t/**\n\t * Gets a cached LF edge label for the specified mode, if one is available. If none\n\t * has been created yet, a new LF edge label is created and cached for later use.\n\t * Access to the cached LF edge labels is synchronized to avoid threading isues.\n\t * \n\t * @param mode The mode to create a label for.\n\t * @return Either a cached LF edge label corresponding to the specified mode, if one\n\t * is available, or a newly created one.\n\t */\n\tpublic static LFEdgeLabel forMode(Mode mode) {\n\t\tsynchronized(LFEdgeLabel.class) {\n\t\t\tLFEdgeLabel l = null;\n\t\t\t\n\t\t\tif(labelCache == null) {\n\t\t\t\t labelCache = new HashMap();\n\t\t\t}\n\t\t\telse {\t\t\t\n\t\t\t\tl = labelCache.get(mode);\n\t\t\t}\n\t\t\t\n\t\t\tif(l == null) {\n\t\t\t\tl = new LFEdgeLabel(mode);\n\t\t\t\tlabelCache.put(mode, l);\n\t\t\t}\n\t\t\t\n\t\t\treturn l;\n\t\t}\n\t}\n\t\n\t/**\n\t * Gets the name of this label, as specified by its underlying \n\t * {@linkplain Mode mode}.\n\t * @return The value of getMode().getName().\n\t */\n\tpublic String getName() {\n\t\treturn mode.getName();\n\t}\n\t\n\t/**\n\t * Gets the mode underlying this edge label.\n\t * @return The mode specified at creation.\n\t * @see #LFEdgeLabel(Mode)\n\t */\n\tpublic Mode getMode() {\n\t\treturn mode;\n\t}\n\n\t/**\n\t * Computes a hash code for this LF edge label based on the hash code of its\n\t * underlying mode.\n\t */\n\t@Override\n\tpublic int hashCode() {\n\t\treturn 31 * mode.hashCode();\n\t}\n\n\t/**\n\t * Tests whether this LF edge label is equivalent to another by comparing their\n\t * modes, using their {@link Mode#equals(Object)} methods.\n\t */\n\t@Override\n\tpublic boolean equals(Object obj) {\n\t\treturn (obj instanceof LFEdgeLabel) && mode.equals(((LFEdgeLabel)obj).mode);\n\t}\n\n\t/**\n\t * Gets a string representation of this LF edge label.\n\t * @return The value of getName().\n\t */\n\t@Override\n\tpublic String toString() {\n\t\treturn getName();\n\t}\n}\n" }, { "path": "src/opennlp/ccg/hylo/graph/LFGraph.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.hylo.graph;\n\nimport java.util.HashMap;\nimport java.util.LinkedHashSet;\nimport java.util.Map;\nimport java.util.Set;\n\nimport opennlp.ccg.hylo.Flattener;\nimport opennlp.ccg.hylo.Nominal;\nimport opennlp.ccg.synsem.LF;\n\nimport org.jgrapht.graph.DirectedMultigraph;\n\n/**\n * A graph representation of an {@link LF}. By default, LF graphs are empty. But populated LF graphs can\n * be easily created using the {@link LFGraphFactory} class.\n *

    \n * This class extends the \n * {@link DirectedMultigraph} class from the org.jgrapht.graph package that provides some\n * specialized methods for dealing with OpenCCG LFs. It provides some flexibility for edge creation\n * by allowing an {@link LFEdgeFactory} to be specified at creation. The original LF structure is also\n * reflected in the {@link #getLFAncestry(LFVertex)}, {@link #highestLFAncestors()}, and\n * {@link #highestLFAncestorOf(LFVertex)} methods.\n *

    \n * As a convenience, the method {@link #findVertexByNominal(Nominal)} provides access to the vertex\n * corresponding to a given nominal. This is implemented by a hash map so that the lookup takes place in \n * constant time. \n * \n * @see JGraphT website\n * \n * @author Scott Martin\n */\npublic class LFGraph extends DirectedMultigraph {\n\n\tprivate static final long serialVersionUID = 1L; //TODO make this really serializable?\n\t\n\t/**\n\t * The map reflecting LF ancestry by giving the highest ancestor for a specified\n\t * vertex, or null if it is the highest ancestor.\n\t */\n\tprotected Map highestAncestorMap = new HashMap();\n\t\n\t\n\t/**\n\t * Convenience cache of nominals mapped to vertices.\n\t * @see #findVertexByNominal(Nominal)\n\t */\n\tprotected Map nominalVertexMap = new HashMap();\n\t\n\t/**\n\t * Creates a new, empty LF graph that uses an implementation of {@link LFEdgeFactory}\n\t * as its edge factory.\n\t * @see #LFGraph(LFEdgeFactory)\n\t */\n\tpublic LFGraph() {\n\t\tthis(LFGraphFactory.DEFAULT_EDGE_FACTORY);\n\t}\n\t\n\t/**\n\t * Creates a new, empty LF graph using the specified edge factory.\n\t * @param lfEdgeFactory The edge factory to use for creating edges.\n\t * \n\t * @see LFGraphFactory\n\t */\n\tpublic LFGraph(LFEdgeFactory lfEdgeFactory) {\n\t\tsuper(lfEdgeFactory);\n\t}\n\t\n\t/**\n\t * Gets the {@link LFEdgeFactory} used to create this LF graph.\n\t * @return The value of {@link #getEdgeFactory()}, cast to an \n\t * {@link LFEdgeFactory}.\n\t */\n\tpublic LFEdgeFactory getLFEdgeFactory() {\n\t\treturn (LFEdgeFactory)getEdgeFactory();\n\t}\n\n\t/**\n\t * Adds a new labeled edge based on the specified source and target vertices and edge label by calling\n\t * getLFEdgeFactory().createLabeledEdge(source, target, label). The new\n\t * edge is added to this graph's {@linkplain #edgeSet() edge set}.\n\t * @param source The source vertex of the new edge.\n\t * @param target The target vertex of the new edge.\n\t * @param label The label of the new edge.\n\t * @return The newly created edge if it was successfully added to this graph, null otherwise.\n\t */\n\tpublic LFEdge addLabeledEdge(LFVertex source, LFVertex target, LFEdgeLabel label) {\n\t\tLFEdge e = getLFEdgeFactory().createLabeledEdge(source, target, label);\n\t\treturn addEdge(source, target, e) ? e : null;\n\t}\n\t\n\t/**\n\t * Finds a vertex by its nominal. This method does a lookup on a dictionary mapping each vertex's \n\t * {@linkplain LFVertex#getNominal() nominal} to members of the{@link #vertexSet()}. \n\t * @param nominal The nominal to test for.\n\t * @return A vertex whose nominal is equivalent to the one specified, or null if none is present.\n\t */\n\tpublic LFVertex findVertexByNominal(Nominal nominal) { \n\t\treturn nominalVertexMap.get(nominal);\n\t}\n\n\t/**\n\t * Overrides the superclass method to add a mapping from the vertex's {@linkplain LFVertex#getNominal() nominal}\n\t * to the vertex itself, for later retrieval via {@link #findVertexByNominal(Nominal)}.\n\t */\n\t@Override\n\tpublic boolean addVertex(LFVertex v) {\n\t\tboolean b = super.addVertex(v); // give this a chance first\n\t\t\n\t\tif(b) {\n\t\t\tnominalVertexMap.put(v.nominal, v);\n\t\t}\n\t\t\n\t\treturn b;\n\t}\n\n\t/**\n\t * Overrides the superclass method to remove any existing mapping from some nominal to the specified vertex.\n\t * @see #addVertex(LFVertex)\n\t * @see #findVertexByNominal(Nominal)\n\t */\n\t@Override\n\tpublic boolean removeVertex(LFVertex v) {\n\t\tboolean b = super.removeVertex(v); // invoke this first\n\t\n\t\tif(b) {\n\t\t\tnominalVertexMap.values().remove(v);\n\t\t}\n\t\t\n\t\treturn b;\n\t}\n\n\t/**\n\t * Gets the highest LF ancestor of the specified vertex, as determined by the LF structure.\n\t * @param vertex The vertex to get the highest ancestor for.\n\t * @return The highest ancestor of the specified vertex, or null if it is the highest in its\n\t * ancestry.\n\t * @see Flattener#getHighestParentMap()\n\t */\n\tpublic LFVertex highestLFAncestorOf(LFVertex vertex) {\n\t\treturn highestAncestorMap.get(vertex);\n\t}\n\t\n\t/**\n\t * Gets the LF ancestry corresponding to the specified vertex.\n\t * \n\t * @return The parents of the specified vertex in the LF ancestry.\n\t * @see #highestLFAncestorOf(LFVertex) \n\t * @see Flattener#getHighestParentMap()\n\t */\n\tpublic Set getLFAncestry(LFVertex vertex) {\n\t\tLFVertex a = highestLFAncestorOf(vertex);\n\t\tSet as = new LinkedHashSet();\n\t\t\n\t\tfor(LFVertex v : vertexSet()) {\n\t\t\tif(!v.equals(vertex) && highestLFAncestorOf(v).equals(a)) {\n\t\t\t\tas.add(v);\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn as;\n\t}\n\t\n\t/**\n\t * Gets the vertex or vertices that are at the top of the LF ancestry hierarchy. \n\t * @return The set of vertices v for which {@link #highestLFAncestorOf(LFVertex)} returns\n\t * null.\n\t * @see Flattener#getHighestParentMap()\n\t */\n\tpublic Set highestLFAncestors() {\n\t\tSet ps = new LinkedHashSet();\n\t\t\n\t\tfor(LFVertex v : vertexSet()) {\n\t\t\tif(highestLFAncestorOf(v) == null) {\n\t\t\t\tps.add(v);\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn ps;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/hylo/graph/LFGraphFactory.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.hylo.graph;\n\nimport static opennlp.ccg.hylo.HyloHelper.isAttrPred;\nimport static opennlp.ccg.hylo.HyloHelper.isElementaryPredication;\nimport static opennlp.ccg.hylo.HyloHelper.isLexPred;\nimport static opennlp.ccg.hylo.HyloHelper.isRelPred;\n\nimport java.util.List;\nimport java.util.Map;\n\nimport opennlp.ccg.hylo.Diamond;\nimport opennlp.ccg.hylo.Flattener;\nimport opennlp.ccg.hylo.HyloHelper;\nimport opennlp.ccg.hylo.Nominal;\nimport opennlp.ccg.hylo.Proposition;\nimport opennlp.ccg.hylo.SatOp;\nimport opennlp.ccg.realize.Realizer;\nimport opennlp.ccg.synsem.LF;\n\nimport org.jdom.Element;\nimport org.jdom.input.DOMBuilder;\n\n/**\n * Factory class for creating LF graphs based on {@link LF}s and XML elements \n * that represent them. Graph factories cannot be instantiated, however, their\n * methods for building LF graphs are statically accessible.\n * \n * @author Scott Martin\n * @see LFGraph\n * @see LFGraphFactory#newGraphFrom(LF)\n */\npublic class LFGraphFactory {\n\t\n\t/**\n\t * An instance of {@link DefaultLFEdgeFactory}.\n\t */\n\tpublic static final LFEdgeFactory DEFAULT_EDGE_FACTORY = new DefaultLFEdgeFactory();\n\tstatic final DOMBuilder DOM_BUILDER = new DOMBuilder();\n\tstatic final Flattener FLATTENER = new Flattener();\n\t\n\tprivate LFGraphFactory() {\n\t\t// can't instantiate\n\t}\n\t\n\t/**\n\t * Builds a new LF graph based on the representation provided in the specified w3c XML element.\n\t * @param lfElement The XML element with root element tagged lf.\n\t * @return The value of {@link #newGraphFrom(Element)} after using a {@link DOMBuilder} to create a\n\t * jdom document.\n\t * \n\t * @see #newGraphFrom(Element)\n\t */\n\tpublic static LFGraph newGraphFrom(org.w3c.dom.Element lfElement) {\n\t\treturn newGraphFrom(DOM_BUILDER.build(lfElement));\n\t}\n\t\n\t/**\n\t * Creates a new LF graph from the corresponding representation contained in the specified \n\t * jdom XML element.\n\t * @param lfElement The jdom element containing the representation of the LF.\n\t * @return An LF graph build from an {@link LF} object obtained by calling \n\t * {@link Realizer#getLfFromElt(Element)}.\n\t * \n\t * @see LFGraphFactory#newGraphFrom(LF)\n\t */\n\tpublic static LFGraph newGraphFrom(Element lfElement) {\n\t\treturn newGraphFrom(Realizer.getLfFromElt(lfElement));\n\t}\n\t\n\t/**\n\t * Creates a new LF graph based on the specified LF object. The LF object is first flattened, and then\n\t * its LF ancestry structure is obtained by calling {@link Flattener#getHighestParentMap()}.\n\t *

    \n\t * This method makes two passes over the list of {@link SatOp}s obtained by flattening the specified LF.\n\t * The first pass adds vertices to the graph for every lexical predication, as determined by calling\n\t * {@link HyloHelper#isLexPred(LF)} on the {@linkplain SatOp#getNominal() SatOp's nominal} and\n\t * {@linkplain SatOp#getArg() proposition argument}.\n\t *

    \n\t * The second pass proceeds by cases, depending on the nature of the SatOp in question:\n\t *

    \n\t * \t
    Lexical predications
    \n\t * \t
    cause the new LF graph to be updated with the corresponding LF ancestry, as determined by\n\t * \t\t{@link Flattener#getHighestParentMap()}.
    \n\t *
    Relation predications
    \n\t *
    cause a new {@link LFEdge} to be added to the LF graph based on the\n\t * \t{@linkplain SatOp#getArg() SatOp's argument} and {@linkplain Diamond#getMode() the argument's \n\t * \tmode}.
    \n\t *
    Attribute-value predications
    \n\t *
    cause the vertex corresponding to the {@linkplain SatOp#getNominal() SatOp's nominal} to have\n\t * \tattributes {@linkplain LFVertex#setAttribute(opennlp.ccg.hylo.Mode, Proposition) added} based on\n\t * \tthe {@linkplain SatOp#getArg() SatOp's argument}.
    \n\t *
    \n\t * where the nature of the SatOp in question is determined using {@link HyloHelper#isLexPred(LF)},\n\t * {@link HyloHelper#isRelPred(LF)}, and {@link HyloHelper#isAttrPred(LF)}.\n\t * \n\t * @param lf The LF object to build an LF graph for.\n\t * @return A new LF graph whose vertices represent the nominals in the LF's flattened representation and\n\t * whose edges represent its relation predications.\n\t * @throws IllegalArgumentException If lf is null.\n\t */\n\tpublic static LFGraph newGraphFrom(LF lf) {\n\t\tif(lf == null) {\n\t\t\tthrow new IllegalArgumentException(\"lf is null\");\n\t\t}\n\t\t\n\t\tLFGraph g = new LFGraph(DEFAULT_EDGE_FACTORY);\n\t\t\n\t\tFlattener f = new Flattener();\n\t\tList satOps = f.flatten(lf);\n\t\tMap ancestorMap = f.getHighestParentMap();\n\t\t\n\t\tfor(SatOp so : satOps) { // first pass adds vertices\n\t\t\tif(isLexPred(so)) {\n\t\t\t\tg.addVertex(new LFVertex(so.getNominal(), (Proposition)so.getArg()));\n\t\t\t}\n\t\t}\n\t\t\n\t\tfor(SatOp so : satOps) { // second pass adds edges and attributes, sets highest parent (if any)\n\t\t\tif(isElementaryPredication(so)) {\n\t\t\t\tNominal soNom = so.getNominal();\n\t\t\t\tLFVertex source = g.findVertexByNominal(soNom);\n\t\t\t\t\n\t\t\t\t// check if node is not yet added (not a lex. pred.)\n\t\t\t\tif(source == null) {\n\t\t\t\t\tsource = new LFVertex(soNom);\n\t\t\t\t\tg.addVertex(source);\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tif(isLexPred(so)) {\n\t\t\t\t\tNominal parent = ancestorMap.get(source.nominal);\n\t\t\t\t\tif(parent != null) {\n\t\t\t\t\t\tg.highestAncestorMap.put(source, g.findVertexByNominal(parent));\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\telse if(isRelPred(so)) {\n\t\t\t\t\tDiamond d = (Diamond)so.getArg();\n\t\t\t\t\t\n\t\t\t\t\tNominal dArg = (Nominal)d.getArg();\n\t\t\t\t\tLFVertex target = g.findVertexByNominal(dArg);\n\t\t\t\t\t\n\t\t\t\t\tif(target == null) { \n\t\t\t\t\t\ttarget = new LFVertex(dArg);\n\t\t\t\t\t\tg.addVertex(target);\n\t\t\t\t\t}\n\t\t\t\t\t\n\t\t\t\t\tg.addLabeledEdge(source, target, LFEdgeLabel.forMode(d.getMode()));\n\t\t\t\t}\n\t\t\t\telse if(isAttrPred(so)) {\n\t\t\t\t\tDiamond d = (Diamond)so.getArg();\n\t\t\t\t\tsource.addAttribute(d.getMode(), (Proposition)d.getArg());\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn g;\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/hylo/graph/LFVertex.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.hylo.graph;\n\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.Map;\nimport java.util.Set;\n\nimport opennlp.ccg.hylo.Mode;\nimport opennlp.ccg.hylo.Nominal;\nimport opennlp.ccg.hylo.Proposition;\nimport opennlp.ccg.hylo.SatOp;\n\n/**\n * A vertex in an {@link LFGraph}. Vertices are based on {@link SatOp}s, encapsulating their\n * {@linkplain SatOp#getNominal() nominal} and {@linkplain SatOp#getArg() proposition argument} (or\n * null if there is no associated proposition).\n * Vertices also maintain a list of attribute/value pairs representing their associated attribute/value\n * predications.\n *

    \n * The {@linkplain #getIndex() index} and {@linkplain #getType() type} of an LF vertex are determined by\n * the specified nominal's {@linkplain Nominal#getName() name}. For example, if a vertex's \n * {@linkplain #getName() name} is w12, then {@link #getIndex()} returns 12 and\n * {@link #getType()} returns {@link LFVertexType#WORD}.\n * \n * @author Scott Martin\n * @see LFGraph\n * @see LFVertexType\n * @see SatOp\n */\npublic class LFVertex {\n\n\tfinal Nominal nominal;\n\tfinal Proposition proposition;\n\t\n\t/**\n\t * This vertex's attribute map.\n\t */\n\tprotected Map attributes;\n\t\n\tprivate Integer index;\n\tprivate LFVertexType type;\n\t\n\t/**\n\t * Creates a new LF vertex based on the specified SatOp.\n\t * @param satOp The SatOp to use for creating a new LF vertex, using its nominal and argument.\n\t * @see #LFVertex(Nominal, Proposition)\n\t */\n\tpublic LFVertex(SatOp satOp) {\n\t\tthis(satOp.getNominal(), (Proposition)satOp.getArg());\n\t}\n\t\n\t/**\n\t * Creates a new LF vertex based on the specified nominal, with a null proposition.\n\t * @see #LFVertex(Nominal, Proposition) \n\t */\n\tpublic LFVertex(Nominal nominal) {\n\t\tthis(nominal, null);\n\t}\n\t\n\t/**\n\t * Creates a new LF vertex based on the specified nominal and proposition.\n\t * @see #LFVertex(Nominal, Proposition, Map)\n\t */\n\tpublic LFVertex(Nominal nominal, Proposition proposition) {\n\t\tthis(nominal, proposition, null);\n\t}\n\t\n\t/**\n\t * Creates a new LF vertex based on the specified nominal and proposition, with the specified attribute\n\t * map (which can be null).\n\t * @throws IllegalArgumentException if nominal is null.\n\t */\n\tpublic LFVertex(Nominal nominal, Proposition proposition, Map attributes) {\n\t\tif(nominal == null) {\n\t\t\tthrow new IllegalArgumentException(\"nominal is null\");\n\t\t}\n\t\t\n\t\tthis.nominal = nominal;\n\t\tthis.proposition = proposition;\n\t\tthis.attributes = attributes;\n\t}\n\n\t/**\n\t * Gets this LF vertex's name, determined by the name of its {@linkplain #getNominal() nominal}.\n\t * @return The value of getNominal().getName().\n\t */\n\tpublic String getName() {\n\t\treturn getNominal().getName();\n\t}\n\t\n\t/**\n\t * Gets this LF vertex's associated predicate, the name of its {@linkplain #getProposition() proposition}.\n\t * @return The value of getProposition().getName(), if this vertex's proposition is non-null,\n\t * and null otherwise.\n\t */\n\tpublic String getPredicate() {\n\t\tProposition p = getProposition();\n\t\treturn (p == null) ? null : p.getName();\n\t}\n\t\n\t/**\n\t * Gets this LF vertex's associated nominal.\n\t */\n\tpublic Nominal getNominal() {\n\t\treturn nominal;\n\t}\n\n\t/**\n\t * Gets this LF vertex's associated proposition.\n\t * @return Possibly null, if no proposition was provided at creation.\n\t * @see #LFVertex(Nominal)\n\t */\n\tpublic Proposition getProposition() {\n\t\treturn proposition;\n\t}\n\t\n\t/**\n\t * Gets the type of this LF vertex, as determined by the prefix of its {@linkplain #getNominal() nominal}'s\n\t * name. For example, if this vertex's {@linkplain Nominal#getName() name} is x3, then\n\t * this method returns\n\t * {@link LFVertexType#fromPrefix(String) LFVertexType.fromPrefix}('x') == {@link LFVertexType#NONWORD}.\n\t * @return The value of {@link LFVertexType#fromPrefix(String)} for this vertex's \n\t * {@linkplain #getNominal() nominal}'s {@linkplain Nominal#getName() name}.\n\t * @see LFVertexType\n\t */\n\tpublic LFVertexType getType() {\n\t\treturn (type == null) ? (type = LFVertexType.fromPrefix(nominal.getName())) : type;\n\t}\n\n\t/**\n\t * Gets the word index associated with this LF vertex, as determined by {@link #parseVertexIndex(Nominal)}.\n\t */\n\tpublic Integer getIndex() {\n\t\treturn (index == null) ? (index = parseVertexIndex(nominal)) : index;\n\t}\n\t\n\t/**\n\t * Parses the word index associated with the specified nominal, obtained by parsing its name. For\n\t * example, if the specified nominal's name is x9, this method returns the integer \n\t * 9.\n\t * @param nominal The nominal to find the word index for.\n\t * @return The integer index corresponding to the specified nominal, determined by parsing its\n\t * {@linkplain Nominal#getName() name}.\n\t */\n\tpublic static Integer parseVertexIndex(Nominal nominal) {\n\t\tString nm = nominal.getName();\n\t\tint colidx = nm.indexOf(':');\n\t\tString s = (colidx == -1) ? nm : nm.substring(0, colidx);\n\t\t\n\t\treturn Integer.parseInt(s.substring(1));\n\t}\n\t\n\t/**\n\t * Gets the attribute map associated with this LF vertex. Note that the returned map is\n\t * not modifiable; to modify a vertex's attributes, the methods\n\t * {@link #setAttribute(Mode, Proposition)} and {@link #removeAttribute(Mode)}\n\t * should be used.\n\t * @return An unmodifiable copy of the attribute map encapsulated by this vertex, or\n\t * {@link Collections#EMPTY_MAP} if no attributes are present.\n\t * @see Collections#unmodifiableMap(Map)\n\t */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic Map getAttributeMap() {\n\t\treturn (attributes == null) ? Collections.EMPTY_MAP : Collections.unmodifiableMap(attributes);\n\t}\n\t\n\t/**\n\t * Gets the names of this LF vertex's attributes. Note that the returned set is not modifiable;\n\t * the methods {@link #setAttribute(Mode, Proposition)} and {@link #removeAttribute(Mode)} should\n\t * be used to modify this vertex's attribute/value pairs.\n\t * @return The value of getAttributeMap().keySet(), or {@link Collections#EMPTY_SET} if\n\t * no attributes are present.\n\t * @see Collections#unmodifiableSet(Set)\n\t */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic Set attributeNames() {\n\t\treturn (attributes == null) ? Collections.EMPTY_SET : Collections.unmodifiableSet(attributes.keySet());\n\t}\n\t\n\t/**\n\t * Tests whether this vertex contains an attribute with the associated attribute name.\n\t * @param attributeName The name of the attribute to test for.\n\t * @return true if this vertex has an attribute named attributeName.\n\t * @see #attributeNames()\n\t */\n\tpublic boolean containsAttribute(Mode attributeName) {\n\t\treturn (attributes != null && attributes.containsKey(attributeName));\n\t}\n\t\n\t/**\n\t * Gets the value of the attribute with the specified mode name.\n\t * @param attributeName The attribute name to retrieve a value for.\n\t * @return The associated attribute value, or null if none is present.\n\t */\n\tpublic Proposition getAttributeValue(Mode attributeName) {\n\t\treturn (attributes == null) ? null : attributes.get(attributeName);\n\t}\n\t\n\t/**\n\t * Adds an attribute/value to this vertex's attributes. \n\t * @param attributeName The name of the new attribute.\n\t * @param value The value of the new attribute.\n\t * @return True if this vertex's attribute/value map changed as a result of the call because\n\t * either (1) no attribute named attributeName was present or (2)\n\t * the value associated with attributeName changed (was different from value).\n\t * @see #setAttribute(Mode, Proposition)\n\t */\n\tpublic boolean addAttribute(Mode attributeName, Proposition value) {\n\t\t// works even when setAttribute() returns null\n\t\treturn !value.equals(setAttribute(attributeName, value)); \n\t}\n\t\n\t/**\n\t * Sets the attribute associated with the specified mode name to the specified proposition value.\n\t * @param attributeName The key to set the value for.\n\t * @param value The value that will be associated with attributeName.\n\t * @return The value previously associated with attributeName, or null if no value\n\t * was previously associated.\n\t * @see Map#put(Object, Object)\n\t */\n\tpublic Proposition setAttribute(Mode attributeName, Proposition value) {\n\t\tif(attributes == null) {\n\t\t\tattributes = new HashMap();\n\t\t}\n\t\t\n\t\treturn attributes.put(attributeName, value);\n\t}\n\t\n\t/**\n\t * Removes and returns the value associated with the specified attribute name.\n\t * @param attributeName The name to remove the value for.\n\t * @return The value previously associated with attributeName, or null if no value\n\t * was associated with it.\n\t */\n\tpublic Proposition removeAttribute(Mode attributeName) {\n\t\treturn (attributes == null) ? null : attributes.remove(attributeName); \n\t}\n\n\t/**\n\t * Gets a hash code for this vertex based on its nominal and proposition, if the proposition is\n\t * non-null.\n\t */\n\t@Override\n\tpublic int hashCode() {\n\t\tint i = 37 * nominal.hashCode();\n\t\t\n\t\tif(proposition != null) {\n\t\t\ti += proposition.hashCode();\n\t\t}\n\t\t\n\t\t// Don't include attributes in hash code calculation. This could cause problems if a vertex is\n\t\t// added to a collection that relies on hashing, and then the attributes are later modified.\n\t\t\n\t\treturn i;\n\t}\n\n\t/**\n\t * Tests whether this LF vertex is equal to another by comparing their nominals and possibly also their\n\t * propositions and attributes, if they are non-null/non-empty.\n\t * @see #getNominal()\n\t * @see #getProposition()\n\t * @see #getAttributeMap()\n\t */\n\t@Override\n\tpublic boolean equals(Object obj) {\n\t\tif(obj instanceof LFVertex) {\n\t\t\tLFVertex v = (LFVertex)obj;\n\t\t\treturn nominal.equals(v.nominal) \n\t\t\t\t\t&& ((proposition == null) ? v.proposition == null : proposition.equals(v.proposition))\n\t\t\t\t\t&& ((attributes == null) ? v.attributes == null : attributes.equals(v.attributes));\n\t\t}\n\t\t\n\t\treturn false;\n\t}\n\n\t/**\n\t * Gets a string representation of this LF vertex. For example, if this vertex's name is w9,\n\t * its proposition's name is walk, and its attribute map contains num=sg and\n\t * det=nil, this method returns w9@walk {num=sg, det=nil}.\n\t */\n\t@Override\n\tpublic String toString() {\n\t\tStringBuilder sb = new StringBuilder(nominal.getName());\n\t\t\n\t\tif(proposition != null) {\n\t\t\tsb.append('@');\n\t\t\tsb.append(proposition.getName());\n\t\t}\n\t\t\n\t\tif(attributes != null && !attributes.isEmpty()) {\n\t\t\tsb.append(' ');\n\t\t\tsb.append(attributes.toString());\n\t\t}\n\t\t\t\n\t\treturn sb.toString();\n\t}\n}\n" }, { "path": "src/opennlp/ccg/hylo/graph/LFVertexType.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.hylo.graph;\n\n/**\n * A set of enum constants indicating the type of an LF vertex. The type of a vertex is\n * also signaled by the character prefix of its name, as in w9 and x3.\n * \n * @author Scott Martin\n */\npublic enum LFVertexType {\n\t\n\t/**\n\t * The type of vertices representing words, with prefix w.\n\t */\n\tWORD('w'),\n\t\n\t/**\n\t * The type of vertices representing nonwords, prefixed x.\n\t */\n\tNONWORD('x');\n\t\n\t/**\n\t * The prefix character for this vertex type.\n\t */\n\tfinal Character prefix;\n\t\n\tprivate LFVertexType(Character prefix) {\n\t\tthis.prefix = prefix;\n\t}\n\t\n\t/**\n\t * Gets the naming prefix used by this vertex type.\n\t */\n\tpublic Character getPrefix() {\n\t\treturn prefix;\n\t}\n\t\n\t/**\n\t * Gets the LF vertex type corresponding to the specified prefix string.\n\t * \n\t * @param string The prefix string.\n\t * @return The value of {@link #fromPrefix(Character)} for the first character in the\n\t * specified string.\n\t * \n\t * @see #fromPrefix(Character)\n\t */\n\tpublic static LFVertexType fromPrefix(String string) {\n\t\treturn fromPrefix(Character.valueOf(string.charAt(0)));\n\t}\n\t\n\t/**\n\t * Gets the LF vertex type corresponding to the specified character prefix.\n\t * @param prefix The prefix character.\n\t * @return The vertex type with the specified prefix, as determined by\n\t * {@link #getPrefix()}, or {@link #NONWORD} if there is no vertex type for the given\n\t * prefix character.\n\t */\n\tpublic static LFVertexType fromPrefix(Character prefix) {\n\t\tfor(LFVertexType type : values()) {\n\t\t\tif(type.prefix.equals(prefix)) {\n\t\t\t\treturn type;\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn NONWORD;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/hylo/graph/package.html", "content": "\n\t\n

    Provides representations of {@link opennlp.ccg.synsem.LF}s as traversible graphs based\non the JGraphT package, as well as customized filters for\nsets of edges in those graphs.

    \n\n

    {@link opennlp.ccg.hylo.graph.LFGraph}s can be created on their own, or from an existing\n{@link opennlp.ccg.synsem.LF} (or XML structure representing one) using the\n{@link opennlp.ccg.hylo.graph.LFGraphFactory}. LF graphs are also\ncustomizable in that different edge factories can be specified by implementing\n{@link opennlp.ccg.hylo.graph.LFEdgeFactory} (a default implementation is provided in\n{@link opennlp.ccg.hylo.graph.DefaultLFEdgeFactory}).

    \n\n\n\n" }, { "path": "src/opennlp/ccg/lexicon/DataItem.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge and Gann Bierner\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.lexicon;\n\nimport org.jdom.*;\n\n/**\n * Data structure for storing information about a lexical entry. Specifically\n * used by LMR grammars.\n *\n * @author Jason Baldridge\n * @version $Revision: 1.2 $, $Date: 2007/12/17 20:02:23 $\n */\npublic class DataItem {\n private String stem = \"\";\n private String pred = \"\";\n \n public DataItem() {}\n public DataItem (String s, String p) {\n\t\tstem = s;\n\t\tpred = p;\n }\n\n \n public DataItem(Element datael) {\n\t\tstem = datael.getAttributeValue(\"stem\");\t\n\t\n\t\tpred = datael.getAttributeValue(\"pred\");\t\n\t\tif (null == pred) {\n\t\t pred = stem;\n\t\t}\n }\n\n public void setStem(String s) { stem = s; }\n public void setPred(String s) { pred = s; }\n\n public String getStem() { return stem; }\n public String getPred() { return pred; }\n}\n" }, { "path": "src/opennlp/ccg/lexicon/DefaultTokenizer.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004-5 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport opennlp.ccg.grammar.Grammar;\nimport opennlp.ccg.util.*;\n\nimport java.text.*;\nimport java.util.*;\nimport javax.xml.datatype.*;\nimport gnu.trove.*;\n\n/**\n * DefaultTokenizer provides a default implementation of the \n * Tokenizer interface.\n *\n * @author Michael White\n * @version $Revision: 1.32 $, $Date: 2010/12/09 04:58:12 $\n **/\n\npublic class DefaultTokenizer implements Tokenizer {\n\n // date format with pattern yyyy.MM.dd, strict parsing\n private DateFormat dateFormat = null;\n\n // date format with pattern *.MM.dd, strict parsing\n private DateFormat dateFormatNoYear = null;\n\n // time format with pattern HH:mm, strict parsing\n private DateFormat timeFormat = null;\n \n // factory for parsing durations, in format \"PnYnMnDTnHnMnS\", as defined in XML Schema 1.0 section 3.2.6.1\n private DatatypeFactory datatypeFactory = null;\n \n /**\n * Map from special token semantic classes to special token constants. \n * The map is initialized in the constructor, where \n * the standard constants (eg Tokenizer.DATE_CLASS and Tokenizer.DATE_VAL) are added.\n */\n protected Map specialTokenMap = null;\n \n /** \n * A set containing semantic classes to replace words with for language models.\n * Equality is checked with identity, for use with interned strings.\n */\n @SuppressWarnings(\"unchecked\")\n\tprotected Set replacementSemClasses = new THashSet(new TObjectIdentityHashingStrategy());\n\n\n /**\n * Constructor.\n */\n public DefaultTokenizer() {\n // init date, time formats\n dateFormat = new SimpleDateFormat(\"yyyy.MM.dd\", Locale.ENGLISH);\n dateFormat.setLenient(false);\n dateFormatNoYear = new SimpleDateFormat(\"*.MM.dd\", Locale.ENGLISH);\n dateFormatNoYear.setLenient(false);\n timeFormat = new SimpleDateFormat(\"HH:mm\", Locale.ENGLISH);\n timeFormat.setLenient(false);\n // init data type factory\n try {\n datatypeFactory = DatatypeFactory.newInstance();\n } catch (DatatypeConfigurationException exc) {\n throw (RuntimeException) new RuntimeException().initCause(exc);\n }\n // init special token map\n specialTokenMap = new HashMap();\n specialTokenMap.put(Tokenizer.DATE_CLASS, Tokenizer.DATE_VAL);\n specialTokenMap.put(Tokenizer.TIME_CLASS, Tokenizer.TIME_VAL);\n specialTokenMap.put(Tokenizer.NUM_CLASS, Tokenizer.NUM_VAL);\n specialTokenMap.put(Tokenizer.AMT_CLASS, Tokenizer.AMT_VAL);\n specialTokenMap.put(Tokenizer.DUR_CLASS, Tokenizer.DUR_VAL);\n specialTokenMap.put(Tokenizer.NE_CLASS, Tokenizer.NE_VAL);\n }\n \n \n /**\n * Adds a semantic class to replace words with for language models.\n */\n public void addReplacementSemClass(String semClass) {\n replacementSemClasses.add(semClass.intern());\n }\n \n /** \n * Returns whether the given semantic class is one to replace words with for language models.\n * The sem class is assumed to have been interned.\n */\n public boolean isReplacementSemClass(String semClass) {\n return replacementSemClasses.contains(semClass);\n }\n\n \n /**\n * Parses an input string into a list of words, \n * including any explicitly given factors, \n * and the semantic class of special tokens.\n * Tokens are parsed into words using parseToken with the strictFactors\n * flag set to false.\n */\n public List tokenize(String s) { return tokenize(s, false); }\n\n /**\n * Parses an input string into a list of words, \n * including any explicitly given factors, \n * and the semantic class of special tokens.\n * Tokens are parsed into words using parseToken, according to the given \n * flag for whether to parse factors strictly.\n * The string is assumed to have white-space delimited tokens.\n */\n public List tokenize(String s, boolean strictFactors) {\n List retval = new ArrayList();\n StringTokenizer st = new StringTokenizer(s);\n while (st.hasMoreTokens()) { retval.add(parseToken(st.nextToken(), strictFactors)); }\n return retval;\n }\n\n \n /** \n * Parses a token into a word, including any explicitly given factors \n * and the semantic class of special tokens.\n * Parsing is performed using parseToken with the strictFactors\n * flag set to false.\n */\n public Word parseToken(String token) { return parseToken(token, false); }\n \n /** \n * Parses a token into a word, including any explicitly given factors \n * and the semantic class of special tokens, according to the given \n * flag for whether to parse factors strictly.\n * Recognized pitch accents may be appended to the word form with an underscore.\n * If the strictFactors flag is true, then colons are always assumed to \n * separate attribute-value pairs, and hyphens are always assumed to \n * separate attributes from values, and thus colons or hyphens not used as separators \n * must be escaped. \n * If the strictFactors flag is false, then there must be at least one colon \n * and at least one hyphen in the token to trigger parsing of factors, \n * in which case colons or hyphens not used as separators \n * must again be escaped; otherwise, colons or hyphens on their own may \n * appear without escaping in the word form.\n * After splitting the token into factors, it is unescaped.\n */\n public Word parseToken(String token, boolean strictFactors) {\n // init\n String form = token;\n String pitchAccent = null;\n List> attrValPairs = null;\n String stem = null;\n String POS = null;\n String supertag = null;\n String semClass = null;\n // handle colon-separated attr-val pairs\n int colonPos = token.indexOf(':');\n int hyphenPos = token.indexOf('-');\n if (strictFactors || (colonPos > 0 && hyphenPos > 0)) {\n // deal with special cases before first colon, if any\n String suffix;\n if (colonPos > 0 && hyphenPos > colonPos) {\n form = token.substring(0, colonPos);\n suffix = token.substring(colonPos+1);\n }\n else if (colonPos < 0 && hyphenPos < 0) {\n form = token;\n suffix = null;\n }\n else {\n form = null;\n suffix = token;\n }\n while (suffix != null) {\n hyphenPos = suffix.indexOf('-');\n String attr = suffix.substring(0, hyphenPos);\n String val = suffix.substring(hyphenPos+1);\n colonPos = suffix.indexOf(':');\n if (colonPos > 0) {\n val = suffix.substring(hyphenPos+1, colonPos);\n suffix = suffix.substring(colonPos+1);\n }\n else suffix = null;\n attr = unescape(attr); val = unescape(val);\n if (attr.equals(Tokenizer.WORD_ATTR)) { form = val; continue; }\n if (attr.equals(Tokenizer.STEM_ATTR)) { stem = val; continue; }\n if (attr.equals(Tokenizer.POS_ATTR)) { POS = val; continue; }\n if (attr.equals(Tokenizer.SUPERTAG_ATTR)) { supertag = val; continue; }\n if (attr.equals(Tokenizer.SEM_CLASS_ATTR)) { semClass = val; continue; }\n if (attr.equals(Tokenizer.PITCH_ACCENT_ATTR)) { pitchAccent = val; continue; }\n if (attrValPairs == null) attrValPairs = new ArrayList>(5);\n attrValPairs.add(new Pair(attr, val));\n }\n }\n // check for pitch accent preceded by an underscore\n int pos = (form != null) ? form.lastIndexOf(\"_\") : -1;\n if (pos > 0) {\n String suffix = form.substring(pos+1);\n if (Grammar.isPitchAccent(suffix)) {\n pitchAccent = suffix;\n form = form.substring(0, pos);\n }\n }\n // unescape form (unless it happens to be \"null\")\n if (!\"null\".equals(form)) form = unescape(form);\n // check for special token\n String specialTokenClass = isSpecialToken(form);\n if (specialTokenClass != null) semClass = specialTokenClass;\n // done\n return Word.createWord(form,pitchAccent,attrValPairs,stem,POS,supertag,semClass);\n }\n \n \n /**\n * Returns a string (eg Tokenizer.DATE_CLASS) indicating the semantic class \n * of special token, if the given token is recognized as a special \n * token; otherwise returns null. \n */\n public String isSpecialToken(String token) {\n if (token == null) return null;\n if (isDate(token)) return Tokenizer.DATE_CLASS;\n if (isTime(token)) return Tokenizer.TIME_CLASS;\n if (isNum(token)) return Tokenizer.NUM_CLASS;\n if (isAmt(token)) return Tokenizer.AMT_CLASS;\n if (isDur(token)) return Tokenizer.DUR_CLASS;\n if (isNamedEntity(token)) return Tokenizer.NE_CLASS;\n return null;\n }\n \n /**\n * Returns the special token constant for the given special token class, \n * or null if none.\n */\n public String getSpecialTokenConstant(String semClass) {\n if (semClass == null) return null;\n return specialTokenMap.get(semClass);\n }\n \n /** \n * Returns true iff the given string is a special token constant \n * (eg Tokenizer.DATE_VAL).\n */\n public boolean isSpecialTokenConstant(String s) {\n return specialTokenMap.containsValue(s);\n }\n \n \n /** \n * Returns true iff the token is recognized as a date.\n * The default implementation recognizes dates in the \n * format yyyy.MM.dd, e.g. \"2004.05.07\", or *.MM.dd, e.g. \"*.05.07\", \n * which is taken to mean the 5th of May (in the contextually \n * appropriate year). Note that by including the \"*.\" prefix, \n * the format avoids being ambiguous between a date or number; \n * that is, with this format, something like \"10.01\" is \n * unambiguously a number, whereas \"*.10.01\" means the 1st of \n * October.\n */\n public boolean isDate(String token) {\n ParsePosition pos = new ParsePosition(0);\n Date date = dateFormat.parse(token, pos);\n if (date != null && pos.getIndex() == token.length()) return true;\n pos = new ParsePosition(0);\n date = dateFormatNoYear.parse(token, pos);\n return (date != null && pos.getIndex() == token.length());\n }\n \n /** \n * Returns true iff the token is recognized as a time. \n * The default implementation recognizes times in the \n * 24-hour format HH:mm, e.g. \"00:12\" or \"15:03\".\n */\n public boolean isTime(String token) {\n ParsePosition pos = new ParsePosition(0);\n Date time = timeFormat.parse(token, pos);\n return (time != null && pos.getIndex() == token.length());\n }\n \n /** \n * Returns true iff the token is recognized as a number.\n * The default implementation returns true if the token \n * can be parsed as an integer or double, though not one \n * in scientific notation.\n */\n public boolean isNum(String token) {\n try {\n Integer.parseInt(token);\n return true; \n }\n catch (NumberFormatException exc) { \n try {\n Double.parseDouble(token);\n if (token.indexOf('E') != -1) return false;\n if (token.indexOf('e') != -1) return false;\n return true;\n }\n catch (NumberFormatException exc2) {\n return false; \n }\n }\n }\n \n /** \n * Returns true iff the token is recognized as an amount.\n * The default implementation only handles currency amounts.\n * The token is recognized as an amount if it begins with \n * a number and ends with an ISO-4217 currency code.\n * (See http://www.xe.com/iso4217.htm.)\n */\n public boolean isAmt(String token) { \n if (token.length() < 4) return false;\n String code = token.substring(token.length()-3);\n try { Currency.getInstance(code); }\n catch (IllegalArgumentException exc) { return false; }\n String num = token.substring(0,token.length()-3).trim();\n return isNum(num);\n }\n\n /**\n * Returns true iff the token is recognized as a duration.\n * The format is \"PnYnMnDTnHnMnS\", as defined in XML Schema 1.0 section 3.2.6.1.\n * For example, \"PT5H30\" is 5 hours and 30 minutes.\n */\n public boolean isDur(String token) {\n try { datatypeFactory.newDuration(token); return true; }\n catch (Exception exc) { return false; }\n }\n \n /** \n * Returns true iff the token is recognized as a named entity (not listed in lexicon). \n * The default implementation always returns false.\n */\n public boolean isNamedEntity(String token) {\n return false;\n }\n \n \n /**\n * Returns a string for the given list of words.\n * A space separates the string for each word, as determined by getOrthography(Word,false).\n */\n public String getOrthography(List words) {\n return getOrthography(words, false);\n }\n \n /**\n * Returns a string for the given list of words, optionally with semantic class replacement.\n * A space separates the string for each word, as determined by getOrthography(Word,semClassReplacement).\n */\n public String getOrthography(List words, boolean semClassReplacement) {\n StringBuffer sb = new StringBuffer();\n for (int i = 0; i < words.size(); i++) {\n Word w = (Word) words.get(i);\n sb.append(getOrthography(w, semClassReplacement));\n if (i < words.size()-1) sb.append(\" \");\n }\n return sb.toString();\n }\n \n\n /**\n * Returns a string for the given word, optionally with semantic class replacement.\n * The default implementation returns the word's form - or semantic class, if apropos - \n * followed by its pitch accent (if non-null) separated by an underscore, \n * followed by any further attribute values, also separated by underscores.\n * With the semantic class replacement option, the word form is replaced with \n * the semantic class, uppercased, if the class is listed as one to replace words with for \n * language models.\n */\n public String getOrthography(Word w, boolean semClassReplacement) {\n StringBuffer sb = new StringBuffer();\n String semClass = w.getSemClass();\n if (semClassReplacement && semClass != null && replacementSemClasses.contains(semClass))\n sb.append(semClass.toUpperCase());\n else\n sb.append(w.getForm());\n if (w.getPitchAccent() != null)\n sb.append(\"_\").append(w.getPitchAccent());\n for (Iterator> it = w.getAttrValPairs(); it.hasNext(); ) {\n Pair p = it.next();\n sb.append(\"_\").append(p.b);\n \n }\n return sb.toString();\n }\n \n /**\n * Returns a string for the given list of words, \n * in the format expected by the SRILM tool for factored language models.\n * A space separates the string for each word, determined by format(Word).\n */\n public String format(List words) { return format(words, false); }\n\n /**\n * Returns a string for the given list of words, \n * in the format expected by the SRILM tool for factored language models, \n * optionally with semantic class replacement.\n * A space separates the string for each word, determined by format(Word,boolean).\n */\n public String format(List words, boolean semClassReplacement) {\n StringBuffer sb = new StringBuffer();\n sb.append(\" \");\n for (int i = 0; i < words.size(); i++) {\n Word w = words.get(i);\n if (w.getForm() == \"\" || w.getForm() == \"\") continue; // skip or \n sb.append(format(w, semClassReplacement));\n sb.append(\" \");\n }\n sb.append(\"\");\n return sb.toString();\n }\n \n /**\n * Returns a string for the given word, \n * in the format expected by the SRILM tool for factored language models.\n * All factors are escaped.\n */\n public String format(Word w) { return format(w, false); }\n \n /**\n * Returns a string for the given word, \n * in the format expected by the SRILM tool for factored language models, \n * optionally with semantic class replacement.\n * All factors are escaped.\n * With the semantic class replacement option, the word form and stem are replaced with \n * the semantic class, uppercased, if the class is listed as one to replace words with for \n * language models.\n */\n public String format(Word w, boolean semClassReplacement) {\n StringBuffer sb = new StringBuffer();\n String form = w.getForm();\n String pitchAccent = w.getPitchAccent();\n String stem = w.getStem();\n String POS = w.getPOS();\n String supertag = w.getSupertag();\n String semClass = w.getSemClass();\n if (semClassReplacement && semClass != null && replacementSemClasses.contains(semClass)) {\n form = escape(semClass.toUpperCase()); stem = form; \n }\n sb.append(escape(form));\n if (pitchAccent != null) sb.append(\":\").append(Tokenizer.PITCH_ACCENT_ATTR).append(\"-\").append(escape(pitchAccent));\n for (Iterator> it = w.getAttrValPairs(); it.hasNext(); ) {\n Pair p = it.next();\n String attr = p.a; String val = p.b;\n if (val != null) sb.append(\":\").append(escape(attr)).append(\"-\").append(escape(val));\n }\n if (stem != null) sb.append(\":\").append(Tokenizer.STEM_ATTR).append(\"-\").append(escape(stem));\n if (POS != null) sb.append(\":\").append(Tokenizer.POS_ATTR).append(\"-\").append(escape(POS));\n if (supertag != null) sb.append(\":\").append(Tokenizer.SUPERTAG_ATTR).append(\"-\").append(escape(supertag));\n if (semClass != null) sb.append(\":\").append(Tokenizer.SEM_CLASS_ATTR).append(\"-\").append(escape(semClass));\n return sb.toString();\n }\n\n \n /**\n * Returns an encoding of the given string where \n * the characters for ampersand, less-than, greater-than, \n * apostrophe, quote, colon and hyphen are escaped \n * using HTML conventions.\n * Null is returned for the null string.\n * An initial substring 'null' is doubled.\n */\n public static String escape(String s) {\n if (s == null) return null;\n StringBuffer output = null; // instantiate only if needed\n if (s.startsWith(\"null\")) {\n output = new StringBuffer();\n output.append(\"null\");\n }\n for(int i=0; i < s.length(); i++) {\n char c = s.charAt(i);\n if (output == null && (c=='<' || c=='>' || c=='&' || c=='\\'' || c=='\"' || c==':' || c=='-')) {\n output = new StringBuffer();\n output.append(s.substring(0,i));\n }\n if (output != null) {\n switch(c) {\n case '<': output.append(\"<\"); break;\n case '>': output.append(\">\"); break;\n case '&': output.append(\"&\"); break;\n case '\\'': output.append(\"'\"); break;\n case '\"': output.append(\""\"); break;\n case ':': output.append(\"&#\").append((int)':').append(\";\"); break;\n case '-': output.append(\"&#\").append((int)'-').append(\";\"); break;\n default: output.append(c);\n }\n }\n }\n return (output != null) ? output.toString() : s;\n }\n \n /**\n * Returns a decoding of the given string where \n * the characters for ampersand, less-than, greater-than, \n * apostrophe, quote, colon and hyphen (and any other \n * character whose code is given numerically) are unescaped \n * using HTML conventions.\n * An exception is that ampersands may be left unescaped \n * for convenience, when there is no following semicolon \n * in the string.\n * Null is returned for the null string and for the string \"null\".\n * An initial substring 'nullnull' is halved.\n */\n public static String unescape(String s) {\n if (s == null || s.equals(\"null\")) return null;\n StringBuffer output = null; // instantiate only if needed\n if (s.startsWith(\"nullnull\")) {\n \ts = s.substring(4);\n output = new StringBuffer();\n }\n for (int i=0; i < s.length(); i++) {\n char c = s.charAt(i);\n if (c == '&') {\n int endPos = s.indexOf(\";\", i);\n if (endPos < 0) {\n \t// allow unescaped ampersands\n \tif (output != null) output.append(c);\n \tcontinue;\n }\n if (output == null) {\n output = new StringBuffer();\n output.append(s.substring(0,i));\n }\n String escaped = s.substring(i+1, endPos);\n if (escaped.equals(\"lt\")) { output.append('<'); i = endPos; continue; }\n if (escaped.equals(\"gt\")) { output.append('>'); i = endPos; continue; }\n if (escaped.equals(\"amp\")) { output.append('&'); i = endPos; continue; }\n if (escaped.equals(\"apos\")) { output.append('\\''); i = endPos; continue; }\n if (escaped.equals(\"quot\")) { output.append('\"'); i = endPos; continue; }\n if (s.charAt(i+1) == '#') {\n escaped = s.substring(i+2, endPos);\n output.append((char)Integer.parseInt(escaped)); i = endPos; continue;\n }\n throw new RuntimeException(\n \"Unable to unescape \" + s.substring(i,endPos+1) + \"at position \" + i + \" in: \" + s\n );\n }\n else if (output != null) output.append(c);\n }\n return (output != null) ? output.toString() : s;\n }\n\n \n /**\n * Returns one or more orthographic words for the given word's form.\n * This method is called from within Sign.getWordsInXml as \n * part of producing the textual output of realization.\n * The default implementation checks the semantic class \n * for a special token class, and if true, returns the result \n * of expandDate, expandTime, expandNum, expandAmt, or \n * expandNamedEntity, as appropriate, after first checking that \n * the corresponding isDate, ..., isNamedEntity method returns true.\n * Otherwise, it splits the word form using underscore as a delimiter.\n */\n public List expandWord(Word word) {\n String token = word.getForm();\n String sc = word.getSemClass();\n if (sc == Tokenizer.DATE_CLASS && isDate(token)) return expandDate(token);\n if (sc == Tokenizer.TIME_CLASS && isTime(token)) return expandTime(token);\n if (sc == Tokenizer.NUM_CLASS && isNum(token)) return expandNum(token);\n if (sc == Tokenizer.AMT_CLASS && isAmt(token)) return expandAmt(token);\n if (sc == Tokenizer.DUR_CLASS && isDur(token)) return expandDur(token);\n if (sc == Tokenizer.NE_CLASS && isNamedEntity(token)) return expandNamedEntity(token);\n String[] words = token.split(\"_\");\n return Arrays.asList(words);\n }\n \n /**\n * Returns one or more orthographic words for the given date token.\n * The default implementation expands the date with \n * EnglishExpander.expandDate, using the long option if the year is \n * present, and the medium option if not.\n */\n public List expandDate(String date) {\n ArrayList retval = new ArrayList();\n try {\n ParsePosition pos = new ParsePosition(0);\n Date dateObj = dateFormat.parse(date, pos);\n if (dateObj != null && pos.getIndex() == date.length()) {\n EnglishExpander.expandDate(dateObj, DateFormat.LONG, retval);\n }\n else {\n dateObj = dateFormatNoYear.parse(date);\n EnglishExpander.expandDate(dateObj, DateFormat.MEDIUM, retval);\n }\n }\n // shouldn't happen if isDate called first\n catch (ParseException exc) {\n // just add date string as a fall-back option\n retval.add(date);\n }\n return retval;\n }\n \n /**\n * Returns one or more orthographic words for the given time token.\n * The default implementation expands the time using \n * EnglishExpander.expandTime.\n */\n public List expandTime(String time) {\n ArrayList retval = new ArrayList();\n try {\n EnglishExpander.expandTime(timeFormat.parse(time), retval);\n }\n // shouldn't happen if isTime called first\n catch (ParseException exc) {\n // just add time string as a fall-back option\n retval.add(time);\n }\n return retval;\n }\n \n /**\n * Returns one or more orthographic words for the given number token.\n * The default implementation expands the number using \n * EnglishExpander.expandNumber.\n */\n public List expandNum(String num) {\n ArrayList retval = new ArrayList();\n EnglishExpander.expandNumber(num, retval);\n return retval;\n }\n \n /**\n * Returns one or more orthographic words for the given amount token.\n * The default implementation expands the number using \n * EnglishExpander.expandAmount.\n */\n public List expandAmt(String amt) {\n String code = amt.substring(amt.length()-3);\n String num = amt.substring(0,amt.length()-3).trim();\n ArrayList retval = new ArrayList();\n EnglishExpander.expandAmount(num, code, retval);\n return retval;\n }\n \n /**\n * Returns one or more orthographic words for the given duration token.\n * The default implementation expands the number using \n * EnglishExpander.expandDuration.\n */\n public List expandDur(String dur) {\n Duration duration = null;\n try {\n duration = datatypeFactory.newDuration(dur);\n } catch (Exception exc) {\n // parsing not expected to fail\n throw (RuntimeException) new RuntimeException().initCause(exc);\n }\n ArrayList retval = new ArrayList();\n EnglishExpander.expandDuration(duration, retval);\n return retval;\n }\n \n /**\n * Returns one or more orthographic words for the given named entity token.\n * The default implementation just splits the token using underscore as a delimiter.\n */\n public List expandNamedEntity(String namedEntity) {\n String[] words = namedEntity.split(\"_\");\n return Arrays.asList(words);\n }\n \n /** Test: tokenize args[0], expand each token; and optionally do parseToken(args[1],true). */\n public static void main(String[] args) {\n Tokenizer tk = new DefaultTokenizer();\n String s = args[0];\n List words = tk.tokenize(s);\n String expw = \"\";\n System.out.println(\"words: \");\n for (int i = 0; i < words.size(); i++) {\n Word word = words.get(i);\n System.out.print(word + \" \");\n List orthWords = tk.expandWord(word);\n for (int j = 0; j < orthWords.size(); j++) {\n expw += orthWords.get(j) + \" \";\n }\n }\n System.out.println();\n System.out.println(\"expanded: \" + expw);\n System.out.println(\"formatted: \" + tk.format(words));\n if (args.length > 1) {\n System.out.println();\n Word strictlyParsed = tk.parseToken(args[1], true);\n System.out.println(\"strictly parsed word: \" + strictlyParsed);\n System.out.println(\"formatted: \" + tk.format(strictlyParsed));\n }\n }\n}\n\n" }, { "path": "src/opennlp/ccg/lexicon/EnglishExpander.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004-5 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport java.util.*;\nimport java.text.*;\nimport javax.xml.datatype.*;\n\n/**\n * EnglishExpander provides methods for expanding numbers, amounts, durations, \n * dates and times as English words.\n * The class com.sun.speech.freetts.en.us.NumberExpander served \n * as a reference point in part, but the implementation has been \n * rewritten from scratch, streamlined and extended.\n *\n * @author Michael White\n * @version $Revision: 1.7 $, $Date: 2005/10/20 17:30:30 $\n **/\npublic class EnglishExpander {\n\n //--------------------------------------------\n // dates \n //\n \n // formats date in month\n private static final DateFormat dateInMonthFormat = new SimpleDateFormat(\"dd\", Locale.ENGLISH);\n\n // formats month in full\n private static final DateFormat monthFormat = new SimpleDateFormat(\"MMMM\", Locale.ENGLISH);\n\n // formats year in full\n private static final DateFormat yearFormat = new SimpleDateFormat(\"yyyy\", Locale.ENGLISH);\n \n /**\n * Expands a date to English words in short, \n * medium or long forms, adding the words to \n * the given list. The style is given by \n * DateFormat.SHORT (e.g., \"the first\"), \n * DateFormat.MEDIUM (e.g., \"the first of May\"), and \n * DateFormat.LONG (e.g., \"the first of May, two thousand and four\").\n */\n public static void expandDate(Date date, int style, List list) {\n list.add(\"the\");\n expandOrdinal(dateInMonthFormat.format(date), list);\n if (style == DateFormat.MEDIUM || style == DateFormat.LONG) {\n list.add(\"of\");\n list.add(monthFormat.format(date));\n }\n if (style == DateFormat.LONG) {\n list.add(\",\");\n String year = yearFormat.format(date);\n // x00y\n if (year.charAt(1) == '0' && year.charAt(2) == '0') {\n expandNumber(year, list);\n }\n // xxyy\n else {\n expandNumber(year.substring(0,2), list);\n expandNumber(year.substring(2), list);\n }\n }\n }\n\n //--------------------------------------------\n // times\n //\n \n // formats hours in range 1-12\n private static final DateFormat hoursFormat = new SimpleDateFormat(\"hh\", Locale.ENGLISH);\n\n // formats minutes\n private static final DateFormat minutesFormat = new SimpleDateFormat(\"mm\", Locale.ENGLISH);\n\n // formats am/pm\n private static final DateFormat amPmFormat = new SimpleDateFormat(\"a\", Locale.ENGLISH);\n\n /**\n * Expands a time to English words, \n * adding the words to the given list.\n * For example, a date object with time set to 23:02 \n * is expanded to \"eleven oh two PM\".\n */\n public static void expandTime(Date time, List list) {\n // add hours\n String hours = hoursFormat.format(time);\n expandNDigitNumber(hours, list);\n // add minutes, with special case for 'oh'\n String minutes = minutesFormat.format(time);\n if (minutes.charAt(0) == '0' && minutes.charAt(1) != '0') {\n list.add(\"oh\");\n expandNDigitNumber(minutes.substring(1), list);\n }\n else {\n expandNDigitNumber(minutes, list);\n }\n // add AM or PM\n list.add(amPmFormat.format(time));\n }\n\n //--------------------------------------------\n // amounts\n //\n \n /**\n * Expands a digit string and currency code to \n * number words and the currency name, which \n * are added to the given list.\n * For example, \"12.50\" and \"GBP\" are expanded to \n * \"twelve pounds and fifty pence\".\n * The codes GBP, USD and EUR are converted to \n * names, while other codes are left as is.\n */\n public static void expandAmount(String digitString, String currencyCode, List list) {\n // establish names for currency (singular and plural), \n // and fractional parts\n String currSing = null; String currPlur = null;\n String fracSing = null; String fracPlur = null;\n if (currencyCode.equals(\"GBP\")) {\n currSing = \"pound\"; currPlur = \"pounds\";\n fracSing = \"penny\"; fracPlur = \"pence\";\n }\n else if (currencyCode.equals(\"USD\")) {\n currSing = \"dollar\"; currPlur = \"dollars\";\n fracSing = \"cent\"; fracPlur = \"cents\";\n }\n else if (currencyCode.equals(\"EUR\")) {\n currSing = \"euro\"; currPlur = \"euros\";\n fracSing = \"cent\"; fracPlur = \"cents\";\n }\n // if none, just expand digit string and append code\n if (currSing == null) {\n expandNumber(digitString, list);\n list.add(currencyCode);\n return;\n }\n // otherwise, get whole and fractional parts of digit string\n String whole = digitString; String frac = null;\n int dotIndex = digitString.indexOf(\".\");\n if (dotIndex != -1) {\n whole = digitString.substring(0,dotIndex); \n frac = digitString.substring(dotIndex+1);\n }\n // expand whole\n expandNumber(whole, list);\n // add currency name\n if (whole.equals(\"1\")) \n list.add(currSing);\n else \n list.add(currPlur);\n // add fractional part, if any\n if (frac != null) {\n // add \"and\"\n list.add(\"and\");\n // expand frac\n expandNDigitNumber(frac, list);\n // add fractional name\n if (frac.equals(\"01\"))\n list.add(fracSing);\n else\n list.add(fracPlur);\n }\n }\n\n\n //--------------------------------------------\n // durations\n //\n \n /**\n * Expands a duration into a string of words.\n * NB: Fractions of seconds are ignored at present.\n */\n public static void expandDuration(Duration duration, List list) {\n int[] durationFields = { \n duration.getYears(), duration.getMonths(), duration.getDays(), \n duration.getHours(), duration.getMinutes(), duration.getSeconds()\n };\n int counter = 0;\n for (int i = 0; i < durationFields.length; i++) {\n if (durationFields[i] > 0) counter++;\n }\n for (int i = 0; i < durationFields.length; i++) {\n if (durationFields[i] > 0) {\n counter--;\n String str = Integer.toString(durationFields[i]);\n expandNumber(str, list);\n String unit = durUnits[i];\n if (durationFields[i] != 1) unit += \"s\";\n list.add(unit);\n if (counter > 1) list.add(\",\");\n if (counter == 1) list.add(\"and\");\n }\n }\n }\n \n // duration units\n private static String[] durUnits = { \"year\", \"month\", \"day\", \"hour\", \"minute\", \"second\" };\n \n \n //--------------------------------------------\n // numbers\n //\n \n /**\n * Expands a digit string to a sequence of digit words, \n * which are added to the given list.\n * For example, \"1234\" is expanded to \"one two three four\".\n */\n public static void expandDigits(String digitString, List list) {\n for (int i = 0; i < digitString.length(); i++) {\n list.add(zeroToNine[digitString.charAt(i)-'0']);\n }\n }\n \n /**\n * Expands a digit string to number words, \n * which are added to the given list.\n * For example, \"1234\" is expanded to \n * \"one thousand two hundred and thirty four\".\n * The digit string may contain a single dot in it, \n * as well as an initial plus or minus character.\n * For example, \"-100.011\" is expanded to \n * \"minus one hundred point zero one one\".\n * Scientific notation is not currently handled.\n */\n public static void expandNumber(String digitString, List list) {\n // do nothing with empty strings\n if (digitString.length() == 0) return;\n // handle plus or minus\n char c0 = digitString.charAt(0);\n if (c0 == '+' || c0 == '-') {\n list.add((c0 == '+') ? \"plus\" : \"minus\");\n digitString = digitString.substring(1);\n }\n // check for dot\n int dotIndex = digitString.indexOf(\".\");\n if (dotIndex != -1) {\n // add numbers \"point\" digits\n expandNDigitNumber(digitString.substring(0,dotIndex), list);\n list.add(\"point\");\n expandDigits(digitString.substring(dotIndex+1), list);\n }\n else {\n // add numbers\n expandNDigitNumber(digitString, list);\n }\n }\n \n /**\n * Expands a digit string to words for an ordinal number, \n * which are added to the given list.\n * For example, \"1234\" is expanded to \n * \"one thousand two hundred and thirty fourth\".\n */\n public static void expandOrdinal(String digitString, List list) {\n // expand number\n expandNDigitNumber(digitString, list);\n // replace last one\n int lastPos = list.size() - 1;\n String ordinal = getOrdinal(list.get(lastPos));\n list.set(lastPos, ordinal);\n }\n\n // n-digit number \n public static void expandNDigitNumber(String digitString, List list) {\n int numDigits = digitString.length();\n if (numDigits == 2) \n expand2DigitNumber(digitString, list); \n else if (numDigits == 3) \n expand3DigitNumber(digitString, list); \n else if (numDigits >= 4 && numDigits <= 12) \n expand4to12DigitNumber(digitString, list);\n else \n expandDigits(digitString, list);\n }\n \n // 2 digit numbers\n private static void expand2DigitNumber(String digitString, List list) {\n // 0x case\n if (digitString.charAt(0) == '0') {\n // do nothing for 00 case\n if (digitString.charAt(1) != '0')\n expandDigits(digitString.substring(1), list);\n }\n // 1x case\n else if (digitString.charAt(0) == '1') {\n list.add(tenToNineteen[digitString.charAt(1)-'0']);\n }\n // xy case, x >= 2\n else {\n list.add(zeroToNinety[digitString.charAt(0)-'0']);\n // do nothing for x0 case\n if (digitString.charAt(1) != '0')\n expandDigits(digitString.substring(1), list);\n }\n }\n\n // 3 digit numbers\n private static void expand3DigitNumber(String digitString, List list) {\n // add hundreds if non-zero\n if (digitString.charAt(0) != '0') {\n // add hundreds digit\n expandDigits(digitString.substring(0,1), list);\n // add unit (\"hundred\")\n list.add(\"hundred\");\n }\n // add \"and\", if final two digits non-zero\n if (digitString.charAt(1) != '0' || digitString.charAt(2) != '0')\n list.add(\"and\");\n // expand final two digits\n expand2DigitNumber(digitString.substring(1), list);\n }\n \n // 4-12 digit numbers\n private static void expand4to12DigitNumber(String digitString, List list) {\n int numDigitsModThree = digitString.length() % 3;\n int numInitialDigits = (numDigitsModThree != 0) ? numDigitsModThree : 3;\n int unitsIndex = ((digitString.length() - 1) / 3) - 1;\n // add initial digits\n expandNDigitNumber(digitString.substring(0, numInitialDigits), list);\n // add unit\n list.add(thousandToBillion[unitsIndex]);\n // add rest\n expandNDigitNumber(digitString.substring(numInitialDigits), list);\n }\n \n \n //------------------------\n // arrays of number words\n //\n \n private static final String[] zeroToNine = {\n \"zero\", \"one\", \"two\", \"three\", \"four\", \n \"five\", \"six\", \"seven\", \"eight\", \"nine\" \n };\n \n private static final String[] tenToNineteen = {\n \"ten\", \"eleven\", \"twelve\", \"thirteen\", \"fourteen\", \n \"fifteen\", \"sixteen\", \"seventeen\", \"eighteen\", \"nineteen\" \n };\n \n private static final String[] zeroToNinety = {\n \"zero\", \"ten\", \"twenty\", \"thirty\", \"forty\", \n \"fifty\", \"sixty\", \"seventy\", \"eighty\", \"ninety\" \n };\n \n private static final String[] thousandToBillion = {\n \"thousand\", \"million\", \"billion\"\n };\n \n private static final String[] zerothToNinth = {\n \"zeroth\", \"first\", \"second\", \"third\", \"fourth\", \n \"fifth\", \"sixth\", \"seventh\", \"eighth\", \"ninth\" \n };\n \n private static final String[] tenthToNineteenth = {\n \"tenth\", \"eleventh\", \"twelfth\", \"thirteenth\", \"fourteenth\", \n \"fifteenth\", \"sixteenth\", \"seventeenth\", \"eighteenth\", \"nineteenth\" \n };\n \n private static final String[] zerothToNinetieth = {\n \"zeroth\", \"tenth\", \"twentieth\", \"thirtieth\", \"fortieth\", \n \"fiftieth\", \"sixtieth\", \"seventieth\", \"eightieth\", \"ninetieth\" \n };\n \n private static final String[] thousandthToBillionth = {\n \"thousandth\", \"millionth\", \"billionth\"\n };\n\n \n //--------------------------------------------\n // corresponding ordinals\n //\n \n // map from numbers to corresponding ordinals\n private static Map ordinalMap = null;\n \n // returns corresponding ordinal\n private static String getOrdinal(String number) {\n // ensure ordinalMap instantiated\n if (ordinalMap == null) {\n ordinalMap = new HashMap();\n for (int i = 0; i < zeroToNine.length; i++) {\n ordinalMap.put(zeroToNine[i], zerothToNinth[i]);\n }\n for (int i = 0; i < tenToNineteen.length; i++) {\n ordinalMap.put(tenToNineteen[i], tenthToNineteenth[i]);\n }\n for (int i = 0; i < zeroToNinety.length; i++) {\n ordinalMap.put(zeroToNinety[i], zerothToNinetieth[i]);\n }\n ordinalMap.put(\"hundred\", \"hundredth\");\n for (int i = 0; i < thousandToBillion.length; i++) {\n ordinalMap.put(thousandToBillion[i], thousandthToBillionth[i]);\n }\n }\n return ordinalMap.get(number);\n }\n}\n\n" }, { "path": "src/opennlp/ccg/lexicon/EntriesItem.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-9 Jason Baldridge, Gann Bierner and \n// University of Edinburgh / Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.lexicon;\n\nimport opennlp.ccg.synsem.*;\nimport org.jdom.*;\n\n/**\n * Data structure for storing information about a category family entry.\n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.10 $, $Date: 2009/10/17 20:46:20 $\n */\npublic class EntriesItem {\n \n private Family family;\n private Boolean active = Boolean.TRUE;\n private String name;\n private String stem;\n private String indexRel;\n private Category cat;\n\n public EntriesItem(Element el, Family family) {\n this.family = family;\n name = el.getAttributeValue(\"name\");\n\n stem = el.getAttributeValue(\"stem\");\n if (stem == null) stem = Lexicon.DEFAULT_VAL; \n\n String isActive = el.getAttributeValue(\"active\");\n if (isActive != null && isActive.equals(\"false\"))\n active = Boolean.FALSE;\n\n String indexRelVal = el.getAttributeValue(\"indexRel\");\n if (indexRelVal != null) indexRel = indexRelVal;\n else indexRel = family.getIndexRel();\n\n cat = CatReader.getCat((Element)el.getChildren().get(0));\n }\n\n public Boolean getActive() { return active; }\n /** Returns the name of this entry. */\n public String getName() { return name; }\n /** Returns the qualified name in the form familyName.name. */ \n public String getQualifiedName() { return getFamilyName() + \".\" + name; }\n /** Returns the name of this entry's family. */\n public String getFamilyName() { return family.getName(); }\n /** Returns the supertag of this entry's category. */\n public String getSupertag() { return cat.getSupertag(); }\n public Family getFamily() { return family; }\n public String getStem() { return stem; }\n public String getIndexRel() { return indexRel; }\n /** Returns this entry's family's coart rel. */\n public String getCoartRel() { return family.getCoartRel(); }\n public Category getCat() { return cat; }\n \n public String toString () {\n return getQualifiedName() + \":\" + stem + \" :- \" + cat;\n }\n}\n" }, { "path": "src/opennlp/ccg/lexicon/FactorChainWord.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport opennlp.ccg.util.*;\n\nimport java.util.*;\n\n/**\n * A FactorChainWord represents a word via a chain of references to \n * factor keys, or just to a string in the case of the word form. \n * Canonical instances are created by a factory method, and stored \n * in a trie map. The factor chain representation should be more \n * space efficient when dealing with large numbers of words.\n *\n *\n * @author Michael White\n * @version $Revision: 1.3 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class FactorChainWord extends Word {\n \n\tprivate static final long serialVersionUID = 952665894357382685L;\n\n\t/** The referenced factor key or string (for the word form). */\n protected Object key;\n \n /** The previous node in the chain. */\n protected FactorChainWord prev;\n \n /** Constructor. */\n protected FactorChainWord(Object key, FactorChainWord prev) {\n this.key = key; this.prev = prev; \n }\n \n \n /** Returns the surface form. */\n public String getForm() { return getValFromInterned(Tokenizer.WORD_ATTR); }\n \n /** Returns the pitch accent. */\n public String getPitchAccent() { return getValFromInterned(Tokenizer.PITCH_ACCENT_ATTR); }\n \n /** Returns the list of extra attribute-value pairs. */\n protected List> getAttrValPairsList() { \n List> retval = null; \n FactorChainWord current = this;\n while (current != null) {\n if (current.key instanceof FactorKey) {\n FactorKey fkey = (FactorKey) current.key;\n if (!isKnownAttr(fkey.factor)) {\n if (retval == null) retval = new ArrayList>(5);\n retval.add(0, new Pair(fkey.factor, fkey.val));\n }\n }\n current = current.prev;\n }\n return retval; \n }\n \n /** Returns the stem. */\n public String getStem() { return getValFromInterned(Tokenizer.STEM_ATTR); }\n \n /** Returns the part of speech. */\n public String getPOS() { return getValFromInterned(Tokenizer.POS_ATTR); }\n \n /** Returns the supertag. */\n public String getSupertag() { return getValFromInterned(Tokenizer.SUPERTAG_ATTR); }\n \n /** Returns the semantic class. */\n public String getSemClass() { return getValFromInterned(Tokenizer.SEM_CLASS_ATTR); }\n\n \n /** Returns the value of the attribute with the given name, or null if none. \n The attribute names Tokenizer.WORD_ATTR, ..., Tokenizer.SEM_CLASS_ATTR \n may be used to retrieve the form, ..., semantic class. */\n public String getVal(String attr) {\n String internedAttr = attr.intern(); // use == on interned attr\n return getValFromInterned(internedAttr); \n }\n \n /** Returns the value of the given interned attr, or null if none. */\n protected String getValFromInterned(String attr) {\n FactorChainWord current = this;\n while (current != null) {\n if (attr == Tokenizer.WORD_ATTR) {\n if (current.key instanceof String) return (String) current.key;\n }\n else if (current.key instanceof FactorKey) {\n FactorKey fkey = (FactorKey) current.key;\n if (fkey.factor == attr) return fkey.val;\n }\n current = current.prev;\n }\n return null;\n }\n \n /** Factory. */\n public static class Factory implements WordFactory {\n \n /** Trie map for canonical instances. */\n protected TrieMap factorChainRoot = new TrieMap(null);\n \n /** Creates a surface word with the given interned form. */\n public synchronized Word create(String form) {\n return create(factorChainRoot, Tokenizer.WORD_ATTR, form);\n }\n \n /** Creates a (surface or full) word with the given normalized attribute name and value.\n The attribute names Tokenizer.WORD_ATTR, ..., Tokenizer.SEM_CLASS_ATTR \n may be used for the form, ..., semantic class. */\n public synchronized Word create(String attr, String val) {\n return create(factorChainRoot, attr, val);\n }\n \n /** Creates a (surface or full) word from the given normalized factors. \n Returns null if no non-null vals. */\n public synchronized Word create(\n String form, String pitchAccent, List> attrValPairs, \n String stem, String POS, String supertag, String semClass \n ) {\n // adds non-null vals from the root, in a rough specificity order \n TrieMap currentNode = factorChainRoot;\n if (POS != null) currentNode = findChild(currentNode, Tokenizer.POS_ATTR, POS);\n if (supertag != null) currentNode = findChild(currentNode, Tokenizer.SUPERTAG_ATTR, supertag);\n if (semClass != null) currentNode = findChild(currentNode, Tokenizer.SEM_CLASS_ATTR, semClass);\n if (stem != null) currentNode = findChild(currentNode, Tokenizer.STEM_ATTR, stem);\n if (form != null) currentNode = findChild(currentNode, Tokenizer.WORD_ATTR, form);\n if (pitchAccent != null) currentNode = findChild(currentNode, Tokenizer.PITCH_ACCENT_ATTR, pitchAccent);\n if (attrValPairs != null) {\n for (int i = 0; i < attrValPairs.size(); i++) {\n \tPair p = attrValPairs.get(i);\n String attr = p.a;\n String val = p.b; \n currentNode = findChild(currentNode, attr, val);\n }\n }\n return currentNode.data;\n }\n \n /** Creates a word from the given node, adding the given interned attr \n and non-null val. */\n protected Word create(TrieMap currentNode, String attr, String val) {\n TrieMap child = findChild(currentNode, attr, val);\n return child.data;\n }\n \n /** Gets or makes a child node from the given node. */\n protected TrieMap findChild(TrieMap currentNode, String attr, String val) {\n Object key = FactorKey.getKey(attr, val);\n TrieMap child = currentNode.findChild(key);\n if (child.data == null) {\n FactorChainWord parent = currentNode.data;\n child.data = new FactorChainWord(key, parent);\n }\n return child;\n }\n }\n}\n\n" }, { "path": "src/opennlp/ccg/lexicon/FactorKey.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport opennlp.ccg.util.*;\n// import java.util.*;\n\n/** \n * Factor key, with identity equals for the factor and value. \n *\n * @author Michael White\n * @version $Revision: 1.1 $, $Date: 2005/03/22 20:59:32 $\n */\npublic class FactorKey {\n\n /** The factor. */\n public final String factor;\n\n /** The value. */\n public final String val;\n\n /** Constructor. Assumes interned components. */\n private FactorKey(String factor, String val) { this.factor = factor; this.val = val; }\n\n /** Makes/retrieves an interned factor key for the given interned attr and val; \n for the word form, the string itself is returned. \n Null vals are replaced with <NULL>. */\n public static Object getKey(String attr, String val) {\n if (val == null) val = \"\";\n if (attr == Tokenizer.WORD_ATTR) return val;\n else return Interner.globalIntern(new FactorKey(attr, val));\n }\n\n /** Returns a hash code constructed from the component identity hash codes. */\n public int hashCode() { \n return System.identityHashCode(factor) - System.identityHashCode(val);\n }\n\n /** Returns true if the given factor key has identical components. */\n public boolean equals(Object obj) {\n if (this == obj) return true;\n if (!(obj instanceof FactorKey)) return false;\n FactorKey key = (FactorKey) obj;\n return factor == key.factor && val == key.val;\n }\n\n /** Returns \"factor-val\". */\n public String toString() { return factor + \"-\" + val; }\n} \n\n" }, { "path": "src/opennlp/ccg/lexicon/Family.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-9 Jason Baldridge, Gann Bierner and \n// University of Edinburgh / Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport org.jdom.*;\nimport java.util.*;\n\n/**\n * Lexicon category family.\n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.11 $, $Date: 2010/11/30 18:51:05 $\n */\npublic class Family {\n\t\n private String name = \"\";\n private Boolean closed = Boolean.FALSE;\n private String pos = \"\";\n private String indexRel = \"\"; \n private String coartRel = \"\"; \n private DataItem[] data;\n private EntriesItem[] entries;\n\n @SuppressWarnings(\"unchecked\")\n\tpublic Family(Element famel) {\n \t\n setName(famel.getAttributeValue(\"name\"));\n pos = famel.getAttributeValue(\"pos\");\n \n String isClosed = famel.getAttributeValue(\"closed\");\n if (isClosed != null && isClosed.equals(\"true\")) {\n setClosed(Boolean.TRUE);\n }\n\n String indexRelVal = famel.getAttributeValue(\"indexRel\");\n if (indexRelVal != null) { indexRel = indexRelVal; }\n\n String coartRelVal = famel.getAttributeValue(\"coartRel\");\n if (coartRelVal != null) { coartRel = coartRelVal; }\n\n List entriesList = famel.getChildren(\"entry\");\n entries = new EntriesItem[entriesList.size()];\n for (int j=0; j < entriesList.size(); j++) {\n entries[j] = new EntriesItem(entriesList.get(j), this);\n }\n \n List members = famel.getChildren(\"member\");\n data = new DataItem[members.size()];\n for (int j=0; j < members.size(); j++) {\n data[j] = new DataItem(members.get(j));\n }\n }\n\n public Family(String s) { setName(s); }\n\n public boolean isClosed() { return closed.booleanValue(); }\n \n public void setName(String s) { name = s; }\n public void setClosed(Boolean b) { closed = b; }\n public void setPOS(String s) { pos = s; }\n public void setIndexRel(String s) { indexRel = s; }\n public void setCoartRel(String s) { coartRel = s; }\n public void setData(DataItem[] dm) { data = dm; }\n public void setEntries(EntriesItem[] em) { entries = em; }\n\n public String getName() { return name; }\n /** Delegates to first entry. */\n public String getSupertag() { return entries[0].getSupertag(); } \n public Boolean getClosed() { return closed; }\n public String getPOS() { return pos; }\n public String getIndexRel() { return indexRel; }\n public String getCoartRel() { return coartRel; }\n public DataItem[] getData() { return data; }\n public EntriesItem[] getEntries() { return entries; }\n}\n" }, { "path": "src/opennlp/ccg/lexicon/FullWord.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004-5 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport opennlp.ccg.util.*;\n\nimport java.util.*;\n\n/**\n * A FullWord object is a word with all possible fields. \n * The factory methods return interned objects.\n *\n * @author Michael White\n * @version $Revision: 1.6 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class FullWord extends WordWithPitchAccent {\n \n\tprivate static final long serialVersionUID = -3115687437782457735L;\n\n\t/** List of attribute-value pairs, which must be strings. */\n protected List> attrValPairs;\n \n /** The stem. */\n protected String stem;\n \n /** The part of speech. */\n protected String POS;\n \n /** The supertag. */\n protected String supertag;\n \n /** The semantic class (optional). */\n protected String semClass;\n\n \n /** Returns the list of extra attribute-value pairs. */\n protected List> getAttrValPairsList() { return attrValPairs; }\n \n /** Returns the stem. */\n public String getStem() { return stem; }\n \n /** Returns the part of speech. */\n public String getPOS() { return POS; }\n \n /** Returns the supertag. */\n public String getSupertag() { return supertag; }\n \n /** Returns the semantic class (may be null). */\n public String getSemClass() { return semClass; }\n\n \n /** Constructor for full word. */\n protected FullWord(\n String form, String pitchAccent, List> attrValPairs, \n String stem, String POS, String supertag, String semClass \n ) {\n super(form, pitchAccent);\n this.attrValPairs = attrValPairs; \n this.stem = stem; this.POS = POS; this.supertag = supertag; this.semClass = semClass; \n }\n \n /** Factory. */\n public static class Factory implements WordFactory {\n\n // reusable word, for looking up already interned ones\n private FullWord w = new FullWord(null, null, null, null, null, null, null);\n \n // sets the form and factors of the reusable word w \n private void setW(\n String form, String pitchAccent, List> attrValPairs, \n String stem, String POS, String supertag, String semClass \n ) {\n w.form = form; w.pitchAccent = pitchAccent;\n w.attrValPairs = attrValPairs;\n w.stem = stem; w.POS = POS; w.supertag = supertag; w.semClass = semClass;\n }\n \n // looks up the word equivalent to w, or if none, returns a new one based on it\n private Word getOrCreateFromW() {\n Word retval = (Word) Interner.getGlobalInterned(w);\n if (retval != null) return retval;\n if (w.isSurfaceWord() && w.attrValPairs == null) {\n if (w.pitchAccent == null) retval = new SimpleWord(w.form);\n else retval = new WordWithPitchAccent(w.form, w.pitchAccent);\n }\n else retval = new FullWord(w.form, w.pitchAccent, w.attrValPairs, w.stem, w.POS, w.supertag, w.semClass);\n return (Word) Interner.globalIntern(retval);\n }\n \n /** Creates a surface word with the given interned form. */\n public synchronized Word create(String form) {\n return create(form, null, null, null, null, null, null);\n }\n \n /** Creates a (surface or full) word with the given normalized attribute name and value.\n The attribute names Tokenizer.WORD_ATTR, ..., Tokenizer.SEM_CLASS_ATTR \n may be used for the form, ..., semantic class. */\n public synchronized Word create(String attr, String val) {\n String form = null; String pitchAccent = null;\n List> attrValPairs = null; \n String stem = null; String POS = null; String supertag = null; String semClass = null;\n if (attr == Tokenizer.WORD_ATTR) form = val;\n else if (attr == Tokenizer.PITCH_ACCENT_ATTR) pitchAccent = val;\n else if (attr == Tokenizer.STEM_ATTR) stem = val;\n else if (attr == Tokenizer.POS_ATTR) POS = val;\n else if (attr == Tokenizer.SUPERTAG_ATTR) supertag = val;\n else if (attr == Tokenizer.SEM_CLASS_ATTR) semClass = val;\n else {\n attrValPairs = new ArrayList>(1);\n attrValPairs.add(new Pair(attr, val));\n }\n return create(form, pitchAccent, attrValPairs, stem, POS, supertag, semClass);\n }\n \n /** Creates a (surface or full) word from the given canonical factors. */\n public synchronized Word create(\n String form, String pitchAccent, List> attrValPairs, \n String stem, String POS, String supertag, String semClass \n ) {\n setW(form, pitchAccent, attrValPairs, stem, POS, supertag, semClass);\n return getOrCreateFromW();\n }\n }\n}\n\n" }, { "path": "src/opennlp/ccg/lexicon/LexException.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-3 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\n/**\n * Any exception thrown if something wrong happens in the lexicon.\n *\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.3 $, $Date: 2005/10/20 17:30:30 $\n */\npublic class LexException extends Exception {\n\n\tprivate static final long serialVersionUID = 1L;\n\t\n\t/** The message. */\n protected String msg;\n\n /**\n * Constructor with message.\n */ \n public LexException(String s) { msg = s; }\n\n /** Returns exception message. */\n public String toString() {\n return \"Lexicon Exception: \" + msg;\n }\n}\n" }, { "path": "src/opennlp/ccg/lexicon/Lexicon.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004-9 Jason Baldridge, Gann Bierner and \n// Michael White (University of Edinburgh, The Ohio State University)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport opennlp.ccg.grammar.*;\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.util.*;\nimport opennlp.ccg.hylo.*;\n\nimport org.jdom.*;\n\nimport java.io.*;\nimport java.net.*;\nimport java.util.*;\n\nimport gnu.trove.*;\n\n\n/**\n * Contains words and their associated categories and semantics.\n * Lookup can be filtered by plugging in a supertagger.\n * \n *\n * @author Gann Bierner\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.78 $, $Date: 2011/10/31 02:01:06 $\n */\npublic class Lexicon { \n \n /** Flag used to indicate a purely syntactic edge, with no associated semantics. */\n public static final String NO_SEM_FLAG = \"*NoSem*\";\n \n /** Constant used to signal the substitution of the stem or pred. */\n public static final String DEFAULT_VAL = \"[*DEFAULT*]\";\n \n // supertagger\n private SupertaggerAdapter _supertagger = null;\n \n // various maps\n private GroupMap _words;\n private GroupMap _stems;\n private GroupMap _macros;\n private HashMap _macroItems;\n\n private GroupMap _posToEntries;\n private GroupMap _stagToEntries;\n private GroupMap _predToWords;\n private GroupMap _relsToPreds;\n private GroupMap _coartRelsToPreds;\n \n // coarticulation attrs\n private Set _coartAttrs;\n private Set _indexedCoartAttrs;\n \n // attrs per atomic category type, across all entries\n private GroupMap _catsToAttrs;\n private Set _lfAttrs;\n \n // distributive attributes\n private String[] _distributiveAttrs = null;\n \n // licensing features\n private LicensingFeature[] _licensingFeatures = null;\n \n // relation sorting\n private HashMap _relationIndexMap = new HashMap();\n \n // interner for caching lex lookups during realization\n private Interner lookupCache = new Interner(true);\n \n /** The grammar that this lexicon is part of. */\n public final Grammar grammar;\n \n /** The tokenizer. (Defaults to DefaultTokenizer.) */\n public final Tokenizer tokenizer;\n \n /** Flag for whether the lexicon is open, ie complete lexical category mappings are not expected. (Defaults to false.) */\n public boolean openlex = false;\n \n /** Flag for whether to show warnings for failed sem class unification. (Defaults to false.) */\n public boolean debugSemClasses = false;\n \n\n /*************************************************************\n * Constructor\n *************************************************************/\n public Lexicon(Grammar grammar) {\n this.grammar = grammar;\n this.tokenizer = new DefaultTokenizer();\n }\n\n /** Constructor with tokenizer. */\n public Lexicon(Grammar grammar, Tokenizer tokenizer) {\n this.grammar = grammar;\n this.tokenizer = tokenizer;\n }\n\n //-------------------------------------------------------------\n \n /** Sets the supertagger (null if none). */\n public void setSupertagger(SupertaggerAdapter supertagger) { _supertagger = supertagger; }\n \n \n /** Loads the lexicon and morph files. */\n public void init(URL lexiconUrl, URL morphUrl) throws IOException {\n \n \tList lexicon = null;\n List morph = null;\n List macroModel = null;\n\n // load category families (lexicon), morph forms and macros\n lexicon = getLexicon(lexiconUrl);\n Pair,List> morphInfo = getMorph(morphUrl);\n morph = morphInfo.a; macroModel = morphInfo.b;\n\n // index words; also index stems to words, as default preds\n // store indexed coarticulation attrs too\n _words = new GroupMap();\n _predToWords = new GroupMap();\n _coartAttrs = new HashSet();\n _indexedCoartAttrs = new HashSet();\n for (MorphItem morphItem : morph) {\n Word surfaceWord = morphItem.getSurfaceWord();\n _words.put(surfaceWord, morphItem);\n _predToWords.put(morphItem.getWord().getStem(), surfaceWord);\n if (morphItem.isCoart()) {\n Word indexingWord = morphItem.getCoartIndexingWord();\n _words.put(indexingWord, morphItem);\n Pair first = indexingWord.getSurfaceAttrValPairs().next();\n _indexedCoartAttrs.add(first.a);\n for (Iterator> it = surfaceWord.getSurfaceAttrValPairs(); it.hasNext(); ) {\n \tPair p = it.next();\n _coartAttrs.add(p.a);\n }\n }\n }\n\n // index entries based on stem+pos\n _stems = new GroupMap();\n _posToEntries = new GroupMap();\n // index entries by supertag+pos, for supertagging\n _stagToEntries = new GroupMap();\n // also index rels and coart rels to preds\n _relsToPreds = new GroupMap();\n _coartRelsToPreds = new GroupMap();\n // and gather list of attributes used per atomic category type \n _catsToAttrs = new GroupMap();\n _lfAttrs = new HashSet();\n // and remember family and ent, names, for checking excluded list on morph items\n HashSet familyAndEntryNames = new HashSet();\n \n // index each family\n for (Family family : lexicon) {\n\n \tfamilyAndEntryNames.add(family.getName());\n EntriesItem[] entries = family.getEntries();\n DataItem[] data = family.getData();\n\n // for generic use when we get an unknown stem\n // from the morphological analyzer\n if (!family.isClosed()) {\n _posToEntries.put(family.getPOS(), entries);\n }\n\n // scan through entries\n for (int j=0; j < entries.length; j++) {\n // index\n EntriesItem eItem = entries[j];\n \t_stagToEntries.put(eItem.getSupertag()+family.getPOS(), eItem);\n if (eItem.getStem().length() > 0) {\n _stems.put(eItem.getStem()+family.getPOS(), eItem);\n }\n try {\n // gather features\n eItem.getCat().forall(gatherAttrs);\n // record names\n familyAndEntryNames.add(eItem.getName());\n familyAndEntryNames.add(eItem.getQualifiedName());\n }\n catch (RuntimeException exc) {\n System.err.println(\"exception for: \" + family.getName() + \": \" + exc);\n }\n }\n\n // scan through data\n for (int j=0; j < data.length; j++) {\n DataItem dItem = data[j];\n _stems.put(dItem.getStem()+family.getPOS(), new Pair(dItem,entries));\n // index non-default preds to words\n if (!dItem.getStem().equals(dItem.getPred())) {\n Collection words = (Collection) _predToWords.get(dItem.getStem());\n if (words == null) {\n \tif (!openlex) {\n\t System.out.print(\"Warning: couldn't find words for pred '\");\n\t System.out.println(dItem.getPred() + \"' with stem '\" + dItem.getStem() + \"'\");\n \t}\n }\n else {\n for (Iterator it = words.iterator(); it.hasNext(); ) {\n _predToWords.put(dItem.getPred(), it.next());\n }\n }\n }\n }\n\n // index rels to preds\n // nb: this covers relational (eg @xe) and featural (eg @epast) \n // elementary predications\n List indexRels = new ArrayList(3);\n String familyIndexRel = family.getIndexRel();\n if (familyIndexRel.length() > 0) { \n indexRels.add(familyIndexRel); \n }\n for (int j=0; j < entries.length; j++) {\n EntriesItem eItem = entries[j];\n String indexRel = eItem.getIndexRel();\n if (indexRel.length() > 0 && !indexRel.equals(familyIndexRel)) {\n indexRels.add(indexRel);\n }\n }\n for (Iterator it = indexRels.iterator(); it.hasNext(); ) {\n String indexRel = it.next();\n // nb: not indexing on entries items, b/c some stems are still defaults \n for (int j=0; j < data.length; j++) {\n DataItem dItem = data[j];\n _relsToPreds.put(indexRel, dItem.getPred());\n }\n }\n \n // index coart rels (features, really) to preds\n String coartRel = family.getCoartRel();\n if (coartRel.length() > 0) {\n for (int j=0; j < data.length; j++) {\n _coartRelsToPreds.put(coartRel, data[j].getPred());\n }\n }\n }\n\n // index the macros\n _macros = new GroupMap();\n // nb: could just index MacroItem objects for feature structures too;\n // this might be a bit cleaner, but life is short\n _macroItems = new HashMap();\n for (MacroItem mi : macroModel) {\n String macName = mi.getName();\n FeatureStructure[] specs = mi.getFeatureStructures();\n for (int j=0; j < specs.length; j++) {\n _macros.put(macName, specs[j]);\n }\n // this is for handling LF part of macros\n _macroItems.put(macName, mi);\n }\n\n // with morph items, check POS, macro names, excluded list for xref\n for (MorphItem morphItem : morph) {\n Word w = morphItem.getWord();\n if (!openlex && \n \t!_stems.containsKey(w.getStem() + w.getPOS()) &&\n !_posToEntries.containsKey(w.getPOS())) \n {\n System.err.println(\n \"Warning: no entries for stem '\" + w.getStem() + \n \"' and POS '\" + w.getPOS() + \n \"' found for word '\" + w + \"'\"\n );\n }\n String[] macroNames = morphItem.getMacros();\n for (int j=0; j < macroNames.length; j++) {\n if (!_macroItems.containsKey(macroNames[j])) {\n System.err.println(\"Warning: macro \" + macroNames[j] + \n \" not found for word '\" + morphItem.getWord() + \"'\");\n }\n }\n String[] excludedNames = morphItem.getExcluded();\n for (int j=0; j < excludedNames.length; j++) {\n if (!familyAndEntryNames.contains(excludedNames[j])) {\n System.err.println(\"Warning: excluded family or entry '\" + excludedNames[j] + \n \"' not found for word '\" + morphItem.getWord() + \"'\");\n }\n }\n }\n }\n \n /** Expands inheritsFrom links to feature equations for those features not explicitly listed. */ \n public void expandInheritsFrom(Category cat) {\n expandInheritsFrom(cat, null);\n }\n \n /** Expands inheritsFrom links to feature equations for those features not explicitly listed. */ \n public void expandInheritsFrom(Category cat, Category cat2) {\n // index feature structures\n featStrucMap.clear();\n cat.forall(indexFeatStrucs);\n if (cat2 != null) { cat2.forall(indexFeatStrucs); }\n // add feature eqs \n cat.forall(doInheritsFrom);\n if (cat2 != null) { cat2.forall(doInheritsFrom); }\n }\n \n // gathers attrs from a category\n private CategoryFcn gatherAttrs = new CategoryFcnAdapter() {\n public void forall(Category c) {\n if (!(c instanceof AtomCat)) return;\n String type = ((AtomCat)c).getType();\n FeatureStructure fs = c.getFeatureStructure();\n if (fs == null) return;\n for (Iterator it = fs.getAttributes().iterator(); it.hasNext(); ) {\n String att = it.next();\n _catsToAttrs.put(type, att);\n if (fs.getValue(att) instanceof LF) {\n _lfAttrs.add(att);\n }\n }\n }\n };\n\n // a map from indices to atomic categories, reset for each category\n private TIntObjectHashMap featStrucMap = new TIntObjectHashMap();\n \n // fills in featStrucMap for a category\n private CategoryFcn indexFeatStrucs = new CategoryFcnAdapter() {\n public void forall(Category c) {\n FeatureStructure fs = c.getFeatureStructure();\n if (fs != null && fs.getIndex() != 0)\n featStrucMap.put(fs.getIndex(), fs);\n }\n };\n\n // adds feature equations to percolate attributes from inheritsFrom feature \n // structure, except for any attributes already present\n private CategoryFcn doInheritsFrom = new CategoryFcnAdapter() {\n public void forall(Category c) {\n // get feature structures\n if (!(c instanceof AtomCat)) return;\n String type = ((AtomCat)c).getType();\n FeatureStructure fs = c.getFeatureStructure();\n GFeatStruc gfs = (GFeatStruc) fs;\n if (gfs == null || gfs.getInheritsFrom() == 0) return;\n int inhf = gfs.getInheritsFrom();\n FeatureStructure inhfFS = (FeatureStructure) featStrucMap.get(inhf);\n if (inhfFS != null) {\n // copy values of features from inhfFS not already present\n for (Iterator it = inhfFS.getAttributes().iterator(); it.hasNext(); ) {\n String att = it.next(); \n if (gfs.hasAttribute(att)) continue;\n gfs.setFeature(att, UnifyControl.copy(inhfFS.getValue(att)));\n }\n // for each possible attr used with this type and not already present, \n // add feature equation\n Collection attrs = (Collection) _catsToAttrs.get(type);\n if (attrs == null) return;\n for (Iterator it = attrs.iterator(); it.hasNext(); ) {\n String att = it.next(); \n if (gfs.hasAttribute(att)) continue;\n String varName = att.toUpperCase() + inhf;\n if (_lfAttrs.contains(att)) {\n gfs.setFeature(att, new HyloVar(varName));\n inhfFS.setFeature(att, new HyloVar(varName));\n }\n else {\n gfs.setFeature(att, new GFeatVar(varName));\n inhfFS.setFeature(att, new GFeatVar(varName));\n }\n }\n }\n else {\n System.err.println(\n \"Warning: no feature structure with inheritsFrom index of \" + inhf + \n \" found in category \" + c\n );\n }\n }\n };\n\n \n /**\n * Returns the lexical signs indexed by the given rel, or null if none. \n */\n public Collection getSignsFromRel(String rel) {\n // check cache (if not doing supertagging)\n \tif (_supertagger == null) {\n\t RelLookup lookup = new RelLookup(rel);\n\t RelLookup retLookup = (RelLookup) lookupCache.getInterned(lookup);\n\t if (retLookup != null) return retLookup.signs;\n \t}\n // lookup signs via preds\n Collection preds = (Collection) _relsToPreds.get(rel);\n if (preds == null) return null;\n Collection retval = getSignsFromRelAndPreds(rel, preds);\n // cache non-null result (if not doing supertagging)\n if (_supertagger == null && retval != null) {\n \tRelLookup lookup = new RelLookup(rel);\n lookup.signs = retval; lookupCache.intern(lookup);\n }\n return retval;\n }\n\n // get signs for rel via preds, or null if none\n private Collection getSignsFromRelAndPreds(String rel, Collection preds) {\n List retval = new ArrayList();\n for (Iterator it = preds.iterator(); it.hasNext(); ) {\n String pred = it.next();\n Collection signs = getSignsFromPredAndTargetRel(pred, rel);\n if (signs != null) retval.addAll(signs);\n }\n // return null if none survive filter\n if (retval.size() > 0) return retval;\n else return null;\n }\n\n /**\n * Returns the lexical signs indexed by the given pred.\n * If the pred is not listed in the lexicon, the tokenizer is \n * consulted to see if it is a special token (date, time, etc.); \n * otherwise, null is returned.\n * Coarticulations are applied for the given rels, if non-null.\n */\n public Collection getSignsFromPred(String pred, List coartRels) {\n // check cache (if not doing supertagging)\n \tif (_supertagger == null) {\n\t PredLookup lookup = new PredLookup(pred, coartRels);\n\t PredLookup retLookup = (PredLookup) lookupCache.getInterned(lookup);\n\t if (retLookup != null) return retLookup.signs;\n \t}\n // lookup pred\n Collection result = getSignsFromPredAndTargetRel(pred, null);\n if (result == null) return null;\n // apply coarts for rels\n if (coartRels != null) applyCoarts(coartRels, result);\n // cache result (if not doing supertagging)\n \tif (_supertagger == null) {\n\t PredLookup lookup = new PredLookup(pred, coartRels);\n \t\tlookup.signs = result; lookupCache.intern(lookup);\n \t}\n // and return\n return result;\n }\n \n // get signs using an additional arg for a target rel\n private Collection getSignsFromPredAndTargetRel(String pred, String targetRel) {\n \n Collection words = (Collection) _predToWords.get(pred);\n String specialTokenConst = null;\n \n // for robustness, when using supertagger, add words for pred sans sense index\n int dotIndex = -1;\n if (_supertagger != null && !Character.isDigit(pred.charAt(0)) && // skip numbers \n \t\t(dotIndex = pred.lastIndexOf('.')) > 0 && pred.length() > dotIndex+1 &&\n \t\tpred.charAt(dotIndex+1) != '_') // skip titles, eg Mr._Smith\n {\n \tString barePred = pred.substring(0, dotIndex);\n \tCollection barePredWords = (Collection) _predToWords.get(barePred);\n \tif (words == null) words = barePredWords;\n \telse if (barePredWords != null) {\n \t\tSet unionWords = new HashSet(words);\n \t\tunionWords.addAll(barePredWords);\n \t\twords = unionWords;\n \t}\n }\n \n if (words == null) {\n specialTokenConst = tokenizer.getSpecialTokenConstant(tokenizer.isSpecialToken(pred));\n if (specialTokenConst == null) return null;\n // lookup words with pred = special token const\n Collection specialTokenWords = (Collection) _predToWords.get(specialTokenConst);\n // replace special token const with pred\n if (specialTokenWords == null) return null;\n words = new ArrayList(specialTokenWords.size());\n for (Iterator it = specialTokenWords.iterator(); it.hasNext(); ) {\n Word stw = it.next();\n Word w = Word.createSurfaceWord(stw, pred);\n words.add(w);\n }\n }\n \n List retval = new ArrayList();\n for (Iterator it = words.iterator(); it.hasNext(); ) {\n Word w = it.next();\n try {\n SignHash signs = getSignsFromWord(w, specialTokenConst, pred, targetRel);\n retval.addAll(signs.asSignSet());\n }\n // shouldn't happen\n catch (LexException exc) {\n System.err.println(\"Unexpected lex exception for word \" + w + \": \" + exc);\n }\n }\n return retval;\n }\n \n // look up and apply coarts for given rels to each sign in result\n private void applyCoarts(List coartRels, Collection result) {\n List inputSigns = new ArrayList(result);\n result.clear();\n List outputSigns = new ArrayList(inputSigns.size());\n // for each rel, lookup coarts and apply to input signs, storing results in output signs\n for (Iterator it = coartRels.iterator(); it.hasNext(); ) {\n String rel = it.next();\n Collection preds = (Collection) _coartRelsToPreds.get(rel);\n if (preds == null) continue; // not expected\n Collection coartResult = getSignsFromRelAndPreds(rel, preds);\n if (coartResult == null) continue;\n for (Iterator it2 = coartResult.iterator(); it2.hasNext(); ) {\n Sign coartSign = it2.next();\n // apply to each input\n for (int j = 0; j < inputSigns.size(); j++) {\n Sign sign = inputSigns.get(j);\n grammar.rules.applyCoart(sign, coartSign, outputSigns);\n }\n }\n // switch output to input for next iteration\n inputSigns.clear();\n inputSigns.addAll(outputSigns);\n outputSigns.clear();\n }\n // add results back\n result.addAll(inputSigns);\n }\n\n \n /**\n * For a string of 1 or more surface words, return all of the lexical\n * entries for each word as a list of sign hashes.\n * Tokenization is performed using the configured tokenizer.\n *\n * @param w the words in string format\n * @return a list of sign hashes\n * @exception LexException thrown if word not found\n */\n public List getEntriesFromWords(String s) throws LexException { \n List entries = new ArrayList();\n List words = tokenizer.tokenize(s);\n for (Iterator it = words.iterator(); it.hasNext(); ) {\n Word w = it.next();\n SignHash signs = getSignsFromWord(w);\n if (signs.size() == 0) {\n throw new LexException(\"Word not in lexicon: \\\"\" + w +\"\\\"\");\n }\n entries.add(signs);\n }\n return entries;\n }\n \n /**\n * For a given word, return all of its surface word's lexical entries.\n * If the word is not listed in the lexicon, the tokenizer is \n * consulted to see if it is a special token (date, time, etc.); \n * otherwise an exception is thrown.\n * If the word has coarticulations, all applicable coarticulation \n * entries are applied to the base word, in an arbitrary order.\n *\n * @param w the word\n * @return a sign hash\n * @exception LexException thrown if word not found\n */\n public SignHash getSignsFromWord(Word w) throws LexException {\n // reduce word to its core, removing coart attrs if any\n \tWord surfaceWord = Word.createSurfaceWord(w);\n Word coreWord = (surfaceWord.attrsIntersect(_coartAttrs)) \n ? Word.createCoreSurfaceWord(surfaceWord, _coartAttrs) \n : surfaceWord;\n // lookup core word\n SignHash result = getSignsFromWord(coreWord, null, null, null);\n if (result.size() == 0) {\n throw new LexException(coreWord + \" not found in lexicon\");\n }\n // return signs if no coart attrs\n if (coreWord == surfaceWord) return result; \n // otherwise apply coarts for word\n applyCoarts(surfaceWord, result);\n return result; \n }\n \n // look up and apply coarts for w to each sign in result\n @SuppressWarnings(\"unchecked\")\n\tprivate void applyCoarts(Word w, SignHash result) throws LexException {\n List inputSigns = new ArrayList(result.asSignSet());\n result.clear();\n List outputSigns = new ArrayList(inputSigns.size());\n // for each surface attr, lookup coarts and apply to input signs, storing results in output signs\n for (Iterator> it = w.getSurfaceAttrValPairs(); it.hasNext(); ) {\n Pair p = it.next();\n String attr = (String) p.a;\n if (!_indexedCoartAttrs.contains(attr)) continue;\n String val = (String) p.b;\n Word coartWord = Word.createWord(attr, val);\n SignHash coartResult = getSignsFromWord(coartWord, null, null, null);\n for (Iterator it2 = coartResult.iterator(); it2.hasNext(); ) {\n Sign coartSign = it2.next();\n // apply to each input\n for (int j = 0; j < inputSigns.size(); j++) {\n Sign sign = inputSigns.get(j);\n grammar.rules.applyCoart(sign, coartSign, outputSigns);\n }\n }\n // switch output to input for next iteration\n inputSigns.clear();\n inputSigns.addAll(outputSigns);\n outputSigns.clear();\n }\n // add results back\n result.addAll(inputSigns);\n }\n \n // get signs with additional args for a known special token const, target pred and target rel \n private SignHash getSignsFromWord(Word w, String specialTokenConst, String targetPred, String targetRel) throws LexException {\n\n Collection morphItems = (specialTokenConst == null)\n ? (Collection) _words.get(w)\n : null;\n\n if (morphItems == null) {\n // check for special tokens\n if (specialTokenConst == null) {\n specialTokenConst = tokenizer.getSpecialTokenConstant(tokenizer.isSpecialToken(w.getForm()));\n targetPred = w.getForm();\n }\n if (specialTokenConst != null) {\n Word key = Word.createSurfaceWord(w, specialTokenConst);\n morphItems = (Collection) _words.get(key);\n }\n // otherwise throw lex exception\n if (morphItems == null)\n throw new LexException(w + \" not in lexicon\");\n }\n\n SignHash result = new SignHash();\n\n for (Iterator MI = morphItems.iterator(); MI.hasNext();) {\n getWithMorphItem(w, MI.next(), targetPred, targetRel, result);\n }\n\n return result;\n }\n\n\n // given MorphItem\n private void getWithMorphItem(Word w, MorphItem mi, String targetPred, String targetRel, SignHash result)\n throws LexException \n {\n \t// get supertags for filtering, if a supertagger is installed\n \tMap supertags = null;\n \tSet supertagsFound = null;\n \tif (_supertagger != null) {\n \t\tsupertags = _supertagger.getSupertags();\n \t\tif (supertags != null) supertagsFound = new HashSet(supertags.size());\n \t}\n \t\n // get macro adder\n MacroAdder macAdder = getMacAdder(mi);\n \n // if we have this stem in our lexicon\n String stem = mi.getWord().getStem();\n String pos = mi.getWord().getPOS();\n Set explicitEntries = null; // for storing entries from explicitly listed family members\n if (_stems.containsKey(stem+pos)) {\n explicitEntries = new HashSet();\n Collection stemItems = (Collection)_stems.get(stem+pos);\n for (Iterator I=stemItems.iterator(); I.hasNext();) {\n Object item = I.next();\n // see if it's an EntriesItem\n if (item instanceof EntriesItem) {\n EntriesItem entry = (EntriesItem) item;\n // do lookup\n getWithEntriesItem(w, mi, stem, stem, targetPred, targetRel, entry, macAdder, supertags, supertagsFound, result);\n } \n // otherwise it has to be a Pair containing a DataItem and \n // an EntriesItem[]\n else {\n @SuppressWarnings(\"rawtypes\")\n\t\t\t\t\tDataItem dItem = (DataItem)((Pair)item).a;\n @SuppressWarnings(\"rawtypes\")\n\t\t\t\t\tEntriesItem[] entries = (EntriesItem[])((Pair)item).b;\n // store entries\n explicitEntries.add(entries);\n // do lookup\n getWithDataItem(w, mi, dItem, entries, targetPred, targetRel, macAdder, supertags, supertagsFound, result);\n }\n }\n }\n \n // for entries that are not explicitly in the lexicon file, we have to create\n // Signs from the open class entries with the appropriate part-of-speech\n Collection entrySets = (Collection)_posToEntries.get(pos);\n if (entrySets != null) {\n\t for (Iterator E=entrySets.iterator(); E.hasNext(); ) {\n\t EntriesItem[] entries = E.next(); \n\t // skip if entries explicitly listed\n\t if (explicitEntries != null && explicitEntries.contains(entries)) continue;\n\t // otherwise get entries with pred = targetPred, or stem if null\n\t String pred = (targetPred != null) ? targetPred : stem;\n\t getWithDataItem(w, mi, new DataItem(stem, pred), entries, targetPred, targetRel, macAdder, supertags, supertagsFound, result);\n\t }\n }\n \n // finally do entries for any remaining supertags\n if (supertags != null) {\n \tfor (String supertag : supertags.keySet()) {\n \t\tif (supertagsFound.contains(supertag)) continue;\n \t\tSet entries = _stagToEntries.get(supertag+pos);\n \t\tif (entries == null) continue; // nb: could be a POS mismatch\n\t // get entries with pred = targetPred, or stem if null\n\t String pred = (targetPred != null) ? targetPred : stem;\n \t\tfor (EntriesItem entry : entries) {\n \t\t\tif (!entry.getStem().equals(DEFAULT_VAL)) continue;\n\t\t getWithEntriesItem(w, mi, stem, pred, targetPred, targetRel, entry, macAdder, supertags, supertagsFound, result);\n \t\t}\n \t}\n }\n }\n\n \n // given DataItem\n private void getWithDataItem(Word w, MorphItem mi, \n DataItem item, EntriesItem[] entries, \n String targetPred, String targetRel, \n MacroAdder macAdder,\n Map supertags,\n Set supertagsFound,\n SignHash result) \n {\n for (int i=0; i < entries.length; i++) {\n EntriesItem entry = entries[i];\n if (entry.getStem().equals(DEFAULT_VAL)) {\n getWithEntriesItem(w, mi, item.getStem(), item.getPred(), targetPred, targetRel, entry, macAdder, supertags, supertagsFound, result);\n }\n }\n }\n\n // given EntriesItem\n private void getWithEntriesItem(Word w, MorphItem mi, \n String stem, String pred, \n String targetPred, String targetRel,\n EntriesItem item,\n MacroAdder macAdder,\n Map supertags,\n Set supertagsFound,\n SignHash result) \n {\n // ensure apropos\n if (targetPred != null && !targetPred.equals(pred)) return; \n if (targetRel != null && !targetRel.equals(item.getIndexRel()) && !targetRel.equals(item.getCoartRel())) return; \n if (!item.getActive().booleanValue()) return;\n if (mi.excluded(item)) return;\n \n try {\n\t // copy and add macros\n\t Category cat = item.getCat().copy();\n\t macAdder.addMacros(cat);\n\t\n\t // replace DEFAULT_VAL with pred, after first \n\t // unifying type of associated nom var(s) with sem class \n\t unifySemClass(cat, mi.getWord().getSemClass());\n\t REPLACEMENT = pred; \n\t cat.deepMap(defaultReplacer);\n\t \n\t // check supertag\n\t // TODO: think about earlier checks for efficiency, for grammars where macros and preds don't matter\n\t //Double lexprob = null; // nb: skipping lex log probs, don't seem to be helpful\n\t if (supertags != null) {\n\t \t// skip if not found\n\t \tString stag = cat.getSupertag();\n\t \tif (!supertags.containsKey(stag)) return;\n\t \t// otherwise update found supertags\n\t \tsupertagsFound.add(stag);\n\t \t// get lex prob\n\t \t//lexprob = supertags.get(stag);\n\t }\n\t \n\t // propagate types of nom vars\n\t propagateTypes(cat);\n\t \n\t // handle distrib attrs and inherits-from\n\t propagateDistributiveAttrs(cat);\n\t expandInheritsFrom(cat);\n\t \n\t // merge stem, pos, sem class from morph item, plus supertag from cat\n\t Word word = Word.createFullWord(w, mi.getWord(), cat.getSupertag());\n\n\t // set origin and lexprob\n\t Sign sign = new Sign(word, cat);\n\t sign.setOrigin();\n\t //if (lexprob != null) {\n\t //\tsign.addData(new SupertaggerAdapter.LexLogProb((float) Math.log10(lexprob)));\n\t //}\n\t // return sign\n\t result.insert(sign);\n }\n catch (RuntimeException exc) {\n \tSystem.err.println(\n \t\t\t\"Warning: ignoring entry: \" + item.getName() + \" of family: \" + item.getFamilyName() + \n \t\t\t\" for stem: \" + stem + \" b/c: \" + exc.toString()\n \t);\n }\n }\n\n // the sem class for defaultNomvarSetter\n private SimpleType SEMCLASS = null;\n \n // unify sem class with default nom var(s)\n private void unifySemClass(Category cat, String semClass) {\n if (semClass == null || cat.getLF() == null) return;\n SEMCLASS = grammar.types.getSimpleType(semClass);\n try {\n cat.getLF().deepMap(defaultNomvarUnifier);\n } catch (TypePropagationException tpe) {\n \tif (debugSemClasses) {\n\t System.err.println(\n\t \"Warning: unable to unify types '\" + tpe.st1 + \"' and '\" + tpe.st2 + \n\t \"' in unifying sem class in cat: \\n\" + cat\n\t );\n \t}\n }\n }\n \n // mod function to unify type of nom var for DEFAULT_VAL with SEMCLASS\n private ModFcn defaultNomvarUnifier = new ModFcn() {\n public void modify(Mutable m) {\n if (!(m instanceof SatOp)) return;\n SatOp satop = (SatOp) m;\n if (!(satop.getArg() instanceof Proposition)) return; \n Proposition prop = (Proposition) satop.getArg();\n if (!prop.getName().equals(DEFAULT_VAL)) return;\n if (!(satop.getNominal() instanceof NominalVar)) return;\n NominalVar nv = (NominalVar) satop.getNominal();\n SimpleType st = nv.getType();\n // check equality\n if (st.equals(SEMCLASS)) return;\n // otherwise unify types, update nv\n try {\n SimpleType stU = (SimpleType) st.unify(SEMCLASS, null);\n nv.setType(stU);\n } catch (UnifyFailure uf) {\n throw new TypePropagationException(st, SEMCLASS);\n }\n }\n };\n\n // the replacement string for defaultReplacer\n private String REPLACEMENT = \"\";\n \n // mod function to replace DEFAULT_VAL with REPLACEMENT\n private ModFcn defaultReplacer = new ModFcn() {\n public void modify(Mutable m) {\n if (m instanceof Proposition) {\n Proposition prop = (Proposition) m; \n if (prop.getName().equals(DEFAULT_VAL)) prop.setAtomName(REPLACEMENT);\n }\n else if (m instanceof FeatureStructure) {\n FeatureStructure fs = (FeatureStructure) m;\n for (Iterator it = fs.getAttributes().iterator(); it.hasNext(); ) {\n String attr = it.next();\n Object val = fs.getValue(attr);\n if (val instanceof SimpleType && \n ((SimpleType)val).getName().equals(DEFAULT_VAL))\n {\n fs.setFeature(attr, grammar.types.getSimpleType(REPLACEMENT));\n }\n }\n }\n }\n };\n\n\n // a cache for macro adders\n private Map macAdderMap = new HashMap();\n \n // returns a macro adder for the given morph item\n private MacroAdder getMacAdder(MorphItem mi) {\n \n // check map\n MacroAdder retval = macAdderMap.get(mi);\n if (retval != null) return retval;\n \n // set up macro adder\n IntHashSetMap macrosFromLex = new IntHashSetMap();\n String[] newMacroNames = mi.getMacros();\n List macroItems = new ArrayList();\n for (int i=0; i < newMacroNames.length; i++) {\n Set featStrucs = (Set)_macros.get(newMacroNames[i]);\n if (featStrucs != null) {\n for (Iterator fsIt = featStrucs.iterator(); fsIt.hasNext();) {\n FeatureStructure fs = fsIt.next();\n macrosFromLex.put(fs.getIndex(), fs);\n }\n }\n MacroItem macroItem = _macroItems.get(newMacroNames[i]);\n if (macroItem != null) { macroItems.add(macroItem); }\n else { \n // should be checked earlier too\n System.err.println(\"Warning: macro \" + newMacroNames[i] + \n \" not found for word '\" + mi.getWord() + \"'\");\n }\n }\n retval = new MacroAdder(macrosFromLex, macroItems);\n \n // update map and return\n macAdderMap.put(mi, retval);\n return retval; \n }\n \n \n //\n // type propagation\n //\n\n /** Propagates types of nomvars in the given category. */\n public void propagateTypes(Category cat) {\n propagateTypes(cat, null);\n } \n \n /** Propagates types of nomvars in the given categories. */\n public void propagateTypes(Category cat, Category cat2) {\n try {\n nomvarMap.clear();\n cat.deepMap(nomvarTypePropagater);\n if (cat2 != null) cat2.deepMap(nomvarTypePropagater);\n cat.deepMap(nomvarTypePropagater);\n if (cat2 != null) cat2.deepMap(nomvarTypePropagater);\n } catch (TypePropagationException tpe) {\n \tif (debugSemClasses) {\n\t System.err.println(\n\t \"Warning: unable to unify types '\" + tpe.st1 + \"' and '\" + tpe.st2 + \n\t \"' in cat: \\n\" + cat\n\t );\n\t if (cat2 != null) System.err.println(\"and cat: \\n\" + cat2);\n \t}\n }\n }\n \n // a map from a cat's nomvars to types, \n // just using the var's name for equality\n @SuppressWarnings(\"unchecked\")\n\tprivate Map nomvarMap = new THashMap(\n new TObjectHashingStrategy() {\n\t\t\tprivate static final long serialVersionUID = 1L;\n\t\t\tpublic int computeHashCode(Object o) {\n return ((NominalVar)o).getName().hashCode();\n }\n public boolean equals(Object o1, Object o2) {\n return ((NominalVar)o1).getName().equals(((NominalVar)o2).getName());\n }\n }\n );\n \n // exception for unification failures in propagating types\n private class TypePropagationException extends RuntimeException {\n\t\tprivate static final long serialVersionUID = 1L;\n\t\tSimpleType st1; SimpleType st2;\n TypePropagationException(SimpleType st1, SimpleType st2) {\n this.st1 = st1; this.st2 = st2;\n }\n }\n \n // mod function to propagate nomvar types; \n // needs to be called twice after clearing nomvarMap\n private ModFcn nomvarTypePropagater = new ModFcn() {\n public void modify(Mutable m) {\n if (m instanceof NominalVar) {\n NominalVar nv = (NominalVar) m;\n SimpleType st = nv.getType();\n SimpleType st0 = nomvarMap.get(nv);\n // add type to map if no type found\n if (st0 == null) { nomvarMap.put(nv, st); return; }\n // check equality\n if (st.equals(st0)) return;\n // otherwise unify types, update nv and map\n try {\n SimpleType stU = (SimpleType) st.unify(st0, null);\n nv.setType(stU);\n nomvarMap.put(nv, stU);\n } catch (UnifyFailure uf) {\n throw new TypePropagationException(st, st0);\n }\n }\n }\n };\n \n\n //\n // distributive attribute propagation\n //\n\n /**\n * Returns the list of distributive attributes, or null if none.\n */\n public String[] getDistributiveAttrs() { return _distributiveAttrs; }\n \n /**\n * Gathers and propagates the unique values of each \n * distributive attribute.\n */\n public void propagateDistributiveAttrs(Category cat) {\n propagateDistributiveAttrs(cat, null);\n }\n \n /**\n * Gathers and propagates the unique values of each \n * distributive attribute.\n */\n public void propagateDistributiveAttrs(Category cat, Category cat2) {\n if (_distributiveAttrs == null) return;\n resetDistrAttrVals();\n cat.forall(gatherDistrAttrVals);\n if (cat2 != null) { cat2.forall(gatherDistrAttrVals); }\n cat.forall(propagateUniqueDistrAttrVals);\n if (cat2 != null) { cat2.forall(propagateUniqueDistrAttrVals); }\n }\n \n // an array of lists, one for each distributive attr \n @SuppressWarnings(\"rawtypes\")\n\tprivate List[] distrAttrVals = null;\n @SuppressWarnings(\"rawtypes\")\n\tprivate void resetDistrAttrVals() {\n if (distrAttrVals == null) { \n distrAttrVals = new List[_distributiveAttrs.length];\n for (int i = 0; i < distrAttrVals.length; i++) {\n distrAttrVals[i] = new ArrayList(3);\n }\n return;\n }\n for (int i = 0; i < distrAttrVals.length; i++) {\n distrAttrVals[i].clear();\n }\n }\n \n // gathers distinct values for each distributive attr\n private CategoryFcn gatherDistrAttrVals = new CategoryFcnAdapter() {\n @SuppressWarnings(\"unchecked\")\n\t\tpublic void forall(Category c) {\n if (!(c instanceof AtomCat)) return;\n FeatureStructure fs = c.getFeatureStructure();\n if (fs == null) return;\n for (int i = 0; i < _distributiveAttrs.length; i++) {\n String attr = _distributiveAttrs[i];\n Object val = fs.getValue(attr);\n if (val != null && !distrAttrVals[i].contains(val)) { \n distrAttrVals[i].add(val); \n }\n }\n }\n };\n\n // propagates unique values for each distributive attr\n private CategoryFcn propagateUniqueDistrAttrVals = new CategoryFcnAdapter() {\n public void forall(Category c) {\n if (!(c instanceof AtomCat)) return;\n FeatureStructure fs = c.getFeatureStructure();\n if (fs == null) return;\n for (int i = 0; i < _distributiveAttrs.length; i++) {\n if (distrAttrVals[i].size() != 1) continue;\n Object distVal = distrAttrVals[i].get(0);\n String attr = _distributiveAttrs[i];\n Object val = fs.getValue(attr);\n if (val == null) {\n fs.setFeature(attr, UnifyControl.copy(distVal));\n }\n }\n }\n };\n\n \n //\n // licensing features\n //\n\n /**\n * Returns the list of licensing features.\n */\n public LicensingFeature[] getLicensingFeatures() { return _licensingFeatures; }\n \n \n /**\n * Returns the index of the given relation in the relation sort order, \n * or the index of \"*\" if the relation is not explicitly listed.\n */\n public Integer getRelationSortIndex(String rel) {\n Integer retval = _relationIndexMap.get(rel);\n if (retval != null) return retval;\n retval = _relationIndexMap.get(\"*\");\n if (retval != null) return retval;\n return new Integer(-1);\n }\n \n \n //\n // access to maps (limited)\n //\n\n /** Returns whether the given rel (semantic feature, really) is one used to signal coarticulation. */\n public boolean isCoartRel(String rel) {\n return _coartRelsToPreds.containsKey(rel);\n }\n \n \n //\n // classes for caching lex lookups during realization\n //\n\n // a class for caching lookups of signs from rels\n // nb: equality is checked just on the rel, to check for a cached lookup\n private static class RelLookup {\n String rel; Collection signs;\n RelLookup(String s) { rel = s; }\n public int hashCode() { return rel.hashCode(); }\n public boolean equals(Object obj) { \n return (obj instanceof RelLookup) && rel.equals(((RelLookup)obj).rel);\n }\n }\n \n // a class for caching lookups of signs from preds and coart rels\n // nb: equality is checked just on the pred and coart rels, to check for a cached lookup\n private static class PredLookup {\n String pred; List coartRels; Collection signs;\n PredLookup(String s, List l) { pred = s; coartRels = l; }\n public int hashCode() { \n return pred.hashCode() + ((coartRels != null) ? coartRels.hashCode() : 0); \n }\n public boolean equals(Object obj) { \n if (!(obj instanceof PredLookup)) return false;\n PredLookup pLook = (PredLookup) obj;\n if (!pred.equals(pLook.pred)) return false;\n if (coartRels == null) return (pLook.coartRels == null);\n return coartRels.equals(pLook.coartRels);\n }\n }\n\n \n //\n // XML loading routines\n //\n \n\tprivate class MorphScanner extends XmlScanner {\n \tList morphItems = new ArrayList();\n \tList macroItems = new ArrayList();\n \tpublic void handleElement(Element e) {\n // create morph item\n\t\t\tif (e.getName().equals(\"entry\")) {\n try { morphItems.add(new MorphItem(e)); }\n catch (RuntimeException exc) {\n System.err.println(\"Skipping morph item: \" + e.getAttributeValue(\"word\"));\n System.err.println(exc.toString());\n }\n }\n // create macro item\n\t\t\telse if (e.getName().equals(\"macro\")) {\n try { macroItems.add(new MacroItem(e)); }\n catch (RuntimeException exc) {\n System.err.println(\"Skipping macro item: \" + e.getAttributeValue(\"name\"));\n System.err.println(exc.toString());\n }\n }\n\t\t}\n\t};\n\t\n private Pair,List> getMorph(URL url) throws IOException {\n \t// scan XML\n \tMorphScanner morphScanner = new MorphScanner();\n \tmorphScanner.parse(url);\n // return morph and macro items\n return new Pair,List>(morphScanner.morphItems, morphScanner.macroItems);\n }\n \n\tprivate class LexiconScanner extends XmlScanner {\n \tList lexicon = new ArrayList();\n \tElement distrElt = null;\n \tElement licensingElt = null;\n \tElement relationSortingElt = null;\n \tpublic void handleElement(Element e) {\n // create family\n\t\t\tif (e.getName().equals(\"family\")) {\n try { lexicon.add(new Family(e)); }\n catch (RuntimeException exc) {\n System.err.println(\"Skipping family: \" + e.getAttributeValue(\"name\"));\n System.err.println(exc.toString());\n }\n }\n // save distributive attributes\n\t\t\telse if (e.getName().equals(\"distributive-features\")) distrElt = e; \n // save licensing features\n\t\t\telse if (e.getName().equals(\"licensing-features\")) licensingElt = e; \n // save relation sort order\n\t\t\telse if (e.getName().equals(\"relation-sorting\")) relationSortingElt = e; \n\t\t}\n\t};\n\t\n private List getLexicon(URL url) throws IOException {\n \t// scan XML, creating families\n \tLexiconScanner lexiconScanner = new LexiconScanner();\n \tlexiconScanner.parse(url);\n // get distributive attributes, if any\n if (lexiconScanner.distrElt != null) {\n String distrAttrs = lexiconScanner.distrElt.getAttributeValue(\"attrs\");\n _distributiveAttrs = distrAttrs.split(\"\\\\s+\");\n }\n // load licensing features\n loadLicensingFeatures(lexiconScanner.licensingElt);\n // load relation sort order\n loadRelationSortOrder(lexiconScanner.relationSortingElt);\n // return families\n return lexiconScanner.lexicon;\n }\n \n // get licensing features, with appropriate defaults\n @SuppressWarnings(\"unchecked\")\n\tprivate void loadLicensingFeatures(Element licensingElt) {\n List licensingFeats = new ArrayList();\n boolean containsLexFeat = false;\n if (licensingElt != null) {\n for (Iterator it = licensingElt.getChildren(\"feat\").iterator(); it.hasNext(); ) {\n Element featElt = it.next();\n String attr = featElt.getAttributeValue(\"attr\");\n if (attr.equals(\"lex\")) containsLexFeat = true;\n String val = featElt.getAttributeValue(\"val\");\n List alsoLicensedBy = null;\n String alsoVals = featElt.getAttributeValue(\"also-licensed-by\");\n if (alsoVals != null) {\n alsoLicensedBy = Arrays.asList(alsoVals.split(\"\\\\s+\"));\n }\n boolean licenseEmptyCats = true;\n boolean licenseMarkedCats = false;\n boolean instantiate = true; \n byte loc = LicensingFeature.BOTH;\n String lmc = featElt.getAttributeValue(\"license-marked-cats\");\n if (lmc != null) {\n licenseMarkedCats = Boolean.valueOf(lmc).booleanValue();\n // change defaults\n licenseEmptyCats = false;\n loc = LicensingFeature.TARGET_ONLY;\n instantiate = false;\n }\n String lec = featElt.getAttributeValue(\"license-empty-cats\");\n if (lec != null) {\n licenseEmptyCats = Boolean.valueOf(lec).booleanValue();\n }\n String inst = featElt.getAttributeValue(\"instantiate\");\n if (inst != null) {\n instantiate = Boolean.valueOf(inst).booleanValue();\n }\n String locStr = featElt.getAttributeValue(\"location\");\n if (locStr != null) {\n if (locStr.equals(\"target-only\")) loc = LicensingFeature.TARGET_ONLY;\n if (locStr.equals(\"args-only\")) loc = LicensingFeature.ARGS_ONLY;\n if (locStr.equals(\"both\")) loc = LicensingFeature.BOTH;\n }\n licensingFeats.add(\n new LicensingFeature(\n attr, val, alsoLicensedBy, \n licenseEmptyCats, licenseMarkedCats, instantiate, \n loc\n )\n );\n }\n }\n if (!containsLexFeat) {\n licensingFeats.add(LicensingFeature.defaultLexFeature);\n }\n _licensingFeatures = new LicensingFeature[licensingFeats.size()];\n licensingFeats.toArray(_licensingFeatures);\n }\n \n \n // default relation sort order\n private static String[] defaultRelationSortOrder = {\n \"BoundVar\", \"PairedWith\", \n \"Restr\", \"Body\", \"Scope\", \n \"*\", \n \"GenRel\", \"Coord\", \"Append\"\n };\n \n // get relation sort order, or use defaults\n private void loadRelationSortOrder(Element relationSortingElt) {\n // use defaults if no order specified\n if (relationSortingElt == null) {\n for (int i = 0; i < defaultRelationSortOrder.length; i++) {\n _relationIndexMap.put(defaultRelationSortOrder[i], new Integer(i));\n }\n return;\n }\n // otherwise load from 'order' attribute\n String orderAttr = relationSortingElt.getAttributeValue(\"order\");\n String[] relSortOrder = orderAttr.split(\"\\\\s+\");\n for (int i = 0; i < relSortOrder.length; i++) {\n _relationIndexMap.put(relSortOrder[i], new Integer(i));\n }\n }\n\t\t/*\n\t\t * Accessor for words map\n\t\t */\n public GroupMap getWords() {\n\t\t\treturn _words;\n\t\t}\n}\n" }, { "path": "src/opennlp/ccg/lexicon/LicensingFeature.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2003-4 University of Edinburgh (Michael White)\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.lexicon;\r\n\r\nimport java.util.*;\r\n\r\n/**\r\n * A licensing feature is one which controls the licensing and \r\n * instantiation of semantically null or marked categories \r\n * in the realizer. Defaults are handled in Lexicon.loadLicensingFeatures.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.7 $, $Date: 2009/12/21 03:27:18 $\r\n */\r\npublic class LicensingFeature\r\n{\r\n\r\n /**\r\n * The name of the licensing feature.\r\n */\r\n public final String attr;\r\n\r\n /**\r\n * The string value of the licensing feature, or null if any value will do.\r\n */\r\n public final String val;\r\n \r\n /**\r\n * A list of string values of other features that suffice to license \r\n * categories with this feature.\r\n */\r\n public final List alsoLicensedBy;\r\n \r\n /**\r\n * A flag indicating whether semantically null categories with the \r\n * licensing feature need to be licensed.\r\n */\r\n public final boolean licenseEmptyCats;\r\n \r\n /**\r\n * A flag indicating whether initial categories with the \r\n * licensing feature are marked and need to be licensed.\r\n */\r\n public final boolean licenseMarkedCats;\r\n \r\n /**\r\n * A flag indicating whether semantically empty categories with the \r\n * licensing feature should be instantiated.\r\n */\r\n public final boolean instantiate;\r\n \r\n /**\r\n * The location of the licensing feature on the category to be licensed.\r\n * The value must be one of TARGET_ONLY, ARGS_ONLY or BOTH.\r\n */\r\n public final byte loc;\r\n \r\n /**\r\n * Location of the feature on the target category only.\r\n */\r\n public static final byte TARGET_ONLY = 1;\r\n \r\n /**\r\n * Location of the feature on the argument categories only.\r\n */\r\n public static final byte ARGS_ONLY = 2;\r\n \r\n /**\r\n * Location of the feature on either the target category \r\n * or the argument categories.\r\n */\r\n public static final byte BOTH = 0;\r\n \r\n \r\n /** Constructor. */\r\n public LicensingFeature(\r\n String attr, String val, List alsoLicensedBy, \r\n boolean licenseEmptyCats, boolean licenseMarkedCats, boolean instantiate, \r\n byte loc\r\n ) \r\n {\r\n this.attr = attr; this.val = val;\r\n List emptyList = Collections.emptyList();\r\n this.alsoLicensedBy = (alsoLicensedBy != null) ? alsoLicensedBy : emptyList;\r\n this.licenseEmptyCats = licenseEmptyCats; \r\n this.licenseMarkedCats = licenseMarkedCats; \r\n this.instantiate = instantiate; \r\n this.loc = loc;\r\n }\r\n \r\n /** Default lex feature. */\r\n public static final LicensingFeature defaultLexFeature = \r\n new LicensingFeature(\"lex\", null, null, true, false, true, BOTH);\r\n \r\n /** Simple lex feature, for comparison purposes. */\r\n public static final LicensingFeature simpleLexFeature = \r\n new LicensingFeature(\"lex\", null, null, true, false, false, BOTH);\r\n}\r\n \r\n" }, { "path": "src/opennlp/ccg/lexicon/ListPairWord.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2010 Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport java.util.*;\n\nimport opennlp.ccg.util.Pair;\n\n/**\n * A ListPairWord represents a word via a list of pairs of interned \n * attributes and values. It is intended to be a simple wrapper of \n * the list to make it act like a word, without requiring canonical \n * instances. \n *\n * @author Michael White\n * @version $Revision: 1.1 $, $Date: 2010/01/17 04:49:24 $\n */\npublic class ListPairWord extends Word {\n\n\tprivate static final long serialVersionUID = 1L;\n\t\n\t/**\n\t * The list of pairs of attributes and values.\n\t */\n\tprotected List> pairsList;\n\t\n\t/** Constructor. */\n\tpublic ListPairWord(List> pairsList) { this.pairsList = pairsList; }\n\t\n /** Returns the surface form. */\n public String getForm() { return getValFromInterned(Tokenizer.WORD_ATTR); }\n \n /** Returns the pitch accent. */\n public String getPitchAccent() { return getValFromInterned(Tokenizer.PITCH_ACCENT_ATTR); }\n \n /** Returns the list of extra attribute-value pairs. */\n protected List> getAttrValPairsList() { \n List> retval = null; \n\t\tfor (Pair pair : pairsList) {\n if (!isKnownAttr(pair.a)) {\n if (retval == null) retval = new ArrayList>(5);\n retval.add(pair);\n }\n\t\t}\n return retval; \n }\n \n /** Returns the stem. */\n public String getStem() { return getValFromInterned(Tokenizer.STEM_ATTR); }\n \n /** Returns the part of speech. */\n public String getPOS() { return getValFromInterned(Tokenizer.POS_ATTR); }\n \n /** Returns the supertag. */\n public String getSupertag() { return getValFromInterned(Tokenizer.SUPERTAG_ATTR); }\n \n /** Returns the semantic class. */\n public String getSemClass() { return getValFromInterned(Tokenizer.SEM_CLASS_ATTR); }\n\n\n /** Returns the value of the attribute with the given name, or null if none. \n The attribute names Tokenizer.WORD_ATTR, ..., Tokenizer.SEM_CLASS_ATTR \n may be used to retrieve the form, ..., semantic class. */\n\tpublic String getVal(String attr) {\n\t\tString internedAttr = attr.intern(); // use == on interned attr\n\t\treturn getValFromInterned(internedAttr); \n\t}\n\n\t/** Returns the value of the given interned attr, or null if none. */\n\tprotected String getValFromInterned(String attr) {\n\t\tfor (Pair pair : pairsList) {\n\t\t\tif (pair.a == attr) return pair.b;\n\t\t}\n\t return null;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/lexicon/MacroAdder.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.util.*;\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.hylo.*;\n\nimport java.util.*;\n\n/**\n * Adds the features from macros to a category.\n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.6 $, $Date: 2011/03/20 20:11:57 $\n */\npublic class MacroAdder {\n\n private IntHashSetMap _specificMacros;\n private List _macroItems; // for LF macros\n\n public MacroAdder(IntHashSetMap sm, List macroItems) {\n _specificMacros = sm;\n _macroItems = macroItems;\n }\n\n public void addMacros(Category cat) {\n // add features \n cat.deepMap(addIndexedFeatures);\n // append preds to LF\n LF lf = cat.getLF();\n for (int i=0; i < _macroItems.size(); i++) {\n MacroItem mi = _macroItems.get(i);\n LF[] preds = mi.getPreds();\n for (int j=0; j < preds.length; j++) {\n LF pred = (LF) preds[j].copy();\n if (!HyloHelper.isElementaryPredication(pred)) {\n System.out.println(\n \"Warning: ignoring LF macro pred, which is not an elementary predication: \" +\n pred\n );\n continue;\n }\n lf = HyloHelper.append(lf, pred);\n }\n }\n // sort and reset LF\n HyloHelper.sort(lf);\n cat.setLF(lf);\n }\n \n private ModFcn addIndexedFeatures = new ModFcn() {\n @SuppressWarnings(\"rawtypes\")\n\t\tpublic void modify(Mutable c) {\n if (c instanceof AtomCat) {\n FeatureStructure fs = ((AtomCat)c).getFeatureStructure();\n int fsIndex = fs.getIndex();\n Set featStrucs = (Set)_specificMacros.get(fsIndex);\n if (null == featStrucs) {\n return;\n }\n FeatureStructure $fs = fs.copy();\n for (Iterator fsIt = featStrucs.iterator(); fsIt.hasNext();) {\n FeatureStructure macroFS = (FeatureStructure) fsIt.next();\n $fs = $fs.inherit(macroFS);\n }\n ((AtomCat)c).setFeatureStructure($fs);\n }\n }\n };\n}\n" }, { "path": "src/opennlp/ccg/lexicon/MacroItem.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-4 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.hylo.HyloHelper;\n\nimport org.jdom.*;\nimport java.util.*;\n\n/**\n * Data structure for storing information about morphological macros.\n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2009/12/21 03:27:18 $\n */\npublic class MacroItem {\n private String name;\n private FeatureStructure[] featStrucs;\n private LF[] preds;\n \n public MacroItem() {};\n\n @SuppressWarnings(\"unchecked\")\n\tpublic MacroItem (Element e) {\n name = e.getAttributeValue(\"name\");\n if (name == null) { name = e.getAttributeValue(\"n\"); }\n List fsEls = e.getChildren(\"fs\");\n featStrucs = new FeatureStructure[fsEls.size()];\n for (int i=0; i predElts = lfElt.getChildren();\n preds = new LF[predElts.size()];\n for (int i=0; i < predElts.size(); i++) {\n preds[i] = HyloHelper.getLF(predElts.get(i));\n }\n }\n }\n \n public void setName(String s) {\n name=s;\n }\n\n //public void setSpecs(ArrayList al) {specs = al; }\n\n public String getName() {\n return name;\n }\n\n public FeatureStructure[] getFeatureStructures() {\n return featStrucs;\n }\n \n public LF[] getPreds() {\n return preds;\n }\n\n //public void addSpec(String s) { specs.add(s); }\n //public void removeSpec(String s) {\n // specs.remove(specs.indexOf(s));\n //}\n \n}\n" }, { "path": "src/opennlp/ccg/lexicon/MorphItem.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-5 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.lexicon;\n\nimport org.jdom.*;\n\nimport opennlp.ccg.grammar.Grammar;\n\n\n/**\n * A data structure for morphological entries.\n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.11 $, $Date: 2009/10/17 20:46:20 $\n */\npublic class MorphItem {\n\n private static final String[] emptyStringArray = new String[0];\n \n private Word surfaceWord;\n private Word word;\n private Word coartIndexingWord = null;\n \n private String[] macros = emptyStringArray;\n private String[] excluded = emptyStringArray; \n\n private boolean coart = false;\n \n /** Constructor. */\n public MorphItem() {};\n\n /** Constructor from XML element. */\n public MorphItem(Element e) {\n \n String coartString = e.getAttributeValue(\"coart\");\n if (\"true\".equals(coartString)) coart = true;\n \n String wordString = e.getAttributeValue(\"word\");\n boolean strictFactors = coart; // parse with flag for strict factors with coart items\n Word tokenizedWord = Grammar.theGrammar.lexicon.tokenizer.parseToken(wordString, strictFactors);\n surfaceWord = Word.createSurfaceWord(tokenizedWord);\n \n String stem = e.getAttributeValue(\"stem\");\n if (stem == null) stem = surfaceWord.getForm();\n \n String POS = e.getAttributeValue(\"pos\");\n String supertag = null; // supertag comes later from syn cat\n String semClass = e.getAttributeValue(\"class\");\n \n word = Word.createFullWord(surfaceWord, stem, POS, supertag, semClass);\n \n String macrosString = e.getAttributeValue(\"macros\");\n if (macrosString != null) {\n macros = macrosString.split(\"\\\\s+\");\n }\n\n String excludedString = e.getAttributeValue(\"excluded\");\n if (excludedString != null) {\n excluded = excludedString.split(\"\\\\s+\");\n }\n\n // index on first attr of coarts\n if (coart) {\n String indexAttr = wordString.substring(0, wordString.indexOf(\"-\"));\n String indexVal = surfaceWord.getVal(indexAttr);\n coartIndexingWord = Word.createWord(indexAttr, indexVal);\n }\n }\n\n /** Returns whether the name, qualified name or family name of the given entries item is in the excluded list. */\n public boolean excluded(EntriesItem eItem) {\n if (excluded.length == 0) return false;\n for (int i = 0; i < excluded.length; i++) {\n if (eItem.getName().equals(excluded[i])) return true;\n if (eItem.getQualifiedName().equals(excluded[i])) return true;\n if (eItem.getFamilyName().equals(excluded[i])) return true;\n }\n return false;\n }\n \n \n /** Returns the full word. */\n public Word getWord() { return word; }\n\n /** Returns the surface word (without the stem, POS and semantic class). */\n public Word getSurfaceWord() { return surfaceWord; }\n\n /** Returns the macro names. */\n public String[] getMacros() { return macros; }\n \n /** Returns the names of the excluded entries. */\n public String[] getExcluded() { return excluded; }\n\n /** Returns whether the morph item is a coarticulation, eg a pitch accent. */\n public boolean isCoart() { return coart; }\n \n /** Returns the word for indexing this coarticulation (or null if not a coarticulation). */\n public Word getCoartIndexingWord() { return coartIndexingWord; }\n\n \n /** Returns a string for this morph item. */\n // nb: excluded not handled\n public String toString() {\n \tStringBuffer sb = new StringBuffer();\n \tsb.append('[');\n \tfor (int i=0; i < macros.length; i++) {\n \t\tsb.append(macros[i]);\n \t\tif (i < macros.length-1) sb.append(',');\n \t}\n \tsb.append(']');\n return \"{\" + word + \" => \" + sb + \"}\";\n }\n}\n" }, { "path": "src/opennlp/ccg/lexicon/SimpleWord.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport opennlp.ccg.util.*;\nimport java.util.*;\n\n/**\n * A SimpleWord object is a surface word which stores just the word form.\n * SimpleWord serves as the base class for concrete instantiations of words.\n *\n * @author Michael White\n * @version $Revision: 1.4 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class SimpleWord extends Word {\n \n\tprivate static final long serialVersionUID = 181491057498517717L;\n\t\n\t/** The surface form. */\n protected String form;\n \n /** Returns the surface form. */\n public String getForm() { return form; }\n \n /** Returns the pitch accent. */\n public String getPitchAccent() { return null; }\n \n /** Returns the list of extra attribute-value pairs. */\n protected List> getAttrValPairsList() { return null; }\n \n /** Returns the stem. */\n public String getStem() { return null; }\n \n /** Returns the part of speech. */\n public String getPOS() { return null; }\n \n /** Returns the supertag. */\n public String getSupertag() { return null; }\n \n /** Returns the semantic class. */\n public String getSemClass() { return null; }\n\n \n /** Returns the value of the attribute with the given name, or null if none. \n The attribute names Tokenizer.WORD_ATTR, ..., Tokenizer.SEM_CLASS_ATTR \n may be used to retrieve the form, ..., semantic class. */\n public String getVal(String attr) {\n String internedAttr = attr.intern(); // use == on interned attr\n if (internedAttr == Tokenizer.WORD_ATTR) return getForm();\n if (internedAttr == Tokenizer.PITCH_ACCENT_ATTR) return getPitchAccent();\n if (internedAttr == Tokenizer.STEM_ATTR) return getStem();\n if (internedAttr == Tokenizer.POS_ATTR) return getPOS();\n if (internedAttr == Tokenizer.SUPERTAG_ATTR) return getSupertag();\n if (internedAttr == Tokenizer.SEM_CLASS_ATTR) return getSemClass();\n List> pairs = getAttrValPairsList(); \n if (pairs == null) return null;\n for (int i = 0; i < pairs.size(); i++) {\n Pair p = pairs.get(i);\n if (p.a == internedAttr) return p.b;\n }\n return null; // not found\n }\n\n \n /** Constructor. */\n protected SimpleWord(String form) {\n this.form = form; \n }\n}\n\n" }, { "path": "src/opennlp/ccg/lexicon/SupertaggerAdapter.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2008-9 Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport java.util.*;\n\n/**\n * The SupertaggerAdapter interface is for plugging a supertagger into the \n * lexicon in order to return only the desired, high probability categories \n * during lexical lookup. Once the supertagger has been plugged in, \n * using Lexicon.setSupertagger, the supertagger will be consulted during \n * each lexical lookup for the desired categories, using getSupertags. \n * Note that this entails that the supertagger must update its state between lexical \n * lookup calls; in this way, identical words in a sentence can have different \n * predicted categories.\n * \n * The supertagger returns beta-best categories for lexical lookup \n * according to a sequence of beta settings it maintains internally.\n * Associated probabilities for each supertag are also returned.\n * The 'include gold' option controls whether gold standard tags are included \n * during training.\n * \n * At present, the lexicon must contain appropriate morph items for all words. \n * However, the supertags assigned to a word need not be limited to those explicitly \n * listed in the lexicon. When there is an explicit entry, it will be used, \n * as doing so allows the specification of a 'pred' which differs from the stem, \n * as well as macros that can affect the supertag. Otherwise, when using a supertagger, \n * it is no longer necessary to list stems with categories in the lexicon, as the \n * supertagger becomes responsible for this mapping.\n *\n * Note also that at present, only one supertagger (for either parsing or realization) \n * may be plugged in to the lexicon at a time.\n * \n * @author Michael White\n * @version $Revision: 1.11 $, $Date: 2010/12/08 15:24:26 $\n */\npublic interface SupertaggerAdapter {\n\t\n\t/**\n\t * Class for caching a lexical item's log prob in a sign.\n\t */\n\tpublic static class LexLogProb {\n\t\t/** The log prob. */\n\t\tpublic final float logprob;\n\t\t/** Constructor. */\n\t\tpublic LexLogProb(float logprob) { this.logprob = logprob; }\n\t}\n\t\n\t/**\n\t * Returns the supertags of the desired categories for the current lexical lookup \n\t * as a map from supertags to contextual probabilities (or null to accept all). \n\t */\n\tpublic Map getSupertags();\n\t\n\t/**\n\t * Sets the flag for whether to include gold tags.\n\t */\n\tpublic void setIncludeGold(boolean includeGold);\n\t\n\t/**\n\t * Resets beta to the most restrictive value.\n\t */\n\tpublic void resetBeta();\n\n\t/**\n\t * Resets beta to the least restrictive value.\n\t */\n\tpublic void resetBetaToMax();\n\n\t/**\n\t * Advances beta to the next most restrictive setting.\n\t */\n\tpublic void nextBeta();\n\t\n\t/**\n\t * Advances beta to the next less restrictive setting.\n\t */\n\tpublic void previousBeta();\n\t\n\t/**\n\t * Returns whether there are any less restrictive beta settings\n\t * remaining in the sequence.\n\t */\n\tpublic boolean hasMoreBetas();\n\n\t/**\n\t * Returns whether there are any more restrictive beta settings\n\t * remaining in the sequence.\n\t */\n\tpublic boolean hasLessBetas();\n\n\t/** Returns all the beta values. */\n\tpublic double[] getBetas();\n\t\n\t/** Sets the beta values. */\n\tpublic void setBetas(double[] betas);\n\t\n\t/** Returns the current beta value. */\n\tpublic double getCurrentBetaValue();\n}\n" }, { "path": "src/opennlp/ccg/lexicon/Tokenizer.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004-5 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport java.util.*;\n\n/**\n * The Tokenizer interface provides a way to customize tokenization \n * and handling of special tokens.\n * A custom tokenizer may be specified in the grammar file.\n * DefaultTokenizer provides a default implementation, which can also \n * be subclassed for custom behavior.\n *\n * @author Michael White\n * @version $Revision: 1.14 $, $Date: 2005/10/20 17:30:30 $\n **/\n\npublic interface Tokenizer {\n\n /** Name used to tokenize word form attribute (usually left implicit). */\n public static final String WORD_ATTR = \"W\";\n \n /** Name used to tokenize stem attribute. */\n public static final String STEM_ATTR = \"S\";\n \n /** Name used to tokenize part-of-speech attribute. */\n public static final String POS_ATTR = \"P\";\n \n /** Name used to tokenize supertag attribute. */\n public static final String SUPERTAG_ATTR = \"T\";\n \n /** Name used to tokenize semantic class attribute. */\n public static final String SEM_CLASS_ATTR = \"C\";\n \n /** Name used to tokenize pitch accent attribute. */\n public static final String PITCH_ACCENT_ATTR = \"A\";\n \n \n /** Constant used to signal the substitution of a date as the pred. */\n public static final String DATE_VAL = \"[*DATE*]\";\n \n /** Constant used to represent the semantic class date. */\n public static final String DATE_CLASS = \"date\";\n \n /** Constant used to signal the substitution of a time as the pred. */\n public static final String TIME_VAL = \"[*TIME*]\";\n \n /** Constant used to represent the semantic class time. */\n public static final String TIME_CLASS = \"time\";\n \n /** Constant used to signal the substitution of a number as the pred. */\n public static final String NUM_VAL = \"[*NUM*]\";\n \n /** Constant used to represent the semantic class number. */\n public static final String NUM_CLASS = \"num\";\n \n /** Constant used to signal the substitution of an amount as the pred. */\n public static final String AMT_VAL = \"[*AMT*]\";\n \n /** Constant used to represent the semantic class amount. */\n public static final String AMT_CLASS = \"amt\";\n \n /** Constant used to signal the substitution of a duration as the pred. */\n public static final String DUR_VAL = \"[*DUR*]\";\n \n /** Constant used to represent the semantic class duration. */\n public static final String DUR_CLASS = \"dur\";\n \n /** \n * Constant used to signal the substitution of a named entity \n * (not listed in lexicon) as the pred. \n */\n public static final String NE_VAL = \"[*NE*]\";\n \n /** Constant used to represent the semantic class (other) named entity. */\n public static final String NE_CLASS = \"ne\";\n \n\n /**\n * Parses an input string into a list of words, \n * including any explicitly given factors, \n * and the semantic class of special tokens.\n * Tokens are parsed into words using parseToken.\n */\n public List tokenize(String s);\n\n /**\n * Parses an input string into a list of words, \n * including any explicitly given factors, \n * and the semantic class of special tokens.\n * Tokens are parsed into words using parseToken, according to the given \n * flag for whether to parse factors strictly.\n */\n public List tokenize(String s, boolean strictFactors);\n \n /** \n * Parses a token into a word, including any explicitly given factors \n * and the semantic class of special tokens.\n */\n public Word parseToken(String token);\n \n /** \n * Parses a token into a word, including any explicitly given factors \n * and the semantic class of special tokens, according to the given \n * flag for whether to parse factors strictly.\n */\n public Word parseToken(String token, boolean strictFactors);\n \n \n /**\n * Returns a string (eg Tokenizer.DATE_CLASS) indicating the semantic class \n * of special token, if the given token is recognized as a special \n * token; otherwise returns null. \n */\n public String isSpecialToken(String token);\n \n /**\n * Returns the special token constant for the given special token class, \n * or null if none.\n */\n public String getSpecialTokenConstant(String semClass);\n \n /** \n * Returns true iff the given string is a special token constant \n * (eg Tokenizer.DATE_VAL).\n */\n public boolean isSpecialTokenConstant(String s);\n \n \n /** Returns true iff the token is recognized as a date. */\n public boolean isDate(String token);\n \n /** Returns true iff the token is recognized as a time. */\n public boolean isTime(String token);\n \n /** Returns true iff the token is recognized as a number. */\n public boolean isNum(String token);\n \n /** Returns true iff the token is recognized as an amount. */\n public boolean isAmt(String token);\n \n /** \n * Returns true iff the token is recognized as a named entity (not listed in lexicon). \n */\n public boolean isNamedEntity(String token);\n \n \n /**\n * Adds a semantic class to replace words with for language models.\n */\n public void addReplacementSemClass(String semClass);\n \n /** \n * Returns whether the given semantic class is one to replace words with for language models.\n * The sem class is assumed to have been interned.\n */\n public boolean isReplacementSemClass(String semClass);\n\n \n /**\n * Returns a string for the given list of words.\n */\n public String getOrthography(List words);\n \n /**\n * Returns a string for the given list of words, optionally with semantic class replacement.\n */\n public String getOrthography(List words, boolean semClassReplacement);\n \n /**\n * Returns a string for the given word, optionally with semantic class replacement.\n */\n public String getOrthography(Word w, boolean semClassReplacement);\n \n /**\n * Returns a string for the given list of words, \n * in the format expected by the SRILM tool for factored language models.\n */\n public String format(List words);\n \n /**\n * Returns a string for the given word, \n * in the format expected by the SRILM tool for factored language models.\n */\n public String format(Word word);\n \n /**\n * Returns a string for the given list of words, \n * in the format expected by the SRILM tool for factored language models, \n * optionally with semantic class replacement.\n */\n public String format(List words, boolean semClassReplacement);\n \n /**\n * Returns a string for the given word, \n * in the format expected by the SRILM tool for factored language models, \n * optionally with semantic class replacement.\n */\n public String format(Word word, boolean semClassReplacement);\n\n /**\n * Returns one or more orthographic words for the given word.\n * This method is called from within Sign.getWordsInXml as \n * part of producing the textual output of realization.\n */\n public List expandWord(Word word);\n \n /**\n * Returns one or more orthographic words for the given date token.\n */\n public List expandDate(String date);\n \n /**\n * Returns one or more orthographic words for the given time token.\n */\n public List expandTime(String time);\n \n /**\n * Returns one or more orthographic words for the given number token.\n */\n public List expandNum(String num);\n \n /**\n * Returns one or more orthographic words for the given amount token.\n */\n public List expandAmt(String amt);\n \n /**\n * Returns one or more orthographic words for the given named entity token.\n */\n public List expandNamedEntity(String namedEntity);\n}\n\n" }, { "path": "src/opennlp/ccg/lexicon/TrueCaser.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2011 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\n/**\n * A simple class that is constructed with a file containing a list of words\n * that are all and only those found (presumably from a large corpus) to be\n * more frequently upper-cased. Any word-string passed to it (that is also\n * not a named entity string) is queried in this list. If the word is there\n * it is restored to the casing found in the list, if it is not, it is converted\n * to lower-case.\n *\n * There is an option, skipAlreadyLower, to skip any word that is already in lower case, \n * so as to avoid mistakenly uppercasing words. This option is enabled by default. \n * \n * @author Dennis N. Mehay\n *\n */\n\nimport java.util.*;\nimport java.io.*;\nimport java.nio.charset.Charset;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.lexicon.DefaultTokenizer;\nimport java.util.zip.GZIPInputStream;\n\npublic class TrueCaser {\n\n /** Flag for skipping words already in lower case (enabled by default). */\n public boolean skipAlreadyLower = true;\n \n /**\n * A map from lower-cased keys to the true-cased forms (from the list passed in during construction).\n */\n private Map caseMap = new HashMap();\n\n /**\n * For the heuristics that determine whether something is in title case: what percentage of the sentence must be\n * cased to make the title-case detector go off?\n */\n private double titleCaseThreshold;\n\n /**\n * Constructor that just takes a path to a list of words that are only upper-case (title-case is default = 0.9).\n */\n public TrueCaser(String pathToOnlyUCWords) {\n \tthis(pathToOnlyUCWords, 0.5);\n }\n\n /**\n * Constructor that takes a path to a list of words that are only upper-case and a title-case threshold.\n */\n public TrueCaser(String pathToOnlyUCWords, double titleCaseThreshold) {\n\t\tthis.titleCaseThreshold = titleCaseThreshold;\n\t\tthis.caseMap = TrueCaser.readInCaseMap(pathToOnlyUCWords);\n\t\tif (this.caseMap == null) {\n\t\t System.err.println(\"Something went wrong.\"+System.getProperty(\"line.separator\")+\n\t\t\t\t \"Make sure you passed in a file of true-cased words, etc.\");\n\t\t System.exit(-1);\n\t\t} else {\n\t\t //System.err.println(\"Reading in list of true-cased words: \"+pathToOnlyUCWords+System.getProperty(\"line.separator\")+\n\t\t //\t\t \" at a title-case heursitic detection threshold of: \"+titleCaseThreshold);\n\t\t}\n }\n\n /**\n * Static method to read in the list of words.\n */ \n public static Map readInCaseMap(String pathToOnlyUCWords) {\n\t\tMap caseMap = new HashMap();\n\t\tBufferedReader inRead = null;\n\t\ttry {\n\t\t Charset utf8 = Charset.availableCharsets().get(\"UTF-8\");\n\t\n\t\t inRead = new BufferedReader(\n\t\t\t\t\t\t(pathToOnlyUCWords.toLowerCase().endsWith(\".gz\")) ?\n\t\t\t\t\t\t(new InputStreamReader(new GZIPInputStream(new FileInputStream(new File(pathToOnlyUCWords))), utf8)) :\n\t\t\t\t\t\t(new InputStreamReader(new FileInputStream(new File(pathToOnlyUCWords)), utf8))\n\t\t\t\t\t\t);\n\t\n\t\t String ln = inRead.readLine();\n\t\t while(ln != null) {\n\t\t\tln = ln.trim();\n\t\t\tif(!ln.startsWith(\"#\")) {\n\t\t\t caseMap.put(ln.toLowerCase(), ln);\n\t\t\t}\n\t\t\t\tln = inRead.readLine();\n\t\t }\n\t\t} catch (IOException ioe) { \n\t\t return null;\n\t\t} finally {\n\t\t try {\n\t\t \tinRead.close();\n\t\t } catch (Exception e) {\n\t\t \t// do nothing.\n\t\t }\n\t\t}\n\t\treturn caseMap;\n }\n\n /**\n * Truecase a candidate word. If the word is in the list of more commonly\n * cased words, then return this cased form. If not, normalize to lowercase if\n * this is the first word or the sentence is in titlecase. Else return the word as-is.\n * If skipAlreadyLower is enabled, skip the word if it is already in lower case.\n */ \n public String trueCase(String theWord, boolean isTitle, boolean isFirstWord) {\n\t\tString loweredWord = theWord.toLowerCase();\n\t\t// skip word in lower case per flag\n\t\tif (skipAlreadyLower && theWord.equals(loweredWord))\n\t\t\treturn theWord;\n\t\t// look up the truecased version; if not there, and title-case or first\n\t\t// word in sentence, lowercase it, otherwise, don't touch it.\n\t\tString trueCasedTheWord = caseMap.get(loweredWord);\n\t\tif (trueCasedTheWord != null) {\n\t\t\treturn trueCasedTheWord;\n\t\t} else {\n\t\t\treturn (isTitle || isFirstWord) ? loweredWord : theWord;\n\t\t}\n }\n\n /**\n * Truecase a whole sentence. If the sentence appears to be in title-case (as determined by a heuristic\n * that is triggered by there being greater than 0.X of the first 10 words, if there are that many,\n * being cased) normalize any word that is not in the true-case list to lower-case. Otherwise, leave all\n * words that are not in the true-case list alone, except the first word (which is normalized to lower-case)\n * The title-case threshold is a creation-time parameter. \n * As always, if skipAlreadyLower is enabled, the word is skipped if it is already in lower case. Note that \n * the second word is counted as the first word if the first token is a left quote (single or double).\n */\n public String trueCaseSentence(String sentence) { \n\t\tString[] parts = sentence.split(\"\\\\s+\");\n\t\tStringBuffer res = new StringBuffer(parts.length);\n\n\t\tboolean isTitle = isTitleCased(parts);\n\t\t// truecase the whole sentence (only normalizing by lowercasing if\n\t\t// titlecase detector went off).\n\t\tint i = 0;\n\t\tfor (String prt : parts) {\n\t\t\tboolean isFirstWord = (i == 0 || (i == 1 && (parts[0].equals(\"``\") || parts[0].equals(\"`\"))));\n\t\t\tres.append(\" \" + trueCase(prt, isTitle, isFirstWord));\n\t\t\ti++;\n\t\t}\n\t\treturn res.toString().trim();\n }\n\n /**\n * Returns true iff the percentage of the first 10 words (or the whole sentence if it's less than 10 words) \n * that have a case distinction is greater than or equal to 'titleCaseThreshold'.\n */\n public boolean isTitleCased(String[] words) {\n\t\tint numCased = 0, cursor = 0;\n\t\tfor (String wd : words) {\n\t\t\tif (cursor >= 10) {\n\t\t\t\tbreak;\n\t\t\t}\n\t\t\tif (!wd.toLowerCase().equals(wd)) {\n\t\t\t\tnumCased++;\n\t\t\t}\n\t\t\tcursor++;\n\t\t}\n\t\treturn (numCased / ((words.length < 10) ? (words.length + 0.0) : 10.0) >= titleCaseThreshold);\n }\n\n public String tcWordToString(String newWordForm, Word oldWord) {\n StringBuffer sb = new StringBuffer();\n sb.append(newWordForm);\n if (oldWord.getPOS() != null) sb.append(\":P-\").append(DefaultTokenizer.escape(oldWord.getPOS()));\n if (oldWord.getSupertag() != null) sb.append(\":T-\").append(DefaultTokenizer.escape(oldWord.getSupertag()));\n if (oldWord.getSemClass() != null) sb.append(\":C-\").append(DefaultTokenizer.escape(oldWord.getSemClass()));\n if (sb.length() == 0) sb.append((String) null);\n return sb.toString();\n }\n\n public static void main(String[] args) throws IOException {\n \tString newline = System.getProperty(\"line.separator\");\n \tString usage = \n \t\t\tnewline + \n \t\t\t\"java TrueCaser -t -r -i -o \" \n \t\t\t+ newline;\n \tif(args.length == 0 || args[0] .equals(\"h\") || args[0] .equals(\"-h\") || args[0] .equals(\"--h\") \n \t\t\t|| args[0] .equals(\"--help\") || args[0] .equals(\"-help\")) {\n\t\t System.err.println(usage);\n\t\t System.exit(0);\n\t\t}\n\n\t\tString truecasefile = null, inputfile = null, outputfile = null;\n\t\tdouble tcThresh = 0.5;\n\t\tfor (int a = 0; a < args.length; a++) {\n\t\t\tif (args[a].equals(\"-t\")) {\n\t\t\t\ttruecasefile = args[++a];\n\t\t\t\tcontinue;\n\t\t\t}\n\t\t\tif (args[a].equals(\"-r\")) {\n\t\t\t\ttcThresh = Double.parseDouble(args[++a]);\n\t\t\t\tcontinue;\n\t\t\t}\n\t\t\tif (args[a].equals(\"-i\")) {\n\t\t\t\tinputfile = args[++a];\n\t\t\t\tcontinue;\n\t\t\t}\n\t\t\tif (args[a].equals(\"-o\")) {\n\t\t\t\toutputfile = args[++a];\n\t\t\t\tcontinue;\n\t\t\t}\n\n\t\t\tSystem.err.println(\"unrecognized option \" + args[a] + \".\");\n\t\t\tSystem.err.println(usage);\n\t\t\tSystem.exit(0);\n\t\t} \n\t\t\n\t\tCharset utf8 = Charset.availableCharsets().get(\"UTF-8\");\n\n\t\t// input of text (assumed to be tokenized utf-8-encoded text).\n\t\tBufferedReader in = new BufferedReader(new InputStreamReader(\n\t\t\t\t((inputfile == null) ? System.in : (new FileInputStream(\n\t\t\t\t\t\tnew File(inputfile)))), utf8));\n\n\t\t// output stream (back to tokenized utf-8-encoded text).\n\t\tBufferedWriter out = new BufferedWriter(new OutputStreamWriter(\n\t\t\t\t((outputfile == null) ? System.out : (new FileOutputStream(\n\t\t\t\t\t\tnew File(outputfile)))), utf8));\n\n\t\t// file of true-cased words is arg0.\n\t\tTrueCaser tc = new TrueCaser(truecasefile, tcThresh);\n\n\t\tString sent = in.readLine();\n\t\twhile (sent != null) {\n\t\t\tsent = tc.trueCaseSentence(sent.trim());\n\t\t\tout.write(sent + System.getProperty(\"line.separator\"));\n\t\t\tsent = in.readLine();\n\t\t}\n\t\tout.close();\n\t\tin.close();\n\t}\n}" }, { "path": "src/opennlp/ccg/lexicon/Word.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004-5 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\nimport opennlp.ccg.util.*;\n\nimport java.io.*;\nimport java.util.*;\nimport gnu.trove.*;\n\n/**\n * A Word object may either be a surface word or a full word.\n * A surface word holds a surface form, an optional pitch accent, and an \n * optional list of extra attribute-value pairs.\n * A full word additionally contains a stem, part of speech, supertag and semantic class.\n * A word may be a multiword consisting of multiple orthographic words, \n * typically separated by underscores in the surface form.\n * For efficient storage and equality checking, Word objects are interned by \n * the factory methods of the configured WordFactory.\n *\n * @author Michael White\n * @version $Revision: 1.21 $, $Date: 2009/12/16 22:39:20 $\n */\nabstract public class Word implements Serializable, Comparable {\n\n\tprivate static final long serialVersionUID = 1L;\n\n\t/** Returns the surface form. */\n abstract public String getForm();\n \n /** Returns the pitch accent. */\n abstract public String getPitchAccent();\n \n // empty iterator\n private static Iterator> emptyIterator = new ArrayList>(0).iterator();\n\n /** Returns an iterator over the extra attribute-value pairs. */\n public Iterator> getAttrValPairs() { \n List> pairs = getAttrValPairsList();\n return (pairs != null) ? pairs.iterator() : emptyIterator; \n } \n \n /** Returns an iterator over the surface attribute-value pairs, including the pitch accent (if any). */\n public Iterator> getSurfaceAttrValPairs() {\n List> pairs = getAttrValPairsList(); String pitchAccent = getPitchAccent();\n if (pairs == null && pitchAccent == null) return emptyIterator; \n else if (pairs == null) { \n List> retval = new ArrayList>(1); \n retval.add(new Pair(Tokenizer.PITCH_ACCENT_ATTR, pitchAccent));\n return retval.iterator();\n }\n else if (pitchAccent == null) return pairs.iterator();\n else {\n List> retval = new ArrayList>(pairs);\n retval.add(new Pair(Tokenizer.PITCH_ACCENT_ATTR, pitchAccent));\n return retval.iterator();\n }\n } \n \n /** Returns the list of extra attribute-value pairs. */\n abstract protected List> getAttrValPairsList();\n \n /** Returns the stem. */\n abstract public String getStem();\n \n /** Returns the part of speech. */\n abstract public String getPOS();\n \n /** Returns the supertag. */\n abstract public String getSupertag();\n \n /** Returns the semantic class. */\n abstract public String getSemClass();\n\n \n /** Returns the value of the attribute with the given name, or null if none. \n The attribute names Tokenizer.WORD_ATTR, ..., Tokenizer.SEM_CLASS_ATTR \n may be used to retrieve the form, ..., semantic class. */\n abstract public String getVal(String attr);\n\n \n // the known attr names\n private static Set knownAttrs = initKnownAttrs(); \n @SuppressWarnings(\"unchecked\")\n\tprivate static Set initKnownAttrs() {\n Set knownAttrs = new THashSet(new TObjectIdentityHashingStrategy());\n String[] names = {\n Tokenizer.WORD_ATTR, Tokenizer.PITCH_ACCENT_ATTR, \n Tokenizer.STEM_ATTR, Tokenizer.POS_ATTR, \n Tokenizer.SUPERTAG_ATTR, Tokenizer.SEM_CLASS_ATTR\n };\n for (int i = 0; i < names.length; i++) { knownAttrs.add(names[i]); }\n return knownAttrs;\n }\n \n /** Returns whether the given attr is a known one (vs an extra one). */\n public static boolean isKnownAttr(String attr) {\n return knownAttrs.contains(attr.intern());\n }\n \n \n /** Returns true if the form is non-null, while the stem, part of speech, supertag and semantic class are null. */\n public boolean isSurfaceWord() {\n return getForm() != null && getStem() == null && getPOS() == null && getSupertag() == null && getSemClass() == null;\n }\n \n \n // factory methods\n \n /** Factory interface. */\n public interface WordFactory {\n /** Creates a surface word with the given interned form. */\n public Word create(String form);\n /** Creates a (surface or full) word with the given normalized attribute name and value.\n The attribute names Tokenizer.WORD_ATTR, ..., Tokenizer.SEM_CLASS_ATTR \n may be used for the form, ..., semantic class. */\n public Word create(String attr, String val);\n /** Creates a (surface or full) word from the given canonical factors. */\n public Word create(\n String form, String pitchAccent, List> attrValPairs, \n String stem, String POS, String supertag, String semClass \n );\n }\n \n /** The word factory to use. */\n protected static WordFactory wordFactory = new FullWord.Factory();\n\n // NB: could try different factory methods for concrete words, but \n // it's unclear whether it makes much difference\n // protected static WordFactory wordFactory = new FactorChainWord.Factory();\n \n /** Creates a surface word with the given form. */\n public static synchronized Word createWord(String form) { \n form = (form != null) ? form.intern() : null; \n return wordFactory.create(form);\n }\n \n /** Creates a (surface or full) word. */\n public static synchronized Word createWord(\n String form, String pitchAccent, List> attrValPairs, \n String stem, String POS, String supertag, String semClass \n ) {\n // normalize factors\n form = (form != null) ? form.intern() : null; \n pitchAccent = (pitchAccent != null) ? pitchAccent.intern() : null;\n if (attrValPairs != null) {\n if (attrValPairs.isEmpty()) attrValPairs = null;\n else {\n attrValPairs = new ArrayList>(attrValPairs);\n sortAttrValPairs(attrValPairs);\n for (int i = 0; i < attrValPairs.size(); i++) {\n Pair p = attrValPairs.get(i);\n String attr = p.a.intern();\n String val = (p.b != null) ? p.b.intern() : null;\n attrValPairs.set(i, new Pair(attr, val));\n }\n }\n }\n stem = (stem != null) ? stem.intern() : null; \n POS = (POS != null) ? POS.intern() : null;\n supertag = (supertag != null) ? supertag.intern() : null;\n semClass = (semClass != null) ? semClass.intern() : null; \n // create word\n return createWordDirectly(form, pitchAccent, attrValPairs, stem, POS, supertag, semClass);\n }\n\n // comparator for attr-val pairs\n private static Comparator> attrValComparator = new Comparator>() {\n public int compare(Pair p1, Pair p2) {\n return p1.a.compareTo(p2.a);\n }\n };\n \n /** Sorts attr-val pairs by attr name. */\n private static void sortAttrValPairs(List> pairs) {\n Collections.sort(pairs, attrValComparator);\n }\n \n /** Creates a (surface or full) word directly, from the given canonical factors. */\n private static synchronized Word createWordDirectly(\n String form, String pitchAccent, List> attrValPairs, \n String stem, String POS, String supertag, String semClass \n ) {\n return wordFactory.create(form, pitchAccent, attrValPairs, stem, POS, supertag, semClass);\n }\n \n /** Creates a (surface or full) word with the given attribute name and value.\n The attribute names Tokenizer.WORD_ATTR, ..., Tokenizer.SEM_CLASS_ATTR \n may be used for the form, ..., semantic class. */\n public static synchronized Word createWord(String attr, String val) {\n attr = attr.intern(); val = (val != null) ? val.intern() : null; \n return wordFactory.create(attr, val);\n }\n \n /** Creates a (surface or full) word from the given one, replacing the word form with the given one. */\n public static synchronized Word createWord(Word word, String form) {\n if (form != null) form = form.intern();\n return createWordDirectly(\n form, word.getPitchAccent(), word.getAttrValPairsList(), \n word.getStem(), word.getPOS(), word.getSupertag(), word.getSemClass()\n );\n }\n \n /** Creates a (surface or full) word from the given one, \n replacing the form and stem with the semantic class, uppercased. */\n public static synchronized Word createWordUsingSemClass(Word word) {\n String form = word.getSemClass().toUpperCase().intern();\n String stem = form;\n return createWordDirectly(\n form, word.getPitchAccent(), word.getAttrValPairsList(), \n stem, word.getPOS(), word.getSupertag(), word.getSemClass()\n );\n }\n \n /** Creates a (surface or full) word from the given surface one, adding the \n second word's additional attr-val pairs. */\n public static synchronized Word createWordWithAttrs(Word word, Word word2) {\n // get accent\n String accent = word.getPitchAccent();\n if (accent == null) accent = word2.getPitchAccent();\n // get attrs\n boolean mixedAttrs = false;\n List> pairs = word.getAttrValPairsList();\n List> pairs2 = word2.getAttrValPairsList(); \n if (pairs == null && pairs2 != null) { pairs = pairs2; }\n else if (pairs2 != null) {\n mixedAttrs = true;\n pairs = new ArrayList>(pairs); \n for (int i = 0; i < pairs2.size(); i++) {\n if (!pairs.contains(pairs2.get(i))) {\n pairs.add(pairs2.get(i)); \n }\n }\n }\n // get rest\n String form = word.getForm(); String stem = word.getStem(); \n String POS = word.getPOS(); String supertag = word.getSupertag(); String semClass = word.getSemClass(); \n // with mixed attrs, need to normalize\n if (mixedAttrs) \n return createWord(form, accent, pairs, stem, POS, supertag, semClass);\n else \n return createWordDirectly(form, accent, pairs, stem, POS, supertag, semClass);\n }\n \n \n /** Creates a full word from the given surface one, adding the given stem, POS and semantic class. */\n public static synchronized Word createFullWord(Word word, String stem, String POS, String supertag, String semClass) {\n stem = (stem != null) ? stem.intern() : null; \n POS = (POS != null) ? POS.intern() : null;\n supertag = (supertag != null) ? supertag.intern() : null;\n semClass = (semClass != null) ? semClass.intern() : null; \n return createWordDirectly(word.getForm(), word.getPitchAccent(), word.getAttrValPairsList(), stem, POS, supertag, semClass);\n }\n \n /** Creates a full word from the given surface one, \n adding the second (full) given word's stem, POS and semantic class, \n as well as the second word's additional attr-val pairs, \n plus the given supertag. */\n public static synchronized Word createFullWord(Word word, Word word2, String supertag) {\n boolean mixedAttrs = false;\n List> pairs = word.getAttrValPairsList(); \n List> pairs2 = word2.getAttrValPairsList(); \n if (pairs == null && pairs2 != null) { pairs = pairs2; }\n else if (pairs2 != null) {\n mixedAttrs = true;\n pairs = new ArrayList>(pairs); \n for (int i = 0; i < pairs2.size(); i++) {\n if (!pairs.contains(pairs2.get(i))) {\n pairs.add(pairs2.get(i)); \n }\n }\n }\n if (mixedAttrs) { \n return createWord(\n word.getForm(), word.getPitchAccent(), pairs, \n word2.getStem(), word2.getPOS(), supertag, word2.getSemClass()\n );\n }\n else {\n supertag = (supertag != null) ? supertag.intern() : null;\n return createWordDirectly(\n word.getForm(), word.getPitchAccent(), pairs, \n word2.getStem(), word2.getPOS(), supertag, word2.getSemClass()\n );\n }\n }\n \n\n /** Creates a surface word from the given one, removing the stem, POS, supertag and semantic class. */\n public static synchronized Word createSurfaceWord(Word word) {\n return createWordDirectly(word.getForm(), word.getPitchAccent(), word.getAttrValPairsList(), null, null, null, null);\n }\n \n /** Creates a surface word from the given one, removing the stem, POS, supertag and semantic class, \n and replacing the form with the given one. */\n public static synchronized Word createSurfaceWord(Word word, String form) {\n form = (form != null) ? form.intern() : null; \n return createWordDirectly(form, word.getPitchAccent(), word.getAttrValPairsList(), null, null, null, null);\n }\n \n /** Creates a surface word from the given one, removing the stem, POS, supertag and semantic class, \n and replacing the form with the semantic class, uppercased. */\n public static synchronized Word createSurfaceWordUsingSemClass(Word word) {\n String form = word.getSemClass().toUpperCase().intern();\n return createWordDirectly(form, word.getPitchAccent(), word.getAttrValPairsList(), null, null, null, null);\n }\n\n \n /** Creates a core surface word from the given one, removing all attrs in the given set. */\n public static synchronized Word createCoreSurfaceWord(Word word, Set attrsSet) {\n String form = word.getForm();\n String accent = word.getPitchAccent();\n if (accent != null && attrsSet.contains(Tokenizer.PITCH_ACCENT_ATTR)) accent = null;\n List> pairs = word.getAttrValPairsList(); \n if (pairs != null) {\n pairs = new ArrayList>(pairs); \n Iterator> pairsIt = pairs.iterator(); \n while (pairsIt.hasNext()) {\n Pair pair = pairsIt.next(); \n if (attrsSet.contains(pair.a)) { pairsIt.remove(); }\n }\n return createWord(form, accent, pairs, null, null, null, null);\n }\n else {\n return createWordDirectly(form, accent, null, null, null, null, null);\n }\n }\n \n\n /** Returns a hash code for this word. */\n public int hashCode() {\n int hc = System.identityHashCode(getForm());\n hc = 31*hc + System.identityHashCode(getPitchAccent());\n for (Iterator> it = getAttrValPairs(); it.hasNext(); ) {\n Pair p = it.next();\n hc = 31*hc + System.identityHashCode(p.a);\n hc = 31*hc + System.identityHashCode(p.b);\n }\n hc = 31*hc + System.identityHashCode(getStem()); \n hc = 31*hc + System.identityHashCode(getPOS()); \n hc = 31*hc + System.identityHashCode(getSupertag()); \n hc = 31*hc + System.identityHashCode(getSemClass()); \n return hc;\n }\n \n /** Returns whether this word equals the given object. */\n public boolean equals(Object obj) {\n if (this == obj) return true;\n // nb: can use ==, since constructor interns all factors\n if (!(obj instanceof Word)) return false;\n Word word = (Word) obj;\n boolean sameFields =\n getForm() == word.getForm() && \n getPitchAccent() == word.getPitchAccent() &&\n getStem() == word.getStem() && \n getPOS() == word.getPOS() && \n getSupertag() == word.getSupertag() && \n getSemClass() == word.getSemClass();\n if (!sameFields) return false;\n List> pairs = getAttrValPairsList();\n List> wordPairs = word.getAttrValPairsList();\n if (pairs == null && wordPairs == null) return true;\n if (pairs == null || wordPairs == null) return false;\n if (pairs.size() != wordPairs.size()) return false;\n for (int i = 0; i < pairs.size(); i++) {\n if (!pairs.get(i).equals(wordPairs.get(i))) return false;\n }\n return true;\n }\n \n /** Returns an int representing lexicographic sorting. */\n public int compareTo(Word word) {\n \tif (this == word) return 0;\n \tint cmp = 0;\n \tcmp = compare(getForm(), word.getForm());\n \tif (cmp != 0) return cmp;\n \tcmp = compare(getPitchAccent(), word.getPitchAccent());\n \tif (cmp != 0) return cmp;\n \tcmp = compare(getStem(), word.getStem());\n \tif (cmp != 0) return cmp;\n \tcmp = compare(getPOS(), word.getPOS());\n \tif (cmp != 0) return cmp;\n \tcmp = compare(getSupertag(), word.getSupertag());\n \tif (cmp != 0) return cmp;\n \tcmp = compare(getSemClass(), word.getSemClass());\n \tif (cmp != 0) return cmp;\n List> pairs = getAttrValPairsList();\n List> wordPairs = word.getAttrValPairsList();\n if (pairs == null && wordPairs == null) return 0;\n if (pairs == null) return -1;\n if (wordPairs == null) return 1;\n if (pairs.size() < wordPairs.size()) return -1;\n if (pairs.size() > wordPairs.size()) return 1;\n for (int i = 0; i < pairs.size(); i++) {\n \tPair p = pairs.get(i);\n \tPair wp = wordPairs.get(i);\n \tcmp = p.a.compareTo(wp.a);\n \tif (cmp != 0) return cmp;\n \tcmp = p.b.compareTo(wp.b);\n \tif (cmp != 0) return cmp;\n }\n \treturn 0;\n }\n \n // compares strings, accounting for nulls\n private int compare(String s1, String s2) {\n \tif (s1 == null && s2 == null) return 0;\n \tif (s1 == null) return -1;\n \tif (s2 == null) return 1;\n \treturn s1.compareTo(s2);\n }\n \n /** Returns whether this word's surface attributes intersect with the given ones. */\n public boolean attrsIntersect(Set attrsSet) {\n if (getPitchAccent() != null && attrsSet.contains(Tokenizer.PITCH_ACCENT_ATTR))\n return true;\n for (Iterator> it = getAttrValPairs(); it.hasNext(); ) {\n Pair p = it.next();\n if (attrsSet.contains(p.a)) return true;\n }\n return false;\n }\n \n /** Returns a hash code for this word's restriction to a surface word. */\n public int surfaceWordHashCode() {\n int hc = System.identityHashCode(getForm());\n hc = 31*hc + System.identityHashCode(getPitchAccent());\n for (Iterator> it = getAttrValPairs(); it.hasNext(); ) {\n Pair p = it.next();\n hc = 31*hc + System.identityHashCode(p.a);\n hc = 31*hc + System.identityHashCode(p.b);\n }\n return hc;\n }\n \n /** Returns whether this word and the given object have equal restrictions to surface words. */\n public boolean surfaceWordEquals(Object obj) {\n if (this == obj) return true;\n // nb: can use ==, since constructor interns all factors\n if (!(obj instanceof Word)) return false;\n Word word = (Word) obj;\n boolean sameFields =\n getForm() == word.getForm() && \n getPitchAccent() == word.getPitchAccent();\n if (!sameFields) return false;\n List> pairs = getAttrValPairsList();\n List> wordPairs = word.getAttrValPairsList();\n if (pairs == null && wordPairs == null) return true;\n if (pairs == null || wordPairs == null) return false;\n if (pairs.size() != wordPairs.size()) return false;\n for (int i = 0; i < pairs.size(); i++) {\n if (!pairs.get(i).equals(wordPairs.get(i))) return false;\n }\n return true;\n }\n \n /** Returns canonical version of deserialized word. */\n public Object readResolve() throws ObjectStreamException {\n \treturn createWord(getForm(), getPitchAccent(), getAttrValPairsList(), getStem(), getPOS(), getSupertag(), getSemClass());\n }\n \n /** Shows non-trivial fields separated by underscores. */\n public String toString() {\n StringBuffer sb = new StringBuffer();\n if (getForm() != null) sb.append(getForm());\n if (getPitchAccent() != null) sb.append('_').append(getPitchAccent());\n for (Iterator> it = getAttrValPairs(); it.hasNext(); ) {\n Pair p = it.next();\n sb.append('_').append(p.b);\n }\n if (getStem() != null && getStem() != getForm()) sb.append('_').append(getStem());\n if (getPOS() != null) sb.append('_').append(getPOS());\n if (getSupertag() != null) sb.append('_').append(getSupertag());\n if (getSemClass() != null) sb.append('_').append(getSemClass());\n if (sb.length() == 0) sb.append((String)null);\n return sb.toString();\n }\n \n /** Tests serialization. */\n public static void main(String[] argv) throws IOException, ClassNotFoundException {\n \t// create words\n \tWord w = createWord(\"ran\");\n \tWord fw = createFullWord(w, \"run\", \"VBD\", \"s\\\\np\", \"MOTION\");\n \tWord wb = createWordWithAttrs(w, createWord(\"B\", \"L\"));\n \t// write to tmp.out\n \tString filename = \"tmp.ser\";\n \tObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(filename));\n \tSystem.out.println(\"Writing w: \" + w);\n \tout.writeObject(w);\n \tSystem.out.println(\"Writing fw: \" + fw);\n \tout.writeObject(fw);\n \tSystem.out.println(\"Writing wb: \" + wb);\n \tout.writeObject(wb);\n \tout.close();\n \t// read from tmp.out\n \tObjectInputStream in = new ObjectInputStream(new FileInputStream(filename));\n \tSystem.out.print(\"Reading w2: \");\n \tWord w2 = (Word) in.readObject();\n \tSystem.out.println(w2);\n \tSystem.out.print(\"Reading fw2: \");\n \tWord fw2 = (Word) in.readObject();\n \tSystem.out.println(fw2);\n \tSystem.out.print(\"Reading wb2: \");\n \tWord wb2 = (Word) in.readObject();\n \tSystem.out.println(wb2);\n \tin.close();\n \t// test identity (and thus readResolve)\n \tSystem.out.println(\"w == w2?: \" + (w == w2));\n \tSystem.out.println(\"fw == fw2?: \" + (fw == fw2));\n \tSystem.out.println(\"wb == wb2?: \" + (wb == wb2));\n }\n}\n\n" }, { "path": "src/opennlp/ccg/lexicon/WordWithPitchAccent.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.lexicon;\n\n/**\n * A WordWithPitchAccent object is a surface word with an optional pitch accent \n * but no further attributes.\n *\n * @author Michael White\n * @version $Revision: 1.4 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class WordWithPitchAccent extends SimpleWord {\n \n\tprivate static final long serialVersionUID = 1510997962756436949L;\n\t\n\t/** The pitch accent. */\n protected String pitchAccent;\n \n /** Returns the pitch accent. */\n public String getPitchAccent() { return pitchAccent; }\n \n /** Constructor. */\n protected WordWithPitchAccent(String form, String pitchAccent) {\n super(form); \n this.pitchAccent = pitchAccent;\n }\n}\n\n" }, { "path": "src/opennlp/ccg/ngrams/AAnFilter.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.ngrams;\n\nimport opennlp.ccg.lexicon.*;\nimport opennlp.ccg.util.*;\n\nimport java.util.*;\nimport java.io.*;\n\nimport gnu.trove.*;\n\n/**\n * AN n-gram filter that detects \"a\" followed by a word beginning with a vowel, \n * or \"an\" followed by a word beginning with a consonant.\n * Note that this filter only makes an approximate check, which may be augmented \n * with a set of exceptions. \n * Exceptions may be culled from a file of bigrams using cullAAnExceptions, \n * which may be accessed from the command line using the -c option.\n * An appropriate bigrams file can be produced using the SRILM ngram-count tool, \n * with the -text and -write2 options.\n *\n * @author Michael White\n * @version $Revision: 1.8 $, $Date: 2011/03/20 20:11:58 $\n */\npublic class AAnFilter implements NgramFilter, Reversible\n{\n // exceptions\n private Set> exceptions = null;\n \n \n /** Constructor. */\n public AAnFilter() {}\n \n /** Constructor that loads a/an exceptions from the given infile of bigrams. */\n public AAnFilter(String infile) throws IOException { \n loadAAnExceptions(infile); \n }\n \n\n /** Flag for whether to reverse words before filtering. */\n protected boolean reverse = false;\n \n /** Get reverse flag. */\n public boolean getReverse() { return reverse; }\n \n /** Set reverse flag. */\n public void setReverse(boolean reverse) { this.reverse = reverse; }\n \n\n /** Returns whether to filter out the given word sequence. */\n public boolean filterOut(List words) {\n for (int i = 0; i < words.size()-1; i++) {\n Word w1 = words.get(i);\n Word w2 = words.get(i+1);\n if (filterOut(w1, w2)) return true;\n }\n return false; \n }\n \n /** Returns whether to filter out the given word bigram (reversed if apropos). */\n public boolean filterOut(Word w1, Word w2) {\n if (reverse) {\n Word tmp = w1; w1 = w2; w2 = tmp;\n }\n String f1 = w1.getForm(); \n if (f1 != \"a\" && f1 != \"an\") return false;\n String f1Alt = (f1 == \"a\") ? \"an\" : \"a\";\n String f2 = w2.getForm();\n boolean defaultRetval = filterOutByDefault(f1, f2);\n if (isException(f1, f2) || isException(f1Alt, f2))\n return !defaultRetval;\n else\n return defaultRetval;\n }\n \n // returns whether to filter out the bigram by default\n private static boolean filterOutByDefault(String w1, String w2) {\n boolean w2StartsWithVowel = startsWithVowel(w2);\n return (w1 == \"a\" && w2StartsWithVowel) || (w1 == \"an\" && !w2StartsWithVowel);\n }\n \n // initial vowel\n private static boolean startsWithVowel(String word) {\n char c = word.charAt(0);\n return (c == 'a' || c == 'e' || c == 'i' || c == 'o' || c == 'u') ||\n (c == 'A' || c == 'E' || c == 'I' || c == 'O' || c == 'U');\n }\n \n // words for a/an\n private static final Word A_WORD = Word.createWord(\"a\");\n private static final Word AN_WORD = Word.createWord(\"an\");\n \n // reusable list for lookup\n private List keyList = new ArrayListWithIdentityEquals(2);\n \n // looks up whether the bigram is an exception\n private boolean isException(String w1, String w2) {\n if (exceptions == null) return false;\n keyList.clear();\n keyList.add((w1 == \"a\") ? A_WORD : AN_WORD);\n keyList.add(Word.createWord(w2));\n return exceptions.contains(keyList);\n }\n \n // singletons for a/an\n @SuppressWarnings(\"unchecked\")\n\tprivate static final List A_SINGLETON = (List) Interner.globalIntern(new SingletonList(A_WORD));\n @SuppressWarnings(\"unchecked\")\n private static final List AN_SINGLETON = (List) Interner.globalIntern(new SingletonList(AN_WORD));\n \n /** Adds an a/an bigram as an exception. */\n @SuppressWarnings(\"unchecked\")\n public void addException(String w1, String w2) {\n // make sure w1 is a/an\n w1 = w1.intern();\n if (w1 != \"a\" && w1 != \"an\") {\n System.err.println(\"Warning: ignoring exception not starting with a/an: \" + w1 + \" \" + w2);\n return;\n }\n // ensure exceptions initialized\n if (exceptions == null) exceptions = new THashSet();\n // intern and add bigram\n List w1Singleton = (w1 == \"a\") ? A_SINGLETON : AN_SINGLETON;\n List w2Singleton = (List) Interner.globalIntern(new SingletonList(Word.createWord(w2)));\n List excBigram = (List) Interner.globalIntern(new StructureSharingList(w1Singleton, w2Singleton));\n exceptions.add(excBigram);\n }\n\n \n /** Culls a/an exceptions from the given infile of bigrams, writing them to the given outfile. */\n public static void cullAAnExceptions(String infile, String outfile) throws IOException {\n Reader in = new BufferedReader(new FileReader(infile));\n StreamTokenizer tokenizer = NgramScorer.initTokenizer(in);\n PrintWriter out = new PrintWriter(new FileWriter(outfile));\n String[] tokens = new String[2];\n // loop through lines\n while (tokenizer.ttype != StreamTokenizer.TT_EOF) {\n // read line into tokens\n NgramScorer.readLine(tokenizer, tokens);\n // check for blank/incomplete line\n if (tokens[1] == null) continue;\n // check for a/an exception\n if (tokens[0].equals(\"a\") || tokens[0].equals(\"an\")) {\n String aan = tokens[0].intern();\n String word = tokens[1];\n if (filterOutByDefault(aan, word)) {\n // write to exceptions file\n out.println(aan + \" \" + word);\n }\n }\n }\n // done\n in.close();\n out.flush(); out.close();\n }\n \n\n /** Loads a/an exceptions from the given infile of bigrams. */\n public void loadAAnExceptions(String infile) throws IOException {\n Reader in = new BufferedReader(new FileReader(infile));\n StreamTokenizer tokenizer = NgramScorer.initTokenizer(in);\n String[] tokens = new String[2];\n // loop through lines\n while (tokenizer.ttype != StreamTokenizer.TT_EOF) {\n // read line into tokens\n NgramScorer.readLine(tokenizer, tokens);\n // check for blank/incomplete line\n if (tokens[1] == null) continue;\n // add a/an exception\n addException(tokens[0], tokens[1]);\n }\n // done\n in.close();\n }\n\n \n /** Test loading and filtering, or cull exceptions from bigrams. */\n public static void main(String[] args) throws IOException {\n \n String usage = \"Usage: java opennlp.ccg.ngrams.AAnFilter () | -c \";\n \n if (args.length > 0 && args[0].equals(\"-h\")) {\n System.out.println(usage);\n System.exit(0);\n }\n \n // cull exceptions with -c\n if (args[0].equals(\"-c\")) {\n String infile = args[1]; String outfile = args[2];\n System.out.println(\"Culling a/an exceptions from \" + infile + \" to \" + outfile);\n cullAAnExceptions(infile, outfile);\n System.exit(0);\n }\n \n // otherwise optionally load exceptions, ... \n AAnFilter aanFilter = new AAnFilter();\n String infile = null; String tokens = null;\n if (args.length >= 2) { infile = args[0]; tokens = args[1]; }\n else tokens = args[0];\n if (infile != null) {\n System.out.println(\"Loading exceptions from: \" + infile);\n System.out.println();\n aanFilter.loadAAnExceptions(infile);\n }\n \n // then filter given tokens\n Tokenizer tokenizer = new DefaultTokenizer();\n List words = tokenizer.tokenize(tokens); //, true);\n System.out.println(\"filtering: \" + tokens);\n System.out.println(\"filter out: \" + aanFilter.filterOut(words));\n }\n}\n\n" }, { "path": "src/opennlp/ccg/ngrams/AbstractStandardNgramModel.java", "content": "/*\n * $Id: AbstractStandardNgramModel.java,v 1.3 2009/12/21 03:27:18 mwhite14850 Exp $ \n */\npackage opennlp.ccg.ngrams;\n\nimport java.util.ArrayList;\nimport java.util.Iterator;\nimport java.util.List;\n\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.util.Pair;\n\n\n/**\n * Abstract class for shared methods used by all standard ngram models.\n * Adapted from the original StandardNgramModel class. \n * \n * @author Scott Martin\n * @version $Revision: 1.3 $\n * @since 0.9.2\n */\nabstract class AbstractStandardNgramModel extends NgramScorer {\n\n\t/** Reusable list of strings to score. */\n protected List stringsToScore = new ArrayList();\n\t\n /**\n * Creates a new ngram model of the given order.\n * @param order The order of the model.\n * @param useSemClasses Whether this model should use semantic classes.\n * @see NgramScorer#NgramScorer(int, boolean)\n */\n protected AbstractStandardNgramModel(int order, boolean useSemClasses) {\n\t\tsuper(order, useSemClasses);\n\t\tnumNgrams = new int[order];\n\t}\n\n /**\n * Creates a new ngram model with the specified order.\n * @see AbstractStandardNgramModel#AbstractStandardNgramModel(int, boolean)\n */\n protected AbstractStandardNgramModel(int order) {\n\t\tthis(order, false);\n\t}\n\n\t/**\n * Converts the words in wordsToScore to strings in stringsToScore, before\n * scoring.\n */\n @Override\n protected void prepareToScoreWords() {\n stringsToScore.clear();\n for (int i = 0; i < wordsToScore.size(); i++) {\n Word w = wordsToScore.get(i);\n String s = w.getForm();\n // check for sem class replacement\n String scr = semClassReplacement(w);\n if (scr != null) s = scr;\n // add pitch accent and attrs, if any\n String pitchAccent = w.getPitchAccent();\n Iterator> pairs = w.getAttrValPairs();\n if (pitchAccent != null || pairs.hasNext()) {\n StringBuffer sb = new StringBuffer();\n sb.append(s);\n if (pitchAccent != null) sb.append('_').append(pitchAccent);\n for (; pairs.hasNext(); ) {\n \tPair p = pairs.next();\n sb.append('_').append(p.b);\n }\n s = sb.toString().intern();\n }\n // check for unknown word\n if (openVocab && trieMapRoot.getChild(s) == null)\n s = \"\";\n // add key\n stringsToScore.add(s);\n }\n }\n \n /**\n * Returns the log prob of the ngram starting at the given index \n * in wordsToScore and with the given order, with backoff. \n * (Assumes words in wordsToScore have already been converted to strings in \n * stringsToScore, via call to prepareToScoreWords.)\n */\n @Override\n protected float logProbFromNgram(int i, int order) {\n // skip initial start tag\n if (i == 0 && order == 1 && stringsToScore.get(0) == \"\") return 0;\n // set keys list\n keysList.clear();\n for (int j = i; j < i+order; j++) {\n keysList.add(stringsToScore.get(j));\n }\n if (debugScore) {\n System.out.print(\"logp( \" + keysList.get(order-1) + \" | \");\n if (order > 1) { \n System.out.print(keysList.get(order-2) + \" ... \");\n }\n System.out.print(\") = \");\n }\n // calc log prob\n float retval = logProb(0, order);\n if (debugScore) System.out.println(\"\" + retval);\n return retval;\n }\n\n}\n" }, { "path": "src/opennlp/ccg/ngrams/ConditionalProbabilityTable.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2010 Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.ngrams;\n\nimport java.io.IOException;\nimport java.util.*;\n\nimport opennlp.ccg.lexicon.*;\nimport opennlp.ccg.util.*;\n\n/**\n * A conditional probability table implemented via a FactoredNgramModelFamily, \n * where probabilities are determined by n-grams of \"factors\" of a single \"word\" \n * (whether the pairs of attributes and values have anything to do with words\n * or not).\n *\n * @author Michael White\n * @version $Revision: 1.2 $, $Date: 2010/02/25 22:26:10 $\n */\npublic class ConditionalProbabilityTable {\n\t\n\t/**\n\t * The model, as a factored n-gram model family.\n\t */\n\tprotected FactoredNgramModelFamily model;\n\t\n\t/**\n\t * Constructor with filename for model as a factored n-gram model family.\n\t */\n\tpublic ConditionalProbabilityTable(String filename) throws IOException {\n\t\tmodel = new FactoredNgramModelFamily(filename, false);\n\t\tif (model.order != 1) {\n\t\t\tthrow new RuntimeException(\"A conditional probability table must have n-gram order 1.\");\n\t\t}\n\t}\n\n /** Sets the debug score flag. */\n public void setDebug(boolean debugScore) { model.setDebug(debugScore); }\n \n\t/**\n\t * Returns a probability from the model for the given list of attribute-value \n\t * pairs, which are assumed to have already been interned, by converting \n\t * the result of the logprob method.\n\t */\n\tpublic double score(List> attrValList) {\n\t\treturn NgramScorer.convertToProb(logprob(attrValList));\n\t}\n\t\n\t/**\n\t * Returns a log probability from the model for the given list of attribute-value \n\t * pairs, which are assumed to have already been interned.\n\t * The order of the list does not matter because the model probabilities are \n\t * defined by the factored n-gram model family specification.\n\t */\n\tpublic double logprob(List> attrValList) {\n\t\treturn model.logprob(new SingletonList(new ListPairWord(attrValList)));\n\t}\n\t\n /** Tests loading and scoring. */\n public static void main(String[] args) throws IOException {\n \n String usage = \"Usage: java opennlp.ccg.ngrams.ConditionalProbabilityTable \";\n \n if (args.length > 0 && args[0].equals(\"-h\")) {\n System.out.println(usage);\n System.exit(0);\n }\n \n String specfile = args[0];\n String tokens = args[1];\n \n List> pairs = new ArrayList>();\n String[] tokenArray = tokens.split(\"\\\\s+\");\n for (int i=0; i < tokenArray.length; i+=2) {\n \tString attr = tokenArray[i].intern();\n \tString val = tokenArray[i+1].intern();\n \tpairs.add(new Pair(attr, val));\n }\n \n System.out.println(\"Loading conditional probability table from: \" + specfile);\n ConditionalProbabilityTable table = new ConditionalProbabilityTable(specfile);\n FactoredNgramModelFamily lmFamily = table.model;\n System.out.println(\"primary child var: \" + lmFamily.primaryGroup.childName);\n if (lmFamily.furtherGroups != null) {\n for (int i = 0; i < lmFamily.furtherGroups.length; i++) {\n System.out.println(\"further child var: \" + lmFamily.furtherGroups[i].childName);\n }\n }\n System.out.println(\"openVocab: \" + lmFamily.openVocab);\n System.out.println();\n \n System.out.println(\"scoring: \" + tokens);\n System.out.println();\n table.setDebug(true);\n double logprob = table.logprob(pairs);\n double score = NgramScorer.convertToProb(logprob);\n System.out.println(\"score: \" + score);\n System.out.println(\"logprob: \" + logprob);\n }\n}\n" }, { "path": "src/opennlp/ccg/ngrams/FactoredNgramModel.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2004-5 University of Edinburgh (Michael White)\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.ngrams;\r\n\r\nimport opennlp.ccg.lexicon.*;\r\nimport opennlp.ccg.util.*;\r\n\r\nimport java.util.*;\r\nimport java.io.*;\r\n\r\n/**\r\n * A scorer for a factored n-gram backoff model. \r\n * The file format is the one generated by the SRILM version 1.4.1 fngram-count tool.\r\n * Only static backoff orders are supported at present, with \r\n * the most distant parent variable dropped at each backoff point. \r\n * Unknown words/factors are mapped to <unk> if the latter is present in \r\n * the model.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.22 $, $Date: 2011/10/11 03:29:42 $\r\n */\r\npublic class FactoredNgramModel extends NgramScorer\r\n{\r\n\t/** Map for caching and reusing individual models by filename. */\r\n\tpublic static Map modelCache = new HashMap();\r\n\t\r\n /** The variable to predict. */\r\n public final ModelVariable child;\r\n \r\n /** The parent variables to condition on, backing off from the end. */\r\n public final ModelVariable[] parents;\r\n \r\n /** A variable in a factored n-gram model. */\r\n public class ModelVariable {\r\n /** The variable name. */\r\n public final String name;\r\n /** The (absolute value of the) position in the history. */\r\n public final int position;\r\n /** Makes a model variable from a string such as \"W\" or \"W(-1)\". */\r\n public ModelVariable(String str) {\r\n int leftparen = str.indexOf(\"(\");\r\n if (leftparen > 0) {\r\n name = str.substring(0,leftparen).intern(); \r\n int rightparen = str.indexOf(\")\");\r\n position = Math.abs(Integer.parseInt(str.substring(leftparen+1,rightparen)));\r\n }\r\n else { name = str.intern(); position = 0; }\r\n }\r\n }\r\n\r\n \r\n /** \r\n * Loads a factored n-gram model for the given child variable, with the given parent variables, \r\n * from the file with the given name, in the SRILM format.\r\n * The flag for using sem classes is defaulted to true.\r\n */\r\n public FactoredNgramModel(String child, String parents[], String filename) throws IOException { \r\n this(child, parents, filename, true);\r\n }\r\n \r\n /** \r\n * Loads a factored n-gram model for the given child variable, with the given parent variables, \r\n * from the file with the given name, in the SRILM format, \r\n * and with the given flag for using sem classes.\r\n */\r\n public FactoredNgramModel(String child, String parents[], String filename, boolean useSemClasses) throws IOException {\r\n this.useSemClasses = useSemClasses;\r\n this.child = new ModelVariable(child);\r\n this.parents = new ModelVariable[parents.length];\r\n order = 1;\r\n for (int i = 0; i < parents.length; i++) {\r\n this.parents[i] = new ModelVariable(parents[i]);\r\n order = Math.max(order, this.parents[i].position + 1);\r\n }\r\n this.numNgrams = new int[(int)Math.pow(2, parents.length)];\r\n // check cache\r\n \tFactoredNgramModel cachedModel = modelCache.get(filename);\r\n \tif (cachedModel != null) {\r\n \t\t// share trie etc.\r\n \t\tnumNgrams = cachedModel.numNgrams;\r\n \t\topenVocab = cachedModel.openVocab;\r\n \t\ttrieMapRoot = cachedModel.trieMapRoot;\r\n \t\tcachedLogProbs = cachedModel.cachedLogProbs;\r\n \t}\r\n \t// otherwise load model\r\n \telse {\r\n Reader in = new BufferedReader(new FileReader(filename));\r\n readModel(in);\r\n \t\tmodelCache.put(filename, this);\r\n \t}\r\n }\r\n \r\n \r\n /**\r\n\t * Returns a list of feature keys for the ngram starting at the given index in\r\n\t * wordsToScore and with the given order, using the keys in keysList after\r\n\t * setting them appropriately with setKeysToNgram; returns null if this\r\n\t * operation does not succeed normally. With factored models, factor keys \r\n\t * are sequenced as two string keys.\r\n\t */\r\n protected List ngram(int i, int order) { \r\n \tboolean ok = setKeysToNgram(i, order);\r\n \tif (!ok) return null;\r\n \tfeatureKeysList.clear();\r\n \tfor (int j=0; j < keysList.size(); j++) {\r\n \t\tObject key = keysList.get(j);\r\n \t\tif (key instanceof String) {\r\n \t\t\tfeatureKeysList.add(Tokenizer.WORD_ATTR);\r\n \t\t\tfeatureKeysList.add((String)key);\r\n \t\t}\r\n \t\telse if (!(key instanceof FactorKey)) {\r\n \t\t\tthrow new RuntimeException(\"Factor keys expected here! key: \" + key);\r\n \t\t}\r\n \t\telse {\r\n \t\t\tFactorKey fkey = (FactorKey) key;\r\n \t\t\tfeatureKeysList.add(fkey.factor);\r\n \t\t\tfeatureKeysList.add(fkey.val);\r\n \t\t}\r\n \t}\r\n \treturn featureKeysList;\r\n }\r\n\r\n \r\n /** Returns the log prob of the ngram starting at the given index \r\n in wordsToScore and with the given order, with backoff. \r\n If using sem classes, then words with replacement sem classes are \r\n mapped to words with the sem class replacing the form and stem.\r\n Any remaining unknown words/factors are mapped to <unk>, \r\n if the latter is present in the model. */\r\n // extracts factor keys from full words according to parents list, \r\n // then determines log prob from the list of factor keys\r\n protected float logProbFromNgram(int i, int order) {\r\n // skip initial start tag\r\n if (i == 0 && order == 1 && ((Word)wordsToScore.get(0)).getForm() == \"\") return 0;\r\n // set up factor keys\r\n keysList.clear();\r\n int i0 = i + order-1; // index of current word\r\n // determine last available parent, if full context not available\r\n int lastParentIndex = parents.length-1;\r\n if (this.order > order) {\r\n for (int j = 0; j < parents.length; j++) {\r\n if (parents[j].position >= order) { lastParentIndex = j-1; break; }\r\n }\r\n }\r\n // go through parents in reverse order, \r\n // extracting and adding factor keys\r\n for (int j = lastParentIndex; j >= 0; j--) {\r\n int pos_j = i0 - parents[j].position;\r\n if (pos_j < i) continue; // skip if pos_j past i\r\n Word w = (Word) wordsToScore.get(pos_j);\r\n keysList.add(makeFactorKey(w, parents[j].name));\r\n }\r\n // add factor key for child\r\n Word current = (Word) wordsToScore.get(i0);\r\n keysList.add(makeFactorKey(current, child.name));\r\n if (debugScore) {\r\n System.out.print(\"logp( \" + keysList.get(keysList.size()-1) + \" | \");\r\n for (int j = keysList.size()-2; j >= 0; j--) {\r\n System.out.print(keysList.get(j) + \" \");\r\n }\r\n System.out.print(\") = \");\r\n }\r\n // calc log prob from factor keys\r\n float retval = logProb(0, keysList.size());\r\n // NB: workaround for apparent bug in SRILM 1.4.1 fngram-count tool, \r\n // whereby prob for does not use higher-order contexts for \r\n // factors other than W: just use zero\r\n if (current.getForm() == \"\" && child.name != \"W\") retval = 0;\r\n if (debugScore) System.out.println(\"\" + retval);\r\n return retval;\r\n }\r\n \r\n // makes a factor key from the given word by extracting \r\n // the attribute with the given name, where\r\n // the delimiter tokens are treated as a special case, \r\n // and the attr val is adjusted if using sem classes\r\n private Object makeFactorKey(Word w, String attr) {\r\n // special cases for and : just return \r\n // a word with this form, regardless of the attr\r\n String form = w.getForm();\r\n if (form == \"\" || form == \"\") {\r\n return FactorKey.getKey(attr, form);\r\n }\r\n // get val for this attr\r\n String val = w.getVal(attr);\r\n // check for sem class replacement for form or stem\r\n if (attr == Tokenizer.WORD_ATTR || attr == Tokenizer.STEM_ATTR) {\r\n String scr = semClassReplacement(w);\r\n if (scr != null) val = scr;\r\n }\r\n // make factor key \r\n Object retval = FactorKey.getKey(attr, val);\r\n // check for unknown val\r\n if (openVocab && trieMapRoot.getChild(retval) == null) {\r\n val = \"\";\r\n retval = FactorKey.getKey(attr, val);\r\n }\r\n // return \r\n return retval;\r\n }\r\n\r\n \r\n // reads in model\r\n private void readModel(Reader in) throws IOException {\r\n // setup\r\n\t\t//Tokenizer wordTokenizer = (Grammar.theGrammar != null) \r\n\t\t// ? Grammar.theGrammar.lexicon.tokenizer\r\n\t\t// : new DefaultTokenizer();\r\n StreamTokenizer tokenizer = initTokenizer(in); \r\n String[] tokens = new String[parents.length+3];\r\n Object[] factorKeys = new Object[parents.length+1];\r\n boolean foundData = false;\r\n int numParents = -1;\r\n int parentsInt = -1;\r\n List currentPrefix = new ArrayList();\r\n List currentKeys = null;\r\n List> currentChildren = null;\r\n // loop through lines\r\n while (tokenizer.ttype != StreamTokenizer.TT_EOF) {\r\n // read line into tokens\r\n readLine(tokenizer, tokens);\r\n // check for blank line\r\n if (tokens[0] == null) continue;\r\n // check for initial delimiter\r\n if (tokens[0].equals(\"\\\\data\\\\\")) { foundData = true; continue; }\r\n if (!foundData) continue;\r\n // read header line\r\n if (tokens[0].equals(\"ngram\")) {\r\n int equalPos = tokens[1].indexOf(\"=\");\r\n int n = Integer.decode(tokens[1].substring(0,equalPos)).intValue(); \r\n int total = Integer.parseInt(tokens[1].substring(equalPos+1));\r\n numNgrams[n] = total;\r\n // init children, keys lists\r\n if (currentChildren == null) { \r\n currentChildren = new ArrayList>(total); \r\n currentKeys = new ArrayList(total); \r\n }\r\n // calc totals (not actually used anymore)\r\n if (n == numNgrams.length-1) {\r\n @SuppressWarnings(\"unused\")\r\n\t\t\t\t\tint totalNgrams = 0;\r\n for (int i = 0; i < numNgrams.length; i++) { totalNgrams += numNgrams[i]; }\r\n // System.out.println(\"totalNgrams: \" + totalNgrams);\r\n }\r\n continue;\r\n }\r\n // check for final delimiter\r\n if (tokens[0].equals(\"\\\\end\\\\\")) {\r\n addTrieMapChildren(currentPrefix, currentKeys, currentChildren);\r\n break;\r\n }\r\n // read line starting new parents context\r\n int gramsPos = -1;\r\n if (tokens[0].startsWith(\"\\\\\") && (gramsPos = tokens[0].indexOf(\"-grams:\")) > 0) {\r\n // add current children\r\n addTrieMapChildren(currentPrefix, currentKeys, currentChildren);\r\n // update num parents\r\n // System.out.println(tokens[0]);\r\n parentsInt = Integer.decode(tokens[0].substring(1,gramsPos)).intValue();\r\n numParents = numParents(parentsInt);\r\n continue;\r\n }\r\n if (numParents < 0) continue;\r\n // current order is num parents plus one for child var\r\n int currentOrder = numParents + 1;\r\n // parse a line of the form\r\n // ... []\r\n // i.e. a log prob, followed by N vars, a child var, and an optional back off weight\r\n // NB: unlike the ARPA format, here the here is\r\n // associated with the backoff of ... \r\n // rather than context consisting of ... \r\n // read logprob\r\n float logprob = Float.parseFloat(tokens[0]);\r\n // unescape, intern factor keys\r\n for (int i = 1; i < currentOrder+1; i++) {\r\n String attr = (i == currentOrder) \r\n ? child.name \r\n : parents[(currentOrder-i)-1].name; \r\n String val = tokens[i];\r\n int hyphenPos = val.indexOf('-');\r\n if (hyphenPos > 0) {\r\n String attrCheck = val.substring(0, hyphenPos).intern();\r\n if (attr != attrCheck) {\r\n System.err.println(\r\n \"Warning: expected attr \" + attr + \" rather than \" + \r\n attrCheck + \" in \" + tokens[i]\r\n );\r\n }\r\n val = val.substring(hyphenPos+1);\r\n }\r\n val = DefaultTokenizer.unescape(val);\r\n if (val != null) val = val.intern();\r\n factorKeys[i-1] = FactorKey.getKey(attr, val);\r\n }\r\n // check prefix\r\n boolean samePrefix = (currentPrefix.size() == currentOrder-1);\r\n for (int i = 0; samePrefix && i < currentOrder-1; i++) {\r\n if (factorKeys[i] != currentPrefix.get(i)) samePrefix = false; \r\n }\r\n // if changed, add current children, reset prefix \r\n if (!samePrefix) {\r\n addTrieMapChildren(currentPrefix, currentKeys, currentChildren);\r\n for (int i = 0; i < currentOrder-1; i++) {\r\n currentPrefix.add(factorKeys[i]);\r\n }\r\n }\r\n Object key = factorKeys[currentOrder-1];\r\n currentKeys.add(key);\r\n currentChildren.add(new TrieMap(new NgramFloats(logprob, 0)));\r\n // read back-off weight, if present\r\n if (tokens[currentOrder+1] != null) {\r\n float bow = Float.parseFloat(tokens[currentOrder+1]);\r\n // add to prefix node\r\n TrieMap prefixNode = trieMapRoot.findChildFromList(currentPrefix);\r\n NgramFloats nfloats = prefixNode.data;\r\n if (nfloats != null) nfloats.bow = bow;\r\n else prefixNode.data = new NgramFloats(0, bow); \r\n }\r\n }\r\n // set openVocab according to presence of child \r\n Object unkKey = FactorKey.getKey(child.name,\"\");\r\n openVocab = (trieMapRoot.getChild(unkKey) != null);\r\n }\r\n\r\n \r\n // returns the number of parents present in the parentsInt spec\r\n private int numParents(int parentsInt) {\r\n int retval = 0;\r\n for (int i = 0; i < parents.length; i++) {\r\n if ((parentsInt & 1) != 0) retval++;\r\n parentsInt = parentsInt >> 1;\r\n }\r\n return retval;\r\n }\r\n \r\n \r\n /** Test loading and scoring. */\r\n // NB: This produces the same scores as the SRILM fngram tool when both \r\n // and tags are used, except that ...\r\n // NB: There is a workaround for an apparent bug in SRILM 1.4.1 fngram-count tool, \r\n // whereby prob for does not use higher-order contexts for \r\n // factors other than W, and thus a log prob of zero is just used instead.\r\n public static void main(String[] args) throws IOException {\r\n \r\n String usage = \"Usage: java opennlp.ccg.ngrams.FactoredNgramModel \";\r\n \r\n if (args.length > 0 && args[0].equals(\"-h\")) {\r\n System.out.println(usage);\r\n System.exit(0);\r\n }\r\n \r\n String child = args[0];\r\n String[] parents = args[1].split(\"\\\\s+\");\r\n String lmfile = args[2];\r\n String tokens = args[3];\r\n \r\n System.out.println(\"Loading n-gram model from: \" + lmfile);\r\n FactoredNgramModel lm = new FactoredNgramModel(child, parents, lmfile, true);\r\n System.out.println(\"child var: \" + lm.child.name);\r\n for (int i = 0; i < lm.parents.length; i++) {\r\n System.out.println(\"parent var: (\" + lm.parents[i].name + \",\" + lm.parents[i].position + \")\");\r\n }\r\n System.out.println(\"order: \" + lm.order);\r\n System.out.println(\"openVocab: \" + lm.openVocab);\r\n System.out.println();\r\n // System.out.println(\"trie map: \");\r\n // System.out.println(lm.trieMapRoot.toString());\r\n // System.out.println();\r\n \r\n Tokenizer tokenizer = new DefaultTokenizer();\r\n List words = tokenizer.tokenize(tokens, true);\r\n System.out.println(\"scoring: \");\r\n for (int i = 0; i < words.size(); i++) {\r\n System.out.println(words.get(i).toString());\r\n }\r\n System.out.println();\r\n lm.debugScore = true;\r\n lm.setWordsToScore(words, true);\r\n lm.prepareToScoreWords();\r\n double logprob = lm.logprob();\r\n double score = convertToProb(logprob);\r\n System.out.println();\r\n System.out.println(\"score: \" + score);\r\n System.out.println(\"logprob: \" + logprob);\r\n System.out.println(\"ppl: \" + NgramScorer.convertToPPL(logprob / (words.size()-1)));\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/ngrams/FactoredNgramModelFamily.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2004-5 University of Edinburgh (Michael White)\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.ngrams;\r\n\r\nimport opennlp.ccg.lexicon.*;\r\nimport opennlp.ccg.perceptron.Alphabet;\r\nimport opennlp.ccg.perceptron.FeatureMap;\r\n\r\nimport java.util.*;\r\nimport java.io.*;\r\n\r\n/**\r\n * A scorer consisting of a family of factored n-gram backoff models.\r\n * The family of models is specified using the factored language model specification \r\n * file format given as input to the SRILM version 1.4.1 fngram-count tool.\r\n * Each individual model is loaded as a FactoredNgramModel instance, and thus \r\n * only static backoff orders are supported at present, with \r\n * the most distant parent variable dropped at each backoff point.\r\n * The first model in the specification file should be the primary one.\r\n * It may be followed by any number of models for the same child variable \r\n * but with lower history orders; these models will be used when the \r\n * full history is unavailable, if their order matches the available \r\n * number of words in the history. For example, if the primary model is \r\n * a trigram model, then a bigram model may also be given (with a potentially \r\n * different backoff order) for scoring bigram word sequences.\r\n * Following these models, there may also be further (sequences of) models for scoring \r\n * different child variables. If present, the scores calculated for these \r\n * child variables will be multipled with the score calculated for the primary \r\n * model's child variable (typically the word form). For example, one or more \r\n * models may be given to calculate the probability of the word's pitch accent, \r\n * independently of the word form (though potentially looking at some of the same history).\r\n * With each individual model, the parents are assumed to be listed in backoff order.\r\n * Unknown words are mapped to <unk> if the latter is present in \r\n * the first model.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.15 $, $Date: 2010/02/25 22:26:11 $\r\n */\r\npublic class FactoredNgramModelFamily extends NgramScorer\r\n{\r\n /** The primary model group. */\r\n public final ModelGroup primaryGroup;\r\n \r\n /** Any additional model groups, for different child variables. */\r\n public final ModelGroup[] furtherGroups;\r\n \r\n /** A factored n-gram model and any secondary ones for the same child variable. */\r\n public class ModelGroup {\r\n /** The child variable name. */\r\n public final String childName;\r\n /** The primary model. */\r\n public final FactoredNgramModel primaryModel;\r\n /** The secondary models. */\r\n public final FactoredNgramModel[] secondaryModels;\r\n /** Makes a model group from the given primary model. */\r\n public ModelGroup(FactoredNgramModel primaryModel, FactoredNgramModel[] secondaryModels) {\r\n this.childName = primaryModel.child.name;\r\n this.primaryModel = primaryModel;\r\n this.secondaryModels = secondaryModels;\r\n }\r\n /** Returns the appropriate model for the given order. */\r\n public FactoredNgramModel getModel(int order) {\r\n if (secondaryModels == null) return primaryModel;\r\n for (int i = 0; i < secondaryModels.length; i++) {\r\n if (secondaryModels[i].order == order) return secondaryModels[i];\r\n }\r\n return primaryModel;\r\n }\r\n /** Propagates the reverse flag. */\r\n public void setReverse(boolean reverse) { \r\n primaryModel.setReverse(reverse); \r\n if (secondaryModels == null) return;\r\n for (int i = 0; i < secondaryModels.length; i++) {\r\n secondaryModels[i].setReverse(reverse);\r\n }\r\n }\r\n /** Propagates the debug score flag. */\r\n public void setDebug(boolean debugScore) { \r\n primaryModel.setDebug(debugScore); \r\n if (secondaryModels == null) return;\r\n for (int i = 0; i < secondaryModels.length; i++) {\r\n secondaryModels[i].setDebug(debugScore);\r\n }\r\n }\r\n /** Propagates wordsToScore to the given list, for sharing purposes. */\r\n protected void shareWordsToScore(List wordsToScore) {\r\n primaryModel.shareWordsToScore(wordsToScore);\r\n if (secondaryModels == null) return;\r\n for (int i = 0; i < secondaryModels.length; i++) {\r\n secondaryModels[i].shareWordsToScore(wordsToScore);\r\n }\r\n }\r\n \t/** Sets the alphabet. */\r\n \tpublic void setAlphabet(Alphabet alphabet) {\r\n \t\tprimaryModel.setAlphabet(alphabet);\r\n if (secondaryModels == null) return;\r\n for (int i = 0; i < secondaryModels.length; i++) {\r\n secondaryModels[i].setAlphabet(alphabet);\r\n }\r\n \t}\r\n }\r\n \r\n \r\n /** \r\n * Loads a family of factored n-gram models\r\n * from the file with the given name, in the SRILM format.\r\n * The flag for using sem classes is defaulted to true.\r\n */\r\n public FactoredNgramModelFamily(String filename) throws IOException { \r\n this(filename, true);\r\n }\r\n \r\n /** \r\n * Loads a family of factored n-gram models\r\n * from the file with the given name, in the SRILM format, \r\n * and with the given flag for using sem classes.\r\n */\r\n public FactoredNgramModelFamily(String filename, boolean useSemClasses) throws IOException {\r\n this.useSemClasses = useSemClasses;\r\n List modelGroups = readModel(filename);\r\n this.primaryGroup = modelGroups.get(0);\r\n if (modelGroups.size() == 1) this.furtherGroups = null;\r\n else {\r\n this.furtherGroups = new ModelGroup[modelGroups.size()-1];\r\n for (int i = 1; i < modelGroups.size(); i++) {\r\n this.furtherGroups[i-1] = modelGroups.get(i);\r\n }\r\n }\r\n order = primaryGroup.primaryModel.order;\r\n openVocab = primaryGroup.primaryModel.openVocab; \r\n }\r\n \r\n \r\n /** Sets reverse flag, and propagates to component models. */\r\n public void setReverse(boolean reverse) { \r\n super.setReverse(reverse);\r\n primaryGroup.setReverse(reverse);\r\n if (furtherGroups == null) return;\r\n for (int i = 0; i < furtherGroups.length; i++) {\r\n furtherGroups[i].setReverse(reverse);\r\n }\r\n }\r\n \r\n /** Sets debug score flag, and propagates to component models. */\r\n public void setDebug(boolean debugScore) { \r\n super.setDebug(debugScore);\r\n primaryGroup.setDebug(debugScore);\r\n if (furtherGroups == null) return;\r\n for (int i = 0; i < furtherGroups.length; i++) {\r\n furtherGroups[i].setDebug(debugScore);\r\n }\r\n }\r\n \r\n\r\n /** Sets wordsToScore to the given list, for sharing purposes. */\r\n protected void shareWordsToScore(List wordsToScore) {\r\n this.wordsToScore = wordsToScore;\r\n primaryGroup.shareWordsToScore(wordsToScore);\r\n if (furtherGroups == null) return;\r\n for (int i = 0; i < furtherGroups.length; i++) {\r\n furtherGroups[i].shareWordsToScore(wordsToScore);\r\n }\r\n }\r\n\r\n \r\n\t/** Sets the alphabet. */\r\n\tpublic void setAlphabet(Alphabet alphabet) {\r\n\t\tsuper.setAlphabet(alphabet);\r\n\t\tprimaryGroup.setAlphabet(alphabet);\r\n if (furtherGroups == null) return;\r\n for (int i = 0; i < furtherGroups.length; i++) {\r\n \tfurtherGroups[i].setAlphabet(alphabet);\r\n }\r\n\t}\r\n\t\r\n\t/**\r\n\t * Increments ngram counts for the ngrams starting at the given index in\r\n\t * wordsToScore and with the given order.\r\n\t */\r\n\tprotected void incNgrams(FeatureMap featmap, int i, int order) {\r\n // do primary group\r\n List ngram = ngram(primaryGroup, i, order);\r\n if (ngram != null) {\r\n\t Alphabet.Feature f = alphabet.index(ngram);\r\n\t\t\tif (f != null) featmap.inc(f);\r\n }\r\n // then any further ones\r\n if (furtherGroups != null) {\r\n for (int j = 0; j < furtherGroups.length; j++) {\r\n List ngram2 = ngram(furtherGroups[j], i, order);\r\n if (ngram2 == null) continue;\r\n Alphabet.Feature f2 = alphabet.index(ngram2);\r\n \t\tif (f2 != null) featmap.inc(f2);\r\n }\r\n }\r\n\t}\r\n\t\r\n // get ngram from a model group\r\n private List ngram(ModelGroup modelGroup, int i, int order) {\r\n FactoredNgramModel modelToUse = modelGroup.primaryModel; \r\n // with less than full history, get possibly different model to use\r\n if (order < modelToUse.order) {\r\n modelToUse = modelGroup.getModel(order); \r\n }\r\n return modelToUse.ngram(i, order);\r\n }\r\n \r\n \r\n /** Returns the log prob of the ngram starting at the given index \r\n in wordsToScore and with the given order, with backoff. */\r\n protected float logProbFromNgram(int i, int order) {\r\n float logProbTotal = 0;\r\n // do primary group\r\n logProbTotal += logProbFromNgram(primaryGroup, i, order);\r\n // then any further ones\r\n if (furtherGroups != null) {\r\n for (int j = 0; j < furtherGroups.length; j++) \r\n logProbTotal += logProbFromNgram(furtherGroups[j], i, order);\r\n }\r\n return logProbTotal;\r\n }\r\n \r\n // calculate the log prob from a model group\r\n private float logProbFromNgram(ModelGroup modelGroup, int i, int order) {\r\n FactoredNgramModel modelToUse = modelGroup.primaryModel; \r\n // with less than full history, get possibly different model to use\r\n if (order < modelToUse.order) {\r\n modelToUse = modelGroup.getModel(order); \r\n if (debugScore && modelToUse != modelGroup.primaryModel) {\r\n int modelNum = Arrays.asList(modelGroup.secondaryModels).indexOf(modelToUse);\r\n System.out.print(\"[2ndary model \" + modelNum + \"] \");\r\n }\r\n }\r\n return modelToUse.logProbFromNgram(i, order);\r\n }\r\n \r\n /** The max number of tokens to allow per line in the spec file. */\r\n public static int MAX_TOKENS_PER_LINE = 64;\r\n \r\n // reads in model, returning model groups\r\n private List readModel(String filename) throws IOException {\r\n // setup\r\n File infile = new File(filename);\r\n Reader in = new BufferedReader(new FileReader(infile));\r\n StreamTokenizer tokenizer = initTokenizer(in); \r\n String[] tokens = new String[MAX_TOKENS_PER_LINE];\r\n // read in models\r\n FactoredNgramModel[] models = null;\r\n int numModels = -1;\r\n int currentModel = 0;\r\n // loop through lines\r\n while (tokenizer.ttype != StreamTokenizer.TT_EOF) {\r\n // read line into tokens\r\n readLine(tokenizer, tokens);\r\n // check for blank line\r\n if (tokens[0] == null) continue;\r\n // check for comment\r\n if (tokens[0].charAt(0) == '#') continue;\r\n // read num models, if not yet found\r\n if (numModels < 0) {\r\n numModels = Integer.parseInt(tokens[0]);\r\n models = new FactoredNgramModel[numModels];\r\n continue;\r\n }\r\n // skip rest if already read in numModels\r\n if (currentModel >= numModels) break;\r\n // read model spec if second token is a colon\r\n // line format is : ... \r\n if (tokens[1] != null && tokens[1].equals(\":\")) {\r\n // read child\r\n String child = tokens[0];\r\n // read parents\r\n int numParents = Integer.parseInt(tokens[2]);\r\n String[] parents = new String[numParents];\r\n for (int i = 0; i < numParents; i++) parents[i] = tokens[i+3];\r\n // read lm filename (skipping count file name)\r\n String lmfn = tokens[numParents+4];\r\n // make filename relative to spec file dir\r\n File lmfile = new File(infile.getParentFile(), lmfn);\r\n lmfn = lmfile.getPath();\r\n // load current model\r\n \tmodels[currentModel] = new FactoredNgramModel(child, parents, lmfn, useSemClasses);\r\n // share wordsToScore\r\n models[currentModel].shareWordsToScore(wordsToScore);\r\n // inc current model\r\n currentModel++;\r\n }\r\n }\r\n // ensure models found\r\n if (models == null) {\r\n throw new IOException(\"No models found in: \" + filename);\r\n }\r\n // check num models\r\n int actualNumModels = 0;\r\n for (int i = 0; i < numModels; i++) {\r\n if (models[i] != null) actualNumModels++;\r\n }\r\n if (actualNumModels != numModels) {\r\n System.err.println(\"Warning: Only found \" + actualNumModels + \"/\" + numModels + \" in \" + filename);\r\n numModels = actualNumModels;\r\n }\r\n // assign models to groups\r\n List modelGroups = new ArrayList();\r\n int modelIndex = 0;\r\n while (modelIndex < numModels) {\r\n // get primary model, remember child name\r\n FactoredNgramModel primaryModel = models[modelIndex];\r\n String childName = primaryModel.child.name;\r\n modelIndex++;\r\n List secondaryModelsList = new ArrayList();\r\n // get secondary models with same child name\r\n while (modelIndex < numModels && models[modelIndex].child.name == childName) {\r\n secondaryModelsList.add(models[modelIndex]);\r\n modelIndex++;\r\n }\r\n FactoredNgramModel[] secondaryModels = new FactoredNgramModel[secondaryModelsList.size()];\r\n secondaryModelsList.toArray(secondaryModels);\r\n // make, add model group\r\n modelGroups.add(new ModelGroup(primaryModel, secondaryModels));\r\n }\r\n // done\r\n return modelGroups;\r\n }\r\n\r\n \r\n /** Test loading and scoring. */\r\n public static void main(String[] args) throws IOException {\r\n \r\n String usage = \"Usage: java opennlp.ccg.ngrams.FactoredNgramModelFamily \";\r\n \r\n if (args.length > 0 && args[0].equals(\"-h\")) {\r\n System.out.println(usage);\r\n System.exit(0);\r\n }\r\n \r\n String specfile = args[0];\r\n String tokens = args[1];\r\n \r\n System.out.println(\"Loading n-gram model family from: \" + specfile);\r\n FactoredNgramModelFamily lmFamily = new FactoredNgramModelFamily(specfile); \r\n System.out.println(\"primary child var: \" + lmFamily.primaryGroup.childName);\r\n if (lmFamily.furtherGroups != null) {\r\n for (int i = 0; i < lmFamily.furtherGroups.length; i++) {\r\n System.out.println(\"further child var: \" + lmFamily.furtherGroups[i].childName);\r\n }\r\n }\r\n System.out.println(\"order: \" + lmFamily.order);\r\n System.out.println(\"openVocab: \" + lmFamily.openVocab);\r\n System.out.println();\r\n \r\n Tokenizer tokenizer = new DefaultTokenizer();\r\n List words = tokenizer.tokenize(tokens, true);\r\n System.out.println(\"scoring: \");\r\n for (int i = 0; i < words.size(); i++) {\r\n System.out.println(words.get(i).toString());\r\n }\r\n System.out.println();\r\n lmFamily.setDebug(true);\r\n lmFamily.setWordsToScore(words, true);\r\n lmFamily.prepareToScoreWords();\r\n double logprob = lmFamily.logprob();\r\n double score = convertToProb(logprob);\r\n System.out.println(\"score: \" + score);\r\n System.out.println(\"logprob: \" + logprob);\r\n System.out.println(\"ppl: \" + NgramScorer.convertToPPL(logprob / (words.size()-1)));\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/ngrams/KenNgramModel.java", "content": "package opennlp.ccg.ngrams;\nimport opennlp.ccg.ngrams.kenlm.jni.KenLM;\nimport opennlp.ccg.ngrams.kenlm.MurmurHash;\nimport java.io.*;\nimport java.util.List;\nimport java.util.ArrayList;\nimport java.util.Map;\nimport java.util.HashMap;\nimport opennlp.ccg.lexicon.DefaultTokenizer;\nimport opennlp.ccg.lexicon.Tokenizer;\nimport opennlp.ccg.lexicon.Word;\n\npublic class KenNgramModel extends AbstractStandardNgramModel {\n \n // the nuts-n-bolts JNI class.\n private KenLM kenlm = null;\n\n // Map of hash of word to integer representation (integerized word used by KenLM).\n private Map hash2ID = new HashMap();\n // Map of String (word to score) to hash of word.\n// private Map word2Hash = new HashMap();\n // Map from hashes to tokens (to see whether we have encountered a new token). \n private Map hash2String = new HashMap();\n // List of vocabulary tokens. New items are added and assigned their index as a representation.\n private List vocabList = new ArrayList();\n // Whether to lowercase text before querying to the language model (e.g., \"Pierre Vinken\" => \"pierre vinken\").\n private boolean lowercaseText = false;\n // Whether to split up named entities before querying the language model (e.g., \"Pierre_Vinken\" => \"Pierre Vinken\")\n private boolean splitNEs = false;\n // What character delimter to use to split NEs on.\n private char neDelim = '_';\n \n // A reusable container for scoring strings.\n// private List someStringsToScore = null;\n\n // Whether to print out messages that trace the scoring process.\n public boolean debugScore = false;\n\n public KenNgramModel(int order, \n\t\t\t String lmFile,\n\t\t\t boolean useSemClasses,\n\t\t\t boolean lowercaseText,\n\t\t\t boolean splitNEs,\n\t\t\t char neDelim,\n\t\t\t boolean useNgramFeatures) throws IOException {\n\tsuper(order, useSemClasses);\n\tthis.lowercaseText = lowercaseText;\n\tthis.splitNEs = splitNEs;\n\tthis.neDelim = neDelim;\n\tkenlm = new KenLM(order, lmFile);\t\n//\tsomeStringsToScore = new ArrayList(order);\n\tthis.useNgramFeatures = useNgramFeatures;\n }\n\n /** Construct with order and filename. (Delegates to superclass for these flags). */\n public KenNgramModel(int order, String lmFile) throws IOException { this(order, lmFile, false); }\n /** Construct with order and filename and an indication of whether to use semantic classes. (Delegates to superclass for these flags). */\n public KenNgramModel(int order, String lmFile, boolean useSemClasses) throws IOException {\n\tthis(order, lmFile, useSemClasses, false, false, '_', false);\n }\n\n\n /**\n * Integerize a word and register it with the LM, if needed.\n */\n public int id(String token) {\n\tsynchronized (this) {\n\t long hash = 0;\n\t try {\n\t\thash = MurmurHash.hash64(token);\n\t } catch (UnsupportedEncodingException e) {\n\t\te.printStackTrace();\n\t }\n\t String hash_word = hash2String.get(hash);\n\t if (hash_word != null) {\n\t\treturn hash2ID.get(hash);\n\t } else {\n\t\tint id = vocabList.size();\n\t\t// let kenlm know about this word's ID.\n\t\tkenlm.registerWord(token, id);\n\t\tvocabList.add(token);\n\t\thash2String.put(hash, token);\n\t\thash2ID.put(hash, id);\n\t\treturn id;\n\t }\n\t}\n }\n\n /**\n * Lowercase each token, if desired, and split each token into a list of tokens\n * (splitting on NE delim token), if desired.\n */\n protected List splitAndLowercase(List words) {\n\tList tmp = new ArrayList(words.size());\n\tif(!(lowercaseText || splitNEs)) {\n\t return words;\n\t} else {\n\t for(Word w : words) {\n\t\tString wdString = w.getForm();\n\t\tString[] parts = wdString.replace(neDelim,' ').split(\"\\\\s+\");\n//\t\tArrayList subTmp = new ArrayList(parts.length);\n\t\tfor(String part : parts) {\n\t\t String newWdForm = (lowercaseText) ? part.toLowerCase() : part;\n\t\t // add null attr/val list, since it is not accessible.\n\t\t tmp.add(Word.createWord(newWdForm, w.getPitchAccent(), null, newWdForm, w.getPOS(), w.getSupertag(), w.getSemClass()));\n\t\t}\n\t }\n\t return tmp;\n\t}\n }\n\n /**\n * Resets wordsToScore to the given ones, reversing them when the reverse\n * flag is true, and adding sentence delimiters if not already present, when\n * the completeness flag is true. Delegates to the superclass \n */\n @Override\n protected void setWordsToScore(List words, boolean complete) {\n wordsToScore.clear();\n tagsAdded = false; \n\tList tmp = splitAndLowercase(words);\n\twords = tmp;\n\tsuper.setWordsToScore(words, complete);\n }\n\n\n /**\n * Calculates a log probability of a delineated substring of the strings\n * to score using KenLM. \n * @param pos The start position (inclusive) within the strings to score.\n * @param len The length, starting from pos, of the string\n * that should be used. \n */\n @Override\n public float logProb(int pos, int len) {\n\ttry {\n\t List range = new ArrayList(keysList.size());\n\t for(Object wts : keysList.subList(pos, pos + len)) range.add((String)wts);\n\n\t int rangeSize = range.size();\n\t if(rangeSize == 0) {\n\t\tthrow new IllegalArgumentException(\"empty range specified for log prob\");\n\t }\n\n\t // Get hashes of words.\n\t int[] wds = new int[range.size()];\n\t int cursor = 0;\n\t for(String s : range) wds[cursor++] = id(s);\n\t \n\t // call KenLM\n\t float result = kenlm.prob(wds);\n\t if(debugScore) {\n\t\tString wd = range.get(range.size()-1);\n\t\tString context = \"\";\n\t\tfor(String contextWord : range.subList(0,range.size()-1)) context += \" \" + contextWord;\n\t\tcontext = context.trim();\n\t\tSystem.out.println(\"logp(\" + wd + \" | \" + context + \") = \" + result);\n\t }\n\t return result;\n\t} catch(IndexOutOfBoundsException e) {\n\t return 0.0f;\n\t}\n }\n\n\n /** Test loading and scoring. */\n // NB: This produces the same scores as the SRILM ngram tool when both \n // and tags are used.\n public static void main(String[] args) throws IOException {\n \n String usage = \"Usage: java opennlp.ccg.ngrams.KenLM \";\n \n if (args.length > 0 && args[0].equals(\"-h\")) {\n System.out.println(usage);\n System.exit(0);\n }\n \n long start = System.currentTimeMillis();\n String order = args[0]; String lmfile = args[1]; String tokens = args[2];\n\tboolean lowercase = true, splitNEs = false;\n\n\t// we want to prove that there are NEs to split and that there are uppercase chars to preserve.\n\tfor(char c : tokens.toCharArray()) {\n\t if (c == '_') {\n\t\tsplitNEs = true;\n\t\tif (!lowercase) break;\n\t }\n\t \n\t if (Character.isUpperCase(c)) {\n\t\tlowercase = false;\n\t\tif(splitNEs) break;\n\t }\n\t}\n\n System.out.println(\"Loading n-gram model with order \" + order + \" from: \" + lmfile);\n KenNgramModel lm = new KenNgramModel(Integer.parseInt(order), lmfile, false, lowercase, splitNEs, '_', false);\n\tlm.debugScore = true;\n int secs = (int) (System.currentTimeMillis() - start) / 1000;\n System.out.println(\"secs: \" + secs);\n System.out.println();\n Tokenizer tokenizer = new DefaultTokenizer();\n List words = tokenizer.tokenize(tokens);\n System.out.println(\"scoring: \" + tokens);\n System.out.println();\n lm.setWordsToScore(words, true);\n lm.prepareToScoreWords();\n double logprob = lm.logprob();\n double score = convertToProb(logprob);\n System.out.println();\n System.out.println(\"score: \" + score);\n System.out.println(\"logprob: \" + logprob);\n\t// Find out how many words there are here.\n\tint size = lm.splitAndLowercase(words).size();\n System.out.println(\"ppl: \" + NgramScorer.convertToPPL(logprob / (size-1)));\n }\n}\n" }, { "path": "src/opennlp/ccg/ngrams/LinearNgramScorerCombo.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-5 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.ngrams;\n\nimport java.util.*;\n\nimport opennlp.ccg.perceptron.*;\nimport opennlp.ccg.synsem.Sign;\nimport opennlp.ccg.lexicon.Word;\n\n/**\n * Linear combination of n-gram probability models, \n * interpolated at the word level; can also be used \n * as a feature extractor in a perceptron model.\n * The models must have the same direction.\n *\n * @author Michael White\n * @version $Revision: 1.17 $, $Date: 2009/06/22 04:32:47 $\n */\npublic class LinearNgramScorerCombo extends NgramScorer implements FeatureExtractor\n{\n /** The component models. */\n protected NgramScorer[] models;\n \n /** The weights. */\n protected double weights[];\n \n /**\n * Creates a new linear combo model with the given component models \n * and with the combination weights determined by the rank order \n * centroid method. The models are assumed to be ordered from \n * most to least important.\n */\n public LinearNgramScorerCombo(NgramScorer[] models) { \n this(models, rankOrderCentroidWeights(models.length));\n }\n \n /**\n * Creates a new linear combo model with the given component models \n * and combination weights. The weights are assumed to sum to 1, \n * and the number of weights is assumed to match the number of models.\n * The wordsToScore list is shared across the component models.\n */\n public LinearNgramScorerCombo(NgramScorer[] models, double[] weights) {\n this.models = models;\n this.weights = weights;\n for (int i = 0; i < models.length; i++) {\n models[i].shareWordsToScore(wordsToScore);\n order = Math.max(order, models[i].order);\n }\n }\n \n /** Set reverse flag, and propagate to component models. */\n public void setReverse(boolean reverse) { \n super.setReverse(reverse);\n for (int i = 0; i < models.length; i++) {\n models[i].setReverse(reverse);\n }\n }\n \n /** Sets wordsToScore to the given list, for sharing purposes. */\n protected void shareWordsToScore(List wordsToScore) {\n this.wordsToScore = wordsToScore;\n for (int i = 0; i < models.length; i++) {\n models[i].shareWordsToScore(wordsToScore);\n }\n }\n \n /** Does further preparation before scoring words for each component model. */\n protected void prepareToScoreWords() {\n for (int i = 0; i < models.length; i++) {\n models[i].prepareToScoreWords();\n }\n }\n \n /** Returns the log prob of the ngram starting at the given index \n in wordsToScore and with the given order, with backoff.\n In particular, returns the linear combination using the established weights \n of the probabilities given by the component models,\n converted back to a log prob (base 10). */\n protected float logProbFromNgram(int i, int order) {\n double prob = 0;\n for (int j = 0; j < models.length; j++) {\n prob += convertToProb(models[j].logProbFromNgram(i, order)) * weights[j];\n }\n return (float) convertToLogProb(prob);\n }\n \n \n\t/**\n\t * Increments ngram counts for the ngrams starting at the given index in\n\t * wordsToScore and with the given order. The implementation \n\t * delegates to the component models.\n\t */\n\tprotected void incNgrams(FeatureMap featmap, int i, int order) {\n for (int j = 0; j < models.length; j++) {\n \tif (models[j].useNgramFeatures) models[j].incNgrams(featmap, i, order);\n }\n\t}\n\t\n\t\n\t/** Flag for including the interpolated log prob as a feature. */\n\tprotected boolean useInterpLogProb = true;\n\t\n\t/** Sets the flag for including interpolated log prob as a feature. */\n\tpublic void setInterpLogProb(boolean useInterpLogProb) { \n\t\tthis.useInterpLogProb = useInterpLogProb; \n\t}\n\t\n /**\n * Returns a feature vector with the log prob features \n * for the given sign and completeness flag.\n\t * In particular, returns the log prob from each model as \n\t * the value of a feature named '$ngramN'.\n\t * The interpolated log prob is also returned as a feature '$ngram' if \n\t * the flag for including the interpolated log prob as a feature is set (the default).\n */\n public FeatureVector extractLogProbs(Sign sign, boolean complete) {\n\t\tFeatureList retval = new FeatureList(models.length+1);\n\t\tif (useInterpLogProb) {\n\t Alphabet.Feature f = alphabet.index(\"$ngram\");\n\t\t\tif (f != null) retval.add(f, (float)logprob(sign, complete));\n\t\t}\n\t\tfor (int i=0; i < models.length; i++) {\n\t Alphabet.Feature f = alphabet.index(\"$ngram\"+i);\n\t\t\tif (f != null) retval.add(f, (float)models[i].logprob(sign, complete));\n\t\t}\n\t\treturn retval;\n }\n \n\t/** Sets the alphabet. */\n\tpublic void setAlphabet(Alphabet alphabet) {\n\t\tsuper.setAlphabet(alphabet);\n\t\tfor (NgramScorer scorer : models) {\n\t\t\tscorer.setAlphabet(alphabet);\n\t\t}\n\t}\n}\n\n" }, { "path": "src/opennlp/ccg/ngrams/NgramDiversityPruningStrategy.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.ngrams;\n\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.realize.*;\nimport opennlp.ccg.synsem.Sign;\n\nimport java.util.*;\n\n/**\n * A diversity pruning strategy that defines signs to be \n * notCompellinglyDifferent if the n-1 initial and final words \n * are the same, where n is the n-gram order of interest.\n * The single arg constructor defaults the singleBestPerGroup flag \n * to true, which can increase efficiency with no loss in quality \n * when only the single best output is of interest (as long as the \n * reduction in the search space outweighs the extra time necessary \n * to check for the same initial and final words).\n *\n * @author Michael White\n * @version $Revision: 1.3 $, $Date: 2009/12/21 03:27:18 $\n */\npublic class NgramDiversityPruningStrategy extends DiversityPruningStrategy\n{\n /** The n-gram order. */\n protected int order;\n \n /** Constructor that defaults singleBestPerGroup to true. */\n public NgramDiversityPruningStrategy(int order) { this(order, true); }\n \n /** Full constructor. */\n public NgramDiversityPruningStrategy(int order, boolean singleBestPerGroup) { \n this.order = order; this.singleBestPerGroup = singleBestPerGroup;\n }\n \n /** Returns true iff the given signs are not compellingly different.\n In particular, returns true iff the n-1 initial and final words are the same. */\n public boolean notCompellinglyDifferent(Sign sign1, Sign sign2) {\n List words1 = sign1.getWords(); List words2 = sign2.getWords();\n int words1Len = words1.size(); int words2Len = words2.size(); \n for (int i = 0; i < order-1 && i < words1Len && i < words2Len; i++) {\n if (words1.get(i) != words2.get(i)) return false;\n }\n int wordsLenDiff = words1Len-words2Len;\n for (int i = words1Len-1; i > words1Len-order && i >= 0 && i >= wordsLenDiff; i--) {\n int j = i - wordsLenDiff;\n if (words1.get(i) != words2.get(j)) return false;\n }\n return true;\n }\n}\n\n" }, { "path": "src/opennlp/ccg/ngrams/NgramFilter.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.ngrams;\n\nimport opennlp.ccg.lexicon.Word;\n\nimport java.util.*;\n\n/**\n * Interface for objects that filter unhappy n-grams.\n *\n * @author Michael White\n * @version $Revision: 1.3 $, $Date: 2005/10/20 18:49:42 $\n */\npublic interface NgramFilter\n{\n /** Returns whether to filter out the given list of words. */\n public boolean filterOut(List words);\n}\n\n" }, { "path": "src/opennlp/ccg/ngrams/NgramPrecisionModel.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-5 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.ngrams;\n\nimport opennlp.ccg.grammar.*;\nimport opennlp.ccg.lexicon.*;\nimport opennlp.ccg.perceptron.FeatureVector;\nimport opennlp.ccg.synsem.Sign;\nimport opennlp.ccg.util.*;\n\nimport gnu.trove.*;\nimport java.util.*;\n\n/**\n * N-gram precision scoring model, using a linear combination of \n * n-grams with rank order centroid weights, and optionally replacing word forms with \n * their semantic classes.\n * Words in the target strings are assumed to contain any desired delimiters.\n * With the exact matches flag set, only exact matches count.\n *\n * @author Michael White\n * @version $Revision: 1.18 $, $Date: 2011/05/15 20:35:06 $\n */\npublic class NgramPrecisionModel extends NgramScorer implements SelfParaphraseBiaser\n{\n // n-grams in the target phrases\n @SuppressWarnings(\"unchecked\")\n\tprivate Set> targetNgrams = new THashSet();\n \n // weights\n private double[] weights = null;\n \n // exact matches flag\n private boolean exactMatches = false;\n \n /** Reusable list of reduced words. */\n protected List reducedWords = new ArrayList();\n \n /** Reusable word list, with identity equals. */\n protected List wordList = new ArrayListWithIdentityEquals();\n\n \n /**\n * Creates a new 4-gram precision model with no initial target strings \n * and with the combination weights determined by the rank order centroid method. \n * Word forms are not replaced by their semantic classes.\n */\n public NgramPrecisionModel() { \n this(new String[]{}, false);\n }\n \n /**\n * Creates a new 4-gram precision model from the given target strings \n * and with the combination weights determined by the rank order centroid method. \n * Word forms are not replaced by their semantic classes.\n */\n public NgramPrecisionModel(String[] targets) { \n this(targets, false);\n }\n \n /**\n * Creates a new n-gram precision model of the given order from the given target strings \n * and with the combination weights determined by the rank order centroid method. \n * Word forms are not replaced by their semantic classes.\n */\n public NgramPrecisionModel(String[] targets, int order) { \n this(targets, order, false);\n }\n \n /**\n * Creates a new 4-gram precision model from the given target strings, \n * with the given flag controlling whether word forms are replaced by their semantic classes, \n * and with the combination weights determined by the rank order centroid method. \n */\n public NgramPrecisionModel(String[] targets, boolean useSemClasses) { \n this(targets, 4, useSemClasses);\n }\n \n /**\n * Creates a new n-gram precision model of the given order from the given target strings, \n * with the given flag controlling whether word forms are replaced by their semantic classes, \n * and with the combination weights determined by the rank order centroid method. \n */\n public NgramPrecisionModel(String[] targets, int order, boolean useSemClasses) { \n this(targets, order, useSemClasses, rankOrderCentroidWeights(order));\n }\n \n /**\n * Creates a new n-gram precision model of the given order from the given target strings, \n * with the given flag controlling whether word forms are replaced by their semantic classes, \n * and with the given combination weights, beginning with the \n * highest-order weight and ending with the lowest-order (unigram) weight.\n */\n public NgramPrecisionModel(String[] targets, int order, boolean useSemClasses, double[] weights) {\n this.useSemClasses = useSemClasses;\n this.order = order;\n this.weights = new double[order];\n for (int i = 0; i < order; i++) {\n this.weights[order-(i+1)] = weights[i];\n }\n initTargetNgrams(targets);\n }\n \n \n /** Sets the exact matches flag. */\n public void setExactMatches(boolean exactMatches) { this.exactMatches = exactMatches; }\n \n /** Returns the exact matches flag. */\n public boolean getExactMatches() { return exactMatches; }\n \n \n /** Reduces the words in wordsToScore to reducedWords, before scoring. */\n protected void prepareToScoreWords() {\n reducedWords.clear();\n for (int i = 0; i < wordsToScore.size(); i++) {\n Word w = wordsToScore.get(i);\n reducedWords.add(reduceWord(w));\n }\n }\n \n /** Returns the given word reduced to a surface word, using the sem class, if apropos. */\n protected Word reduceWord(Word w) {\n if (useSemClasses && isReplacementSemClass(w.getSemClass())) \n return Word.createSurfaceWordUsingSemClass(w);\n else return Word.createSurfaceWord(w);\n }\n \n /** \n * Returns a score between 0 (worst) and 1 (best) for the given sign \n * and completeness flag, based on the n-gram score of the sign's words.\n * If the sign is complete, sentence delimiters are added before \n * scoring the words, if not already present.\n * Returns 0 if any filter flags the n-gram for filtering, or if \n * the sign has no words.\n * Otherwise, sets signToScore, calls prepareToScoreWords, \n * and then calculates and returns the n-gram precision score.\n * In particular, returns the linear combination using the established weights \n * of the various n-gram precision scores (from unigram up to the configured order), \n * where the n-gram precision is the number of n-grams with a match in the target \n * strings divided by the number of n-grams in the word sequence.\n * With the exact matches flag set, only exact matches count.\n * With short sequences (less than the order), the score is adjusted \n * proportionally to the max score.\n */\n public synchronized double score(Sign sign, boolean complete) {\n \t// setup\n List words = sign.getWords(); \n if (words == null) return 0;\n signToScore = sign;\n setWordsToScore(words, complete);\n if (ngramFilters != null) {\n for (int i = 0; i < ngramFilters.size(); i++) {\n NgramFilter filter = ngramFilters.get(i);\n if (filter.filterOut(wordsToScore)) return 0;\n }\n }\n prepareToScoreWords();\n // calc weighted precision score\n double retval = 0;\n for (int i = 0; i < order; i++) {\n retval += weights[i] * ngramPrecision(i+1);\n }\n signToScore = null;\n // adjust score for short sequences\n int numWords = wordsToScore.size();\n if (numWords < order) {\n \tdouble max = 0.0;\n \tfor (int i=0; i < numWords; i++) max += weights[i];\n \tretval = retval / max;\n }\n // done\n return retval;\n }\n\n\n\t/** \n\t * Returns the features as counts of each ngram for the given sign and completeness flag.\n\t * This method returns the feature map as a feature vector.\n\t */\n\tpublic FeatureVector extractFeatures(Sign sign, boolean complete) {\n\t\treturn extractFeatureMap(sign, complete);\n\t}\n\t\n \n /** Not supported; throws an UnsupportedOperationException. */\n protected float logProbFromNgram(int i, int order) {\n throw new UnsupportedOperationException();\n }\n \n \n // returns the n-gram precision of the given order, or zero if too few words\n private double ngramPrecision(int order) {\n int numWords = reducedWords.size();\n int numNgrams = numWords - (order-1);\n if (numNgrams <= 0) return 0;\n int matches = 0;\n for (int i=0; i < numNgrams; i++) {\n setNgram(reducedWords, i, order);\n if (targetNgrams.contains(wordList))\n matches++;\n }\n if (exactMatches) \n \treturn (matches == numNgrams) ? 1.0 : 0.0;\n else \n \treturn (matches * 1.0) / numNgrams;\n }\n\n /** Sets wordList to be the n-gram of the given order using words starting at pos i. */\n protected synchronized void setNgram(List words, int i, int order) {\n wordList.clear();\n for (int j = 0; j < order; j++) {\n wordList.add(words.get(i+j)); \n } \n }\n\n /**\n\t * Sets the keys in keysList to hold the ngram starting at the given index in\n\t * wordsToScore and with the given order; returns true if the operation \n\t * succeeds normally. The implementation uses reducedWords.\n\t */\n\tprotected boolean setKeysToNgram(int i, int order) {\n\t\tkeysList.clear();\n for (int j = 0; j < order; j++) {\n \tkeysList.add(reducedWords.get(i+j).getForm());\n }\n\t\treturn true;\n\t}\n\n \n /** Makes a canonical n-gram of the given order using words starting at pos i. \n Sublists are shared, a la a trie data structure. */\n @SuppressWarnings(\"unchecked\")\n\tprotected List makeNgram(List words, int i, int order) {\n // check for one already interned\n setNgram(words, i, order);\n List alreadyInterned = (List) Interner.getGlobalInterned(wordList);\n if (alreadyInterned != null) return alreadyInterned;\n // if order is 1, intern new singleton list\n if (order == 1) {\n return (List) Interner.globalIntern(new SingletonList(words.get(i)));\n }\n // otherwise, extend list for the first word with suffix list\n List firstOneList = makeNgram(words, i, 1);\n List suffixList = makeNgram(words, i+1, order-1); \n return (List) Interner.globalIntern(new StructureSharingList(firstOneList, suffixList));\n }\n \n\n // initializes the n-grams from the target phrases\n private void initTargetNgrams(String[] targets) {\n for (int j = 0; j < targets.length; j++) {\n \tif (targets[j].length() == 0) continue;\n // parse or tokenize target phrase into words\n List words;\n if (useSemClasses) // use parsed words to get sem classes\n words = Grammar.theGrammar.getParsedWords(targets[j]);\n else\n words = Grammar.theGrammar.lexicon.tokenizer.tokenize(targets[j]);\n // add sentence delimiters, if not already present\n setWordsToScore(words, true);\n // reduce each word to a surface word, using the sem class if apropos\n int numWords = wordsToScore.size();\n for (int i = 0; i < numWords; i++) {\n Word w = wordsToScore.get(i);\n wordsToScore.set(i, reduceWord(w));\n }\n // make and store target n-grams\n for (int k=0; k < order; k++) {\n for (int i=0; i < numWords - k; i++) {\n targetNgrams.add(makeNgram(wordsToScore, i, k+1));\n }\n }\n }\n }\n \n\t/** Sets the target strings for implementing the self-paraphrase bias. */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic void setTargets(String[] targets) {\n\t\ttargetNgrams = new THashSet();\n\t\tinitTargetNgrams(targets);\n\t}\n}\n\n" }, { "path": "src/opennlp/ccg/ngrams/NgramScorer.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-8 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.ngrams;\n\nimport opennlp.ccg.synsem.Sign;\nimport opennlp.ccg.synsem.SignScorer;\nimport opennlp.ccg.grammar.Grammar;\nimport opennlp.ccg.lexicon.*;\nimport opennlp.ccg.util.*;\nimport opennlp.ccg.perceptron.*;\n\nimport java.util.*;\nimport java.io.*;\n\n/**\n * Super class for n-gram scoring models.\n *\n * @author Michael White\n * @version $Revision: 1.37 $, $Date: 2010/02/25 22:26:11 $\n */\npublic abstract class NgramScorer implements SignScorer, Reversible, FeatureExtractor\n{\n\tprotected NgramScorer() {}\n\t\n\tprotected NgramScorer(int order) {\n\t\tthis(order, false);\n\t}\n\t\n\tprotected NgramScorer(int order, boolean useSemClasses) {\n\t\tthis.order = order;\n\t\tthis.useSemClasses = useSemClasses;\n\t}\n\t\n /** The n-gram order of the model. */\n protected int order;\n \n /** Returns the n-gram order of the model. */\n public int getOrder() { return order; }\n\n /** Flag for whether to reverse words before scoring (defaults to false). */\n protected boolean reverse = false;\n \n /** Get reverse flag. */\n public boolean getReverse() { return reverse; }\n \n /** Set reverse flag, and propagate to any reversible filters. */\n public void setReverse(boolean reverse) { \n this.reverse = reverse; \n if (ngramFilters != null) {\n for (int i = 0; i < ngramFilters.size(); i++) {\n NgramFilter filter = ngramFilters.get(i);\n if (filter instanceof Reversible) \n ((Reversible)filter).setReverse(reverse);\n }\n }\n }\n \n /** Root of the n-gram trie. Nodes store NgramFloats instances. */\n protected TrieMap trieMapRoot = new TrieMap(null);\n\n /** An ngram data object, for holding the log prob and backoff weight. */\n public static class NgramFloats {\n /** The log prob. */\n public float logprob;\n /** The backoff weight. */\n public float bow;\n /** Constructor. */\n public NgramFloats(float logprob, float bow) {\n this.logprob = logprob; this.bow = bow; \n }\n \n @Override\n public String toString() { return \"logprob: \" + logprob + \", bow: \" + bow; }\n }\n\n \n /** The n-gram totals for different histories. */\n protected int[] numNgrams = null;\n \n /** Flag for open vocabulary, ie whether the unknown word <unk> is in the model. */\n protected boolean openVocab = false;\n \n /** Flag for whether to show scoring breakdown. */\n protected boolean debugScore = false;\n \n /** Sets the debug score flag. */\n public void setDebug(boolean debugScore) { this.debugScore = debugScore; } \n \n /** List of n-gram filters, for identifying unhappy sequences. */\n protected List ngramFilters = null;\n \n /** Adds an n-gram filter. */\n public void addFilter(NgramFilter filter) { \n if (ngramFilters == null) { ngramFilters = new ArrayList(); }\n ngramFilters.add(filter);\n }\n\n \n /** The alphabet, for filtering features to the relevant ones (when present). */\n protected Alphabet alphabet = null;\n \n /** Sets the alphabet, so that features can be filtered to the relevant ones (when present). */\n public void setAlphabet(Alphabet alphabet) { this.alphabet = alphabet; }\n \n \n /** Weak hash map for cached log probs, keyed from a sign's words. */\n protected Map,Float> cachedLogProbs = null;\n \n /** Reference to current sign to score. */\n protected Sign signToScore = null;\n \n /** Reusable list of words to score. */\n protected List wordsToScore = new ArrayList();\n\n /** Flag for whether start/end tags were added with the current words. */\n protected boolean tagsAdded = false;\n \n /** Reusable list of keys for n-gram lookups. */\n protected List keysList = new ArrayList();\n \n /** Reusable list of keys for n-gram feature lookups. */\n protected List featureKeysList = new ArrayList();\n \n \n /** Gets a cached log prob for the given list of words (or null if none). */\n protected Float getCachedLogProb(List words) {\n if (cachedLogProbs == null) return null;\n return cachedLogProbs.get(words);\n }\n \n /** Caches a log prob for the given list of words. */\n protected void putCachedLogProb(List words, Float logprob) {\n if (cachedLogProbs == null) cachedLogProbs = new WeakHashMap,Float>();\n cachedLogProbs.put(words, logprob);\n }\n\n \n /** \n * Returns a score between 0 (worst) and 1 (best) for the given sign \n * and completeness flag, based on the n-gram score of the sign's words.\n * If the sign is complete, sentence delimiters are added before \n * scoring the words, if not already present.\n * Returns 0 if any filter flags the n-gram for filtering, or if \n * the sign has no words.\n * Otherwise, sets signToScore, calls prepareToScoreWords, \n * and then returns the result of logprob() converted to a probability.\n */\n public synchronized double score(Sign sign, boolean complete) {\n \treturn convertToProb(logprob(sign, complete));\n }\n \n /** \n * Returns a log prob for the given sign and completeness flag, \n * based on the n-gram log prob of the sign's words.\n * If the sign is complete, sentence delimiters are added before \n * scoring the words, if not already present.\n * Returns the log prob for zero probability if any filter flags the n-gram for filtering, or if \n * the sign has no words.\n * Otherwise, sets signToScore, calls prepareToScoreWords, \n * and then returns the result of logProb().\n */\n public synchronized double logprob(Sign sign, boolean complete) {\n List words = sign.getWords(); \n if (words == null) return 0;\n if (!complete) { // check cache\n Float logprob = getCachedLogProb(words);\n if (logprob != null) return logprob;\n }\n signToScore = sign;\n setWordsToScore(words, complete);\n if (ngramFilters != null) {\n for (int i = 0; i < ngramFilters.size(); i++) {\n NgramFilter filter = ngramFilters.get(i);\n if (filter.filterOut(wordsToScore)) return convertToLogProb(0);\n }\n }\n prepareToScoreWords();\n double retval = logprob();\n signToScore = null;\n return retval;\n }\n \n /**\n * Returns an n-gram probability from the given list of words, \n * by converting the result of the logprob method.\n */\n public synchronized double score(List words) {\n \treturn convertToProb(logprob(words));\n }\n \n /** \n * Returns an n-gram log prob for the given list of words.\n * This method is a simplified version of scoring a sign's words \n * that does not cache results, filter n-grams or ever add \n * sentence delimiters.\n */\n public synchronized double logprob(List words) {\n setWordsToScore(words, false);\n prepareToScoreWords();\n return logprob();\n }\n \n /** Sets wordsToScore to the given list, for sharing purposes. */\n protected void shareWordsToScore(List wordsToScore) {\n this.wordsToScore = wordsToScore;\n }\n \n /**\n\t * Resets wordsToScore to the given ones, reversing them when the reverse\n\t * flag is true, and adding sentence delimiters if not already present, when\n\t * the completeness flag is true. Also sets the tagsAdded flag.\n\t */\n protected void setWordsToScore(List words, boolean complete) {\n wordsToScore.clear();\n tagsAdded = false; \n if (complete && (reverse || words.get(0).getForm() != \"\")) { \n wordsToScore.add(Word.createWord(\"\"));\n tagsAdded = true;\n }\n if (reverse) {\n for (int j = words.size()-1; j >= 0; j--) {\n Word w = words.get(j);\n if (w.getForm() == \"\" || w.getForm() == \"\") continue; // skip or \n wordsToScore.add(w);\n }\n }\n else\n wordsToScore.addAll(words);\n if (complete && (reverse || words.get(words.size()-1).getForm() != \"\")) {\n wordsToScore.add(Word.createWord(\"\"));\n tagsAdded = true;\n }\n }\n \n /** Optional step to do further preparation before scoring words. */\n protected void prepareToScoreWords() {}\n \n \n /** Returns a feature map with counts of each ngram for the given sign and completness flag. */\n public FeatureMap extractFeatureMap(Sign sign, boolean complete) {\n \tFeatureMap featmap = new FeatureMap();\n \t// do setup as with scoring\n List words = sign.getWords(); \n if (words == null) return featmap;\n signToScore = sign;\n setWordsToScore(words, complete);\n prepareToScoreWords();\n // count ngrams \n int numWords = wordsToScore.size();\n for (int k=1; k <= order; k++) {\n int numNgrams = numWords - (k-1);\n if (numNgrams <= 0) continue;\n for (int i = 0; i < numNgrams; i++) {\n incNgrams(featmap, i, k);\n }\n }\n // return\n signToScore = null;\n \treturn featmap;\n }\n \n /**\n * Returns a feature vector with the log prob features \n * for the given sign and completeness flag.\n\t * The default implementation returns the log prob as \n\t * the value of a feature named '$ngram' plus the counts of each ngram.\n */\n public FeatureVector extractLogProbs(Sign sign, boolean complete) {\n \tFeatureList retval = new FeatureList(1);\n\t\tAlphabet.Feature f = alphabet.index(\"$ngram\");\n\t\tif (f != null) retval.add(f, (float)logprob(sign, complete));\n\t\treturn retval;\n }\n \n\t/** \n\t * Returns the features for the given sign and completeness flag.\n\t * The default implementation returns the log prob as \n\t * the value of a feature named '$ngram' plus the counts of each ngram.\n\t */\n\tpublic FeatureVector extractFeatures(Sign sign, boolean complete) {\n\t\tFeatureVector logprob = extractLogProbs(sign, complete);\n\t\tif (useNgramFeatures) {\n\t\t\tFeatureMap map = extractFeatureMap(sign, complete);\n\t\t\treturn new ComposedFeatureVector(logprob, map);\n\t\t}\n\t\telse return logprob;\n\t}\n\t\n\t/**\n\t * Increments ngram counts for the ngrams starting at the given index in\n\t * wordsToScore and with the given order. The default implementation \n\t * uses ngram(i, order).\n\t */\n\tprotected void incNgrams(FeatureMap featmap, int i, int order) {\n List ngram = ngram(i, order); \n if (ngram == null) return;\n Alphabet.Feature f = alphabet.index(ngram);\n\t\tif (f != null) featmap.inc(f);\n\t}\n\t\n\t\n /**\n\t * Returns a list of feature keys for the ngram starting at the given index in\n\t * wordsToScore and with the given order, using the keys in keysList after\n\t * setting them appropriately with setKeysToNgram; returns null if this\n\t * operation does not succeed normally.\n\t */\n protected List ngram(int i, int order) { \n \tboolean ok = setKeysToNgram(i, order);\n \tif (!ok) return null;\n \tfeatureKeysList.clear();\n \tfor (int j=0; j < keysList.size(); j++) {\n \t\tObject key = keysList.get(j);\n \t\tif (!(key instanceof String)) {\n \t\t\tthrow new RuntimeException(\"Feature keys must be strings!\");\n \t\t}\n \t\telse featureKeysList.add((String)key);\n \t}\n \treturn featureKeysList;\n }\n \n\t\n /** \n * Returns a log prob for the words in wordsToScore.\n * The default method returns the log prob of the word sequence \n * as determined by this language model's logProbFromNgram method.\n * The probabilities for the first n-1 words are backed off to the \n * lower order probabilities.\n * If the tagsAdded flag is false, the cache is checked to see whether \n * the log prob of the words of signToScore's initial sign has already \n * been calculated, and at the end the log prob of signToScore's words \n * is stored in the cache.\n */\n protected double logprob() {\n float logProbTotal = 0;\n int numCached = 0;\n if (!tagsAdded && signToScore != null) { // check cache for initial words\n Sign[] inputs = signToScore.getDerivationHistory().getInputs();\n if (inputs != null) {\n Sign initialSign = (!reverse) ? inputs[0] : inputs[inputs.length-1];\n List initialWords = initialSign.getWords();\n Float logprob = getCachedLogProb(initialWords);\n if (logprob != null) {\n logProbTotal = logprob.floatValue();\n numCached = initialWords.size();\n }\n }\n }\n for (int i = numCached; i < wordsToScore.size(); i++) {\n int orderToUse = Math.min(order, i+1);\n int startPos = i - (orderToUse-1);\n logProbTotal += logProbFromNgram(startPos, orderToUse);\n }\n if (!tagsAdded && signToScore != null) { // add log prob to cache\n putCachedLogProb(signToScore.getWords(), new Float(logProbTotal));\n }\n return logProbTotal;\n }\n \n \n /**\n\t * Returns the log prob of the ngram starting at the given index in\n\t * wordsToScore and with the given order, with backoff.\n\t */\n abstract protected float logProbFromNgram(int i, int order);\n \n /**\n\t * Sets the keys in keysList to hold the ngram starting at the given index in\n\t * wordsToScore and with the given order; returns true if the operation \n\t * succeeds normally. The default implementation invokes\n\t * logProbFromNgram, and returns false if the log prob is zero.\n\t */\n\tprotected boolean setKeysToNgram(int i, int order) {\n\t\tfloat logprob = logProbFromNgram(i, order);\n\t\treturn logprob != 0;\n\t}\n\n \n\t/** Flag for using ngrams as features. */\n\tprotected boolean useNgramFeatures = true;\n\t\n /** Sets the the flag for using ngrams as features. */\n\tpublic void setNgramFeatures(boolean useNgramFeatures) { \n\t\tthis.useNgramFeatures = useNgramFeatures; \n\t}\n\n\t\n /**\n * Flag whether to use sem classes in place of words.\n * Defaults to false.\n */\n protected boolean useSemClasses = false;\n \n // tokenizer reference\n private Tokenizer tokenizer = null;\n\n private Tokenizer getTokenizer() {\n if (tokenizer != null) return tokenizer;\n if (Grammar.theGrammar != null) tokenizer = Grammar.theGrammar.lexicon.tokenizer;\n else tokenizer = new DefaultTokenizer();\n return tokenizer;\n }\n \n /** Returns whether the given semantic class is a replacement one. */\n protected boolean isReplacementSemClass(String semClass) {\n return semClass != null && getTokenizer().isReplacementSemClass(semClass);\n }\n \n /**\n\t * Returns the semantic class replacement value (the semantic class\n\t * uppercased and interned) for the given word, if apropos, otherwise null.\n\t */\n protected String semClassReplacement(Word w) {\n if (useSemClasses) {\n String semClass = w.getSemClass();\n if (isReplacementSemClass(semClass)) \n return semClass.toUpperCase().intern();\n }\n // otherwise null\n return null;\n }\n \n \n /**\n\t * Adds the TrieMap children, with their keys, under the given prefix, then\n\t * resets the lists.\n\t */\n protected void addTrieMapChildren(List prefix, List keys, List> children) {\n if (!keys.isEmpty()) {\n TrieMap prefixNode = trieMapRoot.findChildFromList(prefix);\n prefixNode.addChildren(keys, children);\n }\n prefix.clear(); keys.clear(); children.clear();\n }\n \n /** Returns the TrieMap node for the given sublist of keysList. */ \n protected TrieMap getNode(int pos, int len) {\n return trieMapRoot.getChildFromList(keysList.subList(pos, pos+len));\n }\n \n \n // from CMU-Cambridge Statistical Language Modeling Toolkit\n //\n // p(wd3|wd1,wd2)= if(trigram exists) p_3(wd1,wd2,wd3)\n // else if(bigram w1,w2 exists) bo_wt_2(w1,w2)*p(wd3|wd2)\n // else p(wd3|w2)\n // \n // p(wd2|wd1)= if(bigram exists) p_2(wd1,wd2)\n // else bo_wt_1(wd1)*p_1(wd2)\n \n /**\n\t * Returns the log prob (base 10) of the given sublist of keysList, with\n\t * backoff, or -99 if not found.\n\t */\n protected float logProb(int pos, int len) {\n TrieMap node = getNode(pos, len);\n if (node != null && node.data != null) return node.data.logprob;\n if (len == 1) return -99;\n float retval = logProb(pos+1, len-1);\n if (debugScore) System.out.print(\"(\" + (len-1) + \"-gram: \" + retval + \") \");\n if (retval > -99) retval += backoffWeight(pos, len-1);\n return retval;\n }\n \n /**\n\t * Returns the back-off weight (log base 10) of the given sublist of\n\t * keysList, or 0 if not found.\n\t */\n protected float backoffWeight(int pos, int len) {\n TrieMap node = getNode(pos, len);\n if (node != null && node.data != null) {\n float retval = node.data.bow;\n // if (debugScore && retval != 0) System.out.print(\"(bow: \" + retval + \") \");\n return retval;\n }\n return 0;\n }\n\n\n\n\t/**\n * Returns the rank order centroid weights for a ranked list of the given length. \n * The weights go from highest to lowest, and sum to 1.\n */\n // ex:\n // weight 1 0.5208333333333333\n // weight 2 0.2708333333333333\n // weight 3 0.14583333333333331\n // weight 4 0.0625\n public static double[] rankOrderCentroidWeights(int length) {\n double[] retval = new double[length];\n for (int i = 0; i < length; i++) {\n double weight_i = 0;\n for (int j = i; j < length; j++) {\n weight_i += 1 / (double) (j+1);\n }\n weight_i = weight_i / (double) length;\n retval[i] = weight_i;\n }\n return retval;\n }\n \n \n /** Converts a base 10 log prob to an actual probability, checking for -99 (not found). */\n public static double convertToProb(double logProb) {\n if (logProb <= -99) { return 0; }\n else return Math.pow(10, logProb);\n }\n \n /** Converts a probability to a base 10 log prob, returning -99 if zero. */\n public static double convertToLogProb(double prob) {\n if (prob == 0) return -99;\n else return Math.log(prob) / Math.log(10);\n }\n \n /** Converts a base 10 log prob to the corresponding perplexity. */\n public static double convertToPPL(double logProb) {\n return Math.exp(- logProb * Math.log(10));\n }\n\n \n /** Sets up tokenizer for reading in language models. */ \n protected static StreamTokenizer initTokenizer(Reader in) {\n StreamTokenizer tokenizer = new StreamTokenizer(in);\n tokenizer.resetSyntax();\n tokenizer.wordChars(0,255);\n tokenizer.whitespaceChars(' ',' ');\n tokenizer.whitespaceChars('\\t','\\t');\n tokenizer.whitespaceChars('\\n','\\n');\n tokenizer.whitespaceChars('\\r','\\r');\n tokenizer.eolIsSignificant(true);\n return tokenizer;\n }\n \n /**\n\t * Reads a line of up to tokens.length tokens using the given tokenizer,\n\t * with the remaining array elements set to null.\n\t */\n protected static void readLine(StreamTokenizer tokenizer, String[] tokens) throws IOException {\n int index = 0;\n int ttype;\n while ( (ttype = tokenizer.nextToken()) != StreamTokenizer.TT_EOF && ttype != StreamTokenizer.TT_EOL ) {\n if (index < tokens.length && ttype == StreamTokenizer.TT_WORD) { \n tokens[index] = tokenizer.sval;\n index++;\n }\n }\n for (int i = index; i < tokens.length; i++) {\n tokens[i] = null;\n }\n }\n}\n\n" }, { "path": "src/opennlp/ccg/ngrams/RepetitionScorer.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.ngrams;\n\nimport opennlp.ccg.synsem.Sign;\nimport opennlp.ccg.synsem.SignScorer;\nimport opennlp.ccg.lexicon.Word;\n\nimport java.util.*;\nimport gnu.trove.*;\n\n/**\n * Scores a sign according to how repetitive its words are given the \n * observed context. Relevant repeated items (eg stems) are counted, \n * with full counts given to items in the previous words or recent context, \n * and fractional counts to older items. The score is then assigned according \n * to the number of repeated items and the configured penalty, as \n * 10 to the minus (penalty times repeated items).\n *\n * @author Michael White\n * @version $Revision: 1.6 $, $Date: 2011/03/20 20:11:58 $\n */\n@SuppressWarnings({\"unchecked\",\"rawtypes\"})\npublic class RepetitionScorer implements SignScorer\n{\n /** The repetition penalty (defaults to 1.0). */\n public double penalty = 1.0;\n\n /** The fractional count for the older items (defaults to 0.5). */\n public double olderCount = 0.5;\n \n /** The fractional count for the even older items (defaults to 0.25). */\n public double evenOlderCount = 0.25;\n \n /** The fractional count for the oldest items (defaults to 0.125). */\n public double oldestCount = 0.125;\n \n /** The interned POS values to use for repetition scoring purposes. */\n protected Set posValsToUse = new THashSet(new TObjectIdentityHashingStrategy());\n \n /** The interned stems to ignore for repetition scoring purposes. */\n protected Set stemsToIgnore = new THashSet(new TObjectIdentityHashingStrategy());\n \n /** The interned items (eg stems) seen in the previous words. */\n protected Set previousItems = new THashSet(new TObjectIdentityHashingStrategy());\n \n /** The interned items (eg stems) seen in the recent context. */\n protected Set contextItems = new THashSet(new TObjectIdentityHashingStrategy());\n \n /** The interned items (eg stems) seen in the older context. */\n protected Set olderContextItems = new THashSet(new TObjectIdentityHashingStrategy());\n \n /** The interned items (eg stems) seen in the even older context. */\n protected Set evenOlderContextItems = new THashSet(new TObjectIdentityHashingStrategy());\n \n /** The interned items (eg stems) seen in the oldest context. */\n protected Set oldestContextItems = new THashSet(new TObjectIdentityHashingStrategy());\n \n /**\n * Default constructor.\n * Adds \"NNP\", \"N\", \"V\", \"Adj\" and \"Adv\" to posValsToUse, \n * and \"do\" and \"not\" to stemsToIgnore.\n */\n public RepetitionScorer() {\n String[] posVals = { \"NNP\", \"N\", \"V\", \"Adj\", \"Adv\" };\n posValsToUse.addAll(Arrays.asList(posVals));\n String[] stems = { \"do\", \"not\" };\n stemsToIgnore.addAll(Arrays.asList(stems));\n }\n \n /** Resets all the context items. */\n public void resetContext() { \n contextItems.clear(); olderContextItems.clear(); \n evenOlderContextItems.clear(); oldestContextItems.clear();\n }\n \n /** Ages the context items, clearing the recent ones. */\n\tpublic void ageContext() { \n oldestContextItems.clear(); oldestContextItems.addAll(evenOlderContextItems);\n evenOlderContextItems.clear(); evenOlderContextItems.addAll(olderContextItems);\n olderContextItems.clear(); olderContextItems.addAll(contextItems);\n contextItems.clear(); \n }\n \n /** Adds the items (eg stems) from the given sign's words to the context items. */\n public void updateContext(Sign sign) {\n List words = sign.getWords(); \n if (words == null) return;\n for (int i = 0; i < words.size(); i++) {\n Word word = (Word) words.get(i);\n updateItems(word, contextItems);\n }\n }\n \n /** \n * Adds the items (eg stems) from the given word to the given set.\n * By default, adds the relevant stems, per the relevantStem method.\n */\n protected void updateItems(Word word, Set set) {\n String stem = relevantStem(word);\n if (stem != null) set.add(stem);\n }\n\n /**\n * Returns the stem of the given word if its POS is in posValsToUse, \n * unless the stem is in stemsToIgnore; otherwise returns null.\n */\n protected String relevantStem(Word word) {\n if (!(posValsToUse.contains(word.getPOS()))) return null;\n String stem = word.getStem();\n if (!(stemsToIgnore.contains(stem))) return stem;\n return null;\n }\n \n /** \n * Returns a score between 0 (worst) and 1 (best) for the given sign \n * and completeness flag, according to how repetitive its word are compared to \n * the observed context. \n * In particular, returns 10 to the minus (penalty times repeated items), \n * or zero if there are no words.\n */\n public double score(Sign sign, boolean complete) {\n List words = sign.getWords(); \n if (words == null) return 0;\n return Math.pow(10, -1 * penalty * repeatedItems(words));\n }\n \n /** \n * Returns the number of repeated items (eg stems) in the given word list, \n * using fractional counts for repetitions of older items.\n * The previous items set is cleared, and then the repeated items \n * are summed for each word, updating the previous items along the way.\n */\n protected double repeatedItems(List words) {\n previousItems.clear();\n double retval = 0;\n for (int i = 0; i < words.size(); i++) {\n Word word = (Word) words.get(i);\n retval += repeatedItems(word);\n updateItems(word, previousItems);\n }\n return retval;\n }\n \n /** \n * Returns the number of repeated items (eg stems) in the given word, \n * using fractional counts for repetitions of older items.\n * By default, returns 1 (or a fractional count) if the stem is relevant, \n * per the relevantStem method. \n */\n protected double repeatedItems(Word word) {\n String stem = relevantStem(word);\n if (stem == null) return 0;\n if (contextItems.contains(stem) || previousItems.contains(stem)) return 1;\n if (olderContextItems.contains(stem)) return olderCount;\n if (evenOlderContextItems.contains(stem)) return evenOlderCount;\n if (oldestContextItems.contains(stem)) return oldestCount;\n return 0;\n }\n}\n\n" }, { "path": "src/opennlp/ccg/ngrams/Reversible.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.ngrams;\n\n\n/**\n * Interface for reversible n-gram classes.\n *\n * @author Michael White\n * @version $Revision: 1.1 $, $Date: 2005/05/26 22:18:05 $\n */\npublic interface Reversible\n{\n /** Get reverse flag. */\n public boolean getReverse();\n \n /** Set reverse flag. */\n public void setReverse(boolean reverse);\n}\n\n" }, { "path": "src/opennlp/ccg/ngrams/SRILMNgramModel.java", "content": "/*\n * $Id: SRILMNgramModel.java,v 1.5 2008/11/09 03:29:36 mwhite14850 Exp $ \n */\npackage opennlp.ccg.ngrams;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.util.List;\nimport java.util.ListIterator;\n\nimport opennlp.ccg.lexicon.DefaultTokenizer;\nimport opennlp.ccg.lexicon.Tokenizer;\nimport opennlp.ccg.lexicon.Word;\n\n\n/**\n * A language model that uses the\n * SRI language modeling\n * toolkit.\n * @author Scott Martin\n * @see SRILM\n * @version $Revision: 1.5 $\n * @since 0.9.2\n */\npublic class SRILMNgramModel extends AbstractStandardNgramModel {\n\n\t/**\n\t * Load the binary, platform-dependent library containing the SRILM JNI\n\t * bridge code. See ${OPENCCG_HOME}/src/srilmbridge.\n\t * @throws UnsatisfiedLinkError If Java can't find the srilmbridge library.\n\t */\n\tstatic {\n\t\tSystem.loadLibrary(\"srilmbridge\");\n\t}\n\t\n\t/**\n\t * Creates a SRILM language model with the specified ngram order and model\n\t * type.\n\t * @param order The ngram order to use.\n\t * @param lmFile The file to read the langauge model from.\n\t * @param useSemClasses Whether or not to use semantic classes.\n\t * @param modelType The type of language model.\n\t * @throws IOException If a problem occurs reading the language model file.\n\t * These include non-existent or unreadable files, file format problems,\n\t * etc. \n\t */\n\tpublic SRILMNgramModel(int order, File lmFile,\n\t\tboolean useSemClasses, SRILMNgramModelType modelType)\n\t\t\tthrows IOException {\n\t\tsuper(order, useSemClasses);\n\t\tloadLMFromFile(order, lmFile, modelType);\n\t}\n\n\t/**\n\t * Creates a new SRILM language model.\n\t * @see SRILMNgramModel#SRILMNgramModel(int, File, boolean,\n\t * SRILMNgramModelType)\n\t */\n\tpublic SRILMNgramModel(int order, File lmFile,\n\t\tSRILMNgramModelType modelType)\n\t\t\tthrows IOException {\n\t\tthis(order, lmFile, false, modelType);\n\t}\n\t\n\t/**\n\t * Loads an LM from a file.\n\t * @param ngramOrder The ngram order to use.\n\t * @param lmFile The file containing the language model.\n\t * @param lmType The type of langauge model to expect.\n\t * @throws IOException If the language model file is non-existent or\n\t * null, or if a problem occurs loading or parsing the file.\n\t */\n\tprotected void loadLMFromFile(int ngramOrder, File lmFile, \n\t\t\t\tSRILMNgramModelType lmType)\n\t\t\tthrows IOException {\n\t\tif(lmFile == null) {\n\t\t\tthrow new IOException(\"null file\");\n\t\t}\n\t\tif(!lmFile.exists()) {\n\t\t\tthrow new IOException(\"file does not exist: \" + lmFile);\n\t\t}\n\t\tif(lmFile.isDirectory()) {\n\t\t\tthrow new IOException(\"file is a directory: \" + lmFile);\n\t\t}\n\t\tif(!lmFile.canRead()) {\n\t\t\tthrow new IOException(\"unable to read file: \" + lmFile);\n\t\t}\n\t\t\n\t\tloadLM(ngramOrder, lmFile.getAbsolutePath(), lmType.ordinal());\n\t}\n\n\t/**\n\t * Calculates a log probability of a delineated substring of the strings\n\t * to score using SRILM. This method reverses the context before passing\n\t * the string to SRILM, as this is the format SRILM expects.\n\t * @param pos The start position (inclusive) within the strings to score.\n\t * @param len The length, starting from pos, of the string\n\t * that should be used. \n\t */\n\t@Override\n\tpublic float logProb(int pos, int len) {\n\t\ttry {\n\t\t\t// create new because reversing list affects keysList\n\t\t\tList range = keysList.subList(pos, pos + len);\n\t\t\tint rangeSize = range.size();\n\t\t\tif(rangeSize == 0) {\n\t\t\t\tthrow new IllegalArgumentException(\n\t\t\t\t\t\t\"empty range specified for log prob\");\n\t\t\t}\n\t\t\t\n\t\t\t// only allocate context array if we have to\n\t\t\tString[] context = (rangeSize > 1)\n\t\t\t\t? new String[rangeSize - 1] : null;\n\t\t\t\n\t\t\tif(context != null) { // reverse for SRILM\t\t\t\t\n\t\t\t\tListIterator contextIterator\n\t\t\t\t\t= range.listIterator(rangeSize - 1);\n\t\t\t\tint i = 0;\n\t\t\t\twhile(contextIterator.hasPrevious()) {\n\t\t\t\t\tcontext[i++] = contextIterator.previous().toString();\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\t// call SRILM to get word in reversed context\n\t\t\treturn doLogProb(range.get(rangeSize - 1).toString(), context);\n\t\t}\n\t\tcatch(IndexOutOfBoundsException e) {\n\t\t\treturn 0.0f;\n\t\t}\n\t}\n\n\t/**\n\t * Invokes SRILM to load a language model.\n\t * @param ngramOrder The order of the language model\n\t * @param fileAbsolutePath The absolute path of the file containing the\n\t * language model.\n\t * @param lmType The language model type.\n\t * @throws IOException If a problem happens with SRILM while trying to\n\t * load the language model.\n\t */\n\tprivate native void loadLM(int ngramOrder, String fileAbsolutePath,\n\t\t\tint lmType)\n\t\tthrows IOException;\n\n\t/**\n\t * Invokes SRILM to calculate the log probability of a string in the\n\t * given context. SRILM will make its calculations based on the language\n\t * model loaded in {@link #loadLM(int, String, int)}.\n\t * @param word The word to calculate a probability for.\n\t * @param context The context, in reverse order. For example, to calculate\n\t * the probability of the word "rain" in the context of the\n\t * string "in the rain", the context should be the array\n\t * {the, in}. If the context is null or\n\t * zero-length, SRILM will assume this means no context should be used.\n\t * @return The log probability of the given word in the given (reversed)\n\t * context, as determined by SRILM.\n\t */\n\tprivate native float doLogProb(String word, String[] context);\n\n\t/**\n\t * Invokes SRILM to clean up any initialized objects.\n\t */\n\t@Override\n\tprotected native void finalize() throws Throwable;\n\t\n\tpublic static void main(String[] args) throws Exception {\n\t\tString usage = \"Usage: java opennlp.ccg.ngrams.SRILMNgramModel\"\n\t\t\t+ \" (-reverse)\";\n \n if (args.length > 0 && args[0].equals(\"-h\")) {\n System.out.println(usage);\n return;\n }\n \n long start = System.currentTimeMillis();\n String order = args[0];\n String lmfile = args[1];\n String lmType = args[2];\n String tokens = args[3];\n String reversed = (args.length >= 5 && args[4].equals(\"-reverse\"))\n \t? \"reversed \" : \"\";\n System.out.println(\"Loading \" + reversed\n \t\t+ \"n-gram model with order \" + order + \" from: \" + lmfile);\n SRILMNgramModel lm = new SRILMNgramModel(Integer.parseInt(order),\n \t\tnew File(lmfile), SRILMNgramModelType.valueOf(lmType));\n if (reversed.length() > 0) lm.setReverse(true);\n System.out.println(\"openVocab: \" + lm.openVocab);\n int secs = (int) (System.currentTimeMillis() - start) / 1000;\n System.out.println(\"secs: \" + secs);\n System.out.println();\n // System.out.println(\"trie map: \");\n // System.out.println(lm.trieMapRoot.toString());\n // System.out.println();\n \n Tokenizer tokenizer = new DefaultTokenizer();\n List words = tokenizer.tokenize(tokens);\n System.out.println(\"scoring: \" + tokens);\n System.out.println();\n lm.debugScore = true;\n lm.setWordsToScore(words, true);\n lm.prepareToScoreWords();\n double logprob = lm.logprob();\n double score = convertToProb(logprob);\n System.out.println();\n System.out.println(\"score: \" + score);\n System.out.println(\"logprob: \" + logprob);\n System.out.println(\"ppl: \" + NgramScorer.convertToPPL(logprob / (words.size()-1)));\n\t}\n}\n" }, { "path": "src/opennlp/ccg/ngrams/SRILMNgramModelType.java", "content": "/*\n * $Id: SRILMNgramModelType.java,v 1.2 2007/05/30 22:53:17 coffeeblack Exp $ \n */\npackage opennlp.ccg.ngrams;\n\n\n/**\n * Used by {@link SRILMNgramModel} to specify the type of language model that\n * should be used.\n * @author Scott Martin\n * @see SRILM\n * @version $LastChangedRevision$\n */\npublic enum SRILMNgramModelType {\n\t/*!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!*\n\t * It is very important that the order of these does not *\n\t * change, as {@link SRILMNgramModel#loadLM(int, String, int)}*\n\t * relies on the ordinal. *\n\t *!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!*/\n\t\n\t/**\n\t * A "standard" ngram model, of the type normally created by\n\t * the SRILM binary ngram-count.\n\t */\n\tSTANDARD,\n\t\n\t/**\n\t * For ngram models based on count LMs. The Google LM format is one of\n\t * these.\n\t * @see Web 1T 5-gram Version 1\n\t */\n\tCOUNT;\n}\n" }, { "path": "src/opennlp/ccg/ngrams/SRILM_FactoredScorerMaker.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.ngrams;\n\nimport opennlp.ccg.synsem.SignScorer;\n\nimport java.io.*;\n\n/**\n * A custom scorer maker that builds and loads factored n-gram models using the \n * SRILM toolkit, which must be separately installed.\n * Most parameters are set in the FLM spec file.\n *\n * @author Michael White\n * @version $Revision: 1.6 $, $Date: 2007/12/21 05:13:37 $\n */\npublic class SRILM_FactoredScorerMaker extends SRILM_ScorerMaker {\n \n /** The base of the model FLM spec file name. Defaults to \"test.flm\". */\n public String flmSpecFileBase = \"test.flm\";\n \n \n /** Returns the model FLM spec filename, extending flmSpecFileBase with the discount options string and \n n-gram order, eg \"test.flm.n4\". */\n protected String flmSpecFilename() {\n return flmSpecFileBase + \".\" + discountOptionsStr + getOrder();\n }\n \n /** Returns the root of the perplexities filename. */\n protected String pplFileRoot() { return flmSpecFilename(); }\n \n /** Creates fold-specific FLM spec filenames from fold numbers, eg \"test.flm.n4.fold1\". */\n protected String filename(int foldNum) {\n return flmSpecFilename() + \".\" + \"fold\" + foldNum;\n }\n \n \n /** Writes a fold-specific FLM spec file, given the tmp dir and fold num.\n The fold-specific file is created by simply replacing \".count\" with \".foldN.count\" \n and \".lm\" with \".foldN.lm\", where N is the fold num, in the model FLM spec file. */\n protected void writeFoldSpecFile(File tmpDir, int foldNum) throws IOException {\n BufferedReader br = new BufferedReader(new FileReader(flmSpecFilename()));\n File foldSpecFile = new File(tmpDir, filename(foldNum));\n PrintWriter out = new PrintWriter(new FileWriter(foldSpecFile));\n String dotFoldN = \".fold\" + foldNum;\n String line = null;\n while ((line = br.readLine()) != null) {\n int countIndex = line.indexOf(\".count\");\n if (countIndex > 0) {\n String foldLine = line.substring(0, countIndex);\n foldLine += dotFoldN;\n int lmIndex = line.indexOf(\".lm\", countIndex);\n foldLine += line.substring(countIndex, lmIndex);\n foldLine += dotFoldN;\n foldLine += line.substring(lmIndex);\n out.println(foldLine);\n }\n else {\n out.println(line);\n }\n }\n out.close();\n br.close();\n }\n \n /**\n * Prepares a scoring model from the training data, \n * by exec-ing the SRILM fngram-count tool with the FLM spec file, \n * and computes perplexity on the test data.\n * The training/test data are written to foldN-train.txt and foldN-test.txt, \n * if not already present.\n * The fold-specific FLM spec file's name is determined by filename(N).\n */\n public void prepScorer(File tmpDir, int foldNum, File trainFile, File testFile) throws IOException {\n // write fold spec file\n writeFoldSpecFile(tmpDir, foldNum);\n // do rest much like standard n-gram scorers\n super.prepScorer(tmpDir, foldNum, trainFile, testFile);\n }\n \n /** Writes training/test targets. */\n protected void writeTargets(File tbFile, String textfile) throws IOException {\n if (!useSemClasses) \n cvr.tester.writeTargetsF(tbFile, textfile); \n else \n cvr.tester.writeTargetsFSC(tbFile, textfile); \n }\n \n /** Returns the command for making an ngram model. */\n protected String countNgrams(int foldNum) {\n String cmd = \"fngram-count -nonull -write-counts -lm \" +\n ((unk) ? \"-unk \" : \"\") + \n \"-factor-file \" + filename(foldNum) + \" \" + \n \"-text \" + trainingfile(foldNum) + \" \" + \n \"-debug \" + debugLevel;\n return cmd;\n }\n \n /** Returns the command for calculating perplexity. \n NB: At present, only the perplexity from the first model is collected. */\n protected String scoreNgrams(int foldNum) {\n String lmfile = filename(foldNum);\n String cmd2 = \"fngram -nonull \" + \n ((unk) ? \"-unk \" : \"\") + \n \"-factor-file \" + lmfile + \" \" + \n \"-ppl \" + testfile(foldNum);\n return cmd2;\n }\n \n \n /**\n * Loads a scoring model created from the training data. \n */\n public SignScorer loadScorer(File tmpDir, int foldNum, File trainFile) throws IOException {\n File foldSpecFile = new File(tmpDir, filename(foldNum));\n String foldSpecPath = foldSpecFile.getCanonicalPath();\n return new FactoredNgramModelFamily(foldSpecPath, useSemClasses);\n }\n}\n\n\n" }, { "path": "src/opennlp/ccg/ngrams/SRILM_ScorerMaker.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.ngrams;\n\nimport opennlp.ccg.synsem.SignScorer;\nimport opennlp.ccg.test.*;\nimport java.io.*;\nimport java.util.*;\n\n/**\n * A custom scorer maker that builds and loads standard n-gram models using the \n * SRILM toolkit, which must be separately installed.\n * This class may be subclassed to set different parameters in the constructor, \n * for use in cross-validation tests with the realizer.\n *\n * @author Michael White\n * @version $Revision: 1.10 $, $Date: 2007/12/21 05:13:37 $\n */\npublic class SRILM_ScorerMaker implements ScorerMaker {\n \n /** Flag specifying whether to use semantic class replacement. */\n public boolean useSemClasses = true;\n \n /** Flag specifying whether to keep <unk> in the LM. */\n public boolean unk = true;\n \n /** String specifying the min counts to use in building the n-gram model. */\n public String minCountOptions = \"-gt1min 1 -gt2min 1 -gt3min 1 -gt4min 1 -gt5min 1 -gt6min 1\";\n \n /** String specifying the discounting parameters. */\n public String discountOptions = N_DISCOUNT_PARAMS;\n \n /** String specifying natural discounting parameters. */\n public static final String N_DISCOUNT_PARAMS = \"-ndiscount1 -ndiscount2 -ndiscount3 -ndiscount4 -ndiscount5 -ndiscount6\";\n\n /** String specifying Witten-Bell discounting parameters. */\n public static final String WB_DISCOUNT_PARAMS = \"-wbdiscount1 -wbdiscount2 -wbdiscount3 -wbdiscount4 -wbdiscount5 -wbdiscount6\";\n \n /** String specifying modified Kneser-Ney natural discounting parameters. */\n public static final String KN_DISCOUNT_PARAMS = \"-kndiscount1 -kndiscount2 -kndiscount3 -kndiscount4 -kndiscount5 -kndiscount6\";\n \n /** String indicating the discounting option in filenames. */\n public String discountOptionsStr = \"n\"; \n \n /** Debug level to use. */\n public int debugLevel = 1;\n \n \n /** The context for this scorer maker. */\n public CrossValidateRealizer cvr = null;\n \n /** Sets the context for this scorer maker. */\n public void setCVR(CrossValidateRealizer cvr) { this.cvr = cvr; }\n \n \n /** Stores perplexities for each fold, after calls to prepScorer. */\n protected List perplexities = new ArrayList();\n \n /** Gets the n-gram order from the context, defaulting to 3. */\n protected int getOrder() {\n int order = cvr.tester.ngramOrder;\n return (order > 0) ? order : 3;\n }\n \n /** Creates LM filenames from fold numbers. */\n protected String filename(int foldNum) {\n String retval = \"fold\" + foldNum + \"-\" + discountOptionsStr; \n if (useSemClasses) retval += \"-sc\";\n retval += \".\" + getOrder() + \"bo\";\n return retval;\n }\n \n /**\n * Prepares a scoring model from the training data, \n * by exec-ing the SRILM ngram-count tool with the current options, \n * and computes perplexity on the test data.\n * The training/test data are written to foldN-train.txt and foldN-test.txt, \n * if not already present.\n * The LM file's name is determined by filename(N).\n */\n public void prepScorer(File tmpDir, int foldNum, File trainFile, File testFile) throws IOException {\n // write training/test files, if not already present\n writeTrainingAndTestFiles(tmpDir, foldNum, trainFile, testFile);\n // make counting command\n String cmd = countNgrams(foldNum);\n // exec command\n System.out.print(\"Writing \" + getOrder() + \"-gram model: \" + filename(foldNum) + \"\\n\");\n Process makeLM = Runtime.getRuntime().exec(cmd, null, tmpDir);\n try {\n InputStream istr = makeLM.getErrorStream(); //.getInputStream();\n int b;\n while ((b = istr.read()) != -1) { System.out.print((char)b); }\n int exitval = makeLM.waitFor();\n if (exitval != 0) {\n System.out.println(\"(f)ngram-count exitval: \" + exitval);\n }\n }\n catch (InterruptedException exc) {\n throw (RuntimeException) new RuntimeException().initCause(exc);\n }\n System.out.println();\n // make scoring command, for perplexity\n String cmd2 = scoreNgrams(foldNum);\n // exec command\n System.out.print(\"Measuring perplexity with: \" + filename(foldNum) + \"\\n\");\n Process measurePPL = Runtime.getRuntime().exec(cmd2, null, tmpDir);\n try {\n InputStream istr = measurePPL.getInputStream();\n int b;\n StringBuffer sb = new StringBuffer();\n while ((b = istr.read()) != -1) { \n System.out.print((char)b); sb.append((char)b); \n }\n int exitval = measurePPL.waitFor();\n // extract perplexity following \"ppl= \"\n String pplOut = sb.toString();\n int pplStart = pplOut.indexOf(\"ppl= \") + \"ppl= \".length();\n int pplEnd = pplOut.indexOf(\" \", pplStart);\n String pplStr = pplOut.substring(pplStart, pplEnd);\n try {\n perplexities.add(new Double(pplStr));\n }\n catch (NumberFormatException exc) { \n System.out.println(\"Warning, unable to extract perplexity from: \" + pplStr);\n System.out.println(exc.toString());\n }\n if (exitval != 0) {\n System.out.println(\"(f)ngram exitval: \" + exitval);\n }\n }\n catch (InterruptedException exc) {\n throw (RuntimeException) new RuntimeException().initCause(exc);\n }\n System.out.println();\n }\n \n /** Writes the training and test files, if not already present. */\n protected void writeTrainingAndTestFiles(File tmpDir, int foldNum, File trainFile, File testFile) throws IOException {\n File trainingFoldFile = new File(tmpDir, trainingfile(foldNum)); \n if (!trainingFoldFile.exists()) {\n String trainingFoldPath = trainingFoldFile.getCanonicalPath();\n writeTargets(trainFile, trainingFoldPath);\n }\n File testFoldFile = new File(tmpDir, testfile(foldNum)); \n if (!testFoldFile.exists()) {\n String testFoldPath = testFoldFile.getCanonicalPath();\n writeTargets(testFile, testFoldPath); \n }\n }\n \n /** Returns the name of the training file. */\n protected String trainingfile(int foldNum) { return \"fold\" + foldNum + \"-train.txt\"; }\n \n /** Returns the name of the test file. */\n protected String testfile(int foldNum) { return \"fold\" + foldNum + \"-test.txt\"; }\n \n /** Writes training/test targets. */\n protected void writeTargets(File tbFile, String textfile) throws IOException {\n if (!useSemClasses) \n cvr.tester.writeTargets(tbFile, textfile); \n else \n cvr.tester.writeTargetsSC(tbFile, textfile); \n }\n\n /** Returns the command for making an ngram model. */\n protected String countNgrams(int foldNum) {\n String lmfile = filename(foldNum);\n String cmd = \"ngram-count -order \" + getOrder() + \" \" + \n ((unk) ? \"-unk \" : \"\") + \n minCountOptions + \" \" + \n discountOptions + \" \" + \n \"-text \" + trainingfile(foldNum) + \" \" + \n \"-lm \" + lmfile + \" \" + \n \"-debug \" + debugLevel;\n return cmd;\n }\n \n /** Returns the command for calculating perplexity. */\n protected String scoreNgrams(int foldNum) {\n String lmfile = filename(foldNum);\n String cmd2 = \"ngram -order \" + getOrder() + \" \" + \n ((unk) ? \"-unk \" : \"\") + \n \"-ppl \" + testfile(foldNum) + \" \" + \n \"-lm \" + lmfile;\n return cmd2;\n }\n \n /** Returns the root of the perplexities filename. */\n protected String pplFileRoot() {\n return discountOptionsStr + getOrder();\n }\n \n /**\n * Summarizes perplexities after all calls to prepScorer.\n */\n public void prepScorersSummary(File tmpDir) throws IOException {\n // summarize to sysout\n double sum = 0;\n System.out.print(\"Perplexities: \");\n for (int i = 0; i < perplexities.size(); i++) {\n double ppl = perplexities.get(i).doubleValue();\n System.out.print(ppl + \" \");\n sum += ppl;\n }\n System.out.println();\n double avg = sum / perplexities.size();\n System.out.println(\"Avg: \" + avg);\n // then to xml\n String filename = \"ppl\"; \n if (useSemClasses) filename += \"-sc\";\n filename += \".\" + pplFileRoot() + \".xml\";\n System.out.println(\"Writing perplexities: \" + filename);\n PrintWriter pw = new PrintWriter(new FileWriter(new File(tmpDir, filename)));\n pw.println(\"\");\n for (int i = 0; i < perplexities.size(); i++) {\n pw.println(\" \");\n }\n pw.println(\"\");\n pw.close();\n }\n \n \n /**\n * Loads a scoring model created from the training data. \n */\n public SignScorer loadScorer(File tmpDir, int foldNum, File trainFile) throws IOException {\n String lmfile = filename(foldNum);\n String lmPath = new File(tmpDir, lmfile).getCanonicalPath();\n return new StandardNgramModel(getOrder(), lmPath, useSemClasses);\n }\n}\n\n\n" }, { "path": "src/opennlp/ccg/ngrams/SelfParaphraseBiaser.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2011 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.ngrams;\r\n\r\n/**\r\n * Interface for scoring models that implement a self-paraphrase bias.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.1 $, $Date: 2011/05/15 20:35:06 $\r\n */\r\npublic interface SelfParaphraseBiaser {\r\n\t/** Sets the target strings for implementing the self-paraphrase bias. */\r\n\tpublic void setTargets(String[] targets);\r\n}\r\n" }, { "path": "src/opennlp/ccg/ngrams/SignScorerInterpolation.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.ngrams;\n\nimport opennlp.ccg.perceptron.*;\nimport opennlp.ccg.synsem.Sign;\nimport opennlp.ccg.synsem.SignScorer;\n\n/**\n * Linear interpolation of sign scorers, some of which may be feature extractors.\n *\n * @author Michael White\n * @version $Revision: 1.3 $, $Date: 2011/01/15 17:52:59 $\n */\npublic class SignScorerInterpolation implements SignScorer, FeatureExtractor\n{\n /** The component models. */\n protected SignScorer[] models;\n \n /** The weights. */\n protected double weights[];\n \n /** The composed feature extractor. */\n protected ComposedFeatureExtractor composedFeatureExtractor;\n \n /**\n * Constructor with component models, which are given uniform weights.\n */\n public SignScorerInterpolation(SignScorer[] models) { \n this.models = models;\n this.weights = new double[models.length];\n for (int i = 0; i < models.length; i++) {\n weights[i] = 1.0 / models.length;\n }\n }\n \n /**\n * Constructor with component models and weights.\n * The weights are assumed to sum to 1, \n * and the number of weights is assumed to match the number of models.\n */\n public SignScorerInterpolation(SignScorer[] models, double[] weights) { \n this.models = models;\n this.weights = weights;\n this.composedFeatureExtractor = new ComposedFeatureExtractor(models);\n }\n \n /** \n * Returns a score between 0 (worst) and 1 (best) for the given sign \n * and completeness flag, as the interpolation of the scores assigned \n * by the component models.\n * In particular, returns the linear combination using the established weights \n * of the scores given by the component models. \n */\n public double score(Sign sign, boolean complete) {\n double retval = 0;\n for (int i = 0; i < models.length; i++) {\n retval += models[i].score(sign, complete) * weights[i];\n }\n return retval;\n }\n \n\t/** Sets the alphabet for the component feature extractors. */\n\tpublic void setAlphabet(Alphabet alphabet) {\n\t\tcomposedFeatureExtractor.setAlphabet(alphabet);\n\t}\n\t\n\t/** Returns the features for the given sign and completeness flag from the component feature extractors. */\n\tpublic FeatureVector extractFeatures(Sign sign, boolean complete) {\n\t\treturn composedFeatureExtractor.extractFeatures(sign, complete);\n\t}\n}\n\n" }, { "path": "src/opennlp/ccg/ngrams/SignScorerProduct.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.ngrams;\n\nimport opennlp.ccg.perceptron.*;\nimport opennlp.ccg.synsem.Sign;\nimport opennlp.ccg.synsem.SignScorer;\n\n// import java.util.*;\n\n/**\n * Product of sign scorers, some of which may be feature extractors..\n *\n * @author Michael White\n * @version $Revision: 1.4 $, $Date: 2011/01/15 17:52:59 $\n */\npublic class SignScorerProduct implements SignScorer, FeatureExtractor\n{\n /** The component models. */\n protected SignScorer[] models;\n \n /** The composed feature extractor. */\n protected ComposedFeatureExtractor composedFeatureExtractor;\n \n /**\n * Constructor with component models.\n */\n public SignScorerProduct(SignScorer[] models) { \n this.models = models;\n this.composedFeatureExtractor = new ComposedFeatureExtractor(models);\n }\n \n /** \n * Returns a score between 0 (worst) and 1 (best) for the given sign \n * and completeness flag, as the product of the scores assigned \n * by the component models.\n */\n public double score(Sign sign, boolean complete) {\n double retval = 1.0;\n for (int i = 0; i < models.length; i++) {\n retval *= models[i].score(sign, complete);\n }\n return retval;\n }\n \n\t/** Sets the alphabet for the component feature extractors. */\n\tpublic void setAlphabet(Alphabet alphabet) {\n\t\tcomposedFeatureExtractor.setAlphabet(alphabet);\n\t}\n\t\n\t/** Returns the features for the given sign and completeness flag from the component feature extractors. */\n\tpublic FeatureVector extractFeatures(Sign sign, boolean complete) {\n\t\treturn composedFeatureExtractor.extractFeatures(sign, complete);\n\t}\n}\n\n" }, { "path": "src/opennlp/ccg/ngrams/StandardNgramModel.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2004-5 University of Edinburgh (Michael White)\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.ngrams;\r\n\r\nimport java.io.BufferedReader;\r\nimport java.io.FileReader;\r\nimport java.io.IOException;\r\nimport java.io.Reader;\r\nimport java.io.StreamTokenizer;\r\nimport java.util.ArrayList;\r\nimport java.util.List;\r\n\r\nimport opennlp.ccg.lexicon.DefaultTokenizer;\r\nimport opennlp.ccg.lexicon.Tokenizer;\r\nimport opennlp.ccg.lexicon.Word;\r\nimport opennlp.ccg.util.TrieMap;\r\n\r\n/**\r\n * A scorer for a standard n-gram backoff model. \r\n * Unknown words are mapped to <unk> if the latter is present in \r\n * the model.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.19 $, $Date: 2011/10/11 03:29:42 $\r\n */\r\npublic class StandardNgramModel extends AbstractStandardNgramModel\r\n{\r\n\t/** \r\n * Loads an n-gram model of the given order in ARPA (Doug Paul) format from\r\n * the given reader, with the given flag controlling whether words are\r\n * replaced by their semantic classes.\r\n */\r\n public StandardNgramModel(int order, Reader in, boolean useSemClasses)\r\n \t\tthrows IOException {\r\n super(order, useSemClasses);\r\n this.numNgrams = new int[order];\r\n readModel(in);\r\n }\r\n\r\n /** \r\n * Loads an n-gram model of the given order in ARPA (Doug Paul) format from\r\n * the given reader. Words are not replaced by their semantic classes.\r\n */\r\n\tpublic StandardNgramModel(int order, Reader in) throws IOException {\r\n\t\tthis(order, in, false);\r\n\t}\r\n\r\n\t/** \r\n * Loads an n-gram model of the given order in ARPA (Doug Paul) format from\r\n * the given file, with the given flag controlling whether words are\r\n * replaced by their semantic classes.\r\n */\r\n\tpublic StandardNgramModel(int order, String filename, boolean useSemClasses)\r\n\t\t\tthrows IOException {\r\n\t\tthis(order, new BufferedReader(new FileReader(filename)),\r\n \t\tuseSemClasses);\r\n\t}\r\n\r\n\t/** \r\n * Loads an n-gram model of the given order in ARPA (Doug Paul) format from\r\n * the given file. Words are not replaced by their semantic classes.\r\n */\r\n\tpublic StandardNgramModel(int order, String filename) throws IOException {\r\n\t\tthis(order, filename, false); \r\n\t}\r\n\t\r\n\t// reads in model\r\n private void readModel(Reader in) throws IOException {\r\n // setup\r\n\t\t//Tokenizer wordTokenizer = (Grammar.theGrammar != null) \r\n\t\t// ? Grammar.theGrammar.lexicon.tokenizer\r\n\t\t// : new DefaultTokenizer();\r\n StreamTokenizer tokenizer = initTokenizer(in); \r\n String[] tokens = new String[order+2];\r\n boolean foundData = false;\r\n int currentOrder = 0;\r\n List currentPrefix = new ArrayList();\r\n List currentKeys = null;\r\n List> currentChildren = null;\r\n // loop through lines\r\n while (tokenizer.ttype != StreamTokenizer.TT_EOF) {\r\n // read line into tokens\r\n readLine(tokenizer, tokens);\r\n // check for blank line\r\n if (tokens[0] == null) continue;\r\n // check for initial delimiter\r\n if (tokens[0].equals(\"\\\\data\\\\\")) { foundData = true; continue; }\r\n if (!foundData) continue;\r\n // read header line\r\n if (tokens[0].equals(\"ngram\")) {\r\n int n = Integer.parseInt(tokens[1].substring(0,1)); \r\n int total = Integer.parseInt(tokens[1].substring(2));\r\n if (n > order) continue;\r\n numNgrams[n-1] = total;\r\n // init children, keys lists\r\n if (currentChildren == null) { \r\n currentChildren = new ArrayList>(total); \r\n currentKeys = new ArrayList(total); \r\n }\r\n // calc totals (not actually used anymore)\r\n if (n == order) {\r\n @SuppressWarnings(\"unused\")\r\n\t\t\t\t\tint totalNgrams = 0;\r\n for (int i = 0; i < order; i++) { totalNgrams += numNgrams[i]; }\r\n // System.out.println(\"totalNgrams: \" + totalNgrams);\r\n }\r\n continue;\r\n }\r\n // check for final delimiter\r\n if (tokens[0].equals(\"\\\\end\\\\\")) {\r\n // add current children\r\n addTrieMapChildren(currentPrefix, currentKeys, currentChildren);\r\n break;\r\n }\r\n // read line starting new order\r\n if (tokens[0].equals(\"\\\\\" + (currentOrder+1) + \"-grams:\")) { \r\n // add current children\r\n addTrieMapChildren(currentPrefix, currentKeys, currentChildren);\r\n // System.out.println(tokens[0]);\r\n currentOrder++; continue; \r\n } \r\n if (currentOrder == 0) continue;\r\n if (currentOrder > order) break;\r\n // read logprob\r\n float logprob = Float.parseFloat(tokens[0]);\r\n // read back-off weight (except with last order)\r\n float bow = 0;\r\n if (currentOrder < order && tokens[currentOrder+1] != null) {\r\n bow = Float.parseFloat(tokens[currentOrder+1]);\r\n }\r\n // intern string tokens\r\n for (int i = 1; i < currentOrder+1; i++) {\r\n tokens[i] = tokens[i].intern();\r\n }\r\n // check prefix\r\n boolean samePrefix = (currentPrefix.size() == currentOrder-1);\r\n for (int i = 1; samePrefix && i < currentOrder; i++) {\r\n if (tokens[i] != currentPrefix.get(i-1)) samePrefix = false; \r\n }\r\n // if changed, add current children, reset prefix \r\n if (!samePrefix) {\r\n addTrieMapChildren(currentPrefix, currentKeys, currentChildren);\r\n for (int i = 1; i < currentOrder; i++) {\r\n currentPrefix.add(tokens[i]);\r\n }\r\n }\r\n String key = tokens[currentOrder];\r\n currentKeys.add(key);\r\n currentChildren.add(new TrieMap(new NgramFloats(logprob, bow)));\r\n }\r\n // set openVocab according to presence of \r\n openVocab = (trieMapRoot.getChild(\"\") != null);\r\n }\r\n \r\n /** Test loading and scoring. */\r\n // NB: This produces the same scores as the SRILM ngram tool when both \r\n // and tags are used.\r\n public static void main(String[] args) throws IOException {\r\n \r\n String usage = \"Usage: java opennlp.ccg.ngrams.StandardNgramModel (-reverse)\";\r\n \r\n if (args.length > 0 && args[0].equals(\"-h\")) {\r\n System.out.println(usage);\r\n System.exit(0);\r\n }\r\n \r\n long start = System.currentTimeMillis();\r\n String order = args[0]; String lmfile = args[1]; String tokens = args[2];\r\n String reversed = (args.length >= 4 && args[3].equals(\"-reverse\")) ? \"reversed \" : \"\";\r\n System.out.println(\"Loading \" + reversed + \"n-gram model with order \" + order + \" from: \" + lmfile);\r\n StandardNgramModel lm = new StandardNgramModel(Integer.parseInt(order), lmfile);\r\n if (reversed.length() > 0) lm.setReverse(true);\r\n System.out.println(\"openVocab: \" + lm.openVocab);\r\n int secs = (int) (System.currentTimeMillis() - start) / 1000;\r\n System.out.println(\"secs: \" + secs);\r\n System.out.println();\r\n // System.out.println(\"trie map: \");\r\n // System.out.println(lm.trieMapRoot.toString());\r\n // System.out.println();\r\n \r\n Tokenizer tokenizer = new DefaultTokenizer();\r\n List words = tokenizer.tokenize(tokens);\r\n System.out.println(\"scoring: \" + tokens);\r\n System.out.println();\r\n lm.debugScore = true;\r\n lm.setWordsToScore(words, true);\r\n lm.prepareToScoreWords();\r\n double logprob = lm.logprob();\r\n double score = convertToProb(logprob);\r\n System.out.println();\r\n System.out.println(\"score: \" + score);\r\n System.out.println(\"logprob: \" + logprob);\r\n System.out.println(\"ppl: \" + NgramScorer.convertToPPL(logprob / (words.size()-1)));\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/ngrams/kenlm/MurmurHash.java", "content": "package opennlp.ccg.ngrams.kenlm;\n\nimport java.io.UnsupportedEncodingException;\n\n/**\n * MurmurHash 2.0.\n * \n * The murmur hash is a relative fast hash function from http://murmurhash.googlepages.com/ for\n * platforms with efficient multiplication.\n * \n * This is a re-implementation of the original C code plus some additional features.\n * \n * Public domain.\n * \n * @author Viliam Holub\n * @version 1.0.2\n * \n */\npublic final class MurmurHash {\n\n private final static String ENCODING = \"UTF-16\";\n\n /**\n * Generates 32 bit hash from byte array of the given length and seed.\n * \n * @param data byte array to hash\n * @param length length of the array to hash\n * @param seed initial seed value\n * @return 32 bit hash of the given array\n */\n public static int hash32(final byte[] data, int length, int seed) {\n // 'm' and 'r' are mixing constants generated offline.\n // They're not really 'magic', they just happen to work well.\n final int m = 0x5bd1e995;\n final int r = 24;\n // Initialize the hash to a random value\n int h = seed ^ length;\n int length4 = length / 4;\n\n for (int i = 0; i < length4; i++) {\n final int i4 = i * 4;\n int k =\n (data[i4 + 0] & 0xff) + ((data[i4 + 1] & 0xff) << 8) + ((data[i4 + 2] & 0xff) << 16)\n + ((data[i4 + 3] & 0xff) << 24);\n k *= m;\n k ^= k >>> r;\n k *= m;\n h *= m;\n h ^= k;\n }\n\n // Handle the last few bytes of the input array\n switch (length % 4) {\n case 3:\n h ^= (data[(length & ~3) + 2] & 0xff) << 16;\n case 2:\n h ^= (data[(length & ~3) + 1] & 0xff) << 8;\n case 1:\n h ^= (data[length & ~3] & 0xff);\n h *= m;\n }\n\n h ^= h >>> 13;\n h *= m;\n h ^= h >>> 15;\n\n return h;\n }\n\n\n /**\n * Generates 32 bit hash from byte array with default seed value.\n * \n * @param data byte array to hash\n * @param length length of the array to hash\n * @return 32 bit hash of the given array\n */\n public static int hash32(final byte[] data, int length) {\n return hash32(data, length, 0x9747b28c);\n }\n\n\n /**\n * Generates 32 bit hash from a string.\n * \n * @param text string to hash\n * @return 32 bit hash of the given string\n * @throws UnsupportedEncodingException\n */\n public static int hash32(final String text) throws UnsupportedEncodingException {\n final byte[] bytes = text.getBytes(ENCODING);\n return hash32(bytes, bytes.length);\n }\n\n\n /**\n * Generates 32 bit hash from a substring.\n * \n * @param text string to hash\n * @param from starting index\n * @param length length of the substring to hash\n * @return 32 bit hash of the given string\n * @throws UnsupportedEncodingException\n */\n public static int hash32(final String text, int from, int length)\n throws UnsupportedEncodingException {\n return hash32(text.substring(from, from + length));\n }\n\n\n /**\n * Generates 64 bit hash from byte array of the given length and seed.\n * \n * @param data byte array to hash\n * @param length length of the array to hash\n * @param seed initial seed value\n * @return 64 bit hash of the given array\n */\n public static long hash64(final byte[] data, int length, int seed) {\n final long m = 0xc6a4a7935bd1e995L;\n final int r = 47;\n\n long h = (seed & 0xffffffffl) ^ (length * m);\n\n int length8 = length / 8;\n\n for (int i = 0; i < length8; i++) {\n final int i8 = i * 8;\n long k =\n ((long) data[i8 + 0] & 0xff) + (((long) data[i8 + 1] & 0xff) << 8)\n + (((long) data[i8 + 2] & 0xff) << 16) + (((long) data[i8 + 3] & 0xff) << 24)\n + (((long) data[i8 + 4] & 0xff) << 32) + (((long) data[i8 + 5] & 0xff) << 40)\n + (((long) data[i8 + 6] & 0xff) << 48) + (((long) data[i8 + 7] & 0xff) << 56);\n\n k *= m;\n k ^= k >>> r;\n k *= m;\n\n h ^= k;\n h *= m;\n }\n\n switch (length % 8) {\n case 7:\n h ^= (long) (data[(length & ~7) + 6] & 0xff) << 48;\n case 6:\n h ^= (long) (data[(length & ~7) + 5] & 0xff) << 40;\n case 5:\n h ^= (long) (data[(length & ~7) + 4] & 0xff) << 32;\n case 4:\n h ^= (long) (data[(length & ~7) + 3] & 0xff) << 24;\n case 3:\n h ^= (long) (data[(length & ~7) + 2] & 0xff) << 16;\n case 2:\n h ^= (long) (data[(length & ~7) + 1] & 0xff) << 8;\n case 1:\n h ^= (long) (data[length & ~7] & 0xff);\n h *= m;\n };\n\n h ^= h >>> r;\n h *= m;\n h ^= h >>> r;\n\n return h;\n }\n\n\n /**\n * Generates 64 bit hash from byte array with default seed value.\n * \n * @param data byte array to hash\n * @param length length of the array to hash\n * @return 64 bit hash of the given string\n */\n public static long hash64(final byte[] data, int length) {\n return hash64(data, length, 0xe17a1465);\n }\n\n\n /**\n * Generates 64 bit hash from a string.\n * \n * @param text string to hash\n * @return 64 bit hash of the given string\n * @throws UnsupportedEncodingException\n */\n public static long hash64(final String text) throws UnsupportedEncodingException {\n byte[] bytes;\n bytes = text.getBytes(ENCODING);\n return hash64(bytes, bytes.length);\n }\n\n\n /**\n * Generates 64 bit hash from a substring.\n * \n * @param text string to hash\n * @param from starting index\n * @param length length of the substring to hash\n * @return 64 bit hash of the given array\n * @throws UnsupportedEncodingException\n */\n public static long hash64(final String text, int from, int length)\n throws UnsupportedEncodingException {\n return hash64(text.substring(from, from + length));\n }\n}\n" }, { "path": "src/opennlp/ccg/ngrams/kenlm/jni/KenLM.java", "content": "package opennlp.ccg.ngrams.kenlm.jni;\n\n// TODO(Joshua devs): include my state object with your LM state then\n// update this API to pass state instead of int[].\n\npublic class KenLM {\n /**\n * Load the binary, platform-dependent library containing the KenLM JNI\n * bridge code. \n * @throws UnsatisfiedLinkError If Java can't find the 'ken' library.\n */\n static {\n System.loadLibrary(\"ken\");\n }\n\n private final long pointer;\n // this is read from the config file, used to set maximum order\n private final int ngramOrder;\n // inferred from model file (may be larger than ngramOrder)\n private final int N;\n\n\n private final static native long construct(String file_name, float fake_oov);\n\n private final static native void destroy(long ptr);\n\n private final static native int order(long ptr);\n\n private final static native boolean registerWord(long ptr, String word, int id);\n\n private final static native float prob(long ptr, int words[]);\n\n private final static native float probString(long ptr, int words[], int start);\n\n public KenLM(int order, String file_name) {\n float lm_ceiling_cost = 99.0f;\n ngramOrder = order;\n pointer = construct(file_name, -lm_ceiling_cost);\n N = order(pointer);\n }\n\n public void destroy() {\n destroy(pointer);\n }\n\n public int getOrder() {\n return N;\n }\n\n public boolean registerWord(String word, int id) {\n return registerWord(pointer, word, id);\n }\n\n public float prob(int words[]) {\n return prob(pointer, words);\n }\n\n}\n" }, { "path": "src/opennlp/ccg/parse/Chart.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-10 Jason Baldridge, Gann Bierner and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse;\n\nimport opennlp.ccg.grammar.*;\nimport opennlp.ccg.synsem.*;\nimport gnu.trove.*;\n\nimport java.io.*;\nimport java.util.*;\n\n/**\n * An implementation of the table (or chart) used for chart parsers like CKY.\n * Special functions are provided for combining cells of the chart into another\n * cell. Time or edge or cell limits can be placed on initial chart construction. \n * A pruning value applies to unpacking, which also limits the number of equivalent \n * edges kept during chart construction.\n * \n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.41 $, $Date: 2011/11/16 03:25:27 $\n */\npublic class Chart {\n\n // maps edges to representative edges, according to their headwords and cats, sans LFs\n\t// NB: using unfilled dependencies in equiv relation appears to unacceptably slow down parsing, \n\t// with a significant drop in complete parses\n @SuppressWarnings(\"unchecked\")\n private static Map createEdgeMap() {\n \treturn new THashMap(11, representativeEdgeStrategy);\n }\n \n private static TObjectHashingStrategy representativeEdgeStrategy = new TObjectHashingStrategy() {\n\t\tprivate static final long serialVersionUID = 1L;\n\t\tpublic int computeHashCode(Object o) {\n Sign sign = ((Edge)o).sign;\n int headpos = Edge.getEdge(sign.getLexHead()).wordPos; \n return 31*headpos + sign.getCategory().hashCodeNoLF(); \n //return 31*headpos + sign.getCategory().hashCodeNoLF() + 17*sign.getUnfilledDeps().hashCode(); \n }\n public boolean equals(Object o1, Object o2) {\n \tif (!(o1 instanceof Edge) || !(o2 instanceof Edge)) return false;\n Sign sign1 = ((Edge)o1).sign; Sign sign2 = ((Edge)o2).sign;\n return Edge.getEdge(sign1.getLexHead()).wordPos == Edge.getEdge(sign2.getLexHead()).wordPos && \n \tsign1.getCategory().equalsNoLF(sign2.getCategory()); \n \t//&& sign1.getUnfilledDeps().equals(sign2.getUnfilledDeps());\n }\n };\n \n // a cell pairs a sorted list with an edge map\n private class Cell implements Serializable {\n\t\tprivate static final long serialVersionUID = 1L;\n\t\tfinal List list = new ArrayList();\n \tfinal Map map = createEdgeMap();\n \tint size() { return list.size(); }\n \tEdge get(Edge edge) { return map.get(edge); }\n \t// add edge, preserving cell limit; return true iff given edge added\n \tboolean add(Edge edge) {\n \t\tif (map.containsKey(edge)) return false;\n \t\treturn addEdgeSorted(edge, list, map, _cellLimit);\n \t}\n \tList getSignsSorted() {\n \tList retval = new ArrayList(list.size());\n \tfor (Edge e : list) retval.add(e.sign);\n \treturn retval;\n \t}\n \tSignHash getSigns() {\n \t\tSignHash retval = new SignHash();\n \t\tfor (Edge e : list) retval.insert(e.sign);\n \treturn retval;\n \t}\n };\t\n \n // adds edge to sorted list and optional map, preserving limit; returns true iff edge added\n // nb: all lexical edges kept\n private boolean addEdgeSorted(Edge edge, List list, Map map, int limit) {\n\t\tint index = Collections.binarySearch(list, edge, edgeComparator);\n\t\t// convert index to insertion point\n index = Math.abs(index) - 1;\n // if somehow negative, use last position\n if (index < 0) index = list.size();\n // check if last and at limit\n boolean limitActive = limit > 0 && !edge.sign.isLexical();\n if (limitActive && index >= limit) return false;\n // otherwise add edge\n list.add(index, edge);\n\t\tif (map != null) map.put(edge, edge);\n\t\t// remove last if over limit\n\t\tif (limitActive && list.size() > limit) {\n\t\t\tEdge last = list.remove(list.size()-1);\n\t\t\tif (map != null) map.remove(last);\n\t\t}\n\t\treturn true; \t\n }\n \n /** Compares edges based on their relative score, in descending order, then their signs. */\n\tpublic static final Comparator edgeComparator = new Comparator() {\n\t\tpublic int compare(Edge edge1, Edge edge2) {\n\t\t\tif (edge1.score != edge2.score)\n\t\t\t\treturn -1 * Double.compare(edge1.score, edge2.score);\n\t\t\telse \n\t\t\t\treturn SignHash.compareTo(edge1.sign, edge2.sign);\n\t\t}\n\t};\n\n /** The chart. */\n\tprotected Cell[][] _table;\n\n\t/** Its size. */\n\tprotected int _size;\n\n\t/** The count of edges created before unpacking. */\n\tprotected int _numEdges = 0;\n\t\n\t/** The count of edges created while unpacking. */\n\tprotected int _numUnpackingEdges = 0;\n\t\n\t/** The max cell size before unpacking. */\n\tprotected int _maxCellSize = 0;\n\t\n\t/** The rules. */\n\tprotected RuleGroup _rules;\n\n\t/** The sign scorer (defaults to the null scorer). */\n\tprotected SignScorer _signScorer = SignScorer.nullScorer;\n\t\n\t/** The \"n\" for n-best pruning (or 0 if none). */\n\tprotected int _pruneVal = 0;\n\t\n\t/** The time limit (0 if none). */\n\tprotected int _timeLimit = 0;\n\t\n\t/** The start time. */\n\tprotected long _startTime = 0;\n\t\n\t/** The edge limit (0 if none). */\n\tprotected int _edgeLimit = 0;\n\t\n\t/** The cell limit on non-lexical edges (0 if none). */\n\tprotected int _cellLimit = 0;\n\t\n\t/** Constructor. */\n\tpublic Chart(int s, RuleGroup _R) {\n\t\t_rules = _R; _size = s;\n\t\t_table = new Cell[_size][_size];\n\t}\n\n\t/** Sets the sign scorer. */\n\tpublic void setSignScorer(SignScorer signScorer) { _signScorer = signScorer; }\n\t\n\t/** Sets the n-best pruning val. */\n\tpublic void setPruneVal(int n) { _pruneVal = n; }\n\t\n\t/** Sets the time limit. */\n\tpublic void setTimeLimit(int timeLimit) { _timeLimit = timeLimit; }\n\t\n\t/** Sets the start time. */\n\tpublic void setStartTime(long startTime) { _startTime = startTime; }\n\t\n\t/** Sets the edge limit. */\n\tpublic void setEdgeLimit(int edgeLimit) { _edgeLimit = edgeLimit; }\n\t\n\t/** Sets the cell limit on non-lexical edges. */\n\tpublic void setCellLimit(int cellLimit) { _cellLimit = cellLimit; }\n\t\n\t/** Returns the edge count prior to unpacking. */\n\tpublic int edgeCount() { return _numEdges; }\n\t\n\t/** Returns the edge count while unpacking. */\n\tpublic int unpackingEdgeCount() { return _numUnpackingEdges; }\n\t\n\t/** Returns the max cell size prior to unpacking. */\n\tpublic int maxCellSize() { return _maxCellSize; }\n\t\n\t\n\t//-----------------------------------------------------------\n\t// Chart construction\n\t\n\t/** \n\t * Inserts a sign at the given cell (modulo pruning). \n\t * Returns true if an edge for the sign is added as a new equiv class. \n\t */\n\tpublic boolean insert(int x, int y, Sign w) {\n\t\tCell cell = get(x, y);\n\t\tboolean retval = false;\n\t\t// make edge\n\t\tEdge edge = new Edge(w);\n\t\tif (w.isLexical()) edge.setWordPos(x);\n\t\t// get representative edge\n\t\tEdge rep = cell.get(edge);\n\t\t// if none, add as representative\n\t\tif (rep == null) {\n\t\t\tedge.initAltEdges();\n\t\t\tretval = cell.add(edge);\n\t\t}\n\t\t// otherwise add as an alternative\n\t\telse {\n\t\t\taddEdgeSorted(edge, rep.altEdges, null, _pruneVal);\n\t\t}\n\t\t// update edge count, max cell size\n\t\t_numEdges++;\n\t\tif (cell.size() > _maxCellSize) _maxCellSize = cell.size();\n\t\t// done\n\t\treturn retval;\n\t}\n\n\t/** Returns the given cell (ensuring non-null). */\n\tprotected Cell get(int x, int y) {\n\t\tif (_table[x][y] == null) _table[x][y] = new Cell();\n\t\treturn _table[x][y];\n\t}\n\n\t/** Returns the signs for a given cell (ensuring non-null). */\n\tprotected SignHash getSigns(int x, int y) {\n\t\tCell cell = get(x, y);\n\t\treturn cell.getSigns();\n\t}\n\n\t/** Inserts edges into (x,y) that result from applying unary rules to those already in (x,y). \n\t * @throws ParseException */ \n\tprotected void insertCell(int x, int y) throws ParseException {\n\t\tif (_table[x][y] == null) return;\n\t\tList inputs = _table[x][y].getSignsSorted();\n\t\tList nextInputs = new ArrayList(inputs.size());\n\t\t// repeat until no more inputs\n\t\twhile (inputs.size() > 0) {\n\t\t\t// apply rules\n\t\t\tfor (Sign sign : inputs) {\n\t\t\t\tcheckLimits();\n\t\t\t\tList results = _rules.applyUnaryRules(sign);\n\t\t\t\tfor (Sign result : results) {\n\t\t\t\t\t// check for unary rule cycle; skip result if found\n\t\t\t\t\tif (!result.getDerivationHistory().containsCycle()) {\n\t\t\t\t\t\t// insert result\n\t\t\t\t\t\tboolean newEdgeClass = insert(x, y, result);\n\t\t\t\t\t\t// add to next inputs if it yielded a new equiv class\n\t\t\t\t\t\tif (newEdgeClass) nextInputs.add(result);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\t// move all results to inputs\n\t\t\tinputs.clear();\n\t\t\tinputs.addAll(nextInputs);\n\t\t\tnextInputs.clear();\n\t\t}\n\t}\n\n\t/** Inserts edges into (x3,y3) resulting from combining those in (x1,y1) and (x2,y2). \n\t * @throws ParseException */\n\tprotected void insertCell(int x1, int y1, int x2, int y2, int x3, int y3) throws ParseException {\n\t\tif (_table[x1][y1] == null) return;\n\t\tif (_table[x2][y2] == null) return;\n\t\tList inputs1 = _table[x1][y1].getSignsSorted();\n\t\tList inputs2 = _table[x2][y2].getSignsSorted();\n\t\tfor (Sign sign1 : inputs1) {\n\t\t\tfor (Sign sign2 : inputs2) {\n\t\t\t\tcheckLimits();\n\t\t\t\tList results = _rules.applyBinaryRules(sign1, sign2);\n\t\t\t\tfor (Sign result : results)\n\t\t\t\t\tinsert(x3, y3, result);\n\t\t\t}\n\t\t}\n\t}\n\n\t/** \n\t * Inserts fragmentary edges into (x3,y3), if non-empty, resulting from combining \n\t * those in (x1,y1) and (x2,y2) using the glue rule. \n\t * @throws ParseException \n\t */\n\tprotected void insertCellFrag(int x1, int y1, int x2, int y2, int x3, int y3) throws ParseException {\n\t\tif (_table[x1][y1] == null) return;\n\t\tif (_table[x2][y2] == null) return;\n\t\tif (!cellIsEmpty(x3, y3)) return;\n\t\tList inputs1 = _table[x1][y1].getSignsSorted();\n\t\tList inputs2 = _table[x2][y2].getSignsSorted();\n\t\tfor (Sign sign1 : inputs1) {\n\t\t\tfor (Sign sign2 : inputs2) {\n\t\t\t\tcheckLimits();\n\t\t\t\tList results = _rules.applyGlueRule(sign1, sign2);\n\t\t\t\tfor (Sign result : results)\n\t\t\t\t\tinsert(x3, y3, result);\n\t\t\t}\n\t\t}\n\t}\n\n // check edge and time limit\n private void checkLimits() throws ParseException {\n \tif (_edgeLimit > 0 && _numEdges > _edgeLimit) {\n \t\tthrow new ParseException(ParseException.EDGE_LIMIT_EXCEEDED);\n \t}\n if (_timeLimit > 0) {\n \tint timeSoFar = (int) (System.currentTimeMillis() - _startTime);\n \tif (timeSoFar > _timeLimit) {\n \t\tthrow new ParseException(ParseException.TIME_LIMIT_EXCEEDED);\n \t}\n }\n }\n \n\t/** Returns whether the given cell is empty. */\n public boolean cellIsEmpty(int x, int y) {\n\t\tCell cell = get(x, y);\n \treturn cell.list.isEmpty();\n }\n \n \n\t//-----------------------------------------------------------\n\t// Unpacking \n\t\n\t/** Unpacks the edges in the given cell as an n-best list. */\n\tpublic List unpack(int x, int y) {\n\t\tCell cell = get(x, y);\n\t\t// recursively unpack each edge\n\t @SuppressWarnings(\"unchecked\")\n Set unpacked = new THashSet(new TObjectIdentityHashingStrategy());\n\t @SuppressWarnings(\"unchecked\")\n Set startedUnpacking = new THashSet(new TObjectIdentityHashingStrategy());\n\t\tfor (Edge edge : cell.list) unpack(edge, unpacked, startedUnpacking); \n\t\t// collect and sort results\n EdgeHash merged = new EdgeHash();\n\t\tfor (Edge edge : cell.list) {\n\t\t\tmerged.addAll(edge.altEdges);\n\t\t}\n\t\tList retval = new ArrayList(merged.asEdgeSet());\n\t\tCollections.sort(retval, edgeComparator);\n // prune\n if (_pruneVal > 0) {\n \twhile (retval.size() > _pruneVal) retval.remove(retval.size()-1);\n }\n // restore alts\n for (Edge edge : cell.list) edge.restoreAltEdges();\n\t\t// return\n\t\treturn retval;\n\t}\n\n\t// recursively unpack edge, unless already visited\n\tprivate void unpack(Edge edge, Set unpacked, Set startedUnpacking) {\n\t\t// check visited\n if (unpacked.contains(edge)) return;\n if (startedUnpacking.contains(edge)) {\n \tSystem.err.println(\"Warning, revisiting edge before unpacking complete: \" + edge);\n \tSystem.err.println(edge.sign.getDerivationHistory().toString());\n \treturn;\n }\n startedUnpacking.add(edge);\n // OR: recursively unpack alts, merging resulting alts\n EdgeHash merged = new EdgeHash();\n for (Edge alt : edge.altEdges) {\n // AND: unpack inputs, make alts, add to merged\n unpackAlt(alt, unpacked, startedUnpacking, merged);\n }\n // score\n boolean complete = (edge.sign.getWords().size() == _size);\n for (Edge m : merged.asEdgeSet()) { m.setScore(_signScorer.score(m.sign, complete)); }\n // sort\n List mergedList = new ArrayList(merged.asEdgeSet());\n Collections.sort(mergedList, edgeComparator);\n // prune\n if (_pruneVal > 0) {\n \twhile (mergedList.size() > _pruneVal) mergedList.remove(mergedList.size()-1);\n }\n // replace edge's alts\n edge.replaceAltEdges(mergedList);\n // add to unpacked set\n unpacked.add(edge);\n }\n \n // recursively unpack inputs, make alt combos and add to merged\n private void unpackAlt(Edge alt, Set unpacked, Set startedUnpacking, EdgeHash merged) {\n // unpack via input signs\n DerivationHistory history = alt.sign.getDerivationHistory(); \n Sign[] inputSigns = history.getInputs();\n // base case: no inputs\n if (inputSigns == null) {\n merged.insert(alt); return;\n }\n // otherwise recursively unpack\n Edge[] inputEdges = new Edge[inputSigns.length];\n for (int i = 0; i < inputSigns.length; i++) {\n inputEdges[i] = Edge.getEdge(inputSigns[i]); \n unpack(inputEdges[i], unpacked, startedUnpacking);\n }\n // then make edges for new combos, using same rule, and add to merged (if unseen)\n Rule rule = history.getRule();\n List altCombos = inputCombos(inputEdges, 0);\n List results = new ArrayList(1);\n for (Sign[] combo : altCombos) {\n \t// use this alt for same combo\n \tif (sameSigns(inputSigns, combo)) {\n \t\tmerged.insert(alt); continue;\n \t}\n \tresults.clear();\n \t((AbstractRule)rule).applyRule(combo, results); // TODO: bypass rule app for efficiency? (requires doing something about var subst)\n \tif (results.isEmpty()) continue; // (rare?)\n Sign sign = results.get(0); // assuming single result\n merged.insert(new Edge(sign)); // make edge for new alt\n _numUnpackingEdges++;\n }\n\t}\n\t\n // returns a list of sign arrays, with each array of length inputEdges.length - i, \n // representing all combinations of alt signs from i onwards\n private List inputCombos(Edge[] inputEdges, int index) {\n Edge edge = inputEdges[index];\n // base case, inputEdges[last]\n if (index == inputEdges.length-1) {\n List altEdges = edge.altEdges; \n List retval = new ArrayList(altEdges.size());\n for (Edge alt : altEdges) {\n retval.add(new Sign[] { alt.sign });\n }\n return retval;\n }\n // otherwise recurse on index+1\n List nextCombos = inputCombos(inputEdges, index+1);\n // and make new combos\n List altEdges = edge.altEdges; \n List retval = new ArrayList(altEdges.size() * nextCombos.size());\n for (Edge alt : altEdges) {\n for (int i = 0; i < nextCombos.size(); i++) {\n Sign[] nextSigns = nextCombos.get(i);\n Sign[] newCombo = new Sign[nextSigns.length+1];\n newCombo[0] = alt.sign;\n System.arraycopy(nextSigns, 0, newCombo, 1, nextSigns.length);\n retval.add(newCombo);\n }\n }\n return retval;\n }\n\n // checks for same signs\n private boolean sameSigns(Sign[] a, Sign[] b) {\n \tif (a.length != b.length) return false;\n \tfor (int i=0; i < a.length; i++)\n \t\tif (a[i] != b[i]) return false;\n \treturn true;\n }\n \n\t\n\t//-----------------------------------------------------------\n\t// Lazy Unpacking \n\t\n\t/** \n\t * Lazily unpacks the edges in the given cell as an n-best list \n\t * using a variant of \"cube pruning\". The algorithm essentially \n\t * follows Algorithm 2 of Huang and Chiang (2005), with checking \n\t * for spurious ambiguity.\n\t */ \n\t@SuppressWarnings(\"unchecked\")\n\tpublic List lazyUnpack(int x, int y) {\n\t\t// if no pruning value set, use basic unpacking algorithm\n\t\tif (_pruneVal <= 0) return unpack(x, y);\n\t\t// recursively sort edge alts\n\t\tCell cell = get(x, y);\n\t\t// make top-level candidate list and derivs map\n\t\tList topcands = new ArrayList(_pruneVal);\n\t\tMap> derivsmap = new THashMap(new TObjectIdentityHashingStrategy());\n\t\tfor (Edge edge : cell.list) {\n\t\t\tList cands = getCandidates(edge, derivsmap);\n\t\t\ttopcands.addAll(cands);\n\t\t}\n\t\tsortAndPrune(topcands);\n\t\t// NB: no single edge for top cell, so must treat it as a special case of findKBest\n\t\tList retval = new ArrayList(_pruneVal);\n \tEdgeHash merged = new EdgeHash();\n \twhile (merged.size() < _pruneVal && !topcands.isEmpty()) {\n \t\tappendNext(topcands, merged, derivsmap);\n \t}\n \tretval.addAll(merged.asEdgeSet());\n \t// rescore edges if apropos\n \tif (_signScorer instanceof ReRankingScorer) {\n \t\tReRankingScorer rescorer = (ReRankingScorer) _signScorer;\n \t\trescorer.setFullModel(true);\n \t\tfor (Edge e : retval) {\n \t\t\te.score = rescorer.score(e.sign, true);\n \t\t}\n \t\trescorer.setFullModel(false);\n \t}\n \tCollections.sort(retval, edgeComparator);\n\t\t// done\n\t\treturn retval;\n\t}\n\t\n // lazily find k-best derivations, if edge not already visited\n private void findKBest(Edge edge, Map> derivsmap) {\n \tif (derivsmap.containsKey(edge)) return;\n \tList cands = getCandidates(edge, derivsmap);\n \tEdgeHash merged = new EdgeHash();\n \twhile (merged.size() < _pruneVal && !cands.isEmpty()) {\n \t\tappendNext(cands, merged, derivsmap);\n \t}\n \tList derivs = new ArrayList(_pruneVal);\n \tderivs.addAll(merged.asEdgeSet());\n \tCollections.sort(derivs, edgeComparator);\n \tderivsmap.put(edge, derivs);\n }\n \n // appends next candidate, expands frontier\n private void appendNext(List cands, EdgeHash merged, Map> derivsmap) {\n \t// append next\n \tCandidate cand = cands.remove(0);\n \tmerged.add(cand.edge);\n\t\t// check for lex cand\n\t\tif (cand.indices == null) return;\n\t\t// enumerate frontier\n\t\tfor (int i=0; i < cand.indices.length; i++) {\n\t\t\t// inc nextIndices at i\n\t\t\tint[] nextIndices = new int[cand.indices.length];\n\t\t\tfor (int m=0; m < nextIndices.length; m++) nextIndices[m] = cand.indices[m];\n\t\t\tnextIndices[i]++;\n\t\t\tEdge next = getEdgeForIndices(cand.edge, cand.inputReps, nextIndices, derivsmap); \n\t\t\t// add next candidate, if any, if not already there\n\t\t\tif (next != null) {\n\t\t\t\tCandidate nextCand = new Candidate(next, cand.inputReps, nextIndices);\n\t\t\t\tif (!cands.contains(nextCand)) {\n\t\t\t\t\tint index = Collections.binarySearch(cands, nextCand);\n\t\t index = Math.abs(index) - 1; // convert index to insertion point\n\t\t if (index >= 0) \n\t\t \tcands.add(index, nextCand);\n\t\t else cands.add(nextCand);\n\t\t\t\t}\n\t\t\t}\n\t\t}\n }\n \n\t// candidate is an edge plus an array of indices for keeping track of \n\t// where to pull candidates from next (or null if lexical), \n // using the input representatives\n\tprivate static class Candidate implements Comparable {\n\t\tEdge edge; Edge[] inputReps; int[] indices;\n\t\tCandidate(Edge edge, Edge[] inputReps, int[] indices) { \n\t\t\tthis.edge = edge; this.inputReps = inputReps; this.indices = indices; \n\t\t}\n\t\tpublic int compareTo(Candidate c) { \n\t\t\tint retval = edgeComparator.compare(edge, c.edge);\n\t\t\tif (retval != 0) return retval;\n\t\t\tif (indices == null && c.indices == null) return 0;\n\t\t\tif (indices == null && c.indices != null) return -1;\n\t\t\tif (indices != null && c.indices == null) return 1;\n\t\t\tif (indices.length < c.indices.length) return -1;\n\t\t\tif (indices.length > c.indices.length) return 1;\n\t\t\tfor (int i=0; i < indices.length; i++) {\n\t\t\t\tif (indices[i] < c.indices[i]) return -1;\n\t\t\t\tif (indices[i] > c.indices[i]) return 1;\n\t\t\t}\n\t\t\treturn 0;\n\t\t}\n\t\tpublic boolean equals(Object o) {\n\t\t\tif (!(o instanceof Candidate)) return false;\n\t\t\tCandidate c = (Candidate)o;\n\t\t\tif (indices != null && c.indices == null) return false;\n\t\t\tif (indices == null && c.indices != null) return false;\n\t\t\tif (indices != null && c.indices != null) {\n\t\t\t\tif (indices.length != c.indices.length) return false;\n\t\t\t\tfor (int i=0; i < indices.length; i++) {\n\t\t\t\t\tif (indices[i] != c.indices[i]) return false;\n\t\t\t\t}\n\t\t\t}\n\t\t\treturn edge.equals(c.edge);\n\t\t}\n\t}\n\t\n\t// get candidates for unpacking an edge\n\tprivate List getCandidates(Edge edge, Map> derivsmap) {\n\t\tList retval = new ArrayList(_pruneVal);\n\t\t// make initial candidate for each alt\n\t\t// nb: should only get initial candidates for representative edges, \n\t\t// but may as well ensure that at least this edge is included\n\t\tList alts = new ArrayList(edge.getAltEdges());\n\t\tif (alts.isEmpty()) alts.add(edge);\n\t\tfor (Edge alt : alts) {\n\t\t\tSign[] inputs = alt.sign.getDerivationHistory().getInputs();\n\t\t\t// lex case: no indices\n\t\t\tif (inputs == null) {\n\t\t\t\tretval.add(new Candidate(alt, null, null));\n\t\t\t\tcontinue;\n\t\t\t}\n\t\t\t// otherwise get edge for best inputs\n\t\t\tEdge[] inputReps = new Edge[inputs.length];\n\t\t\tint[] indices = new int[inputs.length];\n\t\t\tfor (int i=0; i < inputs.length; i++) {\n\t\t\t\tinputReps[i] = Edge.getEdge(inputs[i]);\n\t\t\t\tindices[i] = 0;\n\t\t\t}\n\t\t\tEdge e = getEdgeForIndices(alt, inputReps, indices, derivsmap);\n\t\t\tif (e != null) {\n\t\t\t\tretval.add(new Candidate(e, inputReps, indices));\n\t\t\t}\n\t\t}\n\t\t// sort and prune\n\t\tsortAndPrune(retval);\n\t\t// done\n\t\treturn retval;\n\t}\n \n\t// returns the edge for the given input indices, or null if none\n\tprivate Edge getEdgeForIndices(Edge edge, Edge[] inputReps, int[] indices, Map> derivsmap) {\n\t\tDerivationHistory history = edge.sign.getDerivationHistory();\n Sign[] combo = new Sign[inputReps.length];\n for (int i = 0; i < inputReps.length; i++) {\n Edge inputEdge = inputReps[i];\n // recurse\n findKBest(inputEdge, derivsmap);\n // get derivs\n List inputDerivs = derivsmap.get(inputEdge);\n // check index, return null if out of bounds\n if (indices[i] < inputDerivs.size()) \n \tcombo[i] = inputDerivs.get(indices[i]).sign;\n else return null;\n }\n // return edge if combo is same as input signs\n\t\tSign[] inputSigns = history.getInputs();\n if (sameSigns(inputSigns, combo)) return edge;\n // otherwise return new edge for combo\n Rule rule = history.getRule();\n List results = new ArrayList(1);\n \t((AbstractRule)rule).applyRule(combo, results); // TODO: bypass rule app for efficiency? (requires doing something about var subst)\n \tif (results.isEmpty()) return null; // (rare?)\n Sign sign = results.get(0); // assuming single result\n Edge retval = new Edge(sign); // make edge for new combo\n _numUnpackingEdges++;\n // score it\n boolean complete = (sign.getWords().size() == _size);\n retval.setScore(_signScorer.score(sign, complete));\n // done \n\t\treturn retval;\n\t}\n\t\n\t// sort and prune candidate list\n\tprivate void sortAndPrune(List cands) {\n\t\tCollections.sort(cands);\n\t\twhile (cands.size() > _pruneVal) cands.remove(cands.size()-1);\n\t}\n\t\n \n\t//-----------------------------------------------------------\n\n\t/** Saves the chart entries to the given file. */\n\tpublic void saveChartEntries(File file) throws IOException {\n\t\tObjectOutputStream out = new ObjectOutputStream(new BufferedOutputStream(new FileOutputStream(file)));\n\t\tout.writeObject(_table);\n\t\tout.flush(); out.close();\n\t}\n\t\n\t/** Loads the chart entries from the given file. */\n\tpublic void loadChartEntries(File file) throws IOException {\n\t\tObjectInputStream in = new ObjectInputStream(new BufferedInputStream(new FileInputStream(file)));\n\t\ttry {\n\t\t\t// read entries\n\t\t\t_table = (Cell[][]) in.readObject();\n\t\t\t// restore size, unpacking edge count\n\t\t\t_size = _table.length;\n\t\t\t_numUnpackingEdges = 0;\n\t\t} catch (ClassNotFoundException e) {\n\t\t\tin.close();\n throw (RuntimeException) new RuntimeException().initCause(e);\n\t\t}\n\t\tin.close();\n\t}\n\t\n\t\n\t//-----------------------------------------------------------\n \n\t/** Returns the number of entries in each cell in the chart. */\n\tpublic String toString() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tfor (int i = 0; i < _size; i++) {\n\t\t\tfor (int j = 0; j < _size; j++) {\n\t\t\t\tsb.append(get(i, j).size()).append('\\t');\n\t\t\t}\n\t\t\tsb.append('\\n');\n\t\t}\n\t\treturn sb.toString();\n\t}\n\n\t/** Prints the signs in the chart to System.out. */\n\tpublic void printChart() {\n\t\tint[] sizes = new int[_size];\n\t\tint rows = 0;\n\t\tfor (int i = 0; i < _size; i++) {\n\t\t\tfor (int j = i; j < _size; j++)\n\t\t\t\tif (get(i, j).size() > sizes[i])\n\t\t\t\t\tsizes[i] = get(i, j).size();\n\t\t\trows += sizes[i];\n\t\t}\n\n\t\tString[][] toprint = new String[rows][_size];\n\t\tString[] words = new String[_size];\n\t\tint maxwidth = 0;\n\n\t\tfor (int i = 0, row = 0; i < _size; row += sizes[i++]) {\n\t\t\tfor (int j = 0; j < _size; j++)\n\t\t\t\tfor (int s = 0; s < sizes[i]; s++) {\n\t\t\t\t\tSignHash cell = getSigns(i, j);\n\t\t\t\t\tif (i == j) words[i] = cell.asSignSet().iterator().next().getOrthography();\n\t\t\t\t\tif (cell.size() >= s + 1) {\n\t\t\t\t\t\ttoprint[row + s][j] = ((Sign) cell.toArray()[s])\n\t\t\t\t\t\t\t\t.getCategory().toString();\n\t\t\t\t\t\tif (toprint[row + s][j].length() > maxwidth)\n\t\t\t\t\t\t\tmaxwidth = toprint[row + s][j].length();\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t}\n\n\t\tint fullwidth = _size * (maxwidth + 3) - 1;\n\t\tSystem.out.print(\" \");\n\t\tfor (String w : words) {\n\t\t\tSystem.out.print(w);\n\t\t\tint pad = (maxwidth + 3) - w.length();\n\t\t\tfor (int p = 0; p < pad; p++)\n\t\t\t\tSystem.out.print(\" \");\n\t\t}\n\t\tSystem.out.print(\"|\");\n\t\tSystem.out.println();\n\t\tfor (int p = 0; p < fullwidth; p++)\n\t\t\tSystem.out.print(\"-\");\n\t\tSystem.out.print(\"| \");\n\t\tSystem.out.println();\n\n\t\tfor (int i = 0, entry = sizes[0], e = 0; i < rows; i++) {\n\t\t\tif (i == entry) {\n\t\t\t\tSystem.out.print(\"|\");\n\t\t\t\tfor (int p = 0; p < fullwidth; p++)\n\t\t\t\t\tSystem.out.print(\"-\");\n\t\t\t\tSystem.out.print(\"|\");\n\t\t\t\tSystem.out.println();\n\t\t\t\tentry += sizes[++e];\n\t\t\t}\n\t\t\tSystem.out.print(\"| \");\n\n\t\t\tfor (int j = 0; j < _size; j++) {\n\t\t\t\tint pad = 1 + maxwidth;\n\t\t\t\tif (toprint[i][j] != null) {\n\t\t\t\t\tSystem.out.print(toprint[i][j]);\n\t\t\t\t\tpad -= toprint[i][j].length();\n\t\t\t\t}\n\t\t\t\tfor (int p = 0; p < pad; p++)\n\t\t\t\t\tSystem.out.print(\" \");\n\t\t\t\tSystem.out.print(\"| \");\n\t\t\t}\n\t\t\tSystem.out.println();\n\t\t}\n\t\tSystem.out.print(\"|\");\n\t\tfor (int p = 0; p < fullwidth; p++)\n\t\t\tSystem.out.print(\"-\");\n\t\tSystem.out.print(\"| \");\n\t\tSystem.out.println();\n\t}\n}\n" }, { "path": "src/opennlp/ccg/parse/DerivationHistory.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-5 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse;\n\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.grammar.*;\n\nimport java.io.Serializable;\nimport java.util.*;\n\n/**\n * Record the steps taken in a derivation.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.12 $, $Date: 2009/12/16 22:39:32 $\n */\npublic class DerivationHistory implements Serializable, Comparable {\n\n\tprivate static final long serialVersionUID = 2867339743258182859L;\n\t\n\tprivate Sign[] _inputs;\n private Sign _output;\n private Rule _rule;\n private boolean _noHistory = false;\n private transient int _complexity = -1;\n \n /** Constructor for a sign with no prior history. */\n public DerivationHistory(Sign output) {\n _noHistory = true;\n _output = output;\n }\n \n /** Constructor for a sign created by rule. */\n public DerivationHistory(Sign[] inputs, Sign output, Rule rule) {\n _inputs = new Sign[inputs.length];\n for (int i=0; i < inputs.length; i++) {\n _inputs[i] = inputs[i];\n }\n _output = output;\n _rule = rule;\n }\n\n\n /** Returns true iff the history is empty. */\n public boolean isEmpty() { return _noHistory; }\n \n /** Returns the inputs (or null if none). */\n public Sign[] getInputs() { return _inputs; }\n \n /** Returns the output. */\n public Sign getOutput() { return _output; }\n \n /** Returns the rule. */\n public Rule getRule() { return _rule; }\n \n \n /** Returns the derivation history in vertical list form. */\n public String toString() {\n return toString(maxRuleLen());\n }\n \n // returns the derivation history given the max rule len, for alignment\n private String toString(int maxRuleLen) {\n StringBuffer sb = new StringBuffer();\n // lex item\n if (_noHistory) {\n sb.append(\"(lex) \");\n for (int i = 5; i < maxRuleLen; i++) { sb.append(' '); }\n sb.append(_output.toString()).append('\\n');\n return sb.toString();\n }\n // inputs\n for (int i=0; i < _inputs.length; i++) {\n sb.append(_inputs[i].getDerivationHistory().toString(maxRuleLen));\n }\n // type-changing rule (possibly)\n String ruleName = _rule.name();\n TypeChangingRule tcr = Grammar.theGrammar.rules.getTypeChangingRule(ruleName);\n if (tcr != null) {\n sb.append(\"(gram) \");\n for (int i = 6; i < maxRuleLen; i++) { sb.append(' '); }\n sb.append(tcr.toString()).append('\\n');\n }\n // this rule and result\n sb.append('(').append(ruleName).append(\") \");\n for (int i = (ruleName.length() + 2); i < maxRuleLen; i++) { sb.append(' '); }\n sb.append(_output.toString()).append('\\n');\n // done\n return sb.toString();\n }\n \n // returns the max length of rule names (including parens)\n private int maxRuleLen() {\n if (_noHistory) { return 6; }\n int max = 0;\n for (int i=0; i < _inputs.length; i++) {\n max = Math.max(max, _inputs[i].getDerivationHistory().maxRuleLen());\n }\n max = Math.max(max, _rule.name().length() + 2);\n return max;\n }\n \n /** Returns the complexity of the derivation, as the sum of \n the number of steps, plus the number of composition or \n substitution steps, plus the number of crossing steps. */\n public int complexity() {\n \tif (_complexity > 0) return _complexity;\n if (_noHistory) return 0;\n int retval = 1;\n String ruleName = _rule.name();\n if (ruleName.length() > 1 && (ruleName.charAt(0) == '>' || ruleName.charAt(0) == '<')) {\n if (ruleName.charAt(1) == 'B' || ruleName.charAt(1) == 'S') {\n retval++;\n if (ruleName.length() == 3 && ruleName.charAt(2) == 'x') retval++;\n }\n }\n for (int i=0; i < _inputs.length; i++) {\n retval += _inputs[i].getDerivationHistory().complexity();\n }\n _complexity = retval;\n return retval;\n }\n \n /** Returns whether the derivation contains a unary rule cycle. */\n public boolean containsCycle() {\n \tif (_noHistory || _inputs.length != 1) return false;\n \tList rulesSeen = new ArrayList(4);\n \trulesSeen.add(_rule);\n \treturn _inputs[0].getDerivationHistory().containsCycle(rulesSeen);\n }\n \n // recursive cycle check\n private boolean containsCycle(List rulesSeen) {\n \tif (_noHistory || _inputs.length != 1) return false;\n \tif (rulesSeen.contains(_rule)) return true;\n \trulesSeen.add(_rule);\n \treturn _inputs[0].getDerivationHistory().containsCycle(rulesSeen);\n }\n\n /** Recursively compares derivation histories by their complexity. */\n\tpublic int compareTo(DerivationHistory dh) {\n\t\tint c1 = complexity(); int c2 = dh.complexity();\n\t\tif (c1 < c2) return -1;\n\t\tif (c1 > c2) return 1;\n\t\tif (_noHistory) return 0;\n\t\tif (_inputs.length < dh._inputs.length) return -1;\n\t\tif (_inputs.length > dh._inputs.length) return 1;\n\t\tfor (int i=0; i < _inputs.length; i++) {\n\t\t\tint cmp = _inputs[i].getDerivationHistory().compareTo(dh._inputs[i].getDerivationHistory());\n\t\t\tif (cmp != 0) return cmp;\n\t\t}\n\t\treturn 0;\n\t}\n}\n\n" }, { "path": "src/opennlp/ccg/parse/Edge.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2007 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.parse;\r\n\r\nimport opennlp.ccg.synsem.*;\r\nimport java.util.*;\r\nimport java.io.Serializable;\r\nimport java.text.*;\r\n\r\n/**\r\n *

    \r\n * An edge is a wrapper for a sign, ie a sign together \r\n * with a score, and optionally a list of alternative edges. \r\n * A representative edge is an edge that represents (stands in for) \r\n * other edges with the same category (but different LFs) during the \r\n * chart construction process, stored in the list of alternative edges; \r\n * it is considered disjunctive when there is more than one alternative.\r\n * Note that initially a representative edge will be in its list \r\n * of alternatives, but it can be removed during pruning.\r\n *

    \r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.4 $, $Date: 2009/12/22 22:19:00 $\r\n */\r\npublic class Edge implements Serializable {\r\n\r\n\tprivate static final long serialVersionUID = 1L;\r\n\r\n\t\r\n\t/** Class for storing back-refs from signs. */\r\n\tpublic static class EdgeRef implements Serializable {\r\n\t\tprivate static final long serialVersionUID = 1L;\r\n\t\t/** The edge. */\r\n\t\tpublic final Edge edge;\r\n\t\t/** Constructor. */\r\n\t\tpublic EdgeRef(Edge edge) { this.edge = edge; }\r\n\t}\r\n\t\r\n\t/** Returns the edge associated with this sign, or null if none. */\r\n\tpublic static Edge getEdge(Sign sign) {\r\n\t\tEdgeRef eref = (EdgeRef) sign.getData(EdgeRef.class);\r\n\t\treturn (eref != null) ? eref.edge : null;\r\n\t}\r\n\t\r\n\t\r\n\t/** The sign. */\r\n protected Sign sign;\r\n \r\n /** The edge score. */\r\n protected double score;\r\n \r\n /** Word position, for lexical edges (otherwise -1). */\r\n protected int wordPos = -1;\r\n \r\n /** The alternative edges (none initially). */\r\n protected List altEdges = null;\r\n \r\n /** Saved list of alternative edges, for restoring chart after unpacking. */\r\n protected transient List savedAltEdges = null;\r\n \r\n \r\n /** Constructor (score defaults to 0.0). */\r\n public Edge(Sign sign) { this(sign, 0.0); }\r\n\r\n /** Constructor with score. */\r\n public Edge(Sign sign, double score) {\r\n this.sign = sign; this.score = score; \r\n sign.addData(new EdgeRef(this));\r\n }\r\n\r\n \r\n /** Returns the sign. */\r\n public Sign getSign() { return sign; }\r\n \r\n /** Returns the score. */\r\n public double getScore() { return score; }\r\n \r\n /** Sets the score. */\r\n public void setScore(double score) { this.score = score; }\r\n\r\n /** Returns the word position of a lexical edge (otherwise -1). */\r\n public int getWordPos() { return wordPos; }\r\n \r\n /** Sets the word position of a lexical edge. */\r\n public void setWordPos(int pos) { wordPos = pos; }\r\n \r\n /** Returns whether this edge is a representative. */\r\n public boolean isRepresentative() { return altEdges != null; }\r\n \r\n /** Returns whether this edge is disjunctive. */\r\n public boolean isDisjunctive() { return altEdges != null && altEdges.size() > 1; }\r\n \r\n /** Returns the list of alt edges, or the empty list if none. */\r\n public List getAltEdges() {\r\n if (altEdges == null) return Collections.emptyList(); \r\n else return altEdges;\r\n }\r\n \r\n /** Initializes the alt edges list with a default capacity, adding this edge. */\r\n public void initAltEdges() { initAltEdges(3); }\r\n \r\n /** Initializes the alt edges list with the given capacity, adding this edge. */\r\n public void initAltEdges(int capacity) {\r\n // check uninitialized\r\n if (altEdges != null) throw new RuntimeException(\"Alt edges already initialized!\");\r\n altEdges = new ArrayList(capacity);\r\n altEdges.add(this);\r\n }\r\n \r\n /** Replaces the alt edges, saving the current ones for later restoration. */\r\n public void replaceAltEdges(List newAlts) {\r\n \tsavedAltEdges = altEdges; \r\n \taltEdges = newAlts;\r\n }\r\n \r\n /** Recursively restores saved alt edges, if any. */\r\n public void restoreAltEdges() { \r\n \tif (savedAltEdges != null) {\r\n \t\t// restore\r\n \taltEdges = savedAltEdges;\r\n \tsavedAltEdges = null;\r\n \t// recurse\r\n \tfor (Edge alt : altEdges) {\r\n \t\tSign[] inputs = alt.sign.getDerivationHistory().getInputs();\r\n \t\tif (inputs != null) {\r\n \t\t\tfor (Sign s : inputs) \r\n \t\t\t\tgetEdge(s).restoreAltEdges();\r\n \t\t}\r\n \t}\r\n \t}\r\n }\r\n \r\n \r\n /** Returns a hash code for this edge, based on its sign. (Alternatives and the score are not considered.) */\r\n public int hashCode() {\r\n return sign.hashCode() * 23;\r\n }\r\n \r\n /** Returns a hash code for this edge based on the sign's surface words. (Alternatives and the score are not considered.) */\r\n public int surfaceWordHashCode() {\r\n \treturn sign.surfaceWordHashCode() * 23;\r\n }\r\n \r\n /** Returns whether this edge equals the given object. (Alternatives and the score are not considered.) */\r\n public boolean equals(Object obj) {\r\n if (obj == this) return true;\r\n if (!(obj instanceof Edge)) return false;\r\n Edge edge = (Edge) obj;\r\n return sign.equals(edge.sign);\r\n }\r\n \r\n /** Returns whether this edge equals the given object based on the sign's surface words. (Alternatives and the score are not considered.) */\r\n public boolean surfaceWordEquals(Object obj) {\r\n if (obj == this) return true;\r\n if (!(obj instanceof Edge)) return false;\r\n Edge edge = (Edge) obj;\r\n return sign.surfaceWordEquals(edge.sign);\r\n }\r\n \r\n \r\n /**\r\n * Returns a string for the edge in the format\r\n * [score] orthography :- category. \r\n */\r\n public String toString() {\r\n StringBuffer sbuf = new StringBuffer();\r\n if (score >= 0.001 || score == 0.0) sbuf.append(\"[\" + nf3.format(score) + \"] \");\r\n else sbuf.append(\"[\" + nfE.format(score) + \"] \");\r\n sbuf.append(sign.toString());\r\n return sbuf.toString();\r\n }\r\n \r\n // formats to three decimal places\r\n private static final NumberFormat nf3 = initNF3();\r\n private static NumberFormat initNF3() { \r\n NumberFormat f = NumberFormat.getInstance();\r\n f.setMinimumIntegerDigits(1);\r\n f.setMinimumFractionDigits(3);\r\n f.setMaximumFractionDigits(3);\r\n return f;\r\n }\r\n \r\n // formats to \"0.##E0\"\r\n private static final NumberFormat nfE = new DecimalFormat(\"0.##E0\");\r\n}\r\n" }, { "path": "src/opennlp/ccg/parse/EdgeHash.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-7 Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse;\n\nimport gnu.trove.*;\nimport java.util.*;\n\n/**\n * A set of edges, unique up to surface words.\n * Edges whose signs have lower derivational complexity are kept during insertion.\n * NB: This is just like EdgeHash in the realize package, except that \n * it deals with parse edges.\n *\n * @author Michael White\n * @version $Revision: 1.1 $, $Date: 2007/12/20 05:51:10 $\n */\npublic class EdgeHash extends THashSet {\n\n\tprivate static final long serialVersionUID = 1L;\n\t\n\t/** Hashing strategy that uses Edge's surfaceWordHashCode and surfaceWordEquals methods. */\n protected static TObjectHashingStrategy surfaceWordHashingStrategy = new TObjectHashingStrategy() {\n\t\tprivate static final long serialVersionUID = 1L;\n\t\tpublic int computeHashCode(java.lang.Object o) {\n return ((Edge)o).surfaceWordHashCode();\n }\n public boolean equals(java.lang.Object o1, java.lang.Object o2) {\n return ((Edge)o1).surfaceWordEquals((Edge)o2);\n }\n };\n\n /** Default constructor. */\n public EdgeHash() { super(surfaceWordHashingStrategy); }\n\n /**\n * Returns this as a set of edges.\n */\n @SuppressWarnings(\"unchecked\")\n\tpublic Set asEdgeSet() { return (Set) this; }\n\n /**\n * Adds an edge, keeping the one whose sign has lower derivational complexity \n * if there is an equivalent one there already; returns the old\n * edge if it was displaced, the new edge if there was no equivalent \n * old edge, or null if the edge was not actually added.\n * iff the edge is actually inserted.\n */\n public Edge insert(Edge edge) {\n int pos = index(edge);\n if (pos >= 0) {\n Edge oldEdge = (Edge) _set[pos];\n if (oldEdge == edge) return null; \n int complexity = edge.sign.getDerivationHistory().complexity();\n int oldComplexity = oldEdge.sign.getDerivationHistory().complexity();\n if (complexity < oldComplexity) {\n \t_set[pos] = edge; return oldEdge;\n }\n else return null;\n }\n else {\n \tadd(edge); return edge;\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/ParseException.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-9 Jason Baldridge, Gann Bierner and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse;\n\n/**\n * Any exception having to do with reading the lexicon or rules, etc.\n * \n * @author Gann Bierner\n * @version $Revision: 1.3 $, $Date: 2009/12/20 18:54:41 $\n */\npublic class ParseException extends Exception {\n\t\n\tprivate static final long serialVersionUID = 1L;\n\n\t/** Time limit exceeded message. */\n\tpublic static final String TIME_LIMIT_EXCEEDED = \"Time limit exceeded\";\n\t\n\t/** Edge limit exceeded message. */\n\tpublic static final String EDGE_LIMIT_EXCEEDED = \"Edge limit exceeded\";\n\n\t/**\n\t * Class constructor\n\t * \n\t * @param s\n\t * the error message\n\t */\n\tpublic ParseException(String s) {\n\t\tsuper(s); \n\t}\n\n\tpublic String toString() { return getMessage(); }\n}\n" }, { "path": "src/opennlp/ccg/parse/Parser.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2003-9 Jason Baldridge, Gann Bierner and Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.parse;\r\n\r\nimport opennlp.ccg.TextCCG;\r\nimport opennlp.ccg.lexicon.*;\r\nimport opennlp.ccg.synsem.*;\r\nimport opennlp.ccg.grammar.*;\r\nimport opennlp.ccg.hylo.EPsScorer;\r\nimport opennlp.ccg.hylo.HyloHelper;\r\nimport opennlp.ccg.hylo.Nominal;\r\nimport opennlp.ccg.unify.*;\r\nimport opennlp.ccg.util.Pair;\r\n\r\nimport java.util.*;\r\nimport java.util.prefs.Preferences;\r\n\r\n/**\r\n * The parser is a CKY chart parser for CCG, optionally \r\n * with iterative beta-best supertagging and n-best output.\r\n *\r\n * @author Jason Baldridge\r\n * @author Gann Bierner\r\n * @author Michael White\r\n * @version $Revision: 1.38 $, $Date: 2011/08/27 19:27:00 $\r\n */\r\npublic class Parser \r\n{\r\n /** Preference key for time limit on parsing. */\r\n public static final String PARSE_TIME_LIMIT = \"Parse Time Limit\";\r\n \r\n /** A constant indicating no time limit on parsing. */\r\n public static final int NO_TIME_LIMIT = 0;\r\n\r\n /** Preference key for edge limit on parsing. */\r\n public static final String PARSE_EDGE_LIMIT = \"Parse Edge Limit\";\r\n \r\n /** A constant indicating no edge limit on parsing. */\r\n public static final int NO_EDGE_LIMIT = 0;\r\n\r\n /** Preference key for pruning the number of signs kept per equivalence class. */\r\n public static final String PARSE_PRUNING_VALUE = \"Parse Pruning Value\";\r\n\r\n /** Preference key for pruning the number of edges kept per cell. */\r\n public static final String PARSE_CELL_PRUNING_VALUE = \"Parse Cell Pruning Value\";\r\n\r\n /** A constant indicating no pruning of signs per equivalence class. */\r\n public static final int NO_PRUNING = 0;\r\n\r\n /** Preference key for whether to use lazy unpacking. */\r\n public static final String PARSE_LAZY_UNPACKING = \"Parse Lazy Unpacking\";\r\n \r\n\t/** The grammar. */\r\n\tpublic final Grammar grammar;\r\n\t\r\n /** The lexicon used to create edges. */ \r\n public final Lexicon lexicon;\r\n \r\n /** The rules used to create edges. */\r\n public final RuleGroup rules;\r\n \r\n /** Flag for whether to show the chart for failed parses. */\r\n public boolean debugParse = false;\r\n \r\n\t/** The sign scorer (or null if none). */\r\n\tprotected SignScorer signScorer = null;\r\n\t\r\n\t/** The \"n\" for n-best pruning. (Default is none.) */\r\n\tprotected int pruneVal = -1;\r\n\t\r\n\t/** The cell pruning value. (Default is none.) */\r\n\tprotected int cellPruneVal = -1;\r\n\t\r\n\t/** The lazy unpacking flag. (Default is none.) */\r\n\tprotected Boolean lazyUnpacking = null;\r\n\t\r\n /** Supertagger to use. (Default is none.) */\r\n protected Supertagger supertagger = null;\r\n \r\n /** Flag for whether to use the supertagger in the most-to-least restrictive direction. */\r\n protected boolean stMostToLeastDir = true;\r\n \r\n /** Time limit in milliseconds. (Default is none.) */\r\n protected int timeLimit = -1; \r\n \r\n /** Edge limit. (Default is none.) */\r\n protected int edgeLimit = -1; \r\n \r\n // start time for chart construction\r\n private long startTime = 0;\r\n \r\n // lex lookup time\r\n private int lexTime = 0;\r\n \r\n // parse time\r\n private int parseTime = 0;\r\n \r\n // chart construction time\r\n private int chartTime = 0;\r\n \r\n // unpacking time\r\n private int unpackingTime = 0;\r\n \r\n // time limit to use\r\n private int timeLimitToUse = NO_TIME_LIMIT;\r\n \r\n // edge limit to use\r\n private int edgeLimitToUse = NO_EDGE_LIMIT;\r\n \r\n // pruning value to use\r\n private int pruneValToUse = NO_PRUNING;\r\n \r\n // pruning value to use\r\n private int cellPruneValToUse = NO_PRUNING;\r\n \r\n // lazy unpacking flag to use\r\n private boolean lazyUnpackingToUse = true;\r\n \r\n // current chart\r\n private Chart chart = null;\r\n \r\n // parse results\r\n private ArrayList result;\r\n\r\n // parse scores\r\n private ArrayList scores;\r\n\r\n // flag for whether to glue fragments currently\r\n private boolean gluingFragments = false;\r\n \r\n /** Constructor. */\r\n public Parser(Grammar grammar) {\r\n \tthis.grammar = grammar;\r\n this.lexicon = grammar.lexicon;\r\n this.rules = grammar.rules;\r\n }\r\n \r\n\t/** Sets the sign scorer. */\r\n\tpublic void setSignScorer(SignScorer signScorer) { this.signScorer = signScorer; }\r\n\t\r\n\t/** Sets the time limit. */\r\n\tpublic void setTimeLimit(int timeLimit) { this.timeLimit = timeLimit; }\r\n\t\r\n\t/** Sets the edge limit. */\r\n\tpublic void setEdgeLimit(int edgeLimit) { this.edgeLimit = edgeLimit; }\r\n\t\r\n\t/** Sets the n-best pruning val. */\r\n\tpublic void setPruneVal(int n) { pruneVal = n; }\r\n\t\r\n\t/** Sets the cell pruning val. */\r\n\tpublic void setCellPruneVal(int n) { cellPruneVal = n; }\r\n\t\r\n\t/** Sets the lazy unpacking flag. */\r\n\tpublic void setLazyUnpacking(Boolean b) { this.lazyUnpacking = b; }\r\n\t\r\n\t/** Sets the supertagger. */\r\n\tpublic void setSupertagger(Supertagger supertagger) { this.supertagger = supertagger; }\r\n\r\n\t/** Sets the supertagger most-to-least restrictive direction flag. */\r\n\tpublic void setSupertaggerMostToLeastRestrictiveDirection(boolean bool) {\r\n\t\tstMostToLeastDir = bool;\r\n\t}\r\n\t\r\n /**\r\n * Parses a string.\r\n *\r\n * @param s the string\r\n * @exception ParseException thrown if a parse can't be found for the\r\n * entire string\r\n */\r\n public void parse(String s) throws ParseException {\r\n // tokenize\r\n List words = lexicon.tokenizer.tokenize(s);\r\n // parse words\r\n parse(words);\r\n }\r\n \r\n /**\r\n * Parses a list of words.\r\n */\r\n public void parse(List words) throws ParseException {\r\n \t// set up timing: use limit from prefs unless explicitly set\r\n\t\tPreferences prefs = Preferences.userNodeForPackage(TextCCG.class);\r\n \tif (timeLimit >= 0) timeLimitToUse = timeLimit;\r\n \telse timeLimitToUse = prefs.getInt(PARSE_TIME_LIMIT, NO_TIME_LIMIT);\r\n \tif (edgeLimit >= 0) edgeLimitToUse = edgeLimit;\r\n \telse edgeLimitToUse = prefs.getInt(PARSE_EDGE_LIMIT, NO_EDGE_LIMIT);\r\n \tif (pruneVal >= 0) pruneValToUse = pruneVal; \r\n \telse pruneValToUse = prefs.getInt(PARSE_PRUNING_VALUE, NO_PRUNING);\r\n \tif (cellPruneVal >= 0) cellPruneValToUse = cellPruneVal; \r\n \telse cellPruneValToUse = prefs.getInt(PARSE_CELL_PRUNING_VALUE, NO_PRUNING);\r\n \tif (lazyUnpacking != null) lazyUnpackingToUse = lazyUnpacking;\r\n \telse lazyUnpackingToUse = prefs.getBoolean(PARSE_LAZY_UNPACKING, true);\r\n \t// supertagger case: iterative beta-best\r\n \tif (supertagger != null) {\r\n \t\tparseWithSupertagger(words);\r\n \t\treturn;\r\n \t}\r\n \t// otherwise just once\r\n try {\r\n \t// init\r\n \tlong lexStartTime = System.currentTimeMillis();\r\n UnifyControl.startUnifySequence();\r\n // get entries for each word\r\n List entries = new ArrayList(words.size());\r\n for (Word w : words) {\r\n \tentries.add(lexicon.getSignsFromWord(w));\r\n }\r\n lexTime = (int) (System.currentTimeMillis() - lexStartTime);\r\n // do parsing\r\n parseEntries(entries);\r\n } catch (LexException e) {\r\n \tsetGiveUpTime();\r\n \tString msg = \"Unable to retrieve lexical entries:\\n\\t\" + e.toString();\r\n \tif (debugParse) System.out.println(msg);\r\n \tthrow new ParseException(msg);\r\n }\r\n\t\tcatch (ParseException e) {\r\n \tsetGiveUpTime();\r\n\t\t\t// show chart for failed parse if apropos\r\n\t\t\tif (debugParse) {\r\n\t\t\t\tSystem.out.println(e);\r\n\t\t\t\tSystem.out.println(\"Chart for failed parse:\");\r\n\t\t\t\tchart.printChart();\r\n\t\t\t}\r\n \t// rethrow\r\n\t\t\tthrow e;\r\n\t\t}\r\n }\r\n\r\n // iterative beta-best parsing\r\n private void parseWithSupertagger(List words) throws ParseException {\r\n \t// set supertagger in lexicon\r\n \tgrammar.lexicon.setSupertagger(supertagger);\r\n \t// ensure gluing off\r\n \tgluingFragments = false;\r\n \t// reset beta\r\n \tif (stMostToLeastDir) supertagger.resetBeta();\r\n \telse supertagger.resetBetaToMax();\r\n \t// loop\r\n \tboolean done = false;\r\n \twhile (!done) {\r\n \t\ttry {\r\n \t \t// init\r\n \tlong lexStartTime = System.currentTimeMillis();\r\n \t UnifyControl.startUnifySequence();\r\n // get filtered entries for each word\r\n List entries = new ArrayList(words.size());\r\n supertagger.mapWords(words);\r\n for (int i=0; i < words.size(); i++) {\r\n \tsupertagger.setWord(i);\r\n Word word = words.get(i);\r\n \t\tentries.add(lexicon.getSignsFromWord(word));\r\n }\r\n lexTime = (int) (System.currentTimeMillis() - lexStartTime);\r\n // do parsing\r\n parseEntries(entries);\r\n // done\r\n done = true;\r\n \t// reset supertagger in lexicon, turn gluing off\r\n \tgrammar.lexicon.setSupertagger(null);\r\n \tgluingFragments = false;\r\n \t\t}\r\n \t\tcatch (LexException e) {\r\n \t\t\t// continue if more betas\r\n \t\t\tif (stMostToLeastDir && supertagger.hasMoreBetas()) {\r\n \t\t\t\tsupertagger.nextBeta();\r\n \t\t\t}\r\n \t\t\t// otherwise give up\r\n \t\t\telse {\r\n \t \tsetGiveUpTime();\r\n \t \t// reset supertagger in lexicon, turn gluing off\r\n \t \tgrammar.lexicon.setSupertagger(null);\r\n \tgluingFragments = false;\r\n \t \t// throw parse exception\r\n \t \tString msg = \"Unable to retrieve lexical entries:\\n\\t\" + e.toString();\r\n \t \tif (debugParse) System.out.println(msg);\r\n \t throw new ParseException(msg);\r\n \t\t\t}\r\n \t\t}\r\n \t\tcatch (ParseException e) {\r\n \t\t\t// check if limits exceeded\r\n \t\t\tboolean outwith = e.getMessage() == ParseException.EDGE_LIMIT_EXCEEDED || \r\n\t\t\t\t\te.getMessage() == ParseException.TIME_LIMIT_EXCEEDED;\r\n \t\t\t// continue if more betas and limits not exceeded\r\n \t\t\tif (stMostToLeastDir && supertagger.hasMoreBetas() && !outwith) \r\n \t\t\t\tsupertagger.nextBeta();\r\n \t\t\t// or if limits exceeded and moving in the opposite direction\r\n \t\t\telse if (!stMostToLeastDir && supertagger.hasLessBetas() && outwith)\r\n \t\t\t\tsupertagger.previousBeta();\r\n \t\t\t// otherwise try glue rule, unless already on\r\n \t\t\telse if (!gluingFragments) {\r\n \t\t\t\tsupertagger.resetBeta(); // may as well use most restrictive supertagger setting with glue rule\r\n \t\t\t\tgluingFragments = true;\r\n \t\t\t}\r\n \t\t\t// otherwise give up\r\n \t\t\telse {\r\n \t \tsetGiveUpTime();\r\n \t\t\t\t// show chart for failed parse if apropos\r\n \t\t\t\tif (debugParse) {\r\n \t\t\t\t\tSystem.out.println(e);\r\n\t \t\t\t\tSystem.out.println(\"Chart for failed parse:\");\r\n\t \t\t\t\tchart.printChart();\r\n \t\t\t\t}\r\n \t \t// reset supertagger in lexicon, turn gluing off\r\n \t \tgrammar.lexicon.setSupertagger(null);\r\n \tgluingFragments = false;\r\n \t \t// rethrow\r\n \t\t\t\tthrow e;\r\n \t\t\t}\r\n \t\t}\r\n \t}\r\n }\r\n \r\n /**\r\n * Returns the results of the parse.\r\n */\r\n public List getResult() { return result; }\r\n\r\n /**\r\n * Returns the corresponding scores for the results of the parse.\r\n */\r\n public List getScores() { return scores; }\r\n\r\n\t/** Returns the edge count prior to unpacking. */\r\n\tpublic int edgeCount() { return (chart != null) ? chart.edgeCount() : 0; }\r\n\t\r\n\t/** Returns the edge count while unpacking. */\r\n\tpublic int unpackingEdgeCount() { return (chart != null) ? chart.unpackingEdgeCount() : 0; }\r\n\r\n\t/** Returns the max cell size prior to unpacking. */\r\n\tpublic int maxCellSize() { return (chart != null) ? chart.maxCellSize() : 0; }\r\n\t\r\n\t/** Returns the lexical lookup time for the latest parse. */\r\n\tpublic int getLexTime() { return lexTime; }\r\n\t\r\n\t/** Returns the overall parse time (but excluding lex lookup) for the latest parse. */\r\n\tpublic int getParseTime() { return parseTime; }\r\n\t\r\n\t/** Returns the time spent constructing the chart. */\r\n\tpublic int getChartTime() { return chartTime; }\r\n\t\r\n\t/** Returns the time spent unpacking. */\r\n\tpublic int getUnpackingTime() { return unpackingTime; }\r\n\t\r\n\t/** Returns the supertagger's final beta value (or 0 if none). */\r\n\tpublic double getSupertaggerBeta() {\r\n\t\treturn (supertagger != null) ? supertagger.getCurrentBetaValue() : 0;\r\n\t}\r\n\t\r\n\t\r\n // parses from lex entries\r\n private void parseEntries(List entries) throws ParseException {\r\n \tstartTime = System.currentTimeMillis();\r\n // set up chart\r\n initializeChart(entries);\r\n if (signScorer != null) chart.setSignScorer(signScorer);\r\n chart.setPruneVal(pruneValToUse);\r\n chart.setTimeLimit(timeLimitToUse);\r\n chart.setStartTime(startTime);\r\n chart.setEdgeLimit(edgeLimitToUse);\r\n chart.setCellLimit(cellPruneValToUse);\r\n // do parsing\r\n parse(entries.size());\r\n }\r\n \r\n // initialize the chart\r\n private void initializeChart(List entries) {\r\n chart = new Chart(entries.size(), rules);\r\n for (int i=0; i < entries.size(); i++) {\r\n SignHash wh = entries.get(i);\r\n for (Sign sign : wh.getSignsSorted()) {\r\n Category cat = sign.getCategory();\r\n UnifyControl.reindex(cat);\r\n chart.insert(i, i, sign);\r\n }\r\n }\r\n }\r\n \r\n // actual CKY parsing\r\n private void parse(int size) throws ParseException {\r\n \t// fill in chart\r\n for (int i=0; i=0; i--) {\r\n for (int k=i; k=0; i--) {\r\n for (int k=i; k();\r\n scores = new ArrayList();\r\n // unpack top\r\n List unpacked = (lazyUnpackingToUse) \r\n \t? chart.lazyUnpack(0,size - 1) \r\n\t\t\t: chart.unpack(0, size - 1);\r\n // add signs for unpacked edges\r\n for (Edge edge : unpacked) {\r\n \tresult.add(edge.sign);\r\n \tscores.add(edge.score);\r\n }\r\n // check non-empty\r\n if (result.size() == 0) {\r\n throw new ParseException(\"Unable to parse\");\r\n }\r\n }\r\n \r\n // set parse time when giving up\r\n private void setGiveUpTime() {\r\n chartTime = (int) (System.currentTimeMillis() - startTime);\r\n parseTime = chartTime; \r\n \tunpackingTime = 0;\r\n }\r\n \r\n /**\r\n * Adds the supertagger log probs to the lexical signs of the gold standard parse.\r\n */\r\n public void addSupertaggerLogProbs(Sign gold) {\r\n \tList words = gold.getWords();\r\n supertagger.mapWords(words);\r\n addSupertaggerLogProbs(gold, gold);\r\n for (int i=0; i < words.size(); i++) {\r\n \tsupertagger.setWord(i);\r\n }\r\n }\r\n\r\n // recurses through derivation, adding lex log probs to lexical signs\r\n private void addSupertaggerLogProbs(Sign gold, Sign current) {\r\n \t// lookup and add log prob for lex sign\r\n \tif (current.isLexical()) {\r\n \t\tsupertagger.setWord(gold.wordIndex(current));\r\n \t\tMap stags = supertagger.getSupertags();\r\n \t\tDouble lexprob = stags.get(current.getSupertag());\r\n \t\tif (lexprob != null) {\r\n \t\t\tcurrent.addData(new SupertaggerAdapter.LexLogProb((float) Math.log10(lexprob)));\r\n \t\t}\r\n \t}\r\n \t// otherwise recurse\r\n \telse {\r\n \t\tSign[] inputs = current.getDerivationHistory().getInputs();\r\n \t\tfor (Sign s : inputs) addSupertaggerLogProbs(gold, s);\r\n \t}\r\n }\r\n \r\n /**\r\n * Returns the oracle best sign among those in the n-best list for the given LF, \r\n * using the f-score on all EPs, together with a flag indicating whether the gold LF \r\n * was found (as indicated by an f-score of 1.0).\r\n * NB: It would be better to return the forest oracle, but the nominal conversion would \r\n * be tricky to do correctly. \r\n */\r\n public Pair oracleBest(LF goldLF) {\r\n \tSign retval = null; double bestF = 0.0;\r\n \tfor (Sign sign : result) {\r\n Category cat = sign.getCategory().copy();\r\n Nominal index = cat.getIndexNominal();\r\n LF parsedLF = cat.getLF();\r\n if (parsedLF != null) {\r\n\t index = HyloHelper.convertNominals(parsedLF, sign, index);\r\n\t \t\tEPsScorer.Results score = EPsScorer.score(parsedLF, goldLF);\r\n\t \t\tif (score.fscore > bestF) {\r\n\t \t\t\tretval = sign;\r\n\t \t\t\tbestF = score.fscore;\r\n\t \t\t}\r\n }\r\n \t}\r\n \treturn new Pair(retval, (bestF == 1.0));\r\n }\r\n}\r\n\r\n" }, { "path": "src/opennlp/ccg/parse/Supertagger.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2009 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.parse;\r\n\r\nimport opennlp.ccg.lexicon.*;\r\nimport java.util.*;\r\n\r\n/**\r\n * A parsing supertagger must extend the SupertaggerAdapter interface for plugging a \r\n * supertagger into the lexicon. It must additionally support methods for \r\n * calculating and caching contextual supertagging assignments, so that \r\n * supertags can be retrieved just based on the current word index.\r\n * \r\n * @author Michael White\r\n * @version $Revision: 1.3 $, $Date: 2010/12/08 15:24:26 $\r\n */\r\npublic interface Supertagger extends SupertaggerAdapter {\r\n\t\r\n\t/**\r\n\t * Maps the given words to their predicted categories, \r\n\t * so that the beta-best categories can be returned by calls to setWord\r\n\t * and getSupertags.\r\n\t */\r\n\tpublic void mapWords(List words);\r\n\t\r\n\t/**\r\n\t * Sets the current word to the one with the given index, \r\n\t * so that the beta-best categories for it can be returned by a call to \r\n\t * getSupertags.\r\n\t */\r\n\tpublic void setWord(int index);\r\n}\r\n" }, { "path": "src/opennlp/ccg/parse/postagger/BasicPOSTagger.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse.postagger;\nimport opennlp.ccg.parse.postagger.ml.POSPriorModel;\nimport java.io.BufferedReader;\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.io.InputStreamReader;\nimport java.io.OutputStreamWriter;\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.Comparator;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.parse.tagger.TaggedWord;\nimport opennlp.ccg.parse.tagger.ml.MaxentModel;\nimport opennlp.ccg.parse.supertagger.ml.FeatureExtractor;\nimport opennlp.ccg.util.Pair;\nimport opennlp.ccg.parse.tagger.io.SRILMFactoredBundleCorpusIterator;\nimport opennlp.ccg.parse.tagger.util.ResultSink;\nimport opennlp.ccg.parse.tagger.sequencescoring.SequenceScorer;\nimport opennlp.ccg.parse.tagger.Constants;\n\n/**\n * A non-dummy POS tagger.\n * \n * @author Dennis N. Mehay\n */\npublic class BasicPOSTagger extends POSTagger { \n \n private FeatureExtractor posFex = null;\n private MaxentModel tagMod = null; \n \n private static final Comparator> comp = new Comparator>() {\n public int compare(Pair pr0, Pair pr1) { \n // sorts descending by prob (the double member of the pair).\n if(pr0.a == pr1.a) { return 0; } else if (pr0.a < pr1.a) { return 1; } else { return -1; } \n }\n };\n \n public BasicPOSTagger(MaxentModel tagMod, FeatureExtractor posFex, String tagSequenceModel) {\n this.posFex = posFex;\n this.tagMod = tagMod;\n int ord = SequenceScorer.findOrder(tagSequenceModel);\n try {\n posSeqMod = new SequenceScorer(ord, tagSequenceModel);\n // set the search algorithm.\n posSeqMod.setAlgorithm(Constants.TaggingAlgorithm.FORWARDBACKWARD);\n // set the search beam width\n posSeqMod.setSearchBeam(5);\n \n } catch (IOException ex) {\n Logger.getLogger(BasicPOSTagger.class.getName()).log(Level.SEVERE, null, ex);\n }\n }\n \n public List tagSentence(List sentence) { \n List result = new ArrayList(sentence.size());\n \n // the prob-string taggings (to be filtered, etc. before adding them to the taggings of the TaggedWord list). \n List>> taggings = new ArrayList>>(sentence.size());\n \n Map sentMap = new HashMap(sentence.size());\n int ind = 0; \n for(Word w : sentence) { \n sentMap.put(ind++, new TaggedWord(w)); \n }\n List>> ftss = posFex.getSentenceFeatures(sentMap);\n \n double[] distro = null; \n \n int wordIndex = 0;\n for(Collection> fts : ftss) {\n \n distro = tagMod.eval(fts);\n List> distroList = new ArrayList>(distro.length);\n ind = 0; for(double prob : distro) { distroList.add(new Pair(prob, ind++)); } \n Collections.sort(distroList, comp); \n // widen beta a little bit (we're going to do some fwd-bwd rescoring inp a minute, but we don't\n // want to do the fwd-bwd alg over ALL possible tags -- too inefficient).\n List> tagging = new ArrayList>(distro.length);\n double best = distroList.get(0).a;\n double widenedBeta = beta/8; \n \n String goldPOS = sentence.get(wordIndex).getPOS();\n \n for(Pair outcome : distroList) { \n if( (outcome.a >= (widenedBeta * best)) || (includeGold && tagMod.getOutcome(outcome.b).equals(goldPOS)) ) {\n tagging.add(new Pair(outcome.a, tagMod.getOutcome(outcome.b))); \n } else {\n if(!includeGold) { // if not still potentially fishing for a gold POS tag, then break (they're in sorted order).\n break;\n } \n }\n }\n taggings.add(tagging);\n wordIndex++;\n }\n // rescore using forward-backward.\n taggings = posSeqMod.rescoreSequence(taggings); \n // add these rescored taggings to the list of TaggedWord's.\n int wInd = 0;\n for(List> tagging : taggings) {\n TaggedWord tmpWd = new TaggedWord(sentence.get(wInd++));\n tmpWd.setPOSTagging(tagging); \n result.add(tmpWd);\n }\n // now filter down to the beta-best.\n return betaBestFilter(result); \n }\n \n \n public static void main(String[] args) throws IOException {\n String usage = \"\\nBasicPOSTagger -c (-i [defaults to ]) (-o [defaults to ])\\n\"+\n \" (-e [test tagger; assumes input is gold-standard corpus])\\n\";\n if (args.length > 0 && args[0].equals(\"-h\")) {\n System.out.println(usage);\n System.exit(0);\n }\n\n SRILMFactoredBundleCorpusIterator inp = null;\n BufferedWriter out = null;\n \n try {\n String inputCorp = \"\", output = \"\", \n configFile = null;\n \n boolean test = false;\n\n for (int i = 0; i < args.length; i++) {\n if (args[i].equals(\"-i\")) { inputCorp = args[++i]; continue; }\n if (args[i].equals(\"-o\")) { output = args[++i]; continue; }\n if (args[i].equals(\"-e\")) { test = true; continue; }\n if (args[i].equals(\"-c\")) { configFile = args[++i]; continue; }\n System.out.println(\"Unrecognized option: \" + args[i]);\n }\n\n ResultSink rs = new ResultSink(ResultSink.ResultSinkType.POSTAG);\n \n try { \n inp = new SRILMFactoredBundleCorpusIterator(\n (inputCorp.equals(\"\")) ? \n new BufferedReader(new InputStreamReader(System.in)) : \n new BufferedReader(new FileReader(new File(inputCorp)))); \n } catch (FileNotFoundException ex) {\n System.err.print(\"Input corpus \" + inputCorp + \" not found. Exiting...\");\n Logger.getLogger(POSPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n System.exit(-1);\n } \n\n try {\n out = (output.equals(\"\")) ? new BufferedWriter(new OutputStreamWriter(System.out)) : new BufferedWriter(new FileWriter(new File(output)));\n } catch (IOException ex) {\n System.err.print(\"Output file \" + output + \" not found. Exiting...\");\n Logger.getLogger(POSPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n System.exit(-1);\n }\n\n POSTagger post = POSTagger.posTaggerFactory(configFile);\n \n for (List inLine : inp) {\n List taggedSent = post.tagSentence(inLine);\n List>> sentTagging = new ArrayList>>(taggedSent.size());\n for(TaggedWord tw : taggedSent) { sentTagging.add(tw.getPOSTagging()); }\n if(test) { rs.addSent(sentTagging, inLine); }\n out.write(\"\" + System.getProperty(\"line.separator\"));\n for(TaggedWord tw : taggedSent) {\n out.write(tw.getForm());\n for(Pair tg : tw.getPOSTagging()) {\n out.write(\"\\t\" + tg.b + \"\\t\" + tg.a);\n }\n out.write(System.getProperty(\"line.separator\"));\n }\n out.write(\"\" + System.getProperty(\"line.separator\"));\n }\n out.flush();\n\n if(test) { System.err.println(rs.report()); }\n } catch(Throwable t) {\n t.printStackTrace();\n } finally {\n try { \n inp.close();\n out.close();\n } catch (IOException ex) {\n Logger.getLogger(POSPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n }\n }\n }\n}" }, { "path": "src/opennlp/ccg/parse/postagger/DummyPOSTagger.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2010 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse.postagger;\nimport java.util.ArrayList;\nimport java.util.List;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.parse.tagger.TaggedWord;\nimport opennlp.ccg.util.Pair;\n \n/**\n * A \"DummyPOSTagger\" simply grabs the single POS tag in the Words themselves\n * and puts them into a TaggedWord with a singleton list of probability 1.0 POS\n * tags.\n * \n * @author Dennis N. Mehay\n */\npublic class DummyPOSTagger extends POSTagger { \n\n public List tagSentence(List sentence) { \n List result = new ArrayList(sentence.size());\n for(Word w : sentence) { \n List> tmpTagging = new ArrayList>(1);\n tmpTagging.add(new Pair(1.0,w.getPOS())); \n TaggedWord tmp = new TaggedWord(w);\n tmp.setPOSTagging(tmpTagging);\n result.add(tmp); \n }\n return result; \n }\n}" }, { "path": "src/opennlp/ccg/parse/postagger/POSTagSequenceGetter.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse.postagger;\n\nimport java.io.BufferedReader;\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.util.List;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.parse.tagger.io.SRILMFactoredBundleCorpusIterator;\n\n/**\n * @author Dennis N. Mehay\n */\npublic class POSTagSequenceGetter {\n public static void main(String[] args) throws FileNotFoundException, IOException {\n String usage = \"\\nPOSTagSequenceGetter -i -o \\n\";\n String input = null, output = null;\n if(args == null || args.length == 0 || args[0].equals(\"-h\")) {\n System.err.println(usage);\n System.exit(0);\n }\n for(int i = 0; i < args.length; i++) {\n if(args[i].equals(\"-i\")) { input = args[++i]; continue; }\n if(args[i].equals(\"-o\")) { output = args[++i]; continue; }\n System.err.println(\"unknown command-line option: \" + args[i]);\n }\n \n BufferedReader in = new BufferedReader(new FileReader(new File(input)));\n SRILMFactoredBundleCorpusIterator corp = new SRILMFactoredBundleCorpusIterator(in);\n BufferedWriter out = new BufferedWriter(new FileWriter(new File(output)));\n \n \n for(List sent : corp) {\n out.write(\" \");\n for(Word w : sent) {\n out.write(w.getPOS()+\" \");\n }\n out.write(\"\"+System.getProperty(\"line.separator\"));\n }\n out.close();\n }\n \n}\n" }, { "path": "src/opennlp/ccg/parse/postagger/POSTagger.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2010 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.postagger;\n\nimport opennlp.ccg.parse.tagger.Constants.TaggingAlgorithm;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\nimport opennlp.ccg.parse.tagger.TaggedWord;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.parse.postagger.ml.POSPriorModel;\nimport opennlp.ccg.parse.postagger.ml.POSTagFex;\nimport opennlp.ccg.parse.tagger.ml.MaxentModel;\nimport opennlp.ccg.parse.tagger.ml.ZLMEM;\nimport opennlp.ccg.parse.tagger.sequencescoring.SequenceScorer;\nimport opennlp.ccg.util.Pair;\nimport opennlp.ccg.parse.tagger.util.ConfigFileProcessor;\n\n/**\n * Interface for POS taggers.\n * \n * @author Dennis N. Mehay\n */\npublic abstract class POSTagger {\n\n protected SequenceScorer posSeqMod = null;\n public TaggingAlgorithm alg = TaggingAlgorithm.FORWARDBACKWARD;\n public double beta = 1.0;\n protected boolean includeGold = false;\n\n /** In goes a list of {@code Word}s; out comes a list of {@code TaggedWord}s*/\n public abstract List tagSentence(List sentence);\n\n /** Set the tagging algorithm. */ \n public void setTaggingAlgorithm(TaggingAlgorithm newAlg) { \n alg = newAlg; \n posSeqMod.setAlgorithm(alg);\n }\n \n /** Set the beam width (by default, it's 1.0 -- i.e., single-best). */\n public void setBeta(double beta) {\n this.beta = beta;\n }\n\n /** Say whether or not we will include gold tags (e.g., for training). */\n public void setIncludeGold(boolean includeGoldOrNot) { includeGold = includeGoldOrNot; }\n \n /** \n * Filter the POS tags by the beta filter and return the (potentially) trimmed-down results. \n * It is assumed that the tags of the tagged word are sorted in descending order of\n * probability.\n */\n public List betaBestFilter(List sentence) {\n List res = new ArrayList(sentence.size());\n for (TaggedWord tw : sentence) {\n Word w = tw.getWord();\n double best = tw.getPOSTagging().get(0).a;\n int endIndex = 0;\n for (Pair tagging : tw.getPOSTagging()) {\n if (tagging.a >= (beta * best)) {\n endIndex++;\n } else {\n break;\n }\n }\n List> tmpTagging = new ArrayList>(tw.getPOSTagging().size());\n for (Pair tg : tw.getPOSTagging()) {\n tmpTagging.add(tg);\n }\n tmpTagging.subList(endIndex, tmpTagging.size()).clear();\n TaggedWord twTmp = new TaggedWord(w);\n twTmp.setPOSTagging(tmpTagging);\n res.add(twTmp);\n }\n return res;\n }\n \n /** \n * Build a POS tagger from a config file. \n * A non-dummy config file might contain:\n * ...\n * # this is a comment\n * taggerType=basic\n * priorModel=/home/.../posprior/prior.flm\n * # note that CaSE dOES Not matter for the key (but does for the value, e.g. a file name).\n * PRIORmodelvocab=/home/.../posprior/vocab.voc\n * # you can also repeat opions. the last one will take effect.\n * priormodelvocab=/home/.../posprior/vocab2.voc\n * \n * # did you see that empty line get ignored?\n * # notice that spaces around the '=' get ignored, as well.\n * maxentModel = /home/.../maxentmodels/myposmod.mod\n * # this last must be an ARPA-formatted n-gram model over POS tags (7-grams work well).\n * sequenceModel=/home/.../pos.lm\n * # lastly, the tagging beam width (1.0 means \"single-best\" -- i.e., a unitagger).\n * beta=0.1\n */\n public static POSTagger posTaggerFactory(String configFile) {\n POSTagger res = null;\n try {\n\n String[] pathKeys = { \"maxentmodel\", \"priormodel\", \"priormodelvocab\", \"sequencemodel\"};\n Map opts = ConfigFileProcessor.readInConfig(configFile, pathKeys);\n if (opts.get(\"taggertype\").equalsIgnoreCase(\"dummy\")) {\n return new DummyPOSTagger();\n }\n MaxentModel mem = new ZLMEM(new File(opts.get(\"maxentmodel\")));\n POSPriorModel posPrior = null;\n\t if (opts.get(\"priormodel\") != null) {\n\t\tposPrior = new POSPriorModel(opts.get(\"priormodel\"), opts.get(\"priormodelvocab\"));\n\t }\n POSTagFex fexer = new POSTagFex(posPrior);\n\n res = new BasicPOSTagger(mem, fexer, opts.get(\"sequencemodel\"));\n res.setBeta(Double.parseDouble(opts.get(\"beta\")));\n TaggingAlgorithm alg = (opts.get(\"taggingalgorithm\") == null || opts.get(\"taggingalgorithm\").equals(\"forward-backward\")) ? \n TaggingAlgorithm.FORWARDBACKWARD : TaggingAlgorithm.FORWARD;\n res.setTaggingAlgorithm(alg);\n\n res.setIncludeGold((opts.get(\"includegold\") == null || opts.get(\"includegold\").equals(\"false\")) ? false : true);\n \n } catch (FileNotFoundException ex) {\n Logger.getLogger(POSTagger.class.getName()).log(Level.SEVERE, null, ex);\n } catch (IOException ex) {\n Logger.getLogger(POSTagger.class.getName()).log(Level.SEVERE, null, ex);\n }\n\n return res;\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/postagger/ml/POSPriorModel.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2010 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.postagger.ml;\n\nimport opennlp.ccg.parse.supertagger.util.ProbPairComparator;\n\nimport java.io.BufferedReader;\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.io.InputStreamReader;\nimport java.io.OutputStreamWriter;\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\nimport opennlp.ccg.lexicon.DefaultTokenizer;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.ngrams.ConditionalProbabilityTable;\nimport opennlp.ccg.parse.tagger.io.SRILMFactoredBundleCorpusIterator;\nimport opennlp.ccg.util.Interner;\nimport opennlp.ccg.util.Pair;\n\n/**\n * (c) (2009) Dennis N. Mehay\n * @author Dennis N. Mehay\n * \n * Model for predicting p(POS | word). Uses an ARPA-formatted\n * SRILM-trained \"unigram\" factored LM for this, where each \"unigram\" is\n * a bundle of word:pos.\n */\npublic class POSPriorModel extends ConditionalProbabilityTable {\n\n public static final String WORD = DefaultTokenizer.WORD_ATTR;\n public static final String POS_TAG = DefaultTokenizer.POS_ATTR;\n private Interner> pairs = new Interner>();\n /** \n * Re-usable list for attr-val pairs of word-pos-supertag inputs to the prior model\n * (i.e., for predicting p(STag | word, POS). \n */\n public List> attrVals = new ArrayList>(5);\n /** \n * A comparator for sorting Pair's where the Double is a probability\n * (effectively sorts by descending order of probability).\n */\n private ProbPairComparator ppcomp = new ProbPairComparator();\n /** All the priors. Reference them when getting beta-best, beta-worst, etc. */\n List> priors = new ArrayList>(1000);\n\n /** String[] of all possible POS outcomes. */\n private String[] posVocab = null;\n\n /** Construct a prior model with the FLM config file and corresponding vocab file. */\n public POSPriorModel(String flmFile, String vocabFile) throws IOException {\n super(flmFile);\n String post = null;\n BufferedReader br = new BufferedReader(new FileReader(new File(vocabFile)));\n post = br.readLine().trim();\n\n // get next POS tag from the vocab.\n while ((post != null) && !post.trim().startsWith(POS_TAG + \"-\")) {\n post = br.readLine();\n }\n if (post != null) {\n post = post.trim().split(\"-\")[1];\n }\n\n Collection allSupertags = new HashSet();\n\n // find out how many outcomes we have.\n int cnt = 0;\n while (post != null) {\n cnt++;\n allSupertags.add(post);\n while ((post != null) && !post.trim().startsWith(POS_TAG + \"-\")) {\n post = br.readLine();\n }\n if (post != null) {\n post = post.trim().split(\"-\")[1];\n }\n }\n\n // initialize the arrays to this size.\n posVocab = new String[cnt];\n \n\n cnt = 0;\n // fill the vocab array with all possible POS tags.\n for (String posTag : allSupertags) {\n posVocab[cnt++] = posTag.intern();\n }\n br.close();\n }\n\n /** Get the prior probability of this POS/word combo. */\n public double getPriorOf(String pos, String word) {\n attrVals.clear();\n Pair surfaceForm = pairs.intern(new Pair(WORD, word.intern()));\n attrVals.add(surfaceForm);\n Pair partOfSpeech = pairs.intern(new Pair(POS_TAG, pos.intern()));\n attrVals.add(partOfSpeech);\n return score(attrVals);\n }\n\n /** Get the POS-dict restricted prior distribution (sorted descending by prob.) */\n public List> getPriors(Word w) {\n List> sortedTags = new ArrayList>(posVocab.length);\n for (String postag : posVocab) {\n sortedTags.add(new Pair(getPriorOf(postag, w.getForm()), postag));\n }\n Collections.sort(sortedTags, ppcomp);\n return sortedTags;\n }\n /* added by DCE, to facilitate use in hypertagging\n * Identical to above method, but accepts a String (name of EP) rather than\n * a Word object.\n */\n public List> getPriors(String s) {\n \ts.intern();\n \tList> sortedTags = new ArrayList>(posVocab.length);\n for (String postag : posVocab) {\n sortedTags.add(new Pair(getPriorOf(postag, s), postag));\n }\n Collections.sort(sortedTags, ppcomp);\n return sortedTags;\n }\n \n public static void main(String[] args) throws IOException {\n String usage = \"\\nPOSPriorModel -vocab (-c [default = ]) (-o [default = ])\\n\";\n if (args.length > 0 && args[0].equals(\"-h\")) {\n System.out.println(usage);\n System.exit(0);\n }\n\n SRILMFactoredBundleCorpusIterator in = null;\n BufferedWriter out = null;\n BufferedWriter voc = null;\n\n try {\n String inputCorp = \"\", output = \"\", vocabFile = \"vocab.voc\";\n\n for (int i = 0; i < args.length; i++) {\n if (args[i].equals(\"-c\")) { inputCorp = args[++i]; continue; }\n if (args[i].equals(\"-o\")) { output = args[++i]; continue; }\n if (args[i].equals(\"-vocab\")) { vocabFile = args[++i]; continue; }\n System.out.println(\"Unrecognized option: \" + args[i]);\n }\n\n try {\n in = new SRILMFactoredBundleCorpusIterator(\n (inputCorp.equals(\"\")) ? new BufferedReader(new InputStreamReader(System.in)) : new BufferedReader(new FileReader(new File(inputCorp))));\n } catch (FileNotFoundException ex) {\n System.err.print(\"Input corpus \" + inputCorp + \" not found. Exiting...\");\n Logger.getLogger(POSPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n System.exit(\n -1);\n }\n\n try {\n out = (output.equals(\"\")) ? new BufferedWriter(new OutputStreamWriter(System.out)) : new BufferedWriter(new FileWriter(new File(output)));\n } catch (IOException ex) {\n System.err.print(\"Output file \" + output + \" not found. Exiting...\");\n Logger.getLogger(POSPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n System.exit(\n -1);\n }\n\n try {\n voc = new BufferedWriter(new FileWriter(new File(vocabFile)));\n } catch (IOException ex) {\n Logger.getLogger(POSPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n }\n\n Map vocab = new HashMap();\n for (List inLine : in) {\n for (Word w : inLine) {\n String pos = POS_TAG + \"-\" + DefaultTokenizer.escape(w.getPOS()),\n wform = WORD + \"-\" + DefaultTokenizer.escape(w.getForm());\n\n vocab.put(pos, (vocab.get(pos) == null) ? 1 : vocab.get(pos) + 1);\n vocab.put(wform, (vocab.get(wform) == null) ? 1 : vocab.get(wform) + 1);\n out.write(wform + \":\" + pos + \" \");\n }\n out.write(System.getProperty(\"line.separator\"));\n }\n out.flush();\n\n for (String str : vocab.keySet()) {\n voc.write(str + System.getProperty(\"line.separator\"));\n }\n voc.flush();\n } finally {\n try {\n out.close();\n in.close();\n voc.close();\n } catch (IOException ex) {\n Logger.getLogger(POSPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n }\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/postagger/ml/POSTagFex.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse.postagger.ml;\n\nimport java.io.BufferedReader;\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileReader;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.io.InputStreamReader;\nimport java.io.OutputStreamWriter;\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport opennlp.ccg.parse.tagger.Constants;\nimport opennlp.ccg.parse.supertagger.ml.FeatureExtractor;\nimport opennlp.ccg.parse.tagger.TaggedWord;\nimport opennlp.ccg.util.Pair;\nimport opennlp.ccg.parse.tagger.io.SRILMFactoredBundleCorpusIterator;\nimport opennlp.ccg.lexicon.Word;\n\n/**\n * Feature extractor for POS taggers.\n * \n * The inputs are \"TaggedWord\"s simply for consistency of interface.\n * There should be no tags assigned to the words (short, perhaps, TOBI \n * tags or the like).\n * \n * @author Dennis N. Mehay\n */\npublic class POSTagFex implements FeatureExtractor {\n private POSPriorModel posPrior = null;\n \n /** Constructor with a prior model (replaces tagging dictionary). */\n public POSTagFex(POSPriorModel posPrior) {\n this.posPrior = posPrior;\n }\n \n /** Constructor without prior model. Prior features will not be used. */ \n public POSTagFex() { this(null); }\n \n public static final String curL = \"X\";\n public static final String prevL = \"X-1\";\n public static final String prevPrevL = \"X-2\";\n public static final String nextL = \"X+1\";\n public static final String nextNextL = \"X+2\";\n private static final String[] lxfLabs = {prevPrevL, prevL, curL, nextL, nextNextL};\n private static final String prefix = \"prefix\", suffix = \"suffix\";\n private static final String hyphen = \"containsHyphen\";\n private static final String caps = \"containsUC\";\n private static final String num = \"containsNum\";\n private static final String neConn = \"containsNEConnector\";\n private static final String priorF = \"PPOS\";\n /** The string that connects elements of a fused named entity. */\n private String neConnecter = \"_\";\n /** Get a word's features for applying the tagger (i.e., not training mode). */\n public Collection> getFeatures(Map sentence, Integer wordIndex) {\n return getFeatures(sentence, wordIndex, false);\n }\n\n /** Get a sentence of words' features for applying the tagger (i.e., not training mode). */\n public List>> getSentenceFeatures(Map sentence) {\n return getSentenceFeatures(sentence, false);\n }\n\n /** \n * Get the features for a word in context. training == true iff the output class is to be collected as well. \n * \n * TODO: This and supertagger feature extractor (fex) should be merged into a more general, parameterizable \n * sentence-level contextual feature extractor. (VERY todo-ish, though.)\n */\n public Collection> getFeatures(Map sentence, Integer wordIndex, boolean training) {\n Collection> result = new ArrayList>(30);\n\n TaggedWord current, prev, prevPrev, next, nextNext;\n current = sentence.get(wordIndex);\n // -------- The left periphery ------------\n int wind = wordIndex.intValue();\n if (wind > 1) {\n prev = sentence.get(wind - 1);\n prevPrev = sentence.get(wind - 2);\n } else if (wind > 0) {\n prev = sentence.get(wind - 1);\n prevPrev = Constants.OOB;\n } else {\n prev = prevPrev = Constants.OOB;\n }\n\n // -------- The right periphery -----------\n int tempSize = sentence.size();\n if ((tempSize - (wind + 1)) >= 2) {\n next = sentence.get(wind + 1);\n nextNext = sentence.get(wind + 2);\n } else if (tempSize - (wind + 1) >= 1) {\n next = sentence.get(wind + 1);\n nextNext = Constants.OOB;\n } else {\n next = nextNext = Constants.OOB;\n }\n\n Double activation = Constants.one;\n \n if (training) {\n result.add(new Pair(current.getPOS(), activation));\n }\n\n // we do not use tag-sequence features in this model.\n // these are in a separate sequence model (n-gram model over POS sequences).\n \n // standard contextual features (word to the left, current word, word to the right, etc.).\n // these features are from Ratnaparkhi (1996).\n result.add(new Pair(curL + \"=\" + current.getForm(), activation));\n result.add(new Pair(prevL + \"=\" + prev.getForm(), activation));\n result.add(new Pair(prevPrevL + \"=\" + prevPrev.getForm(), activation));\n result.add(new Pair(nextL + \"=\" + next.getForm(), activation));\n result.add(new Pair(nextNextL + \"=\" + nextNext.getForm(), activation));\n \n // features that replace the tagging dictionary. \n // add real-valued (activation = prior log-prob) features for each of the beta-best prior\n // tags, given this word.\n if(posPrior != null) {\n List> priors = posPrior.getPriors(current.getWord());\n double beta = 0.1;\n double best = priors.get(0).a;\n String wform = current.getForm();\n for(Pair prior : priors) {\n if(prior.a > (beta * best)) {\n // add the features PPOS=: and PPOS_word=_:.\n result.add(new Pair(priorF + \"=\" + prior.b, prior.a));\n result.add(new Pair(priorF + \"_word\" + \"=\" + prior.b + \"_\" + wform, prior.a));\n } else {\n break;\n }\n }\n }\n \n // these are in addition to Ratnaparkhi's (1996) contextual features.\n // now for conjunctions of features: w-2w-1=..., w-1w+1=..., w+1w+2=... (same for posp).\n // (i.e., bigram features over words and parts of speech and bigrams of words and POSs that straddle the current token).\n // N.B. only use single-best POSs (maybe change later).\n TaggedWord[] wds = {prevPrev, prev, current, next, nextNext};\n\n for (int j = 1; j < wds.length; j++) {\n result.add(new Pair(lxfLabs[j - 1] + \"|\" + lxfLabs[j] + \"=\" + wds[j - 1].getForm() + \"|\" + wds[j].getForm(), activation));\n // also, if at the current word slot, add bigrams that straddle the current word.\n if (j == 2) {\n result.add(new Pair(lxfLabs[j - 1] + \"|\" + lxfLabs[j + 1] + \"=\" + wds[j - 1].getForm() + \"|\" + wds[j + 1].getForm(), activation));\n }\n }\n \n // affix features from Ratnaparkhi (1996).\n // if the word's length is > 4, then extract the 1-, 2-, 3- and 4-character affixes. \n if(current.getForm().length() > 4) {\n StringBuffer prefixes = new StringBuffer(4), suffixes = new StringBuffer(4);\n char[] wdForm = current.getForm().toCharArray();\n // prefixes.\n int cursor = 0;\n for(cursor = 0; cursor < 4; cursor++) {\n prefixes.append(wdForm[cursor]);\n result.add(new Pair(prefix+\"=\"+prefixes.toString(), Constants.one));\n }\n // suffixes.\n for(cursor = wdForm.length-1; cursor >= wdForm.length-5; cursor--) {\n suffixes.insert(0, wdForm[cursor]);\n result.add(new Pair(suffix+\"=\"+suffixes.toString(), Constants.one));\n }\n }\n \n // now do \"contains hyphen\", \"contains number\", \"contains uppercase letter\" and contains fused NE connecter (_) features.\n // also from Ratnaparkhi (1996). \n if(current.getForm().contains(\"-\")) { result.add(new Pair(hyphen, Constants.one)); }\n if(current.getForm().matches(\".*[0-9]+.*\")) { result.add(new Pair(num, Constants.one)); }\n if(!current.getForm().toLowerCase().equals(current.getForm())) { result.add(new Pair(caps, Constants.one)); }\n\t// if we see a NE connector, this is likely a NNP (in English, e.g.).\n if(current.getForm().contains(neConnecter)) { result.add(new Pair(neConn, Constants.one)); }\n return result;\n }\n\n /** \n * Get the features for a sentence of words in context. \n * training == true iff the output classes are to be collected as well. \n */\n public List>> getSentenceFeatures(Map sentence, boolean training) {\n List>> result = new ArrayList>>(30);\n List keys = new ArrayList(sentence.keySet().size());\n for(Integer wordIndex : sentence.keySet()) { keys.add(wordIndex); }\n Collections.sort(keys);\n for(Integer wordIndex : keys) {\n result.add(getFeatures(sentence, wordIndex, training));\n }\n return result;\n }\n \n public static void main(String[] args) throws IOException { \n String usage = \n \"POSTagFex (-h [gets this message]) (-i [defaults to ]) (-o [defaults to ])\\n\"+\n \" (-p [.flm] -v )\\n\";\n if(args.length > 0 && args[0].equals(\"-h\")) { System.out.println(usage); System.exit(0); }\n \n String input = \"\", output = \"\", priorModF = null, priorVocab = null;\n for(int j = 0; j < args.length; j++) {\n if(args[j].equals(\"-i\")) { input = args[++j]; continue; }\n if(args[j].equals(\"-o\")) { output = args[++j]; continue; }\n if(args[j].equals(\"-p\")) { priorModF = args[++j]; continue; }\n if(args[j].equals(\"-v\")) { priorVocab = args[++j]; continue; }\n System.err.println(\"Unrecognized option: \" + args[j]); \n }\n SRILMFactoredBundleCorpusIterator corp = new SRILMFactoredBundleCorpusIterator(\n input.equals(\"\") ?\n new BufferedReader(new InputStreamReader(System.in)) :\n new BufferedReader(new FileReader(new File(input))));\n BufferedWriter out = new BufferedWriter(\n output.equals(\"\") ?\n new BufferedWriter(new OutputStreamWriter(System.out)) :\n new BufferedWriter(new FileWriter(new File(output))));\n \n POSPriorModel posPriorMod = null;\n if(priorModF != null) {\n posPriorMod = new POSPriorModel(priorModF, priorVocab);\n }\n POSTagFex fexer = new POSTagFex(posPriorMod); \n for(List sentence : corp) {\n Map sent = new HashMap(sentence.size());\n int index = 0;\n for(Word w : sentence) { sent.put(index++, new TaggedWord(w)); }\n \n List>> ftss = fexer.getSentenceFeatures(sent, true);\n \n for(Collection> fts : ftss) {\n index = 0;\n for(Pair ft : fts) {\n // if we're at the first item, print out the label.\n if (index == 0) {\n out.write(ft.a);\n } else {\n out.write(\" \" + ft.a + \":\" + ft.b);\n }\n index++; \n }\n out.write(System.getProperty(\"line.separator\"));\n }\n }\n\tout.flush();\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/supertagger/JavaSupertaggingApp.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger;\n\nimport opennlp.ccg.parse.tagger.util.ResultSink;\nimport opennlp.ccg.parse.supertagger.ml.STFex;\nimport opennlp.ccg.parse.supertagger.ml.FeatureExtractor;\nimport opennlp.ccg.parse.tagger.ml.ZLMEM;\nimport opennlp.ccg.parse.tagger.io.SRILMFactoredBundleCorpusIterator;\nimport opennlp.ccg.parse.tagger.io.PipeDelimitedFactoredBundleCorpusIterator;\nimport opennlp.ccg.parse.tagger.Constants;\nimport java.io.*;\nimport java.util.*;\nimport static java.util.Arrays.*;\nimport joptsimple.*;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.parse.supertagger.io.*;\nimport opennlp.ccg.parse.supertagger.ml.*;\nimport opennlp.ccg.parse.supertagger.util.*;\nimport opennlp.ccg.util.Pair;\n\n/**\n * @author Dennis N. Mehay\n * @version $Revision: 1.6 $, $Date: 2010/09/21 04:12:41 $\n */\npublic class JavaSupertaggingApp {\n\n public static void main(String[] args) throws Exception {\n try {\n // instantiate command-line option parser, setting up type-safe expectations about\n // what should be passed for the options.\n OptionParser parser = new OptionParser();\n parser.acceptsAll(asList(\"train\", \"R\"), \"extract training features.\");\n parser.acceptsAll(asList(\"tag\", \"T\"), \"supertag a POS-tagged file.\");\n parser.acceptsAll(asList(\"test\",\"E\"), \"test tagger against gold standard.\"); \n parser.acceptsAll(asList(\"tagdictextract\", \"D\"), \"extract tagging dictionaries.\");\n parser.acceptsAll(asList(\"h\", \"?\"), \"show help.\");\n OptionSpec tokenisation = parser.acceptsAll(asList(\"delimiter\", \"d\")).withRequiredArg().ofType(String.class).describedAs(\"SRILM factor bundles or C&C-style\" +\n \"(pipe-delimited) factor bundles [choose one of: \\\"SRILM\\\", \\\"candc\\\"]\");\n OptionSpec goldstandspec = parser.acceptsAll(asList(\"g\",\"gold\")).withRequiredArg().ofType(File.class).\n describedAs(\"the gold standard tagged file [file must have same bundle format as input corpus, \\\"SRILM\\\" or \\\"candc\\\"]\");\n OptionSpec inputspec = parser.acceptsAll(asList(\"i\", \"input\")).withRequiredArg().ofType(File.class).describedAs(\"training or tagging/testing file\");\n OptionSpec outputspec = parser.acceptsAll(asList(\"o\", \"output\")).withRequiredArg().ofType(File.class).describedAs(\"output location (for training feats or tags)\");\n OptionSpec modspec = parser.acceptsAll(asList(\"m\", \"model\")).withRequiredArg().ofType(File.class).describedAs(\"textual model file (ZhangLe maxent-style) [for tagging/testing only]\");\n OptionSpec priormodspec = parser.acceptsAll(asList(\"priorModelF\")).withRequiredArg().ofType(String.class).describedAs(\"config file for ARPA-formatted FLM [for tagging/testing and feature extraction\"+\n \"MUST also give vocab file]\");\n OptionSpec vocabspec = parser.acceptsAll(asList(\"vocabF\")).withRequiredArg().ofType(String.class).describedAs(\"vocab file for ARPA-formatted FLM [for tagging/testing and feature extraction]\");\n OptionSpec kspec = parser.accepts(\"K\").withRequiredArg().ofType(Integer.class).describedAs(\"K parameter of Clark and Curran [for tagging/testing only]\");\n OptionSpec betaspec = parser.accepts(\"beta\").withRequiredArg().ofType(Double.class).describedAs(\"beam width for supertagger [for tagging only]\");\n OptionSpec wdictspec = parser.acceptsAll(asList(\"w\", \"worddict\")).withRequiredArg().ofType(File.class).describedAs(\"path to the word-based tagging dictionary file\");\n OptionSpec pdictspec = parser.acceptsAll(asList(\"p\", \"posdict\")).withRequiredArg().ofType(File.class).describedAs(\"path to the POS-based tagging dictionary file\");\n OptionSpec seqModel = parser.acceptsAll(asList(\"s\",\"seqModel\")).withOptionalArg().ofType(String.class).describedAs(\"the tag sequence model (for forward-backward tagging)\");\n OptionSpec fbBeam = parser.acceptsAll(asList(\"fbBeamWidth\")).withOptionalArg().ofType(Integer.class).describedAs(\"maximum width of the forward-backward beam [default = 5]\");\n OptionSpec tagAlgorithm = parser.acceptsAll(asList(\"taggingAlgorithm\")).withOptionalArg().ofType(String.class).describedAs(\"tagging algorithm. choose from {forward-backward, forward} [default = forward-backward]\");\n OptionSet options = parser.parse(args);\n if (options.has(\"?\") || args.length == 0) {\n parser.printHelpOn(System.out);\n System.exit(0);\n }\n assert (options.valueOf(tokenisation).equalsIgnoreCase(\"candc\") || options.valueOf(tokenisation).equalsIgnoreCase(\"srilm\"));\n // Must say whether we are tagging (or testing) or training (extracting features, actually).\n assert (options.has(\"tag\") || options.has(\"train\") || options.has(\"test\") || options.has(\"D\"));\n \n // Can't both train and tag/test, or train and extract tagging dict, or tag/test and do the last.\n assert !(options.has(\"train\") && (options.has(\"tag\") || options.has(\"test\")));\n assert !(options.has(\"train\") && options.has(\"D\"));\n assert !((options.has(\"tag\") || options.has(\"test\")) && options.has(\"D\"));\n \n // either we're doing forward-backward tagging, or we're not.\n assert (options.has(\"seqModel\") || !(options.has(\"seqInterp\") || options.has(\"fbBeamWidth\")));\n \n // Can't have a model file input when we are training....\n assert !(options.has(\"train\") && options.has(\"m\"));\n // ... or when extracting a tag dict.\n assert !(options.has(\"D\") && options.has(\"m\"));\n \n // Must have tagging dict files when tagging or extracting tag dicts, \n // and additionally beta and K when tagging .\n assert (!(options.has(\"tag\") || options.has(\"test\") || options.has(\"D\")) || (options.has(\"p\") && options.has(\"w\")));\n assert (!(options.has(\"tag\") || options.has(\"test\")) || (options.has(\"K\") && options.has(\"beta\")));\n \n // can't use prior model if no vocab file is given (so that the prior model knows which \n // classes to make probabilistic predictions over) or no POS dictionary is given (so\n // that we can restrict our priors to those supertags that have occurred with a particular\n // POS).\n assert (!(options.has(\"priorModelF\") && (!options.has(\"vocabF\") || !options.has(\"p\"))));\n STPriorModel stPrior = null;\n if (options.has(\"priorModelF\")) {\n stPrior = new STPriorModel(options.valueOf(priormodspec), \n options.valueOf(vocabspec), \n new XMLPOSDictionaryReader(options.valueOf(pdictspec)).read());\n }\n \n if (options.has(\"tag\") || options.has(\"test\")) {\n long start = System.currentTimeMillis();\n // tag (and potentially measure performance against the gold-standard). \n //File mod = options.valueOf(modspec);\n //Integer k = options.valueOf(kspec);\n Double beta = options.valueOf(betaspec);\n \n \n ZLMEM maxentModel;\n String seqMod = options.has(\"seqModel\") ? options.valueOf(seqModel) : null;\n Integer fbWidth = options.has(\"fbBeamWidth\") ? options.valueOf(fbBeam) : 5;\n \n String algStr = options.has(\"taggingAlgorithm\") ? options.valueOf(tagAlgorithm) : \"forward-backward\";\n Constants.TaggingAlgorithm alg = algStr.equalsIgnoreCase(\"forward\") ? \n Constants.TaggingAlgorithm.FORWARD : \n Constants.TaggingAlgorithm.FORWARDBACKWARD;\n \n STTaggerWordDictionary wd = null;\n STTaggerPOSDictionary pd = null;\n \n if(options.has(\"w\")) wd = new XMLWordDictionaryReader(options.valueOf(wdictspec)).read();\n if(options.has(\"p\")) pd = new XMLPOSDictionaryReader(options.valueOf(pdictspec)).read();\n \n WordAndPOSDictionaryLabellingStrategy tagger = new WordAndPOSDictionaryLabellingStrategy(\n wd,\n pd, \n (options.has(\"K\") ? options.valueOf(kspec).intValue() : 20), \n maxentModel = new ZLMEM(options.valueOf(modspec)),\n new STFex(stPrior),\n seqMod,\n alg);\n \n tagger.setMaxSearchBeam(fbWidth);\n maxentModel.verbose = true;\n \n Iterator> corpus = null;\n Iterator> goldCorpus = null;\n \n if(options.valueOf(tokenisation).equalsIgnoreCase(\"srilm\")) {\n corpus = new SRILMFactoredBundleCorpusIterator(new BufferedReader(new FileReader(options.valueOf(inputspec))));\n } else if(options.valueOf(tokenisation).equalsIgnoreCase(\"candc\")) {\n corpus = new PipeDelimitedFactoredBundleCorpusIterator(new BufferedReader(new FileReader(options.valueOf(inputspec))));\n }\n if(options.has(\"test\") && options.valueOf(tokenisation).equalsIgnoreCase(\"srilm\")) {\n goldCorpus = new SRILMFactoredBundleCorpusIterator(new BufferedReader(new FileReader(options.valueOf(goldstandspec))));\n } else if(options.has(\"test\") && options.valueOf(tokenisation).equalsIgnoreCase(\"candc\")) {\n goldCorpus = new PipeDelimitedFactoredBundleCorpusIterator(new BufferedReader(new FileReader(options.valueOf(goldstandspec))));\n }\n \n BufferedWriter outf = new BufferedWriter(new FileWriter(options.valueOf(outputspec)));\n \n boolean test = options.has(\"test\");\n \n ResultSink results = new ResultSink();\n int sentCnt = 0;\n \n tagger.setBetas(new double[] {beta});\n \n while(corpus.hasNext()) {\n sentCnt++;\n List sent = corpus.next();\n \n List>> taggings = tagger.multitag(sent, beta); \n \n if(test) {\n List goldsent = goldCorpus.next();\n results.addSent(taggings, goldsent);\n } \n \n Iterator sentiter = sent.iterator(); \n // output file format = word goldtag tag1 ... tagK \n outf.write(\"\"+System.getProperty(\"line.separator\"));\n for(List> tagging : taggings) { \n Word nextw = sentiter.next();\n outf.write(nextw.getForm() + \"\\t1\\t\" + nextw.getPOS() + \"\\t1.0\\t\" + tagging.size() + \"\\t\");// + nextw.getSupertag() + \" \");\n //outf.write(nextw.getForm() + \"|||\"+ nextw.getStem() + \"|||\" + nextw.getPOS() + \"|||\");\n String tags = \"\";\n for(Pair tg : tagging) {\n //tags+=\"^\"+tg.b+\":\"+tg.a;\n tags+= \"\\t\" + tg.b + \"\\t\"+tg.a;\n }\n // write out the multitagging, minus the initial space (tab).\n outf.write(tags.substring(1) + System.getProperty(\"line.separator\"));\n \n //// write out the multitagging, minus the initial ^.\n //outf.write(tags.substring(1) + \" \");\n } \n \n outf.write(\"\"+System.getProperty(\"line.separator\"));\n if(sentCnt % 10 == 0) {\n outf.flush();\n }\n }\n outf.flush();\n outf.close();\n if(test) {\n System.err.println(results.report());\n }\n long end = System.currentTimeMillis();\n System.err.println(\"Time to tag: \" + ((end - start + 0.0)/1000) + \" seconds.\");\n \n } else if (options.has(\"tagdictextract\")) {\n // extract tagging dictionaries.\n File wd = options.valueOf(wdictspec);\n File pd = options.valueOf(pdictspec);\n File inf = options.valueOf(inputspec);\n TaggingDictionaryExtractor tde = new TaggingDictionaryExtractor(inf,wd,pd,options.valueOf(tokenisation));\n System.err.println(\"Extracting dictionaries from: \"+inf.toString()+\" into files: \"+wd.toString()+\" and: \"+pd.toString()+\"\\n(wdict and posdict, resp.).\");\n tde.extract();\n } else {\n // train (extract features). \n File inf = options.valueOf(inputspec);\n File outf = options.valueOf(outputspec);\n FeatureExtractor fexer = (stPrior == null) ? new STFex() : new STFex(stPrior);\n ZhangLeTrainingExtractor fexApp = new ZhangLeTrainingExtractor(inf, outf, options.valueOf(tokenisation), fexer);\n System.err.println(\"Extracting features from file: \" + inf.toString() + \", and placing extracted features in: \" + outf.toString() + \".\");\n fexApp.writeFeats();\n }\n\n } catch (OptionException e) {\n throw e;\n } catch (Exception e) {\n throw e;\n //System.err.println(\"Something went wrong. Double-check your inputs.\");\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/supertagger/LabellingStrategy.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger;\n\nimport java.util.Collection;\nimport java.util.List;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.util.Pair;\n//import ml.MaxentModel;\n\n/**\n * @author Dennis N. Mehay\n * @version $Revision: 1.2 $, $Date: 2010/09/21 04:12:41 $\n */\npublic interface LabellingStrategy {\n\t\n /**\n * A method to reset the K parameter (for word dictionaries).\n * @param newK An int to replace the int value of K.\n * @returns null.\n */\n public void setK(int newK);\n \n /** Multitag a TaggedWord and throw away the probabilities. */\n public List multitag(Word thisword, Collection> context, double beta);\n \n /** Multitag a TaggedWord and retain the probabilities. */\n public List> multitagWithScores(Word thisword, Collection>contex, double beta);\n \n}\n" }, { "path": "src/opennlp/ccg/parse/supertagger/WordAndPOSDictionaryLabellingStrategy.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger;\n\nimport java.io.BufferedReader;\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.FileWriter;\nimport opennlp.ccg.parse.supertagger.io.XMLWordDictionaryReader;\nimport opennlp.ccg.parse.supertagger.io.XMLPOSDictionaryReader;\nimport opennlp.ccg.parse.supertagger.ml.STFex;\nimport opennlp.ccg.parse.supertagger.ml.FeatureExtractor;\nimport opennlp.ccg.parse.tagger.ProbIndexPair;\nimport opennlp.ccg.parse.tagger.TaggedWord;\nimport opennlp.ccg.parse.tagger.Constants;\nimport opennlp.ccg.parse.tagger.io.SRILMFactoredBundleCorpusIterator;\nimport java.io.IOException;\nimport java.io.InputStreamReader;\nimport java.io.OutputStreamWriter;\nimport java.util.*;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\nimport opennlp.ccg.lexicon.*;\nimport opennlp.ccg.parse.Supertagger;\nimport opennlp.ccg.parse.postagger.DummyPOSTagger;\nimport opennlp.ccg.parse.postagger.POSTagger;\nimport opennlp.ccg.parse.supertagger.ml.STPriorModel;\nimport opennlp.ccg.parse.tagger.ml.MaxentModel;\nimport opennlp.ccg.parse.supertagger.util.*;\nimport opennlp.ccg.parse.tagger.Constants.TaggingAlgorithm;\nimport opennlp.ccg.parse.tagger.ml.ZLMEM;\nimport opennlp.ccg.util.Pair;\nimport opennlp.ccg.parse.tagger.sequencescoring.SequenceScorer;\nimport opennlp.ccg.parse.tagger.util.ConfigFileProcessor;\nimport opennlp.ccg.parse.tagger.util.ResultSink;\n\n/**\n * A `labelling strategy' for a CCG supertagger that\n * restricts the output of the model based on word and POS `tagging\n * dictionaries' in the following way:\n *\n * if a word w occurs at least K times in training, the model's output\n * is constrained to the outcomes seen with w during training. If w\n * did not occur at least K times during training, the model's output is\n * constrained to the outcomes seen with w's POS tag during training.\n * In the off chance that the POS tag was not seen in training, the model's\n * best prediction is used.\n *\n * @author Dennis N. Mehay\n * @version $Revision: 1.22 $, $Date: 2011/03/22 03:20:25 $\n */\npublic class WordAndPOSDictionaryLabellingStrategy implements LabellingStrategy, Supertagger {\n\n // print warnings?\n private boolean verbose = false;\n // use tagging dictionaries?\n private boolean useWordDict = false;\n private boolean usePOSDict = false;\n private SequenceScorer seqScorer = null;\n private STTaggerWordDictionary wd;\n private STTaggerPOSDictionary pd;\n private int K, usualK, finalK; \n private MaxentModel mo;\n // extracts features from the context of a word.\n private FeatureExtractor fexer = new STFex();\n \n // postagger for non-gold-POS supertagging.\n private POSTagger posTagger;\n\n // the current tagging.\n private List tagging;\n \n // POS-specific multipliers to \"tighten\" or \"loosen\" up the tagging beam width\n // (\"beta\") as needed. E.g., the beta for period/full stop might not need to be\n // very permissive, while those for lexical verbs or some fancy punctuation marks\n // might need to be.\n public Map betaMultipliers = new HashMap();\n public double minMultiplier = 1.0;\n \n /** Constructor without n-gram model (for scoring tag sequences). */\n public WordAndPOSDictionaryLabellingStrategy(STTaggerWordDictionary wd, STTaggerPOSDictionary pd, int K, MaxentModel mo,\n FeatureExtractor fexer) {\n this(wd, pd, K, mo, fexer, null, null);\n }\n\n \n /** Constructor WITH n-gram model (for scoring tag sequences). */\n public WordAndPOSDictionaryLabellingStrategy(\n STTaggerWordDictionary wd,\n STTaggerPOSDictionary pd,\n int K,\n MaxentModel mo,\n FeatureExtractor fexer,\n String tagSequenceModel,\n Constants.TaggingAlgorithm alg) {\n this(wd, pd, K, mo, fexer, tagSequenceModel, alg, new DummyPOSTagger());\n }\n /** Constructor with n-gram model and POS tagger */\n public WordAndPOSDictionaryLabellingStrategy(\n STTaggerWordDictionary wd,\n STTaggerPOSDictionary pd,\n int K,\n MaxentModel mo,\n FeatureExtractor fexer,\n String tagSequenceModel,\n Constants.TaggingAlgorithm alg,\n POSTagger posTagger) {\n this.wd = wd;\n this.pd = pd;\n this.mo = mo;\n this.posTagger = posTagger;\n if (K > 0) {\n this.K = K;\n } else {\n this.K = 0;\n }\n usualK = K; finalK = K;\n \n this.fexer = fexer; \n \n try {\n\n if (tagSequenceModel != null) {\n // find the n-gram order of the model.\n int ord = SequenceScorer.findOrder(tagSequenceModel);\n // load it into the SequenceScorer.\n seqScorer = new SequenceScorer(ord, tagSequenceModel);\n seqScorer.setSearchBeam(5);\n seqScorer.setAlgorithm(alg);\n }\n } catch (IOException ex) {\n Logger.getLogger(WordAndPOSDictionaryLabellingStrategy.class.getName()).log(Level.SEVERE, null, ex);\n } \n }\n\n public void useWordDict(boolean useIt) { useWordDict = useIt; }\n public void usePOSDict(boolean useIt) { usePOSDict = useIt; }\n public WordAndPOSDictionaryLabellingStrategy(STTaggerWordDictionary wd, STTaggerPOSDictionary pd, int K, MaxentModel mo) {\n this(wd, pd, K, mo, new STFex());\n }\n\n /**\n * Set the maximum width of the beam in the forward-backward tagger.\n */\n public void setMaxSearchBeam(int maxSearchBeam) { if (seqScorer != null) seqScorer.setSearchBeam(maxSearchBeam); }\n \n /**\n * Reset the K parameter.\n */\n public void setK(int newK) {\n this.K = newK;\n }\n\n /**\n * Set the usual K parameter.\n */\n public void setUsualK(int newK) {\n this.usualK = newK;\n }\n\n /**\n * Set the final K parameter.\n */\n public void setFinalK(int newK) {\n this.finalK = newK;\n }\n\n /**\n * A method that returns all labels given by the model that both (1) are assigned probability `p' s.t.:\n * p>=(`beta'*), where `beta' is a factor passed in by the client of this method and\n * where is the probability of the most probably outcome of the model and (2) (if the word\n * (obtained from the String[] `context') has occured at least K times in training)\n * are in the STTaggerWordDictionary under the entry for said word. If the word did\n * not occur at least K times in training, the output set is constrained by a STTaggerPOSDictionary.\n * In the off chance that a POS did not occur in the training data, the models predictions themselves are\n * submitted to the `beta constraint'.\n *\n * @param context A String[] of contextual predicates (in the maximum entropy modelling sense)\n * @param mo A MaxentModel.\n * @param beta A double specifying how close in probability all returned outcomes must be.\n * @return An ArrayList of labels that meet the above constraints.\n */\n public List multitag(Word w, Collection> context, double beta) {\n List> temp = this.multitagWithScores(w, context, beta);\n ArrayList res = new ArrayList(temp.size());\n for (Pair t : temp) {\n res.add(t.b);\n }\n return res;\n }\n\n /**\n * A method to return the set of labels that are greater than or equal to \n * the best label multiplied by a factor `beta', given a model and a String[]\n * of contextual predicates.\n * @param thisWord a opennlp.ccg.lexicon.Word representing the current word being tagged.\n * @param context A Collection> of contextual predicates \n * (in the maximum entropy modelling sense) with their corresponding activations (real-valued, hence the\n * Double).\n * @param model A model for generating the base predictions.\n * @param beta A positive double specifying how close to the best label\n * each label returned must be.\n * @return An ArrayList> of the outcomes\n * {o: score(o)>=[beta * score(bestLabel)]}.\n */\n public List> multitagWithScores(Word thisWord, Collection> context, double beta) {\n // All the scores of the outcomes (the index of each double score\n // is the key which allows us to retrieve the outcome from the model).\n double[] ocs = mo.eval(context);\n // Sort in descending order of probability.\n ProbIndexPair[] sortedOutcomes = new ProbIndexPair[ocs.length];\n for (int i = 0; i < ocs.length; i++) {\n sortedOutcomes[i] = new ProbIndexPair(new Double(ocs[i]), new Integer(i));\n }\n Arrays.sort(sortedOutcomes);\n String tempOutcome = \"\";\n String word = thisWord.getForm();\n String pos = thisWord.getPOS();\n ArrayList> retVal = new ArrayList>(30);\n\n // Find the best outcomes seen with the word in training that\n // meet the `beta' constraint.\n // *******************************************************************************************\n double bestOutcomeProb, currentOutcomeProb;\n bestOutcomeProb = 0;\n // mww: changed to not always include front of list, as it may not meet dict constraints\n ProbIndexPair temp;\n // Now loop to see how many make the cut.\n // (But make sure to be sensitive to the dictionary, if necessary.)\n // See whether the word has a freq of this.K in the training corpus.\n Collection wordPermittedOutcomes = (wd != null) ? this.wd.getEntry(word, this.K) : null;\n if (wordPermittedOutcomes != null && useWordDict) {\n // The word (lemma) was seen at least K times in training.\n // Get all beta-OK outcomes that are in the dictionary entry.\n for (int ocInd = 0; ocInd < sortedOutcomes.length; ocInd++) {\n temp = sortedOutcomes[ocInd];\n tempOutcome = mo.getOutcome(temp.b.intValue());\n currentOutcomeProb = temp.a.doubleValue();\n if (wordPermittedOutcomes.contains(tempOutcome)) {\n if (bestOutcomeProb == 0) {\n bestOutcomeProb = currentOutcomeProb;\n }\n if (currentOutcomeProb >= (bestOutcomeProb * beta)) { // Beta constraint.\n // The cut-off was met, add the outcome.\n\n retVal.add(new Pair(temp.a, tempOutcome));\n // update max, for first selected outcome\n if (currentOutcomeProb > bestOutcomeProb) {\n bestOutcomeProb = currentOutcomeProb;\n }\n } else {\n // Else, since our ProbIndexPair[] is sorted by probablity, there will be no more\n // outcomes that make the (beta) cut.\n break;\n }\n } // If the word is not in the dictionary specified outcomes, move along. \n\n }\n } else {\n // Revert to the POS dictionary.\n Collection posPermittedOutcomes = null;\n if (pos != null) {\n posPermittedOutcomes = (pd != null) ? this.pd.getEntry(pos) : null;\n } else {\n if(verbose) { System.err.println(\"warning: null POS for: \" + word);} // mww: check for null pos\n\n }\n if (posPermittedOutcomes != null && usePOSDict) {\n // Get all beta-OK outcomes that are in the POS dictionary entry.\n for (int ocInd2 = 0; ocInd2 < sortedOutcomes.length; ocInd2++) {\n temp = sortedOutcomes[ocInd2];\n tempOutcome = mo.getOutcome(temp.b.intValue());\n currentOutcomeProb = temp.a.doubleValue();\n\n if (posPermittedOutcomes.contains(tempOutcome.trim())) {\n if (bestOutcomeProb == 0) {\n bestOutcomeProb = currentOutcomeProb;\n }\n if (currentOutcomeProb >= (bestOutcomeProb * beta)) { // Beta constraint.\n // Made the cut-off, add the outcome.\n\n retVal.add(new Pair(temp.a, tempOutcome));\n // update max, for first selected outcome\n if (currentOutcomeProb > bestOutcomeProb) {\n bestOutcomeProb = currentOutcomeProb;\n }\n } else {\n // Else, since our ProbIndexPair[] is sorted by probablity, there will be no more\n // outcomes that make the (beta) cut.\n break;\n }\n } // If the word is not in the dictionary specified outcomes, move along. \n\n }\n } else {\n // Otherwise, just get all model predictions that meet the beta constraint,\n // ignoring the word and POS dictionaries.\n for (int ocInd3 = 0; ocInd3 < sortedOutcomes.length; ocInd3++) {\n temp = sortedOutcomes[ocInd3];\n currentOutcomeProb = temp.a.doubleValue();\n\n if (bestOutcomeProb == 0) {\n bestOutcomeProb = currentOutcomeProb;\n }\n if (currentOutcomeProb >= (bestOutcomeProb * beta)) {\n // Made the cut-off, add the outcome.\n retVal.add(new Pair(temp.a, mo.getOutcome(temp.b.intValue())));\n // update max, for first selected outcome\n if (currentOutcomeProb > bestOutcomeProb) {\n bestOutcomeProb = currentOutcomeProb;\n }\n } else {\n // Else, since our ProbIndexPair[] is sorted by probability, there will be no more\n // outcomes that make the cut.\n break;\n }\n }\n }\n } \n\n // include the gold standard tag, if not in there.\n if(includeGold) {\n // assume input word has the gold tag in it.\n String gold = thisWord.getSupertag();\n // check whether gold is in the output.\n boolean containsGold = false;\n for(Pair tg : retVal) {\n if(tg.b.equals(gold)) {\n containsGold = true;\n break;\n }\n }\n if(!containsGold) {\n // insert it\n containsGold = false;\n for(ProbIndexPair oc : sortedOutcomes) {\n if(mo.getOutcome(oc.b).equals(gold)) {\n retVal.add(new Pair(oc.a, mo.getOutcome(oc.b)));\n containsGold = true;\n break;\n }\n } \n }\n if(!containsGold) {\n // if the gold-standard still isn't in there, it must not be part of the tag set, add it with epsilon probability.\n // we're assuming that gold tags are needed for a training routine that doesn't care about supertag probabilities\n // (as in Clark and Curran (2007)).\n // check to see whether we are in the log domain (by checking for negative scores -- kind of a hack).\n retVal.add(new Pair((sortedOutcomes[0].a < 0) ? -99 : 1.0112214926104486e-43, thisWord.getSupertag()));\n }\n }\n // *******************************************************************************************\n return retVal;\n }\n\n\n // get the current tagging (now only used to grab the POS tagging). \n public List getCurrentTagging() { return tagging; }\n // set the current tagging (now only used to set the current POS tagging).\n public void setCurrentTagging(List tgging) { tagging = tgging; }\n \n public List>> multitag(List sentence, double beta) {\n List>> results = new ArrayList>>(sentence.size());\n Map sent = new TreeMap();\n int cnt = 0;\n \n List taggedSent = posTagger.tagSentence(sentence);\n setCurrentTagging(taggedSent);\n \n for (TaggedWord werd : taggedSent) {\n sent.put(new Integer(cnt++), werd);\n }\n\n List>> contexts = fexer.getSentenceFeatures(sent);\n\n // Iterate simultaneously through both the words and the contextual features.\n Iterator wds = sentence.iterator();\n Word w = null;\n Iterator>> ctxts = contexts.iterator();\n Collection> context = null;\n\n int cursor = 0;\n while (wds.hasNext() && ctxts.hasNext()) {\n // get the next word.\n w = wds.next();\n if(w.getPOS() == null) {\n w = Word.createFullWord(w, w.getForm(), tagging.get(cursor).getPOSTagging().get(0).b, w.getSupertag(), w.getSemClass()); \n }\n context = ctxts.next();\n if (seqScorer != null) {\n // increase the tag ambiguity (for re-scoring using forward-backward).\n double newBeta = Math.min(beta * minMultiplier, beta / 8);\n if(beta < 0.00001) { newBeta = Math.min(beta * minMultiplier, beta / 2); }\n results.add(multitagWithScores(w, context, newBeta));\n } else { results.add(multitagWithScores(w, context, beta)); }\n cursor++;\n }\n \n List>> finalResults = null;\n if (seqScorer != null) {\n // rescore and filter. pass in input sentence (in case, e.g., we have set the includeGold flag).\n finalResults = betaBestFilter(seqScorer.rescoreSequence(results), beta, sentence);\n } else {\n finalResults = results;\n } \n return finalResults;\n }\n\n /** \n * Return a beta-best filtered subset of the tags in each multitagging list (each multitagging list is assumed to be non-empty). \n */\n private List>> betaBestFilter(List>> multitaggings, double beta, List inputSentence) {\n List>> res = new ArrayList>>(multitaggings.size());\n\n int wordIndex = 0;\n for (List> mtagging : multitaggings) {\n List> tempTagging = new ArrayList>(mtagging.size()); \n Word thisWord = inputSentence.get(wordIndex);\n // set to a (possibly different, possibly less restrictive?) beta if this POS has a beta multiplier set.\n Double bmult = betaMultipliers.get(thisWord.getPOS());\n double possiblyNewBeta = Math.min(1.0, (bmult != null) ? (bmult * beta) : beta);\n \n double best = mtagging.get(0).a;\n for (Pair tg : mtagging) {\n if (tg.a >= (possiblyNewBeta * best) || (includeGold && tg.b.equals(thisWord.getSupertag()))) {\n tempTagging.add(tg);\n } else {\n if(!includeGold) { // if we're not still fishing for gold...\n // ...stop, since they're in sorted order.\n break;\n }\n }\n } \n res.add(tempTagging);\n wordIndex++;\n }\n return res;\n }\n \n \n \n \n //-------------------------------------------------------------------------\n // Supertagger interface methods (added by Michael White)\n \n /**\n * The sequence of beta values to use in tagging.\n */\n protected double[] betas = null;\n \n /**\n * The current betaIndex.\n */\n protected int betaIndex = 0;\n \n /**\n * The current tagging.\n */\n protected List>> currentTagging = null;\n \n /**\n * The current word.\n */\n protected int currentWord = 0;\n \n /**\n * Flag for whether to include gold tags.\n */\n protected boolean includeGold = false;\n\n /** Sets the beta values. */\n public void setBetas(double[] betas) {\n this.betas = betas;\n }\n\n /** Returns all the beta values. */\n public double[] getBetas() {\n return betas;\n }\n\n /** Returns the current beta value. */\n public double getCurrentBetaValue() {\n return betas[betaIndex];\n }\n\n /**\n * Advances beta to the next most restrictive setting.\n */\n public void nextBeta() {\n betaIndex++;\n }\n\n /**\n * Advances beta to the next less restrictive setting.\n */\n public void previousBeta() {\n betaIndex--;\n }\n\n /**\n * Returns whether there are any less restrictive beta settings\n * remaining in the sequence.\n */\n public boolean hasMoreBetas() {\n return betaIndex < betas.length - 1;\n }\n\n /**\n * Returns whether there are any more restrictive beta settings\n * remaining in the sequence.\n */\n public boolean hasLessBetas() {\n return betaIndex > 0;\n }\n\n /**\n * Resets beta to the most restrictive value.\n */\n public void resetBeta() {\n betaIndex = 0;\n }\n\n /**\n * Resets beta to the least restrictive value.\n */\n public void resetBetaToMax() {\n betaIndex = betas.length - 1;\n }\n\n /**\n * Sets the flag for whether to include gold tags.\n */\n public void setIncludeGold(boolean includeGold) { this.includeGold = includeGold; }\n\n /**\n * Maps the given words to their predicted categories, \n * so that the beta-best categories can be returned by calls to setWord\n * and getSupertags.\n */\n public void mapWords(List words) {\n if(hasMoreBetas()) {\n K = usualK;\n } else {\n K = finalK; \n } \n currentTagging = multitag(words, getCurrentBetaValue());\n }\n\n /**\n * Sets the current word to the one with the given index, \n * so that the beta-best categories for it can be returned by a call to \n * getSupertags.\n */\n public void setWord(int index) {\n currentWord = index;\n }\n\n /**\n * Returns the supertags of the desired categories for the current lexical lookup\n * as a map from supertags to contextual probabilities (or null to accept all). \n */\n public Map getSupertags() {\n Map retval = new HashMap();\n List> tags = currentTagging.get(currentWord);\n for (Pair tag : tags) {\n retval.put(tag.b, tag.a);\n }\n return retval;\n }\n \n /** \n * A factory method to make a supertagger from a config file (see the sample config file:\n * \n * $OPENCCG_HOME/ccgbank/models/supertagger/st.config\n * \n * for more information).\n */\n @SuppressWarnings(\"unused\")\n\t\tpublic static WordAndPOSDictionaryLabellingStrategy supertaggerFactory(String configFile) { \n WordAndPOSDictionaryLabellingStrategy res = null;\n String[] pathKeys = { \"priormodel\", \"priormodelvocab\", \"sequencemodel\", \"wdict\", \"posdict\", \"maxentmodel\", \"posconfig\" };\n Map opts = ConfigFileProcessor.readInConfig(configFile, pathKeys);\n boolean verbose = (opts.get(\"verbose\").equals(\"true\")) ? true : false;\n // 'S' is for string repr.\n String priorModS = opts.get(\"priormodel\"), \n priorVocabS = opts.get(\"priormodelvocab\"),\n seqModS = opts.get(\"sequencemodel\"),\n wDictS = opts.get(\"wdict\"),\n pDictS = opts.get(\"posdict\"),\n firstKS = opts.get(\"firstk\"),\n lastKS = opts.get(\"lastk\"),\n maxentModS = opts.get(\"maxentmodel\"),\n posConfigS = opts.get(\"posconfig\"),\n betasS = opts.get(\"betas\"),\n betaMults = opts.get(\"betamultipliers\"), // POS-specific multipliers to \"tighten\" or \"loosen\" up the tagging beam width.\n includeGold = opts.get(\"includegold\");\n \n assert (maxentModS != null) : \"Empty maxent model.\";\n \n // either use prior model (and have prior vocab specified) or not.\n assert (priorModS != null && priorVocabS != null) || (priorModS == null && priorVocabS == null) : \"using prior model with no vocab file.\";\n \n // ensure that there are word- and pos-keyed tagging dicts if there\n // is no st prior model.\n assert (wDictS == null || pDictS == null) && priorModS == null : \"need tagging dicts if no supertagging prior model and prior vocab are specified.\";\n \n // need the POS-keyed tagging dict, no matter what.\n assert (priorModS != null && pDictS == null) : \"need POS-keyed tagging dict for prior model.\";\n \n // need 'K' values if not using tagging dicts.\n assert (priorModS == null || (firstKS != null & lastKS != null)) : \"need to specify first and last 'K' value when not using prior model.\";\n \n // seqMod probably shouldn't be null. warn if in verbose mode.\n if(seqModS == null && verbose) { System.err.println(\"Warning: empty sequence model. Performance will suffer.\"); }\n \n STPriorModel priorM = null;\n if(priorModS != null && priorVocabS != null) {\n try { priorM = new STPriorModel(priorModS, priorVocabS); } \n catch (IOException ex) {\n Logger.getLogger(WordAndPOSDictionaryLabellingStrategy.class.getName()).log(Level.SEVERE, null, ex);\n }\n }\n STFex fex = new STFex(priorM);\n STTaggerWordDictionary wD = (wDictS != null) ? new XMLWordDictionaryReader(new File(wDictS)).read() : null; \n STTaggerPOSDictionary pD = (pDictS != null) ? new XMLPOSDictionaryReader(new File(pDictS)).read() : null;\n int kay = (opts.get(\"firstk\") == null) ? 20 : Integer.parseInt(opts.get(\"firstk\")), firstK, lastK;\n firstK = (opts.get(\"firstk\") == null) ? 20 : Integer.parseInt(opts.get(\"firstk\"));\n lastK = (opts.get(\"lastk\") == null) ? 100 : Integer.parseInt(opts.get(\"lastk\"));\n double[] betaz = new double[betasS.split(\"\\\\s+\").length];\n int cursor = 0;\n for(String beta : betasS.split(\"\\\\s+\")) {\n betaz[cursor++] = Double.parseDouble(beta);\n }\n \n // should we use the tagging dictionaries (yes if there is no prior model).\n boolean useWordDictionary = (wDictS != null);\n boolean usePOSDictionary = (pDictS != null);\n POSTagger pTagger = (posConfigS == null) ? null : POSTagger.posTaggerFactory(posConfigS);\n TaggingAlgorithm alg = (opts.get(\"taggingalgorithm\") == null || opts.get(\"taggingalgorithm\").equals(\"forward-backward\")) ?\n TaggingAlgorithm.FORWARDBACKWARD : TaggingAlgorithm.FORWARD;\n MaxentModel mem = new ZLMEM(new File(maxentModS));\n //STTaggerWordDictionary wd,STTaggerPOSDictionary pd, int K, MaxentModel mo, FeatureExtractor fexer,\n //String tagSequenceModel, Constants.TaggingAlgorithm alg, POSTagger posTagger\n res = (pTagger != null) ? \n new WordAndPOSDictionaryLabellingStrategy(wD, pD, kay, mem, fex, seqModS, alg, pTagger) :\n new WordAndPOSDictionaryLabellingStrategy(wD, pD, kay, mem, fex, seqModS, alg);\n res.setK(kay);\n res.setUsualK(firstK);\n res.setFinalK(lastK);\n res.setBetas(betaz);\n res.useWordDict(useWordDictionary);\n res.usePOSDict(usePOSDictionary);\n res.setIncludeGold((opts.get(\"includegold\") == null || opts.get(\"includegold\").equals(\"false\")) ? false : true); \n \n // get POS-specific beta multipliers (as a string of pairs -- all space delimited).\n if(betaMults != null) {\n String[] bmts = betaMults.split(\"\\\\s+\");\n for(int a=0, b=1; b < bmts.length; a = a + 2, b = b + 2) {\n double mul = Double.parseDouble(bmts[b]);\n res.betaMultipliers.put(bmts[a], mul);\n if(mul < res.minMultiplier) {\n res.minMultiplier = mul;\n }\n }\n }\n return res;\n }\n \n public static void main(String[] args) {\n String usage = \"\\nWordAndPOSDictLabellingStrategy (-h [gets this message]) -e [defaults to not testing] -c > -beta [0.0,1.0]\\n\"+\n \" (-i [default=]) (-o [default=])\\n\";\n \n if (args.length > 0 && args[0].equals(\"-h\")) {\n System.out.println(usage);\n System.exit(0);\n }\n\n SRILMFactoredBundleCorpusIterator in = null;\n BufferedWriter out = null;\n try {\n \n String inputCorp = \"\", output = \"\", \n configFile = null;\n \n double beta = 1.0;\n boolean test = false;\n\n for (int i = 0; i < args.length; i++) {\n if (args[i].equals(\"-i\")) { inputCorp = args[++i]; continue; }\n if (args[i].equals(\"-o\")) { output = args[++i]; continue; }\n if (args[i].equals(\"-e\")) { test = true; continue; }\n if (args[i].equals(\"-c\")) { configFile = args[++i]; continue; }\n if (args[i].equals(\"-beta\")) { beta = Double.parseDouble(args[++i]); continue; }\n System.out.println(\"Unrecognized option: \" + args[i]);\n }\n\n ResultSink rs = new ResultSink(ResultSink.ResultSinkType.SUPERTAG);\n try { \n in = new SRILMFactoredBundleCorpusIterator(\n (inputCorp.equals(\"\")) ? \n new BufferedReader(new InputStreamReader(System.in)) : \n new BufferedReader(new FileReader(new File(inputCorp)))); \n } catch (FileNotFoundException ex) {\n System.err.print(\"Input corpus \" + inputCorp + \" not found. Exiting...\");\n Logger.getLogger(WordAndPOSDictionaryLabellingStrategy.class.getName()).log(Level.SEVERE, null, ex);\n System.exit(-1);\n }\n\n try {\n out = (output.equals(\"\")) ? new BufferedWriter(new OutputStreamWriter(System.out)) : new BufferedWriter(new FileWriter(new File(output)));\n } catch (IOException ex) {\n System.err.print(\"Output file \" + output + \" not found. Exiting...\");\n Logger.getLogger(WordAndPOSDictionaryLabellingStrategy.class.getName()).log(Level.SEVERE, null, ex);\n System.exit(-1);\n }\n\n WordAndPOSDictionaryLabellingStrategy stgger = WordAndPOSDictionaryLabellingStrategy.supertaggerFactory(configFile);\n \n // for each sentence, print out:\n // \n // w1 ... ... \n // ...\n // wN ... ... \n // \n for (List inLine : in) {\n \n List>> taggedSent = stgger.multitag(inLine, beta);\n if(test) { rs.addSent(taggedSent, inLine); }\n // beginning of sentence...\n out.write(\"\" + System.getProperty(\"line.separator\")); \n List posTagging = stgger.getCurrentTagging();\n int cursor = -1;\n while(++cursor < taggedSent.size()) {\n Word wdIn = inLine.get(cursor);\n // word form...\n out.write(wdIn.getForm());\n TaggedWord posT = posTagging.get(cursor);\n // print out number of POS tags, followed by tab-separated probabilized POS tagging.\n out.write(\"\\t\" + posT.getPOSTagging().size());\n for(Pair pt : posT.getPOSTagging()) {\n out.write(\"\\t\" + pt.b + \"\\t\" + pt.a);\n }\n // now print out number of and list of tab-separated, probabilized supertags.\n out.write(\"\\t\" + taggedSent.get(cursor).size());\n for(Pair stg : taggedSent.get(cursor)) {\n out.write(\"\\t\" + stg.b + \"\\t\" + stg.a);\n }\n out.write(System.getProperty(\"line.separator\"));\n }\n out.write(\"\" + System.getProperty(\"line.separator\"));\n }\n out.flush();\n\n if(test) { System.err.println(rs.report()); }\n } catch (IOException ex) {\n Logger.getLogger(WordAndPOSDictionaryLabellingStrategy.class.getName()).log(Level.SEVERE, null, ex);\n } finally {\n try {\n out.close();\n in.close();\n } catch (IOException ex) {\n Logger.getLogger(WordAndPOSDictionaryLabellingStrategy.class.getName()).log(Level.SEVERE, null, ex);\n }\n }\n }\n} // End class WordPOSDictLabellingStrategy\n\n" }, { "path": "src/opennlp/ccg/parse/supertagger/io/XMLPOSDictionaryReader.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger.io;\n\nimport java.io.File;\nimport java.util.HashSet;\nimport java.util.Collection;\nimport java.util.Iterator;\nimport java.util.Map;\nimport java.util.TreeMap;\nimport javax.xml.parsers.*;\n\nimport opennlp.ccg.parse.supertagger.util.STTaggerPOSDictionary;\nimport opennlp.ccg.util.Pair;\nimport org.xml.sax.Attributes;\nimport org.xml.sax.SAXException;\nimport org.xml.sax.XMLReader;\nimport org.xml.sax.helpers.DefaultHandler;\n\n/**\n * @author Dennis N. Mehay\n * @version $Revision: 1.2 $, $Date: 2009/12/21 02:10:57 $\n */\npublic class XMLPOSDictionaryReader {\n \n private File dictFile;\n private XMLReader reader;\n private Map> dict; \n \n /** Creates a new instance of XMLDictionaryReader\n * @param dictFile A String pointing to the location of\n * the XML file specifying the word dictionary.\n */\n public XMLPOSDictionaryReader(File df) {\n if(!df.exists()) {\n throw new RuntimeException(\"File \"+df.getAbsolutePath().toString()+\" does not exist.\");\n }\n this.dictFile = df; \n }\n \n /**\n * Read in the dictionary file and create a new STTaggerPOSDictionary.\n * @return A new STTaggerPOSDictionary.\n */\n public STTaggerPOSDictionary read() {\n SAXParserFactory factory = SAXParserFactory.newInstance();\n try {\n SAXParser parser = factory.newSAXParser();\n reader = parser.getXMLReader();\n reader.setContentHandler(new wdContentHandler());\n reader.parse(this.dictFile.toURI().toString());\n } catch(Exception e) {\n e.printStackTrace();\n }\n return new STTaggerPOSDictionary(this.dict);\n }\n \n public static void main(String[] args) {\n // This is just to verify that the XML doc read in is the one \n // spit out.\n String fname = args[0];\n XMLPOSDictionaryReader rdr = new XMLPOSDictionaryReader(new File(fname));\n STTaggerPOSDictionary dct = rdr.read();\n Iterator>> it = dct.getMappings();\n Pair> tempP = null;\n System.out.println(\"\");\n while(it.hasNext()) {\n tempP = it.next();\n System.out.println(\" \"); \n for(Iterator stgs = tempP.b.iterator(); stgs.hasNext(); ) {\n System.out.println(\" \"+stgs.next().trim()+\" \");\n }\n System.out.println(\" \");\n }\n System.out.print(\"\");\n }\n \n /*\n * A ContentHandler to properly interpret the \"semantics\" of the XML (semantics\n * in the CS sense of formal semantics of a structured document).\n */\n class wdContentHandler extends DefaultHandler {\n private boolean inEntry = false, inSupertag = false;\n private String curPOS = null, currSTFrag = null;\n \n @Override\n public void startDocument() {\n dict = new TreeMap>();\n }\n \n @Override\n public void startElement(String namespaceURI, String lname, String qname, Attributes attrs)\n throws SAXException {\n if(qname.equalsIgnoreCase(\"entry\")) { \n if(this.inEntry) {\n throw new SAXException(\"Something is wrong.\\nThis is not a well-formed dictionary.\");\n } else {\n this.inEntry = true;\n String pos = attrs.getValue(0).trim(); \n dict.\n put(pos,\n new HashSet());\n this.curPOS = pos;\n }\n \n } else if(qname.equalsIgnoreCase(\"supertag\")) {\n if(!this.inEntry) {\n throw new SAXException(\"Something is wrong.\\nThis is not a well-formed dictionary.\");\n } else {\n this.inSupertag = true;\n this.currSTFrag = \"\";\n }\n }\n }\n \n @Override\n public void endElement(String uri, String name, String qName) {\n if(qName.equalsIgnoreCase(\"entry\")) {\n this.inEntry = false; this.curPOS = null;\n } else if(qName.equalsIgnoreCase(\"supertag\")) {\n this.inSupertag = false; \n Collection tempL = dict.get(this.curPOS);\n tempL.add(this.currSTFrag.trim());\n dict.put(this.curPOS, tempL);\n this.currSTFrag = null;\n }\n }\n \n @Override\n public void characters(char[] ch, int start, int length) {\n if(this.inSupertag && this.curPOS!=null) {\n // Get this supertag and add it to the list mapped to by this POS (i.e., the list\n // of supertags seen with this POS in training).\n String temp = new String(ch); \n temp = temp.substring(start, start+length);\n this.currSTFrag += temp;\n } else if(this.inSupertag) {\n System.err.println(\"Something is wrong.\\nThis is not a well-formed dictionary.\");\n }\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/supertagger/io/XMLWordDictionaryReader.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger.io;\n\nimport java.io.File;\nimport java.util.HashSet;\nimport java.util.Iterator;\nimport java.util.Map;\nimport java.util.Collection;\nimport java.util.TreeMap;\nimport javax.xml.parsers.*;\n\nimport opennlp.ccg.parse.supertagger.util.STTaggerWordDictionary;\nimport opennlp.ccg.util.Pair;\nimport org.xml.sax.Attributes;\nimport org.xml.sax.SAXException;\nimport org.xml.sax.XMLReader;\nimport org.xml.sax.helpers.DefaultHandler;\n\n/**\n * @author Dennis N. Mehay\n * @version $Revision: 1.3 $, $Date: 2010/09/21 04:12:41 $\n */\npublic class XMLWordDictionaryReader {\n \n private File dictFile;\n private XMLReader reader;\n private Map>> dict; \n \n /** Creates a new instance of XMLDictionaryReader\n * @param dictFile A String pointing to the location of\n * the XML file specifying the word dictionary.\n */\n public XMLWordDictionaryReader(File df) {\n if(!df.exists()) {\n throw new RuntimeException(\"File \"+df.getAbsolutePath().toString()+\" does not exist.\");\n }\n this.dictFile = df; \n }\n \n /**\n * Read in the dictionary file and create a new STTaggerWordDictionary.\n * @return A new STTaggerWordDictionary.\n */\n public STTaggerWordDictionary read() {\n SAXParserFactory factory = SAXParserFactory.newInstance();\n try {\n SAXParser parser = factory.newSAXParser();\n reader = parser.getXMLReader();\n reader.setContentHandler(new wdContentHandler());\n reader.parse(this.dictFile.toURI().toString());\n } catch(Exception e) {\n e.printStackTrace();\n } \n return new STTaggerWordDictionary(this.dict);\n }\n \n public static void main(String[] args) {\n // This is just to verify that the XML doc read in is the one \n // spit out.\n String fname = args[0];\n XMLWordDictionaryReader rdr = new XMLWordDictionaryReader(new File(fname));\n STTaggerWordDictionary dct = rdr.read();\n Iterator>>> it = dct.getMappings(); \n Pair>> tempP = null;\n System.out.println(\"\"); \n while(it.hasNext()) {\n \t\n tempP = it.next();\n System.out.println(\" \"); \n for(Iterator stgs = tempP.b.b.iterator(); stgs.hasNext(); ) {\n System.out.println(\" \"+stgs.next().trim()+\" \");\n }\n System.out.println(\" \");\n }\n System.out.print(\"\");\n }\n \n /*\n * A ContentHandler to properly interpret the \"semantics\" of the XML (semantics\n * in the CS sense of formal semantics of a structured document).\n */\n class wdContentHandler extends DefaultHandler {\n private boolean inEntry = false, inSupertag = false;\n private String curWord = null, currSTFrag = null;\n \n @Override\n public void startDocument() {\n dict = new TreeMap>>();\n }\n \n @Override\n public void startElement(String namespaceURI, String lname, String qname, Attributes attrs)\n throws SAXException {\n if(qname.equalsIgnoreCase(\"entry\")) { \n if(this.inEntry) {\n throw new SAXException(\"Something is wrong.\\nThis is not a well-formed dictionary.\");\n } else {\n this.inEntry = true;\n String word = attrs.getValue(0).trim(),\n freq = attrs.getValue(1).trim();\n word = word.replace(\"&\",\"&\");\n word = word.replace(\">\",\">\");\n word = word.replace(\"<\",\"<\");\n word = word.replace(\"'\",\"\\'\");\n word = word.replace(\""\",\"\\\"\");\n \n dict.\n put(word,\n new Pair>(\n new Integer(Integer.parseInt(freq)),\n new HashSet()));\n this.curWord = word;\n }\n \n } else if(qname.equalsIgnoreCase(\"supertag\")) {\n if(!this.inEntry) {\n throw new SAXException(\"Something is wrong.\\nThis is not a well-formed dictionary.\");\n } else {\n this.inSupertag = true;\n this.currSTFrag = \"\";\n }\n }\n }\n \n @Override\n public void endElement(String uri, String lName, String qName) {\n if(qName.equalsIgnoreCase(\"entry\")) {\n this.inEntry = false; this.curWord = null;\n } else if(qName.equalsIgnoreCase(\"supertag\")) {\n this.inSupertag = false;\n Pair> tempL = dict.get(this.curWord);\n tempL.b.add(this.currSTFrag.trim());\n \tdict.put(this.curWord, tempL);\n this.currSTFrag = null;\n }\n }\n \n @Override\n public void characters(char[] ch, int start, int length) {\n if(this.inSupertag && this.curWord!=null) {\n // Get this supertag and add it to the list mapped to by this word (i.e., the list\n // of supertags seen with this word in training).\n \tString temp = new String(ch); \n \ttemp = temp.substring(start, start+length);\n \tthis.currSTFrag += temp; \t\n } else if(this.inSupertag) {\n System.err.println(\"Something is wrong.\\nThis is not a well-formed dictionary.\");\n }\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/supertagger/ml/FeatureExtractor.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger.ml;\n\nimport opennlp.ccg.parse.tagger.TaggedWord;\nimport java.util.Collection;\nimport java.util.List;\nimport java.util.Map;\nimport opennlp.ccg.util.Pair;\n\n/**\n * @author Dennis N. Mehay\n * @version $Revision: 1.1 $, $Date: 2010/09/21 04:12:41 $\n */\npublic interface FeatureExtractor {\n\t\n /** \n * @param sentence A {@code Map} giving the (string-indexed) sentence of\n * {@code Word}s to be tagged.\n * @param wordIndex An {@code Integer}, giving the string index of the current word.\n * @return A {@code Collection>} representing the \n * real-valued activations of features (predicates) in the context of a word to be labelled.\n */\n public Collection> getFeatures(Map sentence, Integer wordIndex);\n \n /** \n * Same as getFeatures, but for the whole sentence, returning a List of contextual features, in order, one\n * per Word in sentence\n */\n public List>> getSentenceFeatures(Map sentence);\n \n /** \n * @param sentence A Map giving the (string-indexed) sentence of\n * Words to be tagged.\n * @param wordIndex An Integer, giving the string index of the current word.\n * @param training A boolean indicating whether we are extracting features for training (in which case\n * we need the label too).\n * @return A Collection> representing the \n * real-valued activations of features (predicates) in the context of a word to be labelled.\n */\n public Collection> getFeatures(Map sentence, Integer wordIndex, boolean training);\n \n /** \n * Same as getFeatures, but for the whole sentence, returning a List of contextual features, in order, one\n * per Word in sentence\n */\n public List>> getSentenceFeatures(Map sentence, boolean training);\n}\n" }, { "path": "src/opennlp/ccg/parse/supertagger/ml/STFex.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger.ml;\n\nimport java.io.BufferedReader;\nimport java.io.IOException;\nimport java.io.InputStreamReader;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.parse.tagger.TaggedWord;\nimport opennlp.ccg.parse.tagger.Constants;\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport opennlp.ccg.parse.postagger.POSTagger;\nimport opennlp.ccg.parse.tagger.io.SRILMFactoredBundleCorpusIterator;\nimport opennlp.ccg.util.Pair;\n\n/**\n * @author Dennis N. Mehay\n * @version $Revision: 1.7 $, $Date: 2010/12/15 07:16:03 $\n */\npublic class STFex implements FeatureExtractor {\n\t\n private boolean useMultiPOS = false;\n public static final String LEX = \"X\";\n public static final String POS = \"P\";\n public static final String prevP = POS + \"-1=\";\n public static final String prevPrevP = POS + \"-2=\";\n public static final String nextP = POS + \"+1=\";\n public static final String nextNextP = POS + \"+2=\";\n public static final String curP = POS + \"=\";\n public static final String prevL = LEX + \"-1=\";\n public static final String prevPrevL = LEX + \"-2=\";\n public static final String nextL = LEX + \"+1=\";\n public static final String nextNextL = LEX + \"+2=\";\n public static final String curL = LEX + \"=\";\n public static final String priorST = \"PST\" + \"=\";\n public static final String wordPOSPriorST = \"WPosPST\" + \"=\";\n public static final String wordPriorST = \"WPST\" + \"=\";\n public static final String POSPriorST = \"PosPST\" + \"=\";\n public static final String antiPriorST = \"APST\" + \"=\"; \n public static final String wordPOSAntiPriorST = \"WPosAPST\" + \"=\";\n public static final String wordAntiPriorST = \"WAPST\" + \"=\";\n public static final String POSAntiPriorST = \"PosAPST\" + \"=\";\n public static final TaggedWord outOfBounds = Constants.OOB;\n \n private static final String[] lxfLabs = {LEX + \"-2\", LEX + \"-1\", LEX, LEX + \"+1\", LEX + \"+2\"};\n private static final String[] posfLabs = {POS + \"-2\", POS + \"-1\", POS, POS + \"+1\", POS + \"+2\"}; \n \n /** \n * An object containing a ConditionalProbabilityTable that can give a prior distribution over all\n * known supertags given a POS-tagged word.\n * \n * Trained as a factored LM (presumably using SRILM).\n */\n protected STPriorModel priorMod;\n\n /** Constructor with no prior model. */\n public STFex( ) { this.priorMod = null; }\n \n /** Constructor with a prior model over supertags (to be used as a feature). */\n public STFex(STPriorModel priorMod) { this.priorMod = priorMod; }\n\n /** Pass in true to use multi-POS features, pass in false not to. */\n public void useMultiPOS(boolean trueOrFalse) { useMultiPOS = trueOrFalse; }\n \n /**\n * Extracts an ArrayList representing the contextual\n * predicates (features) of a line of (tokenised) text (each String\n * represents the predicates that fire for a word in the line).\n * Each resulting String will have the form:\n * cp1 cp2 ... cpK\n * @param sentence A List of feature bundles.\n * @param wordIndex An int giving the location of the word to be tagged.\n * @return A Collection> of real-valued feature activations\n * for the word at index wordIndex\n */\n public Collection> getFeatures(Map sentence, Integer wordIndex, boolean training) {\n Collection> result = new ArrayList>(30);\n\n TaggedWord current, prev, prevPrev, next, nextNext;\n current = sentence.get(wordIndex);\n // -------- The left periphery ------------\n int wind = wordIndex.intValue();\n if (wind > 1) {\n prev = sentence.get(wind - 1);\n prevPrev = sentence.get(wind - 2);\n } else if (wind > 0) {\n prev = sentence.get(wind - 1);\n prevPrev = outOfBounds;\n } else {\n prev = prevPrev = outOfBounds;\n }\n\n // -------- The right periphery -----------\n int tempSize = sentence.size();\n if ((tempSize - (wind + 1)) >= 2) {\n next = sentence.get(wind + 1);\n nextNext = sentence.get(wind + 2);\n } else if (tempSize - (wind + 1) >= 1) {\n next = sentence.get(wind + 1);\n nextNext = outOfBounds;\n } else {\n next = nextNext = outOfBounds;\n }\n Double activation = new Double(1.0);\n if (training)\n result.add(new Pair(current.getSupertag(), activation)); \n \n result.add(new Pair(curL + current.getForm(), activation)); \n if(useMultiPOS) { for(Pair tg : current.getPOSTagging()) result.add(new Pair(curP + tg.b, tg.a)); }\n else { result.add(new Pair(curP + current.getPOS(), activation)); }\n \n result.add(new Pair(prevL + prev.getForm(), activation)); \n if(useMultiPOS && prev != Constants.OOB) { for(Pair tg : prev.getPOSTagging()) result.add(new Pair(prevP + tg.b, tg.a)); }\n else { result.add(new Pair(prevP + prev.getPOS(), activation)); } \n \n result.add(new Pair(prevPrevL + prevPrev.getForm(), activation)); \n if(useMultiPOS && prevPrev != Constants.OOB) { for(Pair tg : prevPrev.getPOSTagging()) result.add(new Pair(prevPrevP + tg.b, tg.a)); }\n else { result.add(new Pair(prevPrevP + prevPrev.getPOS(), activation)); }\n \n result.add(new Pair(nextL + next.getForm(), activation)); \n if(useMultiPOS && next != Constants.OOB) { for(Pair tg : next.getPOSTagging()) result.add(new Pair(nextP + tg.b, tg.a)); }\n else { result.add(new Pair(nextP + next.getPOS(), activation)); }\n \n result.add(new Pair(nextNextL + nextNext.getForm(), activation));\n if(useMultiPOS && nextNext != Constants.OOB) { for(Pair tg : nextNext.getPOSTagging()) result.add(new Pair(nextNextP + tg.b, tg.a)); }\n else { result.add(new Pair(nextNextP + nextNext.getPOS(), activation)); }\n \n // now for conjunctions of features: w-2w-1=..., w-1w+1=..., w+1w+2=... (same for posp).\n // (i.e., bigram features over words and parts of speech and bigrams of words and POSs that straddle the current token).\n // N.B. only use single-best POSs (maybe change later). \n TaggedWord[] wds = {prevPrev, prev, current, next, nextNext};\n \n for (int j = 1; j < wds.length; j++) {\n // add bigram features (only for single-best POS). \n result.add(new Pair(lxfLabs[j - 1] + \"|\" + lxfLabs[j] + \"=\" + wds[j - 1].getForm() + \"|\" + wds[j].getForm(), activation));\n result.add(new Pair(posfLabs[j - 1] + \"|\" + posfLabs[j] + \"=\" + wds[j - 1].getPOS() + \"|\" + wds[j].getPOS(), activation));\n // also, if at the current word slot, add bigrams that straddle the current word.\n if (j == 2) {\n result.add(new Pair(lxfLabs[j - 1] + \"|\" + lxfLabs[j + 1] + \"=\" + wds[j - 1].getForm() + \"|\" + wds[j + 1].getForm(), activation));\n result.add(new Pair(posfLabs[j - 1] + \"|\" + posfLabs[j + 1] + \"=\" + wds[j - 1].getPOS() + \"|\" + wds[j + 1].getPOS(), activation));\n }\n }\n \n // If the prior model is not null, extract a feature for the beta-best (beta = 0.1) classes\n // predicted by the prior model (for all output classes -- supertags -- seen with this word's\n // POS).\n // Extract prior features from these. \n if(priorMod != null) {\n priorMod.computePriors(current.getWord());\n for(Pair priorClassActivationPair : priorMod.getRestrictedBetaBestPriors(current.getWord(), 0.1)) { // TODO: make beta parameterizable.\n double act = Math.log(priorClassActivationPair.b); \n String wd = current.getForm().intern(), pos = current.getPOS().intern();\n result.add(new Pair(priorST + priorClassActivationPair.a.intern(), act)); // log(prob)\n result.add(new Pair(wordPriorST + priorClassActivationPair.a.intern()+\"_\"+wd, act)); // log(prob)\n result.add(new Pair(wordPOSPriorST + priorClassActivationPair.a.intern()+\"_\"+wd+\"_\"+pos, act)); // log(prob)\n result.add(new Pair(POSPriorST + priorClassActivationPair.a.intern()+\"_\"+pos, act)); // log(prob)\n result.add(new Pair(antiPriorST + priorClassActivationPair.a.intern(), Math.log(1-Math.exp(act)))); // log(1-prob)\n // TODO: come up with sensible \"anti-prior\" features that simulate the filtering effect of the tagging dict.\n }\n } \n return result;\n }\n\n public List>> getSentenceFeatures(Map sentence, boolean training) {\n List>> res = new ArrayList>>(sentence.size());\n List keys = new ArrayList(sentence.keySet().size());\n for(Integer i : sentence.keySet()) { keys.add(i); }\n Collections.sort(keys);\n for(Integer wordIndex : keys) {\n res.add(getFeatures(sentence, wordIndex, training));\n }\n return res;\n }\n\n public Collection> getFeatures(Map sentence, Integer wordIndex) {\n return getFeatures(sentence, wordIndex, false);\n }\n\n public List>> getSentenceFeatures(Map sentence) {\n return getSentenceFeatures(sentence, false);\n }\n \n // main method for extracting features from a file (for training).\n // pass in a supertag prior model and prior model vocab file, if desired.\n // (these replace tagging dictionaries).\n // pass in a POS tagger config file, if we aren't only using gold POS tags only.\n // input corpus is from stdin, output goes to stdout.\n public static void main(String[] args) {\n // we assume that the training data is being streamed in from stdin (no parse IDs, just SRILM factor bundle lines),\n // and that output will stream to stdout.\n String usage = \n \"\\n | STFex (-h [gets this message]) (-r -v ) (-p ) | \";\n if(args.length > 0 && args[0].equals(\"-h\")) { System.out.println(usage); System.exit(0); }\n \n String priorModF = null, priorVocab = null, posConfig = null;\n for(int j = 0; j < args.length; j++) {\n if(args[j].equals(\"-r\")) { priorModF = args[++j]; continue; }\n if(args[j].equals(\"-v\")) { priorVocab = args[++j]; continue; }\n if(args[j].equals(\"-p\")) { posConfig = args[++j]; continue; }\n System.err.println(\"Unrecognized option: \" + args[j]); \n }\n SRILMFactoredBundleCorpusIterator corp = new SRILMFactoredBundleCorpusIterator(new BufferedReader(new InputStreamReader(System.in))); \n STPriorModel stPriorMod = null;\n if(priorModF != null) {\n try {\n stPriorMod = new STPriorModel(priorModF, priorVocab);\n } catch (IOException ex) {\n Logger.getLogger(STFex.class.getName()).log(Level.SEVERE, null, ex);\n }\n }\n STFex fexer = new STFex(stPriorMod); \n POSTagger posT = (posConfig == null) ? null : POSTagger.posTaggerFactory(posConfig);\n \n if(posT != null) { fexer.useMultiPOS(true); } else { fexer.useMultiPOS(false); }\n \n for(List sentence : corp) {\n Map sent = new HashMap(sentence.size());\n int index = 0; \n if(posT == null) { \n for(Word w : sentence) { sent.put(index++, new TaggedWord(w)); } \n } else {\n List posTagging = posT.tagSentence(sentence);\n for(TaggedWord tw : posTagging) { sent.put(index++, tw); }\n }\n List>> ftss = fexer.getSentenceFeatures(sent, true); \n for(Collection> fts : ftss) {\n index = 0;\n for(Pair ft : fts) {\n // if we're at the first item, print out the label.\n if (index == 0) {\n System.out.print(ft.a);\n } else {\n System.out.print(\" \" + ft.a + \":\" + ft.b);\n }\n index++; \n }\n System.out.println();\n }\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/supertagger/ml/STPriorModel.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2010 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger.ml;\n\nimport opennlp.ccg.parse.supertagger.util.ProbPairComparator;\n\nimport java.io.BufferedReader;\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.io.InputStreamReader;\nimport java.io.OutputStreamWriter;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\nimport opennlp.ccg.lexicon.DefaultTokenizer;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.ngrams.ConditionalProbabilityTable;\nimport opennlp.ccg.parse.tagger.io.SRILMFactoredBundleCorpusIterator;\nimport opennlp.ccg.parse.tagger.ProbIndexPair;\nimport opennlp.ccg.parse.supertagger.util.STTaggerPOSDictionary;\nimport opennlp.ccg.util.Interner;\nimport opennlp.ccg.util.Pair;\n\n/**\n * (c) (2009) Dennis N. Mehay\n * @author Dennis N. Mehay\n * \n * Model for predicting p(supertag | word, pos). Uses an ARPA-formatted\n * SRILM-trained \"unigram\" factored LM for this, where each \"unigram\" is\n * a bundle of word:pos:supertag.\n */\npublic class STPriorModel extends ConditionalProbabilityTable {\n\n public static final String WORD = DefaultTokenizer.WORD_ATTR;\n public static final String POS_TAG = DefaultTokenizer.POS_ATTR;\n public static final String SUPERTAG = DefaultTokenizer.SUPERTAG_ATTR;\n private Interner> pairs = new Interner>();\n /** \n * Re-usable list for attr-val pairs of word-pos-supertag inputs to the prior model\n * (i.e., for predicting p(STag | word, POS). \n */\n public List> attrVals = new ArrayList>(5);\n /** \n * A comparator for sorting Pair's where the Double is a probability\n * (effectively sorts by descending order of probability).\n */\n private ProbPairComparator ppcomp = new ProbPairComparator();\n /** All the priors. Reference them when getting beta-best, beta-worst, etc. */\n List> priors = new ArrayList>(1000);\n /** String[] of all possible supertag outcomes. */\n private String[] stagVocab = null;\n /** double[] containing the probability distro over all supertags. */\n private double[] stagDistro = null;\n /** \n * POS-keyed tagging dictionary (to provide restrictions on what the prior model may consider.\n * No restrictions if null.\n */\n private STTaggerPOSDictionary posDict = null;\n /** \n * Re-usable way of containing the probabilities and a pointer back into where they came from\n * in the probability distro over all supertags. \n */\n private ProbIndexPair[] stagPointers = null;\n\n /** Construct a prior model with the FLM config file and corresponding vocab file. */\n public STPriorModel(String flmFile, String vocabFile) throws IOException {\n // create with a null POS dictionary (i.e., no restrictions on taggings).\n this(flmFile, vocabFile, null);\n }\n\n /** Construct a prior model with the FLM config file and corresponding vocab file. */\n public STPriorModel(String flmFile, String vocabFile, STTaggerPOSDictionary posDict) throws IOException {\n super(flmFile);\n this.posDict = posDict;\n\n String st = null;\n BufferedReader br = new BufferedReader(new FileReader(new File(vocabFile)));\n st = br.readLine().trim();\n\n // get next supertag from the vocab.\n while ((st != null) && !st.trim().startsWith(SUPERTAG + \"-\")) {\n st = br.readLine();\n }\n if (st != null) {\n st = st.trim().split(\"-\")[1];\n }\n\n Collection allSupertags = new HashSet();\n\n // find out how many outcomes we have.\n int cnt = 0;\n while (st != null) {\n cnt++;\n allSupertags.add(st);\n while ((st != null) && !st.trim().startsWith(SUPERTAG + \"-\")) {\n st = br.readLine();\n }\n if (st != null) {\n st = st.trim().split(\"-\")[1];\n }\n }\n br.close();\n\n // initialize the arrays to this size.\n stagVocab = new String[cnt];\n stagPointers = new ProbIndexPair[cnt];\n stagDistro = new double[cnt];\n\n cnt = 0;\n // fill the vocab array with all possible supertags.\n for (String stag : allSupertags) {\n stagVocab[cnt++] = stag.intern();\n }\n }\n\n /** Set the POS-keyed tagging dictionary. */\n public void setPOSDict(STTaggerPOSDictionary posDict) {\n this.posDict = posDict;\n }\n\n /** Get the prior probability of this supertag/POS/word combo. */\n public double getPriorOf(String supertag, String word, String pos) {\n attrVals.clear();\n Pair surfaceForm = pairs.intern(new Pair(WORD, DefaultTokenizer.escape(word).intern()));\n attrVals.add(surfaceForm);\n Pair partOfSpeech = pairs.intern(new Pair(POS_TAG, DefaultTokenizer.escape(pos).intern()));\n attrVals.add(partOfSpeech);\n attrVals.add(pairs.intern(new Pair(SUPERTAG, DefaultTokenizer.escape(supertag).intern())));\n return score(attrVals);\n }\n\n /** Get the beta-best tags for this word, under the prior model. */\n public List> getBetaBestPriors(Word w, double beta) {\n List> allPriors = getAllPriors(w);\n List> betaBestPriors = new ArrayList>(100);\n double best = allPriors.get(0).b;\n for (Pair prior : allPriors) {\n if (best * beta <= prior.b) {\n betaBestPriors.add(prior);\n } else {\n break;\n }\n }\n return betaBestPriors;\n }\n\n /** Compute all priors, subject to the POS dict constraints. */\n public void computePriors(Word w) {\n if (posDict != null) {\n priors = getPOSRestrictedPriors(w);\n }\n }\n\n /** Get the POS-dict restricted prior distribution (sorted descending by prob.) */\n protected List> getPOSRestrictedPriors(Word w) {\n Collection tagsAllowed = posDict.getEntry(w.getPOS());\n if (tagsAllowed == null || tagsAllowed.size() == 0) {\n return priors;\n } else {\n List> sortedTags = new ArrayList>(tagsAllowed.size());\n for (String tag : tagsAllowed) {\n sortedTags.add(new Pair(getPriorOf(tag, w.getForm(), w.getPOS()), tag));\n }\n Collections.sort(sortedTags, ppcomp);\n return sortedTags;\n }\n }\n\n /** \n * Get the beta-best tags (using the prior model) only from among the POS-dictionary-allowed possibilities. \n * beta-best (def'n): {t | p(t) >= beta * p(best-tag) }\n */\n public List> getRestrictedBetaBestPriors(Word w, double beta) {\n if (posDict == null) {\n return getBetaBestPriors(w, beta);\n } else {\n List> rez = new ArrayList>(50);\n double best = priors.get(0).a;\n for(Pair tg : priors) {\n if(tg.a >= (beta * best)) {\n rez.add(new Pair(tg.b,tg.a));\n } else {\n break;\n }\n }\n return rez;\n }\n }\n \n /** \n * Get the beta-WORST tags (using the prior model) only from among the POS-dictionary-allowed possibilities. \n * beta-best (def'n): {t | p(t) >= beta * p(best-tag) }\n * beta-worst (def'n): {t | p(t) * beta <= p(worst-tag)}\n */\n public List> getRestrictedBetaWorstPriors(Word w, double beta) {\n if (posDict == null) {\n throw new UnsupportedOperationException(\"Cannot get beta-worst without a pos-keyed tagging dict.\\nNot yet implemented.\");\n } else {\n List> rez = new ArrayList>(50);\n List> cpy = new ArrayList>(priors);\n Collections.reverse(cpy); \n double worst = cpy.get(0).a;\n for(Pair tg : cpy) {\n if((tg.a * beta) <= worst) {\n rez.add(new Pair(tg.b,tg.a));\n } else {\n break;\n }\n }\n return rez;\n }\n }\n\n public List> getAllPriors(Word w) {\n return getNBestPriors(w, stagVocab.length);\n }\n\n /** Get the n-best supertags on the prior model, given this word (with POS). */\n public List> getNBestPriors(Word w, int n) {\n attrVals.clear();\n Pair surfaceForm = pairs.intern(new Pair(WORD, DefaultTokenizer.escape(w.getForm()).intern()));\n attrVals.add(surfaceForm);\n Pair pos = pairs.intern(new Pair(POS_TAG, DefaultTokenizer.escape(w.getPOS()).intern()));\n attrVals.add(pos);\n\n int cnt = 0;\n for (String st : stagVocab) {\n // remove the last stag factor, if there.\n if (attrVals.size() == 3) {\n attrVals.remove(attrVals.size() - 1);\n }\n\n attrVals.add(pairs.intern(new Pair(SUPERTAG, st)));\n // add the probability of this tag under the prior model to the distro array.\n double sc = score(attrVals);\n stagDistro[cnt] = sc;\n // add this probability with a pointer back to where it came from in the vocab.\n // (so that we can sort by probability, but then retrieve the supertag string).\n stagPointers[cnt] = new ProbIndexPair(sc, cnt);\n cnt++;\n\n }\n // sort descending by probability (achieved by the comparator implementation of ProbIndexPair).\n\n Arrays.sort(stagPointers);\n\n List> result = new ArrayList>(n);\n for (int i = 0; i <\n n; i++) {\n result.add(new Pair(stagVocab[stagPointers[i].b], stagPointers[i].a));\n }\n\n return result;\n }\n\n public static void main(String[] args) throws IOException {\n String usage = \"\\nSTPriorModel -vocab (-c ) (-o ) (-u ) (-v [ or '-verbose'])\\n\";\n if (args.length > 0 && args[0].equals(\"-h\")) {\n System.out.println(usage);\n System.exit(0);\n }\n\n SRILMFactoredBundleCorpusIterator in = null;\n BufferedWriter out = null;\n BufferedWriter voc = null;\n\n try {\n String inputCorp = \"\", output = \"\", vocabFile = \"vocab.voc\";\n int catCutoff = 10;\n\n for (int i = 0; i <\n args.length; i++) {\n if (args[i].equals(\"-c\")) { inputCorp = args[++i]; continue; }\n if (args[i].equals(\"-o\")) { output = args[++i]; continue; }\n if (args[i].equals(\"-vocab\")) {vocabFile = args[++i]; continue; }\n if (args[i].equals(\"-u\")) { catCutoff = Integer.parseInt(args[++i]); continue; }\n System.out.println(\"Unrecognized option: \" + args[i]);\n }\n \n try {\n in = new SRILMFactoredBundleCorpusIterator(\n (inputCorp.equals(\"\")) ? new BufferedReader(new InputStreamReader(System.in)) : new BufferedReader(new FileReader(new File(inputCorp))));\n } catch (FileNotFoundException ex) {\n System.err.print(\"Input corpus \" + inputCorp + \" not found. Exiting...\");\n Logger.getLogger(STPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n System.exit(\n -1);\n }\n\n try {\n out = (output.equals(\"\")) ? new BufferedWriter(new OutputStreamWriter(System.out)) : new BufferedWriter(new FileWriter(new File(output)));\n } catch (IOException ex) {\n System.err.print(\"Output file \" + output + \" not found. Exiting...\");\n Logger.getLogger(STPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n System.exit(\n -1);\n }\n\n try {\n voc = new BufferedWriter(new FileWriter(new File(vocabFile)));\n } catch (IOException ex) {\n Logger.getLogger(STPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n }\n\n Map vocab = new HashMap();\n for (List inLine : in) {\n for (Word w : inLine) {\n String st = SUPERTAG + \"-\" + DefaultTokenizer.escape(w.getSupertag()),\n pos = POS_TAG + \"-\" + DefaultTokenizer.escape(w.getPOS()),\n wform = WORD + \"-\" + DefaultTokenizer.escape(w.getForm());\n\n vocab.put(st, (vocab.get(st) == null) ? 1 : vocab.get(st) + 1);\n vocab.put(pos, (vocab.get(pos) == null) ? 1 : vocab.get(pos) + 1);\n vocab.put(wform, (vocab.get(wform) == null) ? 1 : vocab.get(wform) + 1);\n }\n\n }\n\n // reopen file\n try {\n in = new SRILMFactoredBundleCorpusIterator(\n (inputCorp.equals(\"\")) ? new BufferedReader(new InputStreamReader(System.in)) : new BufferedReader(new FileReader(new File(inputCorp))));\n } catch (FileNotFoundException ex) {\n System.err.print(\"Input corpus \" + inputCorp + \" not found. Exiting...\");\n Logger.getLogger(STPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n System.exit(\n -1);\n }\n for (List inLine : in) {\n for (Word w : inLine) {\n String st = SUPERTAG + \"-\" + DefaultTokenizer.escape(w.getSupertag()),\n pos = POS_TAG + \"-\" + DefaultTokenizer.escape(w.getPOS()),\n wform = WORD + \"-\" + DefaultTokenizer.escape(w.getForm());\n if (vocab.get(st) > catCutoff) {\n out.write(wform + \":\" + pos + \":\" + st + \" \");\n }\n }\n\n out.write(System.getProperty(\"line.separator\"));\n }\n\n out.flush();\n\n for (String str : vocab.keySet()) {\n if (vocab.get(str) > catCutoff) {\n voc.write(str + System.getProperty(\"line.separator\"));\n }\n }\n\n voc.flush();\n } finally {\n try {\n out.close();\n in.close();\n voc.close();\n } catch (IOException ex) {\n Logger.getLogger(STPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n }\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/supertagger/ml/ZhangLeTrainingExtractor.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger.ml;\n\nimport java.io.BufferedReader;\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.util.Collection;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.TreeMap;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.parse.tagger.io.PipeDelimitedFactoredBundleCorpusIterator;\nimport opennlp.ccg.parse.tagger.io.SRILMFactoredBundleCorpusIterator;\nimport opennlp.ccg.parse.supertagger.ml.FeatureExtractor;\nimport opennlp.ccg.parse.supertagger.ml.STFex;\nimport opennlp.ccg.parse.tagger.TaggedWord;\nimport opennlp.ccg.util.Pair;\n\n/**\n * @author Dennis N. Mehay\n * @version $Revision: 1.3 $, $Date: 2010/09/21 04:12:41 $\n */\npublic class ZhangLeTrainingExtractor {\n\n private File outputF;\n private Iterator> incorp;\n private FeatureExtractor fexer = new STFex();\n\n /**\n * Create a training feature extractor that will extract features (with results)\n * for every instance in the input (training) corpus corpusName.\n * \n * @param corpusName A String giving the complete\n * path to the input file of SRILM-compliant factored bundles.\n * @param outputFileName A String giving the complete\n * path to the output file where the features will be written.\n */\n public ZhangLeTrainingExtractor(File corpus, File outputF, String tokenisation) {\n this(corpus, outputF, tokenisation, new STFex());\n }\n \n public ZhangLeTrainingExtractor(File corpus, File outputF, String tokenisation, FeatureExtractor fexer) {\n this.fexer = fexer;\n this.outputF = outputF;\n try {\n if (tokenisation.equalsIgnoreCase(\"srilm\")) {\n incorp = new SRILMFactoredBundleCorpusIterator(new BufferedReader(new FileReader(corpus)));\n } else {\n incorp = new PipeDelimitedFactoredBundleCorpusIterator(new BufferedReader(new FileReader(corpus)));\n }\n } catch (FileNotFoundException ex) {\n Logger.getLogger(ZhangLeTrainingExtractor.class.getName()).log(Level.SEVERE, null, ex);\n }\n }\n\n /**\n * Writes training feats to file.\n */\n public void writeFeats() {\n BufferedWriter bw = null;\n try {\n try {\n bw = new BufferedWriter(new FileWriter(this.outputF));\n } catch (IOException ex) {\n Logger.getLogger(ZhangLeTrainingExtractor.class.getName()).log(Level.SEVERE, null, ex);\n }\n if (bw != null || this.incorp != null) {\n List sent = null;\n Map snt = null;\n\n Iterator> sents = this.incorp;\n while (sents.hasNext()) {\n //for (Iterator> sents = this.incorp; sents.hasNext();) {\n sent = sents.next();\n\n // turn the sent into a map from integer string indices to Words.\n int index = 0;\n snt = new TreeMap();\n for (Word w : sent) {\n snt.put(index++, new TaggedWord(w));\n }\n\n // 'true' says \"we're getting training feats\"\n for (Collection> sentFeatsWithActivation : fexer.getSentenceFeatures(snt, true)) {\n try {\n boolean isLabel = true;\n for (Pair ftWAct : sentFeatsWithActivation) {\n if (isLabel) {\n bw.write(ftWAct.a + \" \");\n isLabel = false;\n } else {\n bw.write(ftWAct.a + \":\" + ftWAct.b.doubleValue() + \" \");\n }\n }\n bw.newLine();\n } catch (IOException ex) {\n Logger.getLogger(ZhangLeTrainingExtractor.class.getName()).log(Level.SEVERE, null, ex);\n }\n\n }\n }\n }\n } finally {\n try {\n bw.flush();\n bw.close();\n } catch (IOException ex) {\n Logger.getLogger(ZhangLeTrainingExtractor.class.getName()).log(Level.SEVERE, null, ex);\n } catch (Exception e) {\n System.out.println(e);\n }\n }\n }\n}" }, { "path": "src/opennlp/ccg/parse/supertagger/util/PipedTokenizer.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger.util;\n\nimport java.util.List;\nimport opennlp.ccg.lexicon.DefaultTokenizer;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.util.Pair;\n\n/**\n * @author Dennis N. Mehay\n * @version $Revision: 1.1 $, $Date: 2009/08/21 17:20:20 $\n */\npublic class PipedTokenizer extends DefaultTokenizer {\n\n public PipedTokenizer() {\n super();\n }\n\n @Override\n public Word parseToken(String token, boolean strictFactors) {\n // init\n String form = token;\n String stem = null;\n String POS = null;\n String pitchAccent = null;\n String supertag = null;\n String semClass = null;\n List> attrValPairs = null;\n \n // handle pipe-separated attr-val pairs\n int pipePos = token.indexOf('|');\n String suffix = null;\n if (pipePos > 0) {\n // get word form\n form = token.substring(0, pipePos);\n // shave off word form\n suffix = token.substring(pipePos + 1);\n // get next | position\n pipePos = suffix.indexOf('|'); \n // get stem [or lemma]. could be null.\n stem = suffix.substring(0,pipePos);\n if (stem.equals(\"\")) { stem = null;}\n // shave off stem/lemma\n suffix = suffix.substring(pipePos + 1);\n // get next | position\n pipePos = suffix.indexOf('|');\n // get POS\n POS = suffix.substring(0,pipePos);\n // shave off POS\n suffix = suffix.substring(pipePos + 1);\n // see whether there is a supertag\n if (suffix != null && !suffix.equals(\"\")) {\n // get supertag\n supertag = suffix.trim();\n }\n } else {\n throw new RuntimeException(\"This file is not in the right format: \\n\"+\n \"form|lemma|POS|(Supertag) ... form|lemma|POS(Supertag).\");\n }\n // done\n return Word.createWord(form, pitchAccent, attrValPairs, stem, POS, supertag, semClass);\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/supertagger/util/ProbPairComparator.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse.supertagger.util;\n\nimport java.util.Comparator;\nimport opennlp.ccg.util.Pair;\n\n/**\n * @author Dennis N. Mehay\n * \n * A little Comparator instance for comparing Pair\n * instances of pair by descending order of the Double value (assuming\n * that they are probabilities of string tags).\n */\npublic class ProbPairComparator implements Comparator> {\n\n /**\n * Implements the Comparator interface's work-horse method.\n * \n * Compares two Pair objects. Crucially, it does NOT\n * ensure that both objects are Pair before casting them.\n * The caller is responsible for ensuring this, and failure to do so may\n * result in a RuntimeException.\n */\n public int compare(Pair pr1, Pair pr2) {\n if (pr1 == pr2) { return 0; }\n return -1 * Double.compare(pr1.a, pr2.a);\n\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/supertagger/util/STTaggerDictionary.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger.util;\n\n//import java.util.ArrayList;\nimport java.util.Collection;\n\n/**\n * An interface for supertagger \"dictionaries\" as described\n * in, e.g., Clark (2002) _Supertagging for CCG_.\n * \n * @author Dennis N. Mehay\n * @version $Revision: 1.1 $, $Date: 2009/08/21 17:20:20 $\n */\npublic interface STTaggerDictionary {\n\n /**\n * A method for getting the dictionary entry for a particular\n * String key. \n * The key will usually be a word, lemma or a part \n * of speech, but you may have other interesting grammatical things \n * to associate with supertags.\n * \n * @param key A String representing a particular\n * grammatical type.\n * @return A Collection containing supertags (CCG lexical \n * categories) seen with the particular grammatical type `key'.\n * Returns null if that word was not seen in the\n * corpus or (if the implementing class has a frequency cut-off)\n * if that type's token frequency was not high enough.\n */\n public Collection getEntry(String key);\n \n /**\n * A method to test whether this STTaggerDictionary contains \n * an entry for a particular String key. \n * The key will usually be a word, lemma or a part of speech, but \n * you may have other interesting grammatical things to associate with \n * supertags.\n * \n * @param key A String representing a particular\n * grammatical type.\n * @return A boolean value of true or \n * false answering the question of whether this\n * dictionary contains an entry for the specified key.\n */\n public boolean containsEntry(String key);\n}\n\n" }, { "path": "src/opennlp/ccg/parse/supertagger/util/STTaggerPOSDictionary.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger.util;\n\n//import util.Pair;\nimport java.io.*;\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.Iterator;\nimport java.util.Map;\nimport opennlp.ccg.util.Pair;\n\n/**\n * @author Dennis N. Mehay\n * @version $Revision: 1.2 $, $Date: 2010/09/21 04:12:41 $\n */\npublic class STTaggerPOSDictionary implements STTaggerDictionary, Serializable {\n\n\tprivate static final long serialVersionUID = -4814356608876054823L;\n\t\n\t/**\n * This object represents our dictionary. The String is the\n * POS we want to look up, and the Collection holds all of \n * of the CCG lex. cat's seen with that POS.\n */\n private Map> dict = null;\n\n \n /**\n * This constructor does not create the dictionary; that must\n * be done using a DictionaryMaker. \n */\n public STTaggerPOSDictionary(Map> dict) { \n \tthis.dict = dict;\n } \n\n /** \n * This method implements the interface STTaggerDictionary\n * by delegating to getEntry(String, int) (see below).\n */\n public Collection getEntry(String POS) {\n\treturn this.dict.get(POS);\n }\n\n /**\n * A method that returns the contents of the mapping embodied in this dictionary.\n * @return An Iterator of supertagger.util.Pairs\n * that represent the pos -> { ... supertags ...} mappings in the dictionary.\n */\n public Iterator>> getMappings() {\n Iterator keyset = this.dict.keySet().iterator();\n ArrayList>> preRes = \n new ArrayList>>();\n String tempS = null;\n while(keyset.hasNext()) {\n tempS = keyset.next();\n preRes.add(new Pair>(tempS, this.dict.get(tempS)));\n }\n return preRes.iterator();\n }\n \n /**\n * A method to test whether this STTaggerDictionary contains \n * an entry for a particular String representing a POS tag. \n *\n * @param key A String representing a particular\n * POS tag.\n * @return A boolean value of true or \n * false answering the question of whether this\n * dictionary contains an entry for the specified POS tag.\n */\n public boolean containsEntry(String POS) { return this.dict.containsKey(POS); }\n \n} // End class STTaggerPOSDictionary\n\n\n" }, { "path": "src/opennlp/ccg/parse/supertagger/util/STTaggerWordDictionary.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger.util;\n\nimport java.io.*;\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.Iterator;\nimport java.util.Map;\nimport opennlp.ccg.util.Pair;\n\n/**\n * @author Dennis N. Mehay\n * @version $Revision: 1.2 $, $Date: 2009/11/15 04:52:26 $\n */\npublic class STTaggerWordDictionary implements STTaggerDictionary, Serializable {\n\t\n\tprivate static final long serialVersionUID = -2474606825228545547L;\n\t\n\t/* This object represents our dictionary. The String is the\n\t * word we want to look up, and the Pair holds\n\t * the word frequency count (= `a') and the Collection of the CCG lex. cat's\n\t * seen with that word (= `b').\n\t */\n\tprivate Map>> dict = null;\n\n\n\t/* This constructor does not create the dictionary; that must\n\t * be done using a DictionaryMaker. \n\t */\n\tpublic STTaggerWordDictionary(Map>> dict) { \n\t\tthis.dict = dict;\n\t} \n\n\t/** \n\t * This method implements the interface STTaggerDictionary\n\t * by delegating to getEntry(String, int) (see below).\n\t */\n\tpublic Collection getEntry(String word) {\n\t\treturn getEntry(word, 1);\n\t}\n\n\t/**\n\t * A method for getting the dictionary entry for a particular\n\t * String word, only if that word appears at least\n\t * `freq' times in the corpus. \n\t * \n\t * @param key A String representing a particular\n\t * word.\n\t * @param freq An int specifying the number of times a \n\t * word should have occured before it is returned.\n\t * @return A String[] containing all supertags (CCG lexical \n\t * categories) seen with the specified word.\n\t * Returns null if the word does not appear at least\n\t * `freq' times in the corpus from which the dictionary was \n\t * created or if the word does not appear at all.\n\t * (N.B. Passing in an int `freq' value of zero will elicit the\n\t * same behavior as passing in a `freq' value of 1.)\n\t */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic Collection getEntry(String word, int freq) {\n\t\tif (word == null) return null; // mww: extra null check\n\t\tObject o = this.dict.get(word);\n\t\tif(o==null) { return null; }\n\n\t\tPair> p = (Pair>)o;\n\t\tint wfreq = p.a.intValue();\n\t\tif(wfreq>=freq) {\n\t\t\treturn p.b;\n\t\t}\n\t\telse {\n\t\t\treturn null;\n\t\t}\n\t} // End method getEntry(String, int)\n\n\t/**\n\t * A method to test whether this STTaggerDictionary contains \n\t * an entry for a particular String representing a word. \n\t *\n\t * @param key A String representing a particular\n\t * word.\n\t * @return A boolean value of true or \n\t * false answering the question of whether this\n\t * dictionary contains an entry for the specified word.\n\t */\n\tpublic boolean containsEntry(String word) { return this.dict.containsKey(word); }\n\n\t/**\n\t * A method that returns the contents of the mapping embodied in this dictionary.\n\t * @return An Iterator of supertagger.util.Pairs\n\t * that represent the word -> (freq, { ... supertags ...}) mappings in the dictionary.\n\t */\n\tpublic Iterator>>> getMappings() {\n\t\tIterator keyset = this.dict.keySet().iterator();\n\t\tArrayList>>> preRes = \n\t\t\tnew ArrayList>>>();\n\t\tString tempS = null;\n\t\twhile(keyset.hasNext()) {\n\t\t\ttempS = keyset.next();\n\t\t\tpreRes.add(\n\t\t\t\t\tnew Pair>>(tempS, this.dict.get(tempS)));\n\t\t}\n\t\treturn preRes.iterator();\n\t}\n\n\t/** \n\t * A method for getting the number of times a word was seen \n\t * in the training data with which this dictionary was created.\n\t * \n\t * @param word A String representing the word in question.\n\t * @return An int count of this word's frequency in the \n\t * corpus with which this dictionary was created.\n\t */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic int getCount(String word) { \n\t\tObject o = dict.get(word);\n\t\tif(o==null) { return 0; }\n\t\telse {\n\t\t\tPair> p = (Pair>)o;\n\t\t\treturn p.a.intValue();\n\t\t}\n\t} // End method getCount(word)\n} // End class STTaggerWordDictionary\n\n\n\n\n\n" }, { "path": "src/opennlp/ccg/parse/supertagger/util/SupertagSequenceGetter.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse.supertagger.util;\n\nimport java.io.BufferedReader;\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.util.List;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.parse.tagger.io.SRILMFactoredBundleCorpusIterator;\n\n/**\n * @author Dennis N. Mehay\n */\npublic class SupertagSequenceGetter {\n public static void main(String[] args) throws FileNotFoundException, IOException {\n String usage = \"\\nSupertagSequenceGetter -i -o \\n\";\n String input = null, output = null;\n if(args == null || args.length == 0 || args[0].equals(\"-h\")) {\n System.err.println(usage);\n System.exit(0);\n }\n for(int i = 0; i < args.length; i++) {\n if(args[i].equals(\"-i\")) { input = args[++i]; continue; }\n if(args[i].equals(\"-o\")) { output = args[++i]; continue; }\n System.err.println(\"unknown command-line option: \" + args[i]);\n }\n \n BufferedReader in = new BufferedReader(new FileReader(new File(input)));\n SRILMFactoredBundleCorpusIterator corp = new SRILMFactoredBundleCorpusIterator(in);\n BufferedWriter out = new BufferedWriter(new FileWriter(new File(output)));\n \n \n for(List sent : corp) {\n out.write(\" \");\n for(Word w : sent) {\n out.write(w.getSupertag()+\" \");\n }\n out.write(\"\"+System.getProperty(\"line.separator\"));\n }\n out.close();\n }\n \n}\n" }, { "path": "src/opennlp/ccg/parse/supertagger/util/TaggingDictionaryExtractor.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.supertagger.util;\n\nimport java.io.BufferedReader;\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.text.CharacterIterator;\nimport java.text.StringCharacterIterator;\nimport java.util.HashMap;\nimport java.util.HashSet;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.parse.tagger.io.PipeDelimitedFactoredBundleCorpusIterator;\nimport opennlp.ccg.parse.tagger.io.SRILMFactoredBundleCorpusIterator;\nimport opennlp.ccg.util.Pair;\n\n/**\n * Extracts and writes out XML files containing tagging\n * dictionary stats.\n * \n * @author Dennis N. Mehay\n * @version $Revision: 1.5 $, $Date: 2010/09/21 04:12:41 $\n */\npublic class TaggingDictionaryExtractor {\n\n /**\n * Create a new dictionary extractor, specifying the word and POS\n * dictionary files, as well as the tokenisation type (SRILM Factored bundle or\n * C&C/Moses-style pipe-delimited factored bundles.\n * @param corpus A File of plain-text, one sentence per line and no additional mark-up beyond\n * the ... for SRILM factored bundle style.\n * @param wd A File where the word-based tagging dictionary will be written.\n * @param posd A File where the POS-based tagging dictionary will be written.\n * @param tokenisationType A String telling us how to tokenise factors in the \n * corpus file.\n */\n // mapping from words to a pairing of their frequencies and the lexical categories they were seen with.\n private Map>> wdmap = new HashMap>>();\n // mapping from POS tags to the lexical categories they were seen with.\n private Map> posmap = new HashMap>();\n private Iterator> incorp = null;\n // writers for dict files.\n private BufferedWriter wbr = null, pbr = null;\n\n // how frequently a cat must occur to make it into the dictionaries.\n private int minCatFreq = 1;\n \n /**\n * Escape characters for text appearing as XML data, between tags.\n * \n *

    The following characters are replaced with corresponding character entities :\n * \n * \n * \n * \n * \n * \n * \n *
    Character Encoding
    < <
    > >
    & &
    \" "
    ' '
    \n * \n *

    Note that JSTL's {@code } escapes the exact same set of \n * characters as this method. That is, {@code }\n * is good for escaping to produce valid XML, but not for producing safe \n * HTML.\n */\n public static String forXML(String aText) {\n \tif (aText == null) return null;\n final StringBuilder result = new StringBuilder();\n final StringCharacterIterator iterator = new StringCharacterIterator(aText);\n char character = iterator.current();\n while (character != CharacterIterator.DONE) {\n if (character == '<') {\n result.append(\"<\");\n } else if (character == '>') {\n result.append(\">\");\n } else if (character == '\\\"') {\n result.append(\""\");\n } else if (character == '\\'') {\n result.append(\"'\");\n } else if (character == '&') {\n result.append(\"&\");\n } else {\n //the char is not a special one\n //add it to the result as is\n result.append(character);\n }\n character = iterator.next();\n }\n return result.toString();\n }\n\n public TaggingDictionaryExtractor(File corpus, File wd, File posd, String tokenisationType) {\n this(corpus, wd, posd, tokenisationType, 10);\n }\n \n public TaggingDictionaryExtractor(File corpus, File wd, File posd, String tokenisationType, int catFreq) {\n try {\n wbr = new BufferedWriter(new FileWriter(wd));\n pbr = new BufferedWriter(new FileWriter(posd));\n minCatFreq = catFreq;\n if (tokenisationType.equalsIgnoreCase(\"srilm\")) {\n incorp = new SRILMFactoredBundleCorpusIterator(new BufferedReader(new FileReader(corpus)));\n } else {\n incorp = new PipeDelimitedFactoredBundleCorpusIterator(new BufferedReader(new FileReader(corpus)));\n }\n } catch (IOException ex) {\n Logger.getLogger(TaggingDictionaryExtractor.class.getName()).log(Level.SEVERE, null, ex);\n }\n }\n\n /**\n * Extract the dictionaries.\n */\n @SuppressWarnings(\"unchecked\")\n\tpublic void extract() {\n try {\n List currsent = null;\n String currForm = null, currPOS = null;\n //Set currFormSTs = null, curPOSSTs = null;\n //Integer currWdCnt = null;\n Pair> currFormFetch = null;\n Set currFormSet = null, currPOSSet = null;\n\n Map catCount = new HashMap();\n \n while (incorp.hasNext()) {\n // for every word in every sentence, update the counts, and add to the word- and POS-based\n // allowable tags.\n currsent = incorp.next();\n for (Word w : currsent) {\n currForm = w.getForm();\n currPOS = w.getPOS();\n Object wfetch = wdmap.get(currForm);\n String stag = w.getSupertag();\n catCount.put(stag, catCount.get(stag)==null ? 1 : catCount.get(stag) + 1);\n \n if (wfetch == null) {\n currFormSet = new HashSet();\n currFormSet.add(w.getSupertag());\n wdmap.put(currForm, new Pair>(new Integer(1), currFormSet));\n } else {\n currFormFetch = ((Pair>) wfetch);\n currFormSet = currFormFetch.b;\n currFormSet.add(w.getSupertag());\n wdmap.put(currForm,\n new Pair>(new Integer(currFormFetch.a.intValue() + 1), currFormSet));\n }\n\n Object pfetch = posmap.get(currPOS);\n if (pfetch == null) {\n currPOSSet = new HashSet();\n currPOSSet.add(w.getSupertag());\n } else {\n currPOSSet = (Set) pfetch;\n currPOSSet.add(w.getSupertag());\n }\n \n posmap.put(currPOS, currPOSSet);\n }\n }\n\n // now write out the dictionaries.\n String wrd = null;\n Pair> lkup = null;\n wbr.write(\"\\n\");\n wbr.write(\"\\n\");\n for (Object wdobj : wdmap.keySet()) {\n wrd = (String) wdobj;\n lkup = (Pair>) (wdmap.get(wdobj));\n wbr.write(\"\\t\\n\"); \n for (String st : lkup.b) {\n if(catCount.get(st) >= minCatFreq) {\n wbr.write(\"\\t\\t \" + forXML(st) + \" \\n\");\n }\n }\n wbr.write(\"\\t\\n\");\n }\n wbr.write(\"\");\n\n String pos = null;\n Set plkup = null;\n pbr.write(\"\\n\");\n pbr.write(\"\\n\");\n for (Object pobj : posmap.keySet()) {\n pos = (String) pobj;\n plkup = (Set) posmap.get(pobj);\n pbr.write(\"\\t\\n\");\n for (String st : plkup) {\n if(catCount.get(st) >= minCatFreq) {\n pbr.write(\"\\t\\t \" + forXML(st) + \" \\n\");\n }\n }\n pbr.write(\"\\t\\n\");\n }\n pbr.write(\"\");\n\n // clean up.\n wbr.flush();\n wbr.close();\n pbr.flush();\n pbr.close();\n // done\n } catch (FileNotFoundException ex) {\n Logger.getLogger(TaggingDictionaryExtractor.class.getName()).log(Level.SEVERE, null, ex);\n } catch (IOException e) {\n Logger.getLogger(TaggingDictionaryExtractor.class.getName()).log(Level.SEVERE, null, e);\n }\n }\n \n public static void main(String[] args) throws Exception {\n String usage =\"\\nTaggingDictionaryExtractor -i -f -p -w \\n\\n\";\n if(args.length > 0 && args[0].equals(\"-h\")) {\n System.out.print(usage);\n System.exit(0);\n }\n \n String inputCorp = null, wOutput = null, pOutput = null;\n // how frequently must a supertag category have been seen to be included in the dictionary?\n int catFreq = 10;\n for(int i = 0; i < args.length; i++) {\n if(args[i].equals(\"-i\")) {inputCorp = args[++i]; continue;}\n if(args[i].equals(\"-w\")) {wOutput = args[++i]; continue;}\n if(args[i].equals(\"-p\")) {pOutput = args[++i]; continue;}\n if(args[i].equals(\"-f\")) {catFreq = Integer.parseInt(args[++i]); continue; }\n System.err.println(\"Unknown command-line option: \"+args[i]);\n }\n \n File in = new File(inputCorp); \n File wout = new File(wOutput);\n File pout = new File(pOutput);\n TaggingDictionaryExtractor tde = new TaggingDictionaryExtractor(in, wout, pout, \"SRILM\", catFreq);\n tde.extract();\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/Constants.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse.tagger;\nimport opennlp.ccg.lexicon.Word;\n\n/**\n * @author Dennis N. Mehay\n */\npublic final class Constants {\n public static final Double one = new Double(1.0);\n public static final Double zero = new Double(0.0);\n public static final TaggedWord OOB = new TaggedWord(Word.createWord(\"OOS\", null, null, \"OOS\", \"OOS\", \"OOS\", null));\n public static enum Domain {PROB, LOGPROB};\n public static enum TaggingAlgorithm {FORWARDBACKWARD, FORWARD};\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/ProbIndexPair.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.tagger;\n\n/**\n * @author Dennis N. Mehay\n * @version $Revision: 1.1 $, $Date: 2010/09/21 04:12:41 $\n */\npublic class ProbIndexPair implements Comparable {\n\t\n public Double a;\n public Integer b;\n \n public ProbIndexPair(Double a, Integer b) { this.a=a; this.b=b; }\n \n public int compareTo(ProbIndexPair p) { return (-1 * (this.a).compareTo(p.a)); }\n \n public static void main(String[] args) {\n ProbIndexPair p1 = new ProbIndexPair(new Double(4.0), new Integer(5));\n ProbIndexPair p2 = new ProbIndexPair(new Double(3.0), new Integer(5));\n ProbIndexPair p3 = new ProbIndexPair(new Double(2.0), new Integer(5));\n ProbIndexPair p4 = new ProbIndexPair(new Double(4.0), new Integer(5));\n System.out.println(\"p1 < p2? \"+(p1.compareTo(p2)<0));\n System.out.println(\"p2 < p3? \"+(p2.compareTo(p3)<0));\n System.out.println(\"p1 < p3? \"+(p1.compareTo(p3)<0));\n System.out.println(\"p1 == p4? \"+(p1.compareTo(p4)==0));\n System.out.println(\"p2 > p1? \"+(p2.compareTo(p1)>0));\n System.out.println(\"p3 > p2? \"+(p3.compareTo(p2)>0));\n System.out.println(\"p3 > p1? \"+(p3.compareTo(p1)>0));\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/TaggedWord.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n/**\n * A wrapper around {@code Word}s that can hold multitaggings (for POSs and for\n * supertags).\n */\n\npackage opennlp.ccg.parse.tagger;\n\nimport java.util.List;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.util.Pair;\n\n/**\n * @author Dennis N. Mehay\n */\npublic class TaggedWord {\n // multitaggings for POSs and supertags (resp).\n private List> postagging; \n private List> stagging; \n \n // old-timey Word that holds the word form (and potentially gold POS and supertag).\n private Word oldWord;\n \n /** Decorators for the core functionality of the underlying word. */\n public String getSupertag() { return oldWord.getSupertag(); }\n public String getForm() { return oldWord.getForm(); }\n public String getPOS() { return oldWord.getPOS(); }\n \n /** Accessor for the underlying vanilla Word. */\n public Word getWord() { return oldWord; }\n \n /** Constructor with a Word. */\n public TaggedWord(Word wd) { \n oldWord = Word.createFullWord(wd, wd.getForm(), wd.getPOS(), wd.getSupertag(), wd.getSemClass()); \n }\n \n /** This does the obvious thing. */\n public void setSupertagging(List> stagging) { this.stagging = stagging; }\n \n /** \n * Set the multi-POS tagging. \n * Also replace the underlying single-best tagging with the\n * first tag of the multitag list.\n */\n public void setPOSTagging(List> postagging) { \n this.postagging = postagging; \n oldWord = Word.createFullWord(oldWord, oldWord.getForm(), this.postagging.get(0).b, oldWord.getSupertag(), oldWord.getSemClass()); \n }\n \n \n /** This does the obvious thing. */\n public List> getSupertagging() { return stagging; }\n \n /** This does the obvious thing. */\n public List> getPOSTagging() { return postagging; }\n \n /** Gets the gold-standard supertag. */\n public String getGoldSuper() { return oldWord.getSupertag(); }\n \n /** Gets the gold-standard POS tag. */\n public String getGoldPOS() { return oldWord.getPOS(); }\n \n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/io/CorpusIterator.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.tagger.io;\n\nimport java.io.IOException;\nimport java.util.Iterator;\nimport java.util.List;\nimport opennlp.ccg.lexicon.Word;\n\n/**\n * An interface that all file iterators must (should?) implement.\n * \n * @author Dennis N. Mehay\n * @version $Revision: 1.1 $, $Date: 2010/09/21 04:12:41 $\n */\npublic interface CorpusIterator {\n \n public List next() throws IOException;\n \n public boolean hasNext();\n \n public void close();\n \n public Iterator> iterator();\n \n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/io/PipeDelimitedFactoredBundleCorpusIterator.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.tagger.io;\n\nimport java.io.BufferedReader;\nimport java.io.IOException;\nimport java.util.Iterator;\nimport java.util.List;\nimport opennlp.ccg.lexicon.Tokenizer;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.parse.supertagger.util.PipedTokenizer;\n\n/**\n * Assuming an input file of n different sentences of the form:\n * wordbundle1 wordbundle2 ... wordbundleM \n * ... [n-2 lines]\n * wordbundle1 ... wordbundleQ \n * \n * where the 'wordbundle's are SRILM factored LM-compliant\n * bundles of factors (wordform, lemma, POS, supertag, semantic class, etc.).\n * \n * We assume one sentence per line, so the ... bracketing is just\n * a formality (what SRILM expects).\n * \n * @author Dennis N. Mehay\n * @version $Revision: 1.2 $, $Date: 2010/09/26 05:50:15 $\n */\npublic class PipeDelimitedFactoredBundleCorpusIterator implements CorpusIterator, Iterator>, Iterable> {\n\n private BufferedReader reader;\n private String nextLine, nextID;\n public static final String SENT_START = \"\", SENT_END = \"\";\n private Tokenizer toker = new PipedTokenizer();\n\n /** Creates a new instance of SRILMFactoredBundleCorpusIterator */\n public PipeDelimitedFactoredBundleCorpusIterator(BufferedReader file) {\n try {\n this.reader = file;\n String line = this.reader.readLine();\n //while (line.length() == 0 && line != null) {\n // line = this.reader.readLine();\n //}\n if (line != null && line.length() > 0) {\n line = line.trim();\n this.nextLine = line;\n } else {\n this.nextLine = this.nextID = null;\n }\n } catch (Exception e) {\n e.printStackTrace();\n }\n }\n\n /**\n * The client of this method is responsible for checking that there\n * is in fact a next line (by calling hasNext before \n * calling this method.\n * \n * @return A String representing the next line in the \n * file.\n * @throws java.io.IOException. \n */\n public List next() {\n List currentSent = null;\n try {\n if (this.hasNext()) {\n String line = this.reader.readLine();\n //while(line != null && (line.length()==0 || line.trim().equals(\"\"))) {\n // line = this.reader.readLine();\n //}\n if (line != null) {\n line = line.trim();\n }\n if (this.nextLine.endsWith(SENT_END)) {\n currentSent = toker.tokenize(this.nextLine.substring(this.nextLine.indexOf(\">\") + 1, this.nextLine.lastIndexOf(\"<\")).trim());\n } else {\n currentSent = toker.tokenize(this.nextLine.substring(this.nextLine.indexOf(\">\") + 1).trim());\n }\n\n if (line != null && !(line.trim().equals(\"\"))) {\n this.nextLine = line;\n } else {\n this.nextLine = this.nextID = null;\n }\n } else {\n throw new IOException(\"There is no next line.\");\n }\n } catch (IOException ex) {\n java.util.logging.Logger.getLogger(\"global\").log(java.util.logging.Level.SEVERE,\n ex.getMessage(), ex);\n }\n return currentSent;\n }\n\n /**\n * @return A String representing the current parse ID\n * (a la CCGbank).\n */\n public String getCurrentID() {\n return this.nextID;\n }\n\n /**\n * @return A boolean as to whether there is a next line\n * in the file. \n */\n public boolean hasNext() {\n return this.nextLine != null;\n }\n\n /**\n * Closes the underlying BufferedReader.\n * \n */\n public void close() {\n try {\n this.reader.close();\n } catch (IOException ex) {\n java.util.logging.Logger.getLogger(\"global\").log(java.util.logging.Level.SEVERE,\n ex.getMessage(), ex);\n }\n }\n\n public void remove() {\n throw new UnsupportedOperationException(\"Not supported yet.\");\n }\n \n public Iterator> iterator() { return this; }\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/io/SRILMFactoredBundleCorpusIterator.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.tagger.io;\n\nimport java.io.BufferedReader;\nimport java.io.IOException;\nimport java.util.Iterator;\nimport java.util.List;\nimport opennlp.ccg.lexicon.DefaultTokenizer;\nimport opennlp.ccg.lexicon.Tokenizer;\nimport opennlp.ccg.lexicon.Word;\n\n/**\n * Assuming an input file of n different sentences of the form:\n * wordbundle1 wordbundle2 ... wordbundleM \n * ... [n-2 lines]\n * wordbundle1 ... wordbundleQ \n * \n * where the 'wordbundle's are SRILM factored LM-compliant\n * bundles of factors (wordform, lemma, POS, supertag, semantic class, etc.).\n * \n * We assume one sentence per line, so the ... bracketing is just\n * a formality (what SRILM expects).\n * \n * @author Dennis N. Mehay\n * @version $Revision: 1.2 $, $Date: 2010/09/26 05:50:15 $\n */\npublic class SRILMFactoredBundleCorpusIterator implements CorpusIterator, Iterator>, Iterable> {\n\n private BufferedReader reader;\n private String nextLine, nextID;\n public static final String SENT_START = \"\", SENT_END = \"\";\n private Tokenizer toker = new DefaultTokenizer();\n\n /** Creates a new instance of SRILMFactoredBundleCorpusIterator */\n public SRILMFactoredBundleCorpusIterator(BufferedReader file) {\n try {\n this.reader = file;\n String line = this.reader.readLine();\n if (line != null && line.length() > 0) {\n line = line.trim();\n this.nextLine = line;\n } else {\n this.nextLine = this.nextID = null;\n }\n } catch (Exception e) {\n e.printStackTrace();\n }\n }\n\n /**\n * The client of this method is responsible for checking that there\n * is in fact a next line (by calling hasNext before \n * calling this method.\n * \n * @return A String representing the next line in the \n * file.\n * @throws java.io.IOException. \n */\n public List next() {\n List currentSent = null;\n try {\n if (this.hasNext()) {\n String line = this.reader.readLine();\n //while(line != null && (line.length()==0 || line.trim().equals(\"\"))) {\n // line = this.reader.readLine();\n //}\n if (line != null) {\n line = line.trim();\n }\n if (this.nextLine.endsWith(SENT_END)) {\n currentSent = toker.tokenize(this.nextLine.substring(this.nextLine.indexOf(\">\") + 1, this.nextLine.lastIndexOf(\"<\")).trim());\n } else {\n currentSent = toker.tokenize(this.nextLine.substring(this.nextLine.indexOf(\">\") + 1).trim());\n }\n\n if (line != null && !(line.trim().equals(\"\"))) {\n this.nextLine = line;\n } else {\n this.nextLine = this.nextID = null;\n }\n } else {\n throw new IOException(\"There is no next line.\");\n }\n } catch (IOException ex) {\n java.util.logging.Logger.getLogger(\"global\").log(java.util.logging.Level.SEVERE,\n ex.getMessage(), ex);\n }\n return currentSent;\n }\n\n /**\n * @return A String representing the current parse ID\n * (a la CCGbank).\n */\n public String getCurrentID() {\n return this.nextID;\n }\n\n /**\n * @return A boolean as to whether there is a next line\n * in the file. \n */\n public boolean hasNext() {\n return this.nextLine != null;\n }\n\n /**\n * Closes the underlying BufferedReader.\n * \n */\n public void close() {\n try {\n this.reader.close();\n } catch (IOException ex) {\n java.util.logging.Logger.getLogger(\"global\").log(java.util.logging.Level.SEVERE,\n ex.getMessage(), ex);\n }\n }\n\n public void remove() {\n throw new UnsupportedOperationException(\"Not supported yet.\");\n }\n\n public Iterator> iterator() {\n return this;\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/ml/MaxentModel.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.tagger.ml;\n\nimport java.util.Collection;\nimport opennlp.ccg.util.Pair;\n\n/**\n * @author Dennis N. Mehay\n * @version $Revision: 1.1 $, $Date: 2010/09/21 04:12:41 $\n */\npublic interface MaxentModel {\n public double[] eval(Collection> context);\n public String getOutcome(int indexOfOutcome);\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/ml/TaggerFeature.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse.tagger.ml;\n\n/**\n * @author Dennis N. Mehay\n */\npublic class TaggerFeature {\n public String name;\n public Double activation;\n \n public TaggerFeature(String name, Double activation) {\n this.name = name.intern();\n this.activation = activation;\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/ml/ZLMEM.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.tagger.ml;\nimport java.io.File;\nimport java.util.Collection;\nimport opennlp.ccg.util.Pair;\nimport opennlp.ccg.parse.tagger.Constants;\n\n/**\n * Decorates ZLMaxentModel, making it a MaxentModel.\n * \n * @author Dennis N. Mehay\n * @version $Revision: 1.2 $, $Date: 2010/09/26 05:50:15 $\n */\npublic class ZLMEM extends ZLMaxentModel implements MaxentModel {\n\t\n public ZLMEM(File model) {\n super(model);\n }\n \n /**\n * @param context: A collection of String,Double pairs, representing the contextual input\n * features and their activations.\n * @return a double[] which represents a probability distribution over output classes, each\n * retrievable by its index with getOutcome(index);\n */\n public double[] eval(Collection> context) {\n // Have to turn a collection of pairs into a String[] of feature:activation Strings.\n // Sloppy and inefficient. \n // TODO: A better solution would be to refactor ZLMaxentModel (DNM)\n String[] inpt = new String[context.size()];\n int index = -1;\n for(Pair inp : context) {\n inpt[++index] = inp.a + \":\" + inp.b;\n }\n \n return super.eval(context, true, Constants.Domain.PROB);\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/ml/ZLMaxentModel.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse.tagger.ml;\n\n/* A nearly literal translation of Zhang Le's pymaxent.py file\n * into Java (D.N. Mehay).\n */\n\nimport opennlp.ccg.parse.tagger.Constants.Domain;\nimport opennlp.ccg.parse.tagger.Constants;\nimport java.io.BufferedReader;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.HashMap;\nimport java.util.Map;\nimport opennlp.ccg.util.Pair;\n\npublic class ZLMaxentModel {\n public boolean verbose = false;\n private ItemMap predMap = null;\n private ItemMap outComeMap = null;\n ArrayList>> paramsMap;\n private boolean loaded = false;\n private double[] probs;\n private int n_outcome;\n // the parameters.\n private double[] theta;\n private Double one = Constants.one;\n\n public ZLMaxentModel() {\n }\n\n public ZLMaxentModel(File model) {\n load(model);\n }\n\n public void load(File modelFile) {\n if (!loaded) {\n loaded = true;\n BufferedReader br = null;\n try {\n br = new BufferedReader(new FileReader(modelFile));\n String line = br.readLine();\n\n if (line.contains(\"#\")) {\n line = br.readLine();\n }\n\n if (verbose) System.err.println(\"\\nReading predicates...\");\n // Read in contextual predicates.\n int numPreds = Integer.parseInt(line);\n predMap = new ItemMap();\n // read in predicates...\n for (int i = 0; i < numPreds; i++) {\n line = br.readLine();\n predMap.add(line);\n }\n\n if (verbose) System.err.println(\"Reading outcomes...\");\n outComeMap = new ItemMap();\n // Read in outcomes (labels).\n line = br.readLine();\n int numOutcomes = Integer.parseInt(line);\n for (int j = 0; j < numOutcomes; j++) {\n line = br.readLine();\n outComeMap.add(line);\n }\n\n if (verbose) System.err.println(\"Reading parameters...\");\n // Read parameters. \n int numParameters = predMap.size();\n paramsMap = new ArrayList>>(numParameters);\n ArrayList> prms;\n int fid = 0;\n String ln = \"\";\n for (int q = 0; q < numParameters; q++) {\n ln = br.readLine();\n String[] lineParts = ln.split(\" \");\n prms = new ArrayList>(Integer.parseInt(lineParts[0]));\n Integer oid;\n for (int p = 1; p < lineParts.length; p++) {\n oid = Integer.valueOf(lineParts[p]); \n prms.add(new Pair(oid, Integer.valueOf(fid))); \n fid++;\n }\n paramsMap.add(prms);\n }\n \n // Load theta.\n int nTheta = Integer.valueOf(br.readLine());\n if (verbose) System.err.println(\"Number of parameters: \" + nTheta);\n theta = new double[nTheta];\n for (int z = 0; z < theta.length; z++) {\n theta[z] = Double.parseDouble(br.readLine());\n }\n\n n_outcome = outComeMap.size();\n // Initialise the array for computing distribution over all labels.\n probs = new double[n_outcome];\n if (verbose) System.err.println(\"Number of outcomes: \" + n_outcome);\n\n } catch (FileNotFoundException e) {\n e.printStackTrace();\n } catch (IOException e) {\n e.printStackTrace();\n } finally {\n try {\n br.close();\n } catch (IOException ioe) {\n ioe.printStackTrace();\n }\n }\n }\n }\n\n public String getBestOutcome(double[] probs) {\n double maxprob = 0.0;\n int maxidx = -1;\n for (int i = 0; i < probs.length; i++) {\n if (probs[i] > maxprob) {\n maxidx = i;\n maxprob = probs[i];\n }\n }\n return getOutcome(maxidx);\n }\n\n public double[] eval(Collection> context, boolean realValued) {\n return eval(context, realValued, Domain.PROB);\n }\n\n public double[] eval(Collection> context, boolean realValued, Domain domain) {\n // Zero out prob distribution over labels.\n for (int i = 0; i < probs.length; i++) {\n probs[i] = 0.0;\n // build up exponentiated scores. \n }\n for (Pair pv : context) {\n Integer predID = predMap.id(pv.a);\n if (predID != null) {\n ArrayList> featClassAssocs = paramsMap.get(predID.intValue());\n for (Pair classAndAssoc : featClassAssocs) {\n if (pv.b == one) {\n // ln(exp(lambda * 1)) = ln(exp(lambda)^1) = ln(exp(lambda)) = lambda\n probs[classAndAssoc.a.intValue()] += theta[classAndAssoc.b.intValue()];\n } else {\n // ln(exp(lambda * )) = ln(exp(lambda)^)\n probs[classAndAssoc.a.intValue()] += Math.log(Math.exp(theta[classAndAssoc.b.intValue()] * pv.b.doubleValue()));\n }\n }\n }\n }\n\n double sum = 0.0;\n // exponentiate the numerators for the denomenator sum.\n for (int p = 0; p < probs.length; p++) {\n sum += Math.exp(probs[p]);\n }\n sum = Math.log(sum);\n for (int q = 0; q < probs.length; q++) {\n probs[q] -= sum;\n }\n if (domain == Domain.PROB) {\n // translate back from the log domain.\n for (int q = 0; q < probs.length; q++) {\n probs[q] = Math.exp(probs[q]);\n }\n }\n return probs;\n }\n\n public String getOutcome(int index) {\n return outComeMap.getItem(index);\n }\n}\n\nclass ItemMap {\n\n private Integer index;\n private Map dict = new HashMap();\n private Map reverseDict = new HashMap();\n\n public ItemMap() {\n index = Integer.valueOf(0);\n }\n\n public int add(String item) {\n if (dict.containsKey(item)) {\n return dict.get(item);\n } else {\n dict.put(item, index);\n reverseDict.put(index, item);\n index = Integer.valueOf(index.intValue() + 1);\n return index.intValue() - 1;\n\n }\n\n }\n\n public Integer id(String item) {\n if (dict.containsKey(item)) {\n return dict.get(item);\n } else {\n return null;\n }\n }\n\n public int size() {\n return dict.size();\n }\n\n public String getItem(int i) {\n return reverseDict.get(Integer.valueOf(i));\n }\n}\n\nclass IntegerPool {\n\n private Integer[] _table;\n\n public IntegerPool(int size) {\n _table = new Integer[size];\n for (int i = 0; i < size; i++) {\n _table[i] = new Integer(i);\n }\n }\n\n public Integer getInt(int i) {\n if (i < _table.length && i >= 0) {\n return _table[i];\n } else {\n return new Integer(i);\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/sequencescoring/Backpointer.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n\npackage opennlp.ccg.parse.tagger.sequencescoring;\n\nimport java.util.List;\n\n/**\n * An abstraction for lists of backpointers in trellises, lattices, etc.\n * Each backpointer is a List of Integer's that give backpointers to the j-th highest\n * scoring paths (where 1 <= j <= N==len(Backpointer) are the indices of the \n * internal list of the backpointer).\n * \n * @author Dennis N. Mehay\n */\npublic class Backpointer {\n private List bkpts;\n public Backpointer(List bkpts) {\n this.bkpts = bkpts;\n }\n public List getBkpts() { return bkpts; }\n public Integer get(int i) { return bkpts.get(i); }\n public int size() { return bkpts.size(); }\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/sequencescoring/FBNode.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n\npackage opennlp.ccg.parse.tagger.sequencescoring;\n\nimport java.util.List;\n\n/**\n * A node in a forward-backward lattice.\n * \n * Holds the current label, the forward score, backward score, a list of Doubles representing\n * this node's contribution to the scores of the following nodes in the next\n * step of the lattice, a list of Strings representing the Markov history of the\n * optimal sequence leading up to this node, and, finally, holds a ranked array of\n * backpointers to the n-best optimal predecessor nodes.\n * \n * @author Dennis N. Mehay\n */\npublic class FBNode { \n /** How far back of a Markov history window do we have? */\n public int markovHistSize; \n /** The label at this node. */\n public String label;\n /** The (normalised) sum of the log-probabilites of all paths leading to this node. */\n public double forwardScore = 0.0;\n /** The (normalised) sum of the log-probabilites of all paths starting at this node. */\n public double backwardScore = 0.0;\n /** \n * How does this node contribute to each of the nodes in the next time step in\n * the lattice?\n */ \n public List forwardContributions; \n /** The list of the optimal Markov history. */\n public List markovHist;\n /** \n * A list of backpointers to the nodes in the previous time step (ranked in order\n * of how likely the sequence including them leading to this node is).\n */\n public List backpointers; \n \n /** \n * Empty constructor. Default Markov history of length 2. \n * All other values are set directly in the fields as they are calculated. \n */\n public FBNode() { this(2); }\n \n /** Constructor that only specifies Markov history size. */\n public FBNode(int markovHistSize) {\n this.markovHistSize = markovHistSize;\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/sequencescoring/SequenceScorer.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse.tagger.sequencescoring;\n\nimport java.io.BufferedReader;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.List;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.ngrams.StandardNgramModel;\nimport opennlp.ccg.util.Interner;\nimport opennlp.ccg.util.Pair;\nimport opennlp.ccg.parse.tagger.ProbIndexPair;\nimport opennlp.ccg.parse.tagger.Constants;\n\n/**\n * Initialise with a language model over the output sequences and,\n * given a List of List>'s initially tagged with \"observation\"\n * probabilities (output probabilities only based on local features),\n * return the forward-pass re-estimated probabilites of the output\n * classes.\n * \n * @author Dennis N. Mehay\n */\npublic class SequenceScorer extends StandardNgramModel {\n\n /**\n * A Trellis to hold sequence labels (wrapped in Word classes)\n * functionality. \n */\n private Trellis seqLabs;\n /** Trellis for initial observation model scores. */\n private Trellis initScores;\n /** Trellis for forward-backward re-estimated scores. */\n private Trellis fbScores;\n /** Trellis of back-pointers (for retrieving n-best sequences). */\n private Trellis backPointers;\n /** How many of the previous (following) best predictions make it into the forward (or backward) search? */\n private int searchBeam = 200;\n /** Re-usable private data structures. */\n private List> tmpInitScores = new ArrayList>(500);\n private List> tmpFwdScores = new ArrayList>(500);\n private List> tmpSeqLabs = new ArrayList>(500);\n private List> tmpBkpointers = new ArrayList>(500);\n /** For interning Word's */\n private Interner words = new Interner();\n private Constants.TaggingAlgorithm alg = Constants.TaggingAlgorithm.FORWARDBACKWARD;\n\n /** Create a ForwardScorer with a sequence model (over supertags, POSs tags, words, etc.) */\n public SequenceScorer(int order, String lmFile) throws IOException {\n super(order, lmFile);\n }\n\n /** \n * A utility method for finding the order of n-gram models (by reading in the ARPA-formatted file.\n * (A bit messy, I know.)\n */\n public static int findOrder(String tagSequenceModel) {\n // find n-gram order of sequence model.\n BufferedReader reader = null;\n String ln = null;\n int ord = 0;\n try {\n reader = new BufferedReader(new FileReader(new File(tagSequenceModel)));\n ln = reader.readLine();\n reader = new BufferedReader(new FileReader(new File(tagSequenceModel)));\n while (ln != null && !ln.startsWith(\"\\\\data\\\\\")) {\n ln = reader.readLine();\n }\n ln = reader.readLine();\n while (ln != null & ln.startsWith(\"ngram \")) {\n ord = Integer.parseInt(ln.split(\" \")[1].split(\"=\")[0]);\n ln = reader.readLine();\n }\n reader.close();\n } catch (FileNotFoundException fnfe) {\n Logger.getLogger(SequenceScorer.class.getName()).log(Level.SEVERE, null, fnfe);\n } catch (IOException ioe) {\n Logger.getLogger(SequenceScorer.class.getName()).log(Level.SEVERE, null, ioe);\n }\n return ord;\n }\n\n /** Set the tagging algorithm (with one of {forward-backward, forward}). */\n public void setAlgorithm(Constants.TaggingAlgorithm newAlg) {\n alg = newAlg;\n }\n\n /**\n * Set the maximum width of the number of previous hypothesized tags to consider\n * in the forward probabilities.\n */\n public void setSearchBeam(int newBeam) {\n searchBeam = newBeam;\n }\n\n /** Rescore an observation sequence of (initially) supertagged Word's using the sequence model. */\n public List>> rescoreSequence(List>> observationSequence) {\n // build up initial trellises.\n tmpInitScores.clear();\n tmpFwdScores.clear();\n tmpSeqLabs.clear();\n tmpBkpointers.clear();\n\n for (List> tw : observationSequence) {\n ArrayList scrs = new ArrayList(tw.size());\n ArrayList fscs = new ArrayList(tw.size());\n ArrayList sLabs = new ArrayList(tw.size());\n ArrayList bpts = new ArrayList(tw.size());\n for (Pair tagging : tw) {\n // add observation score, and convert to log-prob domain, if needed.\n scrs.add((tagging.a > 0) ? Math.log(tagging.a) : tagging.a);\n fscs.add(null);\n sLabs.add(words.intern(Word.createWord(tagging.b, null, null, null, null, null, null)));\n bpts.add(null);\n }\n tmpInitScores.add(scrs);\n tmpSeqLabs.add(sLabs);\n tmpFwdScores.add(fscs);\n tmpBkpointers.add(bpts);\n }\n initScores = new Trellis(tmpInitScores);\n // these are initially null.\n fbScores = new Trellis(tmpFwdScores);\n // these are too.\n backPointers = new Trellis(tmpBkpointers);\n seqLabs = new Trellis(tmpSeqLabs);\n\n // forward loop.\n // for each word...\n for (int u = 0; u < observationSequence.size(); u++) {\n List> tw = observationSequence.get(u);\n double normTot = 0.0;\n // for each of its tags within the search beam.\n for (int v = 0; v < tw.size(); v++) {\n Word currTag = seqLabs.getCoord(u, v);\n List bestHist = null;\n Double seqScore = null;\n\n Double obsScore = initScores.getCoord(u, v);\n if (u == 0) {\n // beginning of sequence. \n bestHist = getBestHist(u, v, order);\n bestHist.add(currTag);\n seqScore = lmScore(bestHist);\n\n double fs = seqScore + obsScore; \n\n normTot += Math.exp(fs);\n fbScores.setCoord(u, v, fs);\n } else {\n // use dynamic programming-computed scores to progress.\n List> prevTaggedWord = observationSequence.get(u - 1);\n ProbIndexPair[] bestPrevScores = new ProbIndexPair[Math.min(prevTaggedWord.size(), searchBeam)];\n\n for (int z = 0; z < Math.min(prevTaggedWord.size(), searchBeam); z++) {\n bestHist = getBestHist(u - 1, z, order - 1);\n bestHist.add(currTag);\n seqScore = lmScore(bestHist);\n double fs = fbScores.getCoord(u - 1, z) + seqScore;\n fs += obsScore; \n bestPrevScores[z] = new ProbIndexPair(\n Double.valueOf(fs),\n Integer.valueOf(z));\n\n }\n\n // sort descending based on score.\n Arrays.sort(bestPrevScores);\n\n // add up the prob's of all sequences leading to this node.\n double fsum = 0.0;\n for (int q = 0; q < bestPrevScores.length; q++) {\n fsum += Math.exp(bestPrevScores[q].a);\n }\n normTot += fsum;\n //fbScores.setCoord(u, v, bestPrevScores[0].a.doubleValue());\n fbScores.setCoord(u, v, Math.log(fsum));\n\n // add n-best backpointers.\n List bks = new ArrayList(bestPrevScores.length);\n for (int q = 0; q < bestPrevScores.length; q++) {\n bks.add(bestPrevScores[q].b);\n }\n backPointers.setCoord(u, v, new Backpointer(bks));\n }\n }\n\n // normalise. \n for (int v = 0; v < tw.size(); v++) {\n fbScores.setCoord(u, v, Math.log(Math.exp(fbScores.getCoord(u, v)) / normTot));\n }\n }\n\n // backward loop.\n int size = observationSequence.size();\n if (alg == Constants.TaggingAlgorithm.FORWARDBACKWARD) {\n // for each word...\n for (int u = size - 1; u >= 0; u--) {\n List> tw = observationSequence.get(u);\n double normTot = 0.0;\n // for each of its tags...\n for (int v = 0; v < tw.size(); v++) {\n List bestHist = null;\n \n Double obsScore = initScores.getCoord(u, v);\n \n if (u == (size - 1)) { // right-hand end of sequence.\n\n bestHist = getBestHist(u, v, order - 1);\n bestHist.add(words.intern(Word.createWord(\"\", null, null, null, null, null, null)));\n double bsc = fbScores.getCoord(u, v) + obsScore;\n normTot += Math.exp(bsc);\n fbScores.setCoord(u, v, bsc);\n } else {\n // use dynamic programming-computed scores to progress backwards.\n bestHist = getBestHist(u, v, order - 1);\n List> followingTaggedWd = observationSequence.get(u + 1);\n double backwardSum = 0.0;\n for (int z = 0; z < followingTaggedWd.size(); z++) {\n Word followingTag = words.intern(Word.createWord(followingTaggedWd.get(z).b.intern(), null, null, null, null, null, null));\n if (z > 0) {\n bestHist.remove(bestHist.size() - 1);\n }\n bestHist.add(followingTag);\n backwardSum += Math.exp(lmScore(bestHist) + fbScores.getCoord(u + 1, z));\n }\n double newSc = Math.log(backwardSum) + obsScore;\n normTot += Math.exp(newSc);\n fbScores.setCoord(u, v, newSc);\n }\n }\n // normalise.\n for (int v = 0; v < tw.size(); v++) {\n fbScores.setCoord(u, v, Math.log(Math.exp(fbScores.getCoord(u, v)) / normTot));\n }\n }\n }\n\n // re-sort based on re-estimated scores. \n for (int i = 0; i < observationSequence.size(); i++) {\n ProbIndexPair[] fwdScrs = new ProbIndexPair[observationSequence.get(i).size()];\n List> tagging = observationSequence.get(i);\n\n for (int j = 0; j < tagging.size(); j++) {\n double probP = Math.exp(fbScores.getCoord(i, j).doubleValue());\n fwdScrs[j] = new ProbIndexPair(probP, new Integer(j));\n }\n Arrays.sort(fwdScrs);\n\n List> newTagging = new ArrayList>(fwdScrs.length);\n for (int z = 0; z < fwdScrs.length; z++) {\n Double renorm = new Double(fwdScrs[z].a.doubleValue());\n if (renorm.equals(Constants.one)) {\n renorm = Constants.one;\n }\n newTagging.add(new Pair(renorm, tagging.get(fwdScrs[z].b.intValue()).b));\n }\n observationSequence.set(i, newTagging);\n }\n return observationSequence;\n }\n\n /** \n * Use the LM to score a sequence of words.\n */\n private double lmScore(List seq) {\n setWordsToScore(seq, false);\n prepareToScoreWords();\n return logprob();\n }\n\n /** Follow the back-pointers to get the best sequence of up to length 'order' leading up to cell (i,j). */\n private List getBestHist(int i, int j, int order) {\n int size = Math.max(order, 0);\n List retVal = null;\n Backpointer bp = backPointers.getCoord(i, j);\n if (i == -1) {\n // base case (off of the end of the sequence). \n retVal = new ArrayList(size);\n retVal.add(words.intern(Word.createWord(\"\", null, null, null, null, null, null)));\n return retVal;\n } else if (i == 0) {\n // base case (at beginning of sequence)\n retVal = getBestHist(i - 1, 0, order - 1);\n retVal.add(seqLabs.getCoord(i, j));\n return retVal;\n } else if (order == 0) {\n // base case (reached back as far as the n-gram model will need to see).\n retVal = new ArrayList(size);\n return retVal;\n } else {\n // recursive case. \n retVal = getBestHist(i - 1, bp.get(0).intValue(), order - 1);\n retVal.add(seqLabs.getCoord(i, j));\n return retVal;\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/sequencescoring/Trellis.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n\npackage opennlp.ccg.parse.tagger.sequencescoring;\n\nimport java.util.ArrayList;\nimport java.util.List;\n\n/**\n * A Trellis for sequence coding (of supertags, e.g.).\n * \n * @author Dennis N. Mehay\n */\npublic class Trellis {\n /** The dimensions of the Trellis. */\n private int cols, rows;\n \n /** The actual nuts and bolts of the Trellis. */\n private ArrayList> trellis;\n\n /** Constructor with passed-in list of lists. */\n public Trellis(List> inpt) {\n this.reshape(inpt.size(), inpt.get(0).size(), inpt);\n }\n \n /** Constructor with dimensions. */\n public Trellis(int cols, int rows, A dummy) {\n List> tr = new ArrayList>(cols);\n for(int i = 0; i < cols; i++) {\n ArrayList tmp = new ArrayList(rows);\n for(int j = 0; j < rows; j++) {\n tmp.add(dummy);\n }\n tr.add(tmp);\n }\n reshape(cols, rows, tr);\n }\n \n /** \n * Reshape the dimensions (e.g., to accomodate a new sequence with a \n * particular max beam width \n */\n public void reshape(int cols, int rows, List> inpt) {\n this.cols = cols;\n this.rows = rows;\n \n this.trellis = new ArrayList>(cols); \n for(List la : inpt) {\n ArrayList row = new ArrayList(rows);\n for(A a : la) {\n row.add(a);\n }\n this.trellis.add(row); \n }\n }\n \n /** What is the max beam width? */\n public int getWidth() { return rows; }\n /** What is the length of the sequence? */\n public int getLength() { return cols; }\n /** Clear out values in the trellis. */\n public void clear() { \n for(int i = 0; i < cols; i++) {\n this.trellis.add(new ArrayList(rows));\n }\n }\n /** Get the sequence options at index i. */\n public ArrayList getOptions(int i) { return this.trellis.get(i); }\n \n /** Get a node in the Trellis (referenced by 2D coordinate). */\n public A getCoord(int i, int j) { \n try {return this.trellis.get(i).get(j); \n } catch(IndexOutOfBoundsException iobe) {\n return null;\n }\n }\n \n /** Set the value at a node in the Trellis (referenced by 2D coordinate) */\n public void setCoord(int i, int j, A val) { this.trellis.get(i).set(j, val); }\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/util/CCGBankToSRILMFLM.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n\n/*\n * Read in the CCGbank (auto format), line by line, and transform each lexical\n * item () into an SRILM factored LM bundle format:\n * W-word:S-word:P-pos1:T:cat1, where every thing has been escaped (e.g., colons),\n * \"W\" stands for word form, \"S\" for stem, \"P\" for POS and \"T\" for super_t_ag.\n */\n\npackage opennlp.ccg.parse.tagger.util;\n\nimport java.io.BufferedReader;\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.util.regex.Matcher;\nimport java.util.regex.Pattern;\nimport opennlp.ccg.lexicon.DefaultTokenizer;\n\n/**\n * @author Dennis N. Mehay\n */\npublic class CCGBankToSRILMFLM {\n public static void main(String[] args) throws FileNotFoundException, IOException {\n String usage = \"\\nCCGBankToSRILMFLM -input -o \\n\";\n if (args.length > 0 && args[0].equals(\"-h\") || args.length == 0) {\n System.out.println(usage);\n System.exit(0);\n }\n\n BufferedReader reader = null;\n BufferedWriter writer = null;\n String inputCorp = \"train.auto\", output = \"train.srilm\";\n for (int i = 0; i < args.length; i++) {\n if (args[i].equals(\"-i\")) {inputCorp = args[++i]; continue;}\n if (args[i].equals(\"-o\")) {output = args[++i]; continue;}\n System.out.println(\"Unrecognized option: \" + args[i]);\n }\n reader = new BufferedReader(new FileReader(new File(inputCorp)));\n writer = new BufferedWriter(new FileWriter(new File(output)));\n String parseIDHeader = \"ID=\";\n Pattern p = Pattern.compile(\"()+?\");\n \n String line = reader.readLine();\n while(line != null) {\n if(line.startsWith(parseIDHeader)) {line = reader.readLine(); continue;}\n line = line.trim();\n Matcher m = p.matcher(line);\n String word = null, pos = null, cat = null;\n int cnt = 0;\n while(m.find()) {\n String toks = m.group();\n // {}\n String[] parts = toks.split(\" \");\n word = parts[4];\n pos = parts[2];\n cat = parts[1];\n if(cnt++ > 0) {\n writer.write(\" \");\n }\n writer.write(\"W-\"+DefaultTokenizer.escape(word)+\":\"\n +\"S-\"+DefaultTokenizer.escape(word)+\":\"\n +\"P-\"+DefaultTokenizer.escape(pos)+\":\"\n +\"T-\"+DefaultTokenizer.escape(cat));\n }\n writer.write(System.getProperty(\"line.separator\"));\n line = reader.readLine();\n }\n writer.close();\n reader.close();\n }\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/util/ConfigFileProcessor.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (c) 2010 Dennis N. Mehay\n//\n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed inp the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.parse.tagger.util;\n\nimport java.io.BufferedReader;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.IOException;\nimport java.util.*;\n//import java.util.Map;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\n\n/**\n * @author Dennis N. Mehay\n */\npublic class ConfigFileProcessor {\n \n \n /** Read in config file as a {@code Map}. */\n public static Map readInConfig(String configFile) {\n \treturn readInConfig(configFile, null);\n }\n\n /**\n * Read in config file as a {@code Map}, resolving the given path keys\n * relative to the config file if not absolute.\n */\n public static Map readInConfig(String configFile, String pathKeys[]) {\n \tSet paths = Collections.emptySet();\n \tif (pathKeys != null) paths = new HashSet(Arrays.asList(pathKeys));\n BufferedReader cf = null;\n Map opts = new HashMap();\n try {\n \tFile infile = new File(configFile);\n \tFile parentDir = infile.getParentFile();\n cf = new BufferedReader(new FileReader(infile));\n \n String ln = cf.readLine();\n // map options to values.\n while (ln != null) {\n if (ln.trim().equals(\"\") || ln.trim().startsWith(\"#\")) {\n ln = cf.readLine();\n continue;\n }\n String[] parts = ln.trim().split(\"=\");\n String key = parts[0].trim().toLowerCase();\n String val = parts[1].trim();\n // resolve path keys\n if (paths.contains(key)) {\n \tFile f = new File(parentDir, val);\n \tif (!f.exists()) {\n \t\tf = new File(val);\n \t\tif (!f.exists()) throw new FileNotFoundException(\"Can't resolve filename: \" + val);\n \t}\n \tval = f.getPath();\n }\n opts.put(key, val);\n ln = cf.readLine();\n }\n } catch (FileNotFoundException ex) {\n Logger.getLogger(ConfigFileProcessor.class.getName()).log(Level.SEVERE, null, ex);\n } catch (IOException ex) {\n Logger.getLogger(ConfigFileProcessor.class.getName()).log(Level.SEVERE, null, ex); \n } finally {\n try {\n cf.close();\n } catch (IOException ex) {\n Logger.getLogger(ConfigFileProcessor.class.getName()).log(Level.SEVERE, null, ex);\n }\n }\n return opts;\n }\n\n}\n" }, { "path": "src/opennlp/ccg/parse/tagger/util/ResultSink.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2009 Dennis N. Mehay\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.parse.tagger.util;\n\nimport java.util.HashMap;\nimport java.util.Iterator;\nimport java.util.List;\nimport java.util.Map;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.util.Pair;\n\n/**\n * Inspired (loosely, based on my recollection) by Jason Baldridge's \n * similar class for tracking classifier performance.\n * Here we simply track the Word-by-Word\n * tagging performance of a CCG supertagger by passing in a multitagging\n * and a gold-standard answer and tabulating the results. The results\n * are reported by a custom report method, which returns\n * a String representation of the results.\n * \n * @author Dennis N. Mehay\n * @version $Revision: 1.1 $, $Date: 2010/09/21 04:12:42 $\n */\npublic class ResultSink {\n\n public static enum ResultSinkType { SUPERTAG, POSTAG };\n \n private int totalTags = 0, totalWords = 0, totalRight = 0;\n // for keeping pos-specific stats.\n private Map posToRight = new HashMap(), posTot = new HashMap();\n // for tracking the total number of sentences, number totally tagged\n // correctly, etc.\n private int sentNum = 0, sentsCorrect = 0;\n private boolean allCorrect = true;\n // what type of tag are we tracking the results over?\n private ResultSinkType whatType;\n // for general pos-specific stats (e.g., N... -> , not NNP -> and NNPS -> , etc.)\n private Map genPOSToRight = new HashMap(), genPOSTot = new HashMap();\n\n /** \n * Nullary constructor. Defaults to supertag result sink.\n * (TODO: add log file logging for more detailed error reporting.)\n */\n public ResultSink() {\n this(ResultSinkType.SUPERTAG);\n }\n\n public ResultSink(ResultSinkType whatType) {\n this.whatType = whatType;\n }\n\n /** \n * Add and store a sentence of tagged words (List>>)\n * wrt a gold-standard tagged word. \n */\n public void addSent(List>> sent, List goldTagging) {\n sentNum++;\n allCorrect = true;\n Iterator gold = goldTagging.iterator();\n for (List> tgging : sent) {\n addResult(tgging, gold.next());\n }\n if (allCorrect) {\n sentsCorrect++;\n }\n }\n\n /**\n * Add a single-word tagging result alongside its gold-standard tagging.\n * Compare and log whether the gold-standard tag is in the beta-best (also\n * log pos-specific error stats).\n */\n public void addResult(List> tagging, Word goldTagging) {\n String goldTag = (whatType == ResultSinkType.SUPERTAG) ? goldTagging.getSupertag() : goldTagging.getPOS();\n totalTags += tagging.size();\n totalWords++;\n // mww: check for missing gold POS (grrr)\n if (goldTagging.getPOS() == null) {\n System.err.println(\"Warning: found null gold POS, skipping word: \" + goldTagging);\n this.allCorrect = false;\n return;\n }\n\n String thisPOS = goldTagging.getPOS(), thisGenPOS = goldTagging.getPOS().substring(0, 1);\n Integer posT = this.posTot.get(thisPOS), gPOST = this.genPOSTot.get(thisGenPOS);\n if (posT == null) {\n this.posTot.put(thisPOS, new Integer(1));\n } else {\n this.posTot.put(thisPOS, new Integer(posT.intValue() + 1));\n }\n if (gPOST == null) {\n this.genPOSTot.put(thisGenPOS, new Integer(1));\n } else {\n this.genPOSTot.put(thisGenPOS, new Integer(gPOST.intValue() + 1));\n }\n // assume this tagging is incorrect, until proven otherwise.\n boolean gotIt = false;\n for (Pair tag : tagging) {\n if (tag.b.equals(goldTag)) {\n gotIt = true;\n totalRight++;\n // add one both to the pos right and total for that pos type.\n Integer posLkup = this.posToRight.get(thisPOS),\n genPOSLkup = this.genPOSToRight.get(thisGenPOS);\n if (posLkup == null) {\n this.posToRight.put(thisPOS, new Integer(1));\n } else {\n this.posToRight.put(thisPOS, new Integer(posLkup.intValue() + 1));\n }\n if (genPOSLkup == null) {\n this.genPOSToRight.put(thisGenPOS, new Integer(1));\n } else {\n this.genPOSToRight.put(thisGenPOS, new Integer(genPOSLkup.intValue() + 1));\n }\n break;\n }\n }\n // mistagged this one word, so tagging the whole sentence correctly -- \n // allCorrect==true -- is not possible. \n if (!gotIt) {\n this.allCorrect = false;\n }\n\n }\n\n public String report() {\n // make sure 0 counts are inserted for POS types that were never got right.\n for (String post : this.posTot.keySet()) {\n if (this.posToRight.get(post) == null) {\n this.posToRight.put(post, new Integer(0));\n }\n }\n for (String post : this.genPOSTot.keySet()) {\n if (this.genPOSToRight.get(post) == null) {\n this.genPOSToRight.put(post, new Integer(0));\n }\n }\n\n String rep = \"\";\n rep += \"\\n\\nAccuracy by POS type:\\n\\n\";\n for (String post : this.posTot.keySet()) {\n rep += post + \": \" + ((this.posToRight.get(post).intValue() + 0.0) / (this.posTot.get(post))) +\n \" <==> \" + this.posToRight.get(post).intValue() + \"/\" + (this.posTot.get(post)) + \" = \" +\n (100 * ((this.posTot.get(post) - this.posToRight.get(post) + 0.0) / (totalWords - totalRight))) + \" (% of total errors) \\n\";\n }\n rep += \"\\nAccuracy by general (truncated) POS type:\\n\\n\";\n for (String post : this.genPOSTot.keySet()) {\n rep += post + \": \" + (this.genPOSToRight.get(post).intValue() + 0.0) / (this.genPOSTot.get(post)) +\n \" <==> \" + this.genPOSToRight.get(post).intValue() + \"/\" + (this.genPOSTot.get(post)) + \" = \" +\n (100 * ((this.genPOSTot.get(post) - this.genPOSToRight.get(post) + 0.0) / (totalWords - totalRight))) + \" (% of total errors) \\n\";\n }\n rep += \"\\nTotal words: \" + totalWords +\n \"\\nTotal sents: \" + this.sentNum +\n \"\\nAggregate total tags: \" + totalTags +\n \"\\nAve. tags/word: \" + ((totalTags + 0.0) / (totalWords + 0.0)) +\n \"\\nWord accuracy: \" + ((totalRight + 0.0) / totalWords) + \"\\n\" +\n \"\\nSent accuracy: \" + ((this.sentsCorrect + 0.0) / (this.sentNum)) + \"\\n\\n\";\n return rep;\n }\n}" }, { "path": "src/opennlp/ccg/perceptron/Alphabet.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2008 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.perceptron;\r\n\r\nimport java.util.*;\r\nimport java.io.*;\r\n\r\nimport opennlp.ccg.util.*;\r\nimport opennlp.ccg.lexicon.DefaultTokenizer;\r\n\r\n/**\r\n * A bidirectional mapping between feature names and indices.\r\n * \r\n * An alphabet can be read from either an alphabet file or a model file.\r\n * An alphabet file starts with the number of features on one line, \r\n * followed by one line per feature pairing the feature name with its \r\n * frequency (which is ignored).\r\n * \r\n * An alphabet can be open or closed. An closed alphabet does not \r\n * accept new features, and thus can be used to filter out features \r\n * not already in the alphabet.\r\n * \r\n * The main routine filters the input event file to an output alphabet file,\r\n * optionally with a table size and pruning threshold.\r\n * \r\n * @author Michael White\r\n * @version $Revision: 1.7 $, $Date: 2009/11/02 03:44:15 $\r\n */\r\npublic class Alphabet {\r\n\t\r\n\t/** Feature as a node in a trie, with each node storing the feature index. */\r\n\tpublic static class Feature extends TrieMap {\r\n\t\t\r\n\t\t/** Constructor with index. */\r\n\t\tpublic Feature(Integer index) { super(index); }\r\n\t\t\r\n\t\t/** Factory method, for adding empty child nodes. */\r\n\t\tprotected Feature createNode() { return new Feature(null); }\r\n\t\t\r\n\t\t/** Returns a string name by concatenating escaped keys using colons. */\r\n\t\tpublic String name() {\r\n\t\t\tStringBuffer namebuf = new StringBuffer();\r\n\t\t\tList keys = traceKeys();\r\n\t\t\tfor (int i=0; i < keys.size(); i++) {\r\n\t\t\t\tnamebuf.append(DefaultTokenizer.escape(keys.get(i)));\r\n\t\t\t\tif (i+1 < keys.size()) namebuf.append(':');\r\n\t\t\t}\r\n\t\t\treturn namebuf.toString();\r\n\t\t}\r\n\t\t\r\n\t\t/** Returns the index. */\r\n\t\tpublic Integer getIndex() { return data; }\r\n\t}\r\n\t\r\n\t\r\n\t// mappings\r\n\tprivate Feature dict;\r\n\tprivate List dictInv;\r\n\t\r\n\t// size\r\n\tprivate int size = 0;\r\n\t\r\n\t// closed flag\r\n\tprivate boolean closed = false;\r\n\t\r\n\t\r\n\t/** Constructor with initial size. The alphabet is left open. */\r\n\tpublic Alphabet(int size) {\r\n\t\tinit(size);\r\n\t}\r\n\r\n\t/** Constructor to load an alphabet from a file. The alphabet is set to closed. */\r\n\tpublic Alphabet(String filename) throws IOException {\r\n\t\tReader reader = EventFile.openReader(new File(filename));\r\n\t\tStreamTokenizer tokenizer = EventFile.initTokenizer(reader);\r\n\t\ttokenizer.nextToken();\r\n\t\tint size = Integer.parseInt(tokenizer.sval);\r\n\t\tinit(size);\r\n\t\tfor (int i=0; i < size; i++) {\r\n\t\t\ttokenizer.nextToken();\r\n\t\t\tadd(tokenizer.sval);\r\n\t\t\ttokenizer.nextToken(); // skip freq or weight\r\n\t\t}\r\n\t\treader.close();\r\n\t\tclosed = true;\r\n\t}\r\n\t\r\n\t// initializes dict, dictInv\r\n\tprivate void init(int size) {\r\n\t\tdict = new Feature(null);\r\n\t\tdictInv = new ArrayList(size);\r\n\t}\r\n\t\r\n\t\r\n\t/** Size. */\r\n\tpublic int size() { return size; }\r\n\t\r\n\t/** Closed. */\r\n\tpublic boolean closed() { return closed; }\r\n\t\r\n\t/** Set closed. */\r\n\tpublic void setClosed(boolean closed) { this.closed = closed; }\r\n\t\r\n\t\r\n\t/** Add feature with given name, if not already present, returning added feature. */\r\n\tpublic Feature add(String feat) {\r\n\t\treturn add(parseKeys(feat));\r\n\t}\r\n\t\r\n\t/** Add equivalent feature, if not already present, returning added feature. */\r\n\tpublic Feature add(Feature f) {\r\n\t\treturn add(f.traceKeys());\r\n\t}\r\n\t\r\n\t/** Add feature with given keys, if not already present, returning added feature. */\r\n\tpublic Feature add(List keys) {\r\n\t\tif (closed) throw new RuntimeException(\"Can't add to a closed alphabet!\");\r\n\t\tFeature node = (Feature) dict.findChildFromList(keys);\r\n\t\treturn addNode(node);\r\n\t}\r\n\t\r\n\t/** Add feature with given keys, if not already present, returning added feature. */\r\n\tpublic Feature addLazy(List> keyExtractors) {\r\n\t\tif (closed) throw new RuntimeException(\"Can't add to a closed alphabet!\");\r\n\t\tFeature node = (Feature) dict.findChildFromLazyList(keyExtractors);\r\n\t\treturn addNode(node);\r\n\t}\r\n\t\r\n\t// adds a feature node\r\n\tprivate Feature addNode(Feature node) {\r\n\t\tif (node.data != null) return node;\r\n\t\tnode.data = size++;\r\n\t\tdictInv.add(node);\r\n\t\treturn node;\r\n\t}\r\n\t\r\n\t/** \r\n\t * Parses a feature name into a list of unescaped interned string keys, \r\n\t * breaking on colons. \r\n\t */\r\n\tpublic static List parseKeys(String feat) {\r\n\t\tList retval = new ArrayList();\r\n\t\tint current = 0; \r\n\t\twhile (current < feat.length()) {\r\n\t\t\tint breakpos = feat.indexOf(\":\", current);\r\n\t\t\tString key;\r\n\t\t\tif (breakpos >= 0) {\r\n\t\t\t\tkey = feat.substring(current, breakpos);\r\n\t\t\t\tcurrent = breakpos + 1;\r\n\t\t\t}\r\n\t\t\telse {\r\n\t\t\t\tkey = feat.substring(current);\r\n\t\t\t\tcurrent = feat.length();\r\n\t\t\t}\r\n\t\t\tretval.add(DefaultTokenizer.unescape(key).intern());\r\n\t\t}\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\t\r\n\t/** Get or add index of feature with given name (null if none when closed). */\r\n\tpublic Feature index(String feat) { \r\n\t\treturn index(parseKeys(feat));\r\n\t}\r\n\t\r\n\t/** Get or add index of equivalent feature (null if none when closed). */\r\n\tpublic Feature index(Feature f) {\r\n\t\treturn index(f.traceKeys());\r\n\t}\r\n\t\r\n\t/** Get or add index of feature with given keys (null if none when closed). */\r\n\tpublic Feature index(List keys) {\r\n\t\tif (!closed) return add(keys);\r\n\t\tFeature node = (Feature) dict.getChildFromList(keys);\r\n\t\treturn node;\r\n\t}\r\n\t\r\n\t/** Get or add index of feature with given key extractors (null if none when closed). */\r\n\tpublic Feature indexLazy(List> keyExtractors) {\r\n\t\tif (!closed) return addLazy(keyExtractors);\r\n\t\tFeature node = (Feature) dict.getChildFromLazyList(keyExtractors);\r\n\t\treturn node;\r\n\t}\r\n\t\r\n\t/** Get indexed feature. */\r\n\tpublic Feature feature(int index) { return dictInv.get(index); }\r\n\t\r\n\t\r\n\t/** \r\n\t * Creates an alphabet file from an event file, \r\n\t * saving it to a file with features sorted by frequency, \r\n\t * using the given table size and pruning threshold. \r\n\t **/\r\n\tpublic static void createAlphabet(String eventfile, String alphabetfile, int tablesize, int pruningthreshold) throws IOException {\r\n\t\t// open files\r\n\t\tEventFile eventFile = new EventFile(eventfile);\r\n\t\tPrintWriter out = EventFile.openWriter(new File(alphabetfile));\r\n\t\t// init freq tally\r\n\t\tMap freqTally = new HashMap(tablesize*2);\r\n\t\t// read event file, incrementing tallies\r\n\t\tEventFile.Block block;\r\n\t\tFeatureMap goldMap = new FeatureMap();\r\n\t\tSet seenFeats = new HashSet();\r\n\t\twhile ( (block = eventFile.nextBlock()) != null ) {\r\n\t\t\t// set gold map (nb: assumes single gold event)\r\n\t\t\tgoldMap.clear();\r\n\t\t\tseenFeats.clear();\r\n\t\t\tfor (EventFile.Event event : block.events) {\r\n\t\t\t\tif (event.count > 0) goldMap.add(event.features);\r\n\t\t\t}\r\n\t\t\t// tally distinct feats not in gold map\r\n\t\t\tfor (EventFile.Event event : block.events) {\r\n\t\t\t\tif (event.count > 0) continue;\r\n\t\t\t\tfor (FeatureVector.Iterator it = event.features.iterator(); it.hasNext(); ) {\r\n\t\t\t\t\tFeature feat = it.nextFeature();\r\n\t\t\t\t\tfloat val = it.nextValue();\r\n\t\t\t\t\tseenFeats.add(feat);\r\n\t\t\t\t\tfloat goldVal = goldMap.get(feat);\r\n\t\t\t\t\tif (val != goldVal) {\r\n\t\t\t\t\t\tInteger tally = freqTally.get(feat);\r\n\t\t\t\t\t\tif (tally != null) freqTally.put(feat, ++tally);\r\n\t\t\t\t\t\telse freqTally.put(feat, 1);\r\n\t\t\t\t\t}\r\n\t\t\t\t}\r\n\t\t\t}\r\n\t\t\t// tally unseen feats from gold event\r\n\t\t\tfor (EventFile.Event event : block.events) {\r\n\t\t\t\tif (event.count == 0) continue;\r\n\t\t\t\tfor (FeatureVector.Iterator it = event.features.iterator(); it.hasNext(); ) {\r\n\t\t\t\t\tFeature feat = it.nextFeature();\r\n\t\t\t\t\tif (!seenFeats.contains(feat)) {\r\n\t\t\t\t\t\tInteger tally = freqTally.get(feat);\r\n\t\t\t\t\t\tif (tally != null) freqTally.put(feat, ++tally);\r\n\t\t\t\t\t\telse freqTally.put(feat, 1);\r\n\t\t\t\t\t}\r\n\t\t\t\t}\r\n\t\t\t}\r\n\t\t}\r\n\t\t// get tallies passing frequency threshold\r\n\t\tList> tallies = new ArrayList>(freqTally.size());\r\n\t\tif (pruningthreshold > 0) {\r\n\t\t\tfor (Map.Entry entry : freqTally.entrySet()) {\r\n\t\t\t\tif (entry.getValue() >= pruningthreshold) tallies.add(entry);\r\n\t\t\t}\r\n\t\t}\r\n\t\telse tallies.addAll(freqTally.entrySet());\r\n\t\t// sort tallies by descending frequency \r\n\t\t// (further sorting alphabetically seems to take too long)\r\n\t\tCollections.sort(\r\n\t\t\ttallies, \r\n\t\t\tnew Comparator>() {\r\n\t\t\t\tpublic int compare(Map.Entry entry1, Map.Entry entry2) {\r\n\t\t\t\t\tint val1 = entry1.getValue(); int val2 = entry2.getValue(); \r\n\t\t\t\t\tif (val1 > val2) return -1;\r\n\t\t\t\t\tif (val1 < val2) return 1;\r\n\t\t\t\t\treturn 0;\r\n\t\t\t\t\t//return entry1.getKey().name().compareTo(entry2.getKey().name());\r\n\t\t\t\t}\r\n\t\t\t}\r\n\t\t);\r\n\t\t// write tallied features to file\r\n\t\tint size = tallies.size();\r\n\t\tout.println(Integer.toString(size));\r\n\t\tfor (int i=0; i < size; i++) {\r\n\t\t\tMap.Entry entry = tallies.get(i);\r\n\t\t\tout.println(entry.getKey().name() + \" \" + entry.getValue());\r\n\t\t}\r\n\t\t// close files\r\n\t\teventFile.close();\r\n\t\tout.close();\r\n\t}\r\n\r\n\t/** Main routine for filtering event file to an alphabet file. */\r\n\tpublic static void main(String[] args) throws IOException {\r\n\t\tif (args.length < 2) {\r\n\t\t\tSystem.out.println(\"Usage: java perceptron.Alphabet (-s ) (-p \");\r\n\t\t\tSystem.exit(0);\r\n\t\t}\r\n\t\tString eventfile = args[0];\r\n\t\tString alphabetfile = args[1];\r\n\t\tint tablesize = 1000000;\r\n\t\tint pruningthreshold = 0;\r\n\t\tfor (int i=2; i < args.length; i++) {\r\n\t\t\tif (args[i].equals(\"-s\")) tablesize = Integer.valueOf(args[++i]);\r\n\t\t\tif (args[i].equals(\"-p\")) pruningthreshold = Integer.valueOf(args[++i]);\r\n\t\t}\r\n\t\tSystem.out.println(\"Writing alphabet to \" + alphabetfile + \" from event file \" + eventfile);\r\n\t\tif (pruningthreshold > 0) System.out.println(\"with pruning threshold \" + pruningthreshold);\r\n\t\tcreateAlphabet(eventfile, alphabetfile, tablesize, pruningthreshold);\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccg/perceptron/ComposedFeatureExtractor.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2009 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.perceptron;\r\n\r\nimport java.util.ArrayList;\r\n\r\nimport opennlp.ccg.synsem.*;\r\n\r\n/** \r\n * Class for composing feature extractors. \r\n * Features from the component feature extractors are assumed to be independent.\r\n * \r\n * @author Michael White\r\n * @version $Revision: 1.5 $, $Date: 2011/01/15 17:52:59 $\r\n */ \r\npublic class ComposedFeatureExtractor implements FeatureExtractor {\r\n\r\n\t/** The feature extractors. */\r\n\tpublic final FeatureExtractor[] featureExtractors;\r\n\t\r\n\t/** Constructor. */\r\n\tpublic ComposedFeatureExtractor(FeatureExtractor[] featureExtractors) {\r\n\t\tthis.featureExtractors = featureExtractors;\r\n\t}\r\n\t\r\n\t/** Binary constructor. */\r\n\tpublic ComposedFeatureExtractor(FeatureExtractor featureExtractor1, FeatureExtractor featureExtractor2) {\r\n\t\tthis.featureExtractors = new FeatureExtractor[]{ featureExtractor1, featureExtractor2 };\r\n\t}\r\n\t\r\n\t/** Constructor for sign scorers, some of which may be feature extractors. */\r\n\tpublic ComposedFeatureExtractor(SignScorer[] models) { \r\n \tArrayList feList = new ArrayList(models.length);\r\n for (int i = 0; i < models.length; i++) {\r\n \tif (models[i] instanceof FeatureExtractor) feList.add((FeatureExtractor)models[i]);\r\n }\r\n this.featureExtractors = feList.toArray(new FeatureExtractor[feList.size()]);\r\n\t}\r\n\t\r\n\t/** Returns the features for the given sign and completeness flag. */\r\n\tpublic FeatureVector extractFeatures(Sign sign, boolean complete) {\r\n\t\tFeatureVector[] featureVectors = new FeatureVector[featureExtractors.length];\r\n\t\tfor (int i=0; i < featureExtractors.length; i++)\r\n\t\t\tfeatureVectors[i] = featureExtractors[i].extractFeatures(sign, complete);\r\n\t\treturn new ComposedFeatureVector(featureVectors);\r\n\t}\r\n\t\r\n\t/** Sets the alphabet. */\r\n\tpublic void setAlphabet(Alphabet alphabet) {\r\n\t\tfor (FeatureExtractor fe : featureExtractors) {\r\n\t\t\tfe.setAlphabet(alphabet);\r\n\t\t}\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccg/perceptron/ComposedFeatureVector.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2009 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.perceptron;\r\n\r\n/** \r\n * Class for composing feature vectors. \r\n * Features from the component feature vectors are assumed to be independent.\r\n * \r\n * @author Michael White\r\n * @version $Revision: 1.2 $, $Date: 2011/01/15 17:52:59 $\r\n */ \r\npublic class ComposedFeatureVector implements FeatureVector {\r\n\r\n\t/** The feature vectors. */\r\n\tpublic final FeatureVector[] featureVectors;\r\n\t\r\n\t/** Constructor. */\r\n\tpublic ComposedFeatureVector(FeatureVector[] featureVectors) {\r\n\t\tthis.featureVectors = featureVectors;\r\n\t}\r\n\t\r\n\t/** Binary constructor. */\r\n\tpublic ComposedFeatureVector(FeatureVector featureVector1, FeatureVector featureVector2) {\r\n\t\tthis.featureVectors = new FeatureVector[]{ featureVector1, featureVector2 };\r\n\t}\r\n\t\r\n\t/** Size. */\r\n\tpublic int size() {\r\n\t\tint retval = 0;\r\n\t\tfor (FeatureVector fv : featureVectors) retval += fv.size();\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\t/** Returns an iterator over the entries. */\r\n\tpublic Iterator iterator() {\r\n\t\tif (featureVectors.length == 0) return EMPTY_ITERATOR;\r\n\t\treturn new Iterator() {\r\n\t\t\tint i = 0;\r\n\t\t\tIterator it = featureVectors[0].iterator();\r\n\t\t\tpublic boolean hasNext() { \r\n\t\t\t\tif (it.hasNext()) return true;\r\n\t\t\t\tif (i == featureVectors.length-1) return false;\r\n\t\t\t\tit = featureVectors[++i].iterator();\r\n\t\t\t\treturn hasNext(); \r\n\t\t\t}\r\n\t\t\tpublic Alphabet.Feature nextFeature() { return it.nextFeature(); }\r\n\t\t\tpublic Float nextValue() { return it.nextValue(); }\r\n\t\t};\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccg/perceptron/EventFile.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2008-2013 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.perceptron;\r\n\r\nimport java.util.*;\r\nimport java.util.zip.*;\r\nimport java.io.*;\r\n\r\nimport opennlp.ccg.synsem.Sign;\r\n\r\n/**\r\n * An abstract representation of an event file, whose syntax is a more readable version \r\n * of what's used in the TADM toolkit. A constructor flag controls whether to keep the \r\n * events in memory (defaults to false).\r\n * \r\n * An event file may be given an alphabet, which allows features to be filtered to \r\n * just those present in the alphabet, when it's closed; otherwise, the \r\n * alphabet is constructed dynamically.\r\n * \r\n * The concrete syntax of an event file is as follows.\r\n * An event file consists of a sequence of blocks.\r\n * A block starts with the number of events on a line by itself.\r\n * It is followed by each event, one per line.\r\n * Each event line has a frequency, followed by the number of feature-value pairs,\r\n * then the sequence of feature-value pairs, \r\n * where the feature name is a string with no white space.\r\n * Each feature can appear only once in an event, and must have a value greater than zero. \r\n * You can have events with a zero frequency -- these are used for dispreferred analyses \r\n * in ranking tasks such as parse selection or realization ranking. \r\n * \r\n * An example file appears below. There are two blocks, corresponding to the \r\n * parses of two different senses. The first block has two possible parses, \r\n * the first of which is correct, while the second block has three possible \r\n * parses, where the second one is the correct one.\r\n * \r\n * 

    \r\n * 2\r\n * 1 2 feat1 1.22 feat2 3\r\n * 0 3 feat1 1.55 feat3 1 feat4 2.7\r\n * 3\r\n * 0 2 feat1 1.44 feat4 2.2\r\n * 1 1 feat1 1.33\r\n * 0 2 feat1 1.32 feat4 3.21\r\n *
    \r\n * \r\n * @author Michael White\r\n * @version $Revision: 1.5 $, $Date: 2009/11/01 22:26:29 $\r\n */\r\npublic class EventFile {\r\n\r\n\t/** A block is a list of events. */\r\n\tpublic static class Block {\r\n\t\t/** The list. */\r\n\t\tpublic List events;\r\n\t\t/** Constructor. */\r\n\t\tpublic Block(List events) { this.events = events; }\r\n\t\t/** The event with the highest count (first tied if ties). */\r\n\t\tpublic Event best() {\r\n\t\t\tEvent retval = null; int max = -1;\r\n\t\t\tfor (Event event : events) {\r\n\t\t\t\tif (event.count > max) { retval = event; max = event.count; }\r\n\t\t\t}\r\n\t\t\treturn retval;\r\n\t\t}\r\n\t}\r\n\t\r\n\t/** An event is a feature vector with a count. */\r\n\tpublic static class Event {\r\n\t\t/** The count. */\r\n\t\tpublic int count; \r\n\t\t/** The feature vector. */\r\n\t\tpublic FeatureVector features;\r\n\t\t/** Constructor. */\r\n\t\tpublic Event(FeatureVector features, int count) {\r\n\t\t\tthis.features = features;\r\n\t\t\tthis.count = count;\r\n\t\t}\r\n\t\t/** toString. */\r\n\t\tpublic String toString() { return \"event: count: \" + count + \" \" + features; }\r\n\t}\r\n\t\r\n\t\r\n\t// the alphabet\r\n\tprivate Alphabet alphabet;\r\n\t\r\n\t// the file, which can be reopened\r\n\tprivate File file;\r\n\t\r\n\t// the current reader\r\n\tprivate Reader reader;\r\n\t\r\n\t// the current tokenizer\r\n\tprivate StreamTokenizer tokenizer;\r\n\t\r\n\t// the saved blocks, if kept in memory\r\n\tprivate List blocks = null;\r\n\t\r\n\t// the iterator over saved blocks\r\n\tprivate Iterator blockIt = null;\r\n\t\r\n\t// whether the end-of-file has been reached\r\n\tprivate boolean eofReached = false;\r\n\t\r\n\t/** Constructor with filename. */\r\n\tpublic EventFile(String filename) throws IOException {\r\n\t\tthis(filename, false);\r\n\t}\r\n\r\n\t/** Constructor with filename and in-memory flag. */\r\n\tpublic EventFile(String filename, boolean inMemory) throws IOException {\r\n\t\tthis(filename, new Alphabet(10000), inMemory);\r\n\t}\r\n\r\n\t/** Constructor with filename and alphabet. */\r\n\tpublic EventFile(String filename, Alphabet alphabet) throws IOException {\r\n\t\tthis(filename, alphabet, false);\r\n\t}\r\n\r\n\t/** Constructor with filename, alphabet and in-memory flag. */\r\n\tpublic EventFile(String filename, Alphabet alphabet, boolean inMemory) throws IOException {\r\n\t\tfile = new File(filename); init();\r\n\t\tthis.alphabet = alphabet;\r\n\t\tif (inMemory) this.blocks = new ArrayList(10000);\r\n\t}\r\n\r\n\t\r\n\t/** Returns the alphabet. */\r\n\tpublic Alphabet getAlphabet() { return alphabet; }\r\n\r\n\t\r\n\t/** Closes the reader. */\r\n\tpublic void close() throws IOException {\r\n\t\treader.close();\r\n\t}\r\n\t\r\n\t/** Resets the event file for reading again. */\r\n\tpublic void reset() throws IOException {\r\n\t\tclose(); init();\r\n\t}\r\n\t\r\n\t// inits the reader and tokenizer, or \r\n\t// if keeping blocks in memory, resets the iterator\r\n\tprivate void init() throws IOException {\r\n\t\t// in-memory case\r\n\t\tif (blocks != null && eofReached) {\r\n\t\t\tblockIt = blocks.iterator(); return;\r\n\t\t}\r\n\t\t// degenerate case: keeping blocks in memory but eof not reached\r\n\t\tif (blocks != null) {\r\n\t\t\t// dump saved blocks\r\n\t\t\tblocks.clear();\r\n\t\t}\r\n\t\t// regular init\r\n\t\treader = openReader(file);\r\n\t\ttokenizer = initTokenizer(reader);\r\n\t}\r\n\t\r\n\t\r\n\t/** Initializes the given tokenizer to recognize most chars as word chars. */\r\n\tpublic static StreamTokenizer initTokenizer(Reader reader) throws IOException {\r\n\t\tStreamTokenizer tokenizer = new StreamTokenizer(reader);\r\n\t\ttokenizer.resetSyntax();\r\n\t\ttokenizer.wordChars(33, 255);\r\n\t\ttokenizer.whitespaceChars(0, 32);\r\n\t\treturn tokenizer;\r\n\t}\r\n\t\r\n\t/** Returns whether EOF has been reached. */\r\n\tpublic boolean endOfFile() throws IOException {\r\n\t\ttokenizer.nextToken();\r\n\t\tboolean eof = (tokenizer.ttype == StreamTokenizer.TT_EOF);\r\n\t\ttokenizer.pushBack();\r\n\t\treturn eof;\r\n\t}\r\n\t\r\n\t\r\n\t/** Reads the next event. Feature are filtered if apropos. */\r\n\tprivate Event nextEvent() throws IOException {\r\n\t\ttokenizer.nextToken();\r\n\t\tint count = Integer.parseInt(tokenizer.sval); \r\n\t\ttokenizer.nextToken();\r\n\t\tint numFeats = Integer.parseInt(tokenizer.sval); \r\n\t\tFeatureList fv = new FeatureList(numFeats);\r\n\t\tfor (int i=0; i < numFeats; i++) {\r\n\t\t\ttokenizer.nextToken();\r\n\t\t\tString feat = tokenizer.sval;\r\n\t\t\ttokenizer.nextToken();\r\n\t\t\tfloat val = Float.parseFloat(tokenizer.sval);\r\n\t\t\tAlphabet.Feature f = alphabet.index(feat);\r\n\t\t\tif (f != null) fv.add(f, val); \r\n\t\t}\r\n\t\treturn new Event(fv, count);\r\n\t}\r\n\t\r\n\t/** Reads the next block, or null if none. */\r\n\tpublic Block nextBlock() throws IOException {\r\n\t\t// first check block iterator for in-mem case\r\n\t\tif (blockIt != null) {\r\n\t\t\treturn (blockIt.hasNext()) ? blockIt.next() : null;\r\n\t\t}\r\n\t\t// otherwise check for eof, noting completion for in-mem case\r\n\t\tif (endOfFile()) {\r\n\t\t\teofReached = true; return null;\r\n\t\t}\r\n\t\t// otherwise parse next block\r\n\t\ttokenizer.nextToken();\r\n\t\tint numEvents = Integer.parseInt(tokenizer.sval);\r\n\t\tList events = new ArrayList(numEvents);\r\n\t\tfor (int i=0; i < numEvents; i++) {\r\n\t\t\tevents.add(nextEvent());\r\n\t\t}\r\n\t\tBlock retval = new Block(events);\r\n\t\t// save block with in-mem case\r\n\t\tif (blocks != null) blocks.add(retval);\r\n\t\t// done\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\t\r\n\t/** Returns a reader for the given file, using gzip inflation if the file's name ends with .gz. */\r\n\tpublic static Reader openReader(File file) throws IOException {\r\n\t\tif (file.getName().endsWith(\".gz\"))\r\n\t\t\treturn new InputStreamReader(new GZIPInputStream(new FileInputStream(file)));\r\n\t\telse\r\n\t\t\treturn new BufferedReader(new FileReader(file));\r\n\t}\r\n\r\n\t/** Returns a printwriter for the given file, using gzip deflation if the file's name ends with .gz. */\r\n\tpublic static PrintWriter openWriter(File file) throws IOException {\r\n\t\tif (file.getName().endsWith(\".gz\"))\r\n\t\t\treturn new PrintWriter(new OutputStreamWriter(new GZIPOutputStream(new FileOutputStream(file))));\r\n\t\telse\r\n\t\t\treturn new PrintWriter(new BufferedWriter(new FileWriter(file)));\r\n\t}\r\n\t\r\n\t/** Writes the events for a given list of signs according to the feature extractor and best sign. */\r\n\tpublic static void writeEvents(PrintWriter pw, List signs, Sign best, FeatureExtractor fe) throws IOException {\r\n \tCollections.shuffle(signs);\r\n \tpw.println(Integer.toString(signs.size()));\r\n \tfor (Sign s : signs) {\r\n \t\tint count = 0;\r\n \t\tif (s == best) count = 1;\r\n \t\tpw.print(count + \" \");\r\n \t\tFeatureVector fvect = fe.extractFeatures(s, true);\r\n \t\tint numfeats = fvect.size();\r\n \t\tpw.print(numfeats + \" \");\r\n\t\t\tfor (FeatureVector.Iterator it = fvect.iterator(); it.hasNext(); ) {\r\n \t\t\tpw.print(it.nextFeature().name() + \" \" + it.nextValue() + \" \");\r\n\t\t\t}\r\n \t\tpw.println();\r\n \t}\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccg/perceptron/FeatureExtractor.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2008 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.perceptron;\r\n\r\nimport opennlp.ccg.synsem.*;\r\n\r\n/** \r\n * Interface for mappings signs to features. \r\n * \r\n * @author Michael White\r\n * @version $Revision: 1.5 $, $Date: 2009/06/22 04:32:50 $\r\n */ \r\npublic interface FeatureExtractor {\r\n\t\r\n\t/** Returns the features for the given sign and completeness flag. */\r\n\tpublic FeatureVector extractFeatures(Sign sign, boolean complete);\r\n\t\r\n\t/** Sets the alphabet. */\r\n\tpublic void setAlphabet(Alphabet alphabet);\r\n}\r\n" }, { "path": "src/opennlp/ccg/perceptron/FeatureList.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2008 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.perceptron;\r\n\r\nimport java.util.*;\r\n\r\n/** \r\n * A feature vector represented by a list of features and a list of values.\r\n * Features are assumed to be unique. \r\n * \r\n * @author\tMichael White\r\n * @version\t$Revision: 1.2 $, $Date: 2011/10/11 03:20:05 $\r\n */ \r\npublic class FeatureList implements FeatureVector {\r\n\t\r\n\t@SuppressWarnings(\"unused\")\r\n\tprivate static final long serialVersionUID = 325935806787566283L;\r\n\t\r\n\t// features\r\n\tprivate ArrayList features;\r\n\t// values\r\n\tprivate ArrayList values;\r\n\t\r\n\t\r\n\t/** Default constructor. */\r\n\tpublic FeatureList() {\r\n\t\tfeatures = new ArrayList();\r\n\t\tvalues = new ArrayList();\r\n\t}\r\n\t\r\n\t/** Constructor with size. */\r\n\tpublic FeatureList(int size) { \r\n\t\tfeatures = new ArrayList(size);\r\n\t\tvalues = new ArrayList(size);\r\n\t}\r\n\t\r\n\t/** Constructor from feature vector. */\r\n\tpublic FeatureList(FeatureVector fv) {\r\n\t\tfor (Iterator it = fv.iterator(); it.hasNext(); ) \r\n\t\t\tadd(it.nextFeature(), it.nextValue());\r\n\t}\r\n\r\n\t\r\n\t/** Add feature-value pair. */\r\n\tpublic void add(Alphabet.Feature feature, Float value) {\r\n\t\tfeatures.add(feature); values.add(value);\r\n\t}\r\n\t\r\n\t/** Add feature vector (features assumed distinct). */\r\n\tpublic void add(FeatureVector fv) {\r\n\t\tfeatures.ensureCapacity(size() + fv.size());\r\n\t\tvalues.ensureCapacity(size() + fv.size());\r\n\t\tfor (Iterator it = fv.iterator(); it.hasNext(); ) \r\n\t\t\tadd(it.nextFeature(), it.nextValue());\r\n\t}\r\n\t\r\n\t/** Get feature at index. */\r\n\tpublic Alphabet.Feature getFeature(int index) { return features.get(index); }\r\n\t\r\n\t/** Get value at index. */\r\n\tpublic Float getValue(int index) { return values.get(index); }\r\n\r\n\t\r\n\t/** Size. */\r\n\tpublic int size() { return features.size(); }\r\n\t\r\n\t/** Iterator. */\r\n\tpublic Iterator iterator() {\r\n\t\treturn new Iterator() {\r\n\t\t\tjava.util.Iterator itF = features.iterator();\r\n\t\t\tjava.util.Iterator itV = values.iterator();\r\n\t\t\tpublic boolean hasNext() { return itF.hasNext(); }\r\n\t\t\tpublic Alphabet.Feature nextFeature() { return itF.next(); }\r\n\t\t\tpublic Float nextValue() { return itV.next(); }\r\n\t\t};\r\n\t}\r\n\t\r\n\t\r\n\t/** toString. */\r\n\tpublic String toString() {\r\n\t\tString retval = \"features: \";\r\n\t\tfor (int i=0; i < features.size(); i++)\r\n\t\t\tretval += \"<\" + getFeature(i).name() + \",\" + getValue(i) + \"> \"; \r\n\t\treturn retval;\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccg/perceptron/FeatureMap.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2008 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.perceptron;\r\n\r\nimport java.util.*;\r\n\r\n/** \r\n * A feature vector represented using a map. \r\n * \r\n * @author Michael White\r\n * @version $Revision: 1.9 $, $Date: 2011/10/11 03:20:06 $\r\n */ \r\npublic class FeatureMap implements FeatureVector {\r\n\r\n\t@SuppressWarnings(\"unused\")\r\n\tprivate static final long serialVersionUID = -5054109887068464041L;\r\n\r\n\t// the map\r\n\tprivate HashMap map;\r\n\t\r\n\t/** Default constructor. */\r\n\tpublic FeatureMap() { map = new LinkedHashMap(); } \r\n\t\r\n\t/** Constructor with size. */\r\n\tpublic FeatureMap(int size) { map = new LinkedHashMap(size*2); }\r\n\t\r\n\t/** Constructor from feature vector. */\r\n\tpublic FeatureMap(FeatureVector fv) { \r\n\t\tthis(fv.size()*4);\r\n\t\tfor (Iterator it = fv.iterator(); it.hasNext(); ) \r\n\t\t\tmap.put(it.nextFeature(), it.nextValue());\r\n\t}\r\n\t\r\n\t/** Constructor from two feature vectors. */\r\n\tpublic FeatureMap(FeatureVector fv1, FeatureVector fv2) { \r\n\t\tthis((fv1.size()+fv2.size())*3);\r\n\t\tfor (Iterator it = fv1.iterator(); it.hasNext(); ) \r\n\t\t\tmap.put(it.nextFeature(), it.nextValue());\r\n\t\tadd(fv2);\r\n\t}\r\n\t\r\n\t\r\n\t/** Increments a feature count. */\r\n\tpublic void inc(Alphabet.Feature feature) {\r\n\t\tfloat count = 1;\r\n\t\tFloat val = map.get(feature);\r\n\t\tif (val != null) count = Math.round(val) + 1;\r\n\t\tmap.put(feature, count);\r\n\t}\r\n\t\r\n\t/** Adds to a feature's value (starting with zero). */\r\n\tpublic void add(Alphabet.Feature feature, Float value) {\r\n\t\tFloat val = map.get(feature);\r\n\t\tif (val != null) map.put(feature, val + value);\r\n\t\telse map.put(feature, value);\r\n\t}\r\n\t\r\n\t/** Adds a feature vector. */\r\n\tpublic void add(FeatureVector fv) {\r\n\t\tfor (Iterator it = fv.iterator(); it.hasNext(); ) \r\n\t\t\tadd(it.nextFeature(), it.nextValue());\r\n\t}\r\n\t\r\n\t/** Returns the feature's value (zero if not present). */\r\n\tpublic float get(Alphabet.Feature feature) {\r\n\t\tFloat retval = map.get(feature);\r\n\t\treturn (retval != null) ? retval : 0;\r\n\t}\r\n\t\r\n\t/** Clears the map. */\r\n\tpublic void clear() { map.clear(); }\r\n\t\r\n\t/** Size. */\r\n\tpublic int size() { return map.size(); }\r\n\t\r\n\t/** Returns an iterator over the entries. */\r\n\tpublic Iterator iterator() {\r\n\t\treturn new Iterator() {\r\n\t\t\tjava.util.Iterator> it = map.entrySet().iterator();\r\n\t\t\tMap.Entry entry = null;\r\n\t\t\tpublic boolean hasNext() { return it.hasNext(); }\r\n\t\t\tpublic Alphabet.Feature nextFeature() { entry = it.next(); return entry.getKey(); }\r\n\t\t\tpublic Float nextValue() { return entry.getValue(); }\r\n\t\t};\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccg/perceptron/FeatureVector.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2008 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.perceptron;\r\n\r\n/** \r\n * A sparse feature vector, with size and an iterator over feature-value pairs. \r\n * \r\n * @author \tMichael White\r\n * @version\t$Revision: 1.6 $, $Date: 2011/01/15 17:52:59 $\r\n */ \r\npublic interface FeatureVector {\r\n\t\r\n\t/**\r\n\t * Iterator over feature-value pairs.\r\n\t * Features and values must be retrieved using a pair of calls \r\n\t * to nextFeature and nextValue, otherwise the behavior is not defined.\r\n\t */\r\n\tpublic interface Iterator {\r\n\t\t\r\n\t\t/** Whether any items remain. */\r\n\t\tpublic boolean hasNext();\r\n\t\t\r\n\t\t/** Returns the next feature. */\r\n\t\tpublic Alphabet.Feature nextFeature();\r\n\t\t\r\n\t\t/** Returns the next value. */\r\n\t\tpublic Float nextValue();\r\n\t}\r\n\t\r\n\t/** Size. */\r\n\tpublic int size();\r\n\r\n\t/** Iterator over feature-value pairs. */\r\n\tpublic Iterator iterator();\r\n\t\r\n\t/** Empty iterator. */\r\n\tpublic static Iterator EMPTY_ITERATOR = new Iterator() {\r\n\t\tpublic boolean hasNext() { return false; }\r\n\t\tpublic Alphabet.Feature nextFeature() { return null; }\r\n\t\tpublic Float nextValue() { return null; }\r\n\t};\r\n}\r\n" }, { "path": "src/opennlp/ccg/perceptron/Model.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2008 Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.perceptron;\n\nimport java.util.*;\nimport java.io.*;\n\nimport opennlp.ccg.perceptron.Alphabet.Feature;\nimport opennlp.ccg.util.Pair;\n\n/**\n * A model is a vector of weights for an alphabet.\n * \n * A model can be read from a file, which starts with the number \n * of features on one line, followed by one line per feature pairing \n * the feature name with its weight.\n * \n * A new model with all zero weights can also be created from an alphabet.\n * \n * The main routine tests the model on an event file.\n * \n * @author Michael White\n * @version $Revision: 1.7 $, $Date: 2009/11/09 18:54:30 $\n */\npublic class Model {\n\n\t/** Filter interface for adjusting feature weights when loading a model. */\n\tpublic interface FeatureFilter {\n\t\t/** Returns the modified feature weight for the given feature. */\n\t\tpublic double adjustedWeight(String name, double weight);\n\t}\n\n\t/** Flag for whether to print debugging info to System.err. */\n\tpublic boolean debug = false;\n\t\n\t// weight vector\n\tprivate double[] weights;\n\t\n\t// alphabet\n\tprivate Alphabet alphabet;\n\n\t\n\t/** Constructor with alphabet, for a new model with all zero weights. */\n\tpublic Model(Alphabet alphabet) {\n\t\tthis.alphabet = alphabet;\n\t\tweights = new double[alphabet.size()];\n\t\tzero();\n\t}\n\t\n\t/** Constructor to load a model from a file. The alphabet is set to closed. */\n\tpublic Model(String filename) throws IOException {\n\t this(filename, null);\n\t}\n\n\t/**\n\t * Constructor to load a model from a file using a feature filter. The\n\t * alphabet is set to closed.\n\t */\n\tpublic Model(String filename, FeatureFilter filter) throws IOException {\n\t\tReader reader = EventFile.openReader(new File(filename));\n\t\tStreamTokenizer tokenizer = EventFile.initTokenizer(reader);\n\t\ttokenizer.nextToken();\n\t\tint size = Integer.parseInt(tokenizer.sval);\n\t\talphabet = new Alphabet(size);\n\t\tweights = new double[size];\n\t\tfor (int i = 0; i < size; i++) {\n\t\t\ttokenizer.nextToken();\n\t\t\tString name = tokenizer.sval;\n\t\t\ttokenizer.nextToken();\n\t\t\tdouble weight = Double.parseDouble(tokenizer.sval);\n\t\t\tif (filter != null)\n\t\t\t\tweight = filter.adjustedWeight(name, weight);\n\t\t\talphabet.add(name);\n\t\t\tweights[i] = weight;\n\t\t}\n\t\treader.close();\n\t\talphabet.setClosed(true);\n\t}\n\t\n\t/** Returns the size of the model. */\n\tpublic int size() { return weights.length; }\n\t\n\t/** Returns the alphabet. */\n\tpublic Alphabet getAlphabet() { return alphabet; }\n\t\n\t\n\t/** Returns the weight for the given index. */\n\tpublic double getWeight(int index) { return weights[index]; }\n\t\n\t/** Returns the weight for the given feature. */\n\tpublic double getWeight(String feat) { return weights[alphabet.index(feat).getIndex()]; }\n\t\n\t/** Returns the weight for the given feature. */\n\tpublic double getWeight(Alphabet.Feature f) { return weights[f.getIndex()]; }\n\t\n\t/** Sets the weight for the given index. */\n\tpublic void setWeight(int index, double weight) { weights[index] = weight; }\n\t\n\t/** Sets the weight for the given feature. */\n\tpublic void setWeight(String feat, double weight) { weights[alphabet.index(feat).getIndex()] = weight; }\n\n\t/** Sets the weight for the given feature. */\n\tpublic void setWeight(Alphabet.Feature f, double weight) { weights[f.getIndex()] = weight; }\n\n\t\n\t/** Returns the dot product of the weights and features. */\n\tpublic double score(FeatureVector fv) {\n\t\tdouble retval = 0.0;\n\t\tfor (FeatureVector.Iterator it = fv.iterator(); it.hasNext(); ) {\n\t\t\tFeature feat = it.nextFeature();\n\t\t\tFloat value = it.nextValue();\n\t\t\tInteger index = feat.getIndex();\n\t\t\tif (index == null) continue;\n\t\t\tretval += weights[index] * value;\n\t\t}\n\t\tif (debug) System.err.println(\"score: \" + retval + \" \" + fv);\n\t\treturn retval;\n\t}\n\t\n\t/** Adds the feature vector values to the weights. */\n\tpublic void add(FeatureVector fv) {\n\t\tfor (FeatureVector.Iterator it = fv.iterator(); it.hasNext(); ) {\n\t\t\tFeature feat = it.nextFeature();\n\t\t\tFloat value = it.nextValue();\n\t\t\tInteger index = feat.getIndex();\n\t\t\tif (index == null) continue;\n\t\t\tweights[index] += value;\n\t\t}\n\t}\n\t\n\t/** Subtracts the feature vector values from the weights. */\n\tpublic void subtract(FeatureVector fv) {\n\t\tfor (FeatureVector.Iterator it = fv.iterator(); it.hasNext(); ) {\n\t\t\tFeature feat = it.nextFeature();\n\t\t\tFloat value = it.nextValue();\n\t\t\tInteger index = feat.getIndex();\n\t\t\tif (index == null) continue;\n\t\t\tweights[index] -= value;\n\t\t}\n\t}\n\n\t\n\t/** Adds the given model's weights to this model. The models are assumed to share the same alphabet. */\n\tpublic void add(Model model) {\n\t\tfor (int i=0; i < weights.length; i++) {\n\t\t\tweights[i] += model.weights[i];\n\t\t}\n\t}\n\t\n\t/** Multiplies the weights by the given number. */\n\tpublic void multiply(double num) {\n\t\tfor (int i=0; i < weights.length; i++) {\n\t\t\tweights[i] *= num;\n\t\t}\n\t}\n\n\t/** Resets the weights to zero. */\n\tpublic void zero() {\n\t\tfor (int i=0; i < weights.length; i++) {\n\t\t\tweights[i] = 0.0;\n\t\t}\n\t}\n\n\t\n\t/** Sets this model's weights to the given model's ones, where the alphabets intersect. */\n\tpublic void set(Model model) {\n\t\tzero();\n\t\tfor (int i=0; i < model.weights.length; i++) {\n\t\t\tAlphabet.Feature f = model.alphabet.feature(i);\n\t\t\tAlphabet.Feature f0 = alphabet.index(f);\n\t\t\tif (f0 == null || f0.getIndex() == null) continue;\n\t\t\tweights[f0.getIndex()] = model.weights[i];\n\t\t}\n\t}\n\t\n\t\n\t/** Returns the best event (first tied if ties). */\n\tpublic FeatureVector best(List fvs) {\n\t\tFeatureVector retval = null; double max = Double.NEGATIVE_INFINITY;\n\t\tfor (FeatureVector fv : fvs) {\n\t\t\tdouble score = score(fv);\n\t\t\tif (score > max) { retval = fv; max = score; }\n\t\t}\n\t\treturn retval;\n\t}\n\t\n\t/** Returns the best event (first tied if ties). */\n\tpublic EventFile.Event best(EventFile.Block block) {\n\t\tEventFile.Event retval = null; double max = Double.NEGATIVE_INFINITY;\n\t\tfor (EventFile.Event event : block.events) {\n\t\t\tdouble score = score(event.features);\n\t\t\tif (score > max) { retval = event; max = score; }\n\t\t}\n\t\treturn retval;\n\t}\n\t\n\t/** Returns the accuracy on the event file. */\n\tpublic double accuracy(EventFile eventFile) throws IOException {\n\t\tif (alphabet != eventFile.getAlphabet()) {\n\t\t\tthrow new RuntimeException(\"Model and EventFile must share the same alphabet!\");\n\t\t}\n\t\tint correct = 0; int total = 0;\n\t\tEventFile.Block block;\n\t\twhile ( (block = eventFile.nextBlock()) != null ) {\n\t\t\ttotal++;\n\t\t\tif (best(block) == block.best()) {\n\t\t\t\tcorrect++;\n\t\t\t\tif (debug) System.err.println(\"CORRECT\");\n\t\t\t}\n\t\t\telse {\n\t\t\t\tif (debug) System.err.println(\"WRONG; best: \" + block.best());\n\t\t\t}\n\t\t}\n\t\tif (debug) System.err.println(\"correct: \" + correct + \" total: \" + total);\n\t\treturn 1.0 * correct / total;\n\t}\n\t\n\t\n\t/** Saves the model to a file, filtering out zero weights. */\n\tpublic void save(String filename) throws IOException { save(filename, 0.0); }\n\t\n\t/** Saves the model to a file, filtering out weights whose absolute value does not exceed the pruning value. */\n\tpublic void save(String filename, double minPrune) throws IOException {\n\t\tFile file = new File(filename);\n\t\tPrintWriter out = EventFile.openWriter(file);\n\t\t// calc pruned size\n\t\tint size = size();\n\t\tint pruned = 0;\n\t\tfor (int i=0; i < size; i++) if (Math.abs(weights[i]) <= minPrune) pruned++;\n\t\tint prunedSize = size - pruned;\n\t\t// write pruned size \n\t\tout.println(Integer.toString(prunedSize));\n\t\t// collect unpruned weights\n\t\tList> featWeights = new ArrayList>(prunedSize);\n\t\tfor (int i=0; i < size; i++) {\n\t\t\tif (Math.abs(weights[i]) <= minPrune) continue;\n\t\t\tfeatWeights.add(new Pair(alphabet.feature(i), weights[i]));\n\t\t}\n\t\t// sort weights by descending absolute value\n\t\t// (further sorting alphabetically may take too long)\n\t\tCollections.sort(\n\t\t\t\tfeatWeights, \n\t\t\t\tnew Comparator>() {\n\t\t\t\t\tpublic int compare(Pair entry1, Pair entry2) {\n\t\t\t\t\t\tdouble val1 = Math.abs(entry1.b); double val2 = Math.abs(entry2.b);\n\t\t\t\t\t\tif (val1 > val2) return -1;\n\t\t\t\t\t\tif (val1 < val2) return 1;\n\t\t\t\t\t\treturn 0;\n\t\t\t\t\t\t//return entry1.a.name().compareTo(entry2.a.name());\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t);\n\t\t// write sorted weights\n\t\tfor (Pair fw : featWeights) {\n\t\t\tout.println(fw.a.name() + \" \" + fw.b);\n\t\t}\n\t\tout.close();\n\t}\n\t\n\t\n\t/**\n\t * Loads a model from a file and tests it on the given event file.\n\t */\n\tpublic static void main(String[] args) throws IOException {\n\t\t// help\n\t\tif (args.length < 2) {\n\t\t\tSystem.out.println(\"Usage: java perceptron.Model (-debug)\");\n\t\t\tSystem.exit(0);\n\t\t}\n\t\t// args\n\t\tString modelfile = args[0];\n\t\tString eventfile = args[1];\n\t\tboolean debug = Arrays.asList(args).contains(\"-debug\");\n\t\t// load model\n\t\tSystem.out.println(\"Loading model from: \" + modelfile);\n\t\tModel model = new Model(modelfile); \n\t\tmodel.debug = debug;\n\t\tSystem.out.println(\"model size: \" + model.size());\n\t\tSystem.out.println(\"debug: \" + debug);\n\t\t// compute accuracy\n\t\tSystem.out.println(\"Scoring events in: \" + eventfile);\n\t\tEventFile eventFile = new EventFile(eventfile, model.alphabet);\n\t\tdouble accuracy = model.accuracy(eventFile);\n\t\tSystem.out.println(\"accuracy: \" + accuracy);\n\t}\n}\n" }, { "path": "src/opennlp/ccg/perceptron/PerceptronScorer.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2008 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.perceptron;\r\n\r\nimport opennlp.ccg.synsem.*;\r\n\r\n/**\r\n * A sign scorer for a perceptron model.\r\n * \r\n * @author Michael White\r\n * @version $Revision: 1.7 $, $Date: 2011/03/21 20:46:43 $\r\n */\r\npublic class PerceptronScorer implements SignScorer {\r\n\r\n\t/** The feature extractor. */\r\n\tpublic final FeatureExtractor featureExtractor;\r\n\t\r\n\t/** The model. */\r\n\tpublic final Model model;\r\n\t\r\n\t/** Constructor. */\r\n\tpublic PerceptronScorer(FeatureExtractor featureExtractor, Model model) {\r\n\t\tthis.featureExtractor = featureExtractor;\r\n\t\tthis.model = model;\r\n\t\tfeatureExtractor.setAlphabet(model.getAlphabet());\r\n\t}\r\n\t\r\n /** \r\n * Returns a score for the given sign and completeness flag, where higher \r\n * numbers are better than lower numbers.\r\n * In particular, returns the score assigned by the model to the features \r\n * extracted from the given sign with the given completeness flag.\r\n */\r\n public double score(Sign sign, boolean complete) {\r\n \treturn model.score(featureExtractor.extractFeatures(sign, complete));\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/perceptron/ReRankingPerceptronScorer.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2011 Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.perceptron;\n\nimport opennlp.ccg.synsem.*;\n\n/**\n * A re-ranking sign scorer for a perceptron model.\n * Note that at present, n-best re-ranking has been found to work better \n * for parsing, but not for realization, where forest re-ranking (ie using \n * the perceptron scorer throughout) seems to work better.\n * \n * @author Michael White\n * @version $Revision: 1.1 $, $Date: 2011/03/21 20:46:32 $\n */\npublic abstract class ReRankingPerceptronScorer extends PerceptronScorer implements ReRankingScorer {\n\n\t/** Flag for whether to use the full model. */\n protected boolean useFullModel = false;\n\n /** Sets the full model flag. */\n public void setFullModel(boolean on) { useFullModel = on; }\n\n\t/** The base scorer, for use when the full model is turned off. */\n protected SignScorer baseScorer;\n\n /** Returns the base scorer, using the given feature extractor if desired. */\n abstract protected SignScorer getBaseScorer(FeatureExtractor featureExtractor);\n\n /** Constructor that configures the base scorer using getBaseScorer. */\n\tpublic ReRankingPerceptronScorer(FeatureExtractor featureExtractor, Model model) {\n\t\tsuper(featureExtractor, model);\n\t\tbaseScorer = getBaseScorer(featureExtractor);\n\t}\n\n /** Scores the sign with the full or base model, according to the full model flag. */\n public double score(Sign sign, boolean complete) {\n\t\tif (useFullModel) return super.score(sign, complete);\n\t\telse return baseScorer.score(sign, complete);\n }\n}\n" }, { "path": "src/opennlp/ccg/perceptron/Trainer.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2008-2013 Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.perceptron;\n\n//import java.util.*;\nimport java.io.*;\n\n/**\n * Trains an averaged perceptron model by iterating through an event file. \n * \n * The main routine trains and saves a model. Options include saving \n * the final, non-averaged model, and saving the averaged model after \n * every N iterations. There's also an option to keep the events in \n * memory for subsequent iterations.\n * \n * An alphabet must be supplied as input. The main routine of the \n * Alphabet class can be used to derive an alphabet from a training\n * file.\n * \n * @author Michael White\n * @version $Revision: 1.4 $, $Date: 2009/06/18 23:38:57 $\n */\npublic class Trainer {\n\n\t/** The training file. */\n\tpublic final String trainfile;\n\t\n\t/** The alphabet. */\n\tpublic final Alphabet alphabet;\n\t\n\t/** The number of training iterations. */\n\tpublic final int iterations;\n\t\n\t/** The model file. */\n\tpublic final String modelfile;\n\t\n\t/** The in-memory flag. */\n\tpublic final boolean inMemory;\n\t\n\t/** The current model. */\n\tpublic final Model currentModel;\n\t\n\t/** The averaged model. */\n\tpublic final Model averagedModel;\n\t\n\t/** The model for averaging over the current iteration. */\n\tprivate Model iterationModel;\n\t\n\t/** The number of iterations to use when saving every N iterations. */\n\tprivate int saveEveryNth = 0;\n\n\t\n\t/** Constructor. */\n\tpublic Trainer(String trainfile, String alphabetfile, int iterations, String modelfile, boolean inMemory) throws IOException {\n\t\tthis.trainfile = trainfile;\n\t\tthis.alphabet = new Alphabet(alphabetfile);\n\t\tthis.iterations = iterations;\n\t\tthis.modelfile = modelfile;\n\t\tthis.inMemory = inMemory;\n\t\tthis.currentModel = new Model(alphabet);\n\t\tthis.averagedModel = new Model(alphabet);\n\t\tthis.iterationModel = new Model(alphabet);\n\t}\n\t\n\t/** Initializes the model. */\n\tpublic void initModel(String initmodelfile) throws IOException {\n\t\tModel model = new Model(initmodelfile);\n\t\tcurrentModel.set(model);\n\t\taveragedModel.set(model);\n\t}\n\t\n\t\n\t/** Train the model, reporting accuracy of the averaged model for each iteration and the final model. */\n\tpublic void train() throws IOException {\n\t\t// open training file\n\t\tEventFile eventFile = new EventFile(trainfile, alphabet, inMemory);\n\t\t// iterate\n\t\tboolean converged = false;\n\t\tfor (int i=0; i < iterations; i++) {\n\t\t\tSystem.out.println(\"iteration: \" + i);\n\t\t\t// reset training file\n\t\t\teventFile.reset();\n\t\t\t// zero iteration model\n\t\t\titerationModel.zero();\n\t\t\t// init counters\n\t\t\tint updates = 0; int correct = 0; int total = 0;\n\t\t\tEventFile.Block block;\n\t\t\t// go through training file\n\t\t\twhile ( (block = eventFile.nextBlock()) != null ) {\n\t\t\t\ttotal++;\n\t\t\t\t// update if wrong\n\t\t\t\tEventFile.Event modelBest = currentModel.best(block);\n\t\t\t\tEventFile.Event actualBest = block.best();\n\t\t\t\tif (modelBest != actualBest) {\n\t\t\t\t\tupdates++;\n\t\t\t\t\tcurrentModel.add(actualBest.features);\n\t\t\t\t\tcurrentModel.subtract(modelBest.features);\n\t\t\t\t}\n\t\t\t\t// see if averaged model correct\n\t\t\t\tEventFile.Event avgModelBest = averagedModel.best(block);\n\t\t\t\tif (avgModelBest == actualBest) correct++;\n\t\t\t\t// update iteration model\n\t\t\t\titerationModel.add(currentModel);\n\t\t\t}\n\t\t\t// divide iteration model by total to yield an average for this iteration, \n\t\t\t// and divide by iteration number (+1) to yield proportion of this \n\t\t\t// iteration to averaged model; combine these two steps into one call\n\t\t\tdouble denominator = 1.0 * total * (i+1);\n\t\t\titerationModel.multiply(1.0 / denominator);\n\t\t\t// multiply averaged model by i/(i+1) for proportion of previous iterations\n\t\t\tif (i > 0) {\n\t\t\t\tdouble mult = 1.0 * i / (i+1);\n\t\t\t\taveragedModel.multiply(mult);\n\t\t\t}\n\t\t\t// updated averaged model\n\t\t\taveragedModel.add(iterationModel);\n\t\t\t// report\n\t\t\tSystem.out.println(\"updates: \" + updates);\n\t\t\tSystem.out.println(\"avg model correct: \" + correct + \" total: \" + total + \" accuracy: \" + (1.0 * correct / total));\n\t\t\tSystem.out.println();\n\t\t\tif (updates == 0) {\n\t\t\t\tSystem.out.println(\"converged\");\n\t\t\t\tSystem.out.println();\n\t\t\t\tconverged = true; break;\n\t\t\t}\n\t\t\t// save every nth\n\t\t\tif (saveEveryNth > 0 && i < (iterations-1) && i % saveEveryNth == 0) {\n\t\t\t\tString nthModelfile = nthFilename(modelfile, i);\n\t\t\t\tSystem.out.println(\"Saving model to \" + nthModelfile);\n\t\t\t\taveragedModel.save(nthModelfile);\n\t\t\t\tSystem.out.println();\n\t\t\t}\n\t\t}\n\t\t// do one more iteration to compute accuracy if not converged\n\t\tif (!converged) {\n\t\t\tSystem.out.println(\"final iteration: \");\n\t\t\t// reset training file\n\t\t\teventFile.reset();\n\t\t\t// init counters\n\t\t\tint finalCorrect = 0; int correct = 0; int total = 0;\n\t\t\tEventFile.Block block;\n\t\t\t// go through training file\n\t\t\twhile ( (block = eventFile.nextBlock()) != null ) {\n\t\t\t\ttotal++;\n\t\t\t\t// see if correct\n\t\t\t\tEventFile.Event modelBest = currentModel.best(block);\n\t\t\t\tEventFile.Event avgModelBest = averagedModel.best(block);\n\t\t\t\tEventFile.Event actualBest = block.best();\n\t\t\t\tif (modelBest == actualBest) finalCorrect++;\n\t\t\t\tif (avgModelBest == actualBest) correct++;\n\t\t\t}\n\t\t\t// report\n\t\t\tSystem.out.println(\"final model correct: \" + finalCorrect + \" total: \" + total + \" accuracy: \" + (1.0 * finalCorrect / total));\n\t\t\tSystem.out.println(\"avg model correct: \" + correct + \" total: \" + total + \" accuracy: \" + (1.0 * correct / total));\n\t\t\tSystem.out.println();\n\t\t}\n\t\t// close training file\n\t\teventFile.close();\n\t}\n\n\t\n\t/** Returns a filename with .N added before the extension, if any. */\n\tpublic static String nthFilename(String filename, int N) {\n\t\tint lastdot = filename.lastIndexOf('.');\n\t\tif (lastdot > 0) return filename.substring(0,lastdot) + \".\" + N + filename.substring(lastdot);\n\t\telse return filename + \".\" + N;\n\t}\n\t\n\t\n\t/**\n\t * Trains an averaged perceptron model from the training file using the alphabet file and the given \n\t * number of iterations, saving the file to model file.\n\t * The final (non-averaged) model can optionally be saved using the -f option, \n\t * and intermediate models can be saved every N iterations using the -n option.\n\t * The -in_mem option keeps the events in memory for subsequent iterations.\n\t */\n\tpublic static void main(String[] args) throws IOException {\n\t\t// help\n\t\tif (args.length < 4) {\n\t\t\tSystem.out.println(\n\t\t\t\t\"Usage: java perceptron.Trainer \" + \n\t\t\t\t\"(-i ) (-f ) (-n ) (-in_mem)\"\n\t\t\t);\n\t\t\tSystem.exit(0);\n\t\t}\n\t\t// args\n\t\tString traineventfile = args[0];\n\t\tString alphabetfile = args[1];\n\t\tint iterations = Integer.parseInt(args[2]);\n\t\tString modelfile = args[3];\n\t\tString initmodelfile = null;\n\t\tString finalmodelfile = null;\n\t\tint saveEveryNth = 0;\n\t\tboolean inMemory = false;\n\t\tfor (int i=4; i < args.length; i++) {\n\t\t\tif (args[i].equals(\"-i\")) initmodelfile = args[++i]; \n\t\t\tif (args[i].equals(\"-f\")) finalmodelfile = args[++i]; \n\t\t\tif (args[i].equals(\"-n\")) saveEveryNth = Integer.parseInt(args[++i]); \n\t\t\tif (args[i].equals(\"-in_mem\")) inMemory = true;\n\t\t}\n\t\t// setup, train\n\t\tSystem.out.println(\"Training on \" + traineventfile + \" using \" + alphabetfile + \" for \" + iterations + \" iterations\");\n\t\tif (initmodelfile != null) System.out.println(\"with \" + initmodelfile + \" as the initial model\");\n\t\tif (inMemory) System.out.println(\"keeping events in memory\");\n\t\tSystem.out.println();\n\t\tTrainer trainer = new Trainer(traineventfile, alphabetfile, iterations, modelfile, inMemory);\n\t\tif (initmodelfile != null) trainer.initModel(initmodelfile);\n\t\ttrainer.saveEveryNth = saveEveryNth;\n\t\ttrainer.train();\n\t\t// save model\n\t\tSystem.out.println(\"Saving model to \" + modelfile);\n\t\ttrainer.averagedModel.save(modelfile);\n\t\tif (finalmodelfile != null) {\n\t\t\tSystem.out.println(\"Saving model to \" + finalmodelfile);\n\t\t\ttrainer.currentModel.save(finalmodelfile);\n\t\t}\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/realize/Chart.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2003-11 University of Edinburgh / Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.realize;\r\n\r\nimport opennlp.ccg.*;\r\nimport opennlp.ccg.synsem.*;\r\nimport opennlp.ccg.util.Pair;\r\nimport opennlp.ccg.ngrams.NgramPrecisionModel;\r\nimport opennlp.ccg.parse.DerivationHistory;\r\nimport opennlp.ccg.hylo.*;\r\n\r\nimport java.io.*;\r\nimport java.util.*;\r\nimport java.util.prefs.*;\r\n\r\nimport gnu.trove.*;\r\n\r\n/**\r\n * The chart manages the creation of edges. Newly added edges are kept on an \r\n * agenda prior to rule applications. In the anytime mode, the chart completion \r\n * process can be interrupted according to the given parameters and preferences \r\n * (see Chart.combine). In the packing mode, completing the chart produces \r\n * a packed representation, which may then be unpacked if the DO_UNPACKING \r\n * preference is set.\r\n *\r\n * A single chart instance per realization request is assumed.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.79 $, $Date: 2011/08/28 04:05:39 $\r\n */\r\npublic class Chart\r\n{\r\n \r\n /** Preference key for time limit on edge combination. */\r\n public static final String TIME_LIMIT = \"Time Limit\";\r\n \r\n /** Preference key for time limit on finding a new best complete realization. \r\n If between o-1, the time limit is interpreted as a percentage of the \r\n time until the first realization is found. */\r\n public static final String NEW_BEST_TIME_LIMIT = \"New Best Time Limit\";\r\n \r\n /** A constant indicating no time limit on edge combination. */\r\n public static final int NO_TIME_LIMIT = 0;\r\n\r\n /** Preference key for edge limit on edge combination. */\r\n public static final String EDGE_LIMIT = \"Edge Limit\";\r\n \r\n /** A constant indicating no edge limit on edge combination. */\r\n public static final int NO_EDGE_LIMIT = 0;\r\n\r\n /** Preference key for pruning the number of signs kept per equivalence class. */\r\n public static final String PRUNING_VALUE = \"Pruning Value\";\r\n\r\n /** Preference key for pruning the number of signs kept per cell. */\r\n public static final String CELL_PRUNING_VALUE = \"Cell Pruning Value\";\r\n\r\n /** A constant indicating no pruning of signs per equivalence class. */\r\n public static final int NO_PRUNING = 0;\r\n\r\n /** Preference key for whether to collect category combinations. */\r\n public static final String USE_COMBOS = \"Use Combos\";\r\n \r\n /** Preference key for whether to create a packed generation forest in the first of \r\n two realization stages (deferring unpacking to the later stage). \r\n This option pre-empts the use of collected combos in a single, anytime realization stage. */\r\n public static final String USE_PACKING = \"Use Packing\";\r\n \r\n /** Preference key for whether to unpack a generation forest in the second of \r\n two realization stages. This option is only relevant if packing is \r\n used in the first stage. */\r\n public static final String DO_UNPACKING = \"Do Unpacking\";\r\n\r\n \r\n /** The edge factory for the realization request. */\r\n public final EdgeFactory edgeFactory;\r\n\r\n /** The pruning strategy. */\r\n public final PruningStrategy pruningStrategy;\r\n \r\n /** Flag for whether to use depth-first search. Defaults to false. */\r\n public boolean depthFirst = false; \r\n \r\n /** New best time limit, in ms. Set from prefs in constructor. */\r\n public int newBestTimeLimit = NO_TIME_LIMIT;\r\n \r\n /** New best time limit, as a percentage of time from the first complete realization. Set from prefs in constructor. */\r\n public double newBestTimeLimitPct = NO_TIME_LIMIT;\r\n \r\n /** Edge limit. Set from prefs in constructor. */\r\n public int edgeLimit = NO_EDGE_LIMIT;\r\n \r\n /** Pruning value. Set from prefs in constructor. */\r\n public int pruningValue = NO_PRUNING;\r\n \r\n /** Cell pruning value. Set from prefs in constructor. */\r\n public int cellPruningValue = NO_PRUNING;\r\n \r\n /** Flag for whether to collect category combos. Set from prefs in constructor. */\r\n public boolean collectCombos = true;\r\n\r\n /** Flag for whether to use packing. Set from prefs in constructor. */\r\n public boolean usePacking = false;\r\n\r\n /** Flag for whether to do unpacking. Set from prefs in constructor. */\r\n public boolean doUnpacking = true;\r\n \r\n /** Flag for whether to join best fragments if no complete realization found. Defaults to false. */\r\n public boolean joinFragments = false;\r\n\r\n /** Flag for whether to glue fragments currently. Defaults to false. */\r\n public boolean gluingFragments = false;\r\n\r\n \r\n // the agenda of edges that have yet to be added to the chart\r\n private List agenda = new ArrayList();\r\n \r\n // the (representative) edges in the chart\r\n private List edges = new ArrayList();\r\n \r\n // all unpruned (and unpacked, if apropos) edges in the chart\r\n private List allEdges = new ArrayList();\r\n\r\n // edges to be removed from the chart, after having been superceded \r\n // by an edge with an equivalent sign (up to surface words) and \r\n // a less complex derivation\r\n private List supercededEdgesPendingRemoval = new ArrayList();\r\n\r\n // maps signs to edges (w/o optional bits marked as covered)\r\n private Map signMap = new IdentityHashMap();\r\n \r\n // the edges seen so far\r\n private EdgeHash edgeHash = new EdgeHash();\r\n \r\n\r\n // maps edges to representative edges, according to their \r\n // coverage vectors and their cats, sans LFs\r\n @SuppressWarnings(\"unchecked\")\r\n private Map catMap = new THashMap(\r\n new TObjectHashingStrategy() {\r\n\t\t\tprivate static final long serialVersionUID = 1L;\r\n\t\t\tpublic int computeHashCode(Object o) {\r\n Edge edge = (Edge) o;\r\n return edge.bitset.hashCode() + edge.sign.getCategory().hashCodeNoLF();\r\n }\r\n public boolean equals(Object o1, Object o2) {\r\n Edge edge1 = (Edge) o1; Edge edge2 = (Edge) o2;\r\n return edge1.bitset.equals(edge2.bitset) &&\r\n edge1.sign.getCategory().equalsNoLF(edge2.sign.getCategory());\r\n }\r\n }\r\n );\r\n \r\n // cell map: based on input coverage vectors\r\n private Map cellMap = new HashMap();\r\n \r\n // non-empty cells: cells to avoid when gluing fragments\r\n private Set nonEmptyCells = null;\r\n \r\n // reusable bitset for checking non-empty cells\r\n private transient BitSet tmpBitSet = new BitSet();\r\n \r\n /** \r\n * The best edge found so far (or null), \r\n * where a complete edge is always given preference \r\n * to an incomplete one.\r\n */\r\n public Edge bestEdge = null;\r\n \r\n /** The best edge created by joining fragments, if necessary. */\r\n public Edge bestJoinedEdge = null;\r\n \r\n /** Whether the realization search has been completed. */\r\n public boolean done = false;\r\n \r\n /** The number of nominals in the input LF. */\r\n public int numNominals = 0;\r\n \r\n /** The number of elementary predications in the input LF. */\r\n public int numPreds = 0;\r\n \r\n /** The number of edges created and added to the agenda. */\r\n public int numEdges = 0;\r\n \r\n /** The number of pruned edges removed from the chart. */\r\n public int numPrunedRemoved = 0;\r\n \r\n /** The number of pruned edges never added to the chart. */\r\n public int numPrunedNeverAdded = 0;\r\n \r\n /** The number of new complete best edges found after the first one. */\r\n public int newBest = 0;\r\n \r\n /** The maximum number of edges in a cell. */\r\n public int cellMax = 0;\r\n \r\n /** The time at which realization started. */\r\n protected long startTime = System.currentTimeMillis();\r\n /** The time in ms until lex lookup was completed. */\r\n public int timeTilLex = 0;\r\n /** The time in ms until the first complete edge was found. */\r\n public int timeTilFirst = 0;\r\n /** The time in ms until the best edge was found. */\r\n public int timeTilBest = 0;\r\n /** The time in ms until the search was stopped. */\r\n public int timeTilStopped = 0;\r\n /** The time in ms until the packed chart was completed. */\r\n public int timeTilPacked = 0;\r\n /** The time in ms until the search was finished. */\r\n public int timeTilDone = 0;\r\n \r\n \r\n /** \r\n * Constructor with explicit pruning strategy. \r\n */\r\n // * NB: Even with a non-default pruning strategy, it could potentially help \r\n // * to set the pruning value to an estimate of the number of \r\n // * edges per equivalent category that will be stored.\r\n public Chart(EdgeFactory edgeFactory, PruningStrategy pruningStrategy) {\r\n this.edgeFactory = edgeFactory;\r\n this.pruningStrategy = pruningStrategy;\r\n Preferences prefs = Preferences.userNodeForPackage(TextCCG.class);\r\n newBestTimeLimitPct = prefs.getDouble(NEW_BEST_TIME_LIMIT, NO_TIME_LIMIT);\r\n if (newBestTimeLimitPct >= 1) {\r\n newBestTimeLimit = (int) newBestTimeLimitPct;\r\n newBestTimeLimitPct = NO_TIME_LIMIT; \r\n }\r\n edgeLimit = prefs.getInt(EDGE_LIMIT, NO_EDGE_LIMIT);\r\n pruningValue = prefs.getInt(PRUNING_VALUE, NO_PRUNING);\r\n cellPruningValue = prefs.getInt(CELL_PRUNING_VALUE, NO_PRUNING);\r\n usePacking = prefs.getBoolean(USE_PACKING, false); \r\n collectCombos = !usePacking && prefs.getBoolean(USE_COMBOS, true);\r\n doUnpacking = usePacking && prefs.getBoolean(DO_UNPACKING, true);\r\n }\r\n \r\n \r\n /** Returns the number of (representative) edges in the chart. */\r\n public int numEdgesInChart() { return edges.size(); }\r\n\r\n /** Returns the number of unpruned (and unpacked, if apropos) edges in the chart. */\r\n public int numUnprunedEdges() { return allEdges.size(); }\r\n\r\n \r\n //-----------------------------------------------------------------\r\n // main algorithm routines \r\n \r\n /** Initializes the agenda. */\r\n public void initialize() {\r\n // record number of nominals\r\n numNominals = edgeFactory.nominals.size();\r\n numPreds = edgeFactory.preds.size();\r\n // create various initial edges and add to the agenda\r\n for (Edge edge : edgeFactory.createInitialEdges()) \r\n addEdgeToAgenda(edge);\r\n // record time 'til lex\r\n long currentTime = System.currentTimeMillis();\r\n timeTilLex = (int) (currentTime - startTime);\r\n }\r\n \r\n /** Returns whether there were no uncovered lexical or featural preds after lex lookup. */\r\n public boolean noUncoveredPreds() { return !edgeFactory.hasUncoveredPreds; }\r\n \r\n \r\n /** \r\n * Reinitializes the agenda for gluing fragments. \r\n * A runtime exception is thrown if not in packing mode.\r\n */\r\n public void reInitForGluing() {\r\n \t// check packing mode\r\n \tif (!usePacking) throw new RuntimeException(\"Packing mode required for gluing fragments.\");\r\n \t// set flags here and in edge factory\r\n \tgluingFragments = true; \r\n \tedgeFactory.gluingFragments = true; edgeFactory.useIndexing = false;\r\n \t// add opt for uncovered preds, unless already done for relaxed relation matching\r\n \tif (!edgeFactory.useRelaxedRelationMatching)\r\n \t\tedgeFactory.addLFOptsForUncoveredPreds();\r\n \t// add opts for rule instances\r\n \tedgeFactory.addLFOptsForRuleInstances();\r\n \t// record non-empty cells\r\n \tnonEmptyCells = new HashSet(cellMap.keySet());\r\n \t// add edges back to agenda, for possible gluing\r\n \tfor (Edge edge : edges) addEdgeToAgenda(edge);\r\n }\r\n \r\n \r\n /** \r\n * Adds to the chart by iteratively moving an edge from the agenda to the chart, \r\n * creating new edges for the agenda by applying available rules, \r\n * while updating the best edge found so far, \r\n * until a stopping criterion is reached.\r\n * The basic stopping criterion is when the agenda becomes empty, and thus the \r\n * search is done.\r\n * Otherwise, the search is stopped either when the edge limit (if any) is reached, \r\n * or the time limit (if any) is reached, \r\n * or the first complete edge is found (if beyond the edge/time limit, and according to the given flag), \r\n * or until the new best time limit (if any; anytime case only) beyond the first \r\n * complete realization is exceeded.\r\n * In the packing case, unpacking is then performed according to the preference setting.\r\n * In the anytime case, if the collect combos option is set, then the combinatory rules \r\n * are only invoked when an edge with a new category is moved to the chart, in which case \r\n * any successful combinations are collected in the edge's combos data structure; \r\n * if the edge instead has an already seen category, new edges are created as \r\n * alternatives to the collected combos in its representative, much as with unpacking.\r\n */\r\n public void combine(int timeLimitMS, boolean waitForCompleteEdge) {\r\n \r\n // until agenda is empty\r\n while (!agenda.isEmpty()) {\r\n\r\n // check for timeout\r\n long currentTime = System.currentTimeMillis();\r\n int timeSoFar = (int) (currentTime - startTime);\r\n int timeSinceFirst = timeSoFar - timeTilFirst;\r\n boolean bestEdgeComplete = (bestEdge != null && bestEdge.complete()); \r\n if (\r\n \t// edge limit case\r\n \t(edgeLimit != NO_EDGE_LIMIT && numEdges > edgeLimit &&\r\n (!waitForCompleteEdge || bestEdgeComplete))\r\n \t||\r\n // timeout case\r\n (timeLimitMS != NO_TIME_LIMIT && timeSoFar > timeLimitMS &&\r\n (!waitForCompleteEdge || bestEdgeComplete))\r\n || \r\n // new best timeout case (anytime only)\r\n (!usePacking && bestEdgeComplete && \r\n ((newBestTimeLimit != NO_TIME_LIMIT && \r\n timeSinceFirst > newBestTimeLimit) ||\r\n (newBestTimeLimitPct != NO_TIME_LIMIT && \r\n (double) timeSinceFirst / timeTilFirst > newBestTimeLimitPct)))\r\n ) {\r\n // ensure best edge in chart\r\n if (!allEdges.contains(bestEdge)) { addEdgeToChart(bestEdge); }\r\n // set timing\r\n timeTilStopped = timeSoFar;\r\n // stop\r\n break;\r\n }\r\n \r\n // take edge from agenda\r\n Edge next = agenda.remove(0);\r\n \r\n // add edge to chart\r\n boolean actuallyAdded = addEdgeToChart(next);\r\n \r\n // skip if edge didn't survive pruning (anytime case), \r\n // or was folded into an existing edge (packing case)\r\n if (!actuallyAdded) { continue; }\r\n \r\n // otherwise do combos \r\n doEdgeCombos(next);\r\n }\r\n\r\n // set done packing time\r\n if (usePacking) {\r\n long donePackingTime = System.currentTimeMillis();\r\n timeTilPacked = (int) (donePackingTime - startTime);\r\n // do unpacking, if apropos\r\n if (doUnpacking) doUnpacking();\r\n }\r\n \r\n // set done flag, timing\r\n done = agenda.isEmpty();\r\n if (done) {\r\n\t long endTime = System.currentTimeMillis();\r\n\t timeTilDone = (int) (endTime - startTime);\r\n }\r\n \r\n // join best fragments, if nec.\r\n if (joinFragments && !bestEdge.complete()) joinBestFragments(); \r\n }\r\n \r\n // does binary combinations with all edges in the chart and unary ones too; \r\n // when collecting combos, invokes the combinatory rules only when nec.; \r\n // prunes superceded edges before returning\r\n private void doEdgeCombos(Edge next) {\r\n \t// skip semantically null edges when gluing fragments\r\n \tif (gluingFragments && next.bitset.isEmpty()) return;\r\n \t// when collecting combos ...\r\n if (collectCombos) {\r\n // existing rep case: just make alt edges from collected combos\r\n Edge nextRep = catMap.get(next);\r\n if (next != nextRep) {\r\n addNewEdges(edgeFactory.createAltEdges(next, nextRep));\r\n // and prune any superceded edges before returning\r\n pruneSupercededEdges();\r\n return;\r\n }\r\n }\r\n // otherwise combine edge with those in chart \r\n List edgesToUse = (usePacking || collectCombos) ? edges : allEdges;\r\n for (Edge edge : edgesToUse) {\r\n if (edge == next) continue; // skip this edge\r\n // skip fragment gluing if semantically null or if result cell non-empty\r\n if (gluingFragments) {\r\n \tif (edge.bitset.isEmpty()) continue;\r\n \ttmpBitSet.clear();\r\n \ttmpBitSet.or(edge.bitset); tmpBitSet.or(next.bitset);\r\n \tif (nonEmptyCells.contains(tmpBitSet)) continue;\r\n }\r\n // add new combos to agenda\r\n addNewEdges(edgeFactory.createNewEdges(edge, next, collectCombos));\r\n }\r\n // combine edge via unary rules and with semantically null edges, \r\n // adding new edges to the agenda\r\n \taddNewEdges(edgeFactory.createNewEdges(next, collectCombos));\r\n // prune any superceded edges before returning\r\n pruneSupercededEdges();\r\n }\r\n\r\n // adds all new edges to the agenda\r\n private void addNewEdges(List newEdges) {\r\n for (Edge newEdge : newEdges) {\r\n addEdgeToAgenda(newEdge);\r\n }\r\n }\r\n \r\n /** Greedily combines best fragments, updating bestJoinedEdge. */\r\n protected void joinBestFragments() {\r\n \t// start with best edge\r\n \tbestJoinedEdge = bestEdge;\r\n \t// greedily find best fragments\r\n \tList fragments = new ArrayList();\r\n \tBitSet bitset = bestEdge.bitset;\r\n \twhile (true) {\r\n \tEdge bestFrag = null;\r\n \tfor (Edge edge : allEdges) bestFrag = chooseBestFrag(bitset, bestFrag, edge);\r\n \tfor (Edge edge : agenda) bestFrag = chooseBestFrag(bitset, bestFrag, edge);\r\n \tif (bestFrag == null) break;\r\n \tfragments.add(bestFrag);\r\n \tbitset = (BitSet) bitset.clone();\r\n \tbitset.or(bestFrag.bitset);\r\n \t}\r\n \t// greedily join\r\n \twhile (fragments.size() > 0) {\r\n \t\tEdge nextJoinedEdge = null; Edge nextFrag = null;\r\n\t \tfor (Edge edge : fragments) {\r\n\t \t\tEdge joinedEdge = edgeFactory.makeJoinedEdge(bestJoinedEdge, edge);\r\n\t \t\tif (nextJoinedEdge == null || nextJoinedEdge.score < joinedEdge.score) {\r\n\t \t\t\tnextJoinedEdge = joinedEdge; nextFrag = edge;\r\n\t \t\t}\r\n\t \t\tEdge joinedEdgeR = edgeFactory.makeJoinedEdge(edge, bestJoinedEdge);\r\n\t \t\tif (nextJoinedEdge.score < joinedEdgeR.score) {\r\n\t \t\t\tnextJoinedEdge = joinedEdgeR; nextFrag = edge;\r\n\t \t\t}\r\n\t \t}\r\n\t \tbestJoinedEdge = nextJoinedEdge;\r\n\t \tfragments.remove(nextFrag);\r\n \t}\r\n }\r\n \r\n // returns edge as the new best frag if it doesn't intersect bitset \r\n // and has a better completeness or better score with same completeness; \r\n // otherwise returns bestFrag\r\n private Edge chooseBestFrag(BitSet bitset, Edge bestFrag, Edge edge) {\r\n\t\tif (edge.bitset.isEmpty() || edge.bitset.intersects(bitset)) return bestFrag;\r\n if (bestFrag == null) return edge;\r\n if (bestFrag.completeness < edge.completeness) return edge;\r\n if (bestFrag.completeness == edge.completeness && bestFrag.score < edge.score) return edge;\r\n return bestFrag; \r\n }\r\n \r\n \r\n \r\n //-----------------------------------------------------------------\r\n // unpacking \r\n \r\n /** Unpack complete edges, if any; otherwise unpack all. */\r\n\tprotected void doUnpacking() {\r\n\t @SuppressWarnings(\"unchecked\")\r\n Set unpacked = new THashSet(new TObjectIdentityHashingStrategy());\r\n\t boolean foundComplete = bestEdge.complete();\r\n // unpack each relevant edge, updating best edge \r\n for (Edge edge : edges) {\r\n if (foundComplete && !edge.complete()) continue;\r\n unpack(edge, unpacked);\r\n updateBestEdge(edge.altEdges.get(0));\r\n }\r\n }\r\n \r\n // recursively unpack and prune edge, unless already visited\r\n private void unpack(Edge edge, Set unpacked) {\r\n if (unpacked.contains(edge)) return;\r\n // add to unpacked set\r\n unpacked.add(edge);\r\n // OR: recursively unpack alts, merging resulting alts\r\n EdgeHash merged = new EdgeHash();\r\n\t if (edge.altEdges == null) {\r\n\t \tthrow new RuntimeException(\"No alts for: \" + edge);\r\n\t }\r\n for (Edge alt : edge.altEdges) {\r\n // AND: unpack inputs, make alts, add to merged\r\n unpackAlt(alt, unpacked, merged);\r\n }\r\n // sort, rescore and prune\r\n List mergedList = new ArrayList(merged.asEdgeSet());\r\n Collections.sort(mergedList, edgeComparator);\r\n List prunedEdges = pruningStrategy.pruneEdges(mergedList);\r\n numPrunedNeverAdded += prunedEdges.size();\r\n // replace edge's alts, add to unpruned edges\r\n edge.altEdges.clear(); edge.altEdges.addAll(mergedList);\r\n allEdges.addAll(mergedList);\r\n // update signMap (for debugging)\r\n for (Edge mergedEdge : mergedList) {\r\n \tif (!signMap.containsKey(mergedEdge.sign))\r\n \t\tsignMap.put(mergedEdge.sign, mergedEdge);\r\n }\r\n }\r\n \r\n // recursively unpack inputs, make alt combos and add to merged\r\n private void unpackAlt(Edge alt, Set unpacked, EdgeHash merged) {\r\n // first check for opt completed edge\r\n if (alt.optCompletes != null) {\r\n // recursively unpack input edge\r\n Edge inputEdge = alt.optCompletes;\r\n unpack(inputEdge, unpacked);\r\n // then make and merge alt edges from input alt edges\r\n for (Edge inputAlt : inputEdge.altEdges) {\r\n Edge edgeToAdd = (inputAlt.sign == alt.sign)\r\n ? alt // use this alt for same sign\r\n : edgeFactory.makeAltEdge(inputAlt.sign, alt); // otherwise make edge for new alt\r\n merged.insert(edgeToAdd);\r\n }\r\n return;\r\n }\r\n // otherwise unpack via input signs\r\n DerivationHistory history = alt.sign.getDerivationHistory(); \r\n Sign[] inputSigns = history.getInputs();\r\n // base case: no inputs\r\n if (inputSigns == null) {\r\n merged.insert(alt); return;\r\n }\r\n // otherwise recursively unpack\r\n Edge[] inputEdges = new Edge[inputSigns.length];\r\n for (int i = 0; i < inputSigns.length; i++) {\r\n inputEdges[i] = signMap.get(inputSigns[i]); // get input edge using signMap\r\n unpack(inputEdges[i], unpacked);\r\n }\r\n // then make edges for new combos, and add to merged (if unseen)\r\n Category resultCat = alt.sign.getCategory();\r\n boolean lefthead = (alt.sign.getLexHead() == inputSigns[0].getLexHead());\r\n List altCombos = inputCombos(inputEdges, 0);\r\n for (Sign[] combo : altCombos) {\r\n \tSign lexHead = (lefthead) ? combo[0].getLexHead() : combo[1].getLexHead();\r\n Sign sign = Sign.createDerivedSignWithNewLF(resultCat, combo, history.getRule(), lexHead);\r\n Edge edgeToAdd = (sign.equals(alt.sign))\r\n ? alt // use this alt for equiv sign\r\n : edgeFactory.makeAltEdge(sign, alt); // otherwise make edge for new alt\r\n merged.insert(edgeToAdd);\r\n }\r\n }\r\n\r\n // returns a list of sign arrays, with each array of length inputEdges.length - i, \r\n // representing all combinations of alt signs from i onwards\r\n private List inputCombos(Edge[] inputEdges, int index) {\r\n Edge edge = inputEdges[index];\r\n // base case, inputEdges[last]\r\n if (index == inputEdges.length-1) {\r\n List altEdges = edge.altEdges; \r\n List retval = new ArrayList(altEdges.size());\r\n for (Edge alt : altEdges) {\r\n retval.add(new Sign[] { alt.sign });\r\n }\r\n return retval;\r\n }\r\n // otherwise recurse on index+1\r\n List nextCombos = inputCombos(inputEdges, index+1);\r\n // and make new combos\r\n List altEdges = edge.altEdges; \r\n List retval = new ArrayList(altEdges.size() * nextCombos.size());\r\n for (Edge alt : altEdges) {\r\n for (int i = 0; i < nextCombos.size(); i++) {\r\n Sign[] nextSigns = nextCombos.get(i);\r\n Sign[] newCombo = new Sign[nextSigns.length+1];\r\n newCombo[0] = alt.sign;\r\n System.arraycopy(nextSigns, 0, newCombo, 1, nextSigns.length);\r\n retval.add(newCombo);\r\n }\r\n }\r\n return retval;\r\n }\r\n \r\n \r\n //-----------------------------------------------------------------\r\n // best edges (single best is available directly as bestEdge)\r\n \r\n // cached best edges\r\n private transient List bestEdges = null;\r\n \r\n /** \r\n * Returns the best complete edges, sorted by their score and \r\n * pruned by the pruning strategy. \r\n */\r\n public List bestEdges() {\r\n \tif (bestEdges != null) return bestEdges;\r\n bestEdges = new ArrayList();\r\n if (!bestEdge.complete()) return bestEdges;\r\n List edgesToUse = (usePacking && !doUnpacking) ? edges : allEdges;\r\n for (Edge edge : edgesToUse) {\r\n if (edge.complete()) bestEdges.add(edge); \r\n }\r\n Collections.sort(bestEdges, edgeComparator);\r\n pruningStrategy.pruneEdges(bestEdges);\r\n return bestEdges;\r\n }\r\n \r\n /** \r\n * Returns the oracle best edge for the given string, together \r\n * with a flag indicating whether the target string was found.\r\n * At present, returns the edge from bestEdges() whose \r\n * orthography equals the given string, otherwise the one \r\n * with the highest 4-gram precision score (breaking ties \r\n * using the list order).\r\n * TODO: It would be better to return the forest oracle. \r\n */\r\n public Pair oracleBest(String target) {\r\n \tList edges = bestEdges();\r\n \t// check orth\r\n \tfor (Edge edge : edges) {\r\n \t\tif (edge.getSign().getOrthography().equals(target)) \r\n \t\t\treturn new Pair(edge, true); \r\n \t}\r\n \t// check 4-gram precision\r\n \tEdge retval = null;\r\n \tdouble bestScore = 0;\r\n \tNgramPrecisionModel oracle = new NgramPrecisionModel(new String[]{target});\r\n \tfor (Edge edge : edges) {\r\n \t\tdouble score = oracle.score(edge.getSign(), true);\r\n \t\tif (score > bestScore) {\r\n \t\t\tretval = edge; bestScore = score;\r\n \t\t}\r\n \t}\r\n \t// done\r\n \treturn new Pair(retval, false);\r\n }\r\n \r\n \r\n //-----------------------------------------------------------------\r\n // printing routines\r\n \r\n /** The PrintWriter to use with the printing routines. Default wraps System.out. */\r\n public PrintWriter out = new PrintWriter(System.out);\r\n\r\n /** Prints the best edge found. */\r\n public void printBestEdge() {\r\n printEdge(bestEdge);\r\n if (!edgeFactory.labeledNominals.isEmpty()) {\r\n try {\r\n ByteArrayOutputStream bstr = new ByteArrayOutputStream();\r\n edgeFactory.grammar.serializeXml(\r\n bestEdge.sign.getWordsInXml(edgeFactory.labeledNominals), bstr\r\n );\r\n out.println(bstr.toString());\r\n }\r\n catch (java.io.IOException exc) { \r\n throw (RuntimeException) new RuntimeException().initCause(exc);\r\n }\r\n }\r\n out.println(bestEdge.sign.getBracketedString());\r\n out.flush();\r\n }\r\n \r\n /** Prints the best joined edge. */\r\n public void printBestJoinedEdge() {\r\n \tif (bestJoinedEdge == null) return;\r\n printEdge(bestJoinedEdge);\r\n out.println(bestJoinedEdge.sign.getBracketedString());\r\n out.flush();\r\n }\r\n \r\n /** Prints the timing (and related) info. */\r\n public void printTiming() {\r\n out.println();\r\n if (!usePacking) {\r\n if (bestEdge != null && bestEdge.complete())\r\n out.println(\"time 'til first (ms): \" + timeTilFirst);\r\n if (bestEdge != null)\r\n out.println(\"time 'til best (ms): \" + timeTilBest);\r\n if (timeTilStopped != 0)\r\n out.println(\"time 'til stopped (ms): \" + timeTilStopped);\r\n }\r\n else {\r\n out.println(\"time 'til packed (ms): \" + timeTilPacked); \r\n }\r\n if (timeTilDone != 0)\r\n out.println(\"time 'til done (ms): \" + timeTilDone); \r\n out.println();\r\n out.println(\"rule apps: \" + edgeFactory.ruleApps());\r\n out.println(\"# edges: \" + edges.size());\r\n out.println(\"# unpruned edges: \" + allEdges.size());\r\n if (!usePacking) {\r\n out.println(\"# pruned: \" + numPrunedRemoved + \" removed, \" + numPrunedNeverAdded + \" never added\");\r\n }\r\n if (doUnpacking) {\r\n out.println(\"# pruned: \" + numPrunedNeverAdded);\r\n }\r\n out.println(\"cell max: \" + cellMax);\r\n out.flush();\r\n }\r\n \r\n /** Prints all chart edges, unsorted. */\r\n public void printEdges() { printEdges(false); }\r\n \r\n /** Prints chart edges unsorted, using the complete edges filter according to the given flag. */ \r\n public void printEdges(boolean complete) { printEdges(complete, false); }\r\n \r\n /** \r\n * Prints chart edges using the complete edges filter according to the given flag \r\n * and sorting according to the given flag.\r\n * In the packing only case, the representative edges are shown, otherwise \r\n * the unpruned (and possibly unpacked) edges are shown.\r\n */ \r\n public void printEdges(boolean complete, boolean sort) {\r\n List edgeList = (usePacking && !doUnpacking) ? edges : allEdges;\r\n if (sort) {\r\n edgeList = new ArrayList(edgeList);\r\n Collections.sort(edgeList, edgeComparator);\r\n }\r\n for (int i=0; i < edgeList.size(); i++) { \r\n \tEdge edge = edgeList.get(i);\r\n if (!complete || edge.complete()) {\r\n \tif (!sort) printEdge(edge, i, edgeList);\r\n \telse printEdge(edge);\r\n }\r\n \r\n }\r\n out.flush();\r\n }\r\n\r\n /**\r\n * Prints the agenda.\r\n */\r\n public void printAgenda() {\r\n for (Edge edge : agenda) {\r\n printEdge(edge);\r\n }\r\n out.flush();\r\n }\r\n \r\n /**\r\n * Prints the initial edges.\r\n */\r\n public void printInitialEdges() {\r\n for (Edge edge : edgeFactory.initialEdges) {\r\n printEdge(edge);\r\n }\r\n out.flush();\r\n }\r\n \r\n // prints edge with incomplete LF chunks and active alts\r\n private void printEdge(Edge edge) { printEdge(edge, -1, null); }\r\n \r\n // prints also with edge index and derivation, if index non-negative\r\n private void printEdge(Edge edge, int index, List edgeList) {\r\n String str = \"\";\r\n if (index >= 0) str += index + \". \";\r\n str += edge.toString();\r\n if (edge.incompleteLfChunk != null) {\r\n int id = edgeFactory.lfChunks.indexOf(edge.incompleteLfChunk);\r\n str += \" <[\" + id + \"]>\";\r\n }\r\n if (edge.activeLfAlts.size() > 0) str += \" \";\r\n for (List altSet : edge.activeLfAlts) {\r\n for (Alt alt : altSet) str += \"?\" + alt.altSet + \".\" + alt.numInSet;\r\n }\r\n str += edgeDerivation(edge, index, edgeList);\r\n out.println(str);\r\n // show alts subordinated in packing only case\r\n if (usePacking && !doUnpacking && edge.isDisjunctive()) {\r\n for (Edge alt : edge.altEdges) {\r\n if (alt != edge) \r\n \tout.println(\" \\\\_ \" + alt + edgeDerivation(alt, index, edgeList));\r\n }\r\n }\r\n }\r\n \r\n // returns derivation, if index non-negative\r\n private String edgeDerivation(Edge edge, int index, List edgeList) {\r\n \tif (index < 0) return \"\";\r\n \tif (edge.optCompletes != null) {\r\n \t\treturn \" (\" + edgeList.indexOf(edge.optCompletes) + \" optC)\";\r\n \t}\r\n \tDerivationHistory history = edge.sign.getDerivationHistory();\r\n \tSign[] inputs = history.getInputs();\r\n \tif (inputs == null) return \" (lex)\";\r\n \tString retval = \" (\";\r\n\t\tfor (Sign sign : inputs) {\r\n\t\t\tEdge repEdge = signMap.get(sign);\r\n\t\t\tif (repEdge != null) retval += edgeList.indexOf(repEdge) + \" \";\r\n\t\t}\r\n\t\tretval += history.getRule().name() + \")\";\r\n \treturn retval;\r\n }\r\n\r\n \r\n /**\r\n * Prints the licensed, marked initial edges.\r\n */\r\n public void printMarkedEdges() {\r\n for (Edge edge : edgeFactory.markedEdges) {\r\n printEdge(edge);\r\n }\r\n out.flush();\r\n }\r\n \r\n /**\r\n * Prints the licensed, instantiated purely syntactic (semantically null) edges.\r\n */\r\n public void printInstantiatedNoSemEdges() {\r\n for (Edge edge : edgeFactory.instantiatedNoSemEdges) {\r\n printEdge(edge);\r\n }\r\n out.flush();\r\n }\r\n \r\n /**\r\n * Prints the licensed, uninstantiated purely syntactic (semantically null) edges.\r\n */\r\n public void printNoSemEdges() {\r\n for (Edge edge : edgeFactory.noSemEdges) {\r\n out.println(edge.toString());\r\n }\r\n out.flush();\r\n }\r\n \r\n /**\r\n * Prints the rule instances, with instantiated semantics.\r\n */\r\n public void printRuleInstances() {\r\n for (Iterator it = edgeFactory.ruleInstances.iterator(); it.hasNext(); ) {\r\n out.println(it.next().toString());\r\n }\r\n out.flush();\r\n }\r\n \r\n /**\r\n * Prints the LF chunks.\r\n */\r\n public void printLfChunks() {\r\n List chunks = edgeFactory.lfChunks;\r\n for (int i = 0; i < chunks.size(); i++) {\r\n BitSet chunk = chunks.get(i);\r\n out.println(\"chunk[\" + i + \"]: \" + Edge.toString(chunk));\r\n }\r\n out.flush();\r\n }\r\n \r\n /**\r\n * Prints the LF alternatives.\r\n */\r\n public void printLfAlts() {\r\n for (List altSet : edgeFactory.lfAlts) {\r\n for (Alt alt : altSet) {\r\n out.print(\"alt[\" + alt.altSet + \".\" + alt.numInSet + \"]: \");\r\n out.println(Edge.toString(alt.bitset));\r\n }\r\n }\r\n out.flush();\r\n }\r\n \r\n /**\r\n * Prints the LF optional parts.\r\n */\r\n public void printLfOpts() {\r\n List opts = edgeFactory.lfOpts;\r\n for (int i = 0; i < opts.size(); i++) {\r\n BitSet opt = opts.get(i);\r\n out.println(\"opt[\" + i + \"]: \" + Edge.toString(opt));\r\n }\r\n out.flush();\r\n }\r\n \r\n /**\r\n * Prints the elementary predications.\r\n */\r\n public void printEPs() {\r\n List preds = edgeFactory.preds; \r\n for (int i=0; i < preds.size(); i++) {\r\n SatOp lf_i = preds.get(i);\r\n out.println(\"ep[\" + i + \"]: \" + lf_i);\r\n }\r\n out.flush();\r\n }\r\n \r\n \r\n \r\n //-----------------------------------------------------------------\r\n // chart management\r\n\r\n // in the anytime case, first checks signs to see whether an edge \r\n // whose sign is equivalent (up to surface words) and which has \r\n // an equal or higher score or equal \r\n // or lower derivational complexity has been seen already, and drops \r\n // the given edge if so (in the packing case, this equivalence check is \r\n // performed during unpacking);\r\n // if the edge replaces an (essentially) equivalent edge of lower score or higher \r\n // derivational complexity, removes the old edge from the agenda \r\n // or removes it from its equivalence class and puts it on a list of \r\n // superceded edges to be pruned from the chart;\r\n // then, in all cases, adds the given edge to the agenda, \r\n // and updates the best edge so far, with preference given to completeness, \r\n // then sign score\r\n private void addEdgeToAgenda(Edge edge) {\r\n \tnumEdges++;\r\n \tif (!usePacking) {\r\n\t \t// update edgeHash, checking for equivalent edge of equal or lower complexity\r\n \t\tEdge retEdge = edgeHash.insert(edge);\r\n \t\tboolean actuallyInserted = (retEdge != null);\r\n\t \tif (!actuallyInserted) { return; } // just drop it\r\n\t \t// remove old edge, if apropos\r\n\t \tEdge oldEdge = (retEdge != edge) ? retEdge : null; \r\n\t \tif (oldEdge != null) {\r\n\t \t\t// check agenda first\r\n\t \t\tboolean onAgenda = agenda.remove(oldEdge);\r\n\t \t\t// if not on agenda, remove from equiv class, if present, \r\n\t \t\t// and add to list of superceded edges pending removal\r\n\t \t\t// nb: delaying pruning of superceded edges from chart \r\n\t \t\t// is nec. to avoid a problem with concurrent access \r\n\t \t\t// to allEdges in doEdgeCombos\r\n\t \t\tif (!onAgenda) {\r\n\t \t Edge repEdge = catMap.get(oldEdge);\r\n\t \t if (repEdge != null) {\r\n\t \t \tboolean inChart = repEdge.altEdges.remove(oldEdge);\r\n\t \t \tif (inChart) supercededEdgesPendingRemoval.add(oldEdge);\r\n\t \t }\r\n\t \t\t}\r\n\t \t}\r\n \t}\r\n if (depthFirst) { agenda.add(0, edge); }\r\n else if (edge.score == 0) { agenda.add(edge); }\r\n else { addSorted(agenda, edge); }\r\n updateBestEdge(edge);\r\n }\r\n \r\n // update bestEdge wrt given edge, and adjust timing info\r\n private void updateBestEdge(Edge edge) {\r\n if (bestEdge == null) {\r\n bestEdge = edge; \r\n long endTime = System.currentTimeMillis();\r\n timeTilBest = (int) (endTime - startTime);\r\n if (bestEdge.complete()) timeTilFirst = timeTilBest; \r\n return;\r\n }\r\n if (bestEdge.completeness > edge.completeness) return;\r\n if (bestEdge.completeness < edge.completeness) {\r\n bestEdge = edge; \r\n long endTime = System.currentTimeMillis();\r\n timeTilBest = (int) (endTime - startTime);\r\n if (bestEdge.complete()) timeTilFirst = timeTilBest; \r\n return;\r\n }\r\n if (edge.score > bestEdge.score) {\r\n bestEdge = edge;\r\n long endTime = System.currentTimeMillis();\r\n timeTilBest = (int) (endTime - startTime);\r\n if (bestEdge.complete()) newBest++;\r\n }\r\n }\r\n \r\n // removes superceded edges from the chart\r\n private void pruneSupercededEdges() {\r\n \tfor (Edge oldEdge : supercededEdgesPendingRemoval) {\r\n \t\tallEdges.remove(oldEdge); numPrunedRemoved++;\r\n \t}\r\n \tsupercededEdgesPendingRemoval.clear();\r\n }\r\n \r\n // adds the edge to the chart and makes it a representative edge if it's the \r\n // first one added for its equiv class; otherwise it's added as an alternative;\r\n // in the anytime case, prunes the edges listed as alts for the representative edge, \r\n // and adds the edge to the list of all unpruned edges, if it survives pruning;\r\n // returns true if the edge is actually added, and false if it doesn't survive \r\n // the pruning (anytime case), or is folded into an existing edge (packing case);\r\n // prunes the edge and returns false if the cell count is exceeded\r\n private boolean addEdgeToChart(Edge edge) {\r\n \t// check cell count\r\n \tif (cellPruningValue != NO_PRUNING && cellCount(edge) >= cellPruningValue) {\r\n \t\tnumPrunedNeverAdded++; return false;\r\n \t}\r\n \t// inc cell count\r\n \tincCellCount(edge);\r\n // get representative edge for this edge\r\n Edge repEdge = catMap.get(edge);\r\n // check for same edge already in chart; pretend it's been added\r\n if (edge == repEdge) return true;\r\n // if none, make this edge into one, adding it to the chart\r\n if (repEdge == null) {\r\n edge.initAltEdges(); // nb: could try capacity of pruningValue+1\r\n if (collectCombos) edge.initEdgeCombos();\r\n catMap.put(edge, edge);\r\n edges.add(edge);\r\n \tsignMap.put(edge.sign, edge);\r\n // anytime case: add to all edges list too\r\n if (!usePacking) allEdges.add(edge);\r\n // and return\r\n return true;\r\n }\r\n // otherwise add edge to alts (sorted)\r\n else {\r\n addSorted(repEdge.altEdges, edge);\r\n // packing case: return false, as edge is simply folded into repEdge\r\n if (usePacking) return false;\r\n }\r\n // anytime case: if not pruning, just add edge to all edges list, and return\r\n if (pruningValue == NO_PRUNING) {\r\n allEdges.add(edge);\r\n \tsignMap.put(edge.sign, edge); // for debugging\r\n return true;\r\n }\r\n // otherwise do pruning\r\n List prunedEdges = pruningStrategy.pruneEdges(repEdge.altEdges);\r\n boolean edgeItselfPruned = false;\r\n for (Edge prunedEdge : prunedEdges) {\r\n if (prunedEdge != edge) {\r\n allEdges.remove(prunedEdge);\r\n numPrunedRemoved++;\r\n }\r\n else edgeItselfPruned = true;\r\n }\r\n // add edge to all edges list, if it was not pruned\r\n if (!edgeItselfPruned) {\r\n allEdges.add(edge);\r\n \tsignMap.put(edge.sign, edge); // for debugging\r\n return true;\r\n }\r\n // otherwise false\r\n numPrunedNeverAdded++;\r\n return false;\r\n }\r\n \r\n // cell count\r\n private int cellCount(Edge edge) {\r\n \tInteger count = cellMap.get(edge.bitset);\r\n \treturn (count == null) ? 0 : count;\r\n }\r\n\r\n // inc cell count\r\n private void incCellCount(Edge edge) {\r\n \tint count = cellCount(edge);\r\n \tcellMap.put(edge.bitset, ++count);\r\n \tif (count > cellMax) cellMax = count;\r\n }\r\n \r\n \r\n //-----------------------------------------------------------------\r\n // edge sorted insertion and comparison\r\n \r\n // adds the given edge into the already sorted list, \r\n // maintaining the sort order;\r\n // when gluing fragments, edges are sorted first by size, \r\n // otherwise by score\r\n private void addSorted(List list, Edge edge) {\r\n \tComparator comparator = (gluingFragments) ? edgeSizeComparator : edgeComparator; \r\n // do binary search\r\n int index = Collections.binarySearch(list, edge, comparator);\r\n // check if search found an edge with the same sort pos\r\n if (index >= 0) {\r\n // if so, advance the index past sort equiv edges\r\n while (index < list.size()) {\r\n Edge existingEdge = list.get(index);\r\n //if (existingEdge.score == edge.score) index++;\r\n if (comparator.compare(existingEdge, edge) == 0) index++;\r\n else break;\r\n }\r\n }\r\n else {\r\n // otherwise, convert index to insertion point\r\n index = Math.abs(index) - 1;\r\n }\r\n // then add edge at index\r\n list.add(index, edge); \r\n }\r\n\r\n /** Compares edges based on their relative score, in descending order. */\r\n public static final Comparator edgeComparator = new Comparator() {\r\n public int compare(Edge edge1, Edge edge2) {\r\n return -1 * Double.compare(edge1.score, edge2.score);\r\n }\r\n };\r\n\r\n /** Compares edges based on their relative size then score, in descending order. */\r\n public static final Comparator edgeSizeComparator = new Comparator() {\r\n public int compare(Edge edge1, Edge edge2) {\r\n \tint retval = -1 * Float.compare(edge1.completeness, edge2.completeness);\r\n \tif (retval != 0) return retval;\r\n return -1 * Double.compare(edge1.score, edge2.score);\r\n }\r\n };\r\n}\r\n \r\n" }, { "path": "src/opennlp/ccg/realize/DiversityPruningStrategy.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004-5 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.realize;\n\nimport opennlp.ccg.synsem.Sign;\n\nimport java.util.*;\n\nimport gnu.trove.*;\n\n/**\n * Abstract n-best edge pruning strategy that keeps edges diversified \n * according to the notCompellinglyDifferent equivalence relation. \n * The edges are clustered into a ranked list of equivalence classes, \n * which are sequentially sampled until the limit n is reached to determine \n * the edges to keep.\n * If the singleBestPerGroup flag is set, then a maximum of one edge \n * per equivalence class is retained.\n *\n * @author Michael White\n * @version $Revision: 1.4 $, $Date: 2009/12/21 03:27:18 $\n */\nabstract public class DiversityPruningStrategy extends NBestPruningStrategy\n{\n /** Flag for whether to keep only the single best edge among those that \n are not compellingly different (defaults to false). */\n public boolean singleBestPerGroup = false;\n \n /** Reusable set of edges to keep. */\n protected THashSet keepers = new THashSet(new TObjectIdentityHashingStrategy());\n \n /** Returns true iff the given signs are not compellingly different. */\n abstract public boolean notCompellinglyDifferent(Sign sign1, Sign sign2);\n \n /**\n * Returns a (possibly empty) list of edges pruned \n * from the given ones, which should have equivalent\n * categories and be sorted by score, from highest to lowest. \n * In particular, prunes and returns the edges that fall below the N-best \n * cutoff when the diversity strategy determined by notCompellinglyDifferent \n * is applied.\n * If the singleBestPerGroup flag is set, no more than one edge \n * per group of equivalent ones will be returned.\n */\n public List pruneEdges(List catEdges) {\n // clear reusable return list\n retval.clear();\n // ensure pruning enabled\n if (CAT_PRUNE_VAL == Chart.NO_PRUNING) return retval;\n // ensure there are edges to prune\n if (!singleBestPerGroup && catEdges.size() <= CAT_PRUNE_VAL) return retval;\n // group edges into ranked equivalence classes, \n // by using a list of lists, preserving order\n List> groups = new ArrayList>();\n for (Edge edge : catEdges) {\n boolean foundGroup = false;\n for (int i = 0; i < groups.size(); i++) {\n List members = groups.get(i);\n Edge first = members.get(0);\n if (notCompellinglyDifferent(first.sign, edge.sign)) {\n members.add(edge); \n foundGroup = true; break;\n }\n }\n if (!foundGroup) {\n List members = new ArrayList();\n members.add(edge);\n groups.add(members);\n }\n }\n // add top n to keepers by sequentially visiting groups, \n // according also to singleBestPerGroup flag\n keepers.clear();\n int counter = 0;\n int numGroups = groups.size();\n while (keepers.size() < CAT_PRUNE_VAL && \n (!singleBestPerGroup || counter < numGroups)) \n {\n int groupNum = counter % numGroups;\n int indexInGroup = counter / numGroups;\n List members = groups.get(groupNum);\n if (indexInGroup < members.size()) { \n keepers.add(members.get(indexInGroup));\n }\n counter++;\n }\n // prune edges not in keepers\n for (Iterator it = catEdges.iterator(); it.hasNext(); ) {\n Edge edge = it.next();\n if (!keepers.contains(edge)) {\n retval.add(edge);\n it.remove();\n }\n }\n // done\n return retval;\n }\n}\n\n" }, { "path": "src/opennlp/ccg/realize/Edge.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2003-5 University of Edinburgh (Michael White)\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.realize;\r\n\r\nimport opennlp.ccg.synsem.*;\r\nimport opennlp.ccg.hylo.*;\r\nimport opennlp.ccg.*;\r\nimport java.util.*;\r\nimport java.util.prefs.*;\r\nimport java.text.*;\r\n\r\n/**\r\n *

    \r\n * An edge is a tracker for a sign, ie a sign together \r\n * with bitsets representing its coverage of the \r\n * input predicates and the semantic indices used, \r\n * along with lists of the active LF alts.\r\n * It also has a completeness percentage and a score, \r\n * as well as its most specific incomplete LF chunk (if any).\r\n * Edges are created by an EdgeFactory and managed by a Chart.\r\n *

    \r\n *

    \r\n * A representative edge is an edge that represents (stands in for) \r\n * other edges with the same category during the chart generation\r\n * process. A representative edge has a list of alternative edges, \r\n * which are assumed to share the same category; it is considered \r\n * disjunctive when there is more than one alternative.\r\n * Note that initially a representative edge will be in its list \r\n * of alternatives, but it can be removed during pruning.\r\n * Finally, for the anytime search, a representative edge can \r\n * maintain a collection of successful combinations, to avoid \r\n * invoking the combinatory rules multiple times with the same \r\n * combinations of categories.\r\n *

    \r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.32 $, $Date: 2010/08/10 04:10:15 $\r\n */\r\npublic class Edge extends Tracker\r\n{\r\n\r\n /** Preference key for showing completeness. */\r\n public static final String SHOW_COMPLETENESS = \"Show Completeness\";\r\n\r\n /** Preference key for showing coverage bitset. */\r\n public static final String SHOW_BITSET = \"Show Bitset\";\r\n\r\n \r\n /** The sign. */\r\n protected Sign sign;\r\n \r\n /** The completeness percentage. */\r\n public final float completeness;\r\n \r\n /** The edge score. */\r\n public final double score;\r\n \r\n /** The most specific incomplete LF chunk (if any). */\r\n public final BitSet incompleteLfChunk;\r\n \r\n /** The alternative edges (none initially). */\r\n protected List altEdges = null;\r\n \r\n /** The edge combos (none initially). */\r\n protected EdgeCombos edgeCombos = null;\r\n \r\n /** The edge, if any, that this edge is constructed from \r\n by marking optional bits as completed. */\r\n protected Edge optCompletes = null;\r\n \r\n \r\n /** Constructor. */\r\n public Edge(Sign sign, BitSet bitset, BitSet indices, \r\n float completeness, double score, \r\n List> activeLfAlts, BitSet incompleteLfChunk) \r\n {\r\n super(bitset, indices, activeLfAlts);\r\n this.sign = sign;\r\n this.completeness = completeness; this.score = score; \r\n this.incompleteLfChunk = incompleteLfChunk;\r\n }\r\n\r\n \r\n /** Returns the sign. */\r\n public Sign getSign() { return sign; }\r\n \r\n /** Returns whether this edge has completeness 1.0, ie, covers all the input preds. */\r\n public boolean complete() {\r\n return (completeness == 1.0);\r\n }\r\n \r\n /**\r\n * Returns the nominal which is the value of the index feature on the \r\n * sign's target cat, or null if none.\r\n */\r\n public Nominal getIndexNominal() { return sign.getCategory().getIndexNominal(); }\r\n \r\n\r\n /**\r\n * Returns true iff this edge can combine with the given tracker \r\n * without violating its LF chunk constraint (if any).\r\n * Specifically, returns true when this edge has no incomplete chunk \r\n * or the tracker is semantically empty; otherwise, returns true \r\n * iff the incomplete chunk intersects with the tracker's \r\n * coverage vector.\r\n */\r\n public boolean meetsLfChunkConstraints(Tracker tracker) {\r\n if (incompleteLfChunk == null || tracker.bitset.isEmpty()) return true;\r\n return incompleteLfChunk.intersects(tracker.bitset);\r\n }\r\n \r\n\r\n /** Returns whether this edge is a representative. */\r\n public boolean isRepresentative() { return altEdges != null; }\r\n \r\n /** Returns whether this edge is disjunctive. */\r\n public boolean isDisjunctive() { return altEdges != null && altEdges.size() > 1; }\r\n \r\n /** Returns the list of alt edges, or the empty list if none. */\r\n public List getAltEdges() {\r\n if (altEdges == null) return Collections.emptyList(); \r\n else return altEdges;\r\n }\r\n \r\n /** Initializes the alt edges list with a default capacity, adding this edge. */\r\n public void initAltEdges() { initAltEdges(3); }\r\n \r\n /** Initializes the alt edges list with the given capacity, adding this edge. */\r\n public void initAltEdges(int capacity) {\r\n // check uninitialized\r\n if (altEdges != null) throw new RuntimeException(\"Alt edges already initialized!\");\r\n altEdges = new ArrayList(capacity);\r\n altEdges.add(this);\r\n }\r\n \r\n \r\n /** Initializes the edge combos. */\r\n public void initEdgeCombos() {\r\n // check representative status\r\n if (!isRepresentative()) throw new RuntimeException(\"Not a representative!\");\r\n if (edgeCombos != null) throw new RuntimeException(\"Edge combos already initialized!\");\r\n edgeCombos = new EdgeCombos();\r\n }\r\n \r\n \r\n /** Returns a hash code for this edge. (Alternatives are not considered.) */\r\n public int hashCode() {\r\n int retval = sign.hashCode() + 31 * bitset.hashCode() + indices.hashCode();\r\n retval += (int) (31000 * score);\r\n return retval;\r\n }\r\n \r\n /** \r\n * Returns a hash code for this edge based on the surface words, \r\n * ignoring the LF and ignoring the score. \r\n */\r\n public int surfaceWordHashCode() {\r\n \treturn sign.surfaceWordHashCode(true) + 31 * bitset.hashCode() + indices.hashCode();\r\n }\r\n \r\n /** Returns whether this edge equals the given object. (Alternatives are not considered.) */\r\n public boolean equals(Object obj) {\r\n if (obj == this) return true;\r\n if (!(obj instanceof Edge)) return false;\r\n Edge edge = (Edge) obj;\r\n return bitset.equals(edge.bitset) && indices.equals(edge.indices) && \r\n \tcompleteness == edge.completeness && score == edge.score &&\r\n \tsign.equals(edge.sign);\r\n }\r\n \r\n /** \r\n * Returns whether this edge equals the given object based on the surface words, \r\n * ignoring the LF and ignoring the score. \r\n */\r\n public boolean surfaceWordEquals(Object obj) {\r\n if (obj == this) return true;\r\n if (!(obj instanceof Edge)) return false;\r\n Edge edge = (Edge) obj;\r\n return bitset.equals(edge.bitset) && indices.equals(edge.indices) &&\r\n sign.surfaceWordEquals(edge.sign, true);\r\n }\r\n \r\n \r\n /**\r\n * Returns a string for the edge in the format\r\n * {completeness} [score] orthography :- category {bitset}. \r\n */\r\n public String toString() {\r\n Preferences prefs = Preferences.userNodeForPackage(TextCCG.class);\r\n boolean showCompleteness = prefs.getBoolean(SHOW_COMPLETENESS, false);\r\n boolean showBitset = prefs.getBoolean(SHOW_BITSET, false);\r\n StringBuffer sbuf = new StringBuffer();\r\n //sbuf.append(indices + \" \");\r\n if (showCompleteness) { sbuf.append(\"{\" + nf2.format(completeness) + \"} \"); }\r\n if (score >= 0.001 || score == 0.0) {\r\n sbuf.append(\"[\" + nf3.format(score) + \"] \");\r\n }\r\n else {\r\n sbuf.append(\"[\" + nfE.format(score) + \"] \");\r\n }\r\n sbuf.append(sign.toString());\r\n if (showBitset) { sbuf.append(' ').append(toString(bitset)); }\r\n return sbuf.toString();\r\n }\r\n \r\n // formats to two decimal places\r\n private static final NumberFormat nf2 = initNF2();\r\n private static NumberFormat initNF2() { \r\n NumberFormat f = NumberFormat.getInstance();\r\n f.setMinimumIntegerDigits(1);\r\n f.setMinimumFractionDigits(2);\r\n f.setMaximumFractionDigits(2);\r\n return f;\r\n }\r\n \r\n // formats to three decimal places\r\n private static final NumberFormat nf3 = initNF3();\r\n private static NumberFormat initNF3() { \r\n NumberFormat f = NumberFormat.getInstance();\r\n f.setMinimumIntegerDigits(1);\r\n f.setMinimumFractionDigits(3);\r\n f.setMaximumFractionDigits(3);\r\n return f;\r\n }\r\n \r\n // formats to \"0.##E0\"\r\n private static final NumberFormat nfE = new DecimalFormat(\"0.##E0\");\r\n \r\n /** Formats bitset compactly, with ranges hyphenated. */\r\n public static String toString(BitSet bitset) {\r\n StringBuffer sbuf = new StringBuffer();\r\n sbuf.append('{');\r\n int j = 0;\r\n for (int i = bitset.nextSetBit(0); i >= 0; i = bitset.nextSetBit(j+1)) {\r\n if (j != 0) sbuf.append(',');\r\n j = bitset.nextClearBit(i);\r\n if (j == i+1) sbuf.append(Integer.toString(i));\r\n else sbuf.append(i + \"-\" + (j-1));\r\n }\r\n sbuf.append('}');\r\n return sbuf.toString();\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/realize/EdgeCombos.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.realize;\n\nimport java.util.*;\n\n/**\n * EdgeCombos is a simple data structure that allows a representative edge \n * to collect the info about successful combinations of the edge (with \n * other edges or unary rules) that is needed to create analogous new result edges \n * for edges that share the same category, without having to invoke \n * the combinatory rules.\n *\n * @author Michael White\n * @version $Revision: 1.3 $, $Date: 2005/10/13 18:20:30 $\n */\npublic class EdgeCombos\n{\n \n /** Info for a collected category combination. */\n public class CatCombo {\n /** The (other) input edge for the category combo. */\n public final Edge inputEdge;\n /** The result edge, to use in making alternative edges. */\n public final Edge resultEdge;\n /** Constructor */\n public CatCombo(Edge inputEdge, Edge resultEdge) {\n this.inputEdge = inputEdge; this.resultEdge = resultEdge; \n }\n }\n \n /** The rightward combos. */\n public final List rightwardCombos = new ArrayList(5);\n \n /** The leftward combos. */\n public final List leftwardCombos = new ArrayList(5);\n \n /** The unary results. */\n public final List unaryResults = new ArrayList(3);\n \n /** The optional results, ie with optional parts marked as completed. */\n public final List optionalResults = new ArrayList(3);\n \n \n /** Adds a rightward combo. */\n public void addRightwardCombo(Edge inputEdge, Edge resultEdge) {\n rightwardCombos.add(new CatCombo(inputEdge, resultEdge));\n }\n \n /** Adds a leftward combo. */\n public void addLeftwardCombo(Edge inputEdge, Edge resultEdge) {\n leftwardCombos.add(new CatCombo(inputEdge, resultEdge));\n }\n}\n\n" }, { "path": "src/opennlp/ccg/realize/EdgeFactory.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2003-11 University of Edinburgh / Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.realize;\r\n\r\nimport opennlp.ccg.*;\r\nimport opennlp.ccg.lexicon.*;\r\nimport opennlp.ccg.grammar.*;\r\nimport opennlp.ccg.unify.*;\r\nimport opennlp.ccg.synsem.*;\r\nimport opennlp.ccg.hylo.*;\r\nimport opennlp.ccg.util.*;\r\n\r\nimport gnu.trove.*;\r\n\r\nimport java.util.*;\r\nimport java.util.prefs.*;\r\n\r\n/**\r\n * The EdgeFactory is responsible for creating edges. \r\n * A single edge factory instance per realization request is assumed.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.87 $, $Date: 2011/10/30 21:06:47 $\r\n */\r\npublic class EdgeFactory\r\n{\r\n\r\n /** Preference key for whether to use indexing to filter edges to combine. */\r\n public static final String USE_INDEXING = \"Use Indexing\";\r\n \r\n /** Preference key for whether to (exceptionally) allow categories with no target cat index nominal to combine. */\r\n public static final String ALLOW_MISSING_INDEX_COMBOS = \"Allow Missing Index Combos\";\r\n \r\n /** Preference key for whether to use LF chunks to filter edges to combine. */\r\n public static final String USE_CHUNKS = \"Use Chunks\";\r\n \r\n /** Preference key for whether to use feature licensing; if false, \r\n the simple lex feature is used for comparison purposes. */\r\n public static final String USE_FEATURE_LICENSING = \"Use Feature Licensing\";\r\n \r\n\r\n /** The grammar used to create edges. */ \r\n public final Grammar grammar;\r\n \r\n /** The elementary predications to be covered. */\r\n public final List preds;\r\n \r\n /** The sign scorer. */\r\n public final SignScorer signScorer;\r\n \r\n /** The hypertagger. */\r\n public final Hypertagger hypertagger;\r\n \r\n /** The initial, unmarked edges instantiated after lexical lookup. */\r\n public final List initialEdges = new ArrayList();\r\n\r\n /** The marked initial edges licensed by features in the other initial edges. */\r\n public final List markedEdges = new ArrayList();\r\n\r\n /** The licensed and instantiated purely syntactic (semantically null) edges. */\r\n public final List instantiatedNoSemEdges = new ArrayList();\r\n \r\n /** The licensed, uninstantiated purely syntactic edges. */\r\n public final List noSemEdges = new ArrayList();\r\n\r\n /** The rule instances, ie the type changing rules with instantiated semantics. */\r\n public final List ruleInstances = new ArrayList();\r\n \r\n /** The LF chunks, represented as bitsets. */\r\n public final List lfChunks = new ArrayList();\r\n \r\n /** The LF alts, represented as a list of lists of alts \r\n (where each list of alts forms an exclusive disjunction). */\r\n public final List> lfAlts = new ArrayList>();\r\n \r\n /** The LF optional parts, represented as bitsets. */\r\n public final List lfOpts = new ArrayList();\r\n \r\n /** Flag indicating whether there are any LF alts or optional parts. */\r\n public boolean hasLfAltsOrOpts = false;\r\n \r\n // a bitset for all preds\r\n private final BitSet allPreds;\r\n \r\n \r\n // the lexicon used to create edges \r\n private final Lexicon lexicon;\r\n \r\n // general rules, ie the ones with no associated semantics\r\n private final RuleGroup generalRules;\r\n \r\n // rule group for rules wrapped by rule instances\r\n private final RuleGroup ruleInstancesGroup;\r\n \r\n // rule for joining fragments\r\n private final FragmentJoining fragmentRule = new FragmentJoining();\r\n \r\n // helper class for licensing features\r\n private final FeatureLicenser featureLicenser;\r\n \r\n \r\n /** Set of nominals whose phrases are marked for labeling in the output (with mark=+). */\r\n public final Set labeledNominals = new HashSet();\r\n \r\n /** Map from nominals to ints, for indexing edges. */\r\n final TObjectIntHashMap nominals = new TObjectIntHashMap();\r\n \r\n // indexes the preds by their position, \r\n // by mapping pred keys to a list of pred indices for that key\r\n private final Map> predMap = new HashMap>();\r\n \r\n // list of paired nominals in the input LF\r\n private final List pairedNominals = new ArrayList();\r\n \r\n // flag for whether there are any paired nominals\r\n private boolean anyPairedNominals = false;\r\n \r\n /** Set of nominals under a BoundVar relation. */\r\n final Set boundVarNominals = new HashSet();\r\n \r\n // list of nominals for a particular cat or pair of cats\r\n private final List catNominals = new ArrayList();\r\n \r\n \r\n /** \r\n * Flag for whether to use indexing. \r\n * Setting retrieved from preferences; turned off when gluing fragments. \r\n */\r\n public boolean useIndexing = true;\r\n\r\n // flag for whether to (exceptionally) allow categories with no target cat index nominal to combine\r\n private boolean allowMissingIndexCombos = false;\r\n \r\n // flag for whether to use chunks\r\n private boolean useChunks = true;\r\n \r\n // flag for whether to use feature licensing\r\n private boolean useFeatureLicensing = true;\r\n\r\n /** \r\n * Flag for whether to debug category instantiation (defaults to false). \r\n * If true, cases of complex categories whose outermost category \r\n * is not instantiated with index nominals are reported to \r\n * System.err. Note that realization is more efficient if such \r\n * categories can be avoided in the grammar.\r\n * Uncovered EPs after lex lookup are also reported to System.err.\r\n */ \r\n public boolean debugInstantiation = false;\r\n \r\n \r\n /* The number of unary rule applications executed. */\r\n private int unaryRuleApps = 0;\r\n\r\n /* The number of unary rule instance applications executed. */\r\n private int unaryRuleInstApps = 0;\r\n\r\n /* The number of binary rule applications executed. */\r\n private int binaryRuleApps = 0;\r\n \r\n\r\n /** Flag for whether to glue fragments currently. Defaults to false. */\r\n public boolean gluingFragments = false;\r\n\r\n /** Bit vector for EPs not covered by a lexical edge or rule instance; null if none. */\r\n protected BitSet uncoveredEPs = null;\r\n \r\n /** Flag indicating whether any lexical or featural EPs are uncovered. */\r\n public boolean hasUncoveredPreds = false;\r\n \r\n /** Flag for whether to use relaxed relation matching. */ // XXX tmp switch\r\n protected boolean useRelaxedRelationMatching = Boolean.getBoolean(\"useRelaxedRelationMatching\");\r\n \r\n \r\n /** Constructor. */\r\n public EdgeFactory(Grammar grammar, List preds, SignScorer signScorer) {\r\n \tthis(grammar, preds, signScorer, null);\r\n }\r\n \r\n /** Constructor with hypertagger. */\r\n public EdgeFactory(Grammar grammar, List preds, SignScorer signScorer, Hypertagger hypertagger) {\r\n this.grammar = grammar;\r\n this.preds = preds;\r\n this.signScorer = signScorer;\r\n this.hypertagger = hypertagger;\r\n \r\n lexicon = grammar.lexicon;\r\n generalRules = new RuleGroup(grammar);\r\n generalRules.borrowSupercatRuleCombos(grammar.rules);\r\n ruleInstancesGroup = new RuleGroup(grammar);\r\n ruleInstancesGroup.borrowSupercatRuleCombos(grammar.rules);\r\n \r\n allPreds = new BitSet(preds.size());\r\n allPreds.set(0, preds.size());\r\n \r\n Preferences prefs = Preferences.userNodeForPackage(TextCCG.class);\r\n useIndexing = prefs.getBoolean(USE_INDEXING, true);\r\n\tallowMissingIndexCombos = prefs.getBoolean(ALLOW_MISSING_INDEX_COMBOS,false);\r\n useChunks = prefs.getBoolean(USE_CHUNKS, true);\r\n useFeatureLicensing = prefs.getBoolean(USE_FEATURE_LICENSING, true);\r\n\r\n if (useFeatureLicensing) {\r\n featureLicenser = new FeatureLicenser(this);\r\n } else {\r\n // if feature licensing off, use simple lex feature for comparison purposes\r\n featureLicenser = new FeatureLicenser(\r\n this, \r\n new LicensingFeature[] { LicensingFeature.simpleLexFeature }\r\n );\r\n }\r\n \r\n UnifyControl.startUnifySequence();\r\n extractLabeledNominals();\r\n indexPreds();\r\n listNominals();\r\n listPairedNominals();\r\n addBoundVarNominals();\r\n fillLfChunks(); \r\n fillLfAlts(); fillLfOpts();\r\n hasLfAltsOrOpts = lfAlts.size() > 0 || lfOpts.size() > 0;\r\n \r\n if (hypertagger != null) hypertagger.mapPreds(preds);\r\n }\r\n\r\n /**\r\n * Adds LF optional parts for any preds not covered by a lex item or rule instance,\r\n * divided up by chunks, for use with fragment gluing or relaxed relation matching.\r\n */\r\n public void addLFOptsForUncoveredPreds() {\r\n \t// see if all preds covered\r\n \tif (uncoveredEPs == null) return;\r\n \t// clone what's not covered by lex items and rule instances\r\n \tBitSet opt = (BitSet) uncoveredEPs.clone();\r\n\t\t// otherwise add opts for what's missing\r\n\t\t// nb: need to split up by chunks\r\n\t\tfor (BitSet chunk : lfChunks) {\r\n\t\t\tif (opt.intersects(chunk)) {\r\n\t\t\t\tBitSet optChunk = (BitSet) opt.clone();\r\n\t\t\t\toptChunk.and(chunk);\r\n\t\t\t\tif (!lfOpts.contains(optChunk)) lfOpts.add(optChunk);\r\n\t\t\t\topt.andNot(optChunk);\r\n\t\t\t}\r\n\t\t}\r\n\t\t// add anything remaining\r\n\t\tif (!opt.isEmpty() && !lfOpts.contains(opt)) lfOpts.add(opt);\r\n \t// ensure hasLfAltsOrOpts set\r\n \thasLfAltsOrOpts = true;\r\n\t\t// TODO deal with lf alts too (may require sorting chunks and alts by size)\r\n }\r\n \r\n /**\r\n * Adds an LF optional part for each instantiated rule instance, \r\n * for use with fragment gluing.\r\n */\r\n public void addLFOptsForRuleInstances() {\r\n \t// do each rule instance\r\n \tfor (RuleInstance ruleInstance : ruleInstances) {\r\n \t\tBitSet opt = (BitSet) ruleInstance.bitset.clone();\r\n \t\tlfOpts.add(opt);\r\n \t}\r\n \t// ensure hasLfAltsOrOpts set\r\n \tif (lfOpts.size() > 0) hasLfAltsOrOpts = true;\r\n }\r\n \r\n // returns the uncovered preds, or null if none\r\n private BitSet uncoveredPreds() {\r\n \t// determine what's covered by lex items and rule instances\r\n \tBitSet retval = new BitSet(preds.size());\r\n \tfor (Edge edge : initialEdges) retval.or(edge.bitset);\r\n \tfor (Edge edge : markedEdges) retval.or(edge.bitset);\r\n \tfor (RuleInstance ruleInstance : ruleInstances) retval.or(ruleInstance.bitset);\r\n \t// see if all preds covered\r\n \tif (retval.equals(allPreds)) return null;\r\n\t\t// otherwise xor what's missing\r\n\t\tretval.xor(allPreds);\r\n\t\t// set missing lex preds flag\r\n\t\tfor (int i = retval.nextSetBit(0); i >= 0; i = retval.nextSetBit(i+1)) {\r\n\t\t\tSatOp pred = preds.get(i);\r\n\t\t\tif (HyloHelper.isLexPred(pred) || HyloHelper.isAttrPred(pred)) {\r\n\t\t\t\thasUncoveredPreds = true; break;\r\n\t\t\t}\r\n\t\t}\r\n\t\t// return\r\n \treturn retval;\r\n }\r\n \r\n \r\n //-----------------------------------------------------------------\r\n // edge construction\r\n //\r\n\r\n /** Makes an edge, computing the completeness percentage, sign score, \r\n and indices, and setting the most specific incomplete LF chunk (if any). */\r\n protected Edge makeEdge(Sign sign, BitSet bitset, List> activeLfAlts) {\r\n BitSet indices = getIndices(sign.getCategory(), null);\r\n float completeness = bitset.cardinality() / (float) preds.size();\r\n boolean complete = (completeness == 1.0);\r\n double score = signScorer.score(sign, complete);\r\n BitSet incompleteLfChunk = getIncompleteLfChunk(bitset, activeLfAlts);\r\n return new Edge(sign, bitset, indices, completeness, score, activeLfAlts, incompleteLfChunk);\r\n }\r\n\r\n /** Makes an edge for the given alt sign from the given edge, after computing the sign's score. */\r\n protected Edge makeAltEdge(Sign altSign, Edge edge) {\r\n double score = signScorer.score(altSign, edge.complete());\r\n return new Edge(\r\n \t\taltSign, edge.bitset, edge.indices, \r\n \t\tedge.completeness, score, \r\n \t\tedge.activeLfAlts, edge.incompleteLfChunk\r\n );\r\n }\r\n \r\n /** Makes an edge consisting of two joined fragments. */\r\n public Edge makeJoinedEdge(Edge edge1, Edge edge2) {\r\n \tSign sign = fragmentRule.applyRule(edge1.sign, edge2.sign);\r\n \tBitSet bitset = (BitSet) edge1.bitset.clone();\r\n \tbitset.or(edge2.bitset);\r\n float completeness = bitset.cardinality() / (float) preds.size();\r\n boolean complete = (completeness == 1.0);\r\n double score = signScorer.score(sign, complete);\r\n return new Edge(\r\n \t\tsign, bitset, edge1.indices, \r\n \t\tcompleteness, score, \r\n \t\tedge1.activeLfAlts, edge1.incompleteLfChunk\r\n );\r\n }\r\n \r\n \r\n //-----------------------------------------------------------------\r\n // active alts\r\n //\r\n\r\n /** From the given LF alts, returns the active ones for the given bitset, \r\n updating the bitset for any completely covered alts. \r\n NB: If the given LF alts list is not the entire list, each alt is assumed to intersect. */\r\n private List> getActiveLfAlts(List> fromLfAlts, BitSet bitset) {\r\n if (fromLfAlts.isEmpty()) return fromLfAlts;\r\n boolean checkingAllAlts = (fromLfAlts == lfAlts);\r\n BitSet tmpBitSet = new BitSet(bitset.size());\r\n List> retval = new ArrayList>(fromLfAlts.size());\r\n // check each 'from' alt \r\n for (List altSet : fromLfAlts) {\r\n List activeAltSet = null; // for collecting active alts\r\n boolean foundCoveredAlt = false;\r\n for (Alt alt : altSet) { \r\n // if checking all alts, check intersection with alt\r\n if (!checkingAllAlts || alt.bitset.intersects(bitset)) {\r\n // check whether alt completely covered\r\n tmpBitSet.clear(); tmpBitSet.or(bitset); tmpBitSet.and(alt.bitset);\r\n if (tmpBitSet.equals(alt.bitset)) {\r\n foundCoveredAlt = true; break;\r\n }\r\n else {\r\n // otherwise add to active alts\r\n if (activeAltSet == null) activeAltSet = new ArrayList(3);\r\n activeAltSet.add(alt);\r\n }\r\n }\r\n }\r\n if (foundCoveredAlt) {\r\n // update coverage bitset to include all alts in this set\r\n List fullAltSet = lfAlts.get(altSet.get(0).altSet);\r\n for (Alt alt : fullAltSet) bitset.or(alt.bitset); \r\n }\r\n else {\r\n // otherwise update active alts, if any\r\n if (activeAltSet != null) retval.add(activeAltSet);\r\n }\r\n }\r\n return retval;\r\n }\r\n \r\n /** Returns the active LF alts that result from combining the given ones, \r\n or null if these are incompatible.\r\n For alt sets in common, the combined alts consist of the intersection \r\n of these alt sets, or null if this intersection is empty.\r\n For alts sets not in common, the active alts are carried through \r\n unchanged. */\r\n private List> getCombinedLfAlts(List> activeLfAlts1, List> activeLfAlts2) {\r\n if (activeLfAlts1.isEmpty()) return activeLfAlts2;\r\n if (activeLfAlts2.isEmpty()) return activeLfAlts1;\r\n List> retval = new ArrayList>(activeLfAlts1.size() + activeLfAlts2.size());\r\n Iterator> it1 = activeLfAlts1.iterator(); Iterator> it2 = activeLfAlts2.iterator();\r\n List altSet1 = it1.next(); List altSet2 = it2.next();\r\n for (int i = 0; i < lfAlts.size(); i++) {\r\n // inc to alt set i, if not yet there (or beyond)\r\n if (altSet1.get(0).altSet < i && it1.hasNext()) altSet1 = it1.next(); \r\n if (altSet2.get(0).altSet < i && it2.hasNext()) altSet2 = it2.next();\r\n // check whether only one or the other has alt set i\r\n if (altSet1.get(0).altSet == i && altSet2.get(0).altSet != i) retval.add(altSet1);\r\n else if (altSet2.get(0).altSet == i && altSet1.get(0).altSet != i) retval.add(altSet2);\r\n else if (altSet1.get(0).altSet == i && altSet2.get(0).altSet == i) {\r\n // take intersection\r\n List combined = new ArrayList(Math.min(altSet1.size(),altSet2.size()));\r\n for (Alt alt : altSet1) {\r\n if (altSet2.contains(alt)) combined.add(alt);\r\n }\r\n // check for empty intersection, returning null \r\n if (combined.isEmpty()) return null;\r\n // otherwise add combined list\r\n retval.add(combined);\r\n }\r\n }\r\n return retval;\r\n }\r\n \r\n \r\n //-----------------------------------------------------------------\r\n // misc bookkeeping\r\n //\r\n\r\n // extracts the nominal atoms marked for labeling in the output\r\n private void extractLabeledNominals() {\r\n for (Iterator it = preds.iterator(); it.hasNext(); ) {\r\n \tSatOp pred = it.next();\r\n if (!HyloHelper.isAttrPred(pred)) continue;\r\n Nominal nom1 = HyloHelper.getPrincipalNominal(pred);\r\n if (!(nom1 instanceof NominalAtom)) continue;\r\n String rel = HyloHelper.getRel(pred);\r\n if (rel == null || !rel.equals(\"mark\")) continue;\r\n labeledNominals.add(nom1);\r\n it.remove();\r\n }\r\n }\r\n \r\n // lists the nominals in the preds\r\n private void listNominals() {\r\n \tfor (SatOp pred : preds) {\r\n Nominal nom1 = HyloHelper.getPrincipalNominal(pred);\r\n Nominal nom2 = HyloHelper.getSecondaryNominal(pred);\r\n if (nom1 instanceof NominalAtom && !nominals.containsKey(nom1)) { \r\n nominals.put(nom1, nominals.size()); \r\n }\r\n if (nom2 instanceof NominalAtom && !nominals.containsKey(nom2)) { \r\n nominals.put(nom2, nominals.size()); \r\n }\r\n }\r\n }\r\n \r\n // create bitset for cat indices\r\n private BitSet getIndices(Category cat, Category cat2) {\r\n catNominals.clear();\r\n cat.forall(gatherIndices);\r\n if (cat2 != null) { cat2.forall(gatherIndices); }\r\n BitSet retval = new BitSet(nominals.size());\r\n for (Iterator it = catNominals.iterator(); it.hasNext(); ) {\r\n Object nom = it.next();\r\n int index = nominals.get(nom);\r\n retval.set(index);\r\n }\r\n return retval;\r\n }\r\n\r\n // check for uninstantiated outer args; if found, \r\n // set the indices to allow all combos, and issue \r\n // a warning if the debugInstantiation flag is set\r\n private void checkInstantiation(List edges) {\r\n for (int i = 0; i < edges.size(); i++) {\r\n Edge edge = edges.get(i);\r\n if (noSemEdges.contains(edge)) continue; // exempt uninstantiated no sem edges\r\n if (outerArgUninstantiated(edge.sign.getCategory())) {\r\n edge.indices.set(0, nominals.size());\r\n if (debugInstantiation) {\r\n \tSystem.err.println(\"Warning: outer arg uninstantiated: \" + edge.sign);\r\n }\r\n }\r\n }\r\n }\r\n \r\n // returns whether the outermost arg is not instantiated\r\n private boolean outerArgUninstantiated(Category cat) {\r\n if (!(cat instanceof ComplexCat)) return false;\r\n Arg outer = ((ComplexCat)cat).getOuterArg();\r\n catNominals.clear();\r\n outer.forall(gatherIndices);\r\n return catNominals.isEmpty();\r\n }\r\n \r\n // gathers values of index feature in atomic cats\r\n private CategoryFcn gatherIndices = new CategoryFcnAdapter() {\r\n public void forall(Category c) {\r\n if (!(c instanceof AtomCat)) return;\r\n FeatureStructure fs = c.getFeatureStructure();\r\n if (fs == null) return;\r\n addCatNominal(fs.getValue(\"index\"));\r\n addCatNominal(fs.getValue(\"mod-index\"));\r\n }\r\n };\r\n \r\n // adds a nominal atom to catNominals\r\n private void addCatNominal(Object indexVal) {\r\n if (indexVal instanceof NominalAtom) { \r\n if (!catNominals.contains(indexVal)) { catNominals.add(indexVal); }\r\n }\r\n }\r\n\r\n // lists the paired nominals in the input LF, as a bitset pair\r\n private void listPairedNominals() {\r\n for (int i=0; i < preds.size(); i++) {\r\n \tSatOp pred = preds.get(i);\r\n if (!\"tup\".equals(HyloHelper.getLexPred(pred))) continue;\r\n Nominal tupNom = HyloHelper.getPrincipalNominal(pred);\r\n Nominal nom1 = null;\r\n Nominal nom2 = null;\r\n for (int j = i+1; j < preds.size(); j++) {\r\n \tSatOp predJ = preds.get(j);\r\n if (!tupNom.equals(HyloHelper.getPrincipalNominal(predJ))) break;\r\n if (\"Item1\".equals(HyloHelper.getRel(predJ))) {\r\n nom1 = HyloHelper.getSecondaryNominal(predJ);\r\n }\r\n if (\"Item2\".equals(HyloHelper.getRel(predJ))) {\r\n nom2 = HyloHelper.getSecondaryNominal(predJ);\r\n }\r\n }\r\n if (nom1 == null || nom2 == null) {\r\n System.err.println(\"Warning, couldn't find paired nominals for tuple: \" + tupNom);\r\n continue;\r\n }\r\n if (!(nom1 instanceof NominalAtom)) continue;\r\n if (!(nom2 instanceof NominalAtom)) continue;\r\n BitSet[] pair = new BitSet[2];\r\n pair[0] = new BitSet(nominals.size());\r\n pair[0].set(nominals.get(nom1));\r\n pair[1] = new BitSet(nominals.size());\r\n pair[1].set(nominals.get(nom2));\r\n pairedNominals.add(pair);\r\n anyPairedNominals = true;\r\n }\r\n }\r\n \r\n // adds the bound var nominals\r\n private void addBoundVarNominals() {\r\n for (int i=0; i < preds.size(); i++) {\r\n \tSatOp pred = preds.get(i);\r\n String rel = HyloHelper.getRel(pred);\r\n if (rel == null || !rel.equals(\"BoundVar\")) continue;\r\n Nominal nom2 = HyloHelper.getSecondaryNominal(pred);\r\n if (!(nom2 instanceof NominalAtom)) continue;\r\n boundVarNominals.add(nom2);\r\n // check if nom2 is a tuple\r\n for (int j = 0; j < preds.size(); j++) {\r\n \tSatOp predJ = preds.get(j);\r\n if (!nom2.equals(HyloHelper.getPrincipalNominal(predJ))) continue;\r\n if (!\"tup\".equals(HyloHelper.getLexPred(predJ))) continue;\r\n // if so, add paired items as bound vars too\r\n for (int k = j+1; k < preds.size(); k++) {\r\n \tSatOp predK = preds.get(k);\r\n if (!nom2.equals(HyloHelper.getPrincipalNominal(predK))) break;\r\n String relK = HyloHelper.getRel(predK);\r\n if (\"Item1\".equals(relK) || \"Item2\".equals(relK)) {\r\n Nominal nom2K = HyloHelper.getSecondaryNominal(predK);\r\n if (!(nom2K instanceof NominalAtom)) continue;\r\n boundVarNominals.add(nom2K);\r\n }\r\n }\r\n }\r\n }\r\n }\r\n \r\n /**\r\n * Returns whether the indices for the two edges are paired in the input LF.\r\n */\r\n public boolean pairedWith(Edge edgeA, Edge edgeB) {\r\n for (int i = 0; i < pairedNominals.size(); i++) {\r\n BitSet[] pair = pairedNominals.get(i);\r\n if (pair[0].equals(edgeA.indices) && pair[1].equals(edgeB.indices)) {\r\n return true;\r\n }\r\n }\r\n return false;\r\n }\r\n \r\n\r\n // indexes the preds by their position into predMap\r\n private void indexPreds() {\r\n for (int i=0; i < preds.size(); i++) {\r\n String[] keys = predKeys(preds.get(i));\r\n for (int j=0; j < keys.length; j++) {\r\n List indices = predMap.get(keys[j]);\r\n if (indices == null) { \r\n indices = new ArrayList(1);\r\n predMap.put(keys[j], indices);\r\n }\r\n indices.add(i);\r\n }\r\n }\r\n }\r\n\r\n // returns 0-2 keys for the given pred (where nom vars are skipped):\r\n // a lex pred is indexed by atom(pred)\r\n // a rel pred is indexed by atom and atom2\r\n // an attr pred is indexed by atom\r\n private static String[] predKeys(LF pred) {\r\n Nominal nom = HyloHelper.getPrincipalNominal(pred);\r\n String lexPred = HyloHelper.getLexPred(pred);\r\n String rel = HyloHelper.getRel(pred);\r\n Nominal nom2 = HyloHelper.getSecondaryNominal(pred);\r\n List keys = new ArrayList(2);\r\n if (nom instanceof NominalAtom && lexPred != null) \r\n keys.add(nom.toString() + \"(\" + lexPred + \")\");\r\n if (nom instanceof NominalAtom && rel != null) \r\n keys.add(nom.toString() + \"<\" + rel + \">\");\r\n if (nom2 instanceof NominalAtom && rel != null) \r\n keys.add(\"<\" + rel + \">\" + nom2.toString());\r\n return (String[]) keys.toArray(new String[keys.size()]);\r\n }\r\n \r\n // fills in the LF chunks list with the chunks for each pred, \r\n // then sorts them by specificity, from most to least\r\n private void fillLfChunks() {\r\n // for each pred, fill in chunks\r\n for (int i=0; i < preds.size(); i++) {\r\n \tSatOp pred = preds.get(i);\r\n \tTIntArrayList chunks = pred.getChunks();\r\n if (chunks == null) continue;\r\n // for each chunk that this pred is part of\r\n for (int j = 0; j < chunks.size(); j++) {\r\n int chunkId = chunks.get(j);\r\n // ensure chunk bitset exists\r\n while (lfChunks.size() < (chunkId + 1)) {\r\n lfChunks.add(new BitSet(preds.size()));\r\n }\r\n // update chunk bitset\r\n BitSet chunk = lfChunks.get(chunkId);\r\n chunk.set(i);\r\n }\r\n }\r\n // do insertion sort, to ensure subset ordering\r\n List tmpList = new ArrayList(lfChunks);\r\n lfChunks.clear();\r\n allChunks: for (BitSet chunk : tmpList) {\r\n for (int i=0; i < lfChunks.size(); i++) {\r\n BitSet sortedChunk = lfChunks.get(i);\r\n if (subset(chunk, sortedChunk)) {\r\n lfChunks.add(i, chunk); continue allChunks;\r\n }\r\n }\r\n lfChunks.add(chunk);\r\n }\r\n }\r\n \r\n // gets the most specific incomplete chunk for an edge, or null\r\n private BitSet getIncompleteLfChunk(BitSet bitset, List> activeLfAlts) {\r\n // check each chunk\r\n allChunks: for (BitSet lfChunk : lfChunks) {\r\n \t// for intersection\r\n if (!lfChunk.intersects(bitset)) { continue; }\r\n // for incomplete coverage\r\n if (!subset(lfChunk, bitset) && subset(bitset, lfChunk)) {\r\n \t// and for part of all active alts\r\n \tfor (List altSet : activeLfAlts) {\r\n \t\tfor (Alt alt : altSet) {\r\n \t\t\tif (!subset(lfChunk, alt.bitset)) continue allChunks;\r\n \t\t}\r\n \t}\r\n \t// return chunk\r\n \treturn lfChunk;\r\n }\r\n }\r\n \t// otherwise null\r\n \treturn null;\r\n }\r\n\r\n // returns true iff bitset1 is a subset of bitset2\r\n\tprivate BitSet tmpBitSet = new BitSet();\r\n\tprivate boolean subset(BitSet bitset1, BitSet bitset2) {\r\n\t\ttmpBitSet.clear();\r\n\t\ttmpBitSet.or(bitset1);\r\n\t\ttmpBitSet.andNot(bitset2);\r\n\t\treturn tmpBitSet.isEmpty();\r\n\t}\r\n \r\n // returns true iff combining the edges would complete a chunk\r\n\tprivate boolean completesChunk(Edge edgeA, Edge edgeB) {\r\n\t\tif (edgeA.incompleteLfChunk != null) {\r\n\t\t\ttmpBitSet.clear();\r\n\t\t\ttmpBitSet.or(edgeA.incompleteLfChunk);\r\n\t\t\ttmpBitSet.andNot(edgeA.bitset); tmpBitSet.andNot(edgeB.bitset);\r\n\t\t\tif (tmpBitSet.isEmpty()) return true;\r\n\t\t}\r\n\t\tif (edgeB.incompleteLfChunk != null) {\r\n\t\t\ttmpBitSet.clear();\r\n\t\t\ttmpBitSet.or(edgeB.incompleteLfChunk);\r\n\t\t\ttmpBitSet.andNot(edgeA.bitset); tmpBitSet.andNot(edgeB.bitset);\r\n\t\t\tif (tmpBitSet.isEmpty()) return true;\r\n\t\t}\r\n\t\treturn false;\r\n\t}\r\n\t\r\n // fills in the LF alts list with the alts for each pred\r\n private void fillLfAlts() {\r\n // for each pred\r\n for (int i=0; i < preds.size(); i++) {\r\n SatOp pred = preds.get(i);\r\n List alts = pred.getAlts();\r\n if (alts == null) continue;\r\n // for each alt that this pred is part of\r\n for (Alt alt : alts) {\r\n // ensure list for alt set exists\r\n while (lfAlts.size() < (alt.altSet + 1)) {\r\n lfAlts.add(null);\r\n }\r\n List altSet = lfAlts.get(alt.altSet);\r\n if (altSet == null) {\r\n altSet = new ArrayList(5);\r\n lfAlts.set(alt.altSet, altSet);\r\n }\r\n // ensure list item for alt num is alt\r\n while (altSet.size() < (alt.numInSet + 1)) {\r\n altSet.add(null);\r\n }\r\n altSet.set(alt.numInSet, alt);\r\n // update alt bitset\r\n alt.bitset.set(i);\r\n }\r\n }\r\n }\r\n \r\n // fills in the LF opts list with the opts for each pred\r\n private void fillLfOpts() {\r\n // for each pred\r\n for (int i=0; i < preds.size(); i++) {\r\n SatOp pred = preds.get(i);\r\n TIntArrayList opts = pred.getOpts();\r\n if (opts == null) continue;\r\n // for each opt that this pred is part of\r\n for (int j = 0; j < opts.size(); j++) {\r\n int optId = opts.get(j);\r\n // ensure opt bitset exists\r\n while (lfOpts.size() < (optId + 1)) {\r\n lfOpts.add(new BitSet(preds.size()));\r\n }\r\n // update opt bitset\r\n BitSet opt = lfOpts.get(optId);\r\n opt.set(i);\r\n }\r\n }\r\n }\r\n \r\n \r\n // returns the list of coart rels for the pred with the given index\r\n // NB: assumes that preds are sorted by their principal nominals, with the lex pred first\r\n private List getCoartRels(int predIndex) {\r\n \tSatOp pred = preds.get(predIndex);\r\n Nominal nom = HyloHelper.getPrincipalNominal(pred);\r\n List retval = null;\r\n for (int i = predIndex+1; i < preds.size(); i++) {\r\n \tSatOp relPred = preds.get(i);\r\n if (!nom.equals(HyloHelper.getPrincipalNominal(relPred))) break;\r\n String rel = HyloHelper.getRel(relPred);\r\n if (rel != null && grammar.lexicon.isCoartRel(rel)) { \r\n if (retval == null) retval = new ArrayList(3);\r\n retval.add(rel);\r\n }\r\n }\r\n return retval;\r\n }\r\n\r\n \r\n //-----------------------------------------------------------------\r\n // createInitialEdges\r\n //\r\n\r\n /** \r\n * Creates and returns all initial edges.\r\n * In particular, initializes all lexical edges that cover some of the input semantics; \r\n * also initializes edges for semantically null lexical items, \r\n * and initializes instances of type changing rules which \r\n * introduce their own semantics.\r\n * If a hypertagger is in place, only the beta-best edges are returned for each EP.\r\n */\r\n public List createInitialEdges() {\r\n\r\n // marked initial edges that need to be licensed\r\n List markedEdgesForLicensing = new ArrayList(); \r\n \r\n // for each pred, create edges for signs indexed \r\n // by lexical preds and by indexed rels; \r\n // and similarly for type changing rules\r\n for (int i=0; i < preds.size(); i++) {\r\n \tSatOp pred = preds.get(i);\r\n String key = HyloHelper.getLexPred(pred);\r\n String rel = HyloHelper.getRel(pred);\r\n // skip if no lex pred or indexed rel (not expected)\r\n if (key == null && rel == null) continue;\r\n // update hypertagger for beta-best lookup\r\n if (hypertagger != null) hypertagger.setPred(i); \r\n Collection signs = new ArrayList();\r\n Collection typeChangingRules = new ArrayList();\r\n // add signs and rules for lex pred\r\n if (key != null) {\r\n List coartRels = getCoartRels(i);\r\n Collection lexPredSigns = lexicon.getSignsFromPred(key, coartRels);\r\n if (lexPredSigns != null) { signs.addAll(lexPredSigns); }\r\n Collection lexPredRules = grammar.rules.getRulesForPred(key);\r\n if (lexPredRules != null) { typeChangingRules.addAll(lexPredRules); }\r\n }\r\n // add signs and rules for indexed rel\r\n if (rel != null) {\r\n Collection indexedRelSigns = lexicon.getSignsFromRel(rel);\r\n if (indexedRelSigns != null) { signs.addAll(indexedRelSigns); }\r\n Collection indexedRelRules = grammar.rules.getRulesForRel(rel);\r\n if (indexedRelRules != null) { typeChangingRules.addAll(indexedRelRules); }\r\n }\r\n // create initial and marked edges for each sign, updating feature map\r\n for (Sign sign : signs) {\r\n List initialEdgesForSign = createInitialEdges(sign, i);\r\n if (initialEdgesForSign != null) {\r\n for (Edge initialEdge : initialEdgesForSign) {\r\n Category cat = initialEdge.sign.getCategory();\r\n if (featureLicenser.needsLicensing(cat))\r\n markedEdgesForLicensing.add(initialEdge);\r\n else {\r\n initialEdges.add(initialEdge);\r\n featureLicenser.updateFeatureMap(cat);\r\n }\r\n }\r\n }\r\n }\r\n // create rules instances for each rule, updating feature map\r\n for (TypeChangingRule rule : typeChangingRules) {\r\n List ruleInstancesForRule = createRuleInstances(rule, i);\r\n if (ruleInstancesForRule != null) {\r\n for (RuleInstance ruleInst : ruleInstancesForRule) { \r\n ruleInstances.add(ruleInst);\r\n featureLicenser.updateFeatureMap(ruleInst.rule.getArg());\r\n featureLicenser.updateFeatureMap(ruleInst.rule.getResult());\r\n }\r\n }\r\n }\r\n }\r\n \r\n // add licensed, marked initial edges\r\n int prevSize;\r\n do { // while list size is changing\r\n prevSize = markedEdgesForLicensing.size();\r\n for (Iterator it = markedEdgesForLicensing.iterator(); it.hasNext(); ) {\r\n // check each edge\r\n Edge edge = it.next();\r\n Category cat = edge.sign.getCategory();\r\n if (featureLicenser.isLicensed(cat)) {\r\n // and add to marked edges if licensed\r\n markedEdges.add(edge);\r\n it.remove();\r\n // updating feature map\r\n featureLicenser.updateFeatureMap(cat);\r\n }\r\n }\r\n } while (markedEdgesForLicensing.size() != prevSize);\r\n \r\n // initialize general rules\r\n initGeneralRules();\r\n\r\n // initialize edges for semantically null lexical items \r\n initNoSemEdges();\r\n \r\n // collect all initial edges\r\n List retval = new ArrayList(\r\n initialEdges.size() + markedEdges.size() + \r\n instantiatedNoSemEdges.size() + noSemEdges.size()\r\n );\r\n retval.addAll(initialEdges);\r\n retval.addAll(markedEdges);\r\n retval.addAll(instantiatedNoSemEdges);\r\n retval.addAll(noSemEdges);\r\n \r\n // check instantiation of outermost cats\r\n checkInstantiation(retval);\r\n \r\n // set uncovered EPs\r\n uncoveredEPs = uncoveredPreds();\r\n // warn if EPs missing and debug instantiation flag set\r\n if (uncoveredEPs != null && debugInstantiation) {\r\n \tSystem.err.println(\"Warning, uncovered preds after lex instantiation: \" + Edge.toString(uncoveredEPs));\r\n }\r\n \r\n // set opts for missing relations, if apropos\r\n if (useRelaxedRelationMatching) addLFOptsForUncoveredPreds();\r\n \t\r\n // return\r\n return retval;\r\n }\r\n \r\n // return null if LF doesn't unify with preds\r\n private List createInitialEdges(Sign sign, int predIndex) {\r\n // get parts of sign \r\n List words = sign.getWords();\r\n Category cat = sign.getCategory();\r\n // instantiate\r\n List> instantiations = instantiate(cat, null, predIndex);\r\n // check for failure\r\n if (instantiations == null) return null;\r\n // otherwise fill cats and make edges\r\n List retval = new ArrayList(instantiations.size());\r\n for (Pair inst : instantiations) {\r\n Substitution subst = inst.a; BitSet bitset = inst.b;\r\n Category filledCat = null;\r\n try {\r\n filledCat = (Category) cat.fill(subst);\r\n } catch (UnifyFailure uf) {\r\n // shouldn't happen\r\n throw new RuntimeException(\"Unable to fill cat: \" + uf);\r\n }\r\n // index subcategorized semantically null words\r\n featureLicenser.indexSemanticallyNullWords(filledCat);\r\n // update lex origins for new sign\r\n Sign newSign = new Sign(words, filledCat);\r\n newSign.setOrigin();\r\n // and add new edge\r\n List> activeLfAlts = getActiveLfAlts(lfAlts, bitset);\r\n retval.add(makeEdge(newSign, bitset, activeLfAlts));\r\n }\r\n // and return them\r\n return retval;\r\n }\r\n \r\n // return null if result LF doesn't unify with preds\r\n private List createRuleInstances(TypeChangingRule rule, int predIndex) {\r\n // get parts of rule\r\n Category result = rule.getResult();\r\n Category arg = rule.getArg();\r\n // instantiate\r\n List> instantiations = instantiate(result, arg, predIndex);\r\n // check for failure\r\n if (instantiations == null) return null;\r\n // otherwise fill cats and make rule instances\r\n List retval = new ArrayList(instantiations.size());\r\n for (Pair inst : instantiations) {\r\n Substitution subst = inst.a; BitSet bitset = inst.b;\r\n Category filledResult = null; Category filledArg = null;\r\n try {\r\n filledResult = (Category) result.fill(subst);\r\n filledArg = (Category) arg.fill(subst);\r\n } catch (UnifyFailure uf) {\r\n // shouldn't happen\r\n throw new RuntimeException(\"Unable to fill cat: \" + uf);\r\n }\r\n // index subcategorized semantically null words\r\n featureLicenser.indexSemanticallyNullWords(filledArg);\r\n featureLicenser.indexSemanticallyNullWords(filledResult);\r\n // and return new rule instance\r\n BitSet indices = getIndices(filledResult, filledArg);\r\n TypeChangingRule newRule = new TypeChangingRule(filledArg, filledResult, rule.name(), rule.getFirstEP());\r\n ruleInstancesGroup.addRule(newRule);\r\n List> activeLfAlts = getActiveLfAlts(lfAlts, bitset);\r\n RuleInstance ruleInst = new RuleInstance(newRule, bitset, indices, activeLfAlts);\r\n retval.add(ruleInst);\r\n }\r\n // and return them\r\n return retval;\r\n }\r\n \r\n\r\n // return null if cat LF doesn't unify with preds\r\n private List> instantiate(Category cat, Category cat2, int predIndex) {\r\n\r\n // unify with indexed pred\r\n UnifyControl.reindex(cat, cat2); \r\n List lfPreds = HyloHelper.getPreds(cat.getLF());\r\n Substitution subst = null;\r\n SatOp indexedPred = preds.get(predIndex);\r\n int lfPredIndex = -1;\r\n for (int i=0; i < lfPreds.size(); i++) {\r\n LF lfPred = lfPreds.get(i);\r\n subst = new SimpleSubstitution();\r\n try {\r\n Unifier.unify(lfPred, indexedPred, subst);\r\n lfPredIndex = i;\r\n break;\r\n } catch (UnifyFailure uf) {}\r\n }\r\n \r\n // if failed, return empty list\r\n if (lfPredIndex == -1) return null;\r\n \r\n // set indexed pred in bitset\r\n BitSet bitset = new BitSet(preds.size());\r\n bitset.set(predIndex);\r\n \r\n // unify with rest of lfPreds, extending subst/bitset\r\n List remainingPreds = new ArrayList(lfPreds.size());\r\n remainingPreds.addAll(lfPreds);\r\n remainingPreds.remove(lfPredIndex);\r\n int prevSize = -1;\r\n List> retval = new ArrayList>(3);\r\n List> prev = new ArrayList>(3);\r\n retval.add(new Pair(subst, bitset));\r\n // loop until empty or no changes, in order to propagate matches\r\n while (!remainingPreds.isEmpty() && remainingPreds.size() != prevSize) {\r\n prevSize = remainingPreds.size();\r\n for (Iterator it = remainingPreds.iterator(); it.hasNext(); ) {\r\n \tSatOp lfPred = it.next();\r\n try {\r\n // fill index\r\n lfPred = (SatOp) lfPred.fill(subst);\r\n } catch (UnifyFailure uf) {\r\n // shouldn't happen\r\n throw new RuntimeException(\"Unable to fill lfPred: \" + uf);\r\n }\r\n // find matching pred\r\n String[] lfPredKeys = predKeys(lfPred);\r\n if (lfPredKeys.length == 0) {\r\n // nb: this means the lfPred is underconstrained; \r\n // will need to check it later!\r\n continue;\r\n }\r\n List matchingPredIndices = new ArrayList(3);\r\n for (int i = 0; i < lfPredKeys.length; i++) {\r\n List indices = predMap.get(lfPredKeys[i]);\r\n if (indices != null) matchingPredIndices.addAll(indices);\r\n }\r\n if (matchingPredIndices.isEmpty()) {\r\n \tif (useRelaxedRelationMatching && HyloHelper.isRelPred(lfPred)) continue; // skip\r\n \telse return null; // fail\r\n }\r\n // try extending each subst/bitset: \r\n // first swap retval, prev, and clear retval\r\n List> tmp = prev;\r\n prev = retval; retval = tmp; \r\n retval.clear();\r\n for (Pair inst : prev) {\r\n Substitution s = inst.a; BitSet b = inst.b;\r\n if (matchingPredIndices.size() == 1) { // reuse current instantiation\r\n int matchingPredIndex = matchingPredIndices.get(0);\r\n b.set(matchingPredIndex);\r\n if (checkAlts(b)) {\r\n try { // unify\r\n \tSatOp matchingPred = preds.get(matchingPredIndex);\r\n Unifier.unify(lfPred, matchingPred, s);\r\n retval.add(inst);\r\n } catch (UnifyFailure uf) {}\r\n }\r\n }\r\n else { // otherwise make copies\r\n for (int matchingPredIndex : matchingPredIndices) {\r\n Substitution s2 = new SimpleSubstitution((SimpleSubstitution)s);\r\n BitSet b2 = (BitSet)b.clone();\r\n b2.set(matchingPredIndex);\r\n if (checkAlts(b2)) {\r\n try { // unify\r\n \tSatOp matchingPred = preds.get(matchingPredIndex);\r\n Unifier.unify(lfPred, matchingPred, s2);\r\n Pair inst2 = new Pair(s2, b2);\r\n retval.add(inst2);\r\n } catch (UnifyFailure uf) {}\r\n }\r\n }\r\n }\r\n }\r\n if (retval.isEmpty()) {\r\n \tif (useRelaxedRelationMatching && HyloHelper.isRelPred(lfPred)) {\r\n \t\tretval.addAll(prev);\r\n \t\tcontinue; // skip\r\n \t}\r\n \telse return null; // fail\r\n }\r\n it.remove();\r\n }\r\n }\r\n \r\n // check for no more than one (rel) pred left over\r\n if (remainingPreds.size() > 1) return null;\r\n // done\r\n return retval;\r\n }\r\n \r\n \r\n // returns true iff no alt exclusions are violated\r\n // nb: needs to check that if there any intersections \r\n // with multiple alts, then these are only in the shared part\r\n private boolean checkAlts(BitSet b) {\r\n for (List altSet : lfAlts) {\r\n int intersects = 0;\r\n for (Alt alt : altSet) {\r\n if (alt.bitset.intersects(b)) intersects++;\r\n }\r\n if (intersects > 1) { // check intersections \r\n for (int i = 0; i < altSet.size(); i++) {\r\n Alt alt = altSet.get(i);\r\n if (alt.bitset.intersects(b)) {\r\n for (int j = i+1; j < altSet.size(); j++) {\r\n Alt alt2 = altSet.get(j);\r\n if (alt2.bitset.intersects(b)) {\r\n BitSet altOnly = (BitSet) alt.bitset.clone();\r\n altOnly.andNot(alt2.bitset);\r\n BitSet alt2Only = (BitSet) alt2.bitset.clone();\r\n alt2Only.andNot(alt.bitset);\r\n if (altOnly.intersects(b) && alt2Only.intersects(b))\r\n return false;\r\n }\r\n }\r\n }\r\n }\r\n }\r\n }\r\n return true;\r\n }\r\n \r\n\r\n //-----------------------------------------------------------------\r\n // createNewEdges\r\n //\r\n\r\n /**\r\n * Returns all edges that can be created by combining the given edges, \r\n * without collecting combos.\r\n */\r\n public List createNewEdges(Edge edge, Edge next) {\r\n return createNewEdges(edge, next, false);\r\n }\r\n \r\n /**\r\n * Returns all edges that can be created by combining the given edges;\r\n * if the collectCombos flag is true, the edges are updated with collected combos, \r\n * and additional alt edges are made for the remaining alternative edges for \r\n * the given first edge.\r\n */\r\n public List createNewEdges(Edge edge, Edge next, boolean collectCombos) {\r\n \r\n // check for sem overlap\r\n if (edge.intersects(next)) return Collections.emptyList();\r\n // check LF chunk constraints\r\n if (useChunks) {\r\n if (!edge.meetsLfChunkConstraints(next) || \r\n !next.meetsLfChunkConstraints(edge)) return Collections.emptyList();\r\n }\r\n \r\n // make new edges ...\r\n List newEdges = null;\r\n // when using indexing:\r\n if (useIndexing) {\r\n // check for intersecting indices\r\n if (edge.indicesIntersect(next)) {\r\n newEdges = createNewEdges(edge, next, collectCombos, true);\r\n }\r\n // check for PairedWith relation\r\n else if (anyPairedNominals && pairedWith(edge, next)) {\r\n newEdges = createNewEdges(edge, next, collectCombos, false);\r\n }\r\n else if (anyPairedNominals && pairedWith(next, edge)) {\r\n newEdges = createNewEdges(next, edge, collectCombos, false);\r\n }\r\n\t // check for a missing index nominal on the target cat,\r\n\t // which can indicate a type-raised category that needs to combine\r\n\t // before its indices become adjacent\r\n else if (allowMissingIndexCombos && (edge.getIndexNominal() == null || next.getIndexNominal() == null))\r\n\t {\r\n newEdges = createNewEdges(edge, next, collectCombos, true);\r\n }\r\n else { return Collections.emptyList(); }\r\n } \r\n // otherwise try everything\r\n else {\r\n newEdges = createNewEdges(edge, next, collectCombos, true);\r\n }\r\n \r\n // make alt edges for rest of edge's alts, with collectCombos option\r\n if (collectCombos && edge.altEdges.size() > 0) {\r\n int numNewEdges = newEdges.size(); // get num before adding any more\r\n for (int i = 0; i < numNewEdges; i++) {\r\n Edge resultEdge = newEdges.get(i);\r\n Sign resultSign = resultEdge.sign;\r\n Category resultCat = resultSign.getCategory();\r\n Rule rule = resultSign.getDerivationHistory().getRule();\r\n Sign[] resultInputs = resultSign.getDerivationHistory().getInputs(); \r\n boolean rightward = (resultInputs[0] == next.sign);\r\n boolean lefthead = (resultSign.getLexHead() == resultInputs[0].getLexHead());\r\n for (int j = 0; j < edge.altEdges.size(); j++) {\r\n Edge furtherEdge = edge.altEdges.get(j);\r\n if (furtherEdge == edge) continue;\r\n Sign[] signs = (rightward) \r\n ? new Sign[] { next.sign, furtherEdge.sign }\r\n : new Sign[] { furtherEdge.sign, next.sign };\r\n Sign lexHead = (rightward == lefthead) \r\n \t? next.sign.getLexHead() \r\n \t\t\t: furtherEdge.sign.getLexHead(); \r\n \tSign altSign = Sign.createDerivedSignWithNewLF(resultCat, signs, rule, lexHead);\r\n newEdges.add(makeAltEdge(altSign, resultEdge));\r\n }\r\n }\r\n }\r\n \r\n // check instantiation of outermost cats\r\n checkInstantiation(newEdges);\r\n \r\n // done\r\n return newEdges;\r\n }\r\n\r\n // creates edges, combining in one or both directions per flag\r\n private List createNewEdges(Edge edgeA, Edge edgeB, boolean collectCombos, boolean bothDirections) {\r\n\r\n // get combined alts, checking compatibility \r\n List> combinedLfAlts = getCombinedLfAlts(edgeA.activeLfAlts, edgeB.activeLfAlts);\r\n if (combinedLfAlts == null) return Collections.emptyList();\r\n \r\n // check whether a chunk is completed when gluing fragments\r\n boolean fragCompletion = false;\r\n if (gluingFragments) fragCompletion = completesChunk(edgeA, edgeB);\r\n \r\n // A B combos\r\n List results;\r\n if (gluingFragments) results = generalRules.applyGlueRule(edgeA.sign, edgeB.sign);\r\n else results = generalRules.applyBinaryRules(edgeA.sign, edgeB.sign);\r\n binaryRuleApps++; \r\n int numResults = results.size();\r\n \r\n // B A combos\r\n List reversedResults = Collections.emptyList();\r\n if (bothDirections) {\r\n \tif (gluingFragments) reversedResults = generalRules.applyGlueRule(edgeB.sign, edgeA.sign);\r\n \telse reversedResults = generalRules.applyBinaryRules(edgeB.sign, edgeA.sign);\r\n binaryRuleApps++; \r\n }\r\n int numReversedResults = reversedResults.size();\r\n \r\n // make edges to return, updating edge combos (if apropos)\r\n List retval = Collections.emptyList();\r\n if (numResults + numReversedResults > 0) {\r\n retval = new ArrayList(numResults + numReversedResults);\r\n BitSet union = (BitSet) edgeA.bitset.clone();\r\n union.or(edgeB.bitset);\r\n int cardBefore = union.cardinality();\r\n List> activeLfAlts = getActiveLfAlts(combinedLfAlts, union);\r\n // check for alt completion when gluing fragments\r\n if (gluingFragments && union.cardinality() > cardBefore) fragCompletion = true;\r\n for (int i = 0; i < numResults; i++) {\r\n Sign sign = results.get(i);\r\n if (fragCompletion) { ((AtomCat)sign.getCategory()).fragCompletion = true; }\r\n Edge resultEdge = makeEdge(sign, union, activeLfAlts); \r\n retval.add(resultEdge);\r\n if (collectCombos) {\r\n edgeA.edgeCombos.addRightwardCombo(edgeB, resultEdge);\r\n edgeB.edgeCombos.addLeftwardCombo(edgeA, resultEdge);\r\n }\r\n }\r\n for (int i = 0; i < numReversedResults; i++) {\r\n Sign sign = reversedResults.get(i);\r\n if (fragCompletion) { ((AtomCat)sign.getCategory()).fragCompletion = true; }\r\n Edge resultEdge = makeEdge(sign, union, activeLfAlts); \r\n retval.add(resultEdge);\r\n if (collectCombos) {\r\n edgeB.edgeCombos.addRightwardCombo(edgeA, resultEdge);\r\n edgeA.edgeCombos.addLeftwardCombo(edgeB, resultEdge);\r\n }\r\n }\r\n }\r\n \r\n // done\r\n return retval;\r\n }\r\n\r\n \r\n /**\r\n * Returns all edges that can be created by applying a unary rule \r\n * to the given edge or by combining it with a purely syntactic edge, \r\n * without collecting combos.\r\n */\r\n public List createNewEdges(Edge edge) {\r\n return createNewEdges(edge, false);\r\n }\r\n \r\n /**\r\n * Returns all edges that can be created by applying a unary rule \r\n * to the given edge, or by combining it with a purely syntactic edge,\r\n * or by completing a realization/chunk/alt with an optional part, \r\n * while updating the given edge with collected combos, \r\n * if the collectCombos flag is true.\r\n * When gluing fragments, only the opt completion step is done.\r\n */\r\n public List createNewEdges(Edge edge, boolean collectCombos) {\r\n \r\n List retval = null; // instantiate on demand\r\n \r\n if (!gluingFragments) {\r\n\t \t\r\n\t List genResults = generalRules.applyUnaryRules(edge.sign);\r\n\t unaryRuleApps++;\r\n\t // make edges for results, updating edge combos\r\n\t if (genResults.size() > 0) {\r\n\t if (retval == null) retval = new ArrayList(genResults.size());\r\n\t for (int i = 0; i < genResults.size(); i++) {\r\n\t Sign sign = genResults.get(i);\r\n\t\t\t\t\t// check for unary rule cycle; skip result if found\r\n\t if (sign.getDerivationHistory().containsCycle()) continue;\r\n\t Edge resultEdge = makeEdge(sign, edge.bitset, edge.activeLfAlts); \r\n\t retval.add(resultEdge);\r\n\t if (collectCombos) edge.edgeCombos.unaryResults.add(resultEdge);\r\n\t }\r\n\t }\r\n\t \r\n\t // do rule instances\r\n\t Sign[] signs = { edge.sign };\r\n\t for (int i = 0; i < ruleInstances.size(); i++) {\r\n\t RuleInstance ruleInst = ruleInstances.get(i);\r\n\t // check sem overlap\r\n\t if (edge.intersects(ruleInst)) continue; \r\n\t // check for indices in common\r\n\t if (useIndexing && !edge.indicesIntersect(ruleInst)) continue; \r\n\t // check LF chunk constraints\r\n\t if (useChunks && !edge.meetsLfChunkConstraints(ruleInst)) continue;\r\n\t // get combined alts, checking compatibility \r\n\t List> combinedLfAlts = getCombinedLfAlts(edge.activeLfAlts, ruleInst.activeLfAlts);\r\n\t if (combinedLfAlts == null) continue;\r\n\t \r\n\t // apply rule\r\n\t List instResults = new ArrayList(1);\r\n\t ruleInst.rule.applyRule(signs, instResults);\r\n\t unaryRuleInstApps++;\r\n\t if (instResults.size() > 0) {\r\n\t if (retval == null) retval = new ArrayList(instResults.size());\r\n\t BitSet union = (BitSet) edge.bitset.clone();\r\n\t union.or(ruleInst.bitset);\r\n\t List> activeLfAlts = getActiveLfAlts(combinedLfAlts, union);\r\n\t for (int j = 0; j < instResults.size(); j++) {\r\n\t Sign sign = instResults.get(j);\r\n\t \t\t\t\t// check for unary rule cycle; skip result if found\r\n\t if (sign.getDerivationHistory().containsCycle()) continue;\r\n\t Edge resultEdge = makeEdge(sign, union, activeLfAlts); \r\n\t retval.add(resultEdge);\r\n\t if (collectCombos) edge.edgeCombos.unaryResults.add(resultEdge);\r\n\t }\r\n\t }\r\n\t }\r\n }\r\n \r\n // do opt completed edges\r\n if (!lfOpts.isEmpty() && !edge.complete()) {\r\n // get completed bitsets for each completed active alt or chunk, and for whole thing\r\n List optCompleted = new ArrayList(2);\r\n addOptCompletedBitSet(edge, allPreds, optCompleted);\r\n for (List altSet : edge.activeLfAlts) {\r\n for (Alt alt : altSet) {\r\n addOptCompletedBitSet(edge, alt.bitset, optCompleted);\r\n }\r\n }\r\n for (BitSet chunk : lfChunks) {\r\n addOptCompletedBitSet(edge, chunk, optCompleted);\r\n }\r\n // for each completed bitset, make complete edge with same sign\r\n for (BitSet completed : optCompleted) {\r\n List> activeLfAlts = getActiveLfAlts(edge.activeLfAlts, completed);\r\n // set frag completion if apropos\r\n if (gluingFragments && edge.sign.getCategory() instanceof AtomCat) {\r\n \tAtomCat ac = (AtomCat) edge.sign.getCategory();\r\n \tif (ac.isFragment()) ac.fragCompletion = true;\r\n }\r\n Edge resultEdge = makeEdge(edge.sign, completed, activeLfAlts);\r\n resultEdge.optCompletes = edge;\r\n if (retval == null) retval = new ArrayList(1);\r\n retval.add(resultEdge);\r\n if (collectCombos) edge.edgeCombos.optionalResults.add(resultEdge);\r\n }\r\n }\r\n\r\n // ensure retval instantiated \r\n if (retval == null) retval = Collections.emptyList();\r\n \r\n // check instantiation of outermost cats\r\n if (!gluingFragments) checkInstantiation(retval);\r\n \r\n // done\r\n return retval;\r\n }\r\n\r\n // bitset for checking completeness\r\n private BitSet tmpBitSetCompleteness = new BitSet();\r\n \r\n // bitset for making retval\r\n private BitSet tmpBitSetRetval = new BitSet();\r\n \r\n // adds a bitset with optional parts completed within the given bitset scope \r\n // to the given list, if the optional parts complete the given edge's bitset\r\n private void addOptCompletedBitSet(Edge edge, BitSet bitset, List optCompleted) {\r\n // check whether already complete\r\n \ttmpBitSetRetval.clear(); tmpBitSetRetval.or(edge.bitset);\r\n \ttmpBitSetRetval.and(bitset);\r\n if (tmpBitSetRetval.cardinality() == bitset.cardinality()) return;\r\n tmpBitSetRetval.or(edge.bitset);\r\n // or retval with opts when apropos\r\n for (BitSet opt : lfOpts) {\r\n if (subset(opt, bitset)) {\r\n \tif (edge.bitset.intersects(opt)) continue; // skip if opt not entirely missing\r\n \ttmpBitSetRetval.or(opt);\r\n }\r\n }\r\n // check completeness, add retval if complete (and distinct)\r\n tmpBitSetCompleteness.clear(); tmpBitSetCompleteness.or(bitset); \r\n tmpBitSetCompleteness.and(tmpBitSetRetval);\r\n if (tmpBitSetCompleteness.cardinality() == bitset.cardinality()) {\r\n if (!optCompleted.contains(tmpBitSetRetval)) \r\n \toptCompleted.add((BitSet)tmpBitSetRetval.clone());\r\n }\r\n }\r\n \r\n \r\n /** Returns the edges that can be made by constructing alternative edges \r\n from the given edge and the collected combos in its representative edge. */\r\n public List createAltEdges(Edge edge, Edge repEdge) {\r\n // instantiate return list with right capacity\r\n EdgeCombos edgeCombos = repEdge.edgeCombos;\r\n int numResults = numResultsFromCombos(edgeCombos.rightwardCombos);\r\n numResults += numResultsFromCombos(edgeCombos.leftwardCombos);\r\n numResults += edgeCombos.unaryResults.size();\r\n numResults += edgeCombos.optionalResults.size();\r\n List retval = new ArrayList(numResults);\r\n // make alt edges\r\n addAltsFromCombos(edge, edgeCombos.rightwardCombos, true, retval);\r\n addAltsFromCombos(edge, edgeCombos.leftwardCombos, false, retval);\r\n addAltsFromUnaryResults(edge, edgeCombos.unaryResults, retval);\r\n addAltsFromOptionalResults(edge, edgeCombos.optionalResults, retval);\r\n // done\r\n return retval;\r\n }\r\n \r\n // returns the number of results from the given combos\r\n private int numResultsFromCombos(List combos) {\r\n int retval = 0;\r\n for (int i = 0; i < combos.size(); i++) {\r\n EdgeCombos.CatCombo combo = combos.get(i);\r\n retval += combo.inputEdge.altEdges.size();\r\n }\r\n return retval;\r\n }\r\n \r\n // adds alt edges for the given edge, combos, and direction to results\r\n private void addAltsFromCombos(Edge edge, List combos, boolean rightward, List results) {\r\n for (EdgeCombos.CatCombo combo : combos) {\r\n Edge resultEdge = combo.resultEdge;\r\n Sign resultSign = resultEdge.sign;\r\n Category resultCat = resultSign.getCategory();\r\n Rule rule = resultSign.getDerivationHistory().getRule();\r\n Sign[] resultInputs = resultSign.getDerivationHistory().getInputs(); \r\n boolean lefthead = (resultSign.getLexHead() == resultInputs[0].getLexHead());\r\n List comboEdges = combo.inputEdge.altEdges;\r\n for (Edge comboEdge : comboEdges) {\r\n Sign[] signs = (rightward) \r\n ? new Sign[] { edge.sign, comboEdge.sign }\r\n : new Sign[] { comboEdge.sign, edge.sign };\r\n Sign lexHead = (rightward == lefthead) \r\n \t? edge.sign.getLexHead() \r\n \t\t\t: comboEdge.sign.getLexHead();\r\n Sign altSign = Sign.createDerivedSignWithNewLF(resultCat, signs, rule, lexHead);\r\n results.add(makeAltEdge(altSign, resultEdge));\r\n }\r\n }\r\n }\r\n \r\n // adds alt edges for the given edge and unary results to results\r\n private void addAltsFromUnaryResults(Edge edge, List unaryResults, List results) {\r\n for (Edge resultEdge : unaryResults) {\r\n Sign resultSign = resultEdge.sign;\r\n Category resultCat = resultSign.getCategory();\r\n Rule rule = resultSign.getDerivationHistory().getRule();\r\n Sign[] signs = { edge.sign };\r\n Sign lexHead = edge.sign.getLexHead();\r\n Sign altSign = Sign.createDerivedSignWithNewLF(resultCat, signs, rule, lexHead);\r\n results.add(makeAltEdge(altSign, resultEdge));\r\n }\r\n }\r\n \r\n // adds alt edges for the given edge and optional results to results\r\n private void addAltsFromOptionalResults(Edge edge, List optionalResults, List results) {\r\n for (Edge resultEdge : optionalResults) { \r\n results.add(makeAltEdge(edge.sign, resultEdge));\r\n }\r\n }\r\n \r\n \r\n /** Returns the number of rule applications executed. */\r\n public int ruleApps() {\r\n return \r\n unaryRuleApps * generalRules.getUnaryRules().size() +\r\n unaryRuleInstApps +\r\n binaryRuleApps * generalRules.getBinaryRules().size();\r\n }\r\n \r\n\r\n //-----------------------------------------------------------------\r\n // initGeneralRules\r\n //\r\n\r\n // separates out general rules with no semantics\r\n // nb: could consider adding feature licensing for type changing rules with no semantics\r\n private void initGeneralRules() {\r\n // add all binary rules to general rules\r\n for (Rule r : grammar.rules.getBinaryRules()) {\r\n generalRules.addRule(r);\r\n }\r\n // add type raising rules, and type changing ones with no semantics too\r\n for (Rule r : grammar.rules.getUnaryRules()) {\r\n // skip type changing rules with semantics\r\n if (r instanceof TypeChangingRule) {\r\n TypeChangingRule rule = (TypeChangingRule) r;\r\n if (rule.getResult().getLF() != null) { continue; }\r\n }\r\n // otherwise add it\r\n generalRules.addRule(r);\r\n }\r\n }\r\n \r\n \r\n //-----------------------------------------------------------------\r\n // initNoSemEdges\r\n //\r\n\r\n // creates edges for signs flagged as having no semantics, \r\n // and with appropriate licensing values in the initial edges\r\n private void initNoSemEdges() {\r\n // lookup signs by special index rel constant NO_SEM_FLAG\r\n lexicon.setSupertagger(null); // turn off hypertagger first\r\n Collection noSemSigns = lexicon.getSignsFromRel(Lexicon.NO_SEM_FLAG);\r\n lexicon.setSupertagger(hypertagger); // reset hypertagger\r\n if (noSemSigns == null) return;\r\n // sets for accumulating no sem edges\r\n Set instEdges = new HashSet();\r\n Set uninstEdges = new HashSet();\r\n // add signs with no LF and with matching licensing values\r\n Set instantiatedCats = new HashSet();\r\n Set uninstantiatedCats = new HashSet();\r\n List> emptyLfAlts = Collections.emptyList();\r\n // loop until no more no sem edges\r\n int numInstEdges, numUninstEdges;\r\n do {\r\n \tnumInstEdges = instEdges.size(); numUninstEdges = uninstEdges.size();\r\n\t for (Sign sign : noSemSigns) {\r\n\t Category cat = sign.getCategory();\r\n\t // get licensed, potentially instantiated cats\r\n\t instantiatedCats.clear();\r\n\t uninstantiatedCats.clear();\r\n\t featureLicenser.licenseEmptyCat(cat, instantiatedCats, uninstantiatedCats);\r\n\t // add edges for instantiated cats to initial edges, updating\r\n\t\t\t\t// feature map\r\n\t for (Category instCat : instantiatedCats) {\r\n featureLicenser.updateFeatureMap(instCat);\r\n featureLicenser.indexSemanticallyNullWords(instCat);\r\n\t Sign instSign = new Sign(sign.getWords(), instCat);\r\n\t instEdges.add(makeEdge(instSign, new BitSet(preds.size()), emptyLfAlts));\r\n\t }\r\n\t // add edges for uninstantiated cats to no-sem edges, updating\r\n\t\t\t\t// feature map\r\n\t for (Category uninstCat : uninstantiatedCats) {\r\n featureLicenser.updateFeatureMap(uninstCat);\r\n featureLicenser.indexSemanticallyNullWords(uninstCat);\r\n\t Sign uninstSign = new Sign(sign.getWords(), uninstCat);\r\n\t Edge noSemEdge = makeEdge(uninstSign, new BitSet(preds.size()), emptyLfAlts); \r\n\t uninstEdges.add(noSemEdge);\r\n\t }\r\n\t }\r\n } while (numInstEdges != instEdges.size() || numUninstEdges != uninstEdges.size());\r\n // update no sem edge lists\r\n instantiatedNoSemEdges.addAll(instEdges);\r\n noSemEdges.addAll(uninstEdges);\r\n }\r\n}\r\n\r\n" }, { "path": "src/opennlp/ccg/realize/EdgeHash.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005 Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.realize;\n\nimport gnu.trove.*;\nimport java.util.*;\n\n/**\n * A set of edges, unique up to surface words.\n * Edges with higher scores or whose signs have lower derivational complexity are kept during insertion.\n *\n * @author Michael White\n * @version $Revision: 1.2 $, $Date: 2010/01/14 22:52:01 $\n */\npublic class EdgeHash extends THashSet {\n\n\tprivate static final long serialVersionUID = 1L;\n\t\n\t/** Hashing strategy that uses Edge's surfaceWordHashCode and surfaceWordEquals methods. */\n protected static TObjectHashingStrategy surfaceWordHashingStrategy = new TObjectHashingStrategy() {\n\t\tprivate static final long serialVersionUID = 1L;\n\t\tpublic int computeHashCode(java.lang.Object o) {\n return ((Edge)o).surfaceWordHashCode();\n }\n public boolean equals(java.lang.Object o1, java.lang.Object o2) {\n return ((Edge)o1).surfaceWordEquals((Edge)o2);\n }\n };\n\n /** Default constructor. */\n public EdgeHash() { super(surfaceWordHashingStrategy); }\n\n /**\n * Returns this as a set of edges.\n */\n @SuppressWarnings(\"unchecked\")\n\tpublic Set asEdgeSet() { return (Set) this; }\n\n /**\n * Adds an edge, keeping the one with a higher score or whose sign has lower derivational complexity \n * if there is an equivalent one there already; returns the old\n * edge if it was displaced, the new edge if there was no equivalent \n * old edge, or null if the edge was not actually added.\n */\n public Edge insert(Edge edge) {\n int pos = index(edge);\n // equiv edge\n if (pos >= 0) {\n Edge oldEdge = (Edge) _set[pos];\n // already there?\n if (oldEdge == edge) return null;\n // check score\n if (edge.score > oldEdge.score) { \n \t_set[pos] = edge; return oldEdge;\n }\n // check complexity\n int complexity = edge.sign.getDerivationHistory().complexity();\n int oldComplexity = oldEdge.sign.getDerivationHistory().complexity();\n if (complexity < oldComplexity) {\n \t_set[pos] = edge; return oldEdge;\n }\n // otherwise toss\n else return null;\n }\n // add new\n else {\n \tadd(edge); return edge;\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/realize/FeatureLicenser.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2003-4 University of Edinburgh (Michael White)\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.realize;\r\n\r\nimport opennlp.ccg.lexicon.*;\r\nimport opennlp.ccg.unify.*;\r\nimport opennlp.ccg.synsem.*;\r\nimport opennlp.ccg.hylo.*;\r\n\r\nimport java.util.*;\r\n\r\n/**\r\n * The feature licenser is a helper class for the edge factory, \r\n * responsible for managing features which license the use and \r\n * instantiation of semantically null or marked categories.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.13 $, $Date: 2009/12/21 03:27:18 $\r\n */\r\npublic class FeatureLicenser\r\n{\r\n\r\n // the edge factory for which this feature licenser is a helper\r\n private final EdgeFactory edgeFactory;\r\n \r\n // the licensing features \r\n private final LicensingFeature[] licensingFeatures;\r\n \r\n /** Constructor. */\r\n public FeatureLicenser(EdgeFactory edgeFactory) { \r\n this.edgeFactory = edgeFactory; \r\n this.licensingFeatures = edgeFactory.grammar.lexicon.getLicensingFeatures();\r\n }\r\n\r\n /** Constructor with licensing features. */\r\n public FeatureLicenser(EdgeFactory edgeFactory, LicensingFeature[] licensingFeatures) { \r\n this.edgeFactory = edgeFactory; \r\n this.licensingFeatures = licensingFeatures;\r\n }\r\n\r\n \r\n //-----------------------------------------------------------------\r\n // semantically null word indexing\r\n\r\n /**\r\n * Adds new nominal atoms for subcategorized semantically null words. \r\n * A check is made for atomic categories with a value for the 'lex' \r\n * feature but with a null or uninstantiated index feature. If found, a new nominal atom is \r\n * created as the value of the index feature, and the nominal is \r\n * added to the edge factory's nominals map, for indexing purposes. \r\n * The same nominal is reused for repeated occurrences of a 'lex' value.\r\n */\r\n public void indexSemanticallyNullWords(Category cat) {\r\n cat.forall(semanticallyNullWordIndexer);\r\n }\r\n \r\n // counter\r\n private int wordCounter = 0;\r\n \r\n // 'lex' value to index map\r\n private Map wordIndexMap = new HashMap();\r\n \r\n // cat function\r\n private CategoryFcn semanticallyNullWordIndexer = new CategoryFcnAdapter() {\r\n public void forall(Category c) {\r\n if (!(c instanceof AtomCat)) return;\r\n FeatureStructure fs = c.getFeatureStructure();\r\n if (fs == null) return;\r\n if (!fs.hasAttribute(\"lex\")) return;\r\n Object indexVal = fs.getValue(\"index\"); \r\n if (indexVal == null || (indexVal instanceof NominalVar)) {\r\n String lexVal = fs.getValue(\"lex\").toString();\r\n String index = wordIndexMap.get(lexVal);\r\n NominalAtom nom;\r\n if (index == null) {\r\n do {\r\n index = \"w\" + ++wordCounter;\r\n nom = new NominalAtom(index);\r\n } while (edgeFactory.nominals.containsKey(nom));\r\n wordIndexMap.put(lexVal, index);\r\n edgeFactory.nominals.put(nom, edgeFactory.nominals.size());\r\n }\r\n else nom = new NominalAtom(index);\r\n fs.setFeature(\"index\", nom);\r\n }\r\n }\r\n };\r\n \r\n\r\n //-----------------------------------------------------------------\r\n // feature map \r\n //\r\n \r\n /**\r\n * Updates the licensed feature map with the info from the given initial category.\r\n */\r\n public void updateFeatureMap(Category cat) {\r\n currentFeatureMap = featureMap; \r\n cat.forall(featureMapUpdater);\r\n currentFeatureMap = null;\r\n }\r\n \r\n // updates the category-specific licensed feature map\r\n private void updateCatFeatureMap(Category cat) {\r\n catFeatureMap.clear(); \r\n currentFeatureMap = catFeatureMap; \r\n cat.forall(featureMapUpdater);\r\n currentFeatureMap = null;\r\n }\r\n \r\n // a map from an attr name to a map from vals to sets of atomic categories \r\n // containing those attr-val pairs\r\n private Map>> featureMap = new HashMap>>();\r\n \r\n // a feature map for a specific category to be checked\r\n private Map>> catFeatureMap = new HashMap>>();\r\n \r\n // working feature map\r\n private Map>> currentFeatureMap = null; \r\n \r\n // list of all initial atom cats checked for main feature map\r\n private List allInitialAtomCats = new ArrayList();\r\n \r\n // feature map updater \r\n private CategoryFcn featureMapUpdater = new CategoryFcnAdapter() {\r\n public void forall(Category c) {\r\n if (!(c instanceof AtomCat)) return;\r\n if (currentFeatureMap == featureMap) allInitialAtomCats.add(c);\r\n FeatureStructure fs = c.getFeatureStructure();\r\n if (fs == null) return;\r\n // for each feature\r\n for (int i = 0; i < licensingFeatures.length; i++) {\r\n String attr = licensingFeatures[i].attr; \r\n Object val = fs.getValue(attr);\r\n if (val != null && !(val instanceof Variable)) {\r\n // check for relevant value\r\n String valStr = val.toString(); \r\n String fVal = licensingFeatures[i].val;\r\n List alsoList = licensingFeatures[i].alsoLicensedBy;\r\n if (fVal != null && !fVal.equals(valStr) && !alsoList.contains(valStr)) continue;\r\n // add to feature map\r\n Map> valMap = currentFeatureMap.get(attr); \r\n if (valMap == null) {\r\n valMap = new HashMap>(); \r\n currentFeatureMap.put(attr, valMap);\r\n }\r\n Set acSet = valMap.get(valStr);\r\n if (acSet == null) {\r\n acSet = new HashSet();\r\n valMap.put(valStr, acSet);\r\n }\r\n acSet.add(c);\r\n }\r\n }\r\n }\r\n };\r\n\r\n \r\n //-----------------------------------------------------------------\r\n // category licensing \r\n //\r\n \r\n /**\r\n * Returns whether the given category contains a feature indicating that \r\n * it needs to be licensed.\r\n */\r\n public boolean needsLicensing(Category cat) {\r\n return checkLicensing(cat, true);\r\n }\r\n\r\n /**\r\n * Returns whether the given category is licensed according to \r\n * the current feature map.\r\n */\r\n public boolean isLicensed(Category cat) {\r\n return checkLicensing(cat, false);\r\n }\r\n\r\n // records the licensing feature which succeeded in licensing the last cat (or null if none)\r\n private LicensingFeature currentLicensingFeature = null; \r\n \r\n // checks the given category according to the given flag, \r\n // updating currentLicensingFeature\r\n private boolean checkLicensing(Category cat, boolean needsLicensing) {\r\n currentLicensingFeature = null;\r\n boolean emptyCat = (cat.getLF() == null);\r\n // set up cat feature map\r\n updateCatFeatureMap(cat);\r\n Category target = getTarget(cat);\r\n // for each feature, look for appropriate attr-val pairs\r\n for (int i = 0; i < licensingFeatures.length; i++) {\r\n // skip when appropriate license flag not set\r\n if (emptyCat && !licensingFeatures[i].licenseEmptyCats) continue; \r\n if (!emptyCat && !licensingFeatures[i].licenseMarkedCats) continue;\r\n String attr = licensingFeatures[i].attr;\r\n Map> valMap = catFeatureMap.get(attr);\r\n if (valMap == null) continue;\r\n String fVal = licensingFeatures[i].val;\r\n Collection vals;\r\n if (fVal != null) {\r\n if (!valMap.containsKey(fVal)) continue;\r\n vals = new ArrayList(1);\r\n vals.add(fVal);\r\n } else {\r\n vals = valMap.keySet();\r\n }\r\n byte loc = licensingFeatures[i].loc;\r\n // for each attr-val pair\r\n for (Iterator it = vals.iterator(); it.hasNext(); ) {\r\n String val = it.next();\r\n Set atomCats = valMap.get(val); \r\n // check loc\r\n if (loc == LicensingFeature.TARGET_ONLY) {\r\n if (atomCats.size() != 1) continue;\r\n if (!atomCats.contains(target)) continue;\r\n } else if (loc == LicensingFeature.ARGS_ONLY) {\r\n if (atomCats.contains(target)) continue;\r\n }\r\n // branch on needs-licensing flag\r\n if (needsLicensing) {\r\n // found a feature needing to be licensed\r\n return true; \r\n }\r\n else {\r\n // check for licensing feature in feature map\r\n Map> fmValMap = featureMap.get(attr);\r\n // return false if not found\r\n if (fmValMap == null) return false;\r\n boolean foundLicensingVal = fmValMap.containsKey(val);\r\n if (!foundLicensingVal) {\r\n List alsoList = licensingFeatures[i].alsoLicensedBy;\r\n for (int j = 0; j < alsoList.size(); j++) {\r\n if (fmValMap.containsKey(alsoList.get(j))) { \r\n foundLicensingVal = true; break;\r\n }\r\n }\r\n }\r\n if (!foundLicensingVal) return false; \r\n // otherwise record licensing feature and return true\r\n currentLicensingFeature = licensingFeatures[i]; \r\n return true;\r\n }\r\n }\r\n }\r\n // otherwise false\r\n return false;\r\n }\r\n\r\n // returns the target cat, if complex, otherwise \r\n // just the cat itself \r\n private Category getTarget(Category cat) {\r\n Category target = cat;\r\n if (cat instanceof ComplexCat) { target = ((ComplexCat)cat).getTarget(); }\r\n return target;\r\n }\r\n\r\n \r\n //-----------------------------------------------------------------\r\n // empty (semantically null) category licensing and instantiation \r\n //\r\n \r\n // reusable simple substitution for instantiating vars on atom cats\r\n private SimpleSubstitution simpleSubst = new SimpleSubstitution();\r\n\r\n /**\r\n * Determines whether the given semantically null category \r\n * is licensed according to the licensed feature map, and if so, returns \r\n * appropriately (un-)instantiated versions of the category. \r\n * The licensing features are checked in priority order.\r\n * NB: Instantiation is limited to the case where there is a single \r\n * value for the operative licensing feature.\r\n */\r\n public void licenseEmptyCat(Category cat, Set instantiatedCats, Set uninstantiatedCats) {\r\n // reindex\r\n UnifyControl.reindex(cat);\r\n // return cat uninstantiated if no licensing features found\r\n if (!needsLicensing(cat)) {\r\n uninstantiatedCats.add(cat); return; \r\n }\r\n // return nothing if not licensed\r\n if (!isLicensed(cat)) return;\r\n // find operative licensing feature, if necessary\r\n if (currentLicensingFeature == null) {\r\n for (int i = 0; i < licensingFeatures.length; i++) {\r\n if (!catFeatureMap.containsKey(licensingFeatures[i].attr)) continue;\r\n Map> valMap = catFeatureMap.get(licensingFeatures[i].attr);\r\n String fVal = licensingFeatures[i].val;\r\n if (fVal != null && !valMap.containsKey(fVal)) continue;\r\n currentLicensingFeature = licensingFeatures[i]; \r\n break;\r\n }\r\n // if still not found, return cat uninstantiated\r\n if (currentLicensingFeature == null) {\r\n uninstantiatedCats.add(cat); \r\n return; \r\n }\r\n }\r\n // return cat uninstantiated if licensing feature does not \r\n // have instantiation flag set\r\n if (!currentLicensingFeature.instantiate) {\r\n uninstantiatedCats.add(cat); return; \r\n }\r\n // return cat uninstantiated if licensing feature has more than one val\r\n String attr = currentLicensingFeature.attr;\r\n Map> valMap = catFeatureMap.get(attr);\r\n if (valMap.size() > 1) {\r\n uninstantiatedCats.add(cat); return; \r\n }\r\n String val = valMap.keySet().iterator().next();\r\n Set atomCats = valMap.get(val);\r\n // for each atom cat, go ahead with instantiation ...\r\n for (Iterator acIt = atomCats.iterator(); acIt.hasNext(); ) {\r\n Category ac = acIt.next(); \r\n // ensure cats with lex feature have an index var\r\n FeatureStructure fs = ac.getFeatureStructure();\r\n if (fs.hasAttribute(\"lex\") && !fs.hasAttribute(\"index\")) {\r\n fs.setFeature(\"index\", new NominalVar(\"W\"));\r\n UnifyControl.reindex(ac);\r\n }\r\n // unify with appropriate initial cats\r\n Collection initialCats = null; \r\n if (!currentLicensingFeature.licenseEmptyCats) \r\n initialCats = allInitialAtomCats;\r\n else { \r\n Map> fmValMap = featureMap.get(attr);\r\n initialCats = fmValMap.get(val);\r\n List alsoList = currentLicensingFeature.alsoLicensedBy;\r\n if (alsoList.size() > 0) { \r\n if (initialCats != null) initialCats = new HashSet(initialCats);\r\n else initialCats = new HashSet();\r\n for (int i = 0; i < alsoList.size(); i++) {\r\n Set alsoSet = fmValMap.get(alsoList.get(i));\r\n if (alsoSet != null) initialCats.addAll(alsoSet);\r\n }\r\n }\r\n }\r\n if (initialCats == null) {\r\n System.out.println(\"Warning, unable to find initial cats for feature \" + attr + \"=\" + val); \r\n uninstantiatedCats.add(cat); return; \r\n }\r\n // for each initial cat\r\n for (Iterator it = initialCats.iterator(); it.hasNext(); ) {\r\n Category initialAC = it.next();\r\n // ensure index instantiated\r\n FeatureStructure initialFS = initialAC.getFeatureStructure();\r\n if (initialFS == null) continue;\r\n Object index = initialFS.getValue(\"index\"); \r\n if (!(index instanceof NominalAtom)) continue;\r\n // block instantiation with bound vars\r\n if (edgeFactory.boundVarNominals.contains(index)) {\r\n instantiatedCats.clear();\r\n uninstantiatedCats.add(cat); \r\n return; \r\n }\r\n // try unifying index ...\r\n simpleSubst.clear(); \r\n try {\r\n Unifier.unify(ac.getFeatureStructure(), initialFS, simpleSubst);\r\n // ensure substitution contains index\r\n if (!simpleSubst.containsValue(index)) continue;\r\n // get rid of other substitutions\r\n for (Iterator it2 = simpleSubst.values().iterator(); it2.hasNext(); ) {\r\n if (!it2.next().equals(index)) it2.remove();\r\n }\r\n // instantiate\r\n Category instCat = (Category) cat.fill(simpleSubst);\r\n // and add instantiated cats\r\n instantiatedCats.add(instCat);\r\n }\r\n catch (UnifyFailure uf) {}\r\n }\r\n }\r\n }\r\n}\r\n \r\n" }, { "path": "src/opennlp/ccg/realize/Hypertagger.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2008-9 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.realize;\r\n\r\nimport opennlp.ccg.lexicon.*;\r\nimport opennlp.ccg.hylo.*;\r\nimport java.util.*;\r\n\r\n/**\r\n * A hypertagger is a realization supertagger. It must extend the \r\n * SupertaggerAdapter interface for plugging a supertagger into the \r\n * lexicon.\r\n * \r\n * @author Michael White\r\n * @version $Revision: 1.5 $, $Date: 2009/08/24 01:56:14 $\r\n */\r\npublic interface Hypertagger extends SupertaggerAdapter {\r\n\t\r\n\t/**\r\n\t * Maps the given elementary predications to their predicted categories, \r\n\t * so that the beta-best categories can be returned by calls to setPred\r\n\t * and getSupertags.\r\n\t */\r\n\tpublic void mapPreds(List preds);\r\n\t\r\n\t/**\r\n\t * Sets the current elementary predication to the one with the given index, \r\n\t * so that the beta-best categories for it can be returned by a call to \r\n\t * getSupertags.\r\n\t */\r\n\tpublic void setPred(int index);\r\n\t\r\n\t/** \r\n\t * Stores the gold standard pred info, for use in discriminative training.\r\n\t * The string consists of space delimited tokens, where each token \r\n\t * is a colon-separated list of fields, with the first field containing \r\n\t * the nominal id, and the second field the gold supertag.\r\n\t */\r\n\tpublic void storeGoldStdPredInfo(String goldStdPredInfo);\r\n}\r\n" }, { "path": "src/opennlp/ccg/realize/LexicalDiversityPruningStrategy.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2011 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.realize;\r\n\r\nimport opennlp.ccg.lexicon.Word;\r\nimport opennlp.ccg.synsem.Sign;\r\n\r\nimport gnu.trove.THashSet;\r\nimport gnu.trove.TObjectIdentityHashingStrategy;\r\n\r\nimport java.util.*;\r\n\r\n/**\r\n * A diversity pruning strategy that defines signs to be \r\n * notCompellinglyDifferent if the set of open class stems \r\n * is the same. The POS classes of interest are set in the \r\n * constructor.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.1 $, $Date: 2011/04/02 16:32:17 $\r\n */\r\npublic class LexicalDiversityPruningStrategy extends DiversityPruningStrategy\r\n{\r\n /** The interned POS values to use for relevant open class stems. */\r\n @SuppressWarnings(\"unchecked\")\r\n\tprotected Set posValsToUse = new THashSet(new TObjectIdentityHashingStrategy());\r\n \r\n /** Reusable set of observed interned stems for comparison purposes. */\r\n @SuppressWarnings(\"unchecked\")\r\n\tprotected Set stemsSeen = new THashSet(new TObjectIdentityHashingStrategy());\r\n \r\n /** Constructor, which sets POS classes of interest. */\r\n public LexicalDiversityPruningStrategy() {\r\n \tString[] poslist = {\r\n \t\t\"JJ\", \"JJR\", \"JJS\",\r\n \t\t\"NN\", \"NNP\", \"NNS\", \"NNPS\",\r\n \t\t\"RB\", \"RBR\", \"RBS\",\r\n \t\t\"VB\", \"VBD\", \"VBG\", \"VBN\", \"VBP\", \"VBZ\"\r\n \t};\r\n \tfor (String pos : poslist) posValsToUse.add(pos);\r\n }\r\n \r\n /** Returns true iff the given signs are not compellingly different.\r\n In particular, returns true iff the set of relevant open class stems are the same. */\r\n public boolean notCompellinglyDifferent(Sign sign1, Sign sign2) {\r\n \tstemsSeen.clear();\r\n \tfor (Word w : sign1.getWords()) {\r\n \t\tif (posValsToUse.contains(w.getPOS())) stemsSeen.add(w.getStem());\r\n \t}\r\n \tfor (Word w : sign2.getWords()) {\r\n \t\tif (posValsToUse.contains(w.getPOS()) && !stemsSeen.contains(w.getStem()))\r\n \t\t\treturn false;\r\n \t}\r\n return true;\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/realize/NBestPruningStrategy.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.realize;\n\nimport opennlp.ccg.TextCCG;\n\nimport java.util.*;\nimport java.util.prefs.*;\n\n/**\n * Default, n-best edge pruning strategy.\n *\n * @author Michael White\n * @version $Revision: 1.9 $, $Date: 2011/03/27 14:45:32 $\n */\npublic class NBestPruningStrategy implements PruningStrategy\n{\n /** The current pruning val. */\n protected int CAT_PRUNE_VAL;\n \n /** Reusable return list. */\n protected List retval = new ArrayList();\n \n /** Constructor with pruning val. */\n public NBestPruningStrategy(int pruningVal) {\n CAT_PRUNE_VAL = pruningVal;\n }\n \n /** Default constructor retrieves pruning val from preferences. */\n public NBestPruningStrategy() {\n Preferences prefs = Preferences.userNodeForPackage(TextCCG.class);\n CAT_PRUNE_VAL = prefs.getInt(Chart.PRUNING_VALUE, Chart.NO_PRUNING);\n }\n \n /**\n * Returns a (possibly empty) list of edges pruned \n * from the given ones, which should be sorted by score, \n * from highest to lowest. \n * In particular, prunes and returns the edges that follow the N-best \n * ones in the given list.\n */\n public List pruneEdges(List catEdges) {\n // clear reusable return list\n retval.clear();\n // ensure pruning enabled\n if (CAT_PRUNE_VAL == Chart.NO_PRUNING) return retval;\n // nb: could add an option to prune all egdes with zero score\n /*\n for (Iterator it = catEdges.iterator(); it.hasNext(); ) {\n Edge edge = it.next();\n if (edge.score == 0) {\n retval.add(edge);\n it.remove();\n }\n }\n */\n // return edges at bottom of list, starting with CAT_PRUNE_VAL (if any)\n while (CAT_PRUNE_VAL < catEdges.size()) {\n retval.add(catEdges.remove(CAT_PRUNE_VAL));\n }\n // done\n return retval;\n }\n}\n\n" }, { "path": "src/opennlp/ccg/realize/PruningStrategy.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.realize;\n\nimport java.util.*;\n\n/**\n * Interface for edge pruning strategies.\n *\n * @author Michael White\n * @version $Revision: 1.4 $, $Date: 2011/03/27 14:45:32 $\n */\npublic interface PruningStrategy\n{\n /**\n * Prunes and returns a (possibly empty) list of edges \n * from the given ones, which should be sorted by score, \n * from highest to lowest.\n */\n public List pruneEdges(List catEdges);\n}\n\n" }, { "path": "src/opennlp/ccg/realize/Realizer.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2003-11 University of Edinburgh / Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.realize;\r\n\r\nimport opennlp.ccg.grammar.*;\r\nimport opennlp.ccg.synsem.*;\r\nimport opennlp.ccg.hylo.*;\r\nimport opennlp.ccg.*;\r\nimport org.jdom.*;\r\nimport java.util.*;\r\nimport java.util.prefs.*;\r\n\r\n/**\r\n * The realizer manages the realization process.\r\n * Realization options may be set for use across calls \r\n * to the realizer.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.31 $, $Date: 2011/07/19 03:40:46 $\r\n */\r\npublic class Realizer\r\n{\r\n \r\n /** The grammar used for realization. */\r\n public final Grammar grammar; \r\n \r\n /** Flag for whether to use depth-first search. Defaults to false. */\r\n public boolean depthFirst = false; \r\n \r\n // the chart used to realize a request\r\n private Chart chart = null;\r\n \r\n \r\n /** Constructor. */\r\n public Realizer(Grammar grammar) { \r\n this.grammar = grammar;\r\n }\r\n \r\n /** Returns the chart used in the latest request, or null if none. */\r\n public Chart getChart() { return chart; }\r\n \r\n \r\n //-----------------------------------------------------------------\r\n // default options, for use when not given in realization request\r\n // nb: as the usual practice is to set these options once \r\n // for reuse across calls to the realizer, only a subset of \r\n // the options may be overridden in different calls to the \r\n // realize method\r\n \r\n /** Time limit in ms. (Default is -1, or none.) */\r\n public int timeLimitMS = -1;\r\n \r\n /** Flag for whether to wait for a complete edge. (Default is false.) */\r\n public boolean waitForCompleteEdge = false;\r\n\r\n /** Sign scorer to use. (Default is none.) */\r\n public SignScorer signScorer = null;\r\n \r\n /** Pruning strategy to use. (Default is none.) */\r\n public PruningStrategy pruningStrategy = null;\r\n \r\n /** Hypertagger to use. (Default is none.) */\r\n public Hypertagger hypertagger = null;\r\n \r\n\r\n //-----------------------------------------------------------------\r\n // get LF from doc \r\n \r\n /**\r\n * Retrieves an input LF from the given XML doc, processing any \r\n * LF chunks along the way.\r\n */\r\n public static LF getLfFromDoc(Document doc) {\r\n Element rootElt = doc.getRootElement();\r\n Element lfElt = (rootElt.getName().equals(\"lf\")) ? rootElt : rootElt.getChild(\"lf\");\r\n return getLfFromElt(lfElt);\r\n }\r\n\r\n /**\r\n * Retrieves an input LF from the given XML element, processing any \r\n * LF chunks along the way.\r\n */\r\n public static LF getLfFromElt(Element lfElt) {\r\n HyloHelper.processChunks(lfElt);\r\n LF lf = HyloHelper.getLF(lfElt);\r\n return lf;\r\n }\r\n\r\n \r\n //-----------------------------------------------------------------\r\n // realization routines \r\n \r\n /**\r\n * Realizes the input LF, \r\n * returning the best edge found (or null if none).\r\n */\r\n public Edge realize(LF lf) {\r\n return realize(lf, this.signScorer);\r\n }\r\n\r\n /**\r\n * Realizes the input LF relative to the given sign scorer, \r\n * returning the best edge found (or null if none).\r\n */\r\n public Edge realize(LF lf, SignScorer signScorer) {\r\n Preferences prefs = Preferences.userNodeForPackage(TextCCG.class);\r\n int timeLimitToUse = (timeLimitMS != -1) \r\n ? timeLimitMS\r\n : prefs.getInt(Chart.TIME_LIMIT, Chart.NO_TIME_LIMIT);\r\n return realize(lf, signScorer, timeLimitToUse, waitForCompleteEdge);\r\n }\r\n \r\n /**\r\n * Realizes the input LF relative to given sign scorer, \r\n * returning the best edge found (or null if none)\r\n * in the given time limit (in ms), potentially waiting \r\n * longer for a complete edge according to the given flag.\r\n * If a hypertagger is employed, realization proceeds \r\n * iteratively through the available beta-best values \r\n * within the overall time or edge limit.\r\n */\r\n public Edge realize(LF lf, SignScorer signScorer, int timeLimitMS, boolean waitForCompleteEdge) {\r\n List preds = HyloHelper.flatten(lf);\r\n SignScorer scorerToUse = (signScorer != null) \r\n ? signScorer : SignScorer.nullScorer;\r\n PruningStrategy strategyToUse = (pruningStrategy != null) \r\n ? pruningStrategy : new NBestPruningStrategy();\r\n // realize iteratively with hypertagger, if present\r\n if (hypertagger != null) { \r\n \treturn realizeWithHypertagger(preds, scorerToUse, strategyToUse, timeLimitMS);\r\n }\r\n // otherwise make chart, set start time\r\n long startTime = System.currentTimeMillis(); \r\n chart = new Chart(new EdgeFactory(grammar, preds, scorerToUse), strategyToUse);\r\n chart.startTime = startTime; \r\n chart.depthFirst = depthFirst;\r\n // run request\r\n chart.initialize();\r\n chart.combine(timeLimitMS, waitForCompleteEdge);\r\n // XXX tmp\r\n \t// if no complete edge, try again gluing fragments\r\n// if (!chart.bestEdge.complete()) {\r\n// \tSystem.out.println(\"Trying to glue fragments ...\");\r\n// \tchart.reInitForGluing();\r\n// \tchart.combine(timeLimitMS, waitForCompleteEdge);\r\n// }\r\n // return best edge\r\n return chart.bestEdge;\r\n }\r\n \r\n // XXX tmp switch for gluing\r\n private boolean useGluing = Boolean.getBoolean(\"useGluing\");\r\n \r\n // iterate through beta-best values until a complete realization is found; \r\n // otherwise return the best fragment using the glue rule, or if all else \r\n // fails (or not using gluing), greedy fragment joining\r\n private Edge realizeWithHypertagger(List preds, SignScorer signScorer, PruningStrategy pruningStrategy, int timeLimitMS) {\r\n // get start time\r\n long startTime = System.currentTimeMillis();\r\n // get edge limit\r\n Preferences prefs = Preferences.userNodeForPackage(TextCCG.class);\r\n int edgeLimit = prefs.getInt(Chart.EDGE_LIMIT, Chart.NO_EDGE_LIMIT);\r\n \t// set supertagger in lexicon\r\n \tgrammar.lexicon.setSupertagger(hypertagger);\r\n \t// reset beta\r\n \thypertagger.resetBeta();\r\n // loop until retval set or need to give up\r\n Edge retval = null;\r\n chart = null;\r\n boolean outOfBetas = false;\r\n boolean pastTimeLimit = false;\r\n boolean exceededEdgeLimit = false;\r\n long iterStartTime = 0, currentTime = 0;\r\n int iterTime = 0;\r\n while (retval == null && !outOfBetas && !pastTimeLimit && !exceededEdgeLimit) {\r\n \t// instantiate chart and set start time for this iteration\r\n chart = new Chart(new EdgeFactory(grammar, preds, signScorer, hypertagger), pruningStrategy);\r\n iterStartTime = System.currentTimeMillis();\r\n \t// do realization in packing mode to see if a complete realization \r\n \t// can be found with this hypertagger setting\r\n chart.usePacking = true; chart.collectCombos = false;\r\n chart.doUnpacking = false; chart.joinFragments = false;\r\n // run request\r\n chart.initialize();\r\n if (chart.noUncoveredPreds()) \r\n \tchart.combine(timeLimitMS, false);\r\n // check time limit\r\n currentTime = System.currentTimeMillis();\r\n iterTime = (int) (currentTime - iterStartTime);\r\n if (timeLimitMS != Chart.NO_TIME_LIMIT && iterTime >= timeLimitMS) {\r\n \tpastTimeLimit = true;\r\n// \tSystem.out.println(\"Went past time limit with ht beta: \" + hypertagger.getCurrentBetaValue());\r\n }\r\n // check edge limit\r\n if (edgeLimit != Chart.NO_EDGE_LIMIT && chart.numEdges >= edgeLimit) {\r\n \texceededEdgeLimit = true;\r\n// \tSystem.out.println(\"Exceeded edge limit with ht beta: \" + hypertagger.getCurrentBetaValue());\r\n }\r\n // if complete, unpack and return best edge\r\n if (chart.bestEdge.complete()) {\r\n \tchart.doUnpacking = true; chart.doUnpacking();\r\n \tretval = chart.bestEdge;\r\n }\r\n // otherwise check beta level if still within limits\r\n else if (!pastTimeLimit && !exceededEdgeLimit) {\r\n \t// progress to next beta setting, if any\r\n \tif (hypertagger.hasMoreBetas()) {\r\n \t\thypertagger.nextBeta();\r\n \t}\r\n \telse {\r\n\t \t// otherwise out of betas\r\n \t\toutOfBetas = true;\r\n//\t \tSystem.out.println(\"Ran out of betas with ht beta: \" + hypertagger.getCurrentBetaValue());\r\n \t}\r\n }\r\n }\r\n // if no result, take desperate measures with fragments\r\n if (retval == null) {\r\n \t// try realization with gluing\r\n if (useGluing) {\r\n//\t System.out.println(\"Num edges for final iteration: \" + chart.numEdges);\r\n//\t System.out.println(\"Trying gluing option after iterTime: \" + iterTime);\r\n\t \tchart.reInitForGluing();\r\n\t \t// double time and space limits, to give gluing option some room\r\n\t \tchart.edgeLimit = edgeLimit * 2;\r\n\t \tchart.combine(timeLimitMS * 2, waitForCompleteEdge);\r\n//\t System.out.println(\"Num edges after gluing: \" + chart.numEdges);\r\n\t currentTime = System.currentTimeMillis();\r\n\t iterTime = (int) (currentTime - iterStartTime);\r\n\t // if complete, unpack and return best edge\r\n\t if (chart.bestEdge.complete()) {\r\n//\t System.out.println(\"Unpacking in final iteration after iterTime: \" + iterTime);\r\n\t \tchart.doUnpacking = true; chart.doUnpacking();\r\n\t \tretval = chart.bestEdge;\r\n\t }\r\n }\r\n // otherwise try a final iteration in an iteration in anytime mode, possibly resorting to joining fragments \r\n if (retval == null) {\r\n// System.out.println(\"Trying a final iteration in anytime mode after iterTime: \" + iterTime);\r\n \t// instantiate chart and set start time for this iteration\r\n chart = new Chart(new EdgeFactory(grammar, preds, signScorer, hypertagger), pruningStrategy);\r\n iterStartTime = System.currentTimeMillis();\r\n // run request\r\n \t\tchart.usePacking = false; chart.joinFragments = true;\r\n chart.initialize();\r\n chart.combine(timeLimitMS, waitForCompleteEdge);\r\n//\t System.out.println(\"Num edges after anytime iteration: \" + chart.numEdges);\r\n\t currentTime = System.currentTimeMillis();\r\n\t iterTime = (int) (currentTime - iterStartTime);\r\n//\t if (chart.bestEdge.complete()) \r\n//\t System.out.println(\"Found complete edge after iterTime: \" + iterTime);\r\n//\t else\r\n//\t System.out.println(\"Resorting to joined fragments after iterTime: \" + iterTime);\r\n // return best edge\r\n retval = chart.bestEdge;\r\n }\r\n }\r\n \t// update end time\r\n long endTime = System.currentTimeMillis();\r\n chart.timeTilDone = (int) (endTime - startTime);\r\n \t// reset supertagger in lexicon\r\n \tgrammar.lexicon.setSupertagger(null);\r\n // return\r\n \treturn retval;\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/realize/RuleInstance.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2003-5 University of Edinburgh (Michael White)\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.realize;\r\n\r\nimport opennlp.ccg.grammar.*;\r\nimport opennlp.ccg.hylo.*;\r\nimport java.util.*;\r\n//import java.util.prefs.*;\r\n//import java.text.*;\r\n\r\n/**\r\n * A rule instance is a tracker for an instantiated version of a type changing \r\n * rule, ie a type changing rule with its semantics instantiated together with \r\n * bitsets representing its coverage of the input predicates \r\n * and the indices in its arg category, along with lists of the active LF alts.\r\n * Such rule instances are created and managed by an EdgeFactory.\r\n * The design follows the Singleton pattern.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.7 $, $Date: 2005/11/24 03:22:08 $\r\n */\r\npublic class RuleInstance extends Tracker\r\n{\r\n /** The instantiated type changing rule. */\r\n public final TypeChangingRule rule;\r\n \r\n /** Constructor. */\r\n public RuleInstance(TypeChangingRule rule, BitSet bitset, BitSet indices, List> activeLfAlts) {\r\n super(bitset, indices, activeLfAlts);\r\n this.rule = rule;\r\n }\r\n\r\n /** Returns '{bitset} name: arg => result'. */\r\n public String toString() {\r\n StringBuffer sb = new StringBuffer();\r\n //sb.append(indices + \" \");\r\n sb.append(bitset + \" \");\r\n sb.append(rule);\r\n return sb.toString();\r\n }\r\n}\r\n \r\n" }, { "path": "src/opennlp/ccg/realize/StemPruningStrategy.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2012 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.realize;\r\n\r\nimport opennlp.ccg.lexicon.Word;\r\nimport opennlp.ccg.synsem.Sign;\r\n\r\nimport java.util.*;\r\n\r\n/**\r\n * A diversity pruning strategy that defines signs to be \r\n * notCompellinglyDifferent if they have the same sequence of\r\n * stems. \r\n * The empty constructor defaults the singleBestPerGroup flag \r\n * to true.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.1 $, $Date: 2011/04/02 16:32:17 $\r\n */\r\npublic class StemPruningStrategy extends DiversityPruningStrategy\r\n{\r\n /** Constructor, defaults singleBestPerGroup to true. */\r\n public StemPruningStrategy() { this(true); }\r\n\r\n /** Full constructor. */\r\n public StemPruningStrategy(boolean singleBestPerGroup) { \r\n this.singleBestPerGroup = singleBestPerGroup;\r\n }\r\n \r\n /** Returns true iff the given signs are not compellingly different.\r\n In particular, returns true iff the signs have the same\r\n sequence of stems. */\r\n public boolean notCompellinglyDifferent(Sign sign1, Sign sign2) {\r\n\tList words1 = sign1.getWords();\r\n\tList words2 = sign2.getWords();\r\n\tif (words1.size() != words2.size()) return false;\r\n \tfor (int i=0; i < words1.size(); i++) {\r\n\t if (words1.get(i).getStem() != words2.get(i).getStem())\r\n\t\treturn false;\r\n \t}\r\n return true;\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/realize/Tracker.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2003-5 University of Edinburgh (Michael White)\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.realize;\r\n\r\nimport opennlp.ccg.hylo.*;\r\nimport java.util.*;\r\n\r\n/**\r\n * A tracker is a wrapper for either a sign (ie, an edge) or \r\n * a type changing rule (ie, a rule instance) which has bitsets for \r\n * representing the coverage of the input predicates and the semantic indices used.\r\n * It also has a list of active LF alts.\r\n * Trackers are created by an EdgeFactory.\r\n * The design follows the Singleton pattern.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.9 $, $Date: 2005/11/24 03:15:26 $\r\n */\r\npublic class Tracker\r\n{\r\n /** The coverage bitset. */\r\n public final BitSet bitset;\r\n \r\n /** The indices bitset. */\r\n public final BitSet indices;\r\n \r\n /** The active LF alts. */\r\n public final List> activeLfAlts;\r\n \r\n /** Constructor. */\r\n public Tracker(BitSet bitset, BitSet indices, List> activeLfAlts) {\r\n this.bitset = bitset; this.indices = indices; this.activeLfAlts = activeLfAlts;\r\n }\r\n \r\n /** \r\n * Returns whether the coverage bitset of this tracker intersects with the \r\n * coverage bitset of the given one.\r\n */\r\n public boolean intersects(Tracker tracker) {\r\n return bitset.intersects(tracker.bitset);\r\n }\r\n\r\n /** \r\n * Returns whether the indices bitset of this tracker intersects with the \r\n * indices bitset of the given one, if both non-empty; otherwise, returns \r\n * true (if either this tracker or the given one has no indices).\r\n */\r\n public boolean indicesIntersect(Tracker tracker) {\r\n return indices.isEmpty() || tracker.indices.isEmpty() || \r\n indices.intersects(tracker.indices);\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/realize/hypertagger/FeatureExtractionException.java", "content": "package opennlp.ccg.realize.hypertagger;\n\npublic class FeatureExtractionException extends Exception {\n\n\tpublic FeatureExtractionException(String string) {\n\t\tsuper(string);\n\t}\n\t\n\tpublic FeatureExtractionException() {\n\t\tsuper();\n\t}\n\n\t/**\n\t * \n\t */\n\tprivate static final long serialVersionUID = 1L;\n\t\n}\n" }, { "path": "src/opennlp/ccg/realize/hypertagger/LFInfo.java", "content": "package opennlp.ccg.realize.hypertagger;\n\nimport opennlp.ccg.synsem.LF;\n\n/**\n * @author espinosa\n *\tThis class encapsulates a LF and its corresponding gold standard predicate info, if available.\n */\npublic class LFInfo {\n\tLF lf;\n\tString fullWords;\n\tString lfNum;\n\tpublic LFInfo(LF lf, String fullWords, String lfNum) {\n\t\tthis.lf = lf;\n\t\tthis.fullWords = fullWords;\n\t\tthis.lfNum = lfNum;\n\t}\n\tpublic LF getLF() {\n\t\treturn this.lf;\n\t}\n\tpublic String getFullWords() {\n\t\treturn this.fullWords;\n\t}\n\tpublic String getLFNum() {\n\t\treturn this.lfNum;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/realize/hypertagger/LFLoader.java", "content": "package opennlp.ccg.realize.hypertagger;\n\nimport java.io.File;\nimport java.io.FileFilter;\nimport java.io.IOException;\nimport java.net.MalformedURLException;\nimport java.net.URL;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.Iterator;\nimport java.util.LinkedList;\nimport java.util.List;\n\nimport opennlp.ccg.grammar.Grammar;\nimport opennlp.ccg.hylo.HyloHelper;\nimport opennlp.ccg.realize.Realizer;\nimport opennlp.ccg.synsem.LF;\n\nimport org.jdom.Document;\nimport org.jdom.Element;\n\n\n/**\n * @author espinosa\n *\tThis class abstracts over a collection of LFs contained in a collection of files.\n */\npublic class LFLoader implements Iterator {\n\tstatic class XmlFilenameFilter implements FileFilter {\n\t\tpublic boolean accept(File f) {\n\t\t\treturn f.getName().toLowerCase().endsWith(\".xml\");\n\t\t}\n\t}\n\tGrammar grammar;\n\tArrayList lfFiles;\n\tint filePos = 0;\n\tLinkedList lfs;\n\tint total = 0;\n\tint skipped = 0;\n\n\t/**\n\t * Constructs a new LFLoader which will load LFs from a collection of files or directories under a base directory.\n\t * @param grammarFile The grammar to use\n\t * @param baseDir The base directory. Paths will be interpreted relative to this directory.\n\t * @param paths The files to load the LFs from. Directories or files can be given. Directories are not searched recursively. Only files ending\n\t * in .xml will be loaded.\n\t */\n\tpublic LFLoader(File grammarFile, File baseDir, List paths) {\n\t\tlfs = new LinkedList();\n\t\tURL grammarURL = null;\n\t\ttry {\n\t\t\tgrammarURL = grammarFile.toURI().toURL();\n\t\t} \n\t\tcatch (MalformedURLException e1) {\n\t\t\te1.printStackTrace();\n\t\t}\n\t\ttry {\n\t\t\tgrammar = new Grammar(grammarURL);\n\t\t} \n\t\tcatch (Exception e) {\n\t\t\te.printStackTrace();\n\t\t}\n\t\tlfFiles = new ArrayList();\n\t\tpaths = normalize(paths);\n\t\tfor (String lfFilename : paths) {\n\t\t\t// if this argument is a directory, load all XML files from it\n\t\t\tFile f = new File(baseDir, lfFilename);\n\t\t\tif(f.isDirectory()) {\n\t\t\t\tlfFiles.addAll(Arrays.asList(f.listFiles(new XmlFilenameFilter())));\n\t\t\t}\n\t\t\telse {\n\t\t\t\tlfFiles.add(f);\n\t\t\t}\n\t\t}\n\t}\n\n\tprivate List normalize(List paths) {\n\t\tArrayList ret = new ArrayList();\n\t\tfor(String s: paths) {\n\t\t\tif(s.indexOf(',') < 0) {\n\t\t\t\tret.add(s.trim());\n\t\t\t}\n\t\t\telse {\n\t\t\t\t// explode comma-separated values into separate strings\n\t\t\t\tString[] fields = s.split(\",\");\n\t\t\t\tfor(String f : fields) {\n\t\t\t\t\tret.add(f.trim());\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\treturn ret;\n\t}\n\n\t@SuppressWarnings(\"unchecked\")\n\tprivate void loadFile(File lfFile) {\n\t\tDocument doc = null;\n\t\tint n = 0;\n\t\ttry {\n\t\t\tdoc = grammar.loadFromXml(lfFile.getAbsolutePath());\n\t\t} \n\t\tcatch (IOException e) {\n\t\t\t// if there's a problem, just skip this file\n\t\t\tSystem.err.println(\"Couldn't open input file \" + lfFile + \", skipping.\\n\");\n\t\t\treturn;\n\t\t}\n\t\tcatch (Exception e) {\n\t\t\te.printStackTrace();\n\t\t\treturn;\n\t\t}\n\t\tElement root = doc.getRootElement();\n\t\tList testItems = root.getChildren();\n\t\t// Iterate through test item LFS and print to file/stdio tags predicted\n\t\t// by the hypertagger\n\t\tfor (Element item : testItems) {\n\t\t\tString lfNum = \"unk\";\n\t\t\tlfNum = item.getAttributeValue(\"info\");\n\t\t\tElement itemLFElt = item.getChild(\"lf\");\n\t\t\t//Element itemFullWordsElt = item.getChild(\"full-words\");\n\t\t\tElement itemPredInfoElt = item.getChild(\"pred-info\");\n\t\t\t//String sentId = itemFullWordsElt.getAttributeValue(\"info\");\n\t\t\t//String fullWords = itemFullWordsElt.getTextNormalize();\n\t\t\t// mww: extra null check\n\t\t\tString predInfo = null;\n\t\t\tif (itemPredInfoElt != null) predInfo = itemPredInfoElt.getAttributeValue(\"data\");\n\t\t\t//String predInfo = itemPredInfoElt.getAttributeValue(\"data\");\n\t\t\tif(predInfo == null || predInfo.equals(\"\")) {\n\t\t\t\t/* because this class is used to load LFs for training purposes, we can't continue without the gold-std info */\n\t\t\t\t// mww: added info: lfNum\n\t\t\t\tSystem.err.println(\"No pred-info found for lf #\" + n + \" (info: \" + lfNum + \") in file \" + lfFile + \", skipping.\");\n\t\t\t\tskipped++;\n\t\t\t\tcontinue;\n\t\t\t}\n\t\t\t// mww: added try-catch block\n\t\t\ttry {\n\t\t\t\tLF lf = Realizer.getLfFromElt(itemLFElt);\n\t\t\t\tLF flatLF = HyloHelper.flattenLF(lf);\n\t\t\t\tlfs.offer(new LFInfo(flatLF, predInfo, lfNum));\n\t\t\t}\n\t\t\tcatch (Exception exc) {\n\t\t\t\tSystem.err.println(\"Skipping lf #\" + n + \" (info: \" + lfNum + \") in file \" + lfFile + \", uncaught exception:\");\n\t\t\t\tSystem.err.println(exc.getMessage());\n\t\t\t\texc.printStackTrace(System.err);\n\t\t\t\tskipped++;\n\t\t\t\tcontinue;\n\t\t\t\t\n\t\t\t}\n\t\t\tn++;\n\t\t\ttotal++;\n\t\t}\n\t\tSystem.err.println(\"LFL: loaded \" + n + \" LFs from \" + lfFile);\n\t}\n\t/* two cases:\n\t * - if there's an LF in the queue, return it\n\t * - if there isn't, load the next file -- BUT -- if the LF queue is still empty, load the next file, and so on\n\t */\n\tpublic boolean hasNext() {\n\t\tif(!lfs.isEmpty()) {\n\t\t\treturn true;\n\t\t}\n\t\t// queue is empty, load next file\n\t\twhile(lfs.isEmpty()) {\n\t\t\tif(filePos == lfFiles.size()) {\n\t\t\t\treturn false; // no more files\n\t\t\t}\n\t\t\tloadFile(lfFiles.get(filePos));\n\t\t\tfilePos++;\n\t\t}\n\t\treturn true;\n\t}\n\n\t// this method returns null when no more LFs can be loaded\n\tpublic LFInfo next() {\n\t\tif(!lfs.isEmpty()) {\n\t\t\treturn lfs.poll();\n\t\t}\n\t\twhile(lfs.isEmpty()) {\n\t\t\tif(filePos == lfFiles.size()) {\n\t\t\t\treturn null;\n\t\t\t}\n\t\t\tloadFile(lfFiles.get(filePos));\n\t\t}\n\t\treturn lfs.poll();\n\t}\n\n\tpublic void remove() {\n\t\t// NOT IMPLEMENTED\n\t\tthrow new RuntimeException(\"Method not implemented\");\n\t}\n\tpublic int getTotal() {\n\t\treturn this.total;\n\t}\n\tpublic int getSkipped() {\n\t\treturn this.skipped;\n\t}\n}" }, { "path": "src/opennlp/ccg/realize/hypertagger/LMFactorExtractor.java", "content": "package opennlp.ccg.realize.hypertagger;\n\nimport java.io.BufferedReader;\nimport java.io.File;\nimport java.io.FileReader;\nimport java.io.IOException;\n\n\n/* this is a copy of TagExtract, modified to extract SRILM-format data using the full-words\n * element from XML-format LFs.\n * The input is a file in which each line is a \"pred-info\"-style line, one per LF.\n * The input file is given as the sole commandline argument. \n * The factors are written to stdout.\n */\npublic class LMFactorExtractor {\n\tpublic static void main(String args[]) {\n\t\tBufferedReader rd = null;\n\t\tint lineNum = 0;\n\t\tint bNum = 0;\n\t\ttry {\n\t\t\trd = new BufferedReader(new FileReader(new File(args[0])));\n\t\t}\n\t\tcatch (Exception e) {\n\t\t\te.printStackTrace();\n\t\t\tSystem.exit(1);\n\t\t}\n\t\tString line = null;\n\t\tStringBuilder out;\n\t\twhile(true) {\n\t\t\ttry { \n\t\t\t\tline = rd.readLine();\n\t\t\t}\n\t\t\tcatch(IOException e) {\n\t\t\t\te.printStackTrace();\n\t\t\t\tSystem.exit(1);\n\t\t\t}\n\t\t\tlineNum++;\n\t\t\tif(line == null) {\n\t\t\t\tbreak;\n\t\t\t}\n\t\t\tif(line.matches(\"^\\\\s*$\")) {\n\t\t\t\tcontinue;\n\t\t\t}\n\t\t\tout = new StringBuilder();\n\t\t\tout.append(\" \");\n\t\t\tString[] fields = line.split(\"\\\\s+\");\n\t\t\tbNum = 0;\n\t\t\tfor(String f : fields) {\n\t\t\t\tbNum++;\n\t\t\t\tString[] info = f.split(\":\");\n\t\t\t\tif(info.length != 4) {\n\t\t\t\t\tSystem.err.println(\"Wrong number of fields encountered in input line \" + lineNum + \", bundle \" + bNum);\n\t\t\t\t\tSystem.exit(1);\n\t\t\t\t}\n\t\t\t\tout.append(info[3]);\n\t\t\t\tout.append(\":S-\");\n\t\t\t\tout.append(info[3]);\n\t\t\t\tout.append(\":P-\");\n\t\t\t\tout.append(info[2]);\n\t\t\t\tout.append(\":T-\");\n\t\t\t\tout.append(info[1]);\n\t\t\t\tout.append(\" \");\n\t\t\t}\n\t\t\tout.append(\"\");\n\t\t\tSystem.out.println(out.toString());\n\t\t}\n\t}\t\n}\n" }, { "path": "src/opennlp/ccg/realize/hypertagger/TagExtract.java", "content": "package opennlp.ccg.realize.hypertagger;\n\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileFilter;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.util.Comparator;\nimport static java.util.Arrays.*;\nimport joptsimple.*;\n\nimport opennlp.ccg.realize.hypertagger.TagExtractor;\nimport opennlp.ccg.synsem.LF;\nimport opennlp.ccg.util.Pair;\n\npublic class TagExtract {\n\tprivate static boolean quiet = false; // when true, suppress stderr messages\n\t@SuppressWarnings(\"unused\")\n\tprivate static File posModelFile;\n\tprivate static File hyperModelFile;\n\tprivate static File posPriorModelFile;\n\tprivate static File htPriorModelFile;\n\tprivate static File posVocabFile;\n\tprivate static File htVocabFile;\n\tprivate static String argnames;\n\tprivate TagExtractor tex;\n\tprivate BufferedWriter output;\n\t@SuppressWarnings({ \"unchecked\", \"rawtypes\" })\n\tstatic class PairComparator implements Comparator {\n\t\t@SuppressWarnings(\"boxing\")\n\t\tpublic int compare(Pair p, Pair q) {\n\t\t\tif(p.a > q.a) {\n\t\t\t\treturn 1;\n\t\t\t}\n\t\t\tif(p.a == q.a) {\n\t\t\t\treturn 0;\n\t\t\t}\n\t\t\treturn -1;\n\t\t}\n\t\tpublic int compare(Object p, Object q) {\n\t\t\treturn this.compare((Pair) p, (Pair)q );\n\t\t}\n\t}\n\tstatic class XmlFilenameFilter implements FileFilter {\n\t\tpublic boolean accept(File f) {\n\t\t\treturn f.getName().toLowerCase().endsWith(\".xml\");\n\t\t}\n\t}\n\tpublic TagExtract(TagExtractor t) {\n\t\tthis.tex = t;\n\t}\n\t/* TODO: this method should probably be rewritten to use LFLoader and a config file */\n\tpublic static void main(String[] args) throws IOException {\n\t\tTagExtract t = null;\n\t\t//PrintStream output = System.out;\n\t\tBufferedWriter output;\n\t\t//int lfcount = 0;\n\t\t//int lfNum = 0;\n\t\t// option processing\n\t\tOptionParser o = new OptionParser();\n\t\to.acceptsAll(asList(\"help\", \"h\"), \"this message\");\n\t\to.acceptsAll(asList(\"quiet\", \"q\"), \"print no messages\");\n\t\to.acceptsAll(asList(\"pos\", \"pos\"), \"extract POS features\");\n\t\tOptionSpec pos_s = o.acceptsAll(asList(\"p\", \"pos-model\")).withRequiredArg().ofType(File.class).describedAs(\"POS model to use\");\n\t\tOptionSpec outf = o.acceptsAll(asList(\"o\", \"output\")).withRequiredArg().ofType(File.class).describedAs(\"output file\");\n\t\tOptionSpec posPrior_s = o.acceptsAll(asList(\"P\", \"pos-prior\")).withRequiredArg().ofType(File.class).describedAs(\"POS prior model to use\");\n\t\tOptionSpec ht_s = o.acceptsAll(asList(\"y\", \"hyper-model\")).withRequiredArg().ofType(File.class).describedAs(\"HT model to use as input to 2-pass model (see README)\");\n\t\tOptionSpec htPrior_s = o.acceptsAll(asList(\"H\", \"ht-prior\")).withRequiredArg().ofType(File.class).describedAs(\"HT prior model to use\");\n\t\tOptionSpec gr_s = o.acceptsAll(asList(\"g\", \"grammar\")).withRequiredArg().ofType(File.class).describedAs(\"grammar filename\");\n\t\tOptionSpec ht_vocab_s = o.acceptsAll(asList(\"V\", \"ht-prior-vocab\")).withRequiredArg().ofType(File.class).describedAs(\"HT prior vocab filename\");\n\t\tOptionSpec pos_vocab_s = o.acceptsAll(asList(\"v\", \"pos-prior-vocab\")).withRequiredArg().ofType(File.class).describedAs(\"POS prior vocab filename\");\n\t\tOptionSpec corpusDir_s = o.acceptsAll(asList(\"d\", \"lf-dir\")).withRequiredArg().ofType(File.class).describedAs(\"Directory to change to before searching for XML files\");\n\t\tOptionSpec argnames_s = o.acceptsAll(asList(\"an\", \"argnames\")).withRequiredArg().describedAs(\"Names of argument roles in format name(:shortname)?\");\n\t\tOptionSet options = o.parse(args);\n\t\t/* if -h (help) is given, print message and exit */\n\t\tif (options.has(\"h\") || args.length == 0) {\n\t\t\to.printHelpOn(System.out);\n\t\t\tSystem.out.println(\"See the README for additional information.\");\n\t\t\tSystem.exit(0);\n\t\t}\n\t\toutput = new BufferedWriter(new FileWriter(options.valueOf(outf)));\n\t\t// some of these will be nulls, depending on what the user is trying to do\n\t\thyperModelFile = options.valueOf(ht_s);\n\t\tposModelFile = options.valueOf(pos_s);\n\t\tposPriorModelFile = options.valueOf(posPrior_s);\n\t\tposVocabFile = options.valueOf(pos_vocab_s);\n\t\thtPriorModelFile = options.valueOf(htPrior_s);\n\t\thtVocabFile = options.valueOf(ht_vocab_s);\n\t\targnames = options.valueOf(argnames_s);\n\t\tif(options.has(\"q\"))\n\t\t\tquiet = true;\n\t\tLFLoader lfs = new LFLoader(options.valueOf(gr_s), options.valueOf(corpusDir_s), options.nonOptionArguments());\n\t\tif(options.has(\"pos\")) {\n\t\t\tTagExtractor tex = new ZLPOSTagger();\n\t\t\t// mww: set arg names\n\t\t\tif (argnames != null) debug(\"Setting arg names to \" + argnames + \"\\n\");\n\t\t\ttex.setArgNames(argnames); // uses default names if null\n\t\t\tif(posPriorModelFile != null && posVocabFile != null) {\n\t\t\t\tdebug(\"Loading POS model priors from \" + posPriorModelFile + \"\\n\");\n\t\t\t\tdebug(\"Loading POS model vocab from \" + posVocabFile + \"\\n\");\n\t\t\t\ttex.loadPriorModel(posPriorModelFile, posVocabFile);\n\t\t\t}\n\t\t\tdebug(\"Extracting POS features...\" + \"\\n\");\n\t\t\tt = new TagExtract(tex);\n\t\t}\n\t\telse {\n\t\t\t// extracting hypertags\n\t\t\t// using GS pos tags\n\t\t\tTagExtractor tex = new ZLMaxentHypertagger();\n\t\t\t// mww: set arg names\n\t\t\tif (argnames != null) debug(\"Setting arg names to \" + argnames + \"\\n\");\n\t\t\ttex.setArgNames(argnames); // uses default names if null\n\t\t\tif(htPriorModelFile != null && htVocabFile != null) {\n\t\t\t\tdebug(\"Loading HT model priors from \" + htPriorModelFile + \"\\n\");\n\t\t\t\tdebug(\"Loading HT model vocab from \" + htVocabFile + \"\\n\");\n\t\t\t\ttex.loadPriorModel(htPriorModelFile,htVocabFile);\n\t\t\t}\n\t\t\tif(hyperModelFile != null) {\n\t\t\t\tdebug(\"Loading proto-HT model from \" + hyperModelFile + \"\\n\");\n\t\t\t\ttex.loadProtoModel(hyperModelFile);\n\t\t\t}\n\t\t\tdebug(\"Extracting hypertagger features...\" + \"\\n\");\n\t\t\tt = new TagExtract(tex);\n\t\t}\n\t\tt.setOutput(output);\n\t\twhile(lfs.hasNext()) {\n\t\t\t\tLFInfo lfi = lfs.next();\n\t\t\t\tLF lf = lfi.getLF();\n\t\t\t\ttry {\n\t\t\t\t\t//lfNum++;\n\t\t\t\t\tt.extract(lf, lfi.getFullWords());\n\t\t\t\t\t//lfcount++;\n\t\t\t\t\t//debug(\"LFs extracted: \" + lfcount + \"\\r\");\n\t\t\t\t} catch (FeatureExtractionException e) {\n\t\t\t\t\tdebug(\"In LF #\" + lfi.getLFNum() + \":\\n\");\n\t\t\t\t\tdebug(e.toString() + \"\\n\");\n\t\t\t\t}\n\t\t}\n\t\toutput.close();\n\t\tdebug(\"\\n\");\n\t}\n\tprivate void extract(LF flatLF, String fullWords) throws FeatureExtractionException {\n\t\ttex.storeGoldStdPredInfo(fullWords);\n\t\ttex.setLF(flatLF);\n\t\ttry {\n\t\t\toutput.write(tex.getAllFeaturesAndAnswer());\n\t\t} catch (IOException e) {\n\t\t\t// TODO Auto-generated catch block\n\t\t\te.printStackTrace();\n\t\t}\n\n\t}\n\tprivate void setOutput(BufferedWriter output) {\n\t\tthis.output = output;\n\n\t}\n\tpublic static void debug(String string) {\n\t\tif(!quiet) \n\t\t\tSystem.err.print(string);\n\t}\n}\n" }, { "path": "src/opennlp/ccg/realize/hypertagger/TagExtractor.java", "content": "package opennlp.ccg.realize.hypertagger;\n\nimport java.io.File;\nimport java.util.*;\n\nimport opennlp.ccg.hylo.*;\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.util.Pair;\n\n/** This class contains methods for extracting features from a logical form\n * \n * @author espinosa\n *\n */\npublic abstract class TagExtractor {\n\tprotected class LfGraphLink {\n\t\tString label; // eg. \"genrel\"\n\t\tString arg; // if applicable\n\t\tLfGraphNode target; // graph node it points to\n\t\tLfGraphNode source; // node it extends from\n\t\tpublic LfGraphLink(String l, String a, LfGraphNode t) {\n\t\t\tlabel = l;\n\t\t\targ = a;\n\t\t\ttarget = t;\n\t\t}\n\t\tpublic LfGraphNode getTarget() {\n\t\t\treturn target;\n\t\t}\n\t\tpublic void setTarget(LfGraphNode t) {\n\t\t\ttarget = t;\n\t\t}\n\t\tpublic String getLabel() {\n\t\t\treturn label;\n\t\t}\n\t\tpublic LfGraphNode getSource() {\n\t\t\treturn source;\n\t\t}\n\t\tpublic void setSource(LfGraphNode source) {\n\t\t\tthis.source = source;\n\t\t}\n\t}\n\tprotected class LfGraphNode {\n\t\tString data;\n\t\tString predicateName;\n\t\tint index; // from original LF\n\t\tSatOp pred; // original predicate object\n\t\tArrayList children;\n\t\tLfGraphLink parentLink;\n\t\tHashMap attribs;\n\t\tprotected String POS; // caches the part-of-speech tag for this node (or GS postag)\n\t\tprotected String ST; // gold-standard supertag\n\t\tArrayList> POSList; // not used atm\n\t\tSet> STList; // for 2-pass tags\n\t\t\n\t\tpublic Set> getSTList() {\n\t\t\treturn STList;\n\t\t}\n\t\tpublic void setSTList(Set> set) {\n\t\t\tSTList = set;\n\t\t}\n\t\tArrayList multiparents;\n\t\tFeatureList features;\n\t\tprivate String id; // e.g. \"w1\"\n\t\tprotected FeatureList getFeatures() {\n\t\t\treturn features;\n\t\t}\n\t\tprotected void setFeatures(FeatureList features) {\n\t\t\tthis.features = features;\n\t\t}\n\t\tpublic LfGraphNode(String s, int idx) {\n\t\t\tdata = s; // this is just \"w1\" or whatever\n\t\t\tindex = idx;\n\t\t\tchildren = new ArrayList();\n\t\t\tattribs = new HashMap();\n\t\t\tPOS = null;\n\t\t\tmultiparents = new ArrayList();\n\t\t}\n\t\tpublic String getData() {\n\t\t\treturn data;\n\t\t}\n\t\tpublic void setData(String s) {\n\t\t\tdata = s;\n\t\t}\n\t\tpublic void addAttribute(String name, String value) {\n\t\t\tattribs.put(name, value);\n\t\t}\n\t\tpublic void addChild(LfGraphLink link) {\n\t\t\tchildren.add(link);\n\t\t}\n\t\tpublic ArrayList getChildren() {\n\t\t\treturn children;\n\t\t}\n\t\tpublic int getNumChildren() {\n\t\t\treturn children.size();\n\t\t}\n\t\tpublic boolean isLeafNode() {\n\t\t\tif(children.isEmpty()) {\n\t\t\t\treturn true;\n\t\t\t}\n\t\t\telse {\n\t\t\t\treturn false;\n\t\t\t}\n\t\t}\n\t\t/*\n\t\tpublic boolean isLexPred() {\n\t\t\treturn HyloHelper.isLexPred(data);\n\t\t}*/\n\t\tpublic LfGraphNode findNode(BitSet b) {\n\t\t\t// intended to be run from the root node; only searches\n\t\t\t// nodes below this one\n\t\t\t// Say we want a node whose index is between 1-4 inclusive\n\t\t\t// pass a bitset with those bits set. Returns the lex pred\n\t\t\t// node whose index is in the bitset.\n\t\t\tif(b.get(index)) {\n\t\t\t\treturn this;\n\t\t\t}\n\t\t\tfor(LfGraphLink n : children) {\n\t\t\t\tn.getTarget().findNode(b);\n\t\t\t}\n\t\t\treturn null;\n\t\t}\n\t\tpublic HashMap getAttribs() {\n\t\t\treturn attribs;\n\t\t}\n\t\tpublic void setAttribs(HashMap attribs) {\n\t\t\tthis.attribs = attribs;\n\t\t}\n\t\tpublic int getIndex() {\n\t\t\treturn index;\n\t\t}\n\t\tpublic void setIndex(int index) {\n\t\t\tthis.index = index;\n\t\t}\n\t\tpublic void setChildren(ArrayList children) {\n\t\t\tthis.children = children;\n\t\t}\n\t\tpublic String getPredicateName() {\n\t\t\treturn predicateName;\n\t\t}\n\t\tpublic void setPredicateName(String predicateName) {\n\t\t\tthis.predicateName = predicateName;\n\t\t}\n\t\tpublic LfGraphLink getParentLink() {\n\t\t\treturn parentLink;\n\t\t}\n\t\tpublic void setParentLink(LfGraphLink parentLink) {\n\t\t\tthis.parentLink = parentLink;\n\t\t}\n\t\tpublic void addMultiParent(LfGraphNode parentNode) {\n\t\t\t// XXX should not be necessary! fix bug & eliminate\n\t\t\tif(parentNode == this) {\n\t\t\t\treturn;\n\t\t\t}\n\t\t\tthis.multiparents.add(parentNode);\n\t\t}\n\t\tpublic ArrayList getMultiParents() {\n\t\t\treturn this.multiparents;\n\t\t}\n\t\tpublic SatOp getPred() {\n\t\t\treturn pred;\n\t\t}\n\t\tprotected void setPred(SatOp pred) {\n\t\t\tthis.pred = pred;\n\t\t}\n\t\tpublic String getPOS() {\n\t\t\treturn POS;\n\t\t}\n\t\tpublic void setPOS(String pos) {\n\t\t\tPOS = new String(pos);\n\t\t}\n\t\tpublic ArrayList> getPOSList() {\n\t\t\treturn POSList;\n\t\t}\n\t\tpublic void setPOSList(ArrayList> plist) {\n\t\t\tthis.POSList = plist;\n\t\t}\n\t\tpublic void setID(String string) {\n\t\t\tthis.id = string;\n\t\t}\n\t\tpublic String getID() {\n\t\t\treturn this.id;\n\t\t}\n\t}\n\tpublic static int LFID = 0;\n\t/** Implements a list of features. Keys are short strings, e.g. \"CN\", \"FO\", ...\n\t * Values are arbitrary strings, but a key can have multiple values. Thus, the values are actually of type ArrayList. \n\t * @author espinosa\n\t *\n\t */\n\tprotected class FeatureList extends HashMap> {\n\t\t/**\n\t\t * \n\t\t */\n\t\tprivate static final long serialVersionUID = 1L;\n\t\tint numFeatures = 0;\n\t\t// deprecated\n\t\tprotected void addFeature(String featureName, String value) {\n\t\t\t/*if(value == null) {\n\t\t\t\treturn;\n\t\t\t}\n\t\t\tif(this.get(featureName) == null) {\n\t\t\t\tthis.put(featureName.trim(), new ArrayList());\n\t\t\t}\n\t\t\tArrayList feats = this.get(featureName);\n\t\t\tfeats.add(value.trim());\n\t\t\tthis.put(featureName.trim(), feats);\n\t\t\tnumFeatures++; */\n\t\t\taddFeatureWithProb(featureName, value);\n\t\t}\n\t\tprotected void addFeatures(String featureName, ArrayList> values) {\n\t\t\tfor(Pair v : values) {\n\t\t\t\tthis.addFeatureWithProb(featureName, v.a, v.b );\n\t\t\t}\n\t\t}\n\t\t// merge f's features into self\n\t\t/*\n\t\tprotected void addFeatures(FeatureList f) {\n\t\t\tSet>> es = this.entrySet();\n\t\t\tfor(Map.Entry> e : es) {\n\t\t\t\tString fn = e.getKey();\n\t\t\t\tfor(String fv : e.getValue()) {\n\t\t\t\t\tthis.addFeature(fn, fv);\n\t\t\t\t}\n\t\t\t}\n\t\t} */\n\t\tprotected void addFeatureWithProb(String featureName, String value, Double prob) {\n\t\t\tif(value == null) {\n\t\t\t\treturn;\n\t\t\t}\n\t\t\tif(this.get(featureName) == null) {\n\t\t\t\tthis.put(featureName.trim(), new ArrayList());\n\t\t\t}\n\t\t\tArrayList feats = this.get(featureName);\n\t\t\tfeats.add(value.trim() + \":\" + prob.toString());\n\t\t\tthis.put(featureName.trim(), feats);\n\t\t\tnumFeatures++;\n\t\t}\n\t\tprotected void addFeatureWithProb(String featureName, String value) {\n\t\t\tthis.addFeatureWithProb(featureName, value, new Double(1.0));\n\t\t}\n\t\tprotected ArrayList getFeature(String featureName) {\n\t\t\treturn this.get(featureName);\n\t\t}\n\t\tprotected String getSingleFeature(String featureName) {\n\t\t\treturn this.get(featureName).get(0);\n\t\t}\n\t\tprotected String getAllFeatures() {\n\t\t\tString output = \"\";\n\t\t\tfor(String k : this.keySet()) {\n\t\t\t\tfor(String v : this.get(k)) {\n\t\t\t\t\toutput = output.concat(k + \"=\" + v);\n\t\t\t\t\toutput = output.concat(\" \");\n\t\t\t\t}\n\t\t\t}\n\t\t\treturn output;\n\t\t}\n\t\tprotected String[] getAllFeaturesForMaxent() {\n\t\t\tString[] out = new String[this.getNumFeatures()];\n\t\t\tint i = 0;\n\t\t\tfor(String k : this.keySet()) {\n\t\t\t\tfor(String v : this.get(k)) {\n\t\t\t\t\tout[i] = k + \"=\" + v;\n\t\t\t\t\ti++;\n\t\t\t\t}\n\t\t\t}\n\t\t\treturn out;\n\t\t}\n\t\tprotected int getNumFeatures() {\n\t\t\treturn numFeatures;\n\t\t}\n\t}\n\t// fields\n\tprotected LfGraphNode lfGraph; // stores the graph structure of the LF\n\tprotected HashMap nomTable;\n\tprotected HashMap varTable;\n\tprotected HashMap lexpairs;\n\tprotected HashMap pospairs;\n\tprotected LF lf;\n\tprotected List preds;\n\tprotected List flatLF;\n\tprotected int maxIndex;\n\tprotected Map argNameMap; // mww: map from arg names to short arg names\n\t\n\n\tpublic TagExtractor() {\targNameMap = new HashMap(); }\n\t\n\tpublic void setLF(LF lf) throws FeatureExtractionException {\n\t\tLFID++;\n\t\t// experimental\n\t\tHyloHelper.convertNominals(lf);\n\t\tsetLF(HyloHelper.getPreds(lf));\n\t}\n\t\n\t/** This method takes an LF and extracts its features, changing the internal state\n\t * of this object accordingly. Other methods such as getSupertag() can then\n\t * be called to obtain the tagger's prediction.\n\t * @param preds A logical form\n\t * @throws FeatureExtractionException when the logical form cannot be processed and the extracted features will not be meaningful\n\t */\n\tpublic void setLF(List preds) throws FeatureExtractionException {\n\t\tthis.preds = preds;\n\t\tLfGraphNode curNode = null;\n\t\tcurNode = null;\n\t\tnomTable = new HashMap();\n\t\tvarTable = new HashMap();\n\t\tint i = 0;\n\t\t// Pass 1: find and store nominals\n\t\tfor(SatOp s: preds) {\n\t\t\tif(s == null) {\n\t\t\t\t\tthrow new FeatureExtractionException(); // ??? why is it sometimes null?\n\t\t\t}\n\t\t\tif(s.getArg() instanceof Proposition) {\n\t\t\t\tLfGraphNode thisNode = new LfGraphNode(s.getNominal().toString(), i);\n\t\t\t\tthisNode.setPredicateName(((Proposition)s.getArg()).getName());\n\t\t\t\tthisNode.setID(s.getNominal().toString());\n\t\t\t\tthisNode.setPred(s);\n\t\t\t\tif(lfGraph == null) {\n\t\t\t\t\t// i.e., the first node processed\n\t\t\t\t\tlfGraph = thisNode;\n\t\t\t\t}\n\t\t\t\tnomTable.put(thisNode.getID(), thisNode);\n\t\t\t}\n\t\t\ti++;\n\t\t}\n\t\tthis.maxIndex = i;\n\t\t// Pass 2: traverse all other nodes, linking to nominals found in pass 1\n\t\ti = 0;\n\t\tfor(SatOp s: preds) {\n\t\t\tif(s == null) {\n\t\t\t\tthrow new FeatureExtractionException();\n\t\t\t}\n\t\t\tif(s.getArg() instanceof Proposition) {\n\t\t\t\tcurNode = nomTable.get(s.getNominal().toString());\n\t\t\t\ti++;\n\t\t\t\tcontinue;\n\t\t\t}\n\t\t\tDiamond d = (Diamond)s.getArg();\n\t\t\t// not sure how this could happen, but it did\n\t\t\tif(d == null || (d.getArg() == null)) { throw new FeatureExtractionException(); }\n\t\t\tif(d.getArg() instanceof NominalVar) {\n\t\t\t\t// XXX all this is probably defunct and can safely be deleted\n\t\t\t\t// add multiparent\n\t\t\t\t// might need getName() instead of toString()\n\t\t\t\t//System.err.println(\"found var: str = \" + d.getArg().toString());\n\t\t\t\tLfGraphNode target = nomTable.get(d.getArg().toString());\n\t\t\t\tLfGraphLink ln = new LfGraphLink(d.getMode().toString(), null, target);\n\t\t\t\tln.setSource(curNode);\n\t\t\t\t//target.addMultiParent(ln);\n\t\t\t}\n\t\t\telse if(d.getArg() instanceof Nominal) {\n\t\t\t\t// make link labeled to nominal\n\t\t\t\tLfGraphNode target = nomTable.get(d.getArg().toString());\n\t\t\t\tLfGraphLink ln = new LfGraphLink(d.getMode().toString(), null, target);\n\t\t\t\tln.setSource(curNode);\n\t\t\t\tif(target != null) {\n\t\t\t\t\t// ??? why is it sometimes null? and if it is, should an exception be thrown?\n\t\t\t\t\ttarget.setParentLink(ln);\n\t\t\t\t\ttarget.addMultiParent(curNode);\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\t//System.err.println(\"TE: target was null: \" + d.getArg().toString());\n\t\t\t\t}\n\t\t\t\tif(curNode == null) { throw new FeatureExtractionException();}\n\t\t\t\tcurNode.addChild(ln);\n\t\t\t}\n\t\t\telse {\n\t\t\t\t// proposition\n\t\t\t\t// add attr to current node\n\t\t\t\ttry {\n\t\t\t\t\tcurNode.addAttribute(d.getMode().toString(), d.getArg().toString());\n\t\t\t\t}\n\t\t\t\tcatch(Exception e) {\n\t\t\t\t\t// ??? this null must occur because there were no attributes, but this\n\t\t\t\t\t// probably isn't the best way to handle it\n\t\t\t\t}\n\t\t\t\t\n\t\t\t}\n\t\t\t// don't change curNode here\n\t\t\ti++;\n\t\t}\n\t}\n\t\n\tpublic LF getLF() {\n\t\treturn this.lf;\n\t}\n\t\n\t/** This method extracts features from a node in the graph and returns\n\t * them as an array of strings. It's implemented slightly differently\n\t * in HyperTagger and POSTagExtractor.\n\t * @param n The node from which to extract the features.\n\t * @return A list of features.\n\t */\n\tprotected abstract FeatureList getFeatures(LfGraphNode n);\n\t\n\t/** Get the features for the index'th node\n\t * \n\t * @param index The index into the LF\n\t */\n\tpublic FeatureList getFeatures(int index) {\n\t\tfor(LfGraphNode n : nomTable.values()) {\n\t\t\tif(n.getIndex() == index) {\n\t\t\t\treturn getFeatures(n);\n\t\t\t}\n\t\t}\n\t\treturn null; // bad index\n\t}\n\t\n\t// these methods only make sense for training, because\n\t// they fetch the gold-standard supertag or POS tag.\n\tpublic String getSupertag(LfGraphNode n) {\n\t\treturn lexpairs.get(n);\n\t}\n\t\n\tpublic String getSupertag(int index) {\n\t\tfor(LfGraphNode n : nomTable.values()) {\n\t\t\tif(n.getIndex() == index) {\n\t\t\t\treturn lexpairs.get(n);\n\t\t\t}\n\t\t}\n\t\treturn null;\n\t}\n\t\n\tpublic String getPOStag(LfGraphNode n) {\n\t\treturn pospairs.get(n);\n\t}\n\t\n\tpublic String getPOStag(int index) {\n\t\tfor(LfGraphNode n : nomTable.values()) {\n\t\t\tif(n.getIndex() == index) {\n\t\t\t\treturn pospairs.get(n);\n\t\t\t}\n\t\t}\n\t\treturn null;\n\t}\n\n\t// return the node with the given index\n\tprotected LfGraphNode findNode(int index) {\n\t\tfor(LfGraphNode n : nomTable.values()) {\n\t\t\tif(n.getIndex() == index) {\n\t\t\t\treturn n;\n\t\t\t}\n\t\t}\n\t\treturn null;\n\t}\n\tprotected int numNodes() {\n\t\treturn nomTable.size();\n\t}\n\tprotected int maxIndex() {\n\t\treturn this.maxIndex;\n\t}\n\t// utility method to do the inverse of java.String.split()\n\tpublic static String join(ArrayList a, String delimiter) {\n\t\tString out = new String();\n\t\tint i;\n\t\tfor(i = 0; i < a.size(); i++) {\n\t\t\tout = out.concat(a.get(i));\n\t\t\tif(i != a.size() - 1) {\n\t\t\t\tout = out.concat(delimiter);\n\t\t\t}\n\t\t}\n\t\treturn out;\n\t}\n\t/**\n\t * @param predInfo\n\t * @throws FeatureExtractionException\n\t * This method stores gold standard predicate info, which is expected to be in the following format:\n\t * predInfo :: (field)+\n\t * field :: wordId:supertag:POStag:predName\n\t * \n\t * The supertag and postag are expected to have been escaped by Lexicon.DefaultTokenizer.escape(), and will be unescaped during storage.\n\t * This method throws a runtime error if the predInfo string cannot be parsed.\n\t * \n\t */\n\tpublic abstract void storeGoldStdPredInfo(String predInfo);\n\n\tpublic abstract String getAllFeaturesAndAnswer();\n\n\tpublic abstract void loadPriorModel(File priorModelFile, File vocabFile);\n\n\tpublic void loadProtoModel(File hyperModelFile) {\n\t\t// TODO refactor TagExtract app so this isn't needed\n\t\treturn;\n\t} \n\n\t// mww: sets configurable arg names \n\t/**\n\t * Sets the arg name map to the given names.\n\t * @param argnames Space-delimited arg names in format name(:shortname)?.\n\t * Defaults to \"Arg0:A0 Arg1:A1 Arg1a:A1a Arg1b:A1b Arg2:A2 Arg2a:A2a Arg2b:A2b Arg3:A3 Arg4:A4 Arg5:A5\".\n\t */\n\tprotected void setArgNames(String argnames) {\n\t\targNameMap.clear();\n\t\t// default is augmented propbank arg names\n\t\tif (argnames == null) argnames = \"Arg0:A0 Arg1:A1 Arg1a:A1a Arg1b:A1b Arg2:A2 Arg2a:A2a Arg2b:A2b Arg3:A3 Arg4:A4 Arg5:A5\";\n\t\tString[] nameslist = argnames.split(\"\\\\s+\");\n\t\tfor (String argname : nameslist) {\n\t\t\tString[] namepair = argname.split(\":\");\n\t\t\tif (namepair.length == 2) argNameMap.put(namepair[0], namepair[1]);\n\t\t\telse if (namepair.length == 1) argNameMap.put(namepair[0], namepair[1]);\n\t\t}\n\t}\n}\n\n\n" }, { "path": "src/opennlp/ccg/realize/hypertagger/ZLMaxentHypertagger.java", "content": "package opennlp.ccg.realize.hypertagger;\n\nimport static java.util.Arrays.asList;\n\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.io.OutputStreamWriter;\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\nimport java.util.regex.Matcher;\nimport java.util.regex.Pattern;\n\nimport joptsimple.OptionParser;\nimport joptsimple.OptionSet;\nimport joptsimple.OptionSpec;\n\nimport opennlp.ccg.hylo.Nominal;\nimport opennlp.ccg.hylo.SatOp;\nimport opennlp.ccg.lexicon.DefaultTokenizer;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.parse.supertagger.io.XMLPOSDictionaryReader;\nimport opennlp.ccg.parse.supertagger.io.XMLWordDictionaryReader;\nimport opennlp.ccg.parse.supertagger.ml.STPriorModel;\nimport opennlp.ccg.parse.supertagger.util.STTaggerPOSDictionary;\nimport opennlp.ccg.parse.supertagger.util.STTaggerWordDictionary;\nimport opennlp.ccg.parse.tagger.util.ConfigFileProcessor;\nimport opennlp.ccg.parse.tagger.util.ResultSink;\nimport opennlp.ccg.realize.Hypertagger;\nimport opennlp.ccg.util.Pair;\n\n/**\n * \n * @author espinosa\n * This class implements the hypertagger. Instantiating a hypertagger requires several external\n * files:\n *
      \n *
    1. A POS model\n *
    2. A prior model for POS tags, and its vocab file\n *
    3. A prior model for hypertags, and its vocab file\n *
    \n * The prior model files are optional.\n *

    \n * To use the hypertagger for realization, instantiate a POS tagger first, then\n * instantiate the hypertagger using that POS tagger. Example in pseudo-code:\n * \n * POSPriorModel ppm = new POSPriorModel(String flmFile, String vocabFile);\n * ZLMaxentModel posMod = new ZLMaxentModel(String fileName);\n * ZLPOSTagger pt = new ZLPOSTagger(posMod, ppm);\n * ZLMaxentHypertagger ht = new ZLMaxentHypertagger(pt, File htModelPath);\n * ht.loadPriorModel(File priorModelFile, File vocabFile);\n * \n * The tagger can also be instantiated from a config file. This is the recommended\n * method for using the tagger as part of the realizer. See the method 'ZLMaxentHypertaggerFactory'.\n * \n * \n */\npublic class ZLMaxentHypertagger extends TagExtractor implements Hypertagger {\n\tFile hypertagModelFilename;\n\tFile posModelFilename;\n\tpublic ZLMaxentModel hypertagModel; // null if extracting feats for training\n\tZLMaxentModel posModel; // null if extracting feats for training\n\tZLPOSTagger postagger;\n\tZLMaxentModel protoHTModel;\n\tdouble protoHTBeta = 0.01; // FIXME what should this be?\n\tLfGraphNode currentPred;\n\tString LFNum;\n\tprotected double[] betas;\n\tint currentBeta;\n\t//Flag which indicates whether gold std tags need to be ensured during tag prediction\n\t//Gold std tag info for perceptron training (event generation)\n\tprivate boolean goldStdTagInsert=false;\n\t//Nominal id to gold std supertag mapping\n\tprivate HashMap goldPred2Tag;\n\tprivate STPriorModel priorModel;\n\tdouble priorBeta = 0.4;\n\tprivate HashMap goldPredPOS;\n\tprivate STTaggerWordDictionary wdict; // word (pred)-level tagging dictionary\n\tprivate STTaggerPOSDictionary posdict; // pos-level tagging dictionary\n\tint dictK; // frequency threshold for tagdict lookups\n\tpublic BufferedWriter tdErr;\n\n\tpublic class ProbIndexPair implements Comparable {\n\t\tpublic double prob;\n\t\tpublic int index;\n\t\tpublic ProbIndexPair(double prob, int index) {\n\t\t\tthis.prob = prob;\n\t\t\tthis.index = index;\n\t\t}\n\t\tpublic int compareTo(Object o) {\n\t\t\tif(prob < ((ProbIndexPair)o).prob) {\n\t\t\t\treturn -1;\n\t\t\t}\n\t\t\telse if(prob == ((ProbIndexPair)o).prob) {\n\t\t\t\treturn 0;\n\t\t\t}\n\t\t\telse {\n\t\t\t\treturn 1;\n\t\t\t}\n\t\t}\n\t}\n\n\t/**\n\t * @author espinosa\n\t * This is a singleton class containing functions for extracting various features\n\t * from the LF graph nodes. These\n\t * feature functions are called from ZLMaxentHypertagger#getFeatures.\n\t * \n\t * Some of the functions extract several features at once, to avoid unnecessary iteration.\n\t * Feature template abbreviations:\n\t * \n\t * FO -- fan-out, i.e. number of children\n\t * PN -- predicate name\n\t * RN -- parent name \n\t * CT -- type of child\n\t * A1N, A2N, ... -- Arg1 name, Arg2 name, ... (by default)\n\t * A1P, A2P, ... -- Arg1 POS tag, Arg2 POS tag, ... (by default)\n\t * MP -- Modifier POS tag (non-arg children)\n\t * PP -- parent's POS tag, if any parent\n\t * CN -- name of child\n\t * NA -- number of Argument children\n\t * PT -- POS tag (see docs) \n\t * ZD -- det=value\n\t * ZM -- mood=value\n\t * ZN -- num=value\n\t * ZT -- tense=value\n\t * ZP -- partic=value\n\t * XC -- semantic class of node, if applicable\n\t * XnD -- semantic class of argument child node n, if applicable (by default)\n\t * XP -- semantic class of parent node, if applicable\n\t * XM -- semantic class of non-arg child node, if applicable\n\t * CS -- child supertag\n\t * PS -- parent supertag\n\t * AS -- argument supertag\n\t * MS -- modifier supertag\n\t * \n\t */\n\t// mww: switched to configurable arg names\n\tvoid fillFeatures(LfGraphNode n, FeatureList f) {\n\n\t\tf.addFeatureWithProb(\"FO\", Integer.toString(n.getNumChildren()));\n\n\t\tfor(String att : n.getAttribs().keySet()) {\n\t\t\tf.addFeatureWithProb(\"Z\" + att.substring(0,1).toUpperCase(), n.getAttribs().get(att));\n\t\t}\n\n\t\tf.addFeatureWithProb(\"PN\", n.getPredicateName());\n\n\t\tif(n.getMultiParents().size() > 0) {\n\t\t\tfor(LfGraphNode parent : n.getMultiParents() ) {\n\t\t\t\tf.addFeatureWithProb(\"RN\", parent.getPredicateName());\n\t\t\t\t/* the line below will add the parent's best-ranked POS tag with prob=1.0 */\n\t\t\t\tf.addFeatureWithProb(\"PP\", getPOS(parent));\n\t\t\t\t/* The code below will add the parent's best-ranked POS tag with its actual probability */\n\t\t\t\t/*\n\t\t\t\t\t ArrayList> poslist = getBetaBestPOS(parent);\n\t\t\t\t\t Pair pospair = poslist.get(0);\n\t\t\t\t\t feats.addFeatureWithProb(\"PP\", pospair.a, pospair.b);\n\t\t\t\t */\n\t\t\t\t// add class name, if available\n\t\t\t\tString cls = parent.getPred().getNominal().toString();\n\t\t\t\t// string is in X:Y:Z format. Remove 'X:' leaving 'Y:Z'.\n\t\t\t\tif(cls != null && cls.indexOf(':') > 0) {\n\t\t\t\t\tString cfeat = cls.substring(cls.indexOf(':') + 1);\n\t\t\t\t\tf.addFeatureWithProb(\"XP\", cfeat);\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\telse {\n\t\t\tf.addFeatureWithProb(\"RN\", \"0\");\n\t\t}\n\t\tint argchildren = 0;\n\t\tfor(LfGraphLink lnk : n.getChildren()) {\n\t\t\tf.addFeatureWithProb(\"CT\", lnk.getLabel());\n\t\t\tif(lnk.getTarget() != null) {\n\t\t\t\t// how could it be null? \n\t\t\t\tf.addFeatureWithProb(\"CN\", lnk.getTarget().getPredicateName());\n\t\t\t\t// mww: use short arg name\n\t\t\t\tString shortArgName = argNameMap.get(lnk.getLabel());\n\t\t\t\tif (shortArgName != null) {\n\t\t\t\t\t// increment argchild count\n\t\t\t\t\targchildren++;\n\t\t\t\t\tf.addFeatureWithProb(shortArgName + \"N\", lnk.getTarget().getPredicateName());\n\t\t\t\t\tf.addFeatureWithProb(shortArgName + \"P\", getPOS(lnk.getTarget()) );\n\t\t\t\t\t// add class info for arg child, if applicable\n\t\t\t\t\tString cls = lnk.getTarget().getPred().getNominal().toString();\n\t\t\t\t\t// string is in X:Y:Z format. Remove 'X:' leaving 'Y:Z'.\n\t\t\t\t\tif(cls != null && cls.indexOf(':') > 0) {\n\t\t\t\t\t\tString cfeat = cls.substring(cls.indexOf(':') + 1);\n\t\t\t\t\t\t// mww: for backwards compatibility\n\t\t\t\t\t\tString argNumOrName = (shortArgName.startsWith(\"A\")) ? shortArgName.substring(1) : shortArgName;\n\t\t\t\t\t\tf.addFeatureWithProb(\"X\" + argNumOrName + \"D\", cfeat);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\t// not an argument child\n\t\t\t\t\tf.addFeatureWithProb(\"MP\", getPOS(lnk.getTarget()));\n\t\t\t\t\t// add class info for non-arg child, if applicable\n\t\t\t\t\tString cls = lnk.getTarget().getPred().getNominal().toString();\n\t\t\t\t\t// string is in X:Y:Z format. Remove 'X:' leaving 'Y:Z'.\n\t\t\t\t\tif(cls != null && cls.indexOf(':') > 0) {\n\t\t\t\t\t\tString cfeat = cls.substring(cls.indexOf(':') + 1);\n\t\t\t\t\t\tf.addFeatureWithProb(\"XM\", cfeat);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tf.addFeatureWithProb(\"NA\", Integer.toString(argchildren));\n\n\t\tif(posModel == null) {\n\t\t\tf.addFeatureWithProb(\"PT\", getPOS(n));\n\t\t}\n\t\telse {\n\t\t\tArrayList> poslist = getBetaBestPOS(n);\n\t\t\tfor(Pair pospair : poslist) {\n\t\t\t\t// add PT=NNP:0.7, PT=NN:0.6, etc\n\t\t\t\tf.addFeatureWithProb(\"PT\", pospair.a, pospair.b);\n\t\t\t\tn.setPOS(pospair.a); // take the top choice to be \"the pos tag\", used later for prior\n\t\t\t}\n\t\t}\n\t\t// class name feature for node, if available\n\t\tNominal idx = n.getPred().getNominal();\n\t\tString cls = idx.toString();\n\t\t// string is in X:Y:Z format. Remove 'X:' leaving 'Y:Z'.\n\t\tif(cls != null && cls.indexOf(':') > 0) {\n\t\t\tString cfeat = cls.substring(cls.indexOf(':') + 1);\n\t\t\tf.addFeatureWithProb(\"XC\", cfeat);\n\t\t}\n\t\t// prior features\n\t\tif(priorModel != null) {\n\t\t\tWord w = Word.createWord(n.getPredicateName(), null, null, n.getPredicateName(),\n\t\t\t\t\tgetPOS(n), null, null);\n\t\t\tpriorModel.computePriors(w);\n\t\t\tList> tags = priorModel.getBetaBestPriors(w, priorBeta);\n\t\t\tfor(Pair t : tags) {\n\t\t\t\tf.addFeatureWithProb(\"PR_ST\", DefaultTokenizer.unescape(t.a), t.b);\n\t\t\t}\n\t\t}\n\t}\n\n\tprivate void fillTwoPassFeatures(LfGraphNode n, FeatureList f) {\n\t\t// get the STs from the STList\n\t\tfor(Pair p : n.getSTList()) {\n\t\t\tf.addFeatureWithProb(\"ST\", p.a, p.b);\n\t\t}\n\t\t// get parent STs\n\t\tfor(LfGraphNode pl : n.getMultiParents()) {\n\t\t\tfor(Pair p : pl.getSTList()) {\n\t\t\t\tf.addFeatureWithProb(\"STP\", p.a, p.b);\n\t\t\t}\n\t\t}\n\t\tfor(LfGraphLink cl : n.getChildren()) {\n\t\t\t/*\n\t\t\tfor(Pair p : cl.getTarget().getSTList()) {\n\t\t\t*/\n\t\t\tLfGraphNode c = cl.getTarget();\n\t\t\t\tif(c != null) {\n\t\t\t\t\tfor(Pair p : c.getSTList()) {\n\t\t\t\t\t\tf.addFeatureWithProb(\"STC\", p.a, p.b);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t}\n\t}\n\t\n\tpublic ZLMaxentHypertagger() {\n\t\tsuper();\n\t\tbetas = new double[7];\n\t\t// values determined by DNM 20 April 2008).\n\t\tbetas[0] = 0.16;\n\t\tbetas[1] = 0.05;\n\t\tbetas[2] = 0.0058;\n\t\tbetas[3] = 0.00175;\n\t\tbetas[4] = 0.000625;\n\t\tbetas[5] = 0.000125;\n\t\tbetas[6] = 0.000058;\n\t\tcurrentBeta = 0;\n\t\tgoldPred2Tag = new HashMap();\n\t\tgoldPredPOS = new HashMap();\n\t}\n\n\t/** This constructor loads both a POS-tagging model and a hypertagging model. \n\t * \n\t * @param posModelFile The filename from which to load the POS-tagging model. If null,\n\t * gold-stardard POS tags will be used (and must be stored via storeGoldStdPredInfo()).\n\t * @param hyperModelFile The filename from which to load the hypertagging model\n\t * @throws IOException If any model fails to load.\n\t */\n\tpublic ZLMaxentHypertagger(File posModelFile, File hyperModelFile) throws IOException {\n\t\tthis();\n\t\tthis.posModelFilename = posModelFile;\n\t\t// load the models\n\t\tif(posModelFile != null) {\n\t\t\tthis.posModel = new ZLMaxentModel();\n\t\t\tthis.posModel.load(posModelFile);\n\t\t\tthis.postagger = new ZLPOSTagger(posModel);\n\t\t\tpostagger.setPrefixLength(4);\n\t\t\tpostagger.setSuffixLength(4);\n\t\t\tpostagger.argNameMap = this.argNameMap; // share the arg name map\n\t\t}\n\t\tthis.hypertagModelFilename = hyperModelFile;\n\t\tthis.hypertagModel = new ZLMaxentModel();\n\t\tthis.hypertagModel.load(hyperModelFile);\n\t}\n\t/* POS tagger can be null. In that case, gold-standard POS tags will be used.\n\t * To use in \"realization mode\", do the following:\n\t * 1) instantiate a ZLPOSTagger as normal\n\t * 2) load the POS prior model into the POS tagger\n\t * 3) Instantiate the ZLMaxentHypertagger with this constructor, passing the POS tagger\n\t * and the path to the trained hypertagging model file\n\t * 4) load the ht-prior model into the ZLMaxentHypertagger using setPriorModel(vocab, priorfile)\n\t * 5) commence tagging via setLF() and getSupertags()\n\t */\n\tpublic ZLMaxentHypertagger(ZLPOSTagger ptag, File hyperModelFile) {\n\t\tthis();\n\t\tthis.postagger = ptag;\n\t\tthis.postagger.argNameMap = this.argNameMap; // share the arg name map\n\t\tthis.hypertagModelFilename = hyperModelFile;\n\t\tthis.hypertagModel = new ZLMaxentModel();\n\t\tthis.hypertagModel.load(hyperModelFile);\n\t}\n\n\t/** This constructor loads only a POS-tagging model. It's useful for extracting features for Maxent training.\n\t * \n\t * @param posModelFile The filename from which to load the model.\n\t * @throws IOException If the model couldn't be loaded for any reason.\n\t */\n\tpublic ZLMaxentHypertagger(File posModelFile) throws IOException {\n\t\tthis();\n\t\tthis.posModelFilename = posModelFile;\n\t\t// load the model\n\t\tthis.posModel = new ZLMaxentModel();\n\t\tthis.posModel.load(posModelFile);\n\t\tthis.postagger = new ZLPOSTagger(posModel);\n\t\tpostagger.setPrefixLength(4);\n\t\tpostagger.setSuffixLength(4);\n\t\tpostagger.argNameMap = this.argNameMap; // share the arg name map\n\t}\n\n\tpublic static ZLMaxentHypertagger ZLMaxentHypertaggerFactory(String configFile) throws IOException {\n\t\tZLMaxentHypertagger hypertagger = new ZLMaxentHypertagger();\n\t\tZLPOSTagger postagger = null;\n\t\tString[] pathKeys = { \"priormodel\", \"priormodelvocab\", \"wdict\", \"posdict\", \"maxentmodel\", \"posconfig\" };\n\t\tMap opts = ConfigFileProcessor.readInConfig(configFile, pathKeys);\n\t\t// load the POS model first\n\t\tString posConfig = opts.get(\"posconfig\");\n\t\tif(posConfig != null) {\n\t\t\tpostagger = ZLPOSTagger.ZLPOSTaggerFactory(posConfig);\n\t\t}\n\t\thypertagger.postagger = postagger;\n\t\t// now load the prior models and/or tag dicts\n\t\t// if prior models are specified, then tagdicts are not used, or even loaded\n\t\tString priorModelPath = opts.get(\"priormodel\");\n\t\tString wdictPath = opts.get(\"wdict\");\n\t\tif(priorModelPath != null) {\n\t\t\tString vocabPath = opts.get(\"priormodelvocab\");\n\t\t\tif(vocabPath == null) {\n\t\t\t\tthrow new IOException(\"You must specify a vocab filename along with the prior model filename.\");\n\t\t\t}\n\t\t\thypertagger.loadPriorModel(new File(priorModelPath), new File(vocabPath));\n\t\t}\n\t\telse if(wdictPath != null) {\n\t\t\tString posdictPath = opts.get(\"posdict\");\n\t\t\tif(posdictPath == null) {\n\t\t\t\tthrow new IOException(\"You must specify both a word-level dict and a POS dict.\");\n\t\t\t}\n\t\t\tXMLWordDictionaryReader wdr = new XMLWordDictionaryReader(new File(wdictPath));\n\t\t\tXMLPOSDictionaryReader pdr = new XMLPOSDictionaryReader(new File(posdictPath));\n\t\t\tSTTaggerWordDictionary wdict = wdr.read();\n\t\t\tSTTaggerPOSDictionary posdict = pdr.read();\n\t\t\thypertagger.wdict = wdict;\n\t\t\thypertagger.posdict = posdict;\n\t\t\tString kstring = opts.get(\"dictk\");\n\t\t\tif(kstring != null) \n\t\t\t\thypertagger.dictK = Integer.parseInt(kstring);\n\t\t}\n\t\tString maxentModelPath = opts.get(\"maxentmodel\");\n\t\tif(maxentModelPath == null) {\n\t\t\tSystem.err.println(\"Maxent model path must be specified with key \\\"maxentmodel\\\".\");\n\t\t\tthrow new IOException();\n\t\t}\n\t\tString protoHTModelPath = opts.get(\"protomodel\");\n\t\t// process betas\n\t\tString betaString = opts.get(\"betas\");\n\t\tif(betaString != null) {\n\t\t\tString[] bs = betaString.split(\"\\\\s+\");\n\t\t\tdouble[] betaVals = new double[bs.length];\n\t\t\tint i;\n\t\t\tfor(i = 0; i < bs.length; i++) {\n\t\t\t\tbetaVals[i] = Double.parseDouble(bs[i]);\n\t\t\t}\n\t\t\t// can't sort descending w/o extra code and vars?\n\t\t\t// for now, assume betas are in correct order in configfile\n\t\t\t//Arrays.sort(betaVals);\n\t\t\thypertagger.betas = betaVals;\n\t\t}\n\t\thypertagger.hypertagModel = new ZLMaxentModel(maxentModelPath);\n\t\tif(protoHTModelPath != null) {\n\t\t\thypertagger.protoHTModel = new ZLMaxentModel(protoHTModelPath);\n\t\t\tSystem.err.println(\"Two-pass model instantiated. Initializing hyperdrive.\");\n\t\t}\n\t\t// mww: add argnames\n\t\tString argnames = opts.get(\"argnames\");\n\t\thypertagger.setArgNames(argnames);\n\t\treturn hypertagger;\n\t}\n\n\tprivate String getPOS(LfGraphNode n) {\n\t\tif(postagger == null) {\n\t\t\t// use gold-standard POS tag\n\t\t\tString posTag = goldPredPOS.get(n.getPred().getNominal().getName());\n\t\t\tif(posTag == null) {\n\t\t\t\t// XXX this is not correct\n\t\t\t\tposTag = \"UNK\";\n\t\t\t}\n\t\t\treturn posTag;\n\t\t}\n\t\t// else use POS model to get beta-best POS\n\t\tif(n.getPOS() == null) {\n\t\t\tFeatureList feats = postagger.getFeatures(n);\n\t\t\t//String pos = new String(postagger.getBestPOS(feats));\n\t\t\t// POS tags for nodes are cached in the node itself\n\t\t\t// so as to avoid repeated calls to the model for the same node\n\t\t\tArrayList> pos = postagger.getBetaBestPOS(feats);\n\t\t\tn.setPOS(pos.get(0).a);\n\t\t}\n\t\treturn n.getPOS();\n\t}\n\n\tprivate String getGoldSupertag(LfGraphNode n) {\n\t\treturn goldPred2Tag.get(n.getPred().getNominal().getName());\n\t}\n\n\tprivate ArrayList> getBetaBestPOS(LfGraphNode n) {\n\t\t// TODO (for training): add gold-standard POS tag, if it is not in the beta-best\n\t\t// list. Or should we use beta-best POS tags in training at all?\n\t\tif(n.getPOSList() == null) {\n\t\t\tFeatureList feats = postagger.getFeatures(n); \n\t\t\tArrayList> pos = null;\t\t\t\n\t\t\t//String pos = new String(postagger.getBestPOS(feats));\n\t\t\tpos = postagger.getBetaBestPOS(feats);\n\t\t\tn.setPOSList(pos);\n\t\t}\n\t\treturn n.getPOSList();\n\t}\n\n\t// computes all features for the graph node, returns a feature list\n\tprotected FeatureList getFeatures(LfGraphNode n) {\n\t\tFeatureList feats; \n\t\tfeats = n.getFeatures();\n\t\tif(feats != null) {\n\t\t\treturn feats;\n\t\t}\n\t\tfeats = new FeatureList();\n\t\tfillFeatures(n, feats);\n\t\treturn feats;\n\t}\n\n\t/** Get all features for the current LF as a single multi-line string.\n\t * Used during extraction of training data.\n\t * @return All features for the current LF.\n\t */\n\tpublic String getAllFeaturesAndAnswer() {\n\t\tStringBuilder output = new StringBuilder();\n\t\tfor(LfGraphNode n : nomTable.values()) {\n\t\t\tFeatureList feats = getFeatures(n);\n\t\t\tString context = feats.getAllFeatures();\n\t\t\t//String tag = n.getPred().getOrigin().getSupertag();\n\t\t\tString tag = goldPred2Tag.get(n.getPred().getNominal().getName());\n\t\t\toutput = output.append(tag + \" \" + context);\n\t\t\toutput = output.append(\"\\n\");\n\t\t}\n\t\treturn output.toString();\n\t}\n\n\t// for every node in the tree, compute its features, and store them inside the nodes\n\tprivate void assignAllFeatures() {\n\t\tfor(LfGraphNode n : nomTable.values()) {\n\t\t\tFeatureList feats = getFeatures(n);\n\t\t\tn.setFeatures(feats);\n\t\t\t// eval the proto-model, set the node's STList to the returned outcomes/probs\n\t\t\tif(protoHTModel != null) {\n\t\t\t\tn.setSTList(getProtoSupertagsAndProbs(n, protoHTBeta));\n\t\t\t}\n\t\t}\n\t\t// now traverse the graph again, calculating the additional STS features,\n\t\t// then unioning that feature list with the cached one\n\t\tif(protoHTModel != null) {\n\t\t\tfor(LfGraphNode n : nomTable.values()) {\n\t\t\t\tFeatureList feats = getFeatures(n); // feats guaranteed cached by first pass\n\t\t\t\tfillTwoPassFeatures(n, feats);\n\t\t\t\tn.setFeatures(feats); // necessary?\n\t\t\t}\n\t\t}\n\t}\n\n\tpublic ZLMaxentModel getPosModel() {\n\t\treturn posModel;\n\t}\n\n\tpublic File getPosModelFilename() {\n\t\treturn posModelFilename;\n\t}\n\n\tpublic void setLF(List preds) throws FeatureExtractionException {\n\t\tsuper.setLF(preds);\n\t\tif(postagger != null) {\n\t\t\tpostagger.setLF(preds);\n\t\t}\n\t\tassignAllFeatures();\n\t}\n\n\tpublic String getLFNum() {\n\t\treturn this.LFNum;\n\t}\n\n\tpublic void setLFNum(String s) {\n\t\tthis.LFNum = s;\n\t}\n\n\t/** Set the POS tagging model to use.\n\t * @param posModelFilename The filename containing the LBFGS model to use for\n\t * computing simple POS tags as part of the featureset.\n\t * @throws IOException when the model cannot be loaded for some reason.\n\t */\n\tpublic void setPOSModelFilename(File posModelFilename) throws IOException {\n\t\tthis.posModelFilename = posModelFilename;\n\t\t// load the model\n\t\tthis.posModel = new ZLMaxentModel();\n\t\tthis.posModel.load(posModelFilename);\n\t\tthis.postagger = new ZLPOSTagger(posModel); \n\t}\n\n\tpublic File getHypertagModelFilename() {\n\t\treturn hypertagModelFilename;\n\t}\n\n\t/** Set the POS tagging model to use.\n\t * @param posModelFilename The filename containing the LBFGS model to use for\n\t * computing simple POS tags as part of the featureset.\n\t * @throws IOException when the model cannot be loaded for some reason.\n\t */\n\tpublic void setHypertagModelFilename(File hypertagModelFilename) throws IOException {\n\t\tthis.hypertagModelFilename = hypertagModelFilename;\n\t\t// load the model\n\t\tthis.hypertagModel = new ZLMaxentModel();\n\t\tthis.hypertagModel.load(hypertagModelFilename);\n\t}\n\n\t/* methods from Hypertagger interface */\n\n\t// should probably just rename setLF to mapPreds in the first place\n\tpublic void mapPreds(List preds) {\n\t\ttry {\n\t\t\tsetLF(preds);\n\t\t}\n\t\tcatch(FeatureExtractionException e) {\n\t\t\tthrow (RuntimeException) new RuntimeException().initCause(e);\n\t\t}\n\t}\n\n\tpublic void setPred(int index) {\n\t\tcurrentPred = findNode(index);\n\t}\n\n\tpublic void resetBeta() {\n\t\tcurrentBeta = 0;\n\t}\n\tpublic void resetBetaToMax() {\n\t\tcurrentBeta = betas.length - 1;\n\t}\n\n\tpublic void nextBeta() {\n\t\tif(currentBeta < betas.length -1) {\n\t\t\tcurrentBeta++;\n\t\t}\n\t}\n\tpublic void previousBeta() {\n\t\tif(currentBeta > 0) {\n\t\t\tcurrentBeta--;\n\t\t}\n\t}\n\n\tpublic boolean hasMoreBetas() {\n\t\tif(currentBeta < betas.length - 1) {\n\t\t\treturn true;\n\t\t}\n\t\treturn false;\n\t}\n\n\tpublic boolean hasLessBetas() {\n\t\tif(currentBeta > 0 && this.hasMoreBetas()) {\n\t\t\treturn true;\n\t\t}\n\t\treturn false;\n\t}\n\n\n\tpublic Map getSupertags() {\n\t\tSet> tp = this.getSupertagsAndProbs(this.currentPred, this.betas[this.currentBeta]);\n\t\tif(tp == null) {\n\t\t\treturn null;\n\t\t}\n\t\tHashMap tagMap = new HashMap(tp.size());\n\t\tfor(Pair p : tp) {\n\t\t\ttagMap.put(p.a, p.b);\n\t\t}\n\t\treturn tagMap;\n\t}\n\n\t/* this method is for use with the ResultSink class, during testing. That code requires the elements of the pairs to be swapped. */\n\tprivate List> getSupertagsAsList() {\n\t\tSet> tp = this.getSupertagsAndProbs(this.currentPred, this.betas[this.currentBeta]);\n\t\tArrayList> ret = new ArrayList>();\n\t\tfor(Pair p : tp) {\n\t\t\tret.add(new Pair(p.b,p.a));\n\t\t}\n\t\treturn ret;\n\t}\n\t\n\t// cannot be made static due to ProbIndexPair\n\tprivate ArrayList getModelOutcomes(FeatureList f, ZLMaxentModel m) {\n\t\tArrayList probList = new ArrayList();\n\t\tdouble[] probs = m.eval(f.getAllFeaturesForMaxent(),true);\n\t\tfor(int i = 0; i < probs.length; i++) {\n\t\t\tprobList.add(new ProbIndexPair(probs[i], i));\n\t\t}\n\t\tCollections.sort(probList);\n\t\tCollections.reverse(probList);\n\t\treturn probList;\n\t}\n\n\tprivate HashSet> betaSearch(ArrayList probList, double beta) {\n\t\tdouble maxProb = probList.get(0).prob;\t\n\t\tHashSet> names = new HashSet>();\n\t\tfor(int i = 0; i < probList.size(); i++) {\n\t\t\tif(probList.get(i).prob >= beta * maxProb) {\n\t\t\t\tnames.add(new Pair(protoHTModel.getOutcome(probList.get(i).index), probList.get(i).prob));\n\t\t\t} else {\n\t\t\t\tbreak;\n\t\t\t}\n\t\t}\n\t\treturn names;\n\t}\n\n\tpublic Set> getProtoSupertagsAndProbs(LfGraphNode n, double beta) {\n\t\tHashSet> names;\n\t\tArrayList probList = new ArrayList();\n\t\tprobList = getModelOutcomes(n.getFeatures(), protoHTModel);\n\t\tnames = betaSearch(probList, beta);\n\t\treturn names;\n\t}\n\t\n\t/* 'MAIN' hypertagging entry point */\n\t@SuppressWarnings(\"boxing\")\n\tpublic Set> getSupertagsAndProbs(LfGraphNode n, double beta) {\n\t\tif(currentPred == null) {\n\t\t\treturn null; // not handling EPs for rels or feats, so return null\n\t\t}\n\t\tdouble bestOCProb = 0.0, curOCProb = 0.0;\n\t\tHashSet> names = new HashSet>();\n\t\tHashSet tagList = new HashSet();\n\t\tArrayList probList = new ArrayList();\n\t\tprobList = getModelOutcomes(n.getFeatures(), hypertagModel);\n\t\t/* check tagdict */\n\t\tif(wdict != null) {\n\t\t\t// get pred name, get tags from dict at K, if none, try pos dict\n\t\t\tString predName = n.getPredicateName();\n\t\t\tCollection permittedOutcomes = wdict.getEntry(predName, this.dictK);\n\t\t\tif(permittedOutcomes == null) {\n\t\t\t\t// back off to POS dict\n\t\t\t\tString pos = getPOS(n);\n\t\t\t\tpermittedOutcomes = posdict.getEntry(pos);\n\t\t\t\t// sanity check\n\t\t\t\tif(permittedOutcomes == null) {\n\t\t\t\t\tSystem.err.println(\"!! No pos dict entries for \" + pos);\n\t\t\t\t}\n\t\t\t}\n\t\t\tArrayList po = new ArrayList();\n\t\t\tfor(String s : permittedOutcomes) {\n\t\t\t\tpo.add(DefaultTokenizer.unescape(s));\n\t\t\t}\n\t\t\tpermittedOutcomes = (Collection)po;\n\t\t\tif(permittedOutcomes != null) {\n\t\t\t\tString oc = \"\";\n\t\t\t\tfor(ProbIndexPair p : probList) {\n\t\t\t\t\toc = hypertagModel.getOutcome(p.index);\n\t\t\t\t\tcurOCProb = p.prob;\n\t\t\t\t\tif(permittedOutcomes.contains(oc)) {\n\t\t\t\t\t\tif (bestOCProb == 0) {\n\t\t\t\t\t\t\tbestOCProb = curOCProb;\n\t\t\t\t\t\t}\n\t\t\t\t\t\tif (curOCProb >= (bestOCProb * beta)) { // Beta constraint.\n\t\t\t\t\t\t\t// The cut-off was met, add the outcome.\n\t\t\t\t\t\t\tnames.add(new Pair(oc, p.prob));\n\t\t\t\t\t\t\ttagList.add(oc);\n\t\t\t\t\t\t\t// update max, for first selected outcome\n\t\t\t\t\t\t\tif (curOCProb > bestOCProb) {\n\t\t\t\t\t\t\t\tbestOCProb = curOCProb;\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t} else {\n\t\t\t\t\t\t\t// Else, since our ProbIndexPair[] is sorted by probablity, there will be no more\n\t\t\t\t\t\t\t// outcomes that make the (beta) cut.\n\t\t\t\t\t\t\tbreak;\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\t// the outcome was ruled out by the tagdict. make a note of it.\n\t\t\t\t\t\tString nomId = currentPred.pred.getNominal().getName();\n\t\t\t\t\t\tString gsTag = \"\";\n\t\t\t\t\t\tString gsPos = \"\";\n\t\t\t\t\t\tif(goldPred2Tag.containsKey(nomId)) {\n\t\t\t\t\t\t\tgsTag = goldPred2Tag.get(nomId);\n\t\t\t\t\t\t\tgsPos = goldPredPOS.get(nomId);\n\t\t\t\t\t\t}\n\t\t\t\t\t\tif(gsTag.equals(oc)) {\n\t\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\t\ttdErr.write(predName + \"\\t\" + currentPred.index + \"\\t\" + LFNum + \"\\t\" + gsTag + \"\\t\" + gsPos + \"\\n\");\n\t\t\t\t\t\t\t\ttdErr.flush();\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\tcatch(Exception e) {\n\t\t\t\t\t\t\t\tthrow new RuntimeException(e);\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\telse { // not using dicts\n\t\t\tdouble maxProb = probList.get(0).prob;\t\n\t\t\tfor(int i = 0; i < probList.size(); i++) {\n\t\t\t\tif(probList.get(i).prob >= beta * maxProb) {\n\t\t\t\t\tnames.add(new Pair(hypertagModel.getOutcome(probList.get(i).index), probList.get(i).prob));\n\t\t\t\t\ttagList.add(hypertagModel.getOutcome(probList.get(i).index));\n\t\t\t\t} else {\n\t\t\t\t\tbreak;\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tString nomId=currentPred.pred.getNominal().getName();\n\t\tif(goldPred2Tag.containsKey(nomId)){\n\t\t\tString goldStdTag=goldPred2Tag.get(nomId);\n\t\t\tif(goldStdTagInsert && !tagList.contains(goldStdTag))\n\t\t\t\tnames.add(new Pair(goldStdTag,1.0));\n\t\t}\n\t\treturn names;\n\t}\n\n\tpublic double getCurrentBetaValue() {\n\t\tif(currentBeta \");\n\t\tfor(LfGraphNode n : nomTable.values()) {\n\t\t\tif(goldPred2Tag.get(n.getPred().getNominal().getName()) == null) {\n\t\t\t\tcontinue; // skip has-rel for now\n\t\t\t}\n\t\t\tout.append(DefaultTokenizer.escape(n.getPredicateName()));\n\t\t\tout.append(\":S-\");\n\t\t\tout.append(DefaultTokenizer.escape(n.getPredicateName()));\n\t\t\tout.append(\":P-\");\n\t\t\tout.append(DefaultTokenizer.escape(getPOS(n)));\n\t\t\tout.append(\":T-\");\n\t\t\tout.append(DefaultTokenizer.escape(goldPred2Tag.get(n.getPred().getNominal().getName())));\n\t\t\tout.append(\" \");\n\t\t}\n\t\tout.append(\"\\n\");\n\t\treturn out.toString();\n\t}\n\n\tpublic Word getPredAsWord(int idx) {\n\t\tLfGraphNode n = findNode(idx);\n\t\tWord w = Word.createWord(n.getPredicateName(), null, null, n.getPredicateName(),\n\t\t\t\tgetPOS(n), getGoldSupertag(n), null);\n\t\treturn w;\n\t}\n\tprivate Word getPredAsWord() {\n\t\tWord w = Word.createWord(currentPred.getPredicateName(), null, null, currentPred.getPredicateName(), getPOS(currentPred), getGoldSupertag(currentPred), null);\n\t\treturn w;\n\t}\n\n\t@SuppressWarnings(\"boxing\")\n\tpublic static void main(String[] args) throws IOException {\n\t\tString usage = \"\\nhypertagger (-i ) (-o [defaults to ]) (-c )\\n\";\n\t\tif (args.length > 0 && args[0].equals(\"-h\")) {\n\t\t\tSystem.out.println(usage);\n\t\t\tSystem.exit(0);\n\t\t}\n\t\tOptionParser o = new OptionParser();\n\t\to.acceptsAll(asList(\"help\", \"h\"), \"this message\");\n\t\to.acceptsAll(asList(\"quiet\", \"q\"), \"print no status messages\");\n\t\tOptionSpec b_s = o.acceptsAll(asList(\"beta\", \"b\"), \"ignore betas in config file and use this value\").withRequiredArg().ofType(Double.class);\n\t\tOptionSpec gr_s = o.acceptsAll(asList(\"g\", \"grammar\")).withRequiredArg().ofType(File.class).describedAs(\"grammar filename\");\n\t\tOptionSpec corpusDir_s = o.acceptsAll(asList(\"d\", \"lf-dir\")).withRequiredArg().ofType(File.class).describedAs(\"Directory to change to before searching for XML files\");\n\t\tOptionSpec configFile_s = o.acceptsAll(asList(\"c\", \"config\")).withRequiredArg().ofType(File.class).describedAs(\"configfilename\");\n\t\tOptionSpec output_s = o.acceptsAll(asList(\"o\", \"output\")).withRequiredArg().ofType(File.class).describedAs(\"output filename\");\n\t\tOptionSpec dump_s = o.acceptsAll(asList(\"dump-tags\", \"T\")).withRequiredArg().ofType(File.class).describedAs(\"dump predicted tags to file\");\n\t\to.acceptsAll(asList(\"goldstd\", \"G\"), \"include gold-standard supertags in tag dump\");\n\t\tOptionSet options = o.parse(args);\n\t\tFile outputF = options.valueOf(output_s);\n\t\tFile dumpF = options.valueOf(dump_s);\n\t\tFile configFile = options.valueOf(configFile_s);\n\t\tBufferedWriter out = null;\n\t\tBufferedWriter dump = null;\n\t\tArrayList resBetas;\n\t\tboolean quiet = options.has(\"q\");\n\t\tint lfcount = 0;\n\t\ttry {\n\t\t\tout = (output_s.equals(\"stdout\")) ? new BufferedWriter(new OutputStreamWriter(System.out)) : new BufferedWriter(new FileWriter(outputF));\n\t\t} \n\t\tcatch (IOException ex) {\n\t\t\tSystem.err.print(\"Output file \" + outputF + \" could not be opened. Exiting...\");\n\t\t\tLogger.getLogger(STPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n\t\t\tSystem.exit(1);\n\t\t} \n\t\tZLMaxentHypertagger ht = ZLMaxentHypertagger.ZLMaxentHypertaggerFactory(configFile.getAbsolutePath());\n\t\tif(options.has(\"T\")) {\n\t\t\ttry {\n\t\t\t\tdump = new BufferedWriter(new FileWriter(dumpF));\n\t\t\t}\n\t\t\tcatch(IOException e) {\n\t\t\t\tSystem.err.print(\"Output file \" + dumpF + \" could not be opened. Exiting...\");\n\t\t\t\tSystem.exit(1);\n\t\t\t}\n\t\t}\n\t\tif(options.has(\"b\")) {\n\t\t\tdouble beta = options.valueOf(b_s);\n\t\t\tht.betas = new double[1];\n\t\t\tht.betas[0] = beta;\n\t\t}\n\t\tresBetas = new ArrayList(ht.betas.length);\n\t\tfor(int i = 0; i < ht.betas.length; i++) {\n\t\t\tResultSink r = new ResultSink();\n\t\t\tresBetas.add(r);\n\t\t}\n\t\tArrayList errFiles = new ArrayList();\n\t\tfor(int i = 0; i < ht.betas.length; i++) {\n\t\t\tFile logdir = new File(\"logs\");\n\t\t\tif (!logdir.exists()) logdir.mkdirs();\n\t\t\tBufferedWriter b = new BufferedWriter(new FileWriter(new File(\"logs/tagdict.err.out.\" + i)));\n\t\t\tb.write(\"### beta = \" + ht.betas[i] + \"\\n\");\n\t\t\terrFiles.add(b);\n\t\t}\n\t\tLFLoader lfs = new LFLoader(options.valueOf(gr_s), options.valueOf(corpusDir_s), options.nonOptionArguments());\n\t\twhile(lfs.hasNext()) {\n\t\t\tlfcount++;\n\t\t\tLFInfo lfi = lfs.next();\n\t\t\ttry {\n\t\t\t\tht.setLF(lfi.getLF());\n\t\t\t\tht.storeGoldStdPredInfo(lfi.getFullWords());\n\t\t\t\tht.setLFNum(lfi.getLFNum());\n\t\t\t} \n\t\t\tcatch (FeatureExtractionException e) {\n\t\t\t\te.printStackTrace();\n\t\t\t}\n\t\t\tList>> lfTagging = new ArrayList>>();\n\t\t\tList gsTagging = new ArrayList();\n\t\t\tList> tags;\n\t\t\tWord w;\n\t\t\tfor(int bi = 0; bi < ht.betas.length; bi++) {\n\t\t\t\tgsTagging = new ArrayList();\n\t\t\t\tlfTagging = new ArrayList>>();\n\t\t\t\tht.setBetaIndex(bi);\n\t\t\t\tht.tdErr = errFiles.get(bi); \n\t\t\t\tfor(int i = 0; i < ht.maxIndex(); i++) {\n\t\t\t\t\tht.setPred(i);\n\t\t\t\t\tif(ht.currentPred == null) {\n\t\t\t\t\t\t//System.err.println(\"Skipping null pred \" + i);\n\t\t\t\t\t\tcontinue;\n\t\t\t\t\t}\n\t\t\t\t\tw = ht.getPredAsWord();\n\t\t\t\t\ttags = ht.getSupertagsAsList();\n\t\t\t\t\tlfTagging.add(tags);\n\t\t\t\t\tgsTagging.add(w);\n\t\t\t\t\tif(dump != null) {\n\t\t\t\t\t\tif(options.has(\"G\") && w.getSupertag() != null) {\n\t\t\t\t\t\t\tdump.write(w.getSupertag() + \" \");\n\t\t\t\t\t\t}\n\t\t\t\t\t\tdump.write(w.getForm() + \" \");\n\t\t\t\t\t\tfor(int j = 0; j < tags.size(); j++) {\n\t\t\t\t\t\t\tdump.write(tags.get(j).a + \" \" + tags.get(j).b + \" \");\n\t\t\t\t\t\t}\n\t\t\t\t\t\tdump.write(\"\\n\");\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tresBetas.get(bi).addSent(lfTagging, gsTagging);\n\t\t\t\tif(!quiet) {\n\t\t\t\t\tSystem.err.println(\"LFs processed: \" + lfcount + \"\\r\");\n\t\t\t\t}\n\t\t\t\tht.tdErr.flush();\n\t\t\t}\n\t\t}\n\t\tif(dump != null) {\n\t\t\tdump.flush();\n\t\t\tdump.close();\n\t\t}\n\t\tfor(int i = 0; i < ht.betas.length; i++) {\n\t\t\terrFiles.get(i).close();\n\t\t}\n\t\tfor(int i = 0; i < ht.betas.length; i++) {\n\t\t\tout.write(\"---------------\\n\");\n\t\t\tout.write(\"BETA: \" + ht.betas[i] + \"\\n\");\n\t\t\tout.write(resBetas.get(i).report());\n\t\t}\n\t\tout.flush();\n\t\tout.close();\n\t}\n\n\tprivate void setBetaIndex(int bi) {\n\t\tthis.currentBeta = bi;\n\t}\n}" }, { "path": "src/opennlp/ccg/realize/hypertagger/ZLMaxentModel.java", "content": "package opennlp.ccg.realize.hypertagger;\nimport java.io.BufferedReader;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.Map;\n\npublic class ZLMaxentModel {\n private MyIntegerPool intPool = new MyIntegerPool(50000);\n private double doubMax = 1.79769E+308;\n private ItemMap predMap = new ItemMap(this.intPool);\n private ItemMap outComeMap = new ItemMap(this.intPool);\n private Map>> params = new HashMap>>();\n private boolean loaded = false;\n private double[] probs;\n private int n_outcome;\n \n public ZLMaxentModel() {\n \n }\n public ZLMaxentModel(String modelFilename) {\n this.load(new File(modelFilename));\n }\n \n public void load(File modelFile) {\n if(!this.loaded) {\n this.loaded = true;\n BufferedReader br = null; \n try {\n br = new BufferedReader(new FileReader(modelFile));\n String line = br.readLine();\n \n if(line.contains(\"#\")) {\n //DEBUG: outf.write(line+\"\\n\");\n line = br.readLine();\n }\n // Read in contextual predicates.\n int numPreds = Integer.parseInt(line.trim());\n for(int i=0; i>> tempParamsMap = new ArrayList>>();\n int numParameters = this.predMap.size();\n ArrayList> prms;\n int fid = 0;\n String ln = \"\";\n for(int q=0; q>();\n Integer oid;\n for(int p=1; p(oid, this.intPool.getInt(fid)));\n fid++;\n }\n tempParamsMap.add(prms);\n }\n // Load theta.\n int nTheta = Integer.parseInt(br.readLine().trim());\n double[] theta = new double[nTheta];\n for(int z=0; z> tmpParamsList;\n int index = 0;\n for(ArrayList> param : tempParamsMap) {\n tmpParamsList = new ArrayList>();\n for(ZPair mapping : param) {\n //System.out.print(mapping.b.intValue()+\" \");\n tmpParamsList.add(new ZPair(mapping.a, new Double(theta[mapping.b.intValue()])));\n }\n this.params.put(this.intPool.getInt(index), tmpParamsList);\n index++;\n \n }\n \n this.n_outcome = this.outComeMap.size();\n this.probs = new double[this.n_outcome];\n // Initialise the array for computing distribution over all labels.\n } catch (FileNotFoundException e) {\n e.printStackTrace();\n } catch (IOException e) {\n e.printStackTrace();\n } finally {\n try {\n br.close();\n } catch(IOException ioe) {\n ioe.printStackTrace();\n }\n }\n }\n }\n \n public String getBestOutcome(double[] probs) {\n double maxprob = 0.0;\n int maxidx = -1;\n for(int i = 0; i < probs.length; i++) {\n if(probs[i] > maxprob) {\n maxidx = i;\n maxprob = probs[i];\n }\n }\n return getOutcome(maxidx);\n }\n \n @SuppressWarnings(\"boxing\")\n\t\tpublic double[] eval(String[] context, boolean realValued) { \n // Zero out prob distribution over labels.\n for(int i=0; i> pred_value =\n new ArrayList>(context.length);\n \n Double one = new Double(1.0);\n // Split up the strings into (pred,double) pairs.\n int splitPoint;\n for(int m=0; m(context[m].substring(0, splitPoint),\n Double.parseDouble(context[m].substring(splitPoint+1, context[m].length()))) );\n } else {\n pred_value.add(\n new ZPair(context[m], \n one) );\n }\n }\n \n \n ArrayList> featureWeights;\n //for(int j=0; j pv : pred_value) {\n Integer predID = this.predMap.id(pv.a);\n if(predID!=null) {\n featureWeights = this.params.get(predID);\n for(ZPair fw : featureWeights) {\n this.probs[fw.a.intValue()] += (fw.b.doubleValue() * pv.b.doubleValue());\n }\n }\n }\n \n double sum = 0.0;\n for(int p=0; p dict = new HashMap();\n private Map reverseDict = new HashMap();\n private MyIntegerPool intPool;\n \n public ItemMap(MyIntegerPool intPool) {\n this.intPool = intPool;\n this.index = intPool.getInt(0);\n }\n @SuppressWarnings(\"boxing\")\n\t\tpublic int add(String item) {\n if(this.dict.containsKey(item)) {\n return this.dict.get(item);\n } else {\n this.dict.put(item, index);\n this.reverseDict.put(index, item);\n this.index = this.intPool.getInt(index.intValue() + 1);\n return index.intValue()-1;\n \n }\n \n }\n \n public Integer id(String item) {\n if(this.dict.containsKey(item)) {\n return this.dict.get(item);\n } else {\n return null;\n }\n }\n \n public int size() {\n return this.dict.size();\n }\n \n public String getItem(int i) {\n return this.reverseDict.get(this.intPool.getInt(i));\n }\n}\n\n\nclass ZPair {\n public A a;\n public B b;\n \n public ZPair(A a, B b) {\n this.a = a;\n this.b = b;\n }\n}\n\n\nclass MyIntegerPool {\n private Integer[] _table;\n public MyIntegerPool(int size) {\n this._table = new Integer[size];\n for(int i=0; i= 0) {\n return this._table[i];\n } else {\n return new Integer(i);\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/realize/hypertagger/ZLPOSTagger.java", "content": "package opennlp.ccg.realize.hypertagger;\nimport java.io.BufferedReader;\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileReader;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.io.OutputStreamWriter;\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\nimport java.util.regex.Matcher;\nimport java.util.regex.Pattern;\n\nimport opennlp.ccg.hylo.Nominal;\nimport opennlp.ccg.lexicon.DefaultTokenizer;\nimport opennlp.ccg.parse.postagger.ml.POSPriorModel;\nimport opennlp.ccg.parse.tagger.util.ConfigFileProcessor;\nimport opennlp.ccg.parse.tagger.util.ResultSink;\nimport opennlp.ccg.util.Pair;\n\npublic class ZLPOSTagger extends TagExtractor {\n\tZLMaxentModel model;\n\tPOSPriorModel priorModel;\n\tint prefixLength = 4;\n\tint suffixLength = 4;\n\tdouble beta = 0.35;\n\tprivate HashMap goldPred2Tag=new HashMap();\n\tpublic class ProbIndexPair implements Comparable {\n\t\tpublic double prob;\n\t\tpublic int index;\n\t\tpublic ProbIndexPair(double prob, int index) {\n\t\t\tthis.prob = prob;\n\t\t\tthis.index = index;\n\t\t}\n\t\tpublic int compareTo(ProbIndexPair o) {\n\t\t\tif(prob < o.prob) {\n\t\t\t\treturn -1;\n\t\t\t}\n\t\t\telse if(prob == o.prob) {\n\t\t\t\treturn 0;\n\t\t\t}\n\t\t\telse {\n\t\t\t\treturn 1;\n\t\t\t}\n\t\t}\n\t}\n\n\tpublic ZLPOSTagger() { super(); }\n\n\tpublic ZLPOSTagger(ZLMaxentModel model) {\n\t\tthis();\n\t\tthis.model = model;\n\t\tthis.prefixLength = 4;\n\t\tthis.suffixLength = 4;\n\t\tthis.beta = 0.35; // 0.4 delivers 1.08 POStags/pred\n\t}\n\tpublic ZLPOSTagger(ZLMaxentModel model, POSPriorModel priorModel) {\n\t\tthis();\n\t\tthis.model = model;\n\t\tthis.prefixLength = 4;\n\t\tthis.suffixLength = 4;\n\t\tthis.beta = 0.35;\n\t\tthis.priorModel = priorModel;\n\t}\n\tpublic POSPriorModel getPriorModel() {\n\t\treturn priorModel;\n\t}\n\n\tpublic void setPriorModel(POSPriorModel priorModel) {\n\t\tthis.priorModel = priorModel;\n\t}\n\t/** get the features for one node\n\t * \n\t * FO -- fan-out, i.e. number of children\n\t * PN -- predicate name\n\t * RN -- parent name \n\t * RT -- parent relation\n\t * CT -- type of child\n\t * CN -- name of child\n\t * NA -- number of Argument children\n\t * A0N, A1N, ... -- names of argument children (by default)\n\t * PX -- prefix (N characters)\n\t * SX -- suffix (M characters)\n\t * HD -- has a digit\n\t * UH -- has an uppercase character or a hyphen\n\t * \n\t * @param n The graph node to extract features from\n\t * @return An array of strings representing the features\n\t */\t\n\t// mww: switched to configurable arg names\n\t@SuppressWarnings(\"boxing\")\n\tprotected FeatureList getFeatures(LfGraphNode n) {\n\t\tFeatureList feats = new FeatureList();\n\t\tint argchildren = 0;\n\t\tfeats.addFeature(\"PN\", n.getPredicateName());\n\t\tfeats.addFeature(\"FO\", Integer.toString(n.getNumChildren()));\n\t\t// add name of parent, if any parent, and parent relation\n\t\tif(n.getMultiParents().size() > 0) {\n\t\t\tfor(LfGraphNode parent : n.getMultiParents() ) {\n\t\t\t\tfeats.addFeature(\"RN\", parent.getPredicateName());\n\t\t\t\t// add class name, if available\n\t\t\t\tString cls = parent.getPred().getNominal().toString();\n\t\t\t\t// string is in X:Y:Z format. Remove 'X:' leaving 'Y:Z'.\n\t\t\t\tif(cls != null && cls.indexOf(':') > 0) {\n\t\t\t\t\tString cfeat = cls.substring(cls.indexOf(':') + 1);\n\t\t\t\t\tfeats.addFeature(\"XP\", cfeat);\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\telse {\n\t\t\tfeats.addFeature(\"RN\", \"0\");\n\t\t}\n\t\t// add types of children, count up argument children\n\t\tfor(LfGraphLink lnk : n.getChildren()) {\n\t\t\tfeats.addFeature(\"CT\", lnk.getLabel());\n\t\t\tif(lnk.getTarget() != null) { \n\t\t\t\tfeats.addFeature(\"CN\", lnk.getTarget().getPredicateName());\n\t\t\t\t// mww: use short arg name\n\t\t\t\tString shortArgName = argNameMap.get(lnk.getLabel());\n\t\t\t\tif (shortArgName != null) {\n\t\t\t\t\t// increment argchild count\n\t\t\t\t\targchildren++;\n\t\t\t\t\tfeats.addFeature(shortArgName + \"N\", lnk.getTarget().getPredicateName());\n\t\t\t\t\t// add class info for arg child, if applicable\n\t\t\t\t\tString cls = lnk.getTarget().getPred().getNominal().toString();\n\t\t\t\t\t// string is in X:Y:Z format. Remove 'X:' leaving 'Y:Z'.\n\t\t\t\t\tif(cls.indexOf(':') > 0) {\n\t\t\t\t\t\tString cfeat = cls.substring(cls.indexOf(':') + 1);\n\t\t\t\t\t\t// mww: for backwards compatibility\n\t\t\t\t\t\tString argNumOrName = (shortArgName.startsWith(\"A\")) ? shortArgName.substring(1) : shortArgName;\n\t\t\t\t\t\tfeats.addFeature(\"X\" + argNumOrName + \"D\", cfeat);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tfeats.addFeature(\"NA\", Integer.toString(argchildren));\n\t\t// class name feature, if available\n\t\tNominal idx = n.getPred().getNominal();\n\t\tString cls = idx.toString();\n\t\t// string is in X:Y:Z format. Remove 'X:' leaving 'Y:Z'.\n\t\tif(cls.indexOf(':') > 0) {\n\t\t\tString cfeat = cls.substring(cls.indexOf(':') + 1);\n\t\t\tfeats.addFeature(\"XC\", cfeat);\n\t\t}\n\t\t// compute prefix and suffix\n\t\tString predname = n.getPredicateName();\n\t\t// chop .04 part, if any\n\t\tPattern pat = Pattern.compile(\"(\\\\w+)\\\\.\\\\d+$\");\n\t\tString basePredName = new String(predname);\n\t\tMatcher mat = pat.matcher(basePredName);\n\t\tif(mat.matches()) {\n\t\t\tbasePredName = mat.group(1);\n\t\t}\n\t\tif(basePredName.length() > prefixLength + 1) {\n\t\t\tString prefix = predname.substring(0, prefixLength);\n\t\t\tfeats.addFeature(\"PX\", prefix);\n\t\t}\n\t\tif(basePredName.length() > suffixLength + 1) {\n\t\t\tString suffix = basePredName.substring(basePredName.length() - suffixLength, basePredName.length());\n\t\t\tfeats.addFeature(\"SX\", suffix);\n\t\t}\n\t\t// check for digit and/or (upcase char and hyphen)\n\t\tif(basePredName.matches(\"\\\\d+\")) {\n\t\t\tfeats.addFeature(\"HD\", \"1\");\n\t\t}\n\t\telse {\n\t\t\tfeats.addFeature(\"HD\", \"0\");\n\t\t}\n\t\tif(predname.matches(\"[A-Z]+\") || predname.matches(\"-+\")) {\n\t\t\t// XXX hack, because I don't think these PASS nodes should have a positive val\n\t\t\t// for this feature\n\t\t\tif(predname != \"PASS\") {\n\t\t\t\tfeats.addFeature(\"UH\", \"1\");\n\t\t\t}\n\t\t\telse {\n\t\t\t\tfeats.addFeature(\"UH\", \"0\");\n\t\t\t}\n\t\t}\n\t\telse {\n\t\t\tfeats.addFeature(\"UH\", \"0\");\n\t\t}\n\t\t// features from prior model, PPOS, if applicable\n\t\tif(priorModel != null) {\n\t\t\tList> priors = priorModel.getPriors(predname);\n\t\t\tdouble beta = 0.1;\n double best = priors.get(0).a;\n for(Pair prior : priors) {\n if(prior.a > (beta * best)) {\n // add the feature PPOS=:\n feats.addFeatureWithProb(\"PPOS\", prior.b, prior.a);\n } else {\n break;\n }\n }\n\t\t}\n\t\treturn feats;\n\t}\n\n\t/** Get the features for the index'th node\n\t * \n\t * @param index The index into the LF\n\t * @return An array of strings representing the features\n\t */\n\tpublic FeatureList getFeatures(int index) {\n\t\tfor(LfGraphNode n : nomTable.values()) {\n\t\t\tif(n.getIndex() == index) {\n\t\t\t\treturn getFeatures(n);\n\t\t\t}\n\t\t}\n\t\treturn null; // bad index\n\t}\n\tpublic int getPrefixLength() {\n\t\treturn prefixLength;\n\t}\n\tpublic void setPrefixLength(int prefixLength) {\n\t\tthis.prefixLength = prefixLength;\n\t}\n\tpublic int getSuffixLength() {\n\t\treturn suffixLength;\n\t}\n\tpublic void setSuffixLength(int suffixLength) {\n\t\tthis.suffixLength = suffixLength;\n\t}\n\t/** Get all features for the current LF as a single multi-line string.\n\t * Used during extraction of training data.\n\t * @return All features for the current LF.\n\t */\n\t// formerly known as getAllFeaturesAndPOS()\n\tpublic String getAllFeaturesAndAnswer() {\n\t\tStringBuilder output = new StringBuilder();\n\t\tfor(LfGraphNode n : nomTable.values()) {\n\t\t\tString feats = getFeatures(n).getAllFeatures();\n\t\t\tString postag = goldPred2Tag.get(n.getPred().getNominal().getName());\n\t\t\toutput = output.append(postag + \" \");\n\t\t\toutput = output.append(feats);\n\t\t\toutput = output.append(\"\\n\");\n\t\t}\n\t\treturn output.toString();\n\t}\n\t\n\tpublic String getAllFeaturesAndPOSWithID() {\n\t\tString output = \"\";\n\t\tfor(LfGraphNode n : nomTable.values()) {\n\t\t\tString feats = getFeatures(n).getAllFeatures();\n\t\t\tString postag = n.getPred().getOrigin().getPOS();\n\t\t\toutput = output.concat(\"<\" + Integer.toString(LFID) + \"> \" + postag + \" \" + feats);\n\t\t\toutput = output.concat(postag + \" \");\n\t\t\toutput = output.concat(feats);\n\t\t\toutput = output.concat(\"\\n\");\n\t\t}\n\t\treturn output;\n\t}\n\n\t/* Returns an arraylist of tuples (POS, probability) */\n\t@SuppressWarnings({ \"boxing\" })\n\tpublic ArrayList> getBetaBestPOS(FeatureList feats) {\n\t\tArrayList> poss = new ArrayList>();\n\t\tArrayList probList = new ArrayList();\n\t\tdouble[] probs = model.eval(feats.getAllFeaturesForMaxent(),true);\n\t\tfor(int i = 0; i < probs.length; i++) {\n\t\t\tprobList.add(new ProbIndexPair(probs[i], i));\n\t\t}\n\t\tCollections.sort(probList);\n\t\tCollections.reverse(probList);\n\t\tdouble maxProb = probList.get(0).prob;\n\t\tfor(int i = 0; i < probList.size(); i++) {\n\t\t\tif(probList.get(i).prob >= this.beta * maxProb) {\n\t\t\t\tposs.add(new Pair(model.getOutcome(probList.get(i).index), probList.get(i).prob));\n\t\t\t}\n\t\t}\n\t\treturn poss;\n\t}\n\t@SuppressWarnings(\"boxing\")\n\tpublic ArrayList> getBetaBestPOS(String[] feats) {\n\t\tArrayList> poss = new ArrayList>();\n\t\tArrayList probList = new ArrayList();\n\t\tdouble[] probs = model.eval(feats,true);\n\t\tfor(int i = 0; i < probs.length; i++) {\n\t\t\tprobList.add(new ProbIndexPair(probs[i], i));\n\t\t}\n\t\tCollections.sort(probList);\n\t\tCollections.reverse(probList);\n\t\tdouble maxProb = probList.get(0).prob;\n\t\tfor(int i = 0; i < probList.size(); i++) {\n\t\t\tif(probList.get(i).prob >= this.beta * maxProb) {\n\t\t\t\tposs.add(new Pair(model.getOutcome(probList.get(i).index), probList.get(i).prob));\n\t\t\t}\n\t\t}\n\t\treturn poss;\n\t}\n\n//\tfor tagging (i.e. evaluating feature list against model)\n\tpublic String getBestPOS(FeatureList feats) {\n\t\t// feed the features to the model, and get the best guess\n\t\t// at the POS tag given those features.\n\t\tdouble[] probs = model.eval(feats.getAllFeaturesForMaxent(),false);\n\t\treturn new String(model.getBestOutcome(probs));\n\t}\n\n\tpublic double getBeta() {\n\t\treturn beta;\n\t}\n\n\tpublic void setBeta(double beta) {\n\t\tthis.beta = beta;\n\t}\n\n\t/* (non-Javadoc)\n\t * @see opennlp.ccg.realize.hypertagger.TagExtractor#storeGoldStdPredInfo(java.lang.String)\n\t */\n\tpublic void storeGoldStdPredInfo(String predInfo) {\n\t\tString[] preds = predInfo.split(\"\\\\s+\");\n\t\tif(preds != null) {\n\t\t\tfor(int i = 1; i < preds.length; i++) {\n\t\t\t\tString[] info = preds[i].split(\":\");\n\t\t\t\tif(info.length != 4) {\n\t\t\t\t\tSystem.err.println(\"Malformed pred-info field, skipping (value was \\\"\" + preds[i] + \"\\\")\");\n\t\t\t\t\tcontinue;\n\t\t\t\t}\n\t\t\t\tgoldPred2Tag.put(info[0], DefaultTokenizer.unescape(info[2]));\n\t\t\t}\n\t\t} \n\t}\n\tpublic void loadPriorModel(File priorModelFile, File vocabFile) {\n\t\ttry {\n\t\t\tpriorModel = new POSPriorModel(priorModelFile.getAbsolutePath(), vocabFile.getAbsolutePath());\n\t\t} catch (IOException e) {\n\t\t\tSystem.err.println(\"Unable to load prior model or vocab file\");\n\t\t\te.printStackTrace();\n\t\t}\n\t}\n\t@SuppressWarnings({\"unused\" })\n\tpublic static void main(String[] args) throws IOException {\n\t\tString usage = \"\\nBasicPOSTagger -c (-i ) (-o [defaults to ])\\n\"+\n\t\t\" (-b beta value) (-m model file)\\n\";\n\t\tif (args.length > 0 && args[0].equals(\"-h\")) {\n\t\t\tSystem.out.println(usage);\n\t\t\tSystem.exit(0);\n\t\t}\n\n\t\t//SRILMFactoredBundleCorpusIterator inp = null;\n\t\t// input is just a file full of maxent events\n\t\t// i.e., we are not really tagging an lf, just testing a model by feeding it pre-extracted\n\t\t// events and comparing with the GS tags\n\t\tBufferedReader inp = null;\n\t\tBufferedWriter out = null;\n\t\ttry {\n\t\t\tString inputCorp = null;\n\t\t\tString configFile = null;\n\t\t\tString output = null;\n\t\t\tString modelFile = null;\n\t\t\tdouble beta = 0;\n\n\t\t\tboolean test = false;\n\n\t\t\tfor (int i = 0; i < args.length; i++) {\n\t\t\t\tif (args[i].equals(\"-i\")) { inputCorp = args[++i]; continue; }\n\t\t\t\tif (args[i].equals(\"-o\")) { output = args[++i]; continue; }\n\t\t\t\tif (args[i].equals(\"-c\")) { configFile = args[++i]; continue; }\n\t\t\t\tif (args[i].equals(\"-m\")) { modelFile = args[++i]; continue; }\n\t\t\t\tif (args[i].equals(\"-b\")) { beta = Double.parseDouble(args[++i]); continue; }\n\t\t\t\tSystem.out.println(\"Unrecognized option: \" + args[i]);\n\t\t\t}\n\n\t\t\tResultSink rs = new ResultSink(ResultSink.ResultSinkType.POSTAG);\n\n\t\t\ttry { \n\t\t\t\tinp = new BufferedReader(new FileReader(new File(inputCorp))); \n\t\t\t} catch (FileNotFoundException ex) {\n\t\t\t\tSystem.err.print(\"Input corpus \" + inputCorp + \" not found. Exiting...\");\n\t\t\t\tLogger.getLogger(POSPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n\t\t\t\tSystem.exit(-1);\n\t\t\t} \n\n\t\t\ttry {\n\t\t\t\tout = (output.equals(\"\")) ? new BufferedWriter(new OutputStreamWriter(System.out)) : new BufferedWriter(new FileWriter(new File(output)));\n\t\t\t} catch (IOException ex) {\n\t\t\t\tSystem.err.print(\"Output file \" + output + \" not found. Exiting...\");\n\t\t\t\tLogger.getLogger(POSPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n\t\t\t\tSystem.exit(-1);\n\t\t\t}\n\t\t\t//String[] pathKeys = { \"maxentmodel\", \"priormodel\", \"priormodelvocab\", \"sequencemodel\" };\n\t\t\t//Map opts = ConfigFileProcessor.readInConfig(configFile, pathKeys);\n\t\t\t//POSPriorModel posPrior = new POSPriorModel(opts.get(\"priormodel\"), opts.get(\"priormodelvocab\"));\n\t\t\tZLPOSTagger post = new ZLPOSTagger(new ZLMaxentModel(modelFile));\n\t\t\tif(beta > 0) { post.setBeta(beta); }\n\t\t\t//post.setPriorModel(posPrior); // unneeded?\n\t\t\tString line;\n\t\t\tint count = 0;\n\t\t\tint wins = 0;\n\t\t\tint tagsPerPred = 0;\n\t\t\twhile(true) {\n\t\t\t\tline = inp.readLine();\n\t\t\t\tif(line == null) { break; }\n\t\t\t\tcount++;\n\t\t\t\tint pos = line.indexOf(' ');\n\t\t\t\tString gs = line.substring(0,pos);\n\t\t\t\t//System.err.println(\"GS: \" + gs);\n\t\t\t\t//System.err.println(\"Featline: \" + line.substring(pos + 1));\n\t\t\t\tString[] feats = line.substring(pos+1).split(\"\\\\s+\");\n\t\t\t\tArrayList> ptags = post.getBetaBestPOS(feats);\n\t\t\t\ttagsPerPred += ptags.size();\n\t\t\t\t// now check for the win...\n\t\t\t\tfor(Pair p : ptags) {\n\t\t\t\t\tif(p.a.equals(gs)) {\n\t\t\t\t\t\twins++;\n\t\t\t\t\t\tbreak;\n\t\t\t\t\t}\n\t\t\t\t}\n\n\t\t\t}\n\t\t\t// for now, just print overall accuracy\n\t\t\tout.write(\"Beta: \" + post.getBeta() + \"\\n\");\n\t\t\tout.write(\"Acc: \" + (double)wins / (double)count * 100.0 + \"\\n\");\n\t\t\tout.write(\"Tags/Pred: \" + (double)tagsPerPred / (double)count + \"\\n\");\n\t\t\tout.flush();\n\t\t} catch(Throwable t) {\n\t\t\tt.printStackTrace();\n\t\t} finally {\n\t\t\ttry { \n\t\t\t\tinp.close();\n\t\t\t\tout.close();\n\t\t\t} catch (IOException ex) {\n\t\t\t\tLogger.getLogger(POSPriorModel.class.getName()).log(Level.SEVERE, null, ex);\n\t\t\t}\n\t\t}\n\t}\n\n\tpublic static ZLPOSTagger ZLPOSTaggerFactory(String configFile) throws IOException {\n\t\tZLPOSTagger postagger = new ZLPOSTagger();\n\t\tString[] pathKeys = { \"priormodel\", \"priormodelvocab\", \"maxentmodel\"};\n\t\tMap opts = ConfigFileProcessor.readInConfig(configFile, pathKeys);\n\t\tString priorModelFile = opts.get(\"priormodel\");\n\t\tif(priorModelFile != null) {\n\t\t\tString vocabFile = opts.get(\"priormodelvocab\");\n\t\t\tif(vocabFile == null) {\n\t\t\t\tthrow new IOException(\"A vocab file must be specified.\");\n\t\t\t}\n\t\t\tpostagger.loadPriorModel(new File(priorModelFile), new File(vocabFile));\n\t\t}\n\t\tString modelFile = opts.get(\"maxentmodel\");\n\t\tif(modelFile == null) {\n\t\t\tthrow new IOException(\"You must specify the maxent model to use.\");\n\t\t}\n\t\tpostagger.model = new ZLMaxentModel(modelFile);\n\t\tString betaString = opts.get(\"beta\");\n\t\tif(betaString != null) {\n\t\t\tdouble beta = Double.parseDouble(betaString);\n\t\t\tpostagger.beta = beta;\n\t\t}\n\t\treturn postagger;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/synsem/AbstractCat.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-5 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\nimport java.io.IOException;\nimport java.io.Serializable;\nimport gnu.trove.*;\nimport org.jdom.*;\n\nimport opennlp.ccg.hylo.*;\nimport opennlp.ccg.unify.*;\n\n/**\n * Abstract category adapter for CCG categories.\n *\n * @author Gann Bierner\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.22 $, $Date: 2011/05/22 03:40:55 $ \n */\npublic abstract class AbstractCat implements Category, Serializable { \n\n\tprivate static final long serialVersionUID = 1L;\n\n\t/** The feature structure, which should only be used with atomic categories. */\n protected FeatureStructure _featStruc;\n \n /** The logical form, which should be used only with the outermost category. */\n protected LF _lf;\n \n /** The hash code, if already computed. */\n private transient int _hashCode = -1;\n\n /** The hash code for the category without its LF, if already computed. */\n private transient int _hashCodeNoLF = -1;\n \n /** The mapping from vars to ints, if already computed. */\n private transient TObjectIntHashMap _varMap = null;\n\n /** The supertag, if already computed. */\n protected String _supertag = null;\n \n /** Default constructor. */\n public AbstractCat() {}\n\n /** Constructor which sets the LF. */\n public AbstractCat(LF lf) { _lf = lf; }\n\n /** \n * Constructor which retrieves the LF from the XML element \n * and flattens it to a conjunction of elementary predications\n * (or a single one). \n */\n public AbstractCat(Element elt) {\n Element lfElt = elt.getChild(\"lf\");\n if (lfElt != null) {\n _lf = HyloHelper.flattenLF(HyloHelper.getLF(lfElt));\n }\n }\n\n /**\n * Adds an XML element for the LF, if any, to the given catElt.\n * Uses {@link HyloHelper#toXml(LF)}.\n */\n public void toXml(Element catElt) {\n \tif (_lf != null) catElt.addContent(HyloHelper.toXml(_lf));\n }\n\n // during deserialization, intern computed supertag, and ensure varmap recomputed\n private void readObject(java.io.ObjectInputStream in) throws IOException, ClassNotFoundException {\n \tin.defaultReadObject();\n \tif (_supertag != null) _supertag = _supertag.intern();\n \t_varMap = null;\n }\n \n \n /** Gets the feature structure. */\n public FeatureStructure getFeatureStructure() { \n return _featStruc; \n }\n\n /** Sets the feature structure. */\n public void setFeatureStructure(FeatureStructure fs) { \n _featStruc = fs; \n }\n \n /** Gets the LF. */\n public LF getLF() { return _lf; }\n \n /** Sets the LF. */\n public void setLF(LF lf) { _lf = lf; }\n \n\n //-----------------------------------------------------------------\n // methods from Category which should be implemented by subclasses of\n // AbstractCat\n \n public abstract String toString();\n \n /**\n * Returns the supertag for the category.\n */\n public abstract String getSupertag();\n \n /**\n * This will return a TeX formatted representation for a category.\n * If toTeX() is not implemented for this category, the normal\n * toString() method is invoked instead.\n */\n public String toTeX()\n {\n return toString();\n }\n \n public abstract Category copy();\n public abstract Category shallowCopy();\n \n public abstract Object fill (Substitution s) throws UnifyFailure;\n public abstract void unifyCheck (Object u) throws UnifyFailure;\n /** NB: The LF does not participate in unification. */\n public abstract Object unify (Object u, Substitution sub) \n throws UnifyFailure;\n\n /**\n * Returns the target category of this category.\n */\n public abstract TargetCat getTarget();\n \n \n //-----------------------------------------------------------------\n // implemented methods from Category\n \n public boolean shallowEquals(Object o) { \n return equals(o); \n }\n\n public void deepMap(ModFcn mf) { \n if (_lf != null) _lf.deepMap(mf);\n mf.modify(this);\n }\n\n public void forall(CategoryFcn f) { \n f.forall(this); \n }\n \n public boolean occurs(Variable v) {\n if (_lf == null) return false;\n return _lf.occurs(v); \n }\n\n\n // methods to support printing of Categories\n public String prettyPrint() { return prettyPrint(\"\"); }\n protected String prettyPrint(String pad) { return pad+toString(); }\n\n protected int prettyLength(String s) {\n int max=0, cur=0;\n for(int i=0; i info) {\n\t\tList args = new ArrayList();\n\t\tfor (Iterator infoIt = info.iterator(); infoIt.hasNext();) {\n\t\t\tElement el = infoIt.next();\n\t\t\tString elName = el.getName();\n\t\t\tif (elName.equals(\"setarg\")) {\n\t\t\t\targs.add(new SetArg(el));\n\t\t\t\t_hasSet = true;\n\t\t\t} else if (elName.equals(\"dollar\")) {\n\t\t\t\tString name = el.getAttributeValue(\"name\");\n\t\t\t\tif (name == null)\n\t\t\t\t\tname = el.getAttributeValue(\"n\");\n\t\t\t\targs.add(new Dollar(name));\n\t\t\t\t_hasDollar = true;\n\t\t\t} else if (elName.equals(\"slash\") || elName.equals(\"sl\")) {\n\t\t\t\tSlash s = new Slash(el);\n\t\t\t\tElement argEl = infoIt.next();\n\t\t\t\tif (argEl.getName().equals(\"dollar\")) {\n\t\t\t\t\tString name = argEl.getAttributeValue(\"name\");\n\t\t\t\t\tif (name == null)\n\t\t\t\t\t\tname = argEl.getAttributeValue(\"n\");\n\t\t\t\t\targs.add(new Dollar(s, name));\n\t\t\t\t\t_hasDollar = true;\n\t\t\t\t} else {\n\t\t\t\t\targs.add(new BasicArg(s, CatReader.getCat(argEl)));\n\t\t\t\t}\n\t\t\t} else {\n\t\t\t\tSystem.out.println(\"Invalid element for creating ArgStack: \"\n\t\t\t\t\t\t+ elName);\n\t\t\t}\n\t\t}\n\t\t_list = new Arg[args.size()];\n\t\targs.toArray(_list);\n\t}\n\n\tpublic void toXml(Element catElt) {\n\t\tfor (Arg arg: _list) {\n\t\t\tif (arg instanceof SetArg)\n\t\t\t\tcatElt.addContent(((SetArg) arg).toXml());\n\t\t\telse if (arg instanceof Dollar) {\n\t\t\t\tDollar dollar = (Dollar) arg;\n\t\t\t\tif (!dollar.getSlash().toString().equals(\"|.\"))\n\t\t\t\t\tcatElt.addContent(dollar.getSlash().toXml());\n\t\t\t\tElement dollarElt = new Element(\"dollar\");\n\t\t\t\tdollarElt.setAttribute(\"name\", dollar.name());\n\t\t\t\tcatElt.addContent(dollarElt);\n\t\t\t}\n\t\t\telse if (arg instanceof BasicArg) {\n\t\t\t\tBasicArg barg = (BasicArg) arg;\n\t\t\t\tcatElt.addContent(barg.getSlash().toXml());\n\t\t\t\tcatElt.addContent(barg.getCat().toXml());\n\t\t\t}\n\t\t}\n\t}\n\t\n\tpublic void addAt(Arg c, int index) {\n\t\tArg[] $list = new Arg[_list.length + 1];\n\t\tinsert(subList(0, index)._list, $list, 0);\n\t\t$list[index] = c;\n\t\tinsert(subList(index)._list, $list, index + 1);\n\t\t_list = $list;\n\t\tif (c instanceof Dollar) {\n\t\t\t_hasDollar = true;\n\t\t} else if (c instanceof SetArg) {\n\t\t\t_hasSet = true;\n\t\t}\n\t}\n\n\tpublic void add(Arg c) {\n\t\tArg[] $list = new Arg[_list.length + 1];\n\t\tint last = insert(_list, $list, 0);\n\t\t$list[last] = c;\n\t\t_list = $list;\n\t\tif (c instanceof Dollar) {\n\t\t\t_hasDollar = true;\n\t\t} else if (c instanceof SetArg) {\n\t\t\t_hasSet = true;\n\t\t}\n\t}\n\n\tpublic void addAt(ArgStack cl, int index) {\n\t\tArg[] $list = new Arg[_list.length + cl._list.length];\n\t\tint last = insert(subList(0, index)._list, $list, 0);\n\t\tlast = insert(cl._list, $list, last);\n\t\tinsert(subList(index)._list, $list, last);\n\t\t_list = $list;\n\t\tif (cl.containsDollarArg()) {\n\t\t\t_hasDollar = true;\n\t\t} else if (cl.containsSetArg()) {\n\t\t\t_hasSet = true;\n\t\t}\n\t}\n\n\tpublic void add(ArgStack cl) {\n\t\tArg[] $list = new Arg[_list.length + cl._list.length];\n\t\tint last = insert(_list, $list, 0);\n\t\tinsert(cl._list, $list, last);\n\t\t_list = $list;\n\t\tif (cl.containsDollarArg()) {\n\t\t\t_hasDollar = true;\n\t\t} else if (cl.containsSetArg()) {\n\t\t\t_hasSet = true;\n\t\t}\n\t}\n\n\tpublic void addFront(Arg c) {\n\t\tArg[] $list = new Arg[_list.length + 1];\n\t\t$list[0] = c;\n\t\tinsert(_list, $list, 1);\n\t\t_list = $list;\n\t\tif (c instanceof Dollar) {\n\t\t\t_hasDollar = true;\n\t\t} else if (c instanceof SetArg) {\n\t\t\t_hasSet = true;\n\t\t}\n\t}\n\n\tpublic void addFront(ArgStack cl) {\n\t\tArg[] $list = new Arg[_list.length + cl._list.length];\n\t\tint last = insert(cl._list, $list, 0);\n\t\tinsert(_list, $list, last);\n\t\t_list = $list;\n\t\tif (cl.containsDollarArg()) {\n\t\t\t_hasDollar = true;\n\t\t} else if (cl.containsSetArg()) {\n\t\t\t_hasSet = true;\n\t\t}\n\t}\n\n\tpublic void insertFront(ArgStack cl) {\n\t\tinsertAt(cl, 0);\n\t}\n\n\tpublic void insertEnd(ArgStack cl) {\n\t\tinsertAt(cl, _list.length - 1);\n\t}\n\n\tpublic void insertAt(ArgStack cl, int index) {\n\t\tArg insertInto = _list[index];\n\t\tif (insertInto instanceof BasicArg) {\n\t\t\tcl.add(insertInto);\n\t\t\t_list[index] = new SetArg(cl);\n\t\t\t_hasSet = true;\n\t\t} else if (insertInto instanceof SetArg) {\n\t\t\t((SetArg) insertInto).add(cl);\n\t\t} else {\n\t\t\tSystem.out.println(\"Problem inserting arg stack: \" + cl);\n\t\t}\n\t}\n\n\tpublic int size() {\n\t\treturn _list.length;\n\t}\n\n\tpublic boolean containsDollarArg() {\n\t\treturn _hasDollar;\n\t}\n\n\tpublic boolean containsSetArg() {\n\t\treturn _hasSet;\n\t}\n\n\tpublic Arg get(int i) {\n\t\treturn _list[i];\n\t}\n\n\tpublic void set(int i, Arg c) {\n\t\t_list[i] = c;\n\t\tif (c instanceof Dollar) {\n\t\t\t_hasDollar = true;\n\t\t} else if (c instanceof SetArg) {\n\t\t\t_hasSet = true;\n\t\t}\n\t}\n\n\tpublic Arg getLast() {\n\t\treturn _list[_list.length - 1];\n\t}\n\n\tpublic void setLast(Arg c) {\n\t\tset(_list.length - 1, c);\n\t}\n\n /** Sets the harmonic composition result of each arg's slash. */\n public void setSlashHarmonicCompositionResult(boolean harmonicResult) {\n \tfor (int i=0; i < _list.length; i++) {\n \t\t_list[i].setSlashHarmonicCompositionResult(harmonicResult);\n \t}\n }\n \n\tpublic ArgStack copy() {\n\t\tArg[] $list = new Arg[_list.length];\n\t\tfor (int i = 0; i < $list.length; i++) {\n\t\t\t$list[i] = _list[i].copy();\n\t\t}\n\t\treturn new ArgStack($list);\n\t}\n\n\tpublic ArgStack copyWithout(int indexToRemove) {\n\t\tArg[] $list = new Arg[_list.length - 1];\n\t\tif ($list.length < 1) {\n\t\t\tSystem.out.println(\"Removing last item from an argument stack!\");\n\t\t}\n\t\tint index = 0;\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t\tif (i != indexToRemove) {\n\t\t\t\t$list[index++] = _list[i].copy();\n\t\t\t}\n\t\t}\n\t\treturn new ArgStack($list);\n\t}\n\n\tpublic ArgStack subList(int from) {\n\t\treturn subList(from, _list.length);\n\t}\n\n\tpublic ArgStack subList(int from, int upto) {\n\t\tArg[] $list;\n\t\tif (upto > from) {\n\t\t\t$list = new Arg[upto - from];\n\t\t\tint index = 0;\n\t\t\tfor (int i = from; i < upto; i++) {\n\t\t\t\t$list[index++] = _list[i];\n\t\t\t}\n\t\t} else {\n\t\t\t$list = new Arg[0];\n\t\t}\n\t\treturn new ArgStack($list);\n\t}\n\n\tpublic ArgStack shallowCopy() {\n\t\treturn new ArgStack(_list);\n\t}\n\n\tpublic boolean occurs(Variable v) {\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t\tif (_list[i].occurs(v)) {\n\t\t\t\treturn true;\n\t\t\t}\n\t\t}\n\t\treturn false;\n\t}\n\n\tpublic ArgStack fill(Substitution s) throws UnifyFailure {\n\t\tArgStack args = new ArgStack();\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t\tObject value = _list[i].fill(s);\n\t\t\tif (value instanceof ArgStack) {\n\t\t\t\targs.add((ArgStack) value);\n\t\t\t} else {\n\t\t\t\targs.add((Arg) value);\n\t\t\t}\n\t\t}\n\t\treturn args;\n\t}\n\n\tpublic void deepMap(ModFcn mf) {\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t\t_list[i].deepMap(mf);\n\t\t}\n\t}\n\n\tpublic boolean containsContrarySlash() {\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t\tif (_list[i] instanceof BasicArg\n\t\t\t\t\t&& !((BasicArg) _list[i]).getSlash().sameDirAsModality()) {\n\t\t\t\treturn true;\n\t\t\t} else if (_list[i] instanceof SetArg\n\t\t\t\t\t&& ((SetArg) _list[i]).containsContrarySlash()) {\n\t\t\t\treturn true;\n\t\t\t}\n\t\t}\n\t\treturn false;\n\t}\n\n\tpublic void slashesUnify(Slash s) throws UnifyFailure {\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t\t_list[i].unifySlash(s);\n\t\t}\n\t}\n\n\tpublic int unifySuffix(ArgStack as, Substitution sub) throws UnifyFailure {\n\n\t\tint asIndex = as.size();\n\t\tfor (int i = _list.length - 1; i >= 0; i--) {\n\t\t\tasIndex--;\n\t\t\tget(i).unify(as.get(asIndex), sub);\n\t\t}\n\t\treturn asIndex;\n\t}\n\n\tpublic ArgStack unify(ArgStack as, Substitution sub) throws UnifyFailure {\n\t\treturn unifyPrefix(as, as.size(), sub);\n\t}\n\n\tpublic ArgStack unifyPrefix(ArgStack as, int upto, Substitution sub)\n\t\t\tthrows UnifyFailure {\n\n\t\tArgStack $args;\n\t\tif (containsDollarArg()) {\n\t\t\tif (as.containsDollarArg()) {\n\t\t\t\t$args = unifyDollarWithDollar(as, upto, sub);\n\t\t\t} else {\n\t\t\t\t$args = unifyDollarWithNoDollar(size(), as, upto, sub);\n\t\t\t}\n\t\t} else if (as.containsDollarArg()) {\n\t\t\t$args = as.unifyDollarWithNoDollar(upto, this, size(), sub);\n\t\t} else if (size() == upto) {\n\t\t\t$args = unifySimple(as, upto, sub);\n\t\t} else {\n\t\t\t$args = unifyComplex(as, upto, sub);\n\t\t\t// throw new UnifyFailure();\n\t\t}\n\t\treturn $args;\n\t}\n\n\tprivate ArgStack unifySimple(ArgStack as, int upto, Substitution sub)\n\t\t\tthrows UnifyFailure {\n\n\t\tArgStack $args = new ArgStack();\n\t\tfor (int i = upto - 1; i >= 0; i--) {\n\t\t\t$args.addFront((Arg) _list[i].unify(as.get(i), sub));\n\t\t}\n\t\treturn $args;\n\t}\n\n\tprivate ArgStack unifyComplex(ArgStack as, int upto, Substitution sub)\n\t\t\tthrows UnifyFailure {\n\n\t\tArgStack $args = new ArgStack();\n\n\t\tint aIndex = size() - 1;\n\t\tint bIndex = upto - 1;\n\t\twhile (aIndex >= 0 && bIndex >= 0) {\n\t\t\t// while (null != aArg && null != bArg) {\n\t\t\tArg aArg = get(aIndex);\n\t\t\tArg bArg = as.get(bIndex);\n\n\t\t\tif ((aArg instanceof BasicArg && bArg instanceof BasicArg)\n\t\t\t\t\t|| (aArg instanceof SetArg && bArg instanceof SetArg)) {\n\t\t\t\t$args.addFront((Arg) aArg.unify(bArg, sub));\n\t\t\t\taIndex--;\n\t\t\t\tbIndex--;\n\t\t\t} else if (aArg instanceof BasicArg && bArg instanceof SetArg) {\n\t\t\t\tint setsize = ((SetArg) bArg).size();\n\t\t\t\tif (setsize <= aIndex + 1) {\n\t\t\t\t\tint stop = aIndex - setsize;\n\t\t\t\t\tfor (; aIndex > stop;) {\n\t\t\t\t\t\taIndex--;\n\t\t\t\t\t\tif (bArg instanceof BasicArg) {\n\t\t\t\t\t\t\t$args.addFront((Arg) aArg.unify(bArg, sub));\n\t\t\t\t\t\t} else {\n\t\t\t\t\t\t\tint idInSet = ((SetArg) bArg)\n\t\t\t\t\t\t\t\t\t.indexOf((BasicArg) aArg);\n\t\t\t\t\t\t\tif (idInSet == -1)\n\t\t\t\t\t\t\t\tthrow new UnifyFailure();\n\t\t\t\t\t\t\t$args.addFront((Arg) aArg.unify(((SetArg) bArg)\n\t\t\t\t\t\t\t\t\t.get(idInSet), sub));\n\t\t\t\t\t\t\taArg = get(aIndex);\n\t\t\t\t\t\t\tbArg = ((SetArg) bArg).copyWithout(idInSet);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\tbIndex--;\n\t\t\t\t} else {\n\t\t\t\t\tthrow new UnifyFailure();\n\t\t\t\t}\n\t\t\t} else if (aArg instanceof SetArg && bArg instanceof BasicArg) {\n\t\t\t\tthrow new UnifyFailure();\n\t\t\t} else {\n\t\t\t\tthrow new UnifyFailure();\n\t\t\t}\n\t\t}\n\t\tif (aIndex > -1 || bIndex > -1) {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\t\treturn $args;\n\t}\n\n\tprivate ArgStack unifyDollarWithNoDollar(int uptoThis, ArgStack otherStack,\n\t\t\tint uptoOther, Substitution sub) throws UnifyFailure {\n\n\t\tif ((!(_hasSet || otherStack._hasSet) && uptoThis > uptoOther + 1)\n\t\t\t\t|| (uptoThis > 1 && uptoOther < 1)) {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\t\tArgStack $args = new ArgStack();\n\t\totherStack = otherStack.subList(0, uptoOther);\n\t\tint otherIndex = uptoOther - 1;\n\n\t\tfor (int i = uptoThis - 1; i >= 0; i--) {\n\t\t\tArg argi = get(i);\n\t\t\tif (argi instanceof Dollar) {\n\t\t\t\tif (i > 0) {\n\t\t\t\t\tthrow new UnifyFailure();\n\t\t\t\t} else {\n\t\t\t\t\tArgStack $subArgs = otherStack.subList(0, otherIndex + 1);\n\t\t\t\t\t// Slash dsl = ((Dollar) argi).getSlash();\n\t\t\t\t\t((Dollar) argi).unify($subArgs.copy(), sub);\n\t\t\t\t\totherIndex = 0;\n\t\t\t\t\t$args.addFront($subArgs);\n\t\t\t\t}\n\t\t\t} else if (argi instanceof BasicArg) {\n\t\t\t\tif (otherIndex < 0) {\n\t\t\t\t\tthrow new UnifyFailure();\n\t\t\t\t}\n\n\t\t\t\tArg otherArg = otherStack.get(otherIndex);\n\t\t\t\tif (otherArg instanceof BasicArg) {\n\t\t\t\t\t$args.addFront((Arg) argi.unify(otherArg, sub));\n\t\t\t\t\totherIndex--;\n\t\t\t\t} else if (otherArg instanceof SetArg) {\n\t\t\t\t\tSetArg sa = (SetArg) otherArg;\n\t\t\t\t\tint id = sa.indexOf((BasicArg) argi);\n\t\t\t\t\tif (id == -1)\n\t\t\t\t\t\tthrow new UnifyFailure();\n\t\t\t\t\t$args.addFront((Arg) argi.unify(sa.get(id), sub));\n\t\t\t\t\totherStack.set(otherIndex, sa.copyWithout(id));\n\t\t\t\t}\n\t\t\t} else {\n\t\t\t\tthrow new UnifyFailure();\n\t\t\t}\n\t\t}\n\t\tif (otherIndex > 0) {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\t\treturn $args;\n\t}\n\n\tprivate ArgStack unifyDollarWithDollar(ArgStack as, int upto,\n\t\t\tSubstitution sub) throws UnifyFailure {\n\n\t\tArgStack $args;\n\t\tif (size() == 1) {\n\t\t\t$args = as.subList(0, upto);\n\t\t\t((Dollar) get(0)).unify($args.copy(), sub);\n\t\t} else if (upto == 1) {\n\t\t\t$args = subList(0, size());\n\t\t\t((Dollar) as.get(0)).unify($args.copy(), sub);\n\t\t} else if (upto == size()) {\n\t\t\t$args = unifySimple(as, upto, sub);\n\t\t} else {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\t\treturn $args;\n\t}\n\n\tpublic void forall(CategoryFcn fcn) {\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t\t_list[i].forall(fcn);\n\t\t}\n\t}\n\n\tprivate void checkForDollar() {\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t\tif (_list[i] instanceof Dollar) {\n\t\t\t\t_hasDollar = true;\n\t\t\t\treturn;\n\t\t\t}\n\t\t}\n\t}\n\n\tprivate void checkForSet() {\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t\tif (_list[i] instanceof SetArg) {\n\t\t\t\t_hasSet = true;\n\t\t\t\treturn;\n\t\t\t}\n\t\t}\n\t}\n\n\tpublic String toString() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t\tsb.append(_list[i].toString());\n\t\t}\n\t\treturn sb.toString();\n\t}\n\n// \tprivate boolean methodExists(Object o, String methodName) {\n// \t\tjava.lang.reflect.Method[] m = o.getClass().getMethods();\n// \t\tfor (int i = 0; i < m.length; i++)\n// \t\t\tif (m[i].getName() == methodName) {\n// \t\t\t\tif (m[i].getDeclaringClass().toString().startsWith(\"class\"))\n// \t\t\t\t\treturn true;\n// \t\t\t\telse\n// \t\t\t\t\treturn false;\n// \t\t\t}\n// \t\treturn false;\n// \t}\n\n\t/**\n\t * Returns the supertag for this arg stack.\n\t */\n\tpublic String getSupertag() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t\tArg arg = (Arg) _list[i];\n\t\t\tsb.append(arg.getSupertag());\n\t\t}\n\t\treturn sb.toString();\n\t}\n\n\t/**\n\t * Returns a TeX-formatted string representation for this arg stack.\n\t */\n\tpublic String toTeX() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t sb.append(_list[i].toTeX());\n\t\t}\n\t\treturn sb.toString();\n\t}\n\n\t/**\n\t * Returns a hash code using the given map from vars to ints.\n\t */\n\tpublic int hashCode(TObjectIntHashMap varMap) {\n\t\tint retval = 0;\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t\tretval += _list[i].hashCode(varMap);\n\t\t}\n\t\treturn retval;\n\t}\n\n\t/**\n\t * Returns whether this arg stack equals the given object up to variable\n\t * names, using the given maps from vars to ints.\n\t */\n\tpublic boolean equals(Object obj, TObjectIntHashMap varMap,\n\t\t\tTObjectIntHashMap varMap2) {\n\t\tif (obj.getClass() != this.getClass()) {\n\t\t\treturn false;\n\t\t}\n\t\tArgStack as = (ArgStack) obj;\n\t\tif (_list.length != as._list.length) {\n\t\t\treturn false;\n\t\t}\n\t\tfor (int i = 0; i < _list.length; i++) {\n\t\t\tif (!_list[i].equals(as._list[i], varMap, varMap2)) {\n\t\t\t\treturn false;\n\t\t\t}\n\t\t}\n\t\treturn true;\n\t}\n\n\tprotected static int insert(Arg[] a, Arg[] b, int pos) {\n\t\tfor (int i = 0; i < a.length; i++) {\n\t\t\tb[pos++] = a[i];\n\t\t}\n\t\treturn pos;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/synsem/AtomCat.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-11 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.util.DisplayPrefs;\nimport opennlp.ccg.grammar.Grammar;\nimport org.jdom.*;\n\nimport gnu.trove.*;\n\n/**\n * The most basic CG category. This corresponds to a category like 'np[acc]',\n * i.e. a category name and associated features.\n *\n * @author Gann Bierner\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.18 $, $Date: 2011/07/15 03:02:53 $\n */\npublic final class AtomCat extends AbstractCat implements TargetCat {\n\n private static final long serialVersionUID = 1L;\n\n private String type;\n\n /** Flag indicating whether this cat is a fragment with completion status true (defaults to false). */\n public boolean fragCompletion = false;\n \n /** Constructor which creates an atomic category with the given type. */\n public AtomCat(String t) {\n this(t, new GFeatStruc());\n }\n\n /** Constructor which creates an atomic category with the given type and feature structure. */\n public AtomCat(String t, FeatureStructure fs) {\n this(t, fs, null); \n }\n\n /** Constructor which creates an atomic category with the given type, feature structure and LF. */\n public AtomCat(String t, FeatureStructure fs, LF lf) {\n super(lf);\n type = t; \n _featStruc = fs; \n }\n\n /** Constructor which retrieves the atomic category from the XML element. */\n public AtomCat(Element acel) {\n // call super to get LF if present\n super(acel);\n // get type\n type = acel.getAttributeValue(\"type\");\n if (type == null) type = acel.getAttributeValue(\"t\"); \n // get feature structure\n Element fsEl = acel.getChild(\"fs\");\n if (fsEl != null) {\n _featStruc = new GFeatStruc(fsEl);\n }\n // or create empty one\n else {\n _featStruc = new GFeatStruc();\n }\n }\n\n /**Returns an XML element representing the category. */\n public Element toXml() {\n \tElement retval = new Element(\"atomcat\");\n \tretval.setAttribute(\"type\", type);\n \tif (!_featStruc.isEmpty() || _featStruc.getIndex() > 0) {\n \t\tif (_featStruc instanceof GFeatStruc) { // only supporting GFeatStruc per xml construction\n \t\t\tretval.addContent(((GFeatStruc) _featStruc).toXml());\n \t\t}\n \t}\n \t// call super to add LF if present\n \tsuper.toXml(retval);\n \treturn retval;\n }\n\n /**\n * Returns this category as the target category.\n */\n public TargetCat getTarget() { return this; }\n \n \n public String getType() {\n return type;\n }\n\n \n public Category copy() {\n \tAtomCat retval = new AtomCat(type, _featStruc.copy(), (_lf == null) ? null : (LF) _lf.copy());\n \tretval.fragCompletion = fragCompletion;\n return retval;\n }\n\n public Category shallowCopy() {\n AtomCat retval = new AtomCat(type, _featStruc, _lf);\n \tretval.fragCompletion = fragCompletion;\n return retval;\n }\n\n public void deepMap(ModFcn mf) { \n super.deepMap(mf);\n _featStruc.deepMap(mf);\n }\n\n public void unifyCheck (Object u) throws UnifyFailure {\n if (u instanceof AtomCat) {\n AtomCat u_ac = (AtomCat)u;\n if (!(type.equals(u_ac.type))) {\n throw new UnifyFailure();\n }\n if (_featStruc != null && u_ac._featStruc != null) {\n _featStruc.unifyCheck(u_ac._featStruc);\n }\n } else if (!(u instanceof Variable)) {\n throw new UnifyFailure();\n }\n }\n\n /** NB: The LF does not participate in unification. */\n public Object unify (Object u, Substitution sub) \n throws UnifyFailure {\n\n if (u instanceof AtomCat && type.equals(((AtomCat)u).type)) {\n AtomCat u_ac = (AtomCat)u;\n FeatureStructure $fs;\n if (_featStruc == null) {\n $fs = u_ac._featStruc;\n } else if (u_ac._featStruc == null) {\n $fs = _featStruc;\n } else {\n $fs = (FeatureStructure)_featStruc.unify(u_ac._featStruc, sub);\n }\n return new AtomCat(type, $fs);\n }\n else {\n throw new UnifyFailure();\n }\n }\n\n public Object fill (Substitution s) throws UnifyFailure {\n AtomCat $ac =\n new AtomCat(type,\n (FeatureStructure)_featStruc.fill(s),\n (_lf == null) ? null : (LF) _lf.fill(s));\n return $ac;\n }\n\n public boolean shallowEquals(Object c) {\n if (c instanceof AtomCat) {\n AtomCat ac = (AtomCat)c;\n return type.equals(ac.type);\n }\n return false;\n }\n \n public String toString() {\n DisplayPrefs prefs = Grammar.theGrammar.prefs;\n\n StringBuffer sb = new StringBuffer();\n sb.append(type);\n if (fragCompletion) sb.append(\"_c\");\n \n if(_featStruc != null && prefs.showFeats)\n sb.append(_featStruc.toString());\n\n if (_lf != null && prefs.showSem) {\n sb.append(\" : \").append(_lf.toString());\n }\n\n if (sb.length() == 0) return \"UnknownCat\";\n return sb.toString();\n }\n\n /**\n * Returns the interned supertag for the category.\n */\n public String getSupertag() {\n\t\tif (_supertag != null) return _supertag;\n StringBuffer sb = new StringBuffer();\n sb.append(type);\n if(_featStruc != null) sb.append(_featStruc.getSupertagInfo());\n if (sb.length() == 0) _supertag = \"UnknownCat\";\n else _supertag = sb.toString().intern();\n\t\treturn _supertag;\n }\n \n public String toTeX() {\n DisplayPrefs prefs = Grammar.theGrammar.prefs;\n StringBuffer sb = new StringBuffer();\n sb.append(type);\n if(_featStruc != null && prefs.showFeats)\n sb.append(_featStruc.toTeX());\n if (sb.length() == 0) return \"UnknownCat\";\n return sb.toString();\n }\n\n \n /**\n * Returns a hash code for this category ignoring the LF, \n * using the given map from vars to ints.\n */\n public int hashCodeNoLF(TObjectIntHashMap varMap) {\n int retval = type.hashCode();\n if (_featStruc != null) { \n if (_featStruc instanceof GFeatStruc) {\n retval += ((GFeatStruc)_featStruc).hashCode(varMap);\n } else { // nb: would be nice to get rid of this case\n retval += _featStruc.hashCode(); \n }\n }\n return retval;\n }\n\n /**\n * Returns whether this category equals the given object \n * up to variable names, using the given maps from vars to ints, \n * ignoring the LFs (if any).\n */\n public boolean equalsNoLF(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2) {\n if (obj.getClass() != this.getClass()) { return false; }\n AtomCat ac = (AtomCat) obj;\n if (_featStruc != null && ac._featStruc == null) { return false; }\n if (_featStruc == null && ac._featStruc != null) { return false; }\n if (!type.equals(ac.type)) { return false; }\n if (_featStruc != null) { \n if (_featStruc instanceof GFeatStruc) {\n if (!((GFeatStruc)_featStruc).equals(ac._featStruc, varMap, varMap2)) { return false; }\n } else { // nb: would be nice to get rid of this case\n if (!_featStruc.equals(ac._featStruc)) { return false; }\n }\n }\n return true;\n }\n\n /**\n * Returns whether this category is a fragment category;\n * returns true iff the type of this cat is \"frag\".\n */\n public boolean isFragment() { return type.equals(\"frag\"); }\n}\n" }, { "path": "src/opennlp/ccg/synsem/BasicArg.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-5 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\nimport java.io.Serializable;\nimport gnu.trove.*;\n\nimport opennlp.ccg.unify.*;\n\n/**\n * A basic argument that contains a slash and a category.\n * \n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.9 $, $Date: 2009/11/28 03:39:27 $\n */\npublic final class BasicArg implements Arg, Serializable {\n\n\tprivate static final long serialVersionUID = -4244825501682166456L;\n\n\tprivate final Slash _slash;\n\n\tprivate final Category _cat;\n\n\tpublic BasicArg(Slash s, Category c) {\n\t\t_slash = s;\n\t\t_cat = c;\n\t}\n\n\tpublic Arg copy() {\n\t\treturn new BasicArg(_slash.copy(), _cat.copy());\n\t}\n\n\tpublic Slash getSlash() {\n\t\treturn _slash;\n\t}\n\n\tpublic void setSlashModifier(boolean modifier) { _slash.setModifier(modifier); }\n\t\n public void setSlashHarmonicCompositionResult(boolean harmonicResult) { _slash.setHarmonicCompositionResult(harmonicResult); }\n\n public Category getCat() {\n\t\treturn _cat;\n\t}\n\n\tpublic boolean occurs(Variable v) {\n\t\treturn _cat.occurs(v);\n\t}\n\n\tpublic Object fill(Substitution sub) throws UnifyFailure {\n\t\treturn new BasicArg((Slash) _slash.fill(sub), (Category) _cat.fill(sub));\n\t}\n\n\tpublic void forall(CategoryFcn fcn) {\n\t\t_cat.forall(fcn);\n\t}\n\n\tpublic void unifySlash(Slash s) throws UnifyFailure {\n\t\t_slash.unifyCheck(s);\n\t}\n\n\tpublic void unifyCheck(Object u) throws UnifyFailure {\n\t}\n\n\tpublic Object unify(Object u, Substitution sub) throws UnifyFailure {\n\t\tif (u instanceof BasicArg) {\n\t\t\treturn new BasicArg((Slash) _slash\n\t\t\t\t\t.unify(((BasicArg) u)._slash, sub), (Category) _cat.unify(\n\t\t\t\t\t((BasicArg) u)._cat, sub));\n\t\t} else {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\n\t}\n\n\tpublic void deepMap(ModFcn mf) {\n\t\t_slash.deepMap(mf);\n\t\t_cat.deepMap(mf);\n\t}\n\n\tpublic String toString() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tsb.append(_slash.toString());\n\t\tif (_cat instanceof ComplexCat) {\n\t\t\tsb.append('(').append(_cat).append(')');\n\t\t} else {\n\t\t\tsb.append(_cat);\n\t\t}\n\t\treturn sb.toString();\n\t}\n\n\t/**\n\t * Returns the supertag for this arg.\n\t */\n\tpublic String getSupertag() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tsb.append(_slash.getSupertag());\n\t\tif (_cat instanceof ComplexCat) {\n\t\t\tsb.append('(').append(_cat.getSupertag()).append(')');\n\t\t} else {\n\t\t\tsb.append(_cat.getSupertag());\n\t\t}\n\t\treturn sb.toString();\n\t}\n\n\t/**\n\t * Returns a TeX-formatted string representation for this arg.\n\t */\n\tpublic String toTeX() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tsb.append(_slash.toTeX());\n\t\tif (_cat instanceof ComplexCat) {\n\t\t\tsb.append('(').append(_cat.toTeX()).append(')');\n\t\t} else {\n\t\t\tsb.append(_cat.toTeX());\n\t\t}\n\t\treturn sb.toString();\n\t}\n\n\t/**\n\t * Returns a hash code for this, using the given map from vars to ints.\n\t */\n\tpublic int hashCode(TObjectIntHashMap varMap) {\n\t\treturn _slash.hashCode(varMap) + _cat.hashCodeNoLF(varMap);\n\t}\n\n\t/**\n\t * Returns whether this arg equals the given object up to variable names,\n\t * using the given maps from vars to ints.\n\t */\n\tpublic boolean equals(Object obj, TObjectIntHashMap varMap,\n\t\t\tTObjectIntHashMap varMap2) {\n\t\tif (obj.getClass() != this.getClass()) {\n\t\t\treturn false;\n\t\t}\n\t\tBasicArg ba = (BasicArg) obj;\n\t\treturn _slash.equals(ba._slash, varMap, varMap2)\n\t\t\t\t&& _cat.equalsNoLF(ba._cat, varMap, varMap2);\n\t}\n}\n" }, { "path": "src/opennlp/ccg/synsem/CatReader.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-4 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.synsem;\n\nimport org.jdom.*;\n\n/**\n * Utility class to build categories.\n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.4 $, $Date: 2005/10/18 22:20:15 $\n */\npublic class CatReader {\n \n public static Category getCat(Element catel) {\n Category cat = null;\n String catType = catel.getName();\n \n if (catType.equals(\"atomcat\") || catType.equals(\"ac\")) { \n cat = new AtomCat(catel);\n }\n \n else if (catType.equals(\"complexcat\") || catType.equals(\"cc\")) { \n cat = new ComplexCat(catel);\n }\n\n return cat;\n }\n}\n" }, { "path": "src/opennlp/ccg/synsem/Category.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-5 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.hylo.*;\nimport gnu.trove.*;\nimport org.jdom.Element;\n\n/**\n * A CCG category.\n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.11 $, $Date: 2011/05/22 03:40:55 $\n */\npublic interface Category extends Unifiable, Mutable, java.io.Serializable {\n\n /**\n * Accessor function for the feature structure associated with this category.\n *\n * @return the feature structure for this cateogory\n */ \n public FeatureStructure getFeatureStructure();\n \n /**\n * Gives this category a new feature structure.\n *\n * @param fs the new feature structure\n */ \n public void setFeatureStructure(FeatureStructure fs);\n\n /** Gets the LF. */\n public LF getLF();\n \n /** Sets the LF. */\n public void setLF(LF lf);\n \n\n /**\n * Determines if this category is equal to another on the top level.\n * It does not check sub categories.\n *\n * @param o object to check for equality\n * @return whether or not this is shallowly equal to object\n */\n public boolean shallowEquals(Object o);\n\n /**\n * Deep copies this category.\n *\n * @return a deep copy of this category\n */\n public Category copy();\n\n /** Shallow copies this category. */\n public Category shallowCopy();\n \n\n /**\n * Iterates through this Category applying a function to this category\n * and every subcategory.\n *\n * @param f a function to be applied\n */ \n public void forall(CategoryFcn f); //to ls\n\n \n /** \n * Returns a hash code for this category. \n * The hash code handles equivalence up to variable names \n * as long as features and predicates are in the same order.\n */\n public int hashCode();\n \n /** \n * Returns a hash code for this category ignoring the LF.\n */\n public int hashCodeNoLF();\n \n /**\n * Returns a hash code for this category ignoring the LF, \n * using the given map from vars to ints, \n * to allow for equivalence up to variable names.\n */\n public int hashCodeNoLF(TObjectIntHashMap varMap);\n\n \n /** \n * Returns whether this category equals the given object. \n * Equivalence up to variable names is handled \n * as long as features and predicates are in the same order.\n */\n public boolean equals(Object obj);\n \n /** \n * Returns whether this category equals the given object, \n * ignoring the LFs (if any).\n */\n public boolean equalsNoLF(Object obj);\n \n /**\n * Returns whether this category equals the given object \n * up to variable names, using the given maps from vars to ints, \n * ignoring the LFs (if any).\n */\n public boolean equalsNoLF(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2);\n \n \n /**\n * Returns the target category of this category.\n */\n public TargetCat getTarget();\n \n /**\n * Returns the nominal which is the value of the index feature on the \n * target cat, or null if none.\n */\n public Nominal getIndexNominal(); \n \n /**\n * Returns the interned supertag for the category.\n */\n public String getSupertag();\n \n /**\n * Returns whether this category is a fragment category.\n */\n public boolean isFragment();\n \n /**\n * Returns a TeX-formatted string representation for the category.\n */\n public String toTeX();\n \n /**\n * Returns an XML element representing the category.\n */\n public Element toXml();\n}\n" }, { "path": "src/opennlp/ccg/synsem/CategoryFcn.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge and Gann Bierner\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\n/**\n * A set of functions that can be applied to Categories. This is a way of\n * getting around the fact that Java doesn't have 1st class functions.\n * Thus, if a method is needed that takes a function that works on Categories,\n * pass one of these instead.\n *\n * @author Gann Bierner\n * @version $Revision: 1.1.1.1 $, $Date: 2003/02/28 18:02:12 $\n */\npublic interface CategoryFcn {\n /**\n * Converts a category to a different category\n *\n * @param c the category to change\n * @return the transformed category\n */\n public Category fcn(Category c);\n \n /**\n * Converts a category to a different category with some additional\n * information about its context.\n *\n * @param a The logical form in which the category appears\n * @param c The category to convert\n * @param i The position of the category in the logical form\n * @return the transformed category\n */\n public void fcn(Category a, Category c, int i);\n \n /**\n * Performs some destructive operation given a category\n *\n * @param c The category used for whatever purpose\n */\n public void forall(Category c);\n}\n" }, { "path": "src/opennlp/ccg/synsem/CategoryFcnAdapter.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge and Gann Bierner\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\n/**\n * An adapter for Category Functions so that you only have to implement the\n * methods you want.\n *\n * @author Gann Bierner\n * @version $Revision: 1.1.1.1 $, $Date: 2003/02/28 18:02:12 $\n */\npublic class CategoryFcnAdapter implements CategoryFcn{\n public Category fcn(Category c) {return c;}\n public void fcn(Category a, Category c, int i) {}\n public void forall(Category c) {};\n}\n" }, { "path": "src/opennlp/ccg/synsem/ComplexCat.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-5 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\nimport opennlp.ccg.unify.*;\nimport opennlp.ccg.util.DisplayPrefs;\nimport opennlp.ccg.grammar.Grammar;\nimport gnu.trove.*;\nimport org.jdom.*;\n\nimport java.util.*;\n\n/**\n * A non-recursive representation of complex categories.\n * \n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.15 $, $Date: 2009/06/18 23:38:57 $\n */\npublic final class ComplexCat extends AbstractCat {\n\n\tprivate static final long serialVersionUID = 1L;\n\n\tprivate TargetCat _target;\n\n\tprivate ArgStack _args;\n\n\t/** Constructor with target and single arg. */\n\tpublic ComplexCat(TargetCat target, Arg arg) {\n\t\tthis(target, new ArgStack(arg));\n\t}\n\n\t/** Constructor with target and arg stack. */\n\tpublic ComplexCat(TargetCat target, ArgStack args) {\n\t\tthis(target, args, null);\n\t}\n\n\t/** Constructor with target, arg stack and LF. */\n\tpublic ComplexCat(TargetCat target, ArgStack args, LF lf) {\n\t\tsuper(lf);\n\t\t_target = target;\n\t\tif (args.size() < 1) {\n\t\t\tSystem.out.println(\"WARNING!!! Creating a ComplexCat with\"\n\t\t\t\t\t+ \" empty argument stack!\");\n\t\t}\n\t\t_args = args;\n\t}\n\n\t/** Constructor which retrieves the complex category from the XML element. */\n\t// also determines modifier slashes\n\t@SuppressWarnings(\"unchecked\")\n\tpublic ComplexCat(Element el) {\n\t\t// call super to get LF if present\n\t\tsuper(el);\n\t\t// get children minus LF elt\n\t\tList info = el.getChildren();\n\t\tElement lfElt = el.getChild(\"lf\");\n\t\tif (lfElt != null) {\n\t\t\tinfo.remove(lfElt);\n\t\t}\n\t\t// get target and args from first and rest of remaining children\n\t\t_target = (TargetCat) CatReader.getCat(info.get(0));\n\t\t_args = new ArgStack(info.subList(1, info.size()));\n\t\t// set modifier slashes\n\t\tsetModifierSlashes();\n\t}\n\t\n /**Returns an XML element representing the category. */\n public Element toXml() {\n \tElement retval = new Element(\"complexcat\");\n \tretval.addContent(_target.toXml());\n \t_args.toXml(retval);\n \t// call super to add LF if present\n \tsuper.toXml(retval);\n \treturn retval;\n }\n\t\n\t// sets modifier slashes based on feat struc ids\n\tprivate void setModifierSlashes() {\n\t\tFeatureStructure targetFS = _target.getFeatureStructure(); \n\t\tint targetIndex = targetFS.getIndex();\n\t\tif (targetIndex == 0) targetIndex = targetFS.getInheritsFrom();\n\t\tif (targetIndex == 0) return;\n\t\tfor (int i=0; i < _args.size(); i++) {\n\t\t\tArg arg = _args.get(i);\n\t\t\tif (arg instanceof BasicArg) {\n\t\t\t\tsetModifierSlash(targetIndex, (BasicArg)arg);\n\t\t\t}\n\t\t\telse if (arg instanceof SetArg) {\n\t\t\t\tSetArg sArg = (SetArg) arg;\n\t\t\t\tfor (int j=0; j < sArg.size(); j++) {\n\t\t\t\t\tsetModifierSlash(targetIndex, sArg.get(j));\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\t\n\t// sets modifier slash based on the target index\n\tprivate void setModifierSlash(int targetIndex, BasicArg arg) {\n\t\tFeatureStructure argFS = arg.getCat().getTarget().getFeatureStructure();\n\t\t// check for matching ids\n\t\tif (targetIndex == argFS.getIndex() || targetIndex == argFS.getInheritsFrom())\n\t\t\targ.getSlash().setModifier(true);\n\t}\n\n\t\n\t/**\n\t * Returns the target category of this category.\n\t */\n\tpublic TargetCat getTarget() {\n\t\treturn _target;\n\t}\n\n\tpublic Arg getArg(int pos) {\n\t\treturn _args.get(pos);\n\t}\n\n\tpublic Arg getOuterArg() {\n\t\treturn _args.getLast();\n\t}\n\n\tpublic Category getResult() {\n\t\treturn getSubResult(arity() - 1);\n\t}\n\n\tpublic Category getSubResult(int upto) {\n\t\tif (upto == 0) {\n\t\t\treturn _target;\n\t\t} else {\n\t\t\treturn new ComplexCat(_target, _args.subList(0, upto));\n\t\t}\n\t}\n\n\tpublic ArgStack getArgStack() {\n\t\treturn _args;\n\t}\n\n\tpublic ArgStack getArgStack(int from) {\n\t\treturn _args.subList(from);\n\t}\n\n\tpublic boolean containsDollarArg() {\n\t\treturn _args.containsDollarArg();\n\t}\n\n\tpublic boolean containsSetArg() {\n\t\treturn _args.containsSetArg();\n\t}\n\n\tpublic void add(Arg a) {\n\t\t_args.add(a);\n\t}\n\n\tpublic void add(ArgStack as) {\n\t\t_args.add(as);\n\t}\n\n\tpublic void addBeforeEnd(ArgStack as) {\n\t\tint size = _args.size();\n\t\tif (size < 1) {\n\t\t\tadd(as);\n\t\t} else {\n\t\t\t_args.addAt(as, size - 1);\n\t\t}\n\t}\n\n\tpublic void addFront(ArgStack as) {\n\t\t_args.addFront(as);\n\t}\n\n\tpublic void insertFront(ArgStack as) {\n\t\t_args.insertFront(as);\n\t}\n\n\tpublic void insertEnd(ArgStack as) {\n\t\t_args.insertEnd(as);\n\t}\n\n\tpublic void set(int index, Arg c) {\n\t\t_args.set(index, c);\n\t}\n\n\tpublic void setOuterArgument(Arg c) {\n\t\t_args.setLast(c);\n\t}\n\n\tpublic int arity() {\n\t\treturn _args.size();\n\t}\n\n\tpublic Category copy() {\n\t\treturn new ComplexCat((TargetCat) _target.copy(), _args.copy(),\n\t\t\t\t(_lf == null) ? null : (LF) _lf.copy());\n\t}\n\n\tpublic Category shallowCopy() {\n\t\treturn new ComplexCat(_target, _args, _lf);\n\t}\n\n\tpublic void deepMap(ModFcn mf) {\n\t\tsuper.deepMap(mf);\n\t\t_target.deepMap(mf);\n\t\t_args.deepMap(mf);\n\t}\n\n\tpublic void forall(CategoryFcn f) {\n\t\tf.forall(this);\n\t\t_target.forall(f);\n\t\t_args.forall(f);\n\t}\n\n\tpublic void unifyCheck(Object u) throws UnifyFailure {\n\t\tif (u instanceof ComplexCat) {\n\t\t\tComplexCat cc = (ComplexCat) u;\n\t\t\t_target.unifyCheck(cc._target);\n\t\t}\n\t}\n\n\t/** NB: The LF does not participate in unification. */\n\tpublic Object unify(Object u, Substitution sub) throws UnifyFailure {\n\n\t\tif (u instanceof AtomCat && arity() == 1 & containsDollarArg()) {\n\t\t\tsub.makeSubstitution((Dollar) _args.get(0), new ArgStack());\n\t\t\treturn GUnifier.unify(_target, (AtomCat) u, sub);\n\t\t} else if (u instanceof ComplexCat) {\n\t\t\tComplexCat cc = (ComplexCat) u;\n\t\t\tArgStack $args = _args.unify(cc._args, sub);\n\t\t\tCategory $target = GUnifier.unify(_target, cc._target, sub);\n\t\t\tif ($args.size() == 0) {\n\t\t\t\treturn $target;\n\t\t\t} else {\n\t\t\t\treturn new ComplexCat((TargetCat) $target, $args);\n\t\t\t}\n\t\t} else {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\t}\n\n\tpublic boolean occurs(Variable v) {\n\t\treturn super.occurs(v) || _target.occurs(v) || _args.occurs(v);\n\t}\n\n\t// nb: not yet sure about calling setLF methods\n\tpublic Object fill(Substitution s) throws UnifyFailure {\n\t\tCategory $target = (Category) _target.fill(s);\n\t\tArgStack $args = _args.fill(s);\n\t\tLF $lf = (_lf == null) ? null : (LF) _lf.fill(s);\n\t\tif ($args.size() == 0) {\n\t\t\t$target.setLF($lf);\n\t\t\treturn $target;\n\t\t}\n\t\tif ($target instanceof TargetCat) {\n\t\t\treturn new ComplexCat((TargetCat) $target, $args, $lf);\n\t\t} else if ($target instanceof ComplexCat) {\n\t\t\t((ComplexCat) $target).add($args);\n\t\t\t$target.setLF($lf);\n\t\t\treturn $target;\n\t\t} else {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\t}\n\n\tpublic String toString() {\n\t DisplayPrefs prefs = Grammar.theGrammar.prefs;\n\t StringBuffer sb = new StringBuffer();\n\t sb.append(_target.toString()).append(_args.toString());\n\t if (_lf != null && prefs.showSem) {\n\t \tsb.append(\" : \").append(_lf.toString());\n\t }\n\t return sb.toString();\n\t}\n\n\t/**\n\t * Returns the interned supertag for the category.\n\t */\n\tpublic String getSupertag() {\n\t\tif (_supertag != null) return _supertag;\n\t\tStringBuffer sb = new StringBuffer();\n\t\tsb.append(_target.getSupertag()).append(_args.getSupertag());\n\t\t_supertag = sb.toString().intern();\n\t\treturn _supertag;\n\t}\n\n\tpublic String toTeX() {\n\t\t// Preferences prefs = Preferences.userNodeForPackage(TextCCG.class);\n\t\t// boolean showSem = prefs.getBoolean(SHOW_SEMANTICS, false);\n\t\tStringBuffer sb = new StringBuffer();\n\t\tsb.append(_target.toTeX()).append(_args.toTeX());\n\t\treturn sb.toString();\n\t}\n\n\t/**\n\t * Returns a hash code for this category ignoring the LF, using the given\n\t * map from vars to ints.\n\t */\n\tpublic int hashCodeNoLF(TObjectIntHashMap varMap) {\n\t\tint retval = _target.hashCodeNoLF(varMap);\n\t\tretval += _args.hashCode(varMap);\n\t\treturn retval;\n\t}\n\n\t/**\n\t * Returns whether this category equals the given object up to variable\n\t * names, using the given maps from vars to ints, ignoring the LFs (if any).\n\t */\n\tpublic boolean equalsNoLF(Object obj, TObjectIntHashMap varMap,\n\t\t\tTObjectIntHashMap varMap2) {\n\t\tif (obj.getClass() != this.getClass()) {\n\t\t\treturn false;\n\t\t}\n\t\tComplexCat cc = (ComplexCat) obj;\n\t\tif (!_target.equalsNoLF(cc._target, varMap, varMap2)) {\n\t\t\treturn false;\n\t\t}\n\t\tif (!_args.equals(cc._args, varMap, varMap2)) {\n\t\t\treturn false;\n\t\t}\n\t\treturn true;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/synsem/DerivationHandler.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2010 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.synsem;\r\n\r\nimport opennlp.ccg.parse.DerivationHistory;\r\n\r\n/** \r\n * A class to simplify implementing a recursive procedure on a derivation.\r\n * Recursion is handled within the implementation of the abstract methods \r\n * by calls to handleDerivation, so that order of traversal \r\n * can be flexibly specified. Results may be optionally cached.\r\n * Note that the top step is only used with complete derivations.\r\n * \r\n * @author \tMichael White\r\n * @version\t$Revision: 1.1 $, $Date: 2010/02/21 16:44:59 $\r\n */ \r\nabstract public class DerivationHandler {\r\n\t\r\n\t/** Top step. */\r\n\tabstract public T topStep(Sign sign);\r\n\t\r\n\t/** Lexical step. */\r\n\tabstract public T lexStep(Sign sign);\r\n\r\n\t/** Unary step. */\r\n\tabstract public T unaryStep(Sign sign, Sign headChild);\r\n\t\r\n\t/** Binary step. */\r\n\tabstract public T binaryStep(Sign sign, boolean left, Sign headChild, Sign siblingChild);\r\n\t\r\n\t/** Checks for cached value, returning null if none. Defaults to null. */\r\n\tpublic T checkCache(Sign sign) { return null; }\r\n\r\n\t/** Caches the result. Default no-op. */\r\n\tpublic void cache(Sign sign, T result) {}\r\n\t\r\n\t/** Handles a complete derivation, invoking the top step. */\r\n\tpublic T handleCompleteDerivation(Sign sign) {\r\n\t\treturn topStep(sign);\r\n\t}\r\n\t\r\n\t/** Handles a sub-derivation, checking and updating cache. */\r\n\tpublic T handleDerivation(Sign sign) {\r\n\t\t// check cache\r\n\t\tT retval = checkCache(sign); \r\n\t\tif (retval != null) return retval;\r\n\t\t// lexical case\r\n\t\tif (sign.isLexical()) {\r\n\t\t\tretval = lexStep(sign);\r\n\t\t\tcache(sign, retval); \r\n\t\t\treturn retval; \r\n\t\t}\r\n\t\t// recursive case\r\n\t\tDerivationHistory dh = sign.getDerivationHistory();\r\n\t\tSign[] inputs = dh.getInputs();\r\n\t\t// unary case\r\n\t\tif (inputs.length == 1) {\r\n\t\t\tSign headChild = inputs[0];\r\n\t\t\tretval = unaryStep(sign, headChild);\r\n\t\t\tcache(sign, retval); \r\n\t\t\treturn retval; \r\n\t\t}\r\n\t\t// binary case\r\n\t\telse {\r\n\t\t\tboolean left;\r\n\t\t\tSign headChild, siblingChild;\r\n\t\t\tif (sign.getLexHead() == inputs[0].getLexHead()) {\r\n\t\t\t\tleft = true; headChild = inputs[0]; siblingChild = inputs[1];\r\n\t\t\t}\r\n\t\t\telse {\r\n\t\t\t\tleft = false; headChild = inputs[1]; siblingChild = inputs[0];\r\n\t\t\t}\r\n\t\t\tretval = binaryStep(sign, left, headChild, siblingChild);\r\n\t\t\tcache(sign, retval); \r\n\t\t\treturn retval; \r\n\t\t}\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccg/synsem/Dollar.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-7 Jason Baldridge and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\nimport java.io.Serializable;\nimport gnu.trove.*;\n\nimport opennlp.ccg.unify.*;\n\n/**\n * A variable representing a stack of arguments\n * \n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.10 $, $Date: 2009/11/28 03:39:27 $\n */\npublic final class Dollar implements Arg, Variable, Mutable, Indexed, Serializable {\n\n\tprivate static final long serialVersionUID = -5936227018184772678L;\n\n\tprivate final Slash _slash;\n\n\tprivate final String _name;\n\n\tprivate int _index = 0;\n\n\tprivate boolean _hasMostGeneralSlash = false;\n\n\tpublic Dollar(String name) {\n\t\tthis(new Slash(), name);\n\t}\n\n\tpublic Dollar(Slash s, String name) {\n\t\tthis(s, name, 0);\n\t}\n\n\tpublic Dollar(Slash s, String name, int id) {\n\t\t_slash = s;\n\t\t_name = name;\n\t\t_index = id;\n\t\tif (s.toString().equals(\"|.\")) {\n\t\t\t_hasMostGeneralSlash = true;\n\t\t}\n\t}\n\n\tpublic String name() {\n\t\treturn _name;\n\t}\n\n\tpublic int getIndex() {\n\t\treturn _index;\n\t}\n\n\tpublic void setIndex(int uniqueIndex) {\n\t\t_index = uniqueIndex;\n\t}\n\n\tpublic Arg copy() {\n\t\treturn new Dollar(_slash.copy(), _name, _index);\n\t}\n\n\tpublic void forall(CategoryFcn fcn) {\n\t}\n\n\tpublic Slash getSlash() {\n\t\treturn _slash;\n\t}\n\n\tpublic void setSlashModifier(boolean modifier) { _slash.setModifier(modifier); }\n\t\n public void setSlashHarmonicCompositionResult(boolean harmonicResult) { _slash.setHarmonicCompositionResult(harmonicResult); }\n\n public boolean equals(Object o) {\n\t\treturn (o instanceof Dollar && _index == ((Dollar) o).getIndex() && _slash\n\t\t\t\t.equals(((Dollar) o).getSlash()));\n\t}\n\n\tpublic int hashCode() {\n\t\treturn 31 * _index + _slash.hashCode() ;\n\t}\n\n\tpublic boolean occurs(Variable v) {\n\t\treturn (v instanceof Dollar && equals(v));\n\t}\n\n\tpublic Object fill(Substitution sub) throws UnifyFailure {\n\t\tObject value = sub.getValue(this);\n\t\tif (value == null) {\n\t\t\treturn this;\n\t\t}\n\t\tif (value instanceof Dollar) {\n\t\t\treturn value;\n\t\t}\n\t\t// nb: must do occurs check here, at least in part b/c ArgStack doesn't\n\t\t// quite implement Unifiable\n\t\tif (value instanceof Arg && !((Arg) value).occurs(this)) {\n\t\t\treturn ((Arg) value).fill(sub);\n\t\t} else if (value instanceof ArgStack\n\t\t\t\t&& !((ArgStack) value).occurs(this)) {\n\t\t\treturn ((ArgStack) value).fill(sub);\n\t\t} else {\n\t\t\t// System.out.println(\"Error in value for dollar: \" + this +\" = \" +\n\t\t\t// value);\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\t}\n\n\tpublic void unifySlash(Slash s) throws UnifyFailure {\n\t\t_slash.unifyCheck(s);\n\t}\n\n\tpublic void unifyCheck(Object u) throws UnifyFailure {\n\t}\n\n\tpublic Object unify(Object u, Substitution sub) throws UnifyFailure {\n\t\tif (u instanceof ArgStack && !((ArgStack) u).occurs(this)) {\n\t\t\t((ArgStack) u).slashesUnify(_slash);\n\t\t} else if (u instanceof Arg && !((Arg) u).occurs(this)) {\n\t\t\t((Arg) u).unifySlash(_slash);\n\t\t} else {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\t\tsub.makeSubstitution(this, u);\n\t\treturn u;\n\t}\n\n\tpublic void deepMap(ModFcn mf) {\n\t\tmf.modify(this);\n\t}\n\n\tpublic String toString() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tif (!_hasMostGeneralSlash)\n\t\t\tsb.append(_slash.toString());\n\t\tsb.append('$').append(_name);// .append(_index);\n\t\treturn sb.toString();\n\t}\n\n\t/**\n\t * Returns the supertag for this dollar arg.\n\t */\n\tpublic String getSupertag() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tif (!_hasMostGeneralSlash)\n\t\t\tsb.append(_slash.getSupertag());\n\t\tsb.append('$');\n\t\treturn sb.toString();\n\t}\n\n\t/**\n\t * Returns a TeX-formatted string representation for this dollar arg.\n\t */\n\tpublic String toTeX() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tif (!_hasMostGeneralSlash)\n\t\t\tsb.append(_slash.toTeX());\n\t\tsb.append(\"\\\\$ \\\\subs{\").append(_name).append(\"}\");// .append(_index);\n\t\treturn sb.toString();\n\t}\n\n\t/**\n\t * Returns a hash code using the given map from vars to ints.\n\t */\n\tpublic int hashCode(TObjectIntHashMap varMap) {\n\t\tint retval = _slash.hashCode(varMap);\n\t\t// see if this already in map\n\t\tif (varMap.containsKey(this)) {\n\t\t\tretval += varMap.get(this);\n\t\t}\n\t\t// otherwise add it\n\t\telse {\n\t\t\tint next = varMap.size() + 1;\n\t\t\tvarMap.put(this, next);\n\t\t\tretval += next;\n\t\t}\n\t\treturn retval;\n\t}\n\n\t/**\n\t * Returns whether this dollar equals the given object up to variable names,\n\t * using the given maps from vars to ints.\n\t */\n\tpublic boolean equals(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2) {\n\t\tif (this == obj) return true;\n\t\tif (obj.getClass() != this.getClass()) return false;\n\t\tDollar d = (Dollar) obj;\n\t\tif (varMap.get(this) != varMap2.get(d)) return false;\n\t\tif (!_slash.equals(d._slash, varMap, varMap2)) return false;\n\t\treturn true;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/synsem/GenerativeSyntacticModel.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2010 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.synsem;\r\n\r\nimport java.io.*;\r\nimport java.net.URL;\r\nimport java.util.*;\r\n\r\nimport opennlp.ccg.grammar.Grammar;\r\nimport opennlp.ccg.lexicon.*;\r\nimport opennlp.ccg.ngrams.ConditionalProbabilityTable;\r\nimport opennlp.ccg.ngrams.NgramScorer;\r\nimport opennlp.ccg.perceptron.*;\r\nimport opennlp.ccg.test.Regression;\r\nimport opennlp.ccg.test.RegressionInfo;\r\nimport opennlp.ccg.test.RegressionInfo.TestItem;\r\nimport opennlp.ccg.util.Pair;\r\n\r\n/** \r\n * A class implementing a variant of Hockenmaier's HWDep generative syntactic model, \r\n * with additional postag variables.\r\n * Note that the top step is only used with complete derivations.\r\n * Also note that for simplicity there is no special treatment of rare words, and thus \r\n * a single unknown word is used in the model, rather than one for each POS tag.\r\n * \r\n * @author \tMichael White\r\n * @version\t$Revision: 1.12 $, $Date: 2010/03/07 03:23:01 $\r\n */ \r\npublic class GenerativeSyntacticModel implements FeatureExtractor, SignScorer {\r\n\r\n\t/** Feature key. */\r\n\tpublic static String genlogprobkey = \"genlogprob\";\r\n\t\r\n\t/** Expansion string constant. */\r\n\tpublic static final String EXPANSION = \"E\";\r\n\t\r\n\t/** Left expansion string constant. */\r\n\tpublic static final String LEFT = \"left\";\r\n\t\r\n\t/** Right expansion string constant. */\r\n\tpublic static final String RIGHT = \"right\";\r\n\t\r\n\t/** Unary expansion string constant. */\r\n\tpublic static final String UNARY = \"unary\";\r\n\t\r\n\t/** Leaf expansion string constant. */\r\n\tpublic static final String LEAF = \"leaf\";\r\n\r\n\t/** Category of parent string constant. */\r\n\tpublic static final String PARENT = \"P\";\r\n\r\n\t/** Category of head string constant. */\r\n\tpublic static final String HEAD = \"H\";\r\n\r\n\t/** Category of sibling string constant. */\r\n\tpublic static final String SIBLING = \"S\";\r\n\r\n\t/** Lexical head category of parent string constant. */\r\n\tpublic static final String LEXCAT_PARENT = \"CP\";\r\n\r\n\t/** Head postag of parent string constant. */\r\n\tpublic static final String POS_PARENT = \"T\";\r\n\r\n\t/** Head word of parent string constant. */\r\n\tpublic static final String WORD_PARENT = \"W\";\r\n\r\n\t/** Lexical head category of sibling string constant. */\r\n\tpublic static final String LEXCAT_SIBLING = \"CS\";\r\n\r\n\t/** Head postag of sibling string constant. */\r\n\tpublic static final String POS_SIBLING = \"TS\";\r\n\r\n\t/** Head word of sibling string constant. */\r\n\tpublic static final String WORD_SIBLING = \"WS\";\r\n\r\n\t/** Lexical head category of top string constant. */\r\n\tpublic static final String LEXCAT_TOP = \"CT\";\r\n\r\n\t/** Head postag top string constant. */\r\n\tpublic static final String POS_TOP = \"TT\";\r\n\r\n\t/** Head word of top string constant. */\r\n\tpublic static final String WORD_TOP = \"WT\";\r\n\r\n\t/** Derivation top string constant. */\r\n\tpublic static final String TOP = \"\";\r\n\t\r\n\t/**\r\n\t * Class for caching the model's log prob in a sign.\r\n\t */\r\n\tpublic static class GenLogProb {\r\n\t\t/** The log prob. */\r\n\t\tpublic final double logprob;\r\n\t\t/** Constructor. */\r\n\t\tpublic GenLogProb(double logprob) { this.logprob = logprob; }\r\n\t}\r\n\t\r\n\t/** The top step model. */\r\n\tprotected ConditionalProbabilityTable topModel;\r\n\t/** The lexical step model. */\r\n\tprotected ConditionalProbabilityTable leafModel;\r\n\t/** The unary step model. */\r\n\tprotected ConditionalProbabilityTable unaryModel;\r\n\t/** The binary step model. */\r\n\tprotected ConditionalProbabilityTable binaryModel;\r\n\t\r\n\t/** Constructor with file names. */\r\n\tpublic GenerativeSyntacticModel(String topModelFN, String leafModelFN, String unaryModelFN, String binaryModelFN) throws IOException {\r\n\t\ttopModel = new ConditionalProbabilityTable(topModelFN);\r\n\t\tleafModel = new ConditionalProbabilityTable(leafModelFN);\r\n\t\tunaryModel = new ConditionalProbabilityTable(unaryModelFN);\r\n\t\tbinaryModel = new ConditionalProbabilityTable(binaryModelFN);\r\n\t}\r\n\r\n /** Flag for whether to show scoring breakdown. */\r\n protected boolean debugScore = false;\r\n \r\n /** Sets the debug score flag, and propagates to component models. */\r\n public void setDebug(boolean debugScore) { \r\n \tthis.debugScore = debugScore; \r\n \ttopModel.setDebug(debugScore);\r\n \tleafModel.setDebug(debugScore);\r\n \tunaryModel.setDebug(debugScore);\r\n \tbinaryModel.setDebug(debugScore);\r\n } \r\n \r\n\t/** The alphabet. */\r\n\tprotected Alphabet alphabet = null;\r\n\r\n\t/** Generative logprob feature. */\r\n\tprotected Alphabet.Feature genlogprobFeature = null;\r\n\t\r\n\t/** Sets the alphabet. */\r\n\tpublic void setAlphabet(Alphabet alphabet) {\r\n\t\tthis.alphabet = alphabet;\r\n\t\tList keys = new ArrayList(1);\r\n\t\tkeys.add(genlogprobkey);\r\n\t\tgenlogprobFeature = alphabet.closed() ? alphabet.index(keys) : alphabet.add(keys);\r\n\t}\r\n\t\r\n\t/** Returns the features for the given sign and completeness flag. */\r\n\tpublic FeatureVector extractFeatures(Sign sign, boolean complete) {\r\n\t\treturn genLogProbVector((float) logprob(sign, complete));\r\n\t}\r\n\t\r\n\t/** Returns a feature vector with the given gen log prob. */\r\n\tprotected FeatureVector genLogProbVector(float logprob) {\r\n\t\tFeatureList retval = new FeatureList(1);\r\n\t\tif (genlogprobFeature != null) retval.add(genlogprobFeature, logprob);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\t/** Derivation handler for getting the log prob for each derivation step as a sum. */\r\n\tpublic class LogProbGetter extends DerivationHandler {\r\n\t\t// reusable list of pairs\r\n\t\tprivate List> pairs = new ArrayList>(); \r\n\t\tprivate String listPairs() {\r\n\t\t\tStringBuffer sb = new StringBuffer();\r\n\t\t\tfor (Pair pair : pairs) sb.append(pair.a).append('-').append(pair.b).append(' ');\r\n\t\t\treturn sb.toString();\r\n\t\t}\r\n\t\t/** Checks for cached value. */\r\n\t\tpublic Double checkCache(Sign sign) {\r\n\t\t\tGenLogProb glp = (GenLogProb) sign.getData(GenLogProb.class);\r\n\t\t\treturn (glp == null) ? null : glp.logprob;\r\n\t\t}\r\n\t\t/** Caches the total. */\r\n\t\tpublic void cache(Sign sign, Double total) {\r\n\t\t\tsign.addData(new GenLogProb(total));\r\n\t\t}\r\n\t\t/** Top step. */\r\n\t\tpublic Double topStep(Sign sign) {\r\n\t\t\tpairs.clear(); addTopFactors(sign, pairs); \r\n\t\t\tif (debugScore) System.out.println(\"[topStep] \" + listPairs());\r\n\t\t\treturn topModel.logprob(pairs) + handleDerivation(sign);\r\n\t\t}\r\n\t\t/** Lexical step. */\r\n\t\tpublic Double lexStep(Sign sign) {\r\n\t\t\tpairs.clear(); addLexFactors(sign, pairs); \r\n\t\t\tif (debugScore) System.out.println(\"[lexStep] \" + listPairs());\r\n\t\t\treturn leafModel.logprob(pairs); \r\n\t\t}\r\n\t\t/** Unary step. */\r\n\t\tpublic Double unaryStep(Sign sign, Sign headChild) {\r\n\t\t\tpairs.clear(); addUnaryFactors(sign, pairs, headChild);\r\n\t\t\tif (debugScore) System.out.println(\"[unaryStep] \" + listPairs());\r\n\t\t\treturn unaryModel.logprob(pairs) + handleDerivation(headChild);\r\n\t\t}\r\n\t\t/** Binary step. */\r\n\t\tpublic Double binaryStep(Sign sign, boolean left, Sign headChild, Sign siblingChild) {\r\n\t\t\tpairs.clear(); addBinaryFactors(sign, pairs, left, headChild, siblingChild); \r\n\t\t\tif (debugScore) System.out.println(\"[binaryStep] \" + listPairs());\r\n\t\t\treturn binaryModel.logprob(pairs) + handleDerivation(headChild) + handleDerivation(siblingChild);\r\n\t\t}\r\n\t}\r\n\t\r\n\t/** Derivation handler for getting the factors for each derivation step as a list of words. */\r\n\tpublic static class FactorsGetter extends DerivationHandler {\r\n\t\t/** The factors. */\r\n\t\tpublic List factors = new ArrayList();\r\n\t\t// reusable list of pairs\r\n\t\tprivate List> pairs = null;\r\n\t\t// new pairs\r\n\t\tprivate void newPairs() { pairs = new ArrayList>(); }\r\n\t\t// adds new word for pairs to result\r\n\t\tprivate void addPairs() { factors.add(new ListPairWord(pairs)); }\r\n\t\t/** Top step. */\r\n\t\tpublic Void topStep(Sign sign) {\r\n\t\t\tnewPairs(); addTopFactors(sign, pairs); addPairs(); \r\n\t\t\thandleDerivation(sign); return null;\r\n\t\t}\r\n\t\t/** Lexical step. */\r\n\t\tpublic Void lexStep(Sign sign) {\r\n\t\t\tnewPairs(); addLexFactors(sign, pairs); addPairs(); return null;\r\n\t\t}\r\n\t\t/** Unary step. */\r\n\t\tpublic Void unaryStep(Sign sign, Sign headChild) {\r\n\t\t\tnewPairs(); addUnaryFactors(sign, pairs, headChild); addPairs(); \r\n\t\t\thandleDerivation(headChild); return null;\r\n\t\t}\r\n\t\t/** Binary step. */\r\n\t\tpublic Void binaryStep(Sign sign, boolean left, Sign headChild, Sign siblingChild) {\r\n\t\t\tnewPairs(); addBinaryFactors(sign, pairs, left, headChild, siblingChild); addPairs(); \r\n\t\t\thandleDerivation(headChild); handleDerivation(siblingChild); return null;\r\n\t\t}\r\n\t}\r\n\t\r\n\t/** Returns the probability of the derivation according to the models. */\r\n\tpublic double score(Sign sign, boolean complete) {\r\n\t\treturn NgramScorer.convertToProb(logprob(sign, complete));\r\n\t}\r\n\t\r\n\t/** Returns the log probability of the derivation according to the models. */\r\n\tpublic double logprob(Sign sign, boolean complete) {\r\n\t\tLogProbGetter lpgetter = new LogProbGetter();\r\n\t\tif (complete) return lpgetter.handleCompleteDerivation(sign);\r\n\t\telse return lpgetter.handleDerivation(sign);\r\n\t}\r\n\t\r\n\t/** Returns the factors from the derivation of the given sign (assumed to be complete). */\r\n\tpublic static List getFactors(Sign sign) {\r\n\t\tFactorsGetter fgetter = new FactorsGetter();\r\n\t\tfgetter.handleCompleteDerivation(sign);\r\n\t\treturn fgetter.factors;\r\n\t}\r\n\t\r\n\t/** Adds the factors for the top step in the derivation of the given sign. */\r\n\tpublic static void addTopFactors(Sign sign, List> pairs) {\r\n\t\tpairs.add(new Pair(EXPANSION, TOP));\r\n\t\tpairs.add(new Pair(PARENT, TOP));\r\n\t\tpairs.add(new Pair(LEXCAT_PARENT, TOP));\r\n\t\tpairs.add(new Pair(WORD_PARENT, TOP));\r\n\t\tpairs.add(new Pair(HEAD, sign.getSupertag()));\r\n\t\tSign lexHead = sign.getLexHead();\r\n\t\tpairs.add(new Pair(LEXCAT_TOP, lexHead.getSupertag()));\r\n\t\tpairs.add(new Pair(POS_TOP, lexHead.getPOS()));\r\n\t\tpairs.add(new Pair(WORD_TOP, lexHead.getWordForm()));\r\n\t}\r\n\t\r\n\t/** Adds the factors for a lexical step in the derivation of the given sign. */\r\n\tpublic static void addLexFactors(Sign sign, List> pairs) {\r\n\t\tpairs.add(new Pair(EXPANSION, LEAF));\r\n\t\taddParentFactors(sign, pairs);\r\n\t}\r\n\t\r\n\t/** Adds the parent factors for a step in the derivation of the given sign. */\r\n\tpublic static void addParentFactors(Sign sign, List> pairs) {\r\n\t\tpairs.add(new Pair(PARENT, sign.getSupertag()));\r\n\t\tSign lexHead = sign.getLexHead();\r\n\t\tpairs.add(new Pair(LEXCAT_PARENT, lexHead.getSupertag()));\r\n\t\tpairs.add(new Pair(POS_PARENT, lexHead.getPOS()));\r\n\t\tpairs.add(new Pair(WORD_PARENT, lexHead.getWordForm()));\r\n\t}\r\n\t\r\n\t/** Returns the factors for a unary step in the derivation of the given sign. */\r\n\tpublic static void addUnaryFactors(Sign sign, List> pairs, Sign headChild) {\r\n\t\tpairs.add(new Pair(EXPANSION, UNARY));\r\n\t\taddParentFactors(sign, pairs);\r\n\t\tpairs.add(new Pair(HEAD, headChild.getSupertag()));\r\n\t}\r\n\t\r\n\t/** Returns the factors for a binary step in the derivation of the given sign. */\r\n\tpublic static void addBinaryFactors(Sign sign, List> pairs, boolean left, Sign headChild, Sign siblingChild) {\r\n\t\tpairs.add(new Pair(EXPANSION, (left) ? LEFT : RIGHT));\r\n\t\taddParentFactors(sign, pairs);\r\n\t\tpairs.add(new Pair(HEAD, headChild.getSupertag()));\r\n\t\tpairs.add(new Pair(SIBLING, siblingChild.getSupertag()));\r\n\t\tSign siblingLexHead = siblingChild.getLexHead();\r\n\t\tpairs.add(new Pair(LEXCAT_SIBLING, siblingLexHead.getSupertag()));\r\n\t\tpairs.add(new Pair(POS_SIBLING, siblingLexHead.getPOS()));\r\n\t\tpairs.add(new Pair(WORD_SIBLING, siblingLexHead.getWordForm()));\r\n\t}\r\n\t\r\n /** Tests loading and scoring. */\r\n public static void main(String[] args) throws IOException {\r\n \r\n \tString argstr = \"(-dir ) (-g ) (-t ) (-verbose)\";\r\n String usage = \"Usage: java opennlp.ccg.synsem.GenerativeSyntacticModel \" + argstr;\r\n \r\n if (args.length > 0 && args[0].equals(\"-h\")) {\r\n System.out.println(usage);\r\n System.exit(0);\r\n }\r\n \r\n String dir = \".\", topfn = \"top.flm\", leaffn = \"leaf.flm\", unaryfn = \"unary.flm\", binaryfn = \"binary.flm\";\r\n String grammarfn = \"grammar.xml\", tbfn = \"testbed.xml\";\r\n boolean verbose = false;\r\n \r\n for (int i=0; i < args.length; i++) {\r\n \tif (args[i].equals(\"-dir\")) { dir = args[++i]; continue; }\r\n \tif (args[i].equals(\"-g\")) { grammarfn = args[++i]; continue; }\r\n \tif (args[i].equals(\"-t\")) { tbfn = args[++i]; continue; }\r\n \tif (args[i].equals(\"-v\") || args[i].equals(\"-verbose\")) { verbose = true; continue; }\r\n \tSystem.out.println(\"Unrecognized option: \" + args[i]);\r\n }\r\n \r\n // load grammar\r\n URL grammarURL = new File(grammarfn).toURI().toURL();\r\n System.out.println(\"Loading grammar from URL: \" + grammarURL);\r\n Grammar grammar = new Grammar(grammarURL);\r\n \r\n // load model\r\n System.out.println(\"Loading syntactic model from: \" + dir);\r\n topfn = dir + \"/\" + topfn; leaffn = dir + \"/\" + leaffn; unaryfn = dir + \"/\" + unaryfn; binaryfn = dir + \"/\" + binaryfn;\r\n GenerativeSyntacticModel model = new GenerativeSyntacticModel(topfn, leaffn, unaryfn, binaryfn);\r\n if (verbose) model.setDebug(true);\r\n \r\n \t// score saved signs\r\n \tdouble logprobttotal = 0.0;\r\n \tint numsents = 0;\r\n \tfor (File f : Regression.getXMLFiles(new File(tbfn))) {\r\n // load testfile\r\n \tSystem.out.println(\"Loading: \" + f.getName());\r\n RegressionInfo rinfo = new RegressionInfo(grammar, f);\r\n // do each item\r\n\t \tfor (int i=0; i < rinfo.numberOfItems(); i++) {\r\n\t \t\tTestItem item = rinfo.getItem(i);\r\n\t \t\tif (item.numOfParses == 0) continue;\r\n\t \t\tnumsents++;\r\n\t \t\tif (verbose) System.out.println(\"scoring: \" + item.sentence);\r\n\t \t\telse System.out.print(\".\");\r\n\t \t\tSign sign = item.sign;\r\n\t double logprob = model.logprob(sign, true);\r\n\t logprobttotal += logprob;\r\n\t if (verbose) {\r\n\t\t \t\tSystem.out.println(sign.getDerivationHistory().toString());\r\n\t\t System.out.println(\"logprob: \" + logprob);\r\n\t }\r\n\t \t}\r\n\t \tSystem.out.println();\r\n \t}\r\n \t\r\n \t// totals\r\n \tSystem.out.println(\"total logprob: \" + logprobttotal);\r\n \tSystem.out.println(\"logprob per sentence: \" + (logprobttotal / numsents));\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/synsem/LF.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-5 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\nimport opennlp.ccg.unify.*;\nimport org.jdom.*;\nimport gnu.trove.*;\n\n/**\n * An interface for objects which represent Logical Forms.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.10 $, $Date: 2005/11/01 22:35:35 $\n */\npublic interface LF extends Unifiable, Mutable {\n\n /**\n * Sets the LF chunks to which this LF belongs.\n * LF chunks are used during realization to ensure \n * that certain edges are semantically complete \n * before combination is attempted with edges \n * with semantics outside the chunk.\n * The chunks are numbered starting with 0, \n * and null represents no chunks.\n */\n public void setChunks(TIntArrayList chunks);\n \n /**\n * Gets the LF chunks to which this LF belongs.\n */\n public TIntArrayList getChunks();\n \n\n /** Returns the simple type of this LF, or null if none. */\n public SimpleType getType();\n \n \n /**\n * Returns a copy of this LF.\n * (LF chunks are not copied.)\n */\n public LF copy();\n\n /**\n * Returns a hash code using the given map from vars to ints.\n */\n public int hashCode(TObjectIntHashMap varMap);\n\n /**\n * Returns whether this LF equals the given object \n * up to variable names, using the given maps from vars to ints.\n */\n public boolean equals(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2);\n \n /**\n * Returns an XML representation of this LF.\n */\n public Element toXml();\n \n /**\n * Returns a pretty-printed string of this LF, with the given indent.\n */\n public String prettyPrint(String indent);\n}\n" }, { "path": "src/opennlp/ccg/synsem/LexLogProbFeatureExtractor.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2009 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.synsem;\r\n\r\nimport java.util.*;\r\n\r\nimport opennlp.ccg.perceptron.*;\r\nimport opennlp.ccg.lexicon.SupertaggerAdapter;\r\n\r\n/** \r\n * A class for extracting total lexical log probabilities from a supertagger \r\n * as a feature. The class may also be used as a sign scorer.\r\n * \r\n * @author \tMichael White\r\n * @version $Revision: 1.3 $, $Date: 2009/11/01 22:26:29 $\r\n */ \r\npublic class LexLogProbFeatureExtractor implements FeatureExtractor, SignScorer {\r\n\r\n\t/** Feature key. */\r\n\tpublic static String lexlogprobkey = \"lexlogprob\";\r\n\t\r\n\t/** The alphabet. */\r\n\tprotected Alphabet alphabet = null;\r\n\r\n\t/** Lexical logprob feature. */\r\n\tprotected Alphabet.Feature lexlogprobFeature = null;\r\n\t\r\n\t/** Sets the alphabet. */\r\n\tpublic void setAlphabet(Alphabet alphabet) {\r\n\t\tthis.alphabet = alphabet;\r\n\t\tList keys = new ArrayList(1);\r\n\t\tkeys.add(lexlogprobkey);\r\n\t\tlexlogprobFeature = alphabet.closed() ? alphabet.index(keys) : alphabet.add(keys);\r\n\t}\r\n\t\r\n\t/** Returns the features for the given sign and completeness flag. */\r\n\tpublic FeatureVector extractFeatures(Sign sign, boolean complete) {\r\n\t\treturn lexLogProbVector(getLexLogProb(sign, complete));\r\n\t}\r\n\t\r\n\t/** Recursively gets lex log prob total for the given sign, if not already present. */\r\n\tprotected float getLexLogProb(Sign sign, boolean complete) {\r\n\t\t// check for stored log prob \r\n\t\tSupertaggerAdapter.LexLogProb lexlogprob = \r\n\t\t\t(SupertaggerAdapter.LexLogProb) sign.getData(SupertaggerAdapter.LexLogProb.class);\r\n\t\tif (lexlogprob != null) return lexlogprob.logprob;\r\n\t\t// otherwise calculate and store one\r\n\t\tfloat logprob = 0;\r\n\t\t// lex case\r\n\t\tif (sign.isLexical()) {\r\n\t\t\t// just use zero if not already there\r\n\t\t}\r\n\t\t// non-terminal\r\n\t\telse {\r\n\t\t\t// use input totals to calculate current one\r\n\t\t\tSign[] inputs = sign.getDerivationHistory().getInputs();\r\n\t\t\tif (inputs.length == 1) \r\n\t\t\t\tlogprob = getLexLogProb(inputs[0], false);\r\n\t\t\telse if (inputs.length == 2) \r\n\t\t\t\tlogprob = getLexLogProb(inputs[0], false) + getLexLogProb(inputs[1], false);\r\n\t\t}\r\n\t\t// store it and return\r\n\t\tsign.addData(new SupertaggerAdapter.LexLogProb(logprob));\r\n\t\treturn logprob;\r\n\t}\r\n\t\r\n\t/** Returns a feature vector with the given lex log prob total. */\r\n\tprotected FeatureVector lexLogProbVector(float logprob) {\r\n\t\tFeatureList retval = new FeatureList(1);\r\n\t\tif (lexlogprobFeature != null) retval.add(lexlogprobFeature, logprob);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n /** \r\n * Returns a score for the given sign and completeness flag; \r\n * specifically, returns the lex log prob total for the sign.\r\n */\r\n public double score(Sign sign, boolean complete) {\r\n \treturn getLexLogProb(sign, complete);\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/synsem/LexSemOrigin.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2007 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.synsem;\r\n\r\n/**\r\n * An interface for items which introduce lexical semantics, covering \r\n * (lexical) signs and unary type changing rules.\r\n *\r\n * @author Michael White\r\n * @version $Revision: 1.2 $, $Date: 2008/01/03 21:30:12 $\r\n */\r\npublic interface LexSemOrigin {\r\n\t\r\n /**\r\n * Returns the supertag.\r\n */\r\n public String getSupertag();\r\n \r\n /**\r\n * Returns the POS tag. \r\n * For unary type changing rules, the constant TypeChangingRule.POS_STRING is \r\n * always returned.\r\n */\r\n public String getPOS();\r\n \r\n /**\r\n * Sets the origin of the elementary predications.\r\n */\r\n public void setOrigin();\r\n}\r\n" }, { "path": "src/opennlp/ccg/synsem/Modality.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.synsem;\n\nimport opennlp.ccg.unify.*;\n\n/**\n * A modality that can decorate a categorial slash.\n *\n * @author Jason Baldridge\n * @version $Revision: 1.2 $, $Date: 2004/05/01 10:40:04 $\n */\npublic interface Modality extends Unifiable {\n public Object copy();\n public byte getDirection();\n public String toString(byte dir);\n public String toTeX(byte dir);\n public String toTeX();\n}\n" }, { "path": "src/opennlp/ccg/synsem/ReRankingScorer.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2011 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.synsem;\r\n\r\n/**\r\n * Interface for signs scorers with both a base model and a full (reranking) model.\r\n * The implementation of the score(Sign, boolean) method should vary according to the \r\n * full model flag. The base model should be the default. \r\n */\r\npublic interface ReRankingScorer extends SignScorer {\r\n\t/** Sets flag for using full (vs. base) model. The base model should be the default. */\r\n\tpublic void setFullModel(boolean on);\r\n}\r\n" }, { "path": "src/opennlp/ccg/synsem/SetArg.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-5 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\nimport opennlp.ccg.unify.*;\nimport org.jdom.*;\nimport java.io.Serializable;\nimport java.util.*;\nimport gnu.trove.*;\n\n/**\n * A category which contains an unordered set of categories.\n * \n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.11 $, $Date: 2009/12/21 02:15:44 $\n */\npublic final class SetArg implements Arg, Serializable {\n\n\tprivate static final long serialVersionUID = -7067480310511294657L;\n\t\n\tprivate ArgStack _args;\n\n\t@SuppressWarnings(\"unchecked\")\n\tpublic SetArg(Element el) {\n\t\tList info = el.getChildren();\n\t\tList args = new ArrayList();\n\t\tfor (Iterator infoIt = info.iterator(); infoIt.hasNext();) {\n\t\t\tSlash s = new Slash(infoIt.next());\n\t\t\tCategory c = CatReader.getCat(infoIt.next());\n\t\t\targs.add(new BasicArg(s, c));\n\t\t}\n\t\tArg[] list = new Arg[args.size()];\n\t\targs.toArray(list);\n\t\t_args = new ArgStack(list);\n\t}\n\n\tpublic SetArg(Arg[] args) {\n\t\t_args = new ArgStack(args);\n\t}\n\n\tpublic SetArg(ArgStack args) {\n\t\t_args = args;\n\t}\n\n\tpublic Element toXml() {\n\t\tElement retval = new Element(\"setarg\");\n\t\tfor (Arg arg : _args._list) {\n\t\t\tif (arg instanceof BasicArg) { // only supporting basic args per xml construction\n\t\t\t\tBasicArg barg = (BasicArg) arg;\n\t\t\t\tretval.addContent(barg.getSlash().toXml());\n\t\t\t\tretval.addContent(barg.getCat().toXml());\n\t\t\t}\n\t\t}\n\t\treturn retval;\n\t}\n\t\n\tpublic Arg copy() {\n\t\treturn new SetArg(_args.copy());\n\t}\n\n\tpublic void add(ArgStack as) {\n\t\t_args.add(as);\n\t}\n\n\tpublic void forall(CategoryFcn fcn) {\n\t\t_args.forall(fcn);\n\t}\n\n\tpublic Arg copyWithout(int pos) {\n\t\tif (_args.size() == 2) {\n\t\t\tif (pos == 0) {\n\t\t\t\treturn _args.get(1);\n\t\t\t} else {\n\t\t\t\treturn _args.get(0);\n\t\t\t}\n\t\t} else {\n\t\t\treturn new SetArg(_args.copyWithout(pos));\n\t\t}\n\t}\n\n\tpublic int size() {\n\t\treturn _args.size();\n\t}\n\n\tpublic BasicArg get(int pos) {\n\t\treturn (BasicArg) _args.get(pos);\n\t}\n\n\tpublic Category getCat(int pos) {\n\t\treturn ((BasicArg) _args.get(pos)).getCat();\n\t}\n\n\tpublic int indexOf(BasicArg a) {\n\t\tint index = -1;\n\t\tfor (int i = 0; i < _args.size() && index < 0; i++) {\n\t\t\ttry {\n\t\t\t\ta.unifySlash(((BasicArg) _args.get(i)).getSlash());\n\t\t\t\tGUnifier.unify(getCat(i), a.getCat());\n\t\t\t\tindex = i;\n\t\t\t} catch (UnifyFailure uf) {\n\t\t\t}\n\t\t}\n\t\t// if (index<0) {\n\t\t// throw new UnifyFailure();\n\t\t// } else {\n\t\t// return index;\n\t\t// }\n\t\treturn index;\n\t}\n\n\tpublic int indexOf(Category cat) {\n\t\tint index = -1;\n\t\tfor (int i = 0; i < _args.size() && index < 0; i++) {\n\t\t\ttry {\n\t\t\t\tGUnifier.unify(getCat(i), cat);\n\t\t\t\tindex = i;\n\t\t\t} catch (UnifyFailure uf) {\n\t\t\t}\n\t\t}\n\t\treturn index;\n\t\t// if (index<0) {\n\t\t// throw new UnifyFailure();\n\t\t// } else {\n\t\t// return index;\n\t\t// }\n\t}\n\n\tpublic void setSlashModifier(boolean modifier) { \n\t\tfor (int i = 0; i < _args.size(); i++) {\n\t\t\tBasicArg arg = get(i);\n\t\t\targ.setSlashModifier(modifier);\n\t\t}\n\t}\n\t\n public void setSlashHarmonicCompositionResult(boolean harmonicResult) { \n\t\tfor (int i = 0; i < _args.size(); i++) {\n\t\t\tBasicArg arg = get(i);\n\t\t\targ.setSlashHarmonicCompositionResult(harmonicResult);\n\t\t}\n\t}\n \n\tpublic boolean containsContrarySlash() {\n\t\tfor (int i = 0; i < _args.size(); i++) {\n\t\t\tif (!((BasicArg) _args.get(i)).getSlash().sameDirAsModality()) {\n\t\t\t\treturn true;\n\t\t\t}\n\t\t}\n\t\treturn false;\n\t}\n\n\tpublic void unifySlash(Slash s) throws UnifyFailure {\n\t\tfor (int i = 0; i < _args.size(); i++) {\n\t\t\t_args.get(i).unifySlash(s);\n\t\t}\n\t}\n\n\tpublic void unifyCheck(Object u) throws UnifyFailure {\n\t}\n\n\t// nb: direct unification not implemented ...\n\tpublic Object unify(Object u, Substitution sub) throws UnifyFailure {\n\t\tthrow new UnifyFailure();\n\t}\n\n\tpublic Object fill(Substitution s) throws UnifyFailure {\n\t\treturn new SetArg(_args.fill(s));\n\t}\n\n\tpublic void deepMap(ModFcn mf) {\n\t\t_args.deepMap(mf);\n\t}\n\n\tpublic boolean occurs(Variable v) {\n\t\treturn _args.occurs(v);\n\t}\n\n\tpublic boolean equals(Object c) {\n\t\treturn false;\n\t}\n\n\tpublic String toString() {\n\t\tStringBuffer sb = new StringBuffer(10);\n\t\tsb.append('{').append(_args.toString()).append('}');\n\t\treturn sb.toString();\n\t}\n\n\t/**\n\t * Returns the supertag for this arg.\n\t */\n\tpublic String getSupertag() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tsb.append(\"{\").append(_args.getSupertag()).append(\"}\");\n\t\treturn sb.toString();\n\t}\n\n\t/**\n\t * Returns a TeX-formatted string representation for this arg.\n\t */\n\tpublic String toTeX() {\n\t\tStringBuffer sb = new StringBuffer(10);\n\t\tsb.append(\"\\\\{\").append(_args.toTeX()).append(\"\\\\}\");\n\t\treturn sb.toString();\n\t}\n\n\t/**\n\t * Returns a hash code for this arg, using the given map from vars to ints.\n\t */\n\tpublic int hashCode(TObjectIntHashMap varMap) {\n\t\treturn _args.hashCode(varMap);\n\t}\n\n\t/**\n\t * Returns whether this arg equals the given object up to variable names,\n\t * using the given maps from vars to ints.\n\t */\n\tpublic boolean equals(Object obj, TObjectIntHashMap varMap,\n\t\t\tTObjectIntHashMap varMap2) {\n\t\tif (obj.getClass() != this.getClass()) {\n\t\t\treturn false;\n\t\t}\n\t\tSetArg sa = (SetArg) obj;\n\t\treturn _args.equals(sa._args, varMap, varMap2);\n\t}\n}\n" }, { "path": "src/opennlp/ccg/synsem/Sign.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-9 Jason Baldridge, University of Edinburgh and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\nimport opennlp.ccg.parse.*;\nimport opennlp.ccg.util.*;\nimport opennlp.ccg.lexicon.*;\nimport opennlp.ccg.grammar.*;\nimport opennlp.ccg.hylo.*;\n\nimport org.jdom.*;\n\nimport gnu.trove.*;\n\nimport java.io.*;\nimport java.util.*;\n\n/**\n * A CCG sign, consisting of a list of words paired with a category.\n * Signs may contain arbitrary data objects which are ignored in equality checking.\n * Non-serializable data objects are filtered during serialization.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.44 $, $Date: 2011/08/27 19:27:01 $\n */\npublic class Sign implements LexSemOrigin, Serializable {\n \n\tprivate static final long serialVersionUID = 1072712272514007274L;\n\n\t/** The words. */\n protected List _words;\n \n /** The category. */\n protected Category _cat;\n \n /** The derivation history. */\n protected DerivationHistory _history;\n \n /** The lexical head. */\n protected Sign _lexHead;\n \n \n /** List of transient data objects, for retrieval by class. */\n protected LinkedList data = null; \n \n\n /** Constructor for subclasses. */\n protected Sign() {}\n \n /** Constructor with derivation history. */\n @SuppressWarnings(\"unchecked\")\n\tprotected Sign(List words, Category cat, DerivationHistory dh, Sign lexHead) {\n _words = (List) Interner.globalIntern(words); \n _cat = cat;\n _history = dh;\n _lexHead = lexHead;\n }\n\n /** Constructor with no additional derivation history. */\n public Sign(List words, Category cat) {\n this(words, cat, null, null);\n _history = new DerivationHistory(this);\n _lexHead = this;\n }\n\n /** Constructor with no additional derivation history. */\n public Sign(Word word, Category cat) {\n this(new SingletonList(word), cat);\n }\n \n \n // during deserialization, interns words\n @SuppressWarnings(\"unchecked\")\n\tprivate void readObject(java.io.ObjectInputStream in) throws IOException, ClassNotFoundException {\n \tin.defaultReadObject();\n _words = (List) Interner.globalIntern(_words); \n }\n \n // during serialization, skips non-serializable data objects\n private void writeObject(java.io.ObjectOutputStream stream) throws IOException {\n \t// save old data objects\n \tLinkedList tmp = data;\n \t// filter non-serializable ones\n \tif (tmp != null) {\n \t\tdata = new LinkedList();\n \t\tfor (Object obj : tmp) {\n \t\t\tif (obj instanceof Serializable) data.add(obj);\n \t\t}\n \t\tif (data.isEmpty()) data = null;\n \t}\n \t// serialize\n \tstream.defaultWriteObject();\n \t// restore old data objects\n \tdata = tmp;\n }\n\n \n /** Factory method for creating a sign from a lexical sign plus a coarticulation one. */\n public static Sign createCoartSign(Category cat, Sign lexSign, Sign coartSign) {\n List words = lexSign.getWords();\n if (words.size() > 1) \n throw new RuntimeException(\"Can't create coarticulation sign from multiple words.\");\n Word word = words.get(0);\n Word coartWord = coartSign.getWords().get(0);\n Word wordPlus = Word.createWordWithAttrs(word, coartWord);\n Sign retval = new Sign(new SingletonList(wordPlus), cat, null, null);\n retval._lexHead = retval;\n Rule coartRule = new Rule() {\n public String name() { return \"coart\"; }\n public int arity() { return 1; }\n public List applyRule(Category[] inputs) { throw new RuntimeException(\"Not supported.\"); }\n public RuleGroup getRuleGroup() { throw new RuntimeException(\"Not supported.\"); }\n public void setRuleGroup(RuleGroup ruleGroup) { throw new RuntimeException(\"Not supported.\"); }\n public Element toXml() { throw new RuntimeException(\"Not supported.\"); }\n };\n retval._history = new DerivationHistory(new Sign[]{lexSign,coartSign}, retval, coartRule);\n return retval;\n }\n \n /** Factory method for creating derived signs with the given cat from the given inputs, rule and lex head. */\n public static Sign createDerivedSign(Category cat, Sign[] inputs, Rule rule, Sign lexHead) {\n return new Sign(cat, inputs, rule, lexHead);\n }\n\n /** Factory method for creating derived signs from the given result cat, inputs, rule and lex head, \n with a new LF constructed from the inputs.\n Note that unlike with rule applications, the result LF is constructed with \n no var substitutions, so it is useful only for creating alternative signs during realization. */\n public static Sign createDerivedSignWithNewLF(Category cat, Sign[] inputs, Rule rule, Sign lexHead) {\n Category copyCat = cat.shallowCopy();\n LF lf = null;\n for (int i = 0; i < inputs.length; i++) {\n lf = HyloHelper.append(lf, inputs[i].getCategory().getLF());\n }\n if (rule instanceof TypeChangingRule) {\n TypeChangingRule tcr = (TypeChangingRule) rule;\n lf = HyloHelper.append(lf, tcr.getResult().getLF());\n }\n if (lf != null) { HyloHelper.sort(lf); }\n copyCat.setLF(lf);\n return new Sign(copyCat, inputs, rule, lexHead);\n }\n \n /** Constructor with words and derivation history formed from the given inputs, rule and lex head. */\n protected Sign(Category cat, Sign[] inputs, Rule rule, Sign lexHead) {\n this(getRemainingWords(inputs, 0), cat, null, lexHead);\n _history = new DerivationHistory(inputs, this, rule);\n }\n \n // returns the remaining words in a structure sharing way\n private static List getRemainingWords(Sign[] inputs, int index) {\n // if (inputs.length == 0) throw new RuntimeException(\"Error: can't make sign from zero inputs\");\n if (index == (inputs.length - 1)) return inputs[index]._words;\n return new StructureSharingList(\n inputs[index]._words,\n getRemainingWords(inputs, index+1)\n );\n }\n\n \n /** Returns the words of the sign. */\n public List getWords() {\n return _words;\n }\n\n /** Returns the words as a string. Delegates to the current tokenizer's getOrthography method. */\n public String getOrthography() {\n return Grammar.theGrammar.lexicon.tokenizer.getOrthography(_words);\n }\n\n /** Returns the sign's category. */\n public Category getCategory() {\n return _cat;\n }\n\n /** Returns whether the sign is lexical. */\n public boolean isLexical() { return _history.isEmpty(); }\n \n /** Sets the derivation history. */\n public void setDerivationHistory(DerivationHistory dh) {\n _history = dh;\n }\n \n /** Returns the derivation history. */\n public DerivationHistory getDerivationHistory() {\n return _history;\n }\n\n /** Returns the lexical head. */\n public Sign getLexHead() { return _lexHead; }\n \n \n /** Returns a hash code for this sign. */ \n public int hashCode() {\n return System.identityHashCode(_words) + _cat.hashCode();\n }\n \n /** Returns whether this sign equals the given object. */\n public boolean equals(Object obj) {\n if (obj == this) return true;\n if (!(obj instanceof Sign)) return false;\n Sign sign = (Sign) obj;\n return _words == sign._words && _cat.equals(sign._cat);\n }\n\n \n /** \n * Returns a hash code for this sign with the words restricted to surface words; \n * with lexical signs, however, the original hash code is returned, so that \n * words with signs that differ just in their pos tags can be distinguished \n * (for robustness).\n */ \n public int surfaceWordHashCode() {\n \treturn surfaceWordHashCode(false);\n }\n \n /** \n * Returns a hash code for this sign with the words restricted to surface words, \n * and with the LF ignored according to the given flag; \n * with lexical signs, however, the original hash code is returned, so that \n * words with signs that differ just in their pos tags can be distinguished \n * (for robustness).\n */ \n public int surfaceWordHashCode(boolean ignoreLF) {\n // original hash code for lex signs\n if (_history.getInputs() == null) return hashCode();\n // otherwise use surface words\n int hc = 1;\n for (int i = 0; i < _words.size(); i++) {\n Word word = _words.get(i);\n hc = 31*hc + word.surfaceWordHashCode();\n }\n hc += (ignoreLF) ? _cat.hashCodeNoLF() : _cat.hashCode();\n return hc;\n }\n \n /** \n * Returns whether this sign and the given object have equal categories and\n * restrictions to surface words; \n * with lexical signs, however, the original equals result is returned, so that \n * words with signs that differ just in their pos tags can be distinguished \n * (for robustness).\n */\n public boolean surfaceWordEquals(Object obj) {\n \treturn surfaceWordEquals(obj, false);\n }\n\n /** \n * Returns whether this sign and the given object have equal categories and\n * restrictions to surface words, \n * with the LF ignored according to the given flag; \n * with lexical signs, however, the original equals result is returned, so that \n * words with signs that differ just in their pos tags can be distinguished \n * (for robustness).\n */\n public boolean surfaceWordEquals(Object obj, boolean ignoreLF) {\n if (obj == this) return true;\n if (!(obj instanceof Sign)) return false;\n Sign sign = (Sign) obj;\n // original equals for lex signs\n if (_history.getInputs() == null || sign._history.getInputs() == null) \n return equals(sign);\n // otherwise use surface words\n if (_words.size() != sign._words.size()) return false;\n for (int i = 0; i < _words.size(); i++) {\n Word word = _words.get(i); \n Word signWord = (Word) sign._words.get(i);\n if (!word.surfaceWordEquals(signWord)) return false;\n }\n return (ignoreLF) ? _cat.equalsNoLF(sign._cat) : _cat.equals(sign._cat);\n }\n\n \n /** Returns 'orthography :- category'. */\n public String toString() {\n return getOrthography() + \" :- \" + _cat.toString(); // for lex head: + \" --> \" + _lexHead.getWordForm();\n }\n \n \n /** \n * Returns the words in an XML doc, with no labeled spans for nominals. \n */\n public Document getWordsInXml() {\n \tSet emptySet = Collections.emptySet();\n \treturn getWordsInXml(emptySet); \n }\n \n /** \n * Returns the words in an XML doc, with labeled spans for the given nominals, \n * and with pitch accents and boundary tones converted to elements. \n * Each orthographic word appears in a separate element, \n * with multiwords grouped under a multiword element.\n * Attribute-value pairs for the word (if any) appear on the word \n * or multiword element.\n * Words are also expanded using the grammar's tokenizer.\n */\n public Document getWordsInXml(Set nominals) {\n TObjectIntHashMap nominalsMap = new TObjectIntHashMap(); \n setMaxOrthLengths(nominals, nominalsMap);\n Document doc = new Document();\n Element root = new Element(\"seg\");\n doc.setRootElement(root);\n addWordsToXml(root, nominalsMap);\n return doc;\n }\n \n // finds the maximum orthography lengths for signs headed by the given nominals\n private void setMaxOrthLengths(Set nominals, TObjectIntHashMap nominalsMap) {\n // update map\n Nominal index = _cat.getIndexNominal();\n if (index != null && nominals.contains(index)) {\n int orthLen = getOrthography().length();\n if (!nominalsMap.containsKey(index) || orthLen > nominalsMap.get(index)) {\n nominalsMap.put(index, orthLen);\n }\n }\n // recurse\n Sign[] inputs = _history.getInputs();\n if (inputs == null) return;\n for (int i = 0; i < inputs.length; i++) {\n inputs[i].setMaxOrthLengths(nominals, nominalsMap); \n }\n }\n \n // recursively adds orthographic words as XML to the given parent, \n // using the nominals map to determine labeled spans\n private void addWordsToXml(Element parent, TObjectIntHashMap nominalsMap) {\n // check for matching nominal as index of target cat; \n // if found, update parent to labeled span element\n Nominal index = _cat.getIndexNominal();\n if (index != null && nominalsMap.containsKey(index) && \n nominalsMap.get(index) == getOrthography().length()) \n {\n // remove index key from map, to avoid duplicate spans with the same length\n nominalsMap.remove(index);\n // make span element, update parent\n Element span = new Element(\"span\");\n span.setAttribute(\"label\", index.toString());\n parent.addContent(span);\n parent = span;\n }\n // process inputs from derivation history\n Sign[] inputs = _history.getInputs();\n if (inputs == null) {\n // in leaf case, word list must be a singleton\n Word word = _words.get(0); \n // check for boundary tone\n if (Grammar.isBoundaryTone(word.getForm())) {\n // add element for boundary tone\n Element boundary = new Element(\"boundary\");\n boundary.setAttribute(\"type\", word.getForm());\n parent.addContent(boundary);\n return;\n }\n // check for pitch accent\n if (word.getPitchAccent() != null) {\n // add pitchaccent element containing word(s) with corresponding accent\n Element pitchaccent = new Element(\"pitchaccent\");\n pitchaccent.setAttribute(\"type\", word.getPitchAccent());\n addWords(pitchaccent, word);\n parent.addContent(pitchaccent);\n return;\n }\n // otherwise add word(s)\n addWords(parent, word);\n return;\n }\n if (inputs.length == 1) {\n inputs[0].addWordsToXml(parent, nominalsMap);\n return;\n }\n for (int i = 0; i < inputs.length; i++) {\n inputs[i].addWordsToXml(parent, nominalsMap);\n }\n }\n \n // adds one or more word elements after expanding surface form; \n // multiwords are enclosed within a multiword element; \n // any attribute-value pairs are added to the word or multiword element\n private void addWords(Element parent, Word word) {\n List orthWords = Grammar.theGrammar.lexicon.tokenizer.expandWord(word);\n Element child;\n if (orthWords.size() == 1) {\n Element wordElt = new Element(\"word\");\n wordElt.addContent(orthWords.get(0));\n child = wordElt;\n }\n else {\n Element multiwordElt = new Element(\"multiword\");\n for (int i = 0; i < orthWords.size(); i++) {\n Element wordElt = new Element(\"word\");\n wordElt.addContent(orthWords.get(i));\n multiwordElt.addContent(wordElt);\n }\n child = multiwordElt;\n }\n for (Iterator> it = word.getAttrValPairs(); it.hasNext(); ) {\n Pair p = it.next();\n String attr = p.a; String val = p.b;\n child.setAttribute(attr, val);\n }\n parent.addContent(child);\n }\n \n\n /**\n * Returns a string showing the bracketings implied by the derivation.\n * See DerivationHistory.toString to see the complete derivation in \n * vertical list form.\n */\n public String getBracketedString() {\n Sign[] inputs = _history.getInputs();\n if (inputs == null) return getOrthography();\n if (inputs.length == 1) return inputs[0].getBracketedString();\n StringBuffer sb = new StringBuffer();\n sb.append(\"(\");\n for (int i = 0; i < inputs.length; i++) {\n sb.append(inputs[i].getBracketedString());\n if (i < (inputs.length - 1)) sb.append(\" \");\n }\n sb.append(\")\");\n return sb.toString();\n }\n \n /**\n * Returns the category's supertag.\n */\n public String getSupertag() { return _cat.getSupertag(); }\n \n /**\n * Returns the word form of the first word. \n */\n public String getWordForm() { return _words.get(0).getForm(); }\n \n /**\n * Returns the POS tag of the first word.\n */\n public String getPOS() { return _words.get(0).getPOS(); }\n \n /**\n * Sets the origin of the elementary predications.\n */\n public void setOrigin() { HyloHelper.setOrigin(_cat.getLF(), this); }\n \n /**\n * Returns the index of the first word of the given lex sign in this sign's \n * list of words, or -1 if the given lex sign is not in this sign's derivation \n * history.\n */\n public int wordIndex(Sign lexSign) {\n \treturn wordIndex(lexSign, new int[]{0});\n }\n \n // returns word index relative to input offset\n private int wordIndex(Sign lexSign, int[] offset) {\n \tif (this == lexSign) return offset[0];\n \tif (isLexical()) {\n \t\toffset[0] += _words.size();\n \t\treturn -1;\n \t}\n Sign[] inputs = _history.getInputs();\n for (int i = 0; i < inputs.length; i++) {\n \tint retval = inputs[i].wordIndex(lexSign, offset);\n \tif (retval >= 0) return retval;\n }\n return -1;\n }\n \n \n /** Adds a data object to the front of the list of data objects. */\n public void addData(Object obj) {\n \tif (data == null) data = new LinkedList();\n \tdata.addFirst(obj);\n }\n \n /** Returns the first data object with the given class, or null if none. */\n public Object getData(Class objClass) {\n \tif (data == null) return null;\n \tfor (Object obj : data) {\n \t\tif (obj.getClass() == objClass) return obj;\n \t}\n \treturn null;\n }\n\n \n\t/** Unfilled dependencies wrapper, for unique retrieval from data objects. */\n\tpublic static class UnfilledDeps {\n\t\tpublic List unfilledDeps; \n\t\tpublic UnfilledDeps(List unfilledDeps) { this.unfilledDeps = unfilledDeps; }\n\t}\n\t\n\t/** Filled dependencies wrapper, for unique retrieval from data objects. */\n\tpublic static class FilledDeps {\n\t\tpublic List filledDeps; \n\t\tpublic FilledDeps(List filledDeps) { this.filledDeps = filledDeps; }\n\t}\n\t\n\t/** Returns the unfilled dependencies for this sign, with caching. */\n\tpublic List getUnfilledDeps() {\n\t\t// check cache\n\t\tUnfilledDeps udeps = (UnfilledDeps) getData(UnfilledDeps.class);\n\t\tif (udeps != null) return udeps.unfilledDeps;\n\t\t// lex case: calculate, store and return\n\t\tif (isLexical()) {\n\t\t\tList unfilledDeps = HyloHelper.getUnfilledLexDeps(_cat.getLF());\n\t\t\taddData(new UnfilledDeps(unfilledDeps));\n\t\t\treturn unfilledDeps;\n\t\t}\n\t\t// otherwise compute filled deps, with unfilled determined as a side effect, and return cached result\n\t\tgetFilledDeps();\n\t\tudeps = (UnfilledDeps) getData(UnfilledDeps.class);\n\t\treturn udeps.unfilledDeps;\n\t}\n \n\t/** Returns the filled dependencies for this sign, with caching. */\n\tpublic List getFilledDeps() {\n\t\t// skip lex case\n\t\tif (isLexical()) return Collections.emptyList();\n\t\t// check cache\n\t\tFilledDeps fdeps = (FilledDeps) getData(FilledDeps.class);\n\t\tif (fdeps != null) return fdeps.filledDeps;\n\t\t// otherwise get unfilled deps from children recursively\n\t\tList unfilledDeps = new ArrayList(5);\n\t\tSign[] inputs = _history.getInputs();\n for (int i = 0; i < inputs.length; i++) {\n \tunfilledDeps.addAll(inputs[i].getUnfilledDeps());\n }\n // calculate filled deps\n List filledDeps = HyloHelper.getFilledLexDeps(unfilledDeps, _cat.getLF());\n // store filled and unfilled, returning filled\n addData(new UnfilledDeps(unfilledDeps));\n addData(new FilledDeps(filledDeps));\n \treturn filledDeps;\n\t}\n\n\t/** \n\t * Returns the sibling filled dependencies for this sign by recursively \n\t * filtering the filled dependencies from the input signs for those with \n\t * the same head.\n\t */\n\tpublic List getSiblingFilledDeps() {\n\t\tList filledDeps = getFilledDeps();\n\t\tif (filledDeps.isEmpty()) return Collections.emptyList();\n\t\tList retval = new ArrayList(5);\n\t\tSign[] inputs = _history.getInputs();\n for (int i = 0; i < inputs.length; i++) {\n \tinputs[i].addSiblingFilledDeps(retval, filledDeps);\n }\n\t\treturn retval;\n\t}\n\t\n\t// recursively adds sibling filled deps until lex items reached or \n\t// sibs with different heads found\n\tprivate void addSiblingFilledDeps(List retval, List filledDeps) {\n\t\tif (isLexical()) return;\n\t\tList candDeps = getFilledDeps();\n\t\tif (!candDeps.isEmpty()) {\n\t\t\tList sibs = LexDependency.filterSameHead(candDeps, filledDeps);\n\t\t\tif (sibs.isEmpty()) return;\n\t\t\tretval.addAll(sibs);\n\t\t\t\n\t\t}\n\t\tSign[] inputs = _history.getInputs();\n for (int i = 0; i < inputs.length; i++) {\n \tinputs[i].addSiblingFilledDeps(retval, filledDeps);\n }\n\t}\n\n\t/**\n\t * Returns the descendant sign headed by the given dependent \n\t * by recursing through the input signs as long as the head remains \n\t * the same as the given head; otherwise returns null.\n\t */\n\tpublic Sign getSignHeadedByDep(LexDependency lexdep) {\n\t\t// check same head\n\t\tif (!isLexical() && _lexHead == lexdep.lexHead) {\n\t\t\tSign[] inputs = _history.getInputs();\n\t for (int i = 0; i < inputs.length; i++) {\n\t \t\t// check for match\n\t \tif (inputs[i]._lexHead == lexdep.lexDep) return inputs[i]; // found it\n\t \t// otherwise recurse\n\t \tSign retval = inputs[i].getSignHeadedByDep(lexdep);\n\t \tif (retval != null) return retval;\n\t }\n\t\t}\n\t\t// otherwise not found\n\t\treturn null;\n\t}\n\t\n\t\n /** Tests serialization of simple types, including resolution. */\n public void debugSerialization() throws IOException, ClassNotFoundException {\n // test serialization\n \tString filename = \"tmp.ser\";\n \tObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(filename));\n \tSystem.out.println(\"Writing this: \" + this);\n \tSystem.out.println(this.getDerivationHistory());\n \tout.writeObject(this);\n \tout.close();\n \tObjectInputStream in = new ObjectInputStream(new FileInputStream(filename));\n \tSystem.out.print(\"Reading sign: \");\n \tSign sign = (Sign) in.readObject();\n \tSystem.out.println(sign);\n \tSystem.out.println(sign.getDerivationHistory());\n \tin.close();\n \t// test identity and equality\n \tSystem.out.println(\"this == sign?: \" + (this == sign));\n \tSystem.out.println(\"this.equals(sign)?: \" + (this.equals(sign)));\n }\n}\n" }, { "path": "src/opennlp/ccg/synsem/SignHash.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-5 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\nimport gnu.trove.*;\nimport java.util.*;\n\nimport opennlp.ccg.lexicon.Word;\n\n/**\n * A set of signs, unique up to surface words.\n * Signs with lower derivational complexity are kept during insertion.\n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.13 $, $Date: 2009/12/21 02:15:44 $\n */\npublic class SignHash extends THashSet {\n\n\tprivate static final long serialVersionUID = 1L;\n\t\n\t/** Hashing strategy that uses Sign's surfaceWordHashCode and surfaceWordEquals methods. */\n protected static TObjectHashingStrategy surfaceWordHashingStrategy = new TObjectHashingStrategy() {\n\t\tprivate static final long serialVersionUID = 1L;\n\t\tpublic int computeHashCode(java.lang.Object o) {\n return ((Sign)o).surfaceWordHashCode();\n }\n public boolean equals(java.lang.Object o1, java.lang.Object o2) {\n return ((Sign)o1).surfaceWordEquals((Sign)o2);\n }\n };\n\n /** Default constructor. */\n public SignHash() { super(surfaceWordHashingStrategy); }\n\n /**\n * Constructor which adds one sign.\n */\n public SignHash(Sign sign) {\n this(); insert(sign);\n }\n\n /**\n * Constructor which adds a collection of signs.\n */\n public SignHash(Collection c) {\n this();\n for (Sign s : c) insert(s);\n }\n \n /**\n * Returns this as a set of signs.\n */\n @SuppressWarnings(\"unchecked\")\n\tpublic Set asSignSet() { return (Set) this; }\n\n /**\n * Adds a sign, keeping the one with lower derivational complexity \n * if there is an equivalent one there already; returns the old \n * sign if it was displaced, the new sign if there was no equivalent \n * old sign, or null if the sign was not actually added.\n */\n public Sign insert(Sign sign) {\n int pos = index(sign);\n if (pos >= 0) {\n Sign oldSign = (Sign) _set[pos];\n if (oldSign == sign) return null;\n if (sign.getDerivationHistory().compareTo(oldSign.getDerivationHistory()) < 0) {\n \t_set[pos] = sign; return oldSign;\n }\n else return null;\n }\n else {\n \tadd(sign); return sign;\n }\n }\n \n /** Returns the signs sorted by their words lexicographically. */\n public List getSignsSorted() {\n \tArrayList retval = new ArrayList(asSignSet());\n \tCollections.sort(retval, signComparator); \n \treturn retval;\n }\n \n /** Comparator for signs to provide a persistent ordering. */\n public static final Comparator signComparator = new Comparator() {\n\t\tpublic int compare(Sign sign1, Sign sign2) {\n\t\t\treturn compareTo(sign1, sign2);\n\t\t}\n };\n\n /** Compares signs by their derivation complexity, lists of words, then (somewhat desperately) cat hash codes. */\n public static int compareTo(Sign sign1, Sign sign2) {\n \tint cmp = 0;\n \tcmp = sign1.getDerivationHistory().compareTo(sign2.getDerivationHistory());\n \tif (cmp != 0) return cmp;\n \tList words1 = sign1.getWords(); \n \tList words2 = sign2.getWords();\n \tcmp = compareTo(words1, words2);\n \tif (cmp != 0) return cmp;\n \t// TODO: implement compareTo method on categories\n \tint h1 = sign1.getCategory().hashCode();\n \tint h2 = sign2.getCategory().hashCode();\n \tif (h1 < h2) return -1;\n \tif (h1 > h2) return 1;\n \treturn 0;\n }\n \n /** Compares lists of words lexicographically. */\n public static int compareTo(List words1, List words2) {\n \tint i=0;\n \twhile (i < words1.size() || i < words2.size()) {\n \t\tif (i == words1.size()) return -1;\n \t\tif (i == words2.size()) return 1;\n \t\tWord w1 = words1.get(i); Word w2 = words2.get(i);\n \t\tint cmp = w1.compareTo(w2);\n \t\tif (cmp != 0) return cmp;\n \t\ti++;\n \t}\n \treturn 0;\n }\n}\n" }, { "path": "src/opennlp/ccg/synsem/SignScorer.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-4 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\nimport java.util.Random;\n\n/**\n * Interface for sign scoring models.\n *\n * @author Michael White\n * @version $Revision: 1.2 $, $Date: 2008/11/09 02:59:49 $\n */\npublic interface SignScorer\n{\n /** \n * Returns a score for the given sign and completeness flag, where higher \n * numbers are better than lower numbers.\n * When normalized, returns a score between 0 (worst) and 1 (best).\n */\n public double score(Sign sign, boolean complete);\n\n /** A scorer that returns 0 for all signs. */\n public static SignScorer nullScorer = new SignScorer() { \n public double score(Sign sign, boolean complete) { return 0; }\n };\n\n /** A scorer that returns a random number in [0,1] for all signs. */\n public static SignScorer randomScorer = new SignScorer() { \n \tRandom random = new Random();\n public double score(Sign sign, boolean complete) { return random.nextDouble(); }\n };\n}\n\n" }, { "path": "src/opennlp/ccg/synsem/Slash.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-7 Jason Baldridge, Gann Bierner and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\nimport java.io.Serializable;\nimport gnu.trove.*;\nimport org.jdom.Element;\n\nimport opennlp.ccg.unify.*;\n\n/**\n * A categorial slash which has an optional mode associated with it.\n * \n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.10 $, $Date: 2009/11/28 03:39:27 $\n */\npublic final class Slash implements Unifiable, Mutable, Serializable {\n\n\tprivate static final long serialVersionUID = -1935688863458012637L;\n\n\tpublic static final byte L = 0;\n\n\tpublic static final byte B = 1;\n\n\tpublic static final byte R = 2;\n\n\tpublic static final byte INERT_OR_ACTIVE = 0;\n\n\tpublic static final byte ACTIVE = 1;\n\n\tpublic static final byte INERT = 2;\n\n\tprivate final byte _dir;\n\n\tprivate final Modality _modality;\n\n\tprivate byte _ability = INERT_OR_ACTIVE;\n\t\n\tprivate boolean _modifier = false;\n\n\tprivate boolean _harmonicCompositionResult = false;\n\t\n\tpublic Slash(Element el) {\n\t\tString d = el.getAttributeValue(\"dir\");\n\t\tif (d == null)\n\t\t\td = el.getAttributeValue(\"d\");\n\t\tif (d == null) {\n\t\t\td = \"|\";\n\t\t}\n\t\t_dir = encode(d.charAt(0));\n\n\t\tString m = el.getAttributeValue(\"mode\");\n\t\tif (m == null)\n\t\t\tm = el.getAttributeValue(\"m\");\n\t\tif (m != null) {\n\t\t\t_modality = new SlashMode(m);\n\t\t} else {\n\t\t\tString vm = el.getAttributeValue(\"varmodality\");\n\t\t\tif (vm == null)\n\t\t\t\tvm = el.getAttributeValue(\"varModality\");\n\t\t\tif (vm != null) {\n\t\t\t\t_modality = new VarModality(vm);\n\t\t\t} else {\n\t\t\t\t_modality = new SlashMode();\n\t\t\t}\n\t\t}\n\n\t\tString ability = el.getAttributeValue(\"ability\");\n\t\tif (null != ability) {\n\t\t\tsetAbility(ability);\n\t\t}\n\t}\n\n\tpublic Slash() {\n\t\tthis('|');\n\t}\n\n\tpublic Slash(char sd) {\n\t\t_dir = encode(sd);\n\t\t_modality = new SlashMode();\n\t}\n\n\tpublic Slash(char sd, String md) {\n\t\t_dir = encode(sd);\n\t\t_modality = new SlashMode(md);\n\t}\n\n\tpublic Slash(char sd, Modality md) {\n\t\t_dir = encode(sd);\n\t\t_modality = md;\n\t}\n\n\tprivate Slash(byte d, Modality m, byte a) {\n\t\t_dir = d;\n\t\t_modality = m;\n\t\t_ability = a;\n\t}\n\t\n\tpublic Element toXml() {\n \tElement retval = new Element(\"slash\");\n \tretval.setAttribute(\"dir\", encode());\n \tString ability = decodeAbility();\n \tif (_modality instanceof SlashMode) {\n \t\tString mode = _modality.toString();\n \t\tif (!mode.equals(\".\")) retval.setAttribute(\"mode\", mode);\n \t}\n \telse if (_modality instanceof VarModality)\n \t\tretval.setAttribute(\"varmodality\", ((VarModality) _modality).name());\n \tif (ability != null) retval.setAttribute(\"ability\", ability);\n \treturn retval;\n\t}\n\n\tpublic Slash copy() {\n\t\tSlash retval = new Slash(_dir, (Modality) _modality.copy(), _ability);\n\t\tretval._modifier = _modifier;\n\t\tretval._harmonicCompositionResult = _harmonicCompositionResult;\n\t\treturn retval;\n\t}\n\n\tpublic boolean occurs(Variable v) {\n\t\treturn _modality.occurs(v);\n\t}\n\n\tpublic void deepMap(ModFcn mf) {\n\t\tmf.modify(this);\n\t}\n\n\tpublic boolean isActive() {\n\t\treturn _ability == ACTIVE || _ability == INERT_OR_ACTIVE;\n\t}\n\n\tpublic boolean setAbility(String ability) {\n\t\tbyte newAbility;\n\t\tif (ability.equals(\"inert\")) {\n\t\t\tnewAbility = INERT;\n\t\t} else if (ability.equals(\"active\")) {\n\t\t\tnewAbility = ACTIVE;\n\t\t} else {\n\t\t\tnewAbility = INERT_OR_ACTIVE;\n\t\t}\n\t\tif (abilitiesMatch(_ability, newAbility)) {\n\t\t\t_ability = newAbility;\n\t\t\treturn true;\n\t\t} else {\n\t\t\treturn false;\n\t\t}\n\t}\n\t\n\t/** Returns a string for the ability or null if not set. */\n\tpublic String decodeAbility() {\n\t\tif (_ability == INERT) return \"inert\";\n\t\telse if (_ability == ACTIVE) return \"active\";\n\t\telse return null;\n\t}\n\t\n\t/** Returns whether this cat is a modifier cat (defaults to false). */\n\tpublic boolean isModifier() { return _modifier; }\n\t\n\t/** Sets whether this cat is a modifier cat. */\n\t// NB: Might want to change this allow lex overrides of defaults\n\tpublic void setModifier(boolean modifier) { _modifier = modifier; }\n\t\n\t/** Returns whether this arg has resulted from harmonic composition. */\n\tpublic boolean isHarmonicCompositionResult() { return _harmonicCompositionResult; }\n\t\n\t/** Sets whether this arg has resulted from harmonic composition. */\n\tpublic void setHarmonicCompositionResult(boolean harmonicResult) { _harmonicCompositionResult = harmonicResult; }\n\t\n\n\tpublic void unifyCheck(Object u) throws UnifyFailure {\n\t\tif (u instanceof Slash) {\n\t\t\tif (!abilitiesMatch(_ability, ((Slash) u)._ability)) {\n\t\t\t\tthrow new UnifyFailure();\n\t\t\t}\n\t\t\tif (!directionsMatch(_dir, ((Slash) u)._dir)) {\n\t\t\t\tthrow new UnifyFailure();\n\t\t\t}\n\t\t\t_modality.unifyCheck(((Slash) u)._modality);\n\t\t} else {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\t}\n\n\tpublic Object unify(Object u, Substitution sub) throws UnifyFailure {\n\n\t\tif (u instanceof Slash) {\n\t\t\tSlash s2 = (Slash) u;\n\t\t\tbyte newAbility = _ability;\n\t\t\tif (_ability == INERT_OR_ACTIVE) {\n\t\t\t\tnewAbility = s2._ability;\n\t\t\t} else if (s2._ability == INERT_OR_ACTIVE) {\n\t\t\t\tnewAbility = _ability;\n\t\t\t} else if (_ability != s2._ability) {\n\t\t\t\tthrow new UnifyFailure();\n\t\t\t}\n\n\t\t\tbyte newDir = _dir;\n\t\t\tif (_dir == B) {\n\t\t\t\tnewDir = s2._dir;\n\t\t\t} else if (s2._dir == B) {\n\t\t\t\tnewDir = _dir;\n\t\t\t} else if (_dir != s2._dir) {\n\t\t\t\tthrow new UnifyFailure();\n\t\t\t}\n\n\t\t\tModality newModality = (Modality) _modality.unify(((Slash) u)._modality, sub);\n\t\t\tSlash retval = new Slash(newDir, newModality, newAbility);\n\t\t\tretval._modifier = _modifier;\n\t\t\treturn retval;\n\t\t} else {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\n\t}\n\n\tpublic Object fill(Substitution sub) throws UnifyFailure {\n\t\tSlash retval = new Slash(_dir, (Modality) _modality.fill(sub), _ability);\n\t\tretval._modifier = _modifier;\n\t\treturn retval;\n\t}\n\n//\tpublic boolean equals(Slash s) {\n//\t\treturn directionsMatch(_dir, s._dir);\n//\t}\n\n\tpublic boolean sameDirAsModality() {\n\t\treturn directionsMatch(_dir, _modality.getDirection());\n\t}\n\n\tprivate static byte encode(char sd) {\n\t\tswitch (sd) {\n\t\tcase '/':\n\t\t\treturn R;\n\t\tcase '\\\\':\n\t\t\treturn L;\n\t\tdefault:\n\t\t\treturn B;\n\t\t}\n\t}\n\n\tpublic static boolean directionsMatch(byte s1, byte s2) {\n\t\tif (s1 == B || s2 == B) {\n\t\t\treturn true;\n\t\t} else {\n\t\t\treturn s1 == s2;\n\t\t}\n\t}\n\n\tprivate static boolean abilitiesMatch(byte ab1, byte ab2) {\n\t\tif (ab1 == INERT_OR_ACTIVE || ab2 == INERT_OR_ACTIVE) {\n\t\t\treturn true;\n\t\t} else {\n\t\t\treturn ab1 == ab2;\n\t\t}\n\t}\n\n\t/**\n\t * Returns a hash code based on the direction, ability and modality.\n\t */\n\tpublic int hashCode() {\n\t\treturn 31 * _dir + 7 * _ability + _modality.hashCode();\n\t}\n\n\t/**\n\t * Returns whether this slash equals the given object \n\t * based on the direction, ability and modality.\n\t */\n\tpublic boolean equals(Object obj) {\n\t\tif (this == obj) return true;\n\t\tif (obj.getClass() != this.getClass()) return false;\n\t\tSlash s = (Slash) obj;\n\t\tif (_dir != s._dir ||_ability != s._ability) return false;\n\t\treturn _modality.equals(s._modality);\n\t}\n\n\t/**\n\t * Returns a hash code using the given map from vars to ints.\n\t */\n\tpublic int hashCode(TObjectIntHashMap varMap) {\n\t\tint retval = 31 * _dir + 7 * _ability;\n\t\tif (_modality instanceof Variable) retval += ((Variable)_modality).hashCode(varMap);\n\t\telse retval += _modality.hashCode();\n\t\treturn retval;\n\t}\n\n\t/**\n\t * Returns whether this slash equals the given object up to variable names,\n\t * using the given maps from vars to ints.\n\t */\n\tpublic boolean equals(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2) {\n\t\tif (this == obj) return true;\n\t\tif (obj.getClass() != this.getClass()) return false;\n\t\tSlash s = (Slash) obj;\n\t\tif (_dir != s._dir || _ability != s._ability) return false;\n\t\tif (_modality instanceof Variable) \n\t\t\treturn ((Variable)_modality).equals(s._modality, varMap, varMap2);\n\t\telse \n\t\t\treturn _modality.equals(s._modality);\n\t}\n\n\t// string for showing ability\n\tprivate String abilityStr() {\n\t\tif (_ability == ACTIVE)\n\t\t\treturn \"@\";\n\t\telse if (_ability == INERT)\n\t\t\treturn \"!\";\n\t\telse\n\t\t\treturn \"\";\n\t}\n\n\tpublic String toString() {\n\t\tswitch (_dir) {\n\t\tcase R:\n\t\t\treturn \"/\" + abilityStr() + _modality.toString(R);\n\t\tcase L:\n\t\t\treturn \"\\\\\" + abilityStr() + _modality.toString(L);\n\t\tdefault:\n\t\t\treturn \"|\" + abilityStr() + _modality;\n\t\t}\n\t}\n\n\t/**\n\t * Returns the direction for this slash as a string.\n\t */\n\tpublic String encode() {\n\t\tswitch (_dir) {\n\t\tcase R:\n\t\t\treturn \"/\";\n\t\tcase L:\n\t\t\treturn \"\\\\\";\n\t\tdefault:\n\t\t\treturn \"|\";\n\t\t}\n\t}\n\n\t/**\n\t * Returns the supertag for this slash.\n\t */\n\tpublic String getSupertag() {\n\t\treturn encode();\n\t}\n\n\t/**\n\t * Returns a TeX-formatted string representation for this slash.\n\t */\n\tpublic String toTeX() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\tString sup = \"\\\\sups\";\n\t\tString sub = \"\\\\subs\";\n\t\tString modTeX = null;\n\t\tswitch (_dir) {\n\t\tcase R:\n\t\t\tsb.append(\"/ \");\n\t\t\tsup = \"\\\\supsb\";\n\t\t\tsub = \"\\\\subsa\";\n\t\t\tmodTeX = _modality.toTeX(R);\n\t\t\tbreak;\n\t\tcase L:\n\t\t\tsb.append(\"\\\\bs \");\n\t\t\tsub = \"\\\\subsb\";\n\t\t\tsup = \"\\\\supsa\";\n\t\t\tmodTeX = _modality.toTeX(L);\n\t\t\tbreak;\n\t\tdefault:\n\t\t\tsb.append(\"| \");\n\t\t\tsub = \"\\\\subs\";\n\t\t\tsup = \"\\\\sups\";\n\t\t\tmodTeX = _modality.toTeX();\n\t\t\tbreak;\n\t\t}\n\n\t\tif (_ability == ACTIVE)\n\t\t\tsb.append(sup).append(\"{\").append(\"+\").append(\"} \");\n\t\tif (_ability == INERT)\n\t\t\tsb.append(sup).append(\"{\").append(\"-\").append(\"} \");\n\t\tif ((modTeX != \"\") && (_ability == ACTIVE)) {\n\t\t\tif ((_dir == R))\n\t\t\t\tsb.append(\"\\\\hspace{-1.45mm} \");\n\t\t\telse if (_dir == L)\n\t\t\t\tsb.append(\"\\\\hspace{-0.50mm} \");\n\t\t}\n\t\tif (modTeX != \"\")\n\t\t\tsb.append(sub).append(\"{\").append(modTeX).append(\"} \");\n\t\treturn sb.toString();\n\t}\n}\n" }, { "path": "src/opennlp/ccg/synsem/SlashMode.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\nimport java.io.Serializable;\nimport org.jdom.Element;\n\nimport opennlp.ccg.unify.*;\n\n/**\n * A mode that can decorate a categorial slash.\n * \n * @author Jason Baldridge\n * @version $Revision: 1.5 $, $Date: 2009/07/17 04:23:30 $\n */\npublic final class SlashMode implements Modality, Serializable {\n\n\tprivate static final long serialVersionUID = -2387797559890373347L;\n\n\tpublic static final byte All = 0;\n\n\tpublic static final byte ApplicationOnly = 1;\n\n\tpublic static final byte Associative = 2;\n\n\tpublic static final byte Permutative = 3;\n\n\tpublic static final byte PermutativeRight = 4;\n\n\tpublic static final byte PermutativeLeft = 5;\n\n\tpublic static final byte APRight = 6;\n\n\tpublic static final byte APLeft = 7;\n\n\tprivate byte _mode;\n\n\tpublic SlashMode(Element el) {\n\t\tString m = el.getAttributeValue(\"mode\");\n\t\tif (m == null)\n\t\t\tm = el.getAttributeValue(\"m\");\n\t\tif (m == null) {\n\t\t\tm = \".\";\n\t\t}\n\n\t\t_mode = byteVal(m);\n\t}\n\n\tpublic SlashMode() {\n\t\tthis(\".\");\n\t}\n\n\tpublic SlashMode(String m) {\n\t\t_mode = byteVal(m);\n\t}\n\n\tprivate SlashMode(byte m) {\n\t\t_mode = m;\n\t}\n\n\tpublic Object copy() {\n\t\treturn new SlashMode(_mode);\n\t}\n\n\t/** Returns a hash code based on the mode. */\n\tpublic int hashCode() { return 31 * _mode; }\n\t\n\t/** Returns whether this slash mode equals the given object based on the mode. */\n\tpublic boolean equals(Object obj) {\n\t\tif (this == obj) return true;\n\t\tif (obj.getClass() != this.getClass()) return false;\n\t\tSlashMode m = (SlashMode) obj;\n\t\treturn _mode == m._mode;\n\t}\n\t\n//\tpublic boolean equals(SlashMode m) {\n//\t\treturn _mode == m._mode;\n//\t}\n\n\tpublic boolean occurs(Variable var) {\n\t\treturn false;\n\t}\n\n\tpublic void unifyCheck(Object o) throws UnifyFailure {\n\t\tif (!(o instanceof VarModality || (o instanceof SlashMode && modesMatch(\n\t\t\t\t_mode, ((SlashMode) o)._mode)))) {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\t}\n\n\tpublic Object unify(Object o, Substitution sub) throws UnifyFailure {\n\t\tif (o instanceof VarModality) {\n\t\t\treturn ((VarModality) o).unify(this, sub);\n\t\t} else if (o instanceof SlashMode) {\n\t\t\tif (modesMatch(_mode, ((SlashMode) o)._mode)) {\n\t\t\t\treturn copy();\n\t\t\t} else {\n\t\t\t\tthrow new UnifyFailure();\n\t\t\t}\n\t\t} else {\n\t\t\tthrow new UnifyFailure();\n\t\t}\n\t}\n\n\tpublic Object fill(Substitution sub) throws UnifyFailure {\n\t\treturn copy();\n\t}\n\n\tpublic String toString(byte slashDir) {\n\t\tif (slashDir == getDirection()) {\n\t\t\tswitch (_mode) {\n\t\t\tcase PermutativeRight:\n\t\t\t\treturn \"x\";\n\t\t\tcase PermutativeLeft:\n\t\t\t\treturn \"x\";\n\t\t\tcase APRight:\n\t\t\t\treturn \"\";\n\t\t\tcase APLeft:\n\t\t\t\treturn \"\";\n\t\t\tdefault:\n\t\t\t\treturn toString();\n\t\t\t}\n\t\t} else {\n\t\t\treturn toString();\n\t\t}\n\t}\n\n\tpublic String toString() {\n\t\tswitch (_mode) {\n\t\tcase All:\n\t\t\treturn \".\";\n\t\tcase ApplicationOnly:\n\t\t\treturn \"*\";\n\t\tcase Associative:\n\t\t\treturn \"^\";\n\t\tcase Permutative:\n\t\t\treturn \"x\";\n\t\tcase PermutativeRight:\n\t\t\treturn \"x>\";\n\t\tcase PermutativeLeft:\n\t\t\treturn \"\";\n\t\tcase APLeft:\n\t\t\treturn \"<\";\n\t\tdefault:\n\t\t\treturn \".\";\n\t\t}\n\t}\n\n\tpublic String toTeX(byte slashDir) {\n\t\tif (slashDir == getDirection()) {\n\t\t\tswitch (_mode) {\n\t\t\tcase PermutativeRight:\n\t\t\t\treturn \"x\";\n\t\t\tcase PermutativeLeft:\n\t\t\t\treturn \"x\";\n\t\t\tcase APRight:\n\t\t\t\treturn \"\";\n\t\t\tcase APLeft:\n\t\t\t\treturn \"\";\n\t\t\tdefault:\n\t\t\t\treturn toTeX();\n\t\t\t}\n\t\t} else {\n\t\t\treturn toTeX();\n\t\t}\n\t}\n\n\tpublic String toTeX() {\n\t\tswitch (_mode) {\n\t\tcase All:\n\t\t\treturn \".\";\n\t\tcase ApplicationOnly:\n\t\t\treturn \"*\";\n\t\tcase Associative:\n\t\t\treturn \"\\\\diamond\";\n\t\tcase Permutative:\n\t\t\treturn \"x\";\n\t\tcase PermutativeRight:\n\t\t\treturn \"x>\";\n\t\tcase PermutativeLeft:\n\t\t\treturn \"\";\n\t\tcase APLeft:\n\t\t\treturn \"<\";\n\t\tdefault:\n\t\t\treturn \".\";\n\t\t}\n\t}\n\n\tprivate static byte byteVal(String m) {\n\t\tif (m.equals(\".\")) {\n\t\t\treturn All;\n\t\t} else if (m.equals(\">\")) {\n\t\t\treturn APRight;\n\t\t} else if (m.equals(\"<\")) {\n\t\t\treturn APLeft;\n\t\t} else if (m.equals(\"*\")) {\n\t\t\treturn ApplicationOnly;\n\t\t} else if (m.equals(\"^\")) {\n\t\t\treturn Associative;\n\t\t} else if (m.equals(\"x\")) {\n\t\t\treturn Permutative;\n\t\t} else if (m.equals(\"x>\")) {\n\t\t\treturn PermutativeRight;\n\t\t} else if (m.equals(\" \" + doMatch);\n\t\treturn doMatch;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/synsem/SyntacticFeatureExtractor.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2009 Michael White\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.synsem;\r\n\r\nimport java.util.*;\r\n\r\nimport opennlp.ccg.perceptron.*;\r\nimport opennlp.ccg.util.TrieMap;\r\nimport opennlp.ccg.lexicon.Word;\r\n\r\n/** \r\n * A class for extracting generic features from derivations,\r\n * inspired by those in the C&C-07 normal form model.\r\n * \r\n * Features are extracted lazily for efficiency.\r\n * \r\n * @author Michael White\r\n * @version $Revision: 1.10 $, $Date: 2011/11/08 14:58:15 $\r\n */ \r\npublic class SyntacticFeatureExtractor implements FeatureExtractor {\r\n\t\r\n\t/** Feature prefix constant: \"syn\". */\r\n\tpublic static final String PREFIX = \"syn\";\r\n\t\r\n\t/** Feature map wrapper, for unique retrieval from a sign's data objects. */\r\n\tpublic static class FeatureMapWrapper {\r\n\t\tpublic FeatureMap featureMap;\r\n\t\tpublic FeatureMapWrapper(FeatureMap featureMap) { this.featureMap = featureMap; }\r\n\t}\r\n\t\r\n\t/** Flag for whether to include distance features (defaults to true). */\r\n\tpublic boolean includeDistFeats = true;\r\n\r\n\t/** The alphabet. */\r\n\tprotected Alphabet alphabet = null;\r\n\r\n\t/** Current feature map. */\r\n\tprotected FeatureMap currentMap = null;\r\n\t\r\n\t/** Current sign (for extracting features). */\r\n\tprotected Sign currentSign = null;\r\n\t\r\n\t/** Current input signs (for extracting features). */\r\n\tprotected Sign[] currentInputs = null;\r\n\t\r\n\t/** Current sibling (for extracting features). */\r\n\tprotected Sign currentSibling = null;\r\n\t\r\n\t/** Current words (for extracting features). */\r\n\tprotected List currentWords = null;\r\n\t\r\n\t/** Current head index (for extracting features). */\r\n\tprotected int currentHeadIndex = -1;\r\n\r\n\t/** Current sibling head index (for extracting features). */\r\n\tprotected int currentSibHeadIndex = -1;\r\n\t\r\n\t/** Current distance in words (for extracting features). */\r\n\tprotected String currentDistW = null;\r\n\r\n\t/** Current distance in puncts (for extracting features). */\r\n\tprotected String currentDistP = null;\r\n\r\n\t/** Current distance in verbs (for extracting features). */\r\n\tprotected String currentDistV = null;\r\n\r\n\t/** Lexical feature extractors. */\r\n\tprotected List>> lexExtractors = new ArrayList>>();\r\n\t\r\n\t/** Rule feature extractors. */\r\n\tprotected List>> unaryRuleExtractors = new ArrayList>>();\r\n\t\r\n\t/** Binary rule feature extractors. */\r\n\tprotected List>> binaryRuleExtractors = new ArrayList>>();\r\n\t\r\n\t/** Distance feature extractors. */\r\n\tprotected List>> distExtractors = new ArrayList>>();\r\n\t\r\n\t\r\n\t/** Constructor. */\r\n\tpublic SyntacticFeatureExtractor() {\r\n\t\t// init lazy feature extractors\r\n\t\tlexExtractors.add(lexcat_word());\r\n\t\tlexExtractors.add(lexcat_pos());\r\n\t\tunaryRuleExtractors.add(unary_rule());\r\n\t\tunaryRuleExtractors.add(unary_rule_word());\r\n\t\tunaryRuleExtractors.add(unary_rule_pos());\r\n\t\tbinaryRuleExtractors.add(binary_rule());\r\n\t\tbinaryRuleExtractors.add(binary_rule_word());\r\n\t\tbinaryRuleExtractors.add(binary_rule_pos());\r\n\t\tbinaryRuleExtractors.add(rule_word_word());\r\n\t\tbinaryRuleExtractors.add(rule_word_pos());\r\n\t\tbinaryRuleExtractors.add(rule_pos_word());\r\n\t\tbinaryRuleExtractors.add(rule_pos_pos());\r\n\t\tdistExtractors.add(rule_word_dist());\r\n\t\tdistExtractors.add(rule_pos_dist());\r\n\t\tdistExtractors.add(rule_word_dist_puncts());\r\n\t\tdistExtractors.add(rule_pos_dist_puncts());\r\n\t\tdistExtractors.add(rule_word_dist_verbs());\r\n\t\tdistExtractors.add(rule_pos_dist_verbs());\r\n\t}\r\n\t\r\n\t/** Sets the alphabet. */\r\n\tpublic void setAlphabet(Alphabet alphabet) {\r\n\t\tthis.alphabet = alphabet;\r\n\t}\r\n\t\r\n\t\r\n\t/** Returns the features for the given sign and completeness flag. */\r\n\tpublic FeatureVector extractFeatures(Sign sign, boolean complete) {\r\n\t\taddFeatures(sign, complete);\r\n\t\treturn getFeatureMap(sign);\r\n\t}\r\n\t\r\n\t/** Recursively adds features to the feature map for the given sign, if not already present. */\r\n\tprotected void addFeatures(Sign sign, boolean complete) {\r\n\t\t// check for existing map, otherwise make one\r\n\t\tif (getFeatureMap(sign) != null) return;\r\n\t\t// lex case\r\n\t\tif (sign.isLexical()) {\r\n\t\t\tcurrentSign = sign;\r\n\t\t\tcurrentMap = new FeatureMap();\r\n\t\t\tinc(lexExtractors);\r\n\t\t}\r\n\t\t// non-terminal\r\n\t\telse {\r\n\t\t\tSign[] inputs = sign.getDerivationHistory().getInputs();\r\n\t\t\t// first recurse\r\n\t\t\tfor (Sign child : inputs) addFeatures(child, false);\r\n\t\t\t// use input maps in making current map\r\n\t\t\tcurrentSign = sign;\r\n\t\t\tcurrentInputs = inputs;\r\n\t\t\tif (inputs.length == 1) {\r\n\t\t\t\tcurrentMap = new FeatureMap(getFeatureMap(inputs[0]));\r\n\t\t\t\tinc(unaryRuleExtractors);\r\n\t\t\t}\r\n\t\t\telse if (inputs.length == 2) {\r\n\t\t\t\tcurrentMap = new FeatureMap(getFeatureMap(inputs[0]), getFeatureMap(inputs[1]));\r\n\t\t\t\tcurrentSibling = sibling(sign, inputs);\r\n\t\t\t\tinc(binaryRuleExtractors);\r\n\t\t\t\t// dist feats\r\n\t\t\t\tif (includeDistFeats) {\r\n\t\t\t\t\tcurrentWords = null; // get words and head indices lazily\r\n\t\t\t\t\tcurrentDistW = null; currentDistP = null; currentDistV = null; // also reset current distances\r\n\t\t\t\t\tinc(distExtractors);\r\n\t\t\t\t}\r\n\t\t\t}\r\n\t\t}\r\n\t\t// store it\r\n\t\tstoreFeatureMap(sign);\r\n\t}\r\n\t\r\n\t/** Stores the current feature map as a data object in the given sign. */\r\n\tprotected void storeFeatureMap(Sign sign) {\r\n\t\tsign.addData(new FeatureMapWrapper(currentMap));\r\n\t}\r\n\t\r\n\t/** Returns the feature map for this extractor from the given sign (null if none). */\r\n\tprotected FeatureMap getFeatureMap(Sign sign) {\r\n\t\tFeatureMapWrapper fmw = (FeatureMapWrapper)sign.getData(FeatureMapWrapper.class);\r\n\t\treturn (fmw != null) ? fmw.featureMap : null;\r\n\t}\r\n\t\r\n\t/**\r\n\t * Increments the count of the given features, if relevant.\r\n\t */\r\n\tprotected void inc(List>> extractors) {\r\n\t\tfor (List> lazyExtractor : extractors) {\r\n\t\t\tAlphabet.Feature f = alphabet.indexLazy(lazyExtractor);\r\n\t\t\tif (f != null) currentMap.inc(f);\r\n\t\t}\r\n\t}\r\n\r\n\t/** Returns the sibling sign from among the two inputs. */\r\n\tprotected Sign sibling(Sign sign, Sign[] inputs) {\r\n\t\tif (sign.getLexHead() == inputs[0].getLexHead()) return inputs[1];\r\n\t\telse return inputs[0];\r\n\t}\r\n\t\r\n\t/** Sets the current words, if null, along with head indices. */\r\n\tprotected void setCurrentWords() {\r\n\t\tif (currentWords != null) return;\r\n\t\tcurrentWords = currentSign.getWords();\r\n\t\tWord head = currentSign.getLexHead().getWords().get(0);\r\n\t\tWord sibHead = currentSibling.getLexHead().getWords().get(0);\r\n\t\tcurrentHeadIndex = find(currentWords, head);\r\n\t\tcurrentSibHeadIndex = find(currentWords, sibHead);\r\n\t}\r\n\t\r\n\t/** Returns the index of the given word in the list, or -1 if not found. */\r\n\tprotected int find(List words, Word word) {\r\n\t\tint len = words.size();\r\n\t\tfor (int i=0; i < len; i++) {\r\n\t\t\tif (words.get(i) == word) return i;\r\n\t\t}\r\n\t\treturn -1;\r\n\t}\r\n\t\r\n\t/** Returns the distance in intervening words as 0w, 1w, 2w or 3w (for 3 or more). */\r\n\tprotected String distWords() {\r\n\t\tif (currentDistW != null) return currentDistW;\r\n\t\tsetCurrentWords();\r\n\t\tint dist = Math.abs(currentHeadIndex - currentSibHeadIndex) - 1;\r\n\t\tswitch (dist) {\r\n\t\t\tcase 0: return currentDistW = \"0w\";\r\n\t\t\tcase 1: return currentDistW = \"1w\";\r\n\t\t\tcase 2: return currentDistW = \"2w\";\r\n\t\t\tdefault: return currentDistW = \"3w\";\r\n\t\t}\r\n\t}\r\n\t\r\n\t/** Returns the distance in intervening punctuation marks as 0p, 1p, 2p or 3p (for 3 or more). */\r\n\tprotected String distPuncts() {\r\n\t\tif (currentDistP != null) return currentDistP;\r\n\t\tsetCurrentWords();\r\n\t\tint min = Math.min(currentHeadIndex, currentSibHeadIndex);\r\n\t\tint max = Math.max(currentHeadIndex, currentSibHeadIndex);\r\n\t\tint count = 0;\r\n\t\tfor (int i=min+1; i < max; i++) {\r\n\t\t\tWord w = currentWords.get(i); \r\n\t\t\tif (isPunct(w)) count++;\r\n\t\t}\r\n\t\tswitch (count) {\r\n\t\t\tcase 0: return currentDistP = \"0p\";\r\n\t\t\tcase 1: return currentDistP = \"1p\";\r\n\t\t\tcase 2: return currentDistP = \"2p\";\r\n\t\t\tdefault: return currentDistP = \"3p\";\r\n\t\t}\r\n\t}\r\n\t\r\n\t/**\r\n\t * Returns whether a word is a punctuation mark that typically signals sentence-internal complexity.\r\n\t * The default implementation tests for commas, dashes (--), semi-colons and colons.\r\n\t */\r\n\tprotected boolean isPunct(Word word) {\r\n\t\t// NB: in principle could use POS, but sometimes punctuation marks seem to end up with IN as the POS tag\r\n\t\tString form = word.getForm();\r\n\t\treturn (form == \",\" || form == \"--\" || form == \";\" || form == \":\");\r\n\t}\r\n\t\r\n\t/** Returns the distance in intervening verbs as 0v, 1v, or 2v (for 2 or more). */\r\n\tprotected String distVerbs() {\r\n\t\tif (currentDistV != null) return currentDistV;\r\n\t\tsetCurrentWords();\r\n\t\tint min = Math.min(currentHeadIndex, currentSibHeadIndex);\r\n\t\tint max = Math.max(currentHeadIndex, currentSibHeadIndex);\r\n\t\tint count = 0;\r\n\t\tfor (int i=min+1; i < max; i++) {\r\n\t\t\tWord w = currentWords.get(i); \r\n\t\t\tif (isVerb(w)) count++;\r\n\t\t}\r\n\t\tswitch (count) {\r\n\t\t\tcase 0: return currentDistV = \"0v\";\r\n\t\t\tcase 1: return currentDistV = \"1v\";\r\n\t\t\tdefault: return currentDistV = \"2v\";\r\n\t\t}\r\n\t}\r\n\t\r\n\t/**\r\n\t * Returns whether a word is a verb.\r\n\t * The default implementation tests for a POS tag beginning with V.\r\n\t */\r\n\tprotected boolean isVerb(Word word) {\r\n\t\tString pos = word.getPOS();\r\n\t\treturn (pos.startsWith(\"V\"));\r\n\t}\r\n\t\r\n\t\r\n\t// lex cat + word\r\n\tprivate List> lexcat_word() {\r\n\t\tList> retval = new ArrayList>(2);\r\n\t\tadd_supertag(retval);\r\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentSign.getWordForm(); }});\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\t// add prefix + supertag\r\n\tprivate void add_supertag(List> retval) {\r\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return PREFIX; }});\r\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentSign.getSupertag(); }});\r\n\t}\r\n\t\r\n\t// lex cat + pos\r\n\tprivate List> lexcat_pos() {\r\n\t\tList> retval = new ArrayList>(2);\r\n\t\tadd_supertag(retval);\r\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentSign.getPOS(); }});\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\t// rule\r\n\tprivate List> unary_rule() {\r\n\t\tList> retval = new ArrayList>(2);\r\n\t\tadd_unary_rule(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\tprivate void add_unary_rule(List> retval) {\r\n\t\tadd_supertag(retval);\r\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentInputs[0].getSupertag(); }});\r\n\t}\r\n\t\r\n\tprivate List> binary_rule() {\r\n\t\tList> retval = new ArrayList>(3);\r\n\t\tadd_binary_rule(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\tprivate void add_binary_rule(List> retval) {\r\n\t\tadd_unary_rule(retval);\r\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentInputs[1].getSupertag(); }});\r\n\t}\r\n\t\r\n\t// rule + head word\r\n\tprivate List> unary_rule_word() {\r\n\t\tList> retval = new ArrayList>(3);\r\n\t\tadd_unary_rule_word(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\tprivate void add_unary_rule_word(List> retval) {\r\n\t\tadd_unary_rule(retval);\r\n\t\tadd_lex_word(retval);\r\n\t}\r\n\t\r\n\tprivate void add_lex_word(List> retval) {\r\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentSign.getLexHead().getWordForm(); }});\r\n\t}\r\n\t\r\n\tprivate List> binary_rule_word() {\r\n\t\tList> retval = new ArrayList>(4);\r\n\t\tadd_binary_rule_word(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\tprivate void add_binary_rule_word(List> retval) {\r\n\t\tadd_binary_rule(retval);\r\n\t\tadd_lex_word(retval);\r\n\t}\r\n\t\r\n\t// rule + head pos\r\n\tprivate List> unary_rule_pos() {\r\n\t\tList> retval = new ArrayList>(3);\r\n\t\tadd_unary_rule_pos(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\tprivate void add_unary_rule_pos(List> retval) {\r\n\t\tadd_unary_rule(retval);\r\n\t\tadd_lex_pos(retval);\r\n\t}\r\n\t\r\n\tprivate void add_lex_pos(List> retval) {\r\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentSign.getLexHead().getPOS(); }});\r\n\t}\r\n\t\r\n\tprivate List> binary_rule_pos() {\r\n\t\tList> retval = new ArrayList>(4);\r\n\t\tadd_binary_rule_pos(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\tprivate void add_binary_rule_pos(List> retval) {\r\n\t\tadd_binary_rule(retval);\r\n\t\tadd_lex_pos(retval);\r\n\t}\r\n\t\r\n\t// rule + head word + sibling word\r\n\tprivate List> rule_word_word() {\r\n\t\tList> retval = new ArrayList>(5);\r\n\t\tadd_binary_rule_word(retval);\r\n\t\tadd_sibling_word(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\tprivate void add_sibling_word(List> retval) {\r\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentSibling.getLexHead().getWordForm(); }});\r\n\t}\r\n\t\r\n\t// rule + head word + sibling pos\r\n\tprivate List> rule_word_pos() {\r\n\t\tList> retval = new ArrayList>(5);\r\n\t\tadd_binary_rule_word(retval);\r\n\t\tadd_sibling_pos(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\tprivate void add_sibling_pos(List> retval) {\r\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return currentSibling.getLexHead().getPOS(); }});\r\n\t}\r\n\t\r\n\t// rule + head pos + sibling word\r\n\tprivate List> rule_pos_word() {\r\n\t\tList> retval = new ArrayList>(5);\r\n\t\tadd_binary_rule_pos(retval);\r\n\t\tadd_sibling_word(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\t// rule + head pos + sibling pos\r\n\tprivate List> rule_pos_pos() {\r\n\t\tList> retval = new ArrayList>(5);\r\n\t\tadd_binary_rule_pos(retval);\r\n\t\tadd_sibling_pos(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\t// rule + head word + dist\r\n\tprivate List> rule_word_dist() {\r\n\t\tList> retval = new ArrayList>(5);\r\n\t\tadd_binary_rule_word(retval);\r\n\t\tadd_dist_words(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\tprivate void add_dist_words(List> retval) {\r\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return distWords(); }});\r\n\t}\r\n\t\r\n\t// rule + head pos + dist\r\n\tprivate List> rule_pos_dist() {\r\n\t\tList> retval = new ArrayList>(5);\r\n\t\tadd_binary_rule_pos(retval);\r\n\t\tadd_dist_words(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\t// rule + head word + dist in puncts\r\n\tprivate List> rule_word_dist_puncts() {\r\n\t\tList> retval = new ArrayList>(5);\r\n\t\tadd_binary_rule_word(retval);\r\n\t\tadd_dist_puncts(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\tprivate void add_dist_puncts(List> retval) {\r\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return distPuncts(); }});\r\n\t}\r\n\t\r\n\t// rule + head pos + dist in puncts\r\n\tprivate List> rule_pos_dist_puncts() {\r\n\t\tList> retval = new ArrayList>(5);\r\n\t\tadd_binary_rule_pos(retval);\r\n\t\tadd_dist_puncts(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\t// rule + head word + dist in verbs\r\n\tprivate List> rule_word_dist_verbs() {\r\n\t\tList> retval = new ArrayList>(5);\r\n\t\tadd_binary_rule_word(retval);\r\n\t\tadd_dist_verbs(retval);\r\n\t\treturn retval;\r\n\t}\r\n\t\r\n\tprivate void add_dist_verbs(List> retval) {\r\n\t\tretval.add(new TrieMap.KeyExtractor(){public String getKey(){ return distVerbs(); }});\r\n\t}\r\n\t\r\n\t// rule + head pos + dist in verbs\r\n\tprivate List> rule_pos_dist_verbs() {\r\n\t\tList> retval = new ArrayList>(5);\r\n\t\tadd_binary_rule_pos(retval);\r\n\t\tadd_dist_verbs(retval);\r\n\t\treturn retval;\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccg/synsem/TargetCat.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.synsem;\n\n/**\n * A category that can be a target of a curried cat. Basically, atomcats or\n * variable cats.\n *\n * @author Jason Baldridge\n * @version $Revision: 1.1.1.1 $, $Date: 2003/02/28 18:02:13 $\n */\npublic interface TargetCat extends Category {\n\n}\n" }, { "path": "src/opennlp/ccg/synsem/VarModality.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-7 Jason Baldridge and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.synsem;\n\nimport java.io.Serializable;\nimport gnu.trove.TObjectIntHashMap;\n\nimport opennlp.ccg.unify.*;\n\n/**\n * A class for variables which can stand for slash modalities.\n *\n * @author Jason Baldridge\n * @author\t\tMichael White\n * @version $Revision: 1.4 $, $Date: 2009/07/17 04:23:30 $\n **/\npublic class VarModality implements Variable, Indexed, Mutable, Modality, Serializable {\n \n\tprivate static final long serialVersionUID = 7465785777802095802L;\n\t\n\tprotected final String _name;\n protected int _index;\n protected int _hashCode;\n \n private static int UNIQUE_STAMP = 0;\n \n public VarModality() {\n this(\"VM\"+UNIQUE_STAMP++);\n }\n \n public VarModality(String name) {\n this(name, 0);\n }\n\n protected VarModality(String name, int index) {\n _name = name;\n _index = index;\n _hashCode = _name.hashCode() + _index;\n }\n \n public String name() {\n return _name;\n }\n\n public Object copy() {\n return new VarModality(_name, _index);\n }\n \n public void deepMap(ModFcn mf) {\n mf.modify(this);\n }\n \n public int getIndex() {\n return _index;\n }\n\n public void setIndex(int index) {\n _hashCode += index - _index;\n _index = index;\n }\n\n public boolean occurs(Variable var) {\n return equals(var);\n }\n\n public int hashCode() {\n return _hashCode;\n }\n \n public boolean equals(Object o) {\n \tif (this == o) return true;\n if (!(o instanceof VarModality)) return false;\n VarModality vm = (VarModality) o;\n return _index == vm._index && _name.equals(vm._name);\n }\n\n /**\n\t * Returns a hash code using the given map from vars to ints.\n\t */\n\tpublic int hashCode(TObjectIntHashMap varMap) {\n\t\t// see if this already in map\n\t\tif (varMap.containsKey(this))\n\t\t\treturn varMap.get(this);\n\t\t// otherwise add it\n\t\tint next = varMap.size() + 1;\n\t\tvarMap.put(this, next);\n\t\treturn next;\n\t}\n \n /**\n\t * Returns whether this var equals the given object up to variable names,\n\t * using the given maps from vars to ints.\n\t */\n public boolean equals(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2) {\n if (this == obj) return true;\n if (obj.getClass() != this.getClass()) { return false; }\n VarModality vm = (VarModality) obj;\n if (varMap.get(this) != varMap2.get(vm)) return false;\n return true;\n }\n \n public void unifyCheck(Object o) throws UnifyFailure {\n if (!(o instanceof SlashMode || o instanceof VarModality)) {\n throw new UnifyFailure();\n }\n }\n \n public Object unify(Object u, Substitution sub) throws UnifyFailure {\n if (u instanceof SlashMode) {\n return sub.makeSubstitution(this, u); \n } else if (u instanceof VarModality) {\n VarModality var2 = (VarModality)u;\n Variable $var = new VarModality(_name+var2._name,\n UnifyControl.getUniqueVarIndex());\n \n sub.makeSubstitution(this, $var);\n sub.makeSubstitution(var2, $var);\n return $var;\n } else {\n throw new UnifyFailure();\n }\n }\n\n public Object fill(Substitution sub) throws UnifyFailure {\n Object val = sub.getValue(this);\n if (val != null) {\n return val;\n } else {\n return this;\n }\n }\n\n public byte getDirection() {\n return Slash.B;\n }\n \n public String toString(byte dir) { \n return toString();\n }\n\n public String toString() { \n return _name;\n }\n \n public String toTeX(byte dir) { \n return toTeX();\n }\n\n public String toTeX() { \n return _name;\n }\n}\n" }, { "path": "src/opennlp/ccg/test/CrossValidateRealizer.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.test;\n\nimport opennlp.ccg.grammar.*;\nimport opennlp.ccg.realize.*;\nimport opennlp.ccg.synsem.SignScorer;\nimport opennlp.ccg.ngrams.*;\n\nimport java.io.*;\nimport java.net.*;\nimport java.util.*;\n//import java.util.prefs.*;\n//import java.text.NumberFormat;\n\nimport org.jdom.*;\nimport org.jdom.input.*;\n\n/**\n * Runs cross-validation tests with the realizer.\n *\n * @author Michael White\n * @version $Revision: 1.30 $, $Date: 2011/03/20 20:11:58 $\n */\npublic class CrossValidateRealizer implements ScorerMaker {\n\n /** The tester to use. */\n public Regression tester = null;\n\n /** The path to the tmp dir. */\n public String tmpdir = \"tmp\";\n \n // the actual tmp dir\n private File tmpDir = null;\n \n /** The number of cross-validation folds, either 1.x or an int of at least 2. */\n public double numFolds = 10;\n\n /** The scorer maker, for preparing and loading scoring models. */\n public ScorerMaker scorerMaker = this;\n \n /** The pruning strategy, if any. */\n public PruningStrategy pruningStrategy = null;\n \n \n /** Sets up the tester with the given grammar. */\n public CrossValidateRealizer(URL grammarURL) throws IOException {\n\n // init tester\n tester = new Regression();\n \n // load grammar\n System.out.println(\"Loading grammar from URL: \" + grammarURL);\n tester.grammar = new Grammar(grammarURL);\n System.out.println();\n }\n\n \n /** Sets up the folds in tmpdir. */\n\t@SuppressWarnings({ \"unchecked\", \"rawtypes\" })\n\tpublic void setupInputs(URL testbedURL) throws IOException {\n \n // ensure tmpdir exists\n tmpDir = new File(tmpdir);\n if (!tmpDir.exists()) { tmpDir.mkdirs(); }\n \n try {\n // load items\n System.out.println(\"Loading testbed from URL: \" + testbedURL);\n SAXBuilder builder = new SAXBuilder();\n Document inputDoc = builder.build(testbedURL.openStream());\n System.out.println();\n Element inputRoot = inputDoc.getRootElement();\n List inputItems = inputRoot.getChildren(\"item\");\n \n System.out.println(\"Setting up inputs in tmpdir: \" + tmpdir);\n \n // make, save shuffled doc\n Document shuffledDoc = new Document();\n Element shuffledRoot = new Element(\"regression\");\n shuffledDoc.setRootElement(shuffledRoot);\n Random rand = new Random();\n while (inputItems.size() > 0) {\n Element rItem = (Element) inputItems.remove(rand.nextInt(inputItems.size()));\n shuffledRoot.addContent(rItem);\n }\n FileOutputStream shuffledOut = new FileOutputStream(new File(tmpDir, \"shuffled.xml\")); \n tester.grammar.serializeXml(shuffledDoc, shuffledOut);\n shuffledOut.close();\n List shuffledItems = shuffledRoot.getChildren(\"item\");\n int numItems = shuffledItems.size();\n Element[] shuffledItemsArray = new Element[numItems];\n shuffledItems.toArray(shuffledItemsArray); // need a non-live listing\n \n // get LF listing\n Element[] shuffledLFsArray = new Element[numItems];\n for (int i = 0; i < numItems; i++) {\n Element item = shuffledItemsArray[i];\n shuffledLFsArray[i] = item.getChild(\"lf\");\n // reduce content to just full-words (if present)\n Element fullWords = item.getChild(\"full-words\");\n item.setContent((List)null);\n if (fullWords != null) item.addContent(fullWords);\n }\n \n // make folds docs, leaving LFs out of training, and \n // removing any exact duplicates with test items \n int itemsPerFold = (int) Math.floor(numItems / (numFolds * 1.0));\n Set testStrings = new HashSet();\n int dups = 0;\n for (int i = 0; i < numFolds; i++) {\n testStrings.clear(); dups = 0;\n int foldStart = i * itemsPerFold;\n int foldLimit = (i < numFolds - 1) ? foldStart + itemsPerFold : numItems;\n Document testDoc = new Document();\n Element testRoot = new Element(\"regression\");\n testDoc.setRootElement(testRoot);\n Document trainDoc = new Document();\n Element trainRoot = new Element(\"regression\");\n trainDoc.setRootElement(trainRoot);\n // split items into train/test\n for (int j = 0; j < numItems; j++) {\n Element item = shuffledItemsArray[j]; \n item.detach();\n Element lf = shuffledLFsArray[j];\n if (foldStart <= j && j < foldLimit) {\n testRoot.addContent(item);\n item.addContent(lf);\n testStrings.add(item.getAttributeValue(\"string\"));\n }\n else { \n // special case for 1.x folds: limit training data \n // to first numItems - itemsPerFold items\n if (numFolds < 2 && i == 1 && j >= (numItems - itemsPerFold))\n continue;\n trainRoot.addContent(item);\n }\n }\n // check for dups in training items\n List trainingItems = trainRoot.getChildren(\"item\");\n for (Iterator it = trainingItems.iterator(); it.hasNext(); ) {\n Element item = (Element) it.next();\n if (testStrings.contains(item.getAttributeValue(\"string\"))) {\n it.remove(); dups++;\n }\n }\n if (dups > 0) { \n System.out.println(\"Removing \" + dups + \" test item duplicate(s) from training, fold \" + i);\n }\n FileOutputStream testOut = new FileOutputStream(new File(tmpDir, testFileName(i))); \n tester.grammar.serializeXml(testDoc, testOut);\n testOut.close();\n FileOutputStream trainOut = new FileOutputStream(new File(tmpDir, trainingFileName(i))); \n tester.grammar.serializeXml(trainDoc, trainOut);\n trainOut.close();\n }\n \n System.out.println();\n }\n catch (JDOMException exc) {\n throw (IOException) new IOException().initCause(exc);\n }\n }\n\n // training/test file names\n private String trainingFileName(int foldNum) { return \"fold\" + foldNum + \"-train.xml\"; }\n private String testFileName(int foldNum) { return \"fold\" + foldNum + \"-test.xml\"; }\n \n /** Does scorer prep (if any) on the folds already set-up in tmpdir. */\n public void prepScorers() throws IOException {\n // ensure tmpDir set\n if (tmpDir == null) tmpDir = new File(tmpdir);\n System.out.println(\"Preparing scorers in tmpdir: \" + tmpdir);\n System.out.println();\n // do each fold\n for (int i = 0; i < numFolds; i++) {\n // make training/test files for fold\n File trainFile = new File(tmpDir, trainingFileName(i));\n File testFile = new File(tmpDir, testFileName(i));\n // prep scorer\n scorerMaker.prepScorer(tmpDir, i, trainFile, testFile);\n }\n // summary\n scorerMaker.prepScorersSummary(tmpDir);\n }\n\n \n /** Default, do-nothing implementation of ScorerMaker.setCVR. */\n public void setCVR(CrossValidateRealizer cvr) {}\n \n /** Default, do-nothing implementation of ScorerMaker.prepScorer. */\n public void prepScorer(File tmpDir, int foldNum, File trainFile, File testFile) throws IOException {}\n \n /** Default, do-nothing implementation of ScorerMaker.prepScorersSummary. */\n public void prepScorersSummary(File tmpDir) throws IOException {}\n \n /**\n * Default implementation of ScorerMaker.loadScorer.\n * Loads an n-gram precision model with semantic class replacement, \n * using targets from the training data. \n */\n public SignScorer loadScorer(File tmpDir, int foldNum, File trainFile) throws IOException {\n RegressionInfo trainingItems = new RegressionInfo(tester.grammar, trainFile);\n String[] targets = new String[trainingItems.numberOfItems()];\n for (int i=0; i < trainingItems.numberOfItems(); i++) {\n targets[i] = trainingItems.getItem(i).sentence;\n }\n NgramPrecisionModel retval = (tester.ngramOrder > 0) \n ? new NgramPrecisionModel(targets, tester.ngramOrder, true) \n : new NgramPrecisionModel(targets, true);\n return retval;\n }\n \n\n /** Run the cross-validation test on the folds already set-up in tmpdir. */\n public void runTest() throws IOException {\n\n // ensure tmpDir set\n if (tmpDir == null) tmpDir = new File(tmpdir);\n \n // turn-off parsing, stats\n tester.doParsing = false;\n tester.showStats = false;\n \n // setup realizer\n tester.realizer = new Realizer(tester.grammar);\n if (pruningStrategy != null) {\n tester.realizer.pruningStrategy = pruningStrategy;\n }\n\n // show realizer settings\n Regression.showRealizerSettings();\n \n // do each fold\n for (int i = 0; i < numFolds; i++) {\n // make files for fold\n File trainFile = new File(tmpDir, trainingFileName(i));\n File testFile = new File(tmpDir, testFileName(i));\n // load scorer\n tester.scorer = scorerMaker.loadScorer(tmpDir, i, trainFile);\n // run test\n tester.runTest(testFile);\n }\n \n // show stats\n tester.showStats();\n }\n \n \n \n /** Command-line routine for cross-validating realizer. */\n public static void main(String[] args) throws IOException { \n \n String usage = \"java opennlp.ccg.test.CrossValidateRealizer \" + \n \"(-folds N) (-tmp ) \" +\n \"(-setuponly) (-skipsetup) \" + \n \"(-preponly) (-skipprep) \" + \n \"(-ngramorder N) \" + \n \"(-scorermaker ) \" + \n \"(-pruningstrategy ) \" +\n \"(-g ) (-s ) ()\";\n \n if (args.length > 0 && args[0].equals(\"-h\")) {\n System.out.println(\"Usage: \" + usage);\n System.exit(0);\n }\n \n // args\n double numFolds = 0;\n String tmpdir = null;\n boolean setupOnly = false;\n boolean skipSetup = false;\n boolean prepOnly = false;\n boolean skipPrep = false;\n int ngramOrder = 0;\n String scorerMakerClass = null;\n String pruningStrategyClass = null; \n String grammarfile = \"grammar.xml\";\n String testbedfile = \"testbed.xml\";\n String statsfile = null;\n for (int i = 0; i < args.length; i++) {\n if (args[i].equals(\"-folds\")) { \n numFolds = Double.parseDouble(args[++i]); \n if (numFolds < 1 || (numFolds >= 2 && numFolds != Math.round(numFolds))) {\n System.out.println(\"Error, folds must be 1.x or an int of at least 2\");\n System.exit(-1);\n }\n continue; \n }\n if (args[i].equals(\"-tmp\")) { tmpdir = args[++i]; continue; }\n if (args[i].equals(\"-setuponly\")) { setupOnly = true; continue; }\n if (args[i].equals(\"-skipsetup\")) { skipSetup = true; continue; }\n if (args[i].equals(\"-preponly\")) { prepOnly = true; continue; }\n if (args[i].equals(\"-skipprep\")) { skipPrep = true; continue; }\n if (args[i].equals(\"-ngramorder\")) { ngramOrder = Integer.parseInt(args[++i]); continue; }\n if (args[i].equals(\"-scorermaker\")) { scorerMakerClass = args[++i]; continue; }\n if (args[i].equals(\"-pruningstrategy\")) { pruningStrategyClass = args[++i]; continue; }\n if (args[i].equals(\"-g\")) { grammarfile = args[++i]; continue; }\n if (args[i].equals(\"-s\")) { statsfile = args[++i]; continue; }\n testbedfile = args[i];\n }\n \n // make cross-validator\n URL grammarURL = new File(grammarfile).toURI().toURL();\n CrossValidateRealizer cvr = new CrossValidateRealizer(grammarURL);\n if (numFolds > 0) cvr.numFolds = numFolds;\n if (tmpdir != null) cvr.tmpdir = tmpdir;\n if (ngramOrder > 0) cvr.tester.ngramOrder = ngramOrder;\n if (scorerMakerClass != null) {\n try {\n cvr.scorerMaker = (ScorerMaker) Class.forName(scorerMakerClass).newInstance();\n cvr.scorerMaker.setCVR(cvr);\n } catch (Exception exc) {\n throw (RuntimeException) new RuntimeException().initCause(exc);\n }\n }\n if (pruningStrategyClass != null) {\n try {\n cvr.pruningStrategy = (PruningStrategy) Class.forName(pruningStrategyClass).newInstance();\n } catch (Exception exc) {\n throw (RuntimeException) new RuntimeException().initCause(exc);\n }\n }\n if (statsfile != null) cvr.tester.statsfile = statsfile;\n\n // set-up inputs\n URL testbedURL = new File(testbedfile).toURI().toURL();\n if (!skipSetup) { cvr.setupInputs(testbedURL); }\n if (setupOnly) { System.exit(0); }\n\n // prep scorers\n if (!skipPrep) { cvr.prepScorers(); }\n if (prepOnly) { System.exit(0); }\n \n // run test\n System.gc();\n cvr.runTest();\n }\n}\n" }, { "path": "src/opennlp/ccg/test/DerivMaker.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2016 Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.test;\n\n//import java.util.*;\nimport org.jdom.*;\nimport opennlp.ccg.synsem.*;\n\n/**\n * Utility class for exporting derivations in xml.\n *\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2009/12/21 04:18:31 $\n */\npublic class DerivMaker {\n\n\t/** \n\t * Returns a derivation in the same format as the converted CCGbank, \n\t * but just with the lexemes, POS tags and supertags.\n\t */\n\tpublic static Element makeDeriv(Sign sign) {\n\t\tElement retval;\n\t\tif (sign.isLexical()) {\n\t\t\tretval = new Element(\"Leafnode\");\n\t\t\tretval.setAttribute(\"lexeme\",sign.getOrthography());\n\t\t\tretval.setAttribute(\"pos\",sign.getPOS());\n\t\t}\n\t\telse {\n\t\t\tretval = new Element(\"Treenode\");\n\t\t\tfor (Sign child: sign.getDerivationHistory().getInputs())\n\t\t\t\tretval.addContent(makeDeriv(child));\n\t\t}\n\t\tretval.setAttribute(\"stag\",sign.getSupertag());\n\t\treturn retval;\n\t}\n}\n\n\n" }, { "path": "src/opennlp/ccg/test/GenTargets.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.test;\n\nimport opennlp.ccg.realize.*;\nimport opennlp.ccg.grammar.*;\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.ngrams.*;\n\nimport org.jdom.*;\n\nimport java.io.*;\nimport java.net.*;\nimport java.util.*;\n\n/**\n * Generates initial target sentences from a list of logical forms. \n * The input is an XML file with a list of top-level LF elements.\n * The output is a testbed file, with the number of parses just \n * set to 1 (rather than actually being computed).\n * The best realization is determined using n-grams from an \n * existing testbed file.\n *\n * @author Michael White\n * @version $Revision: 1.9 $, $Date: 2009/12/21 04:18:31 $\n */\npublic class GenTargets\n{\n /** The grammar. */\n private Grammar grammar; \n\n /** The realizer instance. */\n private Realizer realizer;\n \n /** The n-gram scorer. */\n private NgramScorer ngramScorer;\n \n /** The unique target strings. */\n private Set uniqueTargets = new HashSet();\n \n /** \n * Constructor: loads grammar, instantiates realizer, and sets up n-gram scorer \n * using targets with sem class replacement from the testbed.\n */\n private GenTargets(URL grammarURL, File regressionFile) throws IOException {\n\n // load grammar\n System.out.println(\"Loading grammar from: \" + grammarURL);\n grammar = new Grammar(grammarURL);\n\n // set up n-gram scorer\n System.out.println(\"Loading target phrases from: \" + regressionFile);\n RegressionInfo rinfo = new RegressionInfo(grammar, regressionFile);\n String[] targets = new String[rinfo.numberOfItems()];\n for (int i=0; i < targets.length; i++) {\n String target = rinfo.getItem(i).sentence;\n targets[i] = target;\n }\n // use targets with sem class replacement\n ngramScorer = new NgramPrecisionModel(targets, true);\n \n // instantiate realizer \n realizer = new Realizer(grammar);\n }\n \n // does realization, adds test case\n private void realize(Element lfElt, Element outRoot) throws IOException {\n // get LF\n LF lf = Realizer.getLfFromElt(lfElt);\n \n // run request\n realizer.realize(lf, ngramScorer); \n Chart chart = realizer.getChart();\n \n // make test item (w/o trying to figure out the correct number of parses)\n String target = chart.bestEdge.getSign().getOrthography();\n if (uniqueTargets.contains(target)) {\n System.out.println(\"Duplicate realization: \" + target);\n return;\n }\n uniqueTargets.add(target);\n System.out.println(\"Best realization: \" + target);\n Element testElt = RegressionInfo.makeTestItem(grammar, target, 1, lf); \n \n // add to output\n outRoot.addContent(testElt);\n if (!chart.bestEdge.complete()) {\n System.out.println(\"NB: realization incomplete!\");\n testElt.setAttribute(\"complete\", \"false\");\n }\n }\n \n \n /** Creates generator and runs it on the given input file. */ \n @SuppressWarnings(\"unchecked\")\n\tpublic static void main(String[] args) throws IOException {\n \n String usage = \"Usage: java opennlp.ccg.test.GenTargets (-g ) (-tb ) \";\n \n if (args.length > 0 && args[0].equals(\"-h\")) {\n System.out.println(usage);\n System.exit(0);\n }\n\n // args\n String grammarfile = \"grammar.xml\";\n String testbedfile = \"testbed.xml\";\n String inputfile = null;\n String outputfile = null;\n for (int i = 0; i < args.length; i++) {\n if (args[i].equals(\"-g\")) {\n grammarfile = args[++i]; continue; \n }\n if (args[i].equals(\"-tb\")) {\n testbedfile = args[++i]; continue; \n }\n if (inputfile == null) {\n inputfile = args[i]; continue;\n }\n if (outputfile == null) {\n outputfile = args[i]; continue;\n }\n }\n if (inputfile == null || outputfile == null) {\n System.out.println(usage);\n System.exit(0);\n }\n\n // create Generator\n File gFile = new File(grammarfile);\n URL grammarURL = gFile.toURI().toURL();\n File tbFile = new File(testbedfile);\n if (!tbFile.exists()) {\n tbFile = new File(gFile.getParentFile(), testbedfile);\n }\n GenTargets gen = new GenTargets(grammarURL, tbFile);\n \n // load input LFs\n System.out.println(\"Loading LFs from: \" + inputfile);\n Document doc = gen.grammar.loadFromXml(inputfile);\n \n // create output doc\n Document outDoc = new Document();\n Element outRoot = new Element(\"regression\");\n outDoc.setRootElement(outRoot);\n\n // realize each one\n System.out.println(\"Realizing LFs ...\");\n Element root = doc.getRootElement();\n List lfElts = root.getChildren(\"lf\");\n for (int i = 0; i < lfElts.size(); i++) {\n Element lfElt = (Element) lfElts.get(i); \n try {\n gen.realize(lfElt, outRoot);\n }\n catch (Exception exc) {\n System.out.println(\"Warning: unable to realize LF \" + i + \": \" + exc);\n }\n }\n \n // save file\n System.out.println(\"Saving results to: \" + outputfile);\n FileOutputStream out = new FileOutputStream(outputfile); \n gen.grammar.serializeXml(outDoc, out);\n out.close();\n \n System.out.println(\"Done.\");\n }\n}\n\n" }, { "path": "src/opennlp/ccg/test/Regression.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-9 Jason Baldridge and Michael White (University of Edinburgh / The Ohio State University)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.test;\n\nimport java.io.*;\nimport java.net.URL;\nimport java.text.*;\nimport java.util.*;\nimport java.util.prefs.Preferences;\n\nimport opennlp.ccg.TextCCG;\nimport opennlp.ccg.grammar.Grammar;\nimport opennlp.ccg.hylo.*;\nimport opennlp.ccg.lexicon.Tokenizer;\nimport opennlp.ccg.lexicon.Word;\nimport opennlp.ccg.ngrams.*;\nimport opennlp.ccg.parse.ParseException;\nimport opennlp.ccg.parse.Parser;\nimport opennlp.ccg.parse.Supertagger;\nimport opennlp.ccg.parse.supertagger.WordAndPOSDictionaryLabellingStrategy;\nimport opennlp.ccg.realize.*;\nimport opennlp.ccg.realize.hypertagger.ZLMaxentHypertagger;\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.util.Pair;\nimport opennlp.ccg.util.SingletonList;\nimport opennlp.ccg.perceptron.*;\n\nimport org.jdom.*;\nimport org.jdom.output.Format;\nimport org.jdom.output.XMLOutputter;\n\n/**\n * Automates the testing of CCG grammars.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.151 $, $Date: 2011/12/11 16:51:58 $\n */\npublic class Regression {\n\n /** Flag for whether to do parsing. */\n public boolean doParsing = true;\n \n /** Flag for whether to do realization. */\n public boolean doRealization = true;\n\n /** Flag for whether to just do even items. */\n public boolean evenOnly = false;\n\n /** Flag for whether to just do odd items. */\n public boolean oddOnly = false;\n \n /** Flag for whether to do garbage collection before each iteration. */\n public boolean doGC = false;\n \n /** File to write events to (if any). */\n public String eventfile = null;\n \n /** Flag for whether to include the gold sign when generating events. */\n public boolean includeGoldInEvents = false;\n \n /** Directory for writing APML files (if any). */\n public String apmldir = null;\n \n /** Flag for whether to show realization stats. */\n public boolean showStats = true;\n \n /** Flag for whether to show parsing stats. */\n public boolean showParseStats = false;\n \n /** File to dump realizer stats to (if any). */\n public String statsfile = null;\n \n /** File prefix to write bleu test files to (if any). */\n public String bleufileprefix = null;\n\n /** File to write n-best realizations to (if any). */\n public String nbestrealfile = null;\n \n /** Flag for whether to normalize strings as for BLEU scoring in n-best output. */\n public boolean nbestnormbleu = false;\n \n /** Directory to save best realization serializations to (if any). */\n public String realserdir = null;\n \n /** File to write rescored sign scores to (if any). */\n public String rescorefile = null;\n \n /** Map from info keys to best realization signs for serialization (if any). */\n public Map bestRealMap = null;\n \n /** Flag for whether to include LFs in n-best output. */\n public boolean nbestincludelfs = false;\n \n /** File to write n-best parses to (if any). */\n public String nbestparsefile = null;\n \n /** The grammar to use for testing. */\n public Grammar grammar = null;\n \n /** The parser to use for testing. */\n public Parser parser = null;\n \n /** The realizer to use for testing. */\n public Realizer realizer = null;\n \n /** The scorer to use for realizer testing (or null, for default). */\n public SignScorer scorer = null;\n \n /** The scorer to use for parser testing. */\n public SignScorer parseScorer = null;\n \n /** Flag for whether to only allow exact matches with the default scorer. */\n public boolean exactMatches = false;\n \n /** The n-gram order to use with the default scorer (or 0, for default). */\n public int ngramOrder = 0; \n\n /** The feature extractor to use in extracting events. */\n public FeatureExtractor featureExtractor = null;\n \n //\n // the various totals\n //\n\n public int pCount = 0;\n public int pBadCount = 0;\n public int pFailedCount = 0;\n public int pExactCount = 0;\n public double totalF = 0.0;\n public double totalRecall = 0.0;\n public double totalPrecision = 0.0;\n public double totalDepsF = 0.0;\n public double totalDepsRecall = 0.0;\n public double totalDepsPrecision = 0.0;\n public double totalUnlabeledDepsF = 0.0;\n public double totalUnlabeledDepsRecall = 0.0;\n public double totalUnlabeledDepsPrecision = 0.0;\n public double totalFComplete = 0.0;\n public double totalRecallComplete = 0.0;\n public double totalPrecisionComplete = 0.0;\n public double totalDepsFComplete = 0.0;\n public double totalDepsRecallComplete = 0.0;\n public double totalDepsPrecisionComplete = 0.0;\n public double totalUnlabeledDepsFComplete = 0.0;\n public double totalUnlabeledDepsRecallComplete = 0.0;\n public double totalUnlabeledDepsPrecisionComplete = 0.0;\n public int pTotalEdges = 0;\n public int pTotalEdgesGood = 0;\n public int pMaxEdges = 0;\n public int pMaxEdgesGood = 0;\n public int pTotalUnpackingEdges = 0;\n public int pMaxUnpackingEdges = 0;\n public int pTotalCellMax = 0;\n public int pTotalCellMaxGood = 0;\n public int pMaxCellMax = 0;\n public int pMaxCellMaxGood = 0;\n public int pTotalLexTime = 0;\n public int pTotalParseTime = 0;\n public int pTotalChartTime = 0;\n public int pTotalUnpackingTime = 0;\n public int pMaxLexTime = 0;\n public int pMaxParseTime = 0;\n public int pMaxChartTime = 0;\n public int pMaxUnpackingTime = 0;\n public Map pBetaTallies = null;\n \n public int rCount = 0;\n public int rDoneCount = 0;\n public int rBadCount = 0;\n public int rExactCount = 0;\n public double totalScore = 0.0;\n public double totalScoreComplete = 0.0;\n public double totalReciprocalRank = 0.0;\n public int totalNominals = 0; \n public int totalTokens = 0;\n public int minTokens = 0;\n public int maxTokens = 0;\n public int totalRuleApps = 0;\n public int totalEdges = 0;\n public int totalEdgesCreated = 0;\n public int totalUnprunedEdges = 0;\n public int totalPrunedRemoved = 0;\n public int totalPrunedNeverAdded = 0;\n public int totalCellMax = 0;\n public int totalNewBest = 0;\n public int totalLex = 0;\n public int totalFirst = 0;\n public int totalBest = 0;\n public int totalPacked = 0;\n public int totalStoppedOrDone = 0;\n public int maxLex = 0;\n public int maxFirst = 0;\n public int maxBest = 0;\n public int maxNewBest = 0;\n public int maxPacked = 0;\n public int maxStoppedOrDone = 0;\n public int oracleBetter = 0;\n public int goldMissing = 0;\n public String maxLexStr = null;\n public String maxFirstStr = null;\n public String maxBestStr = null;\n public String maxNewBestStr = null;\n public String maxPackedStr = null;\n public String maxStoppedOrDoneStr = null;\n public List bestEstimatedScores = null;\n public List bestActualScores = null;\n public List itemRanks = null;\n public TimingMap lexMap = null;\n public TimingMap firstMap = null;\n public TimingMap bestMap = null; \n public TimingMap allMap = null; \n \n private PrintWriter events = null;\n private PrintWriter bleuGen = null;\n private PrintWriter bleuRef = null;\n private PrintWriter bleuSrc = null;\n private PrintWriter nbestrealPW = null;\n private PrintWriter rescorePW = null;\n private PrintWriter nbestparsePW = null;\n \n private XMLOutputter xmlOutputter = new XMLOutputter(); // for xml-escaping strings\n \n /** Constructor. */\n public Regression() {\n // init\n resetTotals();\n }\n \n /** Resets the various totals. */\n public void resetTotals() {\n \t// parser\n pCount = 0; pBadCount = 0; pFailedCount = 0; pExactCount = 0;\n totalF = 0.0; totalRecall = 0.0; totalPrecision = 0.0;\n totalDepsF = 0.0; totalDepsRecall = 0.0; totalDepsPrecision = 0.0;\n totalUnlabeledDepsF = 0.0; totalUnlabeledDepsRecall = 0.0; totalUnlabeledDepsPrecision = 0.0;\n totalFComplete = 0.0; totalRecallComplete = 0.0; totalPrecisionComplete = 0.0;\n totalDepsFComplete = 0.0; totalDepsRecallComplete = 0.0; totalDepsPrecisionComplete = 0.0;\n totalUnlabeledDepsFComplete = 0.0; totalUnlabeledDepsRecallComplete = 0.0; totalUnlabeledDepsPrecisionComplete = 0.0;\n pTotalEdges = 0; pTotalEdgesGood = 0; pMaxEdges = 0; pMaxEdgesGood = 0;\n pTotalUnpackingEdges = 0; pMaxUnpackingEdges = 0;\n pTotalCellMax = 0; pTotalCellMaxGood = 0; pMaxCellMax = 0; pMaxCellMaxGood = 0;\n pTotalLexTime = 0; pTotalParseTime = 0; pTotalChartTime = 0; pTotalUnpackingTime = 0;\n pMaxLexTime = 0; pMaxParseTime = 0; pMaxChartTime = 0; pMaxUnpackingTime = 0;\n if (doParsing) {\n \tpBetaTallies = new TreeMap();\n }\n // realizer\n rCount = 0; rDoneCount = 0; rBadCount = 0; rExactCount = 0;\n totalScore = 0.0; totalScoreComplete = 0.0;\n totalReciprocalRank = 0.0;\n totalNominals = 0; \n totalTokens = 0; minTokens = 0; maxTokens = 0;\n totalRuleApps = 0;\n totalEdges = 0; totalEdgesCreated = 0; totalUnprunedEdges = 0; totalPrunedRemoved = 0; totalPrunedNeverAdded = 0; totalCellMax = 0;\n totalNewBest = 0; totalLex = 0; totalFirst = 0; totalBest = 0; totalPacked = 0; totalStoppedOrDone = 0;\n maxLex = 0; maxFirst = 0; maxBest = 0; maxNewBest = 0; maxPacked = 0; maxStoppedOrDone = 0;\n oracleBetter = 0; goldMissing = 0;\n maxLexStr = null; maxFirstStr = null; maxBestStr = null; maxNewBestStr = null; maxPackedStr = null; maxStoppedOrDoneStr = null;\n if (doRealization) {\n bestActualScores = new ArrayList(); bestEstimatedScores = new ArrayList(); itemRanks = new ArrayList();\n lexMap = new TimingMap(\"lex\"); firstMap = new TimingMap(\"first\"); bestMap = new TimingMap(\"best\"); allMap = new TimingMap(\"all\");\n }\n }\n\n \n // sets up bleu output\n private void bleuSetup() throws IOException {\n \t// setup bleu files, if apropos\n \tif (bleufileprefix != null && doRealization) {\n \t\tbleuGen = new PrintWriter(new BufferedWriter(new FileWriter(bleufileprefix + \"-gen.sgm\")));\n \t\tbleuRef = new PrintWriter(new BufferedWriter(new FileWriter(bleufileprefix + \"-ref.sgm\")));\n \t\tbleuSrc = new PrintWriter(new BufferedWriter(new FileWriter(bleufileprefix + \"-src.sgm\")));\n \t\tbleuGen.println(\"\");\n \t\tbleuRef.println(\"\");\n \t\tbleuSrc.println(\"\");\n \t}\n }\n \n // sets up n-best realization output\n private void nbestrealSetup() throws IOException {\n // set up file to write ref & n-best realizations (if any)\n if (nbestrealfile != null && doRealization) {\n nbestrealPW = new PrintWriter(new BufferedWriter(new FileWriter(nbestrealfile)));\n nbestrealPW.println(\"\");\n }\n }\n \n // sets up rescored sign score output\n private void rescoreSetup() throws IOException {\n // set up rescoring file\n if (rescorefile != null) {\n rescorePW = new PrintWriter(new BufferedWriter(new FileWriter(rescorefile)));\n rescorePW.println(\"\");\n }\n }\n \n // sets up n-best parsing output\n private void nbestparseSetup() throws IOException {\n // set up file to write sentence & n-best parses (if any)\n if (nbestparsefile != null && doParsing) {\n nbestparsePW = new PrintWriter(new BufferedWriter(new FileWriter(nbestparsefile)));\n nbestparsePW.println(\"\");\n }\n }\n \n // starts a doc\n private void bleuStartDoc(String id) {\n \tif (bleufileprefix != null && doRealization) {\n\t\t\tbleuGen.println(\"\");\n\t\t\tbleuRef.println(\"\");\n\t\t\tbleuSrc.println(\"\");\n \t}\n }\n\n // ends a doc\n private void bleuEndDoc() {\n \tif (bleufileprefix != null && doRealization) {\n \t\tbleuGen.println(\"\"); \n \t\tbleuRef.println(\"\"); \n \t\tbleuSrc.println(\"\"); \n \t}\n }\n\n // finishes bleu output\n private void bleuFinish() throws IOException {\n \t// finish bleu files, if apropos\n \tif (bleufileprefix != null && doRealization) {\n \t\tbleuGen.println(\"\");\n \t\tbleuRef.println(\"\");\n \t\tbleuSrc.println(\"\");\n \t\tbleuGen.flush(); bleuGen.close();\n \t\tbleuRef.flush(); bleuRef.close();\n \t\tbleuSrc.flush(); bleuSrc.close();\n \t}\n }\n\n // finishes n-best realization output\n private void nbestrealFinish() throws IOException {\n \t// finish n-best real file, if apropos\n \tif (nbestrealfile != null && doRealization) {\n \t\tnbestrealPW.println(\"\");\n \t\tnbestrealPW.flush(); nbestrealPW.close();\n \t}\n }\n\n // finishes rescored sign output\n private void rescoreFinish() throws IOException {\n \t// finish rescoring file\n \tif (rescorefile != null) {\n \t\trescorePW.println(\"\");\n \t\trescorePW.flush(); rescorePW.close();\n \t}\n }\n\n // finishes n-best parsing output\n private void nbestparseFinish() throws IOException {\n \t// finish n-best real file, if apropos\n \tif (nbestparsefile != null && doParsing) {\n \t\tnbestparsePW.println(\"\");\n \t\tnbestparsePW.flush(); nbestparsePW.close();\n \t}\n }\n\n // resets bestRealMap\n private void realserStartDoc() {\n \tif (realserdir != null && doRealization) {\n \t\tbestRealMap = new HashMap();\n \t}\n }\n \n // serializes bestRealMap\n private void realserEndDoc(String testName) throws IOException {\n \tif (realserdir != null && doRealization) {\n\t \tFile serFile = new File(new File(realserdir), testName + \".ser\");\n\t \tObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(serFile));\n\t \toos.writeObject(bestRealMap);\n\t \toos.close();\n \t}\n }\n \n // escapes string for xml output\n private String xmlEscape(String s) { return xmlOutputter.outputString(new Text(s)); }\n \n // normalizes realizations for BLEU scoring\n // at present, this means replacing underscores with spaces and escaping for xml output\n private String norm_bleu(String s) { return xmlEscape(s.replace('_', ' ')); }\n \n \n /** Runs the test on the items in the given file or directory of files. */\n public void runTest(File regressionFile) throws IOException {\n \t// set up event file (if any)\n \tif (eventfile != null) events = EventFile.openWriter(new File(eventfile));\n // set up bleu output, n-best realizations, rescoring, n-best parses (if apropos)\n \tbleuSetup(); nbestrealSetup(); rescoreSetup(); nbestparseSetup();\n \t// do each file or files\n \tfor (File f : getXMLFiles(regressionFile))\n \t\trunSingleTest(f);\n \t// finish bleu, n-best realization output, rescoring, n-best parses (if apropos)\n \tbleuFinish(); nbestrealFinish(); rescoreFinish(); nbestparseFinish();\n \t// close event file (if any)\n \tif (events != null) { events.flush(); events.close(); }\n \t// show stats (if apropos)\n \tif (rescorefile != null) return;\n \tif (doParsing && showParseStats) showParseStats();\n if (doRealization && showStats) showStats();\n }\n \t \n /** Returns a list of xml files from the given file or directory. */\n public static List getXMLFiles(File file) {\n \tif (!file.isDirectory()) return new SingletonList(file);\n \tList retval = new ArrayList();\n\t\tFile[] files = file.listFiles();\n\t\tArrays.sort(files);\n\t\tfor (int i = 0; i < files.length; i++) {\n\t\t\tif (!files[i].isDirectory() && files[i].getName().endsWith(\".xml\"))\n\t\t\t\tretval.add(files[i]);\n\t\t}\n \treturn retval;\n }\n \t\n /** Runs the test on the items in the given file. */\n private void runSingleTest(File regressionFile) throws IOException {\n \t\n \tString testName = regressionFile.getName();\n \tint lastDot = testName.lastIndexOf('.');\n \tif (lastDot > 0) testName = testName.substring(0, lastDot);\n \t\n // load testfile\n \tSystem.out.println(\"Loading: \" + testName);\n \tSystem.out.println();\n RegressionInfo rinfo = new RegressionInfo(grammar, regressionFile);\n\n // start bleu doc (if apropos)\n bleuStartDoc(testName);\n // start storing best realizations for serialization (if apropos) \n realserStartDoc();\n \n // do each test\n int numItems = rinfo.numberOfItems();\n System.out.println(\"Parse\\tRealize\\tString\");\n System.out.println(\"-----\\t-------\\t------\");\n \n for (int i=0; i < numItems; i++) {\n\n // check even/odd only\n if (i % 2 == 1 && evenOnly) continue;\n if (i % 2 == 0 && oddOnly) continue;\n \n RegressionInfo.TestItem testItem = rinfo.getItem(i);\n if (doGC) System.gc();\n \n // short circuit for sign rescoring; realization only at the moment\n if (rescorefile != null) {\n String id = testItem.info;\n double score = scorer.score(testItem.sign, true);\n \trescorePW.println(\"\");\n showOutcome(\"-\", nfE.format(score), \"\", testItem.sign.getOrthography());\n \tcontinue;\n }\n \n List parses = null;\n List parseScores = null;\n LF parsedLF = null;\n LF compactedLF = null;\n LF transformedParsedLF = null;\n boolean parsed = false;\n boolean parsedComplete = false;\n if (doParsing) {\n try {\n \t\t// use full-words or words from stored sign if possible\n \t\tList words = null;\n \t\tif (testItem.fullWords != null) {\n \t\twords = grammar.lexicon.tokenizer.tokenize(testItem.fullWords, true);\n \t\t// strip and \n \t\tif (words.get(0).getForm() == \"\") words.remove(0);\n \t\tif (words.get(words.size()-1).getForm() == \"\") words.remove(words.size()-1);\n \t\t}\n \t\telse if (testItem.sign != null) {\n \t\t\twords = testItem.sign.getWords();\n \t\t}\n \t\tif (words != null) {\n\t \t\t// parse 'em\n\t \t\tparser.parse(words);\n \t\t}\n \telse { \n \t\tparser.parse(testItem.sentence);\n \t}\n \t// retrieve results\n parses = parser.getResult();\n parseScores = parser.getScores();\n parsed = true;\n parsedComplete = !parses.get(0).getCategory().isFragment();\n // get LF of best parse, if needed\n if (showParseStats || (doRealization && testItem.lfElt == null && testItem.sign == null)) {\n\t Sign sign = parses.get(0);\n\t Category cat = sign.getCategory().copy();\n\t Nominal index = cat.getIndexNominal();\n\t parsedLF = cat.getLF();\n\t index = HyloHelper.convertNominals(parsedLF, sign, index);\n\t compactedLF = HyloHelper.compact(parsedLF, index);\n\t // get transformed version if needed\n\t if (testItem.sign == null) {\n\t\t transformedParsedLF = grammar.transformLF(compactedLF); \n\t }\n }\n } catch (ParseException e) {\n \tparses = Collections.emptyList();\n parsed = false;\n } catch (Exception e) {\n \tparses = Collections.emptyList();\n parsed = false;\n System.err.println(\"Uncaught exception in parsing: \" + testItem.sentence);\n e.printStackTrace(System.err);\n }\n \n // update parse stats\n int count = parser.edgeCount();\n pTotalEdges += count;\n if (count > pMaxEdges) pMaxEdges = count;\n if (parsedComplete) {\n \tpTotalEdgesGood += count;\n \tif (count > pMaxEdgesGood) pMaxEdgesGood = count;\n }\n count = parser.unpackingEdgeCount();\n pTotalUnpackingEdges += count;\n if (count > pMaxUnpackingEdges) pMaxUnpackingEdges = count;\n int cellMax = parser.maxCellSize();\n pTotalCellMax += cellMax;\n if (cellMax > pMaxCellMax) pMaxCellMax = cellMax;\n if (parsedComplete) {\n \tpTotalCellMaxGood += cellMax;\n if (cellMax > pMaxCellMaxGood) pMaxCellMaxGood = cellMax;\n }\n int time = parser.getLexTime();\n pTotalLexTime += time;\n if (time > pMaxLexTime) pMaxLexTime = time;\n time = parser.getParseTime();\n pTotalParseTime += time;\n if (time > pMaxParseTime) pMaxParseTime = time;\n time = parser.getChartTime();\n pTotalChartTime += time;\n if (time > pMaxChartTime) pMaxChartTime = time;\n time = parser.getUnpackingTime();\n pTotalUnpackingTime += time;\n if (time > pMaxUnpackingTime) pMaxUnpackingTime = time;\n double beta = parser.getSupertaggerBeta();\n Integer betaTally = pBetaTallies.get(beta);\n pBetaTallies.put(beta, (betaTally != null) ? ++betaTally : 1);\n }\n \n // get test item LF, if needed\n LF testItemLF = null;\n if (testItem.lfElt != null && (doRealization || (showParseStats && parsed && testItem.sign == null))) {\n Element lfElt = testItem.lfElt;\n Document doc = new Document();\n lfElt.detach();\n doc.setRootElement(lfElt);\n testItemLF = grammar.loadLF(doc);\n }\n \n // compare EPs\n EPsScorer.Results parseScore = null;\n LF goldLF = null;\n if (showParseStats && parsedLF != null) {\n // get LF to score, gold LF\n \tLF lfToScore = null;\n if (testItem.sign != null) {\n \t// use LF from stored sign if available\n \tlfToScore = parsedLF;\n Category cat = testItem.sign.getCategory().copy();\n Nominal index = cat.getIndexNominal();\n goldLF = cat.getLF();\n index = HyloHelper.convertNominals(goldLF, testItem.sign, index);\n }\n else {\n \t// otherwise use test item LF\n \tlfToScore = transformedParsedLF;\n \tgoldLF = testItemLF;\n }\n \t\tif (goldLF == null) {\n \t\t\tthrow new RuntimeException(\n \t\t\t\t\"Can't score parse: \" + testItem.sentence + \"!\\n\" +\n \t\t\t\t\"No gold LF.\"\n \t\t\t\t);\n \t\t}\n // score parse\n \tparseScore = EPsScorer.score(lfToScore, goldLF);\n }\n \n // update parsing results\n if (showParseStats && testItem.numOfParses > 0) {\n \tpCount++;\n \tif (parses.size() == 0 || !parsedComplete) pBadCount++;\n \tif (parses.size() == 0) pFailedCount++;\n \tif (parseScore != null) {\n \t\tif (parseScore.fscore == 1.0) pExactCount++;\n \t\ttotalF += parseScore.fscore;\n \t\ttotalRecall += parseScore.recall;\n \t\ttotalPrecision += parseScore.precision;\n \t\ttotalDepsF += parseScore.depsFscore;\n \t\ttotalDepsRecall += parseScore.depsRecall;\n \t\ttotalDepsPrecision += parseScore.depsPrecision;\n \t\ttotalUnlabeledDepsF += parseScore.unlabeledDepsFscore;\n \t\ttotalUnlabeledDepsRecall += parseScore.unlabeledDepsRecall;\n \t\ttotalUnlabeledDepsPrecision += parseScore.unlabeledDepsPrecision;\n \t\tif (parsedComplete) {\n \t\ttotalFComplete += parseScore.fscore;\n \t\ttotalRecallComplete += parseScore.recall;\n \t\ttotalPrecisionComplete += parseScore.precision;\n \t\ttotalDepsFComplete += parseScore.depsFscore;\n \t\ttotalDepsRecallComplete += parseScore.depsRecall;\n \t\ttotalDepsPrecisionComplete += parseScore.depsPrecision;\n \t\ttotalUnlabeledDepsFComplete += parseScore.unlabeledDepsFscore;\n \t\ttotalUnlabeledDepsRecallComplete += parseScore.unlabeledDepsRecall;\n \t\ttotalUnlabeledDepsPrecisionComplete += parseScore.unlabeledDepsPrecision;\n \t\t}\n \t}\n }\n \n // events output\n if (events != null && doParsing) {\n \t// nb: only dealing with complete parses at the moment\n \t// nb: gold LF must come from saved sign\n \tif (parses.size() > 0 && testItem.sign != null) {\n \t\tList bestSigns = new ArrayList(parses);\n \t\tSign best = parses.get(0);\n \t\t// update best if not exact match\n \t\tif (parseScore.fscore != 1.0) {\n \t\t\t// check oracle best\n \t\t\tPair bestPair = parser.oracleBest(goldLF); \n \t\t\tif (bestPair.a != null) oracleBetter++;\n \t\t\tif (bestPair.b) {\n \t\t\t\tbest = bestPair.a;\n \t\t\t\tif (!bestSigns.contains(best)) bestSigns.add(best);\n \t\t\t}\n \t\t\t// add gold if missing, if apropos\n \t\t\telse {\n \t\t\t\tgoldMissing++;\n \t\t\t\tif (includeGoldInEvents) {\n\t \t\t\tbest = testItem.sign;\n\t \t\t//parser.addSupertaggerLogProbs(best); \n\t \t\tbestSigns.add(best);\n \t\t\t\t}\n \t\t\t}\n \t\t}\n \t\t\tEventFile.writeEvents(events, bestSigns, best, featureExtractor);\n \t}\n }\n \n // n-best parses output\n if (nbestparsePW != null) {\n XMLOutputter outputter = new XMLOutputter();\n outputter.setFormat(Format.getPrettyFormat());\n \t// header for item\n String extras = \"\";\n if (parsedComplete) extras += \" complete=\\\"true\\\"\";\n String id = testItem.info;\n if (id == null) id = \"\" + i;\n nbestparsePW.println(\"\");\n \tString tagend = (nbestincludelfs) ? \">\" : \"/>\";\n \t// add best parse\n \tif (parseScore != null) { \n \t\tdouble edgeScore = parseScores.get(0);\n \tString scores = \"score=\\\"\" + nf.format(parseScore.fscore) + \"\\\" edge-score=\\\"\" + nfE.format(edgeScore) + \"\\\"\";\n \tnbestparsePW.println(\"\"); \t\t\n \t}\n \t}\n \t// add remaining n-best \n \tfor (int k=1; k < parses.size(); k++) {\n Sign sign = parses.get(k);\n \t\tdouble edgeScore = parseScores.get(k);\n Category cat = sign.getCategory().copy();\n Nominal index = cat.getIndexNominal();\n LF parsedLFk = cat.getLF();\n index = HyloHelper.convertNominals(parsedLFk, sign, index);\n LF compactedLFk = HyloHelper.compact(parsedLFk, index);\n LF lfToScore = parsedLFk;\n if (testItem.sign != null) {\n lfToScore = grammar.transformLF(compactedLFk); \n }\n EPsScorer.Results parseScoreK = EPsScorer.score(lfToScore, goldLF);\n \tString scores = \"score=\\\"\" + nf.format(parseScoreK.fscore) + \"\\\" edge-score=\\\"\" + nfE.format(edgeScore) + \"\\\"\";\n \tnbestparsePW.println(\"\");\n \t}\n \t}\n // close item\n nbestparsePW.println(\"\");\n }\n \n // determine string to show for parse result\n String starForBadSentence = \"\";\n if (testItem.numOfParses == 0) starForBadSentence = \"*\";\n \n String parseResult;\n if (!doParsing) {\n parseResult = \"-\";\n } else if (parseScore != null) {\n parseResult = nf.format(parseScore.fscore);\n if (!parsedComplete) parseResult = \"[\" + parseResult + \"]\";\n } else if (testItem.numOfParses == parses.size()) {\n parseResult = \"ok\";\n } else if (testItem.numOfParses > 0 && parses.size() > 0) {\n // show num parses, if not the number expected\n parseResult = \"(\" + parses.size() + \")\";\n } else if (testItem.knownFailure) {\n parseResult = \"(known)\";\n } else {\n parseResult = \"FAILED\";\n }\n \n if (!doRealization || (doParsing && !parsed) || testItem.numOfParses == 0) {\n showOutcome(parseResult, \"-\", starForBadSentence, testItem.sentence);\n continue;\n }\n \n LF inputLF = null;\n // use given LF\n if (testItemLF != null) inputLF = testItemLF;\n // or LF from stored sign\n else if (testItem.sign != null) {\n Sign sign = testItem.sign;\n Category cat = sign.getCategory().copy();\n Nominal index = cat.getIndexNominal();\n LF convertedLF = HyloHelper.compactAndConvertNominals(cat.getLF(), index, sign);\n inputLF = grammar.transformLF(convertedLF);\n }\n // otherwise use first parse\n else if (transformedParsedLF != null) inputLF = transformedParsedLF;\n // otherwise give up\n else {\n String suggestion = (!doParsing) ? \"Try leaving off -noparsing option.\" : \"\";\n throw new RuntimeException(\"No LF to realize! \" + suggestion);\n }\n \n // set up n-gram precision scorer for default scoring and/or scoring results\n String[] targets = (testItem.alt == null) \n \t? new String[] { testItem.sentence }\n \t: new String[] { testItem.sentence, testItem.alt };\n NgramPrecisionModel defaultNgramScorer = new NgramPrecisionModel(targets);\n SignScorer scorerToUse = scorer;\n if (scorerToUse == null) {\n if (ngramOrder > 0 || exactMatches) {\n \tif (ngramOrder > 0) scorerToUse = new NgramPrecisionModel(targets, ngramOrder);\n \telse scorerToUse = new NgramPrecisionModel(targets);\n \t((NgramPrecisionModel)scorerToUse).setExactMatches(exactMatches);\n }\n else scorerToUse = defaultNgramScorer;\n }\n // set targets for self-paraphrase biasing\n else if (scorerToUse instanceof SelfParaphraseBiaser) {\n \t((SelfParaphraseBiaser)scorerToUse).setTargets(targets);\n }\n \n if (doGC) System.gc(); \n try {\n \t//Add gold std pred supertag mapping to the hypertagger class\n \tif (events != null && realizer.hypertagger != null && testItem.predInfo != null) {\n \trealizer.hypertagger.storeGoldStdPredInfo(testItem.predInfo);\n }\n \trealizer.realize(inputLF, scorerToUse);\n }\n catch (Throwable thrwbl) {\n System.out.println(\"Unable to process: \" + testItem.sentence);\n thrwbl.printStackTrace(System.out);\n continue;\n }\n opennlp.ccg.realize.Chart chart = realizer.getChart();\n String realizeResult = \"ok\"; boolean gramcomplete = true; boolean joined = false;\n Edge bestEdge = chart.bestEdge;\n if (!bestEdge.complete() || bestEdge.getSign().getCategory().isFragment()) {\n realizeResult = \"[ok]\"; gramcomplete = false;\n rBadCount++;\n if (chart.joinFragments) { \n \tbestEdge = chart.bestJoinedEdge;\n \tjoined = true;\n }\n }\n String bestRealization = bestEdge.getSign().getOrthography();\n double score = defaultNgramScorer.score(bestEdge.getSign(), false); // nb: use default n-gram precision score for reporting \n \n // events output\n if (events != null) {\n \tList bestEdges = chart.bestEdges();\n \t// nb: only dealing with complete realizations at the moment\n \tif (bestEdges.size() > 0) {\n \t\tPair bestPair = chart.oracleBest(testItem.sentence); \n \t\tEdge oracleBest = bestPair.a;\n \t\tif (oracleBest != null) {\n \tSign best = oracleBest.getSign();\n \tList bestSigns = new ArrayList(bestEdges.size()+1);\n \tfor (Edge e : bestEdges) bestSigns.add(e.getSign());\n \t\t\tif (bestEdge != oracleBest) oracleBetter++;\n \t\t\tif (!bestPair.b) {\n \t\t\t\tgoldMissing++;\n \t\t\t\tif (includeGoldInEvents) {\n \t\t\t\t\tbest = testItem.sign;\n \t\t\t\t}\n \t\t\t}\n \t\t\tif (!bestSigns.contains(best)) bestSigns.add(best);\n \t\t\tEventFile.writeEvents(events, bestSigns, best, featureExtractor);\n \t\t}\n \t}\n }\n \n // bleu output\n if (bleufileprefix != null) {\n String extras = \" time=\\\"\";\n if (chart.done) extras += chart.timeTilDone;\n else extras += chart.timeTilStopped;\n extras += \"\\\"\";\n extras += \" score=\\\"\" + nf.format(score) + \"\\\"\";\n if (gramcomplete) extras += \" complete=\\\"true\\\"\";\n if (joined) extras += \" joined=\\\"true\\\"\";\n String id = testItem.info;\n if (id == null) id = \"\" + i;\n \tbleuGen.println(\"\" + norm_bleu(bestRealization) + \"\");\n \tString sent = norm_bleu(testItem.sentence);\n \tbleuRef.println(\"\" + sent + \"\");\n \tbleuSrc.println(\"\" + sent + \"\");\n }\n \n // n-best realization output\n if (nbestrealPW != null) {\n XMLOutputter outputter = new XMLOutputter();\n outputter.setFormat(Format.getPrettyFormat());\n \t// header for item\n String extras = \"\";\n if (gramcomplete) extras += \" complete=\\\"true\\\"\";\n if (joined) extras += \" joined=\\\"true\\\"\";\n String id = testItem.info;\n if (id == null) id = \"\" + i;\n \tnbestrealPW.println(\"\");\n \t// add ref sentence\n \tString ref = (nbestnormbleu) ? norm_bleu(testItem.sentence) : xmlEscape(testItem.sentence); \n \tnbestrealPW.println(\"\" + ref + \"\");\n \t// add best realization\n \tString scores = \"score=\\\"\" + nf.format(score) + \"\\\" edge-score=\\\"\" + nfE.format(bestEdge.score) + \"\\\"\";\n \tString best = (nbestnormbleu) ? norm_bleu(bestRealization) : xmlEscape(bestRealization);\n \tif (!nbestincludelfs)\n \t\tnbestrealPW.println(\"\" + best + \"\");\n \telse {\n \t\tnbestrealPW.println(\"\");\n \t\tnbestrealPW.println(\"\" + best + \"\");\n \tSign sign = bestEdge.getSign();\n Category cat = sign.getCategory().copy();\n Nominal index = cat.getIndexNominal();\n LF lf = cat.getLF();\n \tindex = HyloHelper.convertNominalsToVars(lf, index);\n index = HyloHelper.convertNominals(lf, sign, index);\n LF lfc = HyloHelper.compact(lf, index);\n \tElement lfElt = grammar.makeLfElt(lfc);\n \tnbestrealPW.println(outputter.outputString(lfElt));\n \t\tnbestrealPW.println(\"\");\n \t\t\n \t}\n // if complete, add remaining n-best\n if (bestEdge.complete()) {\n List bestEdges = chart.bestEdges();\n for (int j=1; j < bestEdges.size(); j++) {\n \tEdge e = bestEdges.get(j);\n String eSent = e.getSign().getOrthography();\n double eScore = defaultNgramScorer.score(e.getSign(), false); // nb: use default n-gram precision score for reporting\n \tString eScores = \" score=\\\"\" + nf.format(eScore) + \"\\\" edge-score=\\\"\" + nfE.format(e.score) + \"\\\"\";\n \t// add next realization\n \tString next = (nbestnormbleu) ? norm_bleu(eSent) : xmlEscape(eSent);\n \tif (!nbestincludelfs)\n \t\tnbestrealPW.println(\"\" + next + \"\");\n \telse {\n \t\tnbestrealPW.println(\"\");\n \t\tnbestrealPW.println(\"\" + next + \"\");\n \tSign sign = e.getSign();\n Category cat = sign.getCategory().copy();\n Nominal index = cat.getIndexNominal();\n LF lf = cat.getLF();\n \tindex = HyloHelper.convertNominalsToVars(lf, index);\n index = HyloHelper.convertNominals(lf, sign, index);\n LF lfc = HyloHelper.compact(lf, index);\n \tElement lfElt = grammar.makeLfElt(lfc);\n \tnbestrealPW.println(outputter.outputString(lfElt));\n \t\tnbestrealPW.println(\"\");\n \t}\n }\n }\n // close item\n \tnbestrealPW.println(\"\");\n }\n \n \t// if apmldir non-null, output APML as apmldir/ex(i+1).apml\n if (apmldir != null) {\n String apmlfn = apmldir + \"/ex\" + (i+1) + \".apml\";\n grammar.saveToApml(bestEdge.getSign(), apmlfn);\n }\n \n // store best realization, if apropos and grammatically complete, keyed by info string or item position\n if (realserdir != null && gramcomplete) {\n \tString id = testItem.info;\n \tif (id == null) id = \"i\" + i;\n \tbestRealMap.put(testItem.info, bestEdge.getSign());\n }\n \n // compute stats, show outcome\n rCount++;\n totalScore += score; \n if (gramcomplete) totalScoreComplete += score;\n int itemRank = 1;\n Tokenizer tokenizer = grammar.lexicon.tokenizer;\n String itemOrth = tokenizer.getOrthography(tokenizer.tokenize(testItem.sentence));\n if (!bestRealization.equals(itemOrth)) {\n itemRank = 0;\n List bestEdges = chart.bestEdges();\n for (int j = 0; j < bestEdges.size(); j++) {\n Edge edge = bestEdges.get(j);\n String str = edge.getSign().getOrthography();\n if (str.equals(itemOrth)) {\n itemRank = j+1; break;\n }\n }\n if (itemRank > 0) totalReciprocalRank += (1.0 / itemRank);\n \tif (gramcomplete) {\n\t realizeResult = nf.format(score);\n\t if (itemRank > 0 && itemRank < 10) realizeResult += \" \";\n\t if (itemRank > 0 && itemRank < 100) realizeResult += \"#\" + itemRank;\n \t}\n \telse {\n \t\trealizeResult = \"[\" + nf.format(score) + \"]\";\n \t\tif (joined) realizeResult += \"j\";\n \t}\n showOutcome(parseResult, realizeResult, starForBadSentence, testItem.sentence, bestRealization);\n }\n else {\n rExactCount++;\n totalReciprocalRank += 1.0;\n showOutcome(parseResult, realizeResult, starForBadSentence, testItem.sentence);\n }\n \n totalNominals += chart.numNominals;\n int tokens = testItem.sentence.split(\"\\\\s+\").length;\n totalTokens += tokens;\n if (tokens < minTokens || minTokens == 0) minTokens = tokens;\n if (tokens > maxTokens) maxTokens = tokens;\n totalRuleApps += chart.edgeFactory.ruleApps();\n totalEdges += chart.numEdgesInChart();\n totalEdgesCreated += chart.numEdges;\n totalUnprunedEdges += chart.numUnprunedEdges();\n totalPrunedRemoved += chart.numPrunedRemoved;\n totalPrunedNeverAdded += chart.numPrunedNeverAdded;\n totalCellMax += chart.cellMax;\n totalNewBest += chart.newBest;\n \n bestActualScores.add(new Double(score));\n bestEstimatedScores.add(new Double(bestEdge.score));\n itemRanks.add(new Integer(itemRank));\n \n totalLex += chart.timeTilLex;\n if (chart.timeTilLex > maxLex) {\n maxLex = chart.timeTilLex;\n maxLexStr = testItem.sentence;\n }\n lexMap.add(chart.numNominals, chart.timeTilLex);\n \n totalFirst += chart.timeTilFirst;\n if (chart.timeTilFirst > maxFirst) {\n maxFirst = chart.timeTilFirst;\n maxFirstStr = testItem.sentence;\n }\n firstMap.add(chart.numNominals, chart.timeTilFirst);\n \n totalBest += chart.timeTilBest;\n if (chart.timeTilBest > maxBest) {\n maxBest = chart.timeTilBest;\n maxBestStr = testItem.sentence;\n }\n bestMap.add(chart.numNominals, chart.timeTilBest);\n \n if (chart.newBest > 0 && (chart.timeTilBest - chart.timeTilFirst) >= maxNewBest) {\n maxNewBest = chart.timeTilBest - chart.timeTilFirst;\n maxNewBestStr = testItem.sentence;\n }\n \n totalPacked += chart.timeTilPacked;\n if (chart.timeTilPacked > maxPacked) {\n maxPacked = chart.timeTilPacked;\n maxPackedStr = testItem.sentence;\n }\n \n if (chart.done) {\n rDoneCount++;\n totalStoppedOrDone += chart.timeTilDone;\n if (chart.timeTilDone > maxStoppedOrDone) {\n maxStoppedOrDone = chart.timeTilDone;\n maxStoppedOrDoneStr = testItem.sentence;\n }\n allMap.add(chart.numNominals, chart.timeTilDone);\n }\n else {\n totalStoppedOrDone += chart.timeTilStopped;\n if (chart.timeTilStopped > maxStoppedOrDone) {\n maxStoppedOrDone = chart.timeTilStopped;\n maxStoppedOrDoneStr = testItem.sentence;\n }\n allMap.add(chart.numNominals, chart.timeTilStopped);\n }\n }\n \n // end bleu doc (if apropos)\n bleuEndDoc();\n System.out.println();\n \n // serialize best realizations (if apropos)\n realserEndDoc(testName);\n }\n\n /** Shows the various parsing totals. */\n public void showParseStats() {\n \tint pCompleteCount = pCount - pBadCount;\n \tint pFragCount = pBadCount - pFailedCount;\n String pComplete = \"\" + pCompleteCount;\n String pCompletePct = \"\" + nf.format(100.0 * pCompleteCount / pCount) + \"%\";\n System.out.println(\"Strings parsed completely (in fragments, failed): \" + pComplete + \" (\" + pFragCount + \", \" + pFailedCount + \") \" + pCompletePct);\n String pInexact = \"\" + (pCount - pExactCount);\n String pExactPct = \"\" + nf.format(100.0 * pExactCount / pCount) + \"%\";\n System.out.println(\"Strings parsed exactly (inexactly): \" + pExactCount + \" (\" + pInexact + \") \" + pExactPct);\n String avgEdges = nf.format(1.0 * pTotalEdges / pCount);\n System.out.println(\"Average edge count (before unpacking): \" + avgEdges);\n String avgEdgesGood = nf.format(1.0 * pTotalEdgesGood / pCompleteCount);\n System.out.println(\"Average edge count (before unpacking) for complete parses: \" + avgEdgesGood);\n System.out.println(\"Max edge count: \" + pMaxEdges);\n System.out.println(\"Max edge count for complete parses: \" + pMaxEdgesGood);\n String avgUnpacked = nf.format(1.0 * pTotalUnpackingEdges / pCount);\n System.out.println(\"Average edges unpacked (created while unpacking): \" + avgUnpacked);\n System.out.println(\"Max unpacked edges: \" + pMaxUnpackingEdges);\n String avgLexTime = nf.format(1.0 * pTotalLexTime / pCount);\n String avgCellMax = nf.format(1.0 * pTotalCellMax / pCount);\n System.out.println(\"Average max cell size (before unpacking): \" + avgCellMax);\n String avgCellMaxGood = nf.format(1.0 * pTotalCellMaxGood / pCompleteCount);\n System.out.println(\"Average max cell size (before unpacking) for complete parses: \" + avgCellMaxGood);\n System.out.println(\"Max max cell size: \" + pMaxCellMax);\n System.out.println(\"Max max cell size for complete parses: \" + pMaxCellMaxGood);\n System.out.println(\"Average lex lookup time: \" + avgLexTime);\n String avgParseTime = nf.format(1.0 * pTotalParseTime / pCount);\n System.out.println(\"Max lex lookup time: \" + pMaxLexTime);\n System.out.println(\"Average parse time: \" + avgParseTime);\n System.out.println(\"Max parse time: \" + pMaxParseTime);\n String avgChartTime = nf.format(1.0 * pTotalChartTime / pCount);\n System.out.println(\"Average chart construction time: \" + avgChartTime);\n System.out.println(\"Max chart construction time: \" + pMaxChartTime);\n String avgUnpackingTime = nf.format(1.0 * pTotalUnpackingTime / pCount);\n System.out.println(\"Average unpacking time: \" + avgUnpackingTime);\n System.out.println(\"Max unpacking time: \" + pMaxUnpackingTime);\n System.out.println(\"Supertagger beta tallies:\");\n Set betas = pBetaTallies.keySet();\n for (double beta : betas) {\n \tint tally = pBetaTallies.get(beta);\n System.out.println(beta + \"\\t\" + tally);\n }\n String avgF = nf.format(100.0 * totalF / pCount);\n System.out.println(\"Labeled f-score: \" + avgF);\n String avgRecall = nf.format(100.0 * totalRecall / pCount);\n System.out.println(\"Labeled recall: \" + avgRecall);\n String avgPrecision = nf.format(100.0 * totalPrecision / pCount);\n System.out.println(\"Labeled precision: \" + avgPrecision);\n String avgDepsF = nf.format(100.0 * totalDepsF / pCount);\n System.out.println(\"Labeled f-score deps only: \" + avgDepsF);\n String avgDepsRecall = nf.format(100.0 * totalDepsRecall / pCount);\n System.out.println(\"Labeled recall deps only: \" + avgDepsRecall);\n String avgDepsPrecision = nf.format(100.0 * totalDepsPrecision / pCount);\n System.out.println(\"Labeled precision deps only: \" + avgDepsPrecision);\n String avgUnlabeledDepsF = nf.format(100.0 * totalUnlabeledDepsF / pCount);\n System.out.println(\"Unlabeled deps f-score: \" + avgUnlabeledDepsF);\n String avgUnlabeledDepsRecall = nf.format(100.0 * totalUnlabeledDepsRecall / pCount);\n System.out.println(\"Unlabeled deps recall: \" + avgUnlabeledDepsRecall);\n String avgUnlabeledDepsPrecision = nf.format(100.0 * totalUnlabeledDepsPrecision / pCount);\n System.out.println(\"Unlabeled deps precision: \" + avgUnlabeledDepsPrecision);\n String avgFComplete = nf.format(100.0 * totalFComplete / pCompleteCount);\n System.out.println(\"Labeled f-score for complete parses: \" + avgFComplete);\n String avgRecallComplete = nf.format(100.0 * totalRecallComplete / pCompleteCount);\n System.out.println(\"Labeled recall for complete parses: \" + avgRecallComplete);\n String avgPrecisionComplete = nf.format(100.0 * totalPrecisionComplete / pCompleteCount);\n System.out.println(\"Labeled precision for complete parses: \" + avgPrecisionComplete);\n String avgDepsFComplete = nf.format(100.0 * totalDepsFComplete / pCompleteCount);\n System.out.println(\"Labeled f-score deps only for complete parses: \" + avgDepsFComplete);\n String avgDepsRecallComplete = nf.format(100.0 * totalDepsRecallComplete / pCompleteCount);\n System.out.println(\"Labeled recall deps only for complete parses: \" + avgDepsRecallComplete);\n String avgDepsPrecisionComplete = nf.format(100.0 * totalDepsPrecisionComplete / pCompleteCount);\n System.out.println(\"Labeled precision deps only for complete parses: \" + avgDepsPrecisionComplete);\n String avgUnlabeledDepsFComplete = nf.format(100.0 * totalUnlabeledDepsFComplete / pCompleteCount);\n System.out.println(\"Unlabeled deps f-score for complete parses: \" + avgUnlabeledDepsFComplete);\n String avgUnlabeledDepsRecallComplete = nf.format(100.0 * totalUnlabeledDepsRecallComplete / pCompleteCount);\n System.out.println(\"Unlabeled deps recall for complete parses: \" + avgUnlabeledDepsRecallComplete);\n String avgUnlabeledDepsPrecisionComplete = nf.format(100.0 * totalUnlabeledDepsPrecisionComplete / pCompleteCount);\n System.out.println(\"Unlabeled deps precision for complete parses: \" + avgUnlabeledDepsPrecisionComplete);\n if (oracleBetter > 0) System.out.println(\"Oracle better: \" + oracleBetter);\n if (goldMissing > 0) System.out.println(\"Gold missing: \" + goldMissing);\n System.out.println();\n }\n \n /** Shows the various realization totals. */\n public void showStats() {\n \tint rCompleteCount = rCount - rBadCount;\n String rComplete = \"\" + rCompleteCount;\n String rCompletePct = \"\" + nf.format(100.0 * rCompleteCount / rCount) + \"%\";\n System.out.println(\"Strings realized completely (in fragments): \" + rComplete + \" (\" + rBadCount + \") \" + rCompletePct);\n String rInexact = \"\" + (rCount - rExactCount);\n String rExactPct = \"\" + nf.format(100.0 * rExactCount / rCount) + \"%\";\n System.out.println(\"Strings realized exactly (inexactly): \" + rExactCount + \" (\" + rInexact + \") \" + rExactPct);\n System.out.println(\"Strings where realization finished: \" + rDoneCount);\n String avgScore = nf.format(totalScore / rCount);\n System.out.println(\"Avg score: \" + avgScore);\n String avgScoreComplete = nf.format(totalScoreComplete / rCompleteCount);\n System.out.println(\"Avg score for complete realizations: \" + avgScoreComplete);\n String meanReciprocalRank = nf.format(totalReciprocalRank / rCount);\n System.out.println(\"Mean reciprocal rank: \" + meanReciprocalRank);\n String residualMRR = (rCount == rExactCount) \n ? \"n/a\" \n : nf.format((totalReciprocalRank - rExactCount) / (rCount - rExactCount));\n System.out.println(\"Residual mean reciprocal rank: \" + residualMRR);\n String avgNodes = nf.format(totalNominals * 1.0 / rCount); \n String avgTokens = nf.format(totalTokens * 1.0 / rCount); \n System.out.println(\"Avg num nodes, words: \" + avgNodes + \", \" + avgTokens);\n System.out.println(\"Num words (min-max): \" + minTokens + \"-\" + maxTokens);\n String avgRuleApps = nf.format(totalRuleApps * 1.0 / rCount);\n System.out.println(\"Avg num rule apps: \" + avgRuleApps);\n String avgEdges = nf.format(totalEdges * 1.0 / rCount);\n String avgEdgesCreated = nf.format(totalEdgesCreated * 1.0 / rCount);\n String avgUnprunedEdges = nf.format(totalUnprunedEdges * 1.0 / rCount);\n System.out.println(\"Avg num edges in chart: \" + avgEdges);\n System.out.println(\"Avg num edges created: \" + avgEdgesCreated);\n System.out.println(\"Avg num unpruned edges: \" + avgUnprunedEdges);\n String avgRemoved = nf.format(totalPrunedRemoved * 1.0 / rCount);\n String avgNeverAdded = nf.format(totalPrunedNeverAdded * 1.0 / rCount);\n System.out.println(\"Avg num pruned edges removed, never added: \" + avgRemoved + \", \" + avgNeverAdded);\n String avgCellMax = nf.format(totalCellMax * 1.0 / rCount);\n System.out.println(\"Avg cell max: \" + avgCellMax);\n String avgNewBest = nf.format(totalNewBest * 1.0 / rCount);\n System.out.println(\"Total, avg num new best realizations: \" + totalNewBest + \", \" + avgNewBest);\n String avgLex = nf.format(lexMap.mean());\n String stdLex = nf.format(lexMap.sigma());\n System.out.println(\"Avg (std) time 'til lex lookup finished: \" + avgLex + \" (\" + stdLex + \")\");\n System.out.println(\"Max time 'til lex lookup finished: \" + maxLex + \" (\" + maxLexStr + \")\");\n String avgFirst = nf.format(firstMap.mean());\n String stdFirst = nf.format(firstMap.sigma());\n System.out.println(\"Avg (std) time 'til first realization: \" + avgFirst + \" (\" + stdFirst + \")\");\n System.out.println(\"Max time 'til first realization: \" + maxFirst + \" (\" + maxFirstStr + \")\");\n String avgBest = nf.format(bestMap.mean());\n String stdBest = nf.format(bestMap.sigma());\n System.out.println(\"Avg (std) time 'til best realization: \" + avgBest + \" (\" + stdBest + \")\");\n System.out.println(\"Max time 'til best realization: \" + maxBest + \" (\" + maxBestStr +\")\");\n System.out.println(\"Max time 'til new best realization: \" + maxNewBest + \" (\" + maxNewBestStr +\")\");\n String avgPacked = nf.format(totalPacked / rCount);\n System.out.println(\"Avg time 'til done packing: \" + avgPacked);\n System.out.println(\"Max time 'til done packing: \" + maxPacked + \" (\" + maxPackedStr +\")\");\n String avgStoppedOrDone = nf.format(allMap.mean());\n String stdStoppedOrDone = nf.format(allMap.sigma());\n System.out.println(\"Avg (std) time 'til stopped/done with realizations: \" + avgStoppedOrDone + \" (\" + stdStoppedOrDone + \")\");\n System.out.println(\"Max time 'til stopped/done with realizations: \" + maxStoppedOrDone + \" (\" + maxStoppedOrDoneStr +\")\");\n if (oracleBetter > 0) System.out.println(\"Oracle better: \" + oracleBetter);\n if (goldMissing > 0) System.out.println(\"Gold missing: \" + goldMissing);\n if (statsfile != null) {\n Document doc = new Document();\n Element root = new Element(\"rstats\");\n doc.setRootElement(root);\n Element counts = new Element(\"counts\");\n root.addContent(counts);\n counts.setAttribute(\"realized\", \"\" + rCount);\n counts.setAttribute(\"complete\", \"\" + rComplete);\n counts.setAttribute(\"fragments\", \"\" + rBadCount);\n counts.setAttribute(\"exact\", \"\" + rExactCount);\n counts.setAttribute(\"inexact\", rInexact);\n counts.setAttribute(\"finished\", \"\" + rDoneCount);\n Element overall = new Element(\"overall\");\n root.addContent(overall);\n overall.setAttribute(\"avg-score\", avgScore);\n overall.setAttribute(\"mean-reciprocal-rank\", meanReciprocalRank);\n overall.setAttribute(\"residual-mrr\", residualMRR);\n overall.setAttribute(\"avg-nodes\", avgNodes);\n overall.setAttribute(\"avg-words\", avgTokens);\n overall.setAttribute(\"min-words\", \"\" + minTokens);\n overall.setAttribute(\"max-words\", \"\" + maxTokens);\n Element rules = new Element(\"rules\");\n root.addContent(rules);\n rules.setAttribute(\"avg-apps\", avgRuleApps);\n Element edges = new Element(\"edges\");\n root.addContent(edges);\n edges.setAttribute(\"avg\", avgEdges);\n edges.setAttribute(\"avg-unpruned\", avgUnprunedEdges);\n edges.setAttribute(\"avg-removed\", avgRemoved);\n edges.setAttribute(\"avg-never-added\", avgNeverAdded);\n edges.setAttribute(\"avg-cell-max\", avgCellMax);\n if (oracleBetter > 0) edges.setAttribute(\"oracle-better\", \"\" + oracleBetter);\n if (goldMissing > 0) edges.setAttribute(\"gold-missing\", \"\" + goldMissing);\n Element newBest = new Element(\"new-best\");\n root.addContent(newBest);\n newBest.setAttribute(\"total\", \"\" + totalNewBest);\n newBest.setAttribute(\"avg\", avgNewBest);\n Element times = new Element(\"times-summary\");\n root.addContent(times);\n times.setAttribute(\"avg-lex\", avgLex);\n times.setAttribute(\"std-lex\", stdLex);\n times.setAttribute(\"avg-first\", avgFirst);\n times.setAttribute(\"std-first\", stdFirst);\n times.setAttribute(\"max-first\", \"\" + maxFirst);\n times.setAttribute(\"avg-best\", avgBest);\n times.setAttribute(\"std-best\", stdBest);\n times.setAttribute(\"max-best\", \"\" + maxBest);\n times.setAttribute(\"max-new-best\", \"\" + maxNewBest);\n times.setAttribute(\"avg-packed\", avgPacked);\n times.setAttribute(\"max-packed\", \"\" + maxPacked);\n times.setAttribute(\"avg-stopped-or-done\", avgStoppedOrDone);\n times.setAttribute(\"std-stopped-or-done\", stdStoppedOrDone);\n times.setAttribute(\"max-stopped-or-done\", \"\" + maxStoppedOrDone);\n Element strings = new Element(\"max-strings\");\n root.addContent(strings);\n Element lex = new Element(\"lex\");\n strings.addContent(lex);\n lex.addContent(maxLexStr);\n Element first = new Element(\"first\");\n strings.addContent(first);\n first.addContent(maxFirstStr);\n Element best = new Element(\"best\");\n strings.addContent(best);\n best.addContent(maxBestStr);\n Element newBest2 = new Element(\"new-best\");\n strings.addContent(newBest2);\n newBest2.addContent(maxNewBestStr);\n Element packed = new Element(\"packed\");\n strings.addContent(packed);\n packed.addContent(maxPackedStr);\n Element stoppedOrDone = new Element(\"stopped-or-done\");\n strings.addContent(stoppedOrDone);\n stoppedOrDone.addContent(maxStoppedOrDoneStr);\n Element scores = new Element(\"scores\");\n root.addContent(scores);\n for (int i = 0; i < bestActualScores.size(); i++) {\n Element score = new Element(\"score\");\n scores.addContent(score);\n score.setAttribute(\"val\", bestActualScores.get(i).toString());\n score.setAttribute(\"est\", bestEstimatedScores.get(i).toString());\n score.setAttribute(\"rank\", itemRanks.get(i).toString());\n }\n firstMap.saveTimes(root);\n bestMap.saveTimes(root);\n allMap.saveTimes(root);\n try {\n FileOutputStream out = new FileOutputStream(statsfile);\n grammar.serializeXml(doc, out);\n out.flush();\n }\n catch (IOException exc) {\n System.out.println(\"Unable to write stats to: \" + statsfile + \" (\" + exc + \")\");\n }\n }\n }\n \n // show outcome, with wrapping\n private static void showOutcome(String parseResult, String realizeResult, String starForBadSentence, String str) {\n showOutcome(parseResult, realizeResult, starForBadSentence, str, null);\n }\n \n // show outcome including best realization\n private static void showOutcome(String parseResult, String realizeResult, String starForBadSentence, \n String str, String bestRealization) \n {\n System.out.print(parseResult + \"\\t\" + realizeResult + \"\\t\");\n simpleWrap(starForBadSentence + str);\n if (bestRealization != null) {\n System.out.print(\"\\t\\t\");\n simpleWrap(\"(best: \" + bestRealization + \")\");\n }\n }\n \n // does simple wrapping at TEXTWIDTH\n private static void simpleWrap(String str) {\n int TEXTWIDTH = 60;\n for (int i = 0; i <= (str.length()-1)/TEXTWIDTH; i++) {\n if (i != 0) {\n System.out.print(\"\\t\\t\");\n }\n System.out.println(str.substring(i*TEXTWIDTH, Math.min(i*TEXTWIDTH + TEXTWIDTH, str.length())));\n }\n }\n \n // formats to three decimal places\n private static final NumberFormat nf = initNF();\n private static NumberFormat initNF() { \n NumberFormat f = NumberFormat.getInstance();\n f.setMinimumIntegerDigits(1);\n f.setMinimumFractionDigits(1);\n f.setMaximumFractionDigits(2);\n return f;\n }\n \n // formats to three decimal places in scientific notation\n private static final NumberFormat nfE = initNFE();\n private static NumberFormat initNFE() {\n \tDecimalFormat f = new DecimalFormat(\"0.###E0\");\n return f;\n }\n \n /** Shows realizer settings for current test. */\n static void showRealizerSettings() {\n // get, show prefs\n Preferences prefs = Preferences.userNodeForPackage(TextCCG.class);\n boolean useIndexing = prefs.getBoolean(EdgeFactory.USE_INDEXING, true);\n boolean useChunks = prefs.getBoolean(EdgeFactory.USE_CHUNKS, true);\n boolean useLicensing = prefs.getBoolean(EdgeFactory.USE_FEATURE_LICENSING, true);\n boolean useCombos = prefs.getBoolean(opennlp.ccg.realize.Chart.USE_COMBOS, true);\n boolean usePacking = prefs.getBoolean(opennlp.ccg.realize.Chart.USE_PACKING, false);\n int timeLimit = prefs.getInt(opennlp.ccg.realize.Chart.TIME_LIMIT, opennlp.ccg.realize.Chart.NO_TIME_LIMIT);\n double nbTimeLimit = prefs.getDouble(opennlp.ccg.realize.Chart.NEW_BEST_TIME_LIMIT, opennlp.ccg.realize.Chart.NO_TIME_LIMIT);\n int pruningVal = prefs.getInt(opennlp.ccg.realize.Chart.PRUNING_VALUE, opennlp.ccg.realize.Chart.NO_PRUNING);\n int cellPruningVal = prefs.getInt(opennlp.ccg.realize.Chart.CELL_PRUNING_VALUE, opennlp.ccg.realize.Chart.NO_PRUNING);\n String msg = \"Timing realization with index filtering \" + ((useIndexing) ? \"on\" : \"off\") + \", \"; \n msg += \"chunks \" + ((useChunks) ? \"on\" : \"off\") + \", \"; \n msg += \"licensing \" + ((useLicensing) ? \"on\" : \"off\") + \", \";\n if (usePacking) msg += \"packing on, \";\n else {\n msg += \"combos \" + ((useCombos) ? \"on\" : \"off\") + \", \";\n if (timeLimit == opennlp.ccg.realize.Chart.NO_TIME_LIMIT) msg += \"no time limit, \";\n else msg += \"a time limit of \" + timeLimit + \" ms, \";\n if (nbTimeLimit == opennlp.ccg.realize.Chart.NO_TIME_LIMIT) msg += \"no new best time limit, \";\n else {\n msg += \"a new best time limit of \";\n if (nbTimeLimit >= 1) msg += ((int)nbTimeLimit) + \" ms, \";\n else msg += nbTimeLimit + \" of first, \";\n }\n }\n if (pruningVal == opennlp.ccg.realize.Chart.NO_PRUNING) msg += \"no pruning, \";\n else msg += \"a pruning value of \" + pruningVal + \", \";\n msg += \"and \";\n if (cellPruningVal == opennlp.ccg.realize.Chart.NO_PRUNING) msg += \"no cell pruning\";\n else msg += \"a cell pruning value of \" + cellPruningVal;\n System.out.println(msg);\n System.out.println();\n }\n \n\n /** \n * Writes the target strings from the given testbed to the given textfile.\n */\n public void writeTargets(File tbFile, String textfile) throws IOException {\n writeTargets(tbFile, textfile, false, false, false);\n }\n \n /** \n * Writes the target strings with semantic class replacement \n * from the given testbed to the given textfile. \n */\n public void writeTargetsSC(File tbFile, String textfile) throws IOException {\n writeTargets(tbFile, textfile, true, false, false);\n }\n \n /** \n * Writes the target strings with all associated factors \n * from the given testbed to the given textfile. \n */\n public void writeTargetsF(File tbFile, String textfile) throws IOException {\n writeTargets(tbFile, textfile, false, true, false);\n }\n \n /** \n * Writes the target strings with all associated factors with semantic class replacement \n * from the given testbed to the given textfile. \n */\n public void writeTargetsFSC(File tbFile, String textfile) throws IOException {\n writeTargets(tbFile, textfile, true, true, false);\n }\n \n // writes targets, optionally with sem class replacement or factors, \n // and optionally reversing the words; ungrammatical options are filtered out \n private void writeTargets(\n \t\tFile tbFile, String filename, \n \t\tboolean semClassReplacement, boolean withFactors, \n \t\tboolean reverse\n ) throws IOException {\n // open text file\n String option = \"\";\n if (withFactors) option = \" with factors\";\n if (semClassReplacement) option += \" with semantic class replacement\";\n if (reverse) option += \", reversed\";\n System.out.println(\"Writing text file\" + option + \": \" + filename);\n System.out.println();\n PrintWriter tOut = new PrintWriter(new BufferedWriter(new FileWriter(filename)));\n HashSet unique = new HashSet(); \n Tokenizer tokenizer = grammar.lexicon.tokenizer;\n // loop through files\n for (File f : getXMLFiles(tbFile)) {\n\t // load testbed\n\t System.out.println(\"Loading testbed from: \" + f);\n\t RegressionInfo tbInfo = new RegressionInfo(grammar, f);\n\t int numItems = tbInfo.numberOfItems();\n\t // do each test item\n\t for (int i = 0; i < numItems; i++) {\n\t // check even/odd only\n\t if (i % 2 == 1 && evenOnly) continue;\n\t if (i % 2 == 0 && oddOnly) continue;\n\t RegressionInfo.TestItem testItem = tbInfo.getItem(i); \n\t \t// check grammatical\n\t \tif (testItem.numOfParses == 0) continue;\n\t String s = testItem.sentence;\n\t // get parsed words if doing more than just text\n\t List words = null;\n\t if (semClassReplacement || withFactors) {\n\t // use words from sign or pre-parsed full words if available\n\t \tif (testItem.sign != null) \n\t \t\twords = testItem.sign.getWords();\n\t \telse if (testItem.fullWords != null) \n\t words = tokenizer.tokenize(testItem.fullWords, true);\n\t // otherwise parse\n\t else words = grammar.getParsedWords(s);\n\t }\n\t else words = tokenizer.tokenize(s);\n\t // reverse, if apropos\n\t if (reverse) {\n\t List tmp = words;\n\t words = new ArrayList(words.size());\n\t words.add(Word.createWord(\"\"));\n\t for (int j = tmp.size()-1; j >= 0; j--) {\n\t Word w = tmp.get(j);\n\t if (w.getForm() == \"\" || w.getForm() == \"\") continue; // skip or \n\t words.add(w);\n\t }\n\t words.add(Word.createWord(\"\"));\n\t }\n\t // write str, add to unique set\n\t String str = (!withFactors)\n\t ? tokenizer.getOrthography(words, semClassReplacement)\n\t : tokenizer.format(words, semClassReplacement);\n\t tOut.println(str);\n\t unique.add(str);\n\t System.out.print(\".\"); // indicate progress\n\t }\n\t System.out.println();\n }\n tOut.flush(); tOut.close();\n System.out.println();\n System.out.println(\"Unique strings: \" + unique.size());\n System.out.println();\n }\n \n private void writeDerivationFactors(File tbFile, String filename) throws IOException {\n // open text file\n System.out.println(\"Writing derivation factors file: \" + filename);\n System.out.println();\n PrintWriter tOut = new PrintWriter(new BufferedWriter(new FileWriter(filename)));\n Tokenizer tokenizer = grammar.lexicon.tokenizer;\n // loop through files\n for (File f : getXMLFiles(tbFile)) {\n\t // load testbed\n\t System.out.println(\"Loading testbed from: \" + f);\n\t RegressionInfo tbInfo = new RegressionInfo(grammar, f);\n\t int numItems = tbInfo.numberOfItems();\n\t // do each test item, using the saved sign\n\t for (int i = 0; i < numItems; i++) {\n\t RegressionInfo.TestItem testItem = tbInfo.getItem(i); \n\t \tif (testItem.numOfParses == 0) continue; // check grammatical\n\t \tSign sign = testItem.sign;\n\t \tList factors = GenerativeSyntacticModel.getFactors(sign);\n\t \tfor (Word w : factors) {\n\t \t\ttOut.print(tokenizer.format(w));\n\t \t\ttOut.print(\" \");\n\t \t}\n\t \ttOut.println();\n\t System.out.print(\".\"); // indicate progress\n\t }\n\t System.out.println();\n }\n tOut.flush(); tOut.close();\n System.out.println();\n }\n \n \n /** Command-line routine for regression testing. */\n public static void main(String[] args) throws IOException { \n\n String usage = \"java opennlp.ccg.test.Regression \\n\" + \n \" (-noparsing) (-norealization) (-even|-odd) (-gc) \\n\" + \n \" (-nullscorer) (-randomscorer) \\n\" + \n \" (-depthfirst) (-exactmatches) (-aanfilter ()) \\n\" +\n \" (-scorer ) \\n\" +\n \" (-parsescorer ) \\n\" +\n \" (-extractor ) \\n\" +\n \" (-ngrampruningstrategy) (-pruningstrategy ) \\n\" +\n \" (-hypertagger | -htconfig ) (-htgold) \\n\" +\n \" (-supertagger | -stconfig ) \\n\" +\n \" (-ngramorder N) (-lm|-lmsc ) \\n\" + \n \" (-srilm \" + Arrays.toString(SRILMNgramModelType.values()) + \")\\n\"+\n \" (-flm|-flmsc ) \\n\" + \n \" (-text|-textsc|-textf|-textfsc ) (-reverse) \\n\" +\n \" (-derivf ) \\n\" +\n \" (-2events ) (-includegoldinevents) \\n\" +\n \" (-2apml ) (-bleu ) \\n\" +\n \" (-nbestrealfile ) (-nbestnormbleu) (-realserdir ) \\n\" + \n \" (-nbestincludelfs) \\n\" +\n \" (-rescorefile ) \\n\" + \n \" (-nbestparsefile ) \\n\" + \n \" (-g ) (-s ) (|)\";\n \n if (args.length > 0 && args[0].equals(\"-h\")) {\n System.out.println(\"Usage: \\n\\n\" + usage);\n System.exit(0);\n }\n \n // setup Regression tester\n Regression tester = new Regression();\n\n // args\n String grammarfile = \"grammar.xml\";\n String regressionfile = \"testbed.xml\";\n boolean depthFirst = false;\n boolean aanfilter = false;\n String excfile = null;\n String scorerClass = null;\n String parseScorerClass = null;\n String extractorClass = null;\n boolean ngrampruningstrategy = false;\n String pruningStrategyClass = null; \n String hypertaggerClass = null, htconfig = null; \n boolean htgold = false;\n String supertaggerClass = null, stconfig = null;\n String lmfile = null;\n String flmfile = null;\n boolean useSemClasses = false;\n boolean withFactors = false;\n boolean reverse = false;\n String textfile = null;\n String derivfactorsfile = null;\n boolean srilm = false;\n SRILMNgramModelType srilmModelType = SRILMNgramModelType.STANDARD;\n \n for (int i = 0; i < args.length; i++) {\n \tif (args[i].startsWith(\"-D\")) {\n \t\tString prop = args[i].substring(2); int equalpos = prop.indexOf(\"=\");\n \t\tString key = prop.substring(0, equalpos); String val = prop.substring(equalpos+1);\n \t\tSystem.setProperty(key, val); continue;\n \t}\n if (args[i].equals(\"-noparsing\")) { tester.doParsing = false; continue; }\n if (args[i].equals(\"-norealization\")) { tester.doRealization = false; continue; }\n if (args[i].equals(\"-even\")) { tester.evenOnly = true; continue; }\n if (args[i].equals(\"-odd\")) { tester.oddOnly = true; continue; }\n if (args[i].equals(\"-gc\")) { tester.doGC = true; continue; }\n if (args[i].equals(\"-nullscorer\")) { \n \ttester.scorer = SignScorer.nullScorer; tester.parseScorer = SignScorer.nullScorer; continue; \n }\n if (args[i].equals(\"-randomscorer\")) { \n \ttester.scorer = SignScorer.randomScorer; tester.parseScorer = SignScorer.randomScorer; continue; \n }\n if (args[i].equals(\"-depthfirst\")) { depthFirst = true; continue; }\n if (args[i].equals(\"-exactmatches\")) { tester.exactMatches = true; continue; }\n if (args[i].equals(\"-aanfilter\")) {\n aanfilter = true; \n if (i < args.length-1 && args[i+1].charAt(0) != '-') excfile = args[++i]; \n continue;\n }\n if (args[i].equals(\"-scorer\")) { scorerClass = args[++i]; continue; }\n if (args[i].equals(\"-parsescorer\")) { parseScorerClass = args[++i]; continue; }\n if (args[i].equals(\"-extractor\")) { extractorClass = args[++i]; continue; }\n if (args[i].equals(\"-ngrampruningstrategy\")) { ngrampruningstrategy = true; continue; }\n if (args[i].equals(\"-pruningstrategy\")) { pruningStrategyClass = args[++i]; continue; }\n if (args[i].equals(\"-hypertagger\")) { hypertaggerClass = args[++i]; continue; }\n if (args[i].equals(\"-htconfig\")) { htconfig = args[++i]; continue; }\n if (args[i].equals(\"-htgold\")) { htgold = true; continue; }\n if (args[i].equals(\"-supertagger\")) { supertaggerClass = args[++i]; continue; }\n if (args[i].equals(\"-stconfig\")) { stconfig = args[++i]; continue; }\n if (args[i].equals(\"-ngramorder\")) { tester.ngramOrder = Integer.parseInt(args[++i]); continue; }\n if (args[i].equals(\"-lm\")) { lmfile = args[++i]; continue; }\n if (args[i].equals(\"-lmsc\")) { lmfile = args[++i]; useSemClasses = true; continue; }\n if (args[i].equals(\"-flm\")) { flmfile = args[++i]; continue; }\n if (args[i].equals(\"-flmsc\")) { flmfile = args[++i]; useSemClasses = true; continue; }\n if (args[i].equals(\"-reverse\")) { reverse = true; continue; }\n if (args[i].equals(\"-text\")) { textfile = args[++i]; continue; }\n if (args[i].equals(\"-textsc\")) { textfile = args[++i]; useSemClasses = true; continue; }\n if (args[i].equals(\"-textf\")) { textfile = args[++i]; withFactors = true; continue; }\n if (args[i].equals(\"-textfsc\")) { textfile = args[++i]; useSemClasses = true; withFactors = true; continue; }\n if (args[i].equals(\"-derivf\")) { derivfactorsfile = args[++i]; continue; }\n if (args[i].equals(\"-2events\")) { tester.eventfile = args[++i]; continue; }\n if (args[i].equals(\"-includegoldinevents\")) { tester.includeGoldInEvents = true; continue; }\n if (args[i].equals(\"-2apml\")) { tester.apmldir = args[++i]; continue; }\n if (args[i].equals(\"-bleu\")) { tester.bleufileprefix = args[++i]; continue; }\n if (args[i].equals(\"-nbestrealfile\")) { tester.nbestrealfile = args[++i]; continue; }\n if (args[i].equals(\"-nbestnormbleu\")) { tester.nbestnormbleu = true; continue; }\n if (args[i].equals(\"-realserdir\")) { tester.realserdir = args[++i]; continue; }\n if (args[i].equals(\"-nbestincludelfs\")) { tester.nbestincludelfs = true; continue; }\n if (args[i].equals(\"-rescorefile\")) { tester.rescorefile = args[++i]; continue; }\n if (args[i].equals(\"-nbestparsefile\")) { tester.nbestparsefile = args[++i]; continue; }\n if (args[i].equals(\"-g\")) { grammarfile = args[++i]; continue; }\n if (args[i].equals(\"-s\")) { tester.statsfile = args[++i]; continue; }\n if (args[i].equals(\"-srilm\")) { \n \tsrilm = true;\n \tif(i < (args.length - 1)) {\n\t \tString type = args[i + 1];\n\t \ttry {\n\t \t\tsrilmModelType = SRILMNgramModelType.valueOf(type);\n\t \t\ti++;\n\t \t}\n\t \tcatch(IllegalArgumentException iae) {\n\t \t\tsrilmModelType = SRILMNgramModelType.STANDARD;\n\t \t\tSystem.err.println(\n\t \t\t\t\"Warning: unknown SRILM n-gram model type \" + type\n\t \t\t\t\t+ \" specified, using default (\"\n\t \t\t\t\t+ srilmModelType + \")\");\n\t \t}\n \t}\n \tcontinue;\n }\n regressionfile = args[i];\n }\n \n // load grammar\n URL grammarURL = new File(grammarfile).toURI().toURL();\n System.out.println(\"Loading grammar from URL: \" + grammarURL);\n tester.grammar = new Grammar(grammarURL);\n System.out.println();\n \n // with -aanfilter ( 0) ? tester.ngramOrder : 3;\n String reversedStr = (reverse) ? \"reversed \" : \"\";\n System.out.println(\"Loading \" + reversedStr + order\n \t\t+ \"-gram model from file: \" + lmfile);\n NgramScorer lmScorer = (srilm)\n \t? new SRILMNgramModel(order, new File(lmfile), useSemClasses, \n \t\t\tsrilmModelType)\n \t: new StandardNgramModel(order, lmfile, useSemClasses);\n if (reverse) lmScorer.setReverse(true);\n if (aanfilter) lmScorer.addFilter(aanFilter);\n tester.scorer = lmScorer;\n System.out.println();\n }\n\n // with -flm|-flmsc options, load factored n-gram model family\n if (flmfile != null) {\n String reversedStr = (reverse) ? \"reversed \" : \"\";\n System.out.println(\"Loading \" + reversedStr + \"factored n-gram model family from file: \" + flmfile);\n NgramScorer flmScorer = new FactoredNgramModelFamily(flmfile, useSemClasses);\n if (reverse) flmScorer.setReverse(true);\n if (aanfilter) flmScorer.addFilter(aanFilter);\n tester.scorer = flmScorer;\n tester.ngramOrder = flmScorer.getOrder();\n System.out.println();\n }\n\n // with -text|-textsc|-textf|-textfsc options, just write text file and exit\n if (textfile != null) {\n \tFile tbFile = new File(regressionfile);\n tester.writeTargets(tbFile, textfile, useSemClasses, withFactors, reverse);\n System.exit(0);\n }\n \n // with -derivf option, just write derivation factors file and exit\n if (derivfactorsfile != null) {\n \tFile tbFile = new File(regressionfile);\n \ttester.writeDerivationFactors(tbFile, derivfactorsfile);\n System.exit(0);\n }\n \n // setup parser\n if (tester.doParsing) {\n tester.parser = new Parser(tester.grammar);\n // instantiate scorer, if any\n if (parseScorerClass != null) {\n try {\n System.out.println(\"Instantiating parsing sign scorer from class: \" + parseScorerClass);\n tester.parseScorer = (SignScorer) Class.forName(parseScorerClass).newInstance();\n tester.showParseStats = true; // turn parsing stats on\n System.out.println();\n } catch (Exception exc) {\n throw (RuntimeException) new RuntimeException().initCause(exc);\n }\n }\n // set parser scorer, if any\n if (tester.parseScorer != null) tester.parser.setSignScorer(tester.parseScorer);\n // also turn on parse stats if doing n-best output\n if (tester.nbestparsefile != null) tester.showParseStats = true;\n // instantiate supertagger, if any\n if (supertaggerClass != null || stconfig != null) {\n try {\n \tSupertagger supertagger;\n \tif (supertaggerClass != null) {\n\t System.out.println(\"Instantiating supertagger from class: \" + supertaggerClass);\n\t supertagger = (Supertagger) Class.forName(supertaggerClass).newInstance();\n \t}\n \telse {\n \t\tSystem.out.println(\"Instantiating supertagger from config file: \" + stconfig);\n \t\tsupertagger = WordAndPOSDictionaryLabellingStrategy.supertaggerFactory(stconfig);\n \t}\n tester.parser.setSupertagger(supertagger);\n if (tester.eventfile != null) {\n \tsupertagger.setIncludeGold(true); // use gold tags during training\n \ttester.parser.setSupertaggerMostToLeastRestrictiveDirection(false); // reverse direction to start with least restrictive setting\n }\n System.out.println();\n } catch (Exception exc) {\n throw (RuntimeException) new RuntimeException().initCause(exc);\n }\n }\n }\n \n // setup realizer, show settings\n if (tester.doRealization) {\n tester.realizer = new Realizer(tester.grammar);\n tester.realizer.depthFirst = depthFirst;\n // instantiate pruning strategy, if any\n if (ngrampruningstrategy) {\n int order = (tester.ngramOrder > 0) ? tester.ngramOrder : 3;\n System.out.println(\"Instantiating n-gram diversity pruning strategy with order \" + order);\n tester.realizer.pruningStrategy = new NgramDiversityPruningStrategy(order);\n System.out.println();\n }\n if (pruningStrategyClass != null) {\n try {\n System.out.println(\"Instantiating pruning strategy from class: \" + pruningStrategyClass);\n tester.realizer.pruningStrategy = (PruningStrategy) Class.forName(pruningStrategyClass).newInstance();\n System.out.println();\n } catch (Exception exc) {\n throw (RuntimeException) new RuntimeException().initCause(exc);\n }\n }\n if (hypertaggerClass != null || htconfig != null) {\n try {\n \tHypertagger hypertagger;\n \tif (hypertaggerClass != null) {\n System.out.println(\"Instantiating hypertagger from class: \" + hypertaggerClass);\n hypertagger = (Hypertagger) Class.forName(hypertaggerClass).newInstance();\n \t}\n \telse {\n \t\tSystem.out.println(\"Instantiating hypertagger from config file: \" + htconfig);\n \t\thypertagger = ZLMaxentHypertagger.ZLMaxentHypertaggerFactory(htconfig);\n \t}\n tester.realizer.hypertagger = hypertagger;\n if (tester.eventfile != null) {\n \thypertagger.setIncludeGold(true); // use gold tags during training\n // NB: could investigate most-to-least-restrictive direction\n }\n if (htgold) hypertagger.setIncludeGold(true); // use gold tags, eg for paraphrasing\n System.out.println();\n } catch (Exception exc) {\n throw (RuntimeException) new RuntimeException().initCause(exc);\n }\n }\n showRealizerSettings();\n }\n \n // ensure dir for event file exists; \n // set up feature extractor\n if (tester.eventfile != null) {\n \t\tFile tmp = new File(tester.eventfile); \n \t\tFile tmpParent = tmp.getParentFile(); \n \t\tif (tmpParent != null) tmpParent.mkdirs(); \n System.out.println(\"Writing event file to: \" + tester.eventfile);\n System.out.println();\n }\n \n // instantiate feature extractor, if generating events \n if (tester.eventfile != null) {\n \t// ensure just doing parsing or realization\n \tif (tester.doParsing && tester.doRealization) {\n \t\tthrow new RuntimeException(\"Events can't be generated for parsing and realization at the same time.\");\n \t}\n \tFeatureExtractor extractor = null;\n \tif (extractorClass != null) {\n\t try {\n\t System.out.println(\"Instantiating feature extractor from class: \" + extractorClass);\n\t extractor = (FeatureExtractor) Class.forName(extractorClass).newInstance();\n\t tester.featureExtractor = extractor;\n\t System.out.println();\n\t } catch (Exception exc) {\n\t throw (RuntimeException) new RuntimeException().initCause(exc);\n\t }\n \t}\n \t// use or combine with scorer, if it's also a feature extractor\n if (tester.scorer instanceof FeatureExtractor) {\n \tif (extractor != null)\n \t\ttester.featureExtractor = new ComposedFeatureExtractor(\n \t\t\t\t\tnew FeatureExtractor[] {\n \t\t\t\t\t\t\t(FeatureExtractor)tester.scorer, extractor\n \t\t\t\t\t}\n \t\t\t\t);\n \telse tester.featureExtractor = (FeatureExtractor) tester.scorer;\n }\n else if (tester.parseScorer instanceof FeatureExtractor) {\n \tif (extractor != null)\n \t\ttester.featureExtractor = new ComposedFeatureExtractor(\n \t\t\t\t\tnew FeatureExtractor[] {\n \t\t\t\t\t\t\t(FeatureExtractor)tester.parseScorer, extractor\n \t\t\t\t\t}\n \t\t\t\t);\n \telse tester.featureExtractor = (FeatureExtractor) tester.parseScorer;\n }\n // otherwise use an n-gram precision model \n if (tester.featureExtractor == null)\n \ttester.featureExtractor = new NgramPrecisionModel(new String[]{\"\"}, true);\n // set new alphabet\n tester.featureExtractor.setAlphabet(new Alphabet(10000));\n }\n \n // ensure apmldir exists\n if (tester.apmldir != null) {\n File apmlDir = new File(tester.apmldir);\n if (!apmlDir.exists()) { apmlDir.mkdirs(); }\n System.out.println(\"Writing APML files to dir: \" + tester.apmldir);\n System.out.println();\n }\n\n // ensure dir for bleu files exists\n if (tester.bleufileprefix != null) {\n \t\tFile tmp = new File(tester.bleufileprefix + \"-gen.sgm\"); \n \t\tFile tmpParent = tmp.getParentFile(); \n \t\tif (tmpParent != null) tmpParent.mkdirs(); \n System.out.println(\"Writing BLEU files to: \" + tester.bleufileprefix + \"-*.sgm\");\n System.out.println();\n }\n \n // ensure dir for nbestrealfile exists\n if (tester.nbestrealfile != null) {\n \t\tFile tmp = new File(tester.nbestrealfile);\n \t\tFile tmpParent = tmp.getParentFile(); \n \t\tif (tmpParent != null) tmpParent.mkdirs(); \n System.out.println(\"Writing N-best realizations to: \" + tester.nbestrealfile);\n System.out.println();\n }\n \n // ensure realserdir exists\n if (tester.realserdir != null) {\n File realserDir = new File(tester.realserdir);\n if (!realserDir.exists()) { realserDir.mkdirs(); }\n System.out.println(\"Writing best realization serialization files to dir: \" + tester.realserdir);\n System.out.println();\n }\n\n // ensure dir for rescorefile exists\n if (tester.rescorefile != null) {\n \t\tFile tmp = new File(tester.rescorefile);\n \t\tFile tmpParent = tmp.getParentFile(); \n \t\tif (tmpParent != null) tmpParent.mkdirs(); \n System.out.println(\"Writing rescored sign scores to: \" + tester.rescorefile);\n System.out.println();\n }\n \n // run test\n tester.runTest(new File(regressionfile));\n }\n}\n" }, { "path": "src/opennlp/ccg/test/RegressionInfo.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-4 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.test;\n\nimport opennlp.ccg.grammar.Grammar;\nimport opennlp.ccg.synsem.*;\n\nimport org.jdom.*;\nimport org.jdom.input.*;\nimport java.io.*;\nimport java.util.*;\n\n/**\n * Manages the info in a regression test file.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.16 $, $Date: 2011/08/27 19:27:01 $\n */\npublic class RegressionInfo {\n\n // the grammar\n private Grammar grammar;\n \n // the test items\n private TestItem[] testItems;\n \n /** Test item. */\n public class TestItem {\n /** The test sentence/phrase. */\n public String sentence;\n /** An alternative paraphrase to target, or null if none. */\n public String alt = null;\n /** The desired number of parses. */\n public int numOfParses = 1;\n /** Whether the sentence/phrase is known to fail to parse. */\n public boolean knownFailure = false;\n /** The full words for the sentence/phrase, or null if none, formatted by the configured tokenizer. */\n public String fullWords = null;\n /** The LF, in XML, for the sentence/phrase, or null if none. */\n public Element lfElt = null;\n /** Any additionally id info, or null if none. */\n public String info = null;\n\t\t/** The gold std nominal id name & supertag for LF predicates. @deprecated Should use fullWords. */\n\t\tpublic String predInfo = null;\n\t\t/** The gold standard derivation. */\n\t\tpublic Sign sign = null;\n\t\t/** Returns the id from info, without prefixed \"ID=\" if present. */\n\t\tpublic String getId() {\n\t\t\tif (info == null) return null;\n\t\t\tint pos = info.indexOf('=');\n\t\t\treturn (pos < 0) ? info : info.substring(pos+1);\n\t\t}\n }\n\n /** Reads in the given regression test file and corresponding .ser file, if any. */ \n public RegressionInfo(Grammar grammar, File regressionFile) throws FileNotFoundException {\n \tthis(grammar, new FileInputStream(regressionFile), serStream(regressionFile));\n }\n \n /** Reads in a regression test from the given input stream and object input stream. */ \n\t@SuppressWarnings(\"unchecked\")\n\tpublic RegressionInfo(Grammar grammar, InputStream istr, ObjectInputStream serStream) {\n this.grammar = grammar;\n SAXBuilder builder = new SAXBuilder();\n try {\n \tMap signMap = readSerStream(serStream);\n Document doc = builder.build(istr);\n Element root = doc.getRootElement();\n List items = root.getChildren(\"item\");\n testItems = new TestItem[items.size()];\n for (int i = 0; i < items.size(); i++) {\n Element item = (Element) items.get(i);\n TestItem testItem = new TestItem();\n testItems[i] = testItem;\n testItem.sentence = item.getAttributeValue(\"string\");\n testItem.alt = item.getAttributeValue(\"alt\");\n testItem.numOfParses = Integer.parseInt(item.getAttributeValue(\"numOfParses\"));\n testItem.knownFailure = (\"true\".equals(item.getAttributeValue(\"known\"))) ? true : false;\n Element fullWordsElt = item.getChild(\"full-words\");\n if (fullWordsElt != null) testItem.fullWords = fullWordsElt.getTextNormalize();\n testItem.lfElt = item.getChild(\"lf\");\n testItem.info = item.getAttributeValue(\"info\");\n\t\t\t\tElement predInfoElt = item.getChild(\"pred-info\");\n\t\t\t\tif (predInfoElt != null) testItem.predInfo = predInfoElt.getAttributeValue(\"data\");\n\t\t\t\tif (signMap != null && testItem.info != null) testItem.sign = signMap.get(testItem.info);\n }\n } catch (Exception e) {\n throw (RuntimeException) new RuntimeException().initCause(e);\n }\n }\n\n \n /** Returns the corresponding .ser file for loading sign objects. */\n public static File serFile(File regressionFile) {\n \tString name = regressionFile.getName();\n \tString prefix = name.substring(0, name.lastIndexOf('.'));\n \treturn new File(regressionFile.getParentFile(), prefix + \".ser\");\n }\n\n /** Returns object input stream for corresponding .ser file, or null if none. */\n public static ObjectInputStream serStream(File regressionFile) {\n \tFile serFile = serFile(regressionFile);\n \tif (serFile.exists()) {\n\t\t\ttry {\n\t\t\t\treturn new ObjectInputStream(new FileInputStream(serFile));\n\t\t\t} catch (FileNotFoundException e) {\n\t throw (RuntimeException) new RuntimeException().initCause(e); \n\t\t\t} catch (IOException e) {\n\t throw (RuntimeException) new RuntimeException().initCause(e);\n\t\t\t}\n \t}\n\t\telse return null;\n }\n \n /** Reads in a map of info keys and gold standard signs from the given stream, or returns null if the stream is null. */\n @SuppressWarnings(\"unchecked\")\n\tpublic static Map readSerStream(ObjectInputStream serStream) throws IOException {\n \tif (serStream == null) return null;\n \ttry {\n\t\t\treturn (Map) serStream.readObject();\n\t\t} catch (ClassNotFoundException e) {\n throw (RuntimeException) new RuntimeException().initCause(e);\n\t\t}\n }\n \n /** Writes the map of info keys and gold standard signs to the corresponding .ser file. */\n public static void writeSerFile(Map signMap, File regressionFile) throws FileNotFoundException, IOException {\n \tFile serFile = serFile(regressionFile);\n \tObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(serFile));\n \toos.writeObject(signMap);\n \toos.close();\n }\n \n \n /** Returns the number of test items. */\n public int numberOfItems() {\n return testItems.length;\n }\n \n /** Returns the test item with the given index. */ \n public TestItem getItem(int i) {\n return testItems[i];\n }\n \n \n /**\n * Makes an XML test item from the given test item object.\n */\n public static Element makeTestItem(TestItem testItem) {\n Element item = new Element(\"item\");\n item.setAttribute(\"numOfParses\", \"\" + testItem.numOfParses);\n if (testItem.knownFailure) item.setAttribute(\"known\", \"true\");\n item.setAttribute(\"string\", testItem.sentence);\n if (testItem.alt != null) item.setAttribute(\"alt\", testItem.alt);\n if (testItem.fullWords != null) {\n Element fullWordsElt = new Element(\"full-words\");\n item.addContent(fullWordsElt);\n fullWordsElt.addContent(testItem.fullWords);\n }\n if (testItem.lfElt != null) {\n testItem.lfElt.detach();\n item.addContent(testItem.lfElt);\n }\n if (testItem.info != null) item.setAttribute(\"info\", testItem.info);\n return item;\n }\n \n /**\n * Makes an XML test item with the given string, number of parses and LF, \n * applying the configured to-XML transformations.\n */\n public Element makeTestItem(String target, int numParses, LF lf) throws IOException { \n return makeTestItem(grammar, target, numParses, lf);\n }\n \n /**\n * Makes an XML test item with the given string, number of parses and LF, \n * applying the configured to-XML transformations.\n */\n public static Element makeTestItem(Grammar grammar, String target, int numParses, LF lf) throws IOException { \n return makeTestItem(grammar, target, numParses, lf, null);\n }\n\n /**\n * Makes an XML test item with the given string, number of parses, LF and info attribute, \n * applying the configured to-XML transformations.\n */\n public static Element makeTestItem(Grammar grammar, String target, int numParses, LF lf, String info) throws IOException { \n Element item = new Element(\"item\");\n item.setAttribute(\"numOfParses\", \"\" + numParses);\n item.setAttribute(\"string\", target);\n if (lf != null) item.addContent(grammar.makeLfElt(lf));\n if (info != null) item.setAttribute(\"info\", info);\n return item;\n }\n\n /**\n * Adds the given sign with its string, number of parses and converted LF \n * as a test item to the testbed with the given filename, applying the configured to-XML\n * transformations.\n */\n public static void addToTestbed(Grammar grammar, Sign sign, int numParses, LF lf, String filename) throws IOException { \n\n // ensure dirs exist for filename\n File file = new File(filename);\n File parent = file.getParentFile();\n if (parent != null && !parent.exists()) { parent.mkdirs(); }\n \n // load or make doc\n Document doc; \n Element root;\n boolean newDoc = false;\n if (file.exists()) {\n // read XML\n SAXBuilder builder = new SAXBuilder();\n try {\n doc = builder.build(file);\n } catch (JDOMException jde) {\n throw (IOException) new IOException().initCause(jde);\n }\n root = doc.getRootElement();\n }\n else {\n doc = new Document();\n root = new Element(\"regression\");\n doc.setRootElement(root);\n newDoc = true;\n }\n \n // load or make sign map\n \tMap signMap = readSerStream(serStream(file));\n \tif (signMap == null) signMap = new HashMap();\n \n \t// find unique id\n \tint count = 0;\n \tString id = \"i\" + count;\n \twhile (signMap.containsKey(id)) id = \"i\" + ++count;\n \t\n // make test item\n \tString target = sign.getOrthography();\n Element item = makeTestItem(grammar, target, numParses, lf, id);\n \n // append new item\n if (!newDoc) root.addContent(\" \"); // nb: for some reason, this gets the indenting right\n root.addContent(item);\n \n // add sign to map\n signMap.put(id, sign);\n \n // save\n FileOutputStream out = new FileOutputStream(file); \n grammar.serializeXml(doc, out);\n out.close();\n writeSerFile(signMap, file);\n }\n}\n" }, { "path": "src/opennlp/ccg/test/ScorerMaker.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.test;\n\nimport opennlp.ccg.synsem.SignScorer;\n\nimport java.io.*;\n\n/**\n * Interface for making custom scorers for use in cross-validation tests with the realizer.\n *\n * @author Michael White\n * @version $Revision: 1.6 $, $Date: 2007/12/21 05:13:37 $\n */\npublic interface ScorerMaker {\n\n /**\n * Sets the context for this scorer maker.\n */\n public void setCVR(CrossValidateRealizer cvr);\n \n /**\n * Prepares a scoring model from the training data. \n * The data can be accessed by creating a RegressionInfo \n * object from the given training file.\n * The test data is also made available to optionally \n * compute perplexity or other measures.\n * The model can be stored in a file in tmpDir, keyed off of foldNum.\n */\n public void prepScorer(File tmpDir, int foldNum, File trainFile, File testFile) throws IOException;\n \n /**\n * Optionally summarizes perplexity or other measures \n * after all calls to prepScorer.\n */\n public void prepScorersSummary(File tmpDir) throws IOException;\n \n /**\n * Loads a scoring model created from the training data. \n */\n public SignScorer loadScorer(File tmpDir, int foldNum, File trainFile) throws IOException;\n}\n\n\n" }, { "path": "src/opennlp/ccg/test/TimingMap.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-4 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.test;\n\nimport java.util.*;\n\nimport org.jdom.*;\n\n/**\n * Utility class for managing average times per number of nodes.\n *\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2009/12/21 04:18:31 $\n */\npublic class TimingMap {\n \n private String label;\n private List times = new ArrayList();\n private HashMap> map = new HashMap>();\n \n /** Constructor, with label. */\n public TimingMap(String label) {\n this.label = label;\n }\n \n /** Adds a number, time pair. */\n public void add(int num, int time) {\n Integer timeInt = new Integer(time);\n times.add(timeInt); \n Integer key = new Integer(num);\n List timesPerNum = map.get(key);\n if (timesPerNum == null) {\n timesPerNum = new ArrayList();\n map.put(key, timesPerNum);\n }\n timesPerNum.add(timeInt);\n }\n \n /** Returns the mean time. */\n public double mean() {\n int total = 0;\n for (int i = 0; i < times.size(); i++) {\n Integer time = times.get(i);\n total += time.intValue();\n }\n return (1.0 * total) / times.size(); \n }\n \n /** Returns the standard deviation. */\n public double sigma() {\n if (times.size() < 2) return -1; // NA\n double mean = mean();\n double numerator = 0;\n for (int i = 0; i < times.size(); i++) {\n Integer time = times.get(i);\n numerator += Math.pow(time.intValue() - mean, 2);\n }\n int denominator = times.size() - 1;\n return Math.sqrt(numerator / denominator);\n }\n \n /** Saves the times and times per number (with average) as XML elements under the given one. */\n public void saveTimes(Element root) {\n Element timesElt = new Element(\"times\");\n root.addContent(timesElt);\n timesElt.setAttribute(\"label\", label);\n Element listElt = new Element(\"list\");\n timesElt.addContent(listElt);\n listElt.setAttribute(\"mean\", \"\" + mean());\n listElt.setAttribute(\"sigma\", \"\" + sigma());\n for (int i = 0; i < times.size(); i++) {\n Element timeElt = new Element(\"time\");\n listElt.addContent(timeElt);\n timeElt.setAttribute(\"val\", times.get(i).toString());\n }\n Element perNumsElt = new Element(\"per-nums\");\n timesElt.addContent(perNumsElt);\n Set keys = map.keySet();\n List nums = new ArrayList(keys.size());\n nums.addAll(keys);\n Collections.sort(nums);\n int min = nums.get(0).intValue();\n int max = nums.get(nums.size()-1).intValue();\n for (int num = min; num <= max; num++) {\n Element perNumElt = new Element(\"per\");\n perNumsElt.addContent(perNumElt);\n perNumElt.setAttribute(\"num\", \"\" + num);\n Integer numKey = new Integer(num);\n List timesPer = map.get(numKey);\n if (timesPer == null) { \n perNumElt.setAttribute(\"count\", \"0\");\n continue; \n }\n int sum = 0;\n int count = timesPer.size();\n perNumElt.setAttribute(\"count\", \"\" + count);\n for (int i = 0; i < timesPer.size(); i++) {\n Integer time = (Integer) timesPer.get(i);\n sum += time.intValue();\n Element timeElt = new Element(\"time\");\n perNumElt.addContent(timeElt);\n timeElt.setAttribute(\"val\", time.toString());\n }\n double mean = (sum * 1.0) / count;\n perNumElt.setAttribute(\"mean\", \"\" + mean);\n }\n }\n}\n\n\n" }, { "path": "src/opennlp/ccg/test/UpdateTestbed.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.test;\n\nimport opennlp.ccg.grammar.*;\nimport opennlp.ccg.lexicon.*;\n\nimport org.jdom.*;\n\nimport java.io.*;\nimport java.net.*;\nimport java.util.*;\n\n/**\n * Utility class to update testbed files; just adds full words based on \n * sign or parse at present.\n *\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2009/12/21 04:18:31 $\n */\npublic class UpdateTestbed\n{\n /** The grammar. */\n private Grammar grammar; \n\n /** Constructor. */\n private UpdateTestbed(URL grammarURL) throws IOException {\n // load grammar\n System.out.println(\"Loading grammar from URL: \" + grammarURL);\n grammar = new Grammar(grammarURL);\n }\n \n \n /** Adds full words for each test item, if missing, based on the sign or parse. */\n private void addFullWords(File tbFile) throws IOException {\n \n // load testbed\n System.out.println(\"Loading testbed from: \" + tbFile);\n RegressionInfo tbInfo = new RegressionInfo(grammar, tbFile);\n \n // create output doc\n Document outDoc = new Document();\n Element outRoot = new Element(\"regression\");\n outDoc.setRootElement(outRoot);\n\n // update each one\n int numItems = tbInfo.numberOfItems();\n Tokenizer tokenizer = grammar.lexicon.tokenizer;\n System.out.print(\"Adding full words \");\n for (int i = 0; i < numItems; i++) {\n RegressionInfo.TestItem testItem = tbInfo.getItem(i);\n if (testItem.fullWords == null) {\n List words = (testItem.sign != null) \n \t? testItem.sign.getWords() \n \t: grammar.getParsedWords(testItem.sentence);\n testItem.fullWords = tokenizer.format(words);\n }\n outRoot.addContent(RegressionInfo.makeTestItem(testItem));\n System.out.print(\".\"); // indicate progress\n }\n System.out.println();\n \n // save file, backing up original\n File tbFileBackup = new File(tbFile.getParentFile(), tbFile.getName() + \"~\");\n System.out.println(\"Backing up testbed to: \" + tbFileBackup);\n tbFile.renameTo(tbFileBackup);\n System.out.println(\"Saving results to: \" + tbFile);\n FileOutputStream out = new FileOutputStream(tbFile); \n grammar.serializeXml(outDoc, out);\n out.close();\n }\n\n \n /** Updates the given input file. */ \n public static void main(String[] args) throws IOException {\n \n String usage = \"Usage: java opennlp.ccg.test.UpdateTestbed (-g ) (-add-full-words) ()\";\n \n if (args.length > 0 && args[0].equals(\"-h\")) {\n System.out.println(usage);\n System.exit(0);\n }\n\n // args\n String grammarfile = \"grammar.xml\";\n String testbedfile = \"testbed.xml\";\n boolean addFullWords = false;\n for (int i = 0; i < args.length; i++) {\n if (args[i].equals(\"-g\")) {\n grammarfile = args[++i]; continue; \n }\n if (args[i].equals(\"-add-full-words\")) {\n addFullWords = true; continue; \n }\n testbedfile = args[i];\n }\n\n // create updater, check testbed exists\n File gFile = new File(grammarfile);\n URL grammarURL = gFile.toURI().toURL();\n File tbFile = new File(testbedfile);\n if (!tbFile.exists()) {\n tbFile = new File(gFile.getParentFile(), testbedfile);\n }\n if (!tbFile.exists()) {\n System.out.println(\"Unable to find testbed file: \" + testbedfile);\n System.exit(-1);\n }\n UpdateTestbed updater = new UpdateTestbed(grammarURL); \n \n // do tasks\n if (addFullWords) updater.addFullWords(tbFile);\n \n System.out.println(\"Done.\");\n }\n}\n\n" }, { "path": "src/opennlp/ccg/test/Validator.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.test;\n\n// import javax.xml.parsers.*;\nimport org.xml.sax.*;\nimport org.xml.sax.helpers.*;\n// import java.net.*;\n\n\n/**\n * Validates XML files against their declared schemas.\n *\n * @author Michael White\n * @version $Revision: 1.4 $, $Date: 2005/10/20 18:49:42 $\n */\npublic class Validator {\n\n public static void main(String[] args) throws Exception {\n \n if (args.length == 0) {\n System.out.println(\"Usage: java opennlp.ccg.test.Validator \");\n System.exit(1);\n }\n\n // configure schema validating XML parser\n XMLReader parser = getXercesSchemaValidatingParser();\n \n // parse\n for (int i = 0; i < args.length; i++) {\n // System.out.println(\"Parsing: \" + args[i]);\n parser.parse(args[i]);\n }\n }\n\n // NB: this requires xercesImpl.jar, but on the other hand it does not seem \n // possible to validate with the version of JAXP that comes with JDK 1.4.1\n // (cf. JAXP sample SAXLocalNameCount.java)\n private static XMLReader getXercesSchemaValidatingParser() throws Exception {\n String DEFAULT_PARSER_NAME = \"org.apache.xerces.parsers.SAXParser\";\n XMLReader parser = XMLReaderFactory.createXMLReader(DEFAULT_PARSER_NAME);\n String VALIDATION_FEATURE_ID = \"http://xml.org/sax/features/validation\";\n String SCHEMA_VALIDATION_FEATURE_ID = \"http://apache.org/xml/features/validation/schema\";\n parser.setFeature(VALIDATION_FEATURE_ID, true);\n parser.setFeature(SCHEMA_VALIDATION_FEATURE_ID, true);\n return parser;\n }\n}\n\n" }, { "path": "src/opennlp/ccg/unify/EmptySubstitution.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.unify;\n\nimport java.util.Iterator;\n\n\n/**\n * A Substitution which does not hold any substitutions.\n * \n * @author Jason Baldridge\n * @version $Revision: 1.2 $, $Date: 2005/10/20 17:30:30 $\n */\npublic class EmptySubstitution implements Substitution {\n\n\tpublic Object makeSubstitution(Variable var, Object u) throws UnifyFailure {\n\t\treturn u;\n\t}\n\n\tpublic Object getValue(Variable var) {\n\t\treturn null;\n\t}\n\n\tpublic Iterator varIterator() {\n\t\treturn null;\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/unify/Feature.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge and Gann Bierner\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.unify;\n\n/**\n * An attribute paired with a value, possibly with an index that ties\n * it to another feature.\n *\n * @author Jason Baldridge\n * @version $Revision: 1.1.1.1 $, $Date: 2003/02/28 18:02:13 $\n **/\npublic interface Feature {\n\n public String getAttribute ();\n public Unifiable getValue ();\n public short getIndex ();\n\n public void setValue (Unifiable u);\n public void setIndex (short index);\n\n}\n" }, { "path": "src/opennlp/ccg/unify/FeatureStructure.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-5 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.unify;\n\nimport java.util.Set;\n\n/**\n * A feature structure containing attributes and their associated values.\n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2009/02/16 17:31:38 $\n */\npublic interface FeatureStructure extends Unifiable, Mutable {\n\n /**\n * Store a attribute/value pair\n *\n * @param attribute the attribute of the feature\n * @param value the value of the feature\n */\n public void setFeature(String attribute, Object value);\n\n\n /**\n * Get the value corresponding to an attribute.\n *\n * @param attribute the attribute of the feature\n * @return the value of the feature\n */\n public Object getValue(String attribute);\n \n /**\n * Checks to see if the feature structure contains a feature with \n * the given attribute.\n *\n * @param attribute the attribute\n * @return if this structure contains that attribute\n */\n public boolean hasAttribute(String attribute);\n\n /**\n * Checks to see if an attribute has a particular value in this structure\n *\n * @param attribute the attribute\n * @param value the value\n * @return if this structure contains that attribute/value pair\n */\n public boolean attributeHasValue(String attribute, Object value);\n\n\n /**\n * The all attributes in set form\n *\n * @return the set of attributes\n */\n public Set getAttributes();\n\n \n /**\n * The number of features in this feature structure\n *\n * @return number of features in this feature structure\n */\n public int size();\n\n \n /**\n * Returns whether or not this feature structure contains any features\n *\n * @return whether this feature structure contains any features\n */\n public boolean isEmpty();\n\n \n /**\n * Explictly clear the attribute value mappings in this feature structure\n *\n * @param b the empty value\n */\n public void clear();\n \n\n /**\n * Makes a deep copy of this feature structure.\n *\n * @return a copy of this feature structure\n */\n public FeatureStructure copy();\n\n\n /**\n * Computes whether this feature structure contains (is a superset\n * of) another feature structure.\n *\n * @param fs the possibly contained feature structure\n * @return if that structure is a subset of this one \n */\n public boolean contains(FeatureStructure fs);\n\n /**\n * Changes this feature structure such that all its features that\n * are in another feature structure are changed to have the values of\n * the other feature structure. This is destructive.\n *\n * @param f the feature structure to inherit from\n * @return the changed feature structure\n */\n public FeatureStructure inherit(FeatureStructure fs);\n\n /**\n * Determines if this feature structure is exactly the same as another. This\n * means that for every feature, the structures have exactly the same value.\n *\n * @param f the other feature structure\n * @return if this structure is the same as the other\n */\n public boolean equals(FeatureStructure fs);\n\n /** Returns the index. */\n public int getIndex();\n /** Sets the index. */\n public void setIndex(int index);\n /** Returns the inheritsFrom (default unification) index. */\n public int getInheritsFrom();\n \n /**\n * Returns the supertag info for this feature structure.\n */\n public String getSupertagInfo();\n \n /**\n * Returns a TeX formatted feature structure.\n */\n public String toTeX();\n}\n" }, { "path": "src/opennlp/ccg/unify/GFeatStruc.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-7 Jason Baldridge, Gann Bierner, Michael White and Gunes Erkan\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.unify;\n\nimport opennlp.ccg.synsem.LF;\nimport opennlp.ccg.hylo.*;\nimport opennlp.ccg.grammar.*;\nimport gnu.trove.*;\nimport org.jdom.*;\nimport java.util.*;\n\n\n/**\n * A feature structure for use with CCG categories.\n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @author Gunes Erkan\n * @version $Revision: 1.29 $, $Date: 2011/11/11 15:30:42 $\n */\npublic class GFeatStruc extends HashMap implements FeatureStructure {\n\n\tprivate static final long serialVersionUID = 1L;\n\n\tboolean _empty = true;\n int _index = 0;\n int _inheritsFrom = 0;\n \n public GFeatStruc() {\n super(3);\n }\n\n public GFeatStruc(int i) {\n super(i);\n }\n\n @SuppressWarnings(\"unchecked\")\n\tpublic GFeatStruc(Element fsEl) {\n super(fsEl.getChildren().size());\n String index = fsEl.getAttributeValue(\"id\");\n if (index != null) {\n _index = Integer.parseInt(index);\n }\n String inheritsFrom = fsEl.getAttributeValue(\"inheritsFrom\");\n if (inheritsFrom != null) {\n _inheritsFrom = Integer.parseInt(inheritsFrom);\n }\n List feats = fsEl.getChildren();\n if (feats.size() == 0) {\n setFeature(fsEl);\n }\n else {\n for (Iterator featIt=feats.iterator(); featIt.hasNext();) {\n setFeature((Element)featIt.next());\n }\n }\n }\n \n public Element toXml() {\n \tElement retval = new Element(\"fs\");\n \tif (_index > 0) retval.setAttribute(\"id\", Integer.toString(_index));\n \tif (_inheritsFrom > 0) retval.setAttribute(\"inheritsFrom\", Integer.toString(_inheritsFrom));\n \tList keys = new ArrayList(keySet()); \n\t\tCollections.sort(keys);\n \tif (size() == 1 && get(keys.get(0)) instanceof SimpleType) {\n \t\tString attr = keys.get(0); SimpleType val = (SimpleType) get(attr);\n \t\tretval.setAttribute(\"attr\", attr);\n \t\tretval.setAttribute(\"val\", val.getName());\n \t}\n \telse {\n \t\tfor (String attr : keys) {\n \t\t\tElement featElt = new Element(\"feat\");\n \t\t\tfeatElt.setAttribute(\"attr\", attr);\n \t\t\tretval.addContent(featElt);\n \t\t\tObject val = get(attr);\n \t\t\tif (val instanceof SimpleType) \n \t\t\t\tfeatElt.setAttribute(\"val\", ((SimpleType) val).getName());\n \t\t\telse {\n \t\t\t\tif (val instanceof GFeatVar) {\n \t\t\t\t\tGFeatVar var = (GFeatVar) val;\n \t\t\t\t\tElement varElt = new Element(\"featvar\");\n \t\t\t\t\tfeatElt.addContent(varElt);\n \t\t\t\t\tString name = var.name();\n \t\t\t\t\tString typeName = var.getType().getName();\n \t\t\t\t\tif (!typeName.equals(Types.TOP_TYPE)) name += \":\" + typeName;\n \t\t\t\t\tvarElt.setAttribute(\"name\", name);\n \t\t\t\t}\n \t\t\t\telse if (val instanceof LF)\n \t\t\t\t\tfeatElt.addContent(HyloHelper.toXml((LF)val));\n \t\t\t\telse \n \t\t\t\t\tthrow new RuntimeException(\"Unsupported feature value type in constructing XML: \" + val);\n \t\t\t}\n \t\t}\n \t}\n \treturn retval;\n }\n\n public void deepMap(ModFcn mf) {\n for (Iterator attributes=keySet().iterator(); attributes.hasNext();) {\n Object val1 = getValue(attributes.next());\n if (val1 instanceof Mutable) {\n ((Mutable)val1).deepMap(mf);\n }\n }\n mf.modify(this);\n }\n\n public void setFeature(String attribute, Object val) { \n put(attribute, val);\n _empty = false;\n }\n\n private void setFeature(Element e) {\n String attr = e.getAttributeValue(\"attr\");\n if (attr == null) attr = e.getAttributeValue(\"a\");\n if (attr == null) {\n return;\n }\n String val = e.getAttributeValue(\"val\");\n if (val == null) val = e.getAttributeValue(\"v\");\n Object value;\n if (val != null) {\n value = Grammar.theGrammar.types.getSimpleType(val);\n } else {\n Element valEl = (Element)e.getChildren().get(0);\n if (valEl.getName().equals(\"featvar\") || valEl.getName().equals(\"fvar\")) {\n String[] name = valEl.getAttributeValue(\"name\").split(\":\");\n if (name[0]==null) name = valEl.getAttributeValue(\"n\").split(\":\",2);\n if (name.length<2) {\n value = new GFeatVar(name[0]);\n }\n else\n value = new GFeatVar(name[0], Grammar.theGrammar.types.getSimpleType(name[1]));\n } else {\n value = HyloHelper.getLF((Element)e.getChildren().get(0));\n }\n }\n setFeature(attr, value);\n }\n \n public Object getValue(String attribute) {\n return get(attribute);\n }\n\n public boolean hasAttribute(String attribute) {\n return containsKey(attribute);\n }\n\n public boolean attributeHasValue(String attribute, Object val) {\n return val.equals(getValue(attribute));\n }\n\n public Set getAttributes() {\n return keySet();\n }\n \n public void clear() { \n clear();\n _empty = true;\n }\n\n /** Returns true iff this feature structure has the same index and set of attr-val pairs. */\n public boolean equals(FeatureStructure fs) {\n if (!(fs instanceof GFeatStruc)) return false;\n GFeatStruc bfs = (GFeatStruc)fs;\n \n if (_index != bfs._index) return false; \n\n if (size() != bfs.size()) return false;\n Set atts1 = getAttributes();\n Set atts2 = bfs.getAttributes();\n if (!atts1.containsAll(atts2)) return false;\n \n for (Iterator it = atts1.iterator(); it.hasNext(); ) {\n String att = it.next();\n if (!getValue(att).equals(bfs.getValue(att))) return false;\n }\n \n return true;\n }\n \n /** Returns a hash code consistent with equals. */\n public int hashCode() { return super.hashCode() + _index; }\n\n public FeatureStructure copy() { \n GFeatStruc $fs = new GFeatStruc(size());\n $fs.setIndex(_index);\n $fs._inheritsFrom = _inheritsFrom;\n for (Iterator i=getAttributes().iterator(); i.hasNext();) {\n String a = i.next();\n $fs.setFeature(a, UnifyControl.copy(getValue(a)));\n }\n return $fs;\n }\n\n public boolean contains(FeatureStructure fs) { \n if (size() < fs.size())\n return false;\n \n Set atts1 = getAttributes();\n Set atts2 = fs.getAttributes();\n if (atts1.containsAll(atts2)) {\n for (Iterator i2 = atts2.iterator(); i2.hasNext();) {\n String a2 = i2.next();\n boolean foundA2 = false;\n for (Iterator i1 = atts1.iterator(); !foundA2 && i1.hasNext();) {\n String a1 = i1.next();\n if (a1.equals(a2)) {\n if (!getValue(a1).equals(fs.getValue(a2)))\n return false;\n foundA2 = true;\n }\n }\n }\n return true;\n }\n else {\n return false;\n }\n }\n\n public boolean occurs(Variable v) {\n for (Iterator i = values().iterator(); i.hasNext();) {\n Object $_ = i.next();\n if ($_ instanceof Unifiable && ((Unifiable)$_).occurs(v))\n return true;\n }\n return false;\n }\n\n public void unifyCheck(Object u) throws UnifyFailure {\n if (!(u instanceof FeatureStructure)) {\n throw new UnifyFailure();\n }\n // look for incompatible string-valued features\n FeatureStructure fs2 = (FeatureStructure)u;\n Set keys1 = getAttributes();\n for (Iterator i1=keys1.iterator(); i1.hasNext();) {\n String k1 = i1.next();\n Object val1 = getValue(k1);\n if (!(val1 instanceof SimpleType)) continue;\n Object val2 = fs2.getValue(k1);\n if (!(val2 instanceof SimpleType)) continue;\n ((SimpleType)val1).unifyCheck(val2);\n }\n }\n\n public Object unify(Object u, Substitution sub) throws UnifyFailure { \n\n if (!(u instanceof FeatureStructure)) {\n throw new UnifyFailure();\n } \n\n FeatureStructure fs2 = (FeatureStructure)u;\n FeatureStructure $fs = new GFeatStruc(size());\n Set keys1 = getAttributes();\n Set keys2 = fs2.getAttributes();\n for (Iterator i1=keys1.iterator(); i1.hasNext();) {\n String k1 = i1.next();\n Object val1 = getValue(k1);\n Object val2 = fs2.getValue(k1);\n if (val2 != null) {\n $fs.setFeature(k1, Unifier.unify(val1, val2, sub));\n }\n else {\n $fs.setFeature(k1, UnifyControl.copy(val1));\n }\n }\n for (Iterator i2=keys2.iterator(); i2.hasNext();) {\n String k2 = i2.next();\n if (!keys1.contains(k2))\n $fs.setFeature(k2, UnifyControl.copy(fs2.getValue(k2)));\n }\n\n int fs2Index = fs2.getIndex();\n int newIndex = 0;\n if (_index == 0) {\n newIndex = fs2Index;\n } else if (fs2Index == 0) {\n newIndex = _index;\n } else if (sub instanceof GSubstitution) {\n newIndex = ((GSubstitution)sub).makeNewIndex(_index, fs2Index);\n }\n $fs.setIndex(newIndex);\n\n if (sub instanceof GSubstitution && newIndex > 0) {\n ((GSubstitution)sub).addIndexedObject(newIndex,$fs);\n }\n return $fs;\n }\n\n public Object fill(Substitution sub) throws UnifyFailure {\n FeatureStructure $fs = copy();\n for (Iterator i = $fs.getAttributes().iterator(); i.hasNext();) {\n String a = i.next();\n Object value = getValue(a);\n if (value instanceof Variable) {\n Object varVal = sub.getValue((Variable)value);\n if (null != varVal) {\n $fs.setFeature(a, Unifier.unify(value,varVal,sub));\n }\n }\n }\n if (_index > 0 && sub instanceof GSubstitution) {\n FeatureStructure otherVals =\n (FeatureStructure)((GSubstitution)sub).getIndexedObject(_index);\n if (null != otherVals) {\n if (!$fs.equals(otherVals)) {\n $fs = (FeatureStructure)$fs.unify(otherVals, sub);\n $fs.setIndex(otherVals.getIndex());\n }\n }\n }\n return $fs;\n }\n\n public FeatureStructure inherit(FeatureStructure fs) { \n FeatureStructure $fs = copy();\n for (Iterator i = fs.getAttributes().iterator(); i.hasNext();) {\n String a = i.next();\n $fs.setFeature(a, UnifyControl.copy(fs.getValue(a)));\n }\n return $fs;\n }\n\n public int getIndex() {\n return _index;\n }\n\n public void setIndex(int index) {\n _index = index;\n }\n\n public int getInheritsFrom() {\n return _inheritsFrom;\n }\n\n private void addFeatureString(String attribute, StringBuffer sb) {\n Object val = getValue(attribute);\n sb.append(attribute).append('=').append(val.toString());\n }\n \n public String toString() {\n\n // if (_empty) return \"\";\n\n StringBuffer sb = new StringBuffer(size()*4);\n \n if (_index > 0) {\n sb.append('<'); sb.append(_index); sb.append('>');\n }\n \n if (_empty) return sb.toString();\n\n String featsToShow = Grammar.theGrammar.prefs.featsToShow;\n\n sb.append('{');\n\n List filteredKeys = new ArrayList(size());\n if (featsToShow.length() == 0) {\n filteredKeys.addAll(keySet());\n }\n else {\n for (Iterator it = keySet().iterator(); it.hasNext(); ) {\n String key = it.next();\n if (featsToShow.indexOf(key) != -1) \n filteredKeys.add(key);\n }\n }\n String[] keys = new String[filteredKeys.size()];\n filteredKeys.toArray(keys);\n Arrays.sort(keys);\n \n for (int i=0; i < keys.length; i++) {\n addFeatureString(keys[i], sb);\n if (i < keys.length - 1) sb.append(\", \");\n }\n \n sb.append('}');\n \n return sb.toString();\n }\n\n /**\n * Returns the supertag info for this feature structure.\n * In particular, returns the values of any non-variable \n * features of interest, within square brackets.\n * The features of interest are configurable \n * at the grammar level.\n */\n public String getSupertagInfo() {\n if (_empty) return \"\";\n StringBuffer sb = new StringBuffer();\n ArrayList attrs = new ArrayList(getAttributes());\n Collections.sort(attrs);\n Set supertagFeatures = Grammar.theGrammar.supertagFeatures;\n for (int i = 0; i < attrs.size(); i++) {\n String attr = attrs.get(i);\n if (!supertagFeatures.contains(attr)) continue;\n Object val = getValue(attr);\n if (val instanceof Variable) continue;\n String s = val.toString();\n if (s.equals(\"+\") || s.equals(\"-\")) s = s + attr;\n sb.append('[').append(s).append(']');\n }\n return sb.toString();\n }\n \n private void addFeatureTeX(String attribute, StringBuffer sb) {\n Object val = getValue(attribute);\n String s = cleanText(val.toString());\n if (s.equals(\"+\") || s.equals(\"-\")) s = attribute + s;\n sb.append(\" \").append(s);\n }\n \n // makes sure every special character is handled correctly in LaTeX\n private String cleanText(String s) {\n String str = s;\n try {\n //order matters!!\n str = str.replaceAll(\"\\\\\\\\\", \" \\\\\\\\\\\\backslash \");\n str = str.replaceAll(\"\\\\{\", \" \\\\\\\\\\\\{ \");\n str = str.replaceAll(\"\\\\}\", \" \\\\\\\\\\\\} \");\n str = str.replaceAll(\"\\\\$\", \" \\\\\\\\\\\\$ \");\n str = str.replaceAll(\"\\\\#\", \" \\\\\\\\\\\\# \");\n str = str.replaceAll(\"\\\\%\", \" \\\\\\\\\\\\% \");\n str = str.replaceAll(\"\\\\&\", \" \\\\\\\\\\\\& \");\n str = str.replaceAll(\"\\\\~\", \" \\\\\\\\\\\\tilde\\\\{\\\\} \");\n str = str.replaceAll(\"\\\\_\", \" \\\\\\\\\\\\_ \");\n str = str.replaceAll(\"\\\\^\", \" \\\\\\\\\\\\hat\\\\{\\\\} \");\n }\n catch (Exception e) {\n System.out.println(\"Error while evaluating RegExp: \" + e.toString());\n }\n return str;\n }\n \n public String toTeX() {\n StringBuffer sb = new StringBuffer();\n if ((_index > 0)&&(_empty)) {\n sb.append(\" \\\\subsf{ < \"); sb.append(_index); sb.append(\" > } \");\n }\n if (_empty) return sb.toString();\n String featsToShow = Grammar.theGrammar.prefs.featsToShow;\n sb.append(\" \\\\subsf{ \");\n if ((_index > 0)) {\n sb.append(\" < \"); sb.append(_index); sb.append(\" > \");\n }\n List filteredKeys = new ArrayList(size());\n if (featsToShow.length() == 0) {\n filteredKeys.addAll(keySet());\n }\n else {\n for (Iterator it = keySet().iterator(); it.hasNext(); ) {\n String key = it.next();\n if (featsToShow.indexOf(key) != -1) \n filteredKeys.add(key);\n }\n }\n String[] keys = new String[filteredKeys.size()];\n filteredKeys.toArray(keys);\n Arrays.sort(keys);\n for (int i=0; i < keys.length; i++) {\n addFeatureTeX(keys[i], sb);\n if (i < keys.length - 1) sb.append(\" , \");\n }\n sb.append(\" } \");\n return sb.toString();\n }\n \n \n /**\n * Returns a hash code using the given map from vars to ints, \n * to allow for equivalence up to variable names.\n */\n public int hashCode(TObjectIntHashMap varMap) {\n\n int retval = 0;\n \n // nb: treat index as a regular var\n if (_index != 0) {\n \t\t// see if index already in map\n \t\tif (varMap.containsKey(_index))\n \t\t\tretval = varMap.get(_index);\n \t\t// otherwise add it\n \t\telse {\n\t \t\tint next = varMap.size() + 1;\n\t \t\tvarMap.put(_index, next);\n\t \t\tretval = next;\n \t\t}\n }\n // otherwise treat missing index as unique, keyed to negative identity hash\n\t\telse {\n \t\tint next = varMap.size() + 1;\n \t\tvarMap.put(-1 * Math.abs(System.identityHashCode(this)), next);\n \t\tretval = next;\n\t\t}\n \n if (_empty) { return retval; }\n\n // sort keys\n Set keySet = keySet();\n String[] keys = new String[keySet.size()];\n keySet.toArray(keys);\n Arrays.sort(keys);\n \n // do each key\n for (int i=0; i atts1 = getAttributes();\n Set atts2 = fs.getAttributes();\n if (!atts1.containsAll(atts2)) return false;\n \n for (Iterator it = atts1.iterator(); it.hasNext(); ) {\n String att = it.next();\n Object val = getValue(att);\n Object val2 = fs.getValue(att);\n if (val instanceof Variable && val2 instanceof Variable) {\n \tif (!((Variable)val).equals(val2, varMap, varMap2)) return false;\n }\n else {\n if (!val.equals(val2)) return false;\n }\n }\n \n return true;\n }\n}\n" }, { "path": "src/opennlp/ccg/unify/GFeatVar.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-7 Jason Baldridge, Gunes Erkan and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.unify;\n\nimport java.io.Serializable;\nimport gnu.trove.TObjectIntHashMap;\nimport opennlp.ccg.grammar.*;\n\n/**\n * A class for variables which can stand for any feature.\n *\n * @author Jason Baldridge\n * @author Gunes Erkan\n * @author Michael White\n * @version $Revision: 1.9 $, $Date: 2009/07/17 04:23:30 $\n **/\npublic class GFeatVar implements Variable, Indexed, Mutable, Serializable {\n \n\tprivate static final long serialVersionUID = -5526887599728099988L;\n\t\n\tprotected final String _name;\n protected int _index;\n protected int _hashCode;\n protected SimpleType type;\n \n public GFeatVar(String name) {\n this(name, 0, null);\n }\n\n public GFeatVar(String name, SimpleType st) {\n this(name, 0, st);\n }\n\n protected GFeatVar(String name, int index, SimpleType st) {\n _name = name;\n _index = index;\n type = (st != null) ? st : Grammar.theGrammar.types.getSimpleType(Types.TOP_TYPE);\n _hashCode = _name.hashCode() + _index + type.getIndex();\n }\n \n public String name() {\n return _name;\n }\n\n public Object copy() {\n return new GFeatVar(_name, _index, type);\n }\n \n public void deepMap(ModFcn mf) {\n mf.modify(this);\n }\n \n public int getIndex() {\n return _index;\n }\n\n public SimpleType getType() {\n return type;\n }\n\n public void setIndex(int index) {\n _hashCode += index - _index;\n _index = index;\n }\n\n public boolean occurs(Variable var) {\n return equals(var);\n }\n\n public int hashCode() {\n return _hashCode;\n }\n \n public boolean equals(Object o) {\n if (this == o) return true;\n if (!(o instanceof GFeatVar)) return false;\n GFeatVar var = (GFeatVar) o;\n return _index == var._index && _name.equals(var._name) && type.equals(var.type);\n }\n\n /**\n\t * Returns a hash code using the given map from vars to ints.\n\t */\n\tpublic int hashCode(TObjectIntHashMap varMap) {\n\t\t// see if this already in map\n\t\tif (varMap.containsKey(this))\n\t\t\treturn varMap.get(this);\n\t\t// otherwise add it\n\t\tint next = varMap.size() + 1;\n\t\tvarMap.put(this, next);\n\t\treturn next;\n\t}\n \n /**\n\t * Returns whether this var equals the given object up to variable names,\n\t * using the given maps from vars to ints.\n\t * (Note that the name and index may differ, but the types must be equal.)\n\t */\n public boolean equals(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2) {\n if (this == obj) return true;\n if (obj.getClass() != this.getClass()) { return false; }\n GFeatVar gv = (GFeatVar) obj;\n if (varMap.get(this) != varMap2.get(gv)) return false;\n if (!this.type.equals(gv.type)) return false;\n return true;\n }\n \n public void unifyCheck(Object o) throws UnifyFailure {}\n \n public Object unify(Object u, Substitution sub) throws UnifyFailure {\n if (equals(u)) {\n return this;\n }\n else if (u instanceof SimpleType) {\n SimpleType st1 = getType();\n SimpleType st2 = (SimpleType)u;\n return sub.makeSubstitution(this, st2.unify(st1, sub));\n }\n else if (u instanceof GFeatVar) {\n GFeatVar var = (GFeatVar) u;\n if (var.occurs(this)) throw new UnifyFailure();\n SimpleType st1 = getType();\n SimpleType st2 = var.getType();\n SimpleType st3 = (SimpleType) st2.unify(st1, sub);\n // substitute var with most specific type\n if (st3.equals(st2)) return sub.makeSubstitution(this, var);\n else if (st3.equals(st1)) return sub.makeSubstitution(var, this);\n else {\n // need a new var with intersection type\n GFeatVar var3 = new GFeatVar(_name, UnifyControl.getUniqueVarIndex(), st3);\n sub.makeSubstitution(var, var3);\n return sub.makeSubstitution(this, var3);\n }\n }\n else {\n return sub.makeSubstitution(this, u);\n }\n }\n\n public Object fill(Substitution sub) throws UnifyFailure {\n Object val = sub.getValue(this);\n if (val != null) {\n return val;\n } else {\n return this;\n }\n }\n\n\n public String toString() { \n String retval = _name;\n if (!type.getName().equals(Types.TOP_TYPE)) retval += \":\" + type.getName();\n return retval;\n }\n}\n" }, { "path": "src/opennlp/ccg/unify/GSubstitution.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge, University of Edinburgh (Michael White) \n// and Gunes Erkan\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.unify;\n\nimport gnu.trove.*;\nimport java.util.*;\n\n/**\n * Implementation of Substitution interface which ensures that all\n * the categories it contains are updated as new substitutions are\n * made.\n *\n * @author Jason Baldridge\n * @author Michael White\n * @author Gunes Erkan\n * @version $Revision: 1.13 $, $Date: 2009/12/21 03:27:19 $ \n*/\npublic class GSubstitution extends THashMap implements Substitution {\n\n\tprivate static final long serialVersionUID = 1L;\n\n\tprivate TIntObjectHashMap _indexedObjects = new TIntObjectHashMap();\n private TIntIntHashMap _newFeatStrucIndexes = new TIntIntHashMap();\n\n /**\n * Request the Substitution to identify a variable with an\n * object. Automagically condenses the Substitution so that all\n * other values in this Substitution get the new value for the\n * variable if they contain it.\n *\n * @param var the variable whose value has been determined\n * @param o the Object identified with the variable\n * @return the Object identified with the variable, which has\n * potentially undergone further unifications as a result of\n * making the substitution\n * @exception throws UnifyFailure if the Object cannot be unified\n * with a previous value substituted for the Variable.\n */\n @SuppressWarnings(\"unchecked\")\n\tpublic Object makeSubstitution(Variable var, Object u) throws UnifyFailure {\n Object val1 = getValue(var);\n if (u instanceof Variable) {\n Variable var2 = (Variable)u;\n Object val2 = getValue(var2);\n // check if var -> u already\n if (val1 != null && val1.equals(u)) return u;\n // check if u -> var already\n if (val2 != null && val2.equals(var)) return var;\n // otherwise continue \n if (val1 != null) {\n if (val1 instanceof Unifiable && ((Unifiable)val1).occurs(var2)) {\n throw new UnifyFailure();\n }\n if (val2 != null) {\n u = Unifier.unify(var, val2, this);\n } else {\n u = makeSubstitution(var2, val1);\n }\n } else if (val2 != null) {\n if (val2 instanceof Unifiable && ((Unifiable)val2).occurs(var)) {\n throw new UnifyFailure(); \n }\n makeSubstitution(var, val2);\n } \n } else if (val1 != null) {\n u = Unifier.unify(val1, u, this);\n }\n put(var, u);\n for (Iterator i=keySet().iterator(); i.hasNext();) {\n Variable v = (Variable)i.next();\n Object res = getValue(v);\n if (res instanceof Unifiable) {\n res = ((Unifiable)res).fill(this);\n }\n put(v, res);\n }\n if (u instanceof Unifiable) {\n u = ((Unifiable)u).fill(this);\n }\n return u;\n }\n\n /**\n * Try to get the value of a variable from this Substitution.\n * Returns null if the variable is unknown to the Substitution.\n *\n * @param var the variable whose value after unification is desired\n * @return the Object which this variable has been unified with \n */\n public Object getValue(Variable var) {\n Object val = get(var);\n if (null != val) {\n if (val instanceof Variable) {\n Object deepVal = getValue((Variable)val);\n if (null != deepVal) {\n val = deepVal;\n }\n }\n }\n return val;\n }\n\n @SuppressWarnings(\"unchecked\")\n\tpublic Iterator varIterator() {\n return keySet().iterator();\n }\n\n public int makeNewIndex(int fs1Index, int fs2Index) {\n int index = UnifyControl.getUniqueFeatureStructureIndex();\n int fs1IndexUpdated = getUpdatedIndex(fs1Index);\n int fs2IndexUpdated = getUpdatedIndex(fs2Index);\n addReindex(fs1IndexUpdated, index);\n addReindex(fs2IndexUpdated, index);\n return index;\n }\n\n public void addReindex(int oldIndex, int newIndex) {\n // avoid creating a pointer cycle\n if (oldIndex == newIndex) return;\n if (_newFeatStrucIndexes.containsKey(newIndex)) {\n throw new RuntimeException(\n \"Whoops! Index map already contains newIndex: \" + newIndex + \"\\n\" + this\n );\n }\n _newFeatStrucIndexes.put(oldIndex, newIndex);\n }\n\n public int getUpdatedIndex(int oldIndex) {\n if (!_newFeatStrucIndexes.containsKey(oldIndex)) return oldIndex;\n return getUpdatedIndex(_newFeatStrucIndexes.get(oldIndex));\n }\n \n public void addIndexedObject(int index, Object o) {\n _indexedObjects.put(index, o);\n }\n \n public Object getIndexedObject(int index) {\n return _indexedObjects.get(getUpdatedIndex(index));\n }\n\n public void condense() throws UnifyFailure {\n int[] keys = _indexedObjects.keys();\n for (int i=0; i < keys.length; i++) {\n Object obj = _indexedObjects.get(keys[i]);\n if (obj instanceof Unifiable) {\n Object filled = ((Unifiable)obj).fill(this);\n _indexedObjects.put(keys[i], filled);\n }\n }\n // drop old indexed objects\n for (int i = 0; i < keys.length; i++) {\n if (_newFeatStrucIndexes.containsKey(keys[i])) {\n _indexedObjects.remove(keys[i]);\n }\n }\n }\n\n @SuppressWarnings(\"unchecked\")\n\tpublic String toString() {\n StringBuffer sb = new StringBuffer();\n sb.append(\"vars: \\t\");\n for (Iterator keys=keySet().iterator(); keys.hasNext();) {\n Object key = keys.next();\n sb.append(key).append('=').append(get(key)).append('\\t');\n }\n sb.append('\\n');\n sb.append(\"indexes: \\t\");\n int indexKeys[] = _newFeatStrucIndexes.keys();\n for (int i = 0; i < indexKeys.length; i++) {\n sb.append(indexKeys[i] + \"->\" + _newFeatStrucIndexes.get(indexKeys[i]) + \"\\t\");\n }\n return sb.toString();\n }\n}\n" }, { "path": "src/opennlp/ccg/unify/GUnifier.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.unify;\n\nimport opennlp.ccg.synsem.*;\n\n/**\n * A unifier for CCG categories.\n *\n * @author Jason Baldridge\n * @version $Revision: 1.1.1.1 $, $Date: 2003/02/28 18:02:13 $\n */\npublic class GUnifier { \n\n public static Category unify (Category c1, Category c2) throws UnifyFailure {\n return (Category)unify(c1, c2, new EmptySubstitution());\n }\n\n public static Category unify (Category c1, Category c2, Substitution sub) throws UnifyFailure {\n if (c1 instanceof AtomCat && c2 instanceof ComplexCat) {\n c2.unifyCheck(c1);\n return (Category)c2.unify(c1, sub);\n }\n c1.unifyCheck(c2);\n return (Category)Unifier.unify(c1, c2, sub);\n }\n}\n" }, { "path": "src/opennlp/ccg/unify/Indexed.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.unify;\n\n/**\n * An object which can be indexed.\n *\n * @author Jason Baldridge\n * @version $Revision: 1.1.1.1 $, $Date: 2003/02/28 18:02:13 $\n **/\npublic interface Indexed {\n\n /**\n * Gets the index of this Indexed object.\n *\n * @return the index \n **/\n public int getIndex ();\n\n\n /**\n * Sets the index of this Indexed object.\n *\n * @param index An int which provides a unique index\n **/\n public void setIndex (int index);\n \n}\n" }, { "path": "src/opennlp/ccg/unify/ModFcn.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.unify;\n\n/**\n * Interface for functions which modify Mutable objects.\n *\n * @author Jason Baldridge\n * @version $Revision: 1.1.1.1 $, $Date: 2003/02/28 18:02:13 $\n */\npublic interface ModFcn {\n\n /**\n * Changes a Mutable object in some way.\n *\n * @param mf a function to be applied\n **/\n public void modify (Mutable m);\n\n}\n" }, { "path": "src/opennlp/ccg/unify/Mutable.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.unify;\n\n/**\n * Interface for objects which might contain recursive structure and are\n * mutable.\n *\n * @author Jason Baldridge\n * @version $Revision: 1.1.1.1 $, $Date: 2003/02/28 18:02:13 $\n */\npublic interface Mutable {\n\n /**\n * Applies a ModFcn to this Mutable and then applies it to all fields\n * which are themselves Mutables.\n *\n * @param mf a function to be applied\n **/\n public void deepMap (ModFcn mf);\n\n}\n" }, { "path": "src/opennlp/ccg/unify/SelfCondensingSub.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.unify;\n\nimport java.util.*;\n\n/**\n * Implementation of Substitution interface which ensures that all the\n * categories it contains are updated as new substitutions are made.\n * \n * @author Jason Baldridge\n * @version $Revision: 1.3 $, $Date: 2009/12/21 03:27:19 $\n */\npublic class SelfCondensingSub extends HashMap implements Substitution {\n\n\tprivate static final long serialVersionUID = 1L;\n\n\t/**\n\t * Request the Substitution to identify a variable with an object.\n\t * Automagically condenses the Substitution so that all other values in this\n\t * Substitution get the new value for the variable if they contain it.\n\t * \n\t * @param var\n\t * the variable whose value has been determined\n\t * @param o\n\t * the Object identified with the variable\n\t * @return the Object identified with the variable, which has potentially\n\t * undergone further unifications as a result of making the\n\t * substitution\n\t * @exception throws\n\t * UnifyFailure if the Object cannot be unified with a\n\t * previous value substituted for the Variable.\n\t */\n\tpublic Object makeSubstitution(Variable var, Object u) throws UnifyFailure {\n\n\t\tObject val1 = getValue(var);\n\n\t\tif (u instanceof Variable) {\n\t\t\tVariable var2 = (Variable) u;\n\t\t\tObject val2 = getValue(var2);\n\t\t\tif (val1 != null) {\n\t\t\t\tif (val2 != null)\n\t\t\t\t\tu = Unifier.unify(var, val2, this);\n\t\t\t\telse\n\t\t\t\t\tu = makeSubstitution(var2, val1);\n\t\t\t} else {\n\t\t\t\tif (val2 != null)\n\t\t\t\t\tmakeSubstitution(var, val2);\n\t\t\t\telse\n\t\t\t\t\tput(var, var2);\n\t\t\t}\n\t\t} else if (val1 != null) {\n\t\t\tu = Unifier.unify(val1, u, this);\n\t\t}\n\t\tput(var, u);\n\t\tfor (Iterator i = keySet().iterator(); i.hasNext();) {\n\t\t\tVariable v = i.next();\n\t\t\tObject res = getValue(v);\n\t\t\tif (res instanceof Unifiable) {\n\t\t\t\tres = ((Unifiable) res).fill(this);\n\t\t\t}\n\t\t\tput(v, res);\n\t\t}\n\t\tif (u instanceof Unifiable) {\n\t\t\tu = ((Unifiable) u).fill(this);\n\t\t}\n\t\treturn u;\n\t}\n\n\t/**\n\t * Try to get the value of a variable from this Substitution. Returns null\n\t * if the variable is unknown to the Substitution.\n\t * \n\t * @param var\n\t * the variable whose value after unification is desired\n\t * @return the Object which this variable has been unified with\n\t */\n\tpublic Object getValue(Variable var) {\n\t\treturn get(var);\n\t}\n\n\tpublic Iterator varIterator() {\n\t\treturn keySet().iterator();\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/unify/SimpleSubstitution.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-5 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.unify;\n\nimport java.util.*;\n\n/**\n * Simple implementation of Substitution interface.\n * \n * @author Jason Baldridge\n * @version $Revision: 1.3 $, $Date: 2005/10/20 17:30:30 $\n */\npublic class SimpleSubstitution extends HashMap implements Substitution {\n\n\tprivate static final long serialVersionUID = 1L;\n\n\tpublic SimpleSubstitution() {}\n\n\tpublic SimpleSubstitution(Map map) {\n\t\tsuper(map);\n\t}\n\n\tpublic Object makeSubstitution(Variable var, Object u) throws UnifyFailure {\n\t\tif (u instanceof Unifiable) {\n\t\t\tu = ((Unifiable) u).fill(this);\n\t\t}\n\t\tput(var, u);\n\t\treturn u;\n\t}\n\n\tpublic Object getValue(Variable var) {\n\t\treturn get(var);\n\t}\n\n\tpublic Iterator varIterator() {\n\t\treturn keySet().iterator();\n\t}\n}\n" }, { "path": "src/opennlp/ccg/unify/SimpleType.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n//// Copyright (C) 2003-9 Gunes Erkan and Michael White\r\n//// \r\n//// This library is free software; you can redistribute it and/or\r\n//// modify it under the terms of the GNU Lesser General Public\r\n//// License as published by the Free Software Foundation; either\r\n//// version 2.1 of the License, or (at your option) any later version.\r\n//// \r\n//// This library is distributed in the hope that it will be useful,\r\n//// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n//// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n//// GNU Lesser General Public License for more details.\r\n//// \r\n//// You should have received a copy of the GNU Lesser General Public\r\n//// License along with this program; if not, write to the Free Software\r\n//// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n////////////////////////////////////////////////////////////////////////////////\r\n\r\npackage opennlp.ccg.unify;\r\n\r\nimport opennlp.ccg.grammar.*;\r\n\r\nimport java.io.ObjectStreamException;\r\nimport java.io.Serializable;\r\nimport java.util.*;\r\n\r\n/**\r\n * A simple type for feature values in CCG categories.\r\n * \r\n * Note that during deserialization, the type is resolved using the current grammar.\r\n * \r\n * @author Gunes Erkan\r\n * @author \t\tMichael White\r\n * @version $Revision: 1.8 $, $Date: 2009/07/17 04:23:30 $\r\n */\r\npublic class SimpleType implements Unifiable, Serializable {\r\n \r\n\tprivate static final long serialVersionUID = 7028285176993549672L;\r\n\t\r\n\tprivate int index;\r\n private String name;\r\n private BitSet bitset;\r\n private BitSet tempBitset = new BitSet();\r\n private transient Types types;\r\n\r\n public SimpleType(int i, String n, BitSet bs, Types t) {\r\n index = i;\r\n name = n;\r\n bitset = bs;\r\n types = t;\r\n }\r\n\t\r\n public int getIndex() { return index; }\r\n\r\n public BitSet getBitSet() { return bitset; }\r\n\r\n public String getName() { return name; }\r\n\r\n public String toString() { return name; }\r\n \r\n public void unifyCheck(Object u) throws UnifyFailure {\r\n if (!(u instanceof SimpleType)) {\r\n throw new UnifyFailure();\r\n }\r\n }\r\n\r\n public Object unify(Object u, Substitution sub) throws UnifyFailure {\r\n if (!(u instanceof SimpleType)) {\r\n throw new UnifyFailure();\r\n }\r\n if (this == u) return this;\r\n SimpleType st2 = (SimpleType) u;\r\n tempBitset.clear();\r\n tempBitset.or(bitset);\r\n tempBitset.and(st2.getBitSet());\r\n int resultTypeIndex = tempBitset.nextSetBit(0);\r\n if (resultTypeIndex == -1) {\r\n \tthrow new UnifyFailure();\r\n }\r\n return types.getIndexMap().get(resultTypeIndex);\r\n }\r\n\r\n public Object fill(Substitution s) throws UnifyFailure {\r\n return this;\r\n }\r\n \r\n public boolean occurs(Variable v) { return false; }\r\n \r\n public int hashCode() { return index; } \r\n \r\n public boolean equals(Object o) {\r\n if (!(o instanceof SimpleType)) return false;\r\n if (index == ((SimpleType)o).getIndex()) return true;\r\n else return false;\r\n }\r\n \r\n /** Returns canonical version of deserialized type based on current grammar. */\r\n public Object readResolve() throws ObjectStreamException {\r\n \treturn Grammar.theGrammar.types.getSimpleType(name);\r\n }\r\n}\r\n" }, { "path": "src/opennlp/ccg/unify/Substitution.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge and Gann Bierner\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.unify;\n\nimport java.util.*;\n\n/**\n * Specifies how variable are to be replaced to make two objects unify.\n * \n * @author Gann Bierner & Jason Baldridge\n * @version $Revision: 1.2 $, $Date: 2005/10/20 17:30:30 $\n */\npublic interface Substitution {\n\n\t/**\n\t * Request the Substitution to identify a variable with an object.\n\t * \n\t * @param var\n\t * the variable whose value has been determined\n\t * @param o\n\t * the Object identified with the variable\n\t * @return the Object identified with the variable, which has potentially\n\t * undergone further unifications as a result of making the\n\t * substitution\n\t * @exception throws\n\t * UnifyFailure if the Object cannot be unified with a\n\t * previous value substituted for the Variable.\n\t */\n\tpublic Object makeSubstitution(Variable var, Object u) throws UnifyFailure;\n\n\t/**\n\t * Try to get the value of a variable from this Substitution. Should return\n\t * null if the variable is unknown to the Substitution.\n\t * \n\t * @param var\n\t * the variable whose value after unification is desired\n\t * @return the Object which this variable has been unified with\n\t */\n\tpublic Object getValue(Variable var);\n\n\tpublic Iterator varIterator();\n\n}\n" }, { "path": "src/opennlp/ccg/unify/Unifiable.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-5 Jason Baldridge and Gann Bierner\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.unify;\n\n/**\n * An interface for classes that may be unified.\n * \n * @author Gann Bierner\n * @version $Revision: 1.2 $, $Date: 2005/10/19 21:27:16 $\n */\npublic interface Unifiable {\n\n\t/**\n\t * Determines if a Variable occurs within this Unifiable\n\t * \n\t * @param v\n\t * the Variable to check for\n\t * @return whether or not the Variable occurs\n\t */\n\tpublic boolean occurs(Variable v);\n\n\t/**\n\t * Tests for equality with the given Object.\n\t * \n\t * @param o\n\t * object to test for equality\n\t * @return true if this Unifiable is equal to o, false if\n\t * not.\n\t */\n\tpublic boolean equals(Object o);\n\n\t/**\n\t * Unify this Unfiable with another Object.\n\t * \n\t * @param o\n\t * object to unify with\n\t * @param s\n\t * Substitution containing the variable resolutions\n\t * @exception UnifyFailure\n\t * if this Unifiable cannot be unified with the Object\n\t * @return an object which represents the unification of this Unifiable with\n\t * the Object\n\t */\n\tpublic Object unify(Object u, Substitution s) throws UnifyFailure;\n\n\t/**\n\t * Check if this Unifiable can unify with another Object. This should be\n\t * implemented as a quick check to allow users of the Unifiable to scan a\n\t * group of Unifications to rapidly see if the entire group is at least\n\t * possible before descending into each one with a full unification\n\t * procedure. Thus, if a call to this method does not result in a\n\t * UnifyFailure exception being thrown, it doesn't mean that the Object can\n\t * definitely be unified with this Unifiable -- what is important is that\n\t * when a call to this method throws a UnifyFailure exception, it permits\n\t * one to avoid calling the unify() method on other Unifiables in a group\n\t * because the quick check failed on this one.\n\t * \n\t * @param o\n\t * object to check for unifiability\n\t * @exception UnifyFailure\n\t * if this Unifiable cannot be unified with the Object\n\t */\n\tpublic void unifyCheck(Object u) throws UnifyFailure;\n\n\t/**\n\t * Replaces any variables in this Unifiable with the values found for them\n\t * in the Substitution argument.\n\t * \n\t * @param s\n\t * Substitution containing the variable resolutions\n\t * @return a copy of this Unifiable with all variables from the Substitution\n\t * replaced by their values.\n\t */\n\tpublic Object fill(Substitution s) throws UnifyFailure;\n\n}\n" }, { "path": "src/opennlp/ccg/unify/Unifier.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002 Jason Baldridge and Gann Bierner\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.unify;\n\n/**\n * A unification utility that abstracts a few basic issues such\n * Variables and not needed to pass a substitution object explictly.\n *\n * @author Jason Baldridge\n * @version $Revision: 1.2 $, $Date: 2004/11/11 17:50:13 $\n **/\npublic class Unifier {\n \n /**\n * Uses a SelfCondensingSub underlyingly so that it\n * is not necessary to pass a substitution object explictly.\n *\n * @param u1 the first of two Unifiables to unify\n * @param u2 the second of two Unifiables to unify\n * @return the result of unifying u1 and u2\n **/\n public static final Object unify (Object u1, Object u2) throws UnifyFailure {\n Substitution sub = new SelfCondensingSub();\n Object result = unify(u1, u2, sub);\n if (result instanceof Unifiable) {\n result = ((Unifiable)result).fill(sub);\n }\n return result;\n }\n\n /**\n * Method which handles ordering to make sure that the Unifiable\n * unify() method is called on the Variable if either of the\n * arguments is a Variable. This way, under a unification scheme\n * for a set of classes, you don't have to have each Unifiable\n * check to see if the thing it is trying to be unified with is a\n * Variable.\n *\n * @param u1 the first of two Unifiables to unify\n * @param u2 the second of two Unifiables to unify\n * @param sub the substitution object holding global unification\n * information\n * @return the result of unifying u1 and u2\n **/\n public static final Object unify (Object u1, Object u2, Substitution sub) throws UnifyFailure {\n \n // !!!!!!!!!!!!!!!!!!!!!!!! CAUTION !!!!!!!!!!!!!!!!!!!!!!!!\n // the order of this if-else statement is important, so be\n // careful before you change it!\n // !!!!!!!!!!!!!!!!!!!!!!!! CAUTION !!!!!!!!!!!!!!!!!!!!!!!!\n if (u2 instanceof Variable) {\n return ((Variable)u2).unify(u1, sub);\n } else if (u1 instanceof Unifiable) {\n return ((Unifiable)u1).unify(u2, sub);\n } else if (u2 instanceof Unifiable) {\n return ((Unifiable)u2).unify(u1, sub);\n } else if (u1.equals(u2)) {\n return u1;\n } else {\n throw new UnifyFailure();\n }\n }\n}\n" }, { "path": "src/opennlp/ccg/unify/UnifyControl.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge and University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.unify;\n\nimport opennlp.ccg.synsem.*;\n\nimport gnu.trove.*;\n\n/**\n * Center of command for the unification process. \n * Needs work to allow for multithreading. \n *\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.6 $, $Date: 2005/10/20 17:30:30 $\n */\npublic class UnifyControl { \n\n /**\n * An integer used to keep variables unique in lexical items.\n */\n private static int _varIndex = 0;\n\n /**\n * A function that makes variables unique.\n */\n private static ModFcn uniqueFcn = new ModFcn() {\n public void modify (Mutable m) {\n if (m instanceof Indexed && m instanceof Variable) {\n ((Indexed)m).setIndex(_varIndex);\n }\n }};\n\n\n /**\n * An integer used to keep feature structure indexes unique.\n */\n private static int _fsIndex = 1;\n private static TIntIntHashMap _reindexed = new TIntIntHashMap();\n\n private static CategoryFcn indexFcn = new CategoryFcnAdapter() {\n public void forall (Category c) {\n FeatureStructure fs = c.getFeatureStructure();\n if (fs != null) {\n int index = fs.getIndex();\n if (index > 0) {\n int $index = _reindexed.get(index);\n if ($index == 0) {\n $index = _fsIndex++;\n _reindexed.put(index, $index); \n }\n fs.setIndex($index);\n }\n }\n }\n };\n\n /** Resets the uniqueness counters. */\n public static void startUnifySequence() {\n _varIndex = 0;\n _fsIndex = 1;\n }\n \n /** Sets the var and feature structure indices to unique values. */\n public static void reindex(Category cat) { \n reindex(cat, null); \n }\n\n /** Sets the var and feature structure indices to unique values. */\n public static void reindex(Category cat, Category anotherCat) {\n _reindexed.clear();\n cat.forall(indexFcn);\n cat.deepMap(uniqueFcn);\n if (cat != anotherCat && anotherCat != null) {\n anotherCat.forall(indexFcn);\n anotherCat.deepMap(uniqueFcn);\n }\n _varIndex++;\n }\n\n public static int getUniqueVarIndex() {\n return ++_varIndex;\n }\n\n public static int getUniqueFeatureStructureIndex() {\n return ++_fsIndex;\n }\n \n public static Object copy(Object o) {\n if (o instanceof Category) {\n return ((Category)o).copy();\n } else if (o instanceof GFeatVar) {\n return ((GFeatVar)o).copy();\n } else if (o instanceof LF) {\n return ((LF)o).copy();\n } else if (o instanceof GFeatStruc) {\n return ((GFeatStruc)o).copy();\n } else {\n return o;\n }\n }\n}\n\n" }, { "path": "src/opennlp/ccg/unify/UnifyFailure.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-3 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.unify;\n\n/**\n * @author Jason Baldridge\n * @author Michael White\n * @version $Revision: 1.4 $, $Date: 2009/11/28 03:39:27 $\n */\npublic class UnifyFailure extends Exception {\n\n\tprivate static final long serialVersionUID = 1L;\n\n /** Constructor. */\n public UnifyFailure() {}\n\n /** Constructor with message. */\n public UnifyFailure(String m) { super(m); }\n\n /** Constructor with two args that failed to unify. */\n public UnifyFailure(String arg1, String arg2) {\n \tsuper(\"Unable to unify \" + arg1 + \" with \" + arg2 + \".\");\n }\n\n /** Returns exception message. */\n public String toString() {\n \tString msg = getMessage();\n return \"Unify Failure: \" + (msg != null ? msg : \"(no message)\");\n }\n}\n" }, { "path": "src/opennlp/ccg/unify/Variable.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2002-7 Jason Baldridge, Gann Bierner and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.unify;\n\nimport gnu.trove.TObjectIntHashMap;\n\n/**\n * A variable that can stand for some class of Unifiable objects.\n *\n * @author Gann Bierner\n * @author \t\tMichael White\n * @version $Revision: 1.2 $, $Date: 2007/12/20 21:30:22 $\n **/\npublic interface Variable extends Unifiable {\n\n /**\n * Returns the name of this variable.\n *\n * @return the variable's name\n **/ \n public String name();\n\n /**\n\t * Returns a hash code using the given map from vars to ints.\n\t */\n\tpublic int hashCode(TObjectIntHashMap varMap);\n \n /**\n\t * Returns whether this var equals the given object up to variable names,\n\t * using the given maps from vars to ints.\n\t */\n public boolean equals(Object obj, TObjectIntHashMap varMap, TObjectIntHashMap varMap2);\n}\n" }, { "path": "src/opennlp/ccg/util/ArrayListWithIdentityEquals.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-4 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport java.util.*;\n\n/**\n * An array list where equality is checked using == on the list elements, which \n * are assumed to be canonical.\n * The hashCode method is compatible with SingletonList and StructureSharingList.\n *\n * @author Michael White\n * @version $Revision: 1.4 $, $Date: 2009/12/21 03:27:18 $\n */\npublic class ArrayListWithIdentityEquals extends ArrayList {\n \n\tprivate static final long serialVersionUID = 1L;\n\n\t/** Default constructor. */\n public ArrayListWithIdentityEquals() {}\n \n /** Constructor with initial collection. */\n public ArrayListWithIdentityEquals(Collection c) { super(c); }\n \n /** Constructor with initial capacity. */\n public ArrayListWithIdentityEquals(int initialCapacity) { super(initialCapacity); }\n \n \n /** Returns a hash code for this list, using identity hash codes of the list elements. */\n public int hashCode() {\n int hc = 1;\n for (int i = 0; i < size(); i++) {\n hc = 31*hc + System.identityHashCode(get(i));\n }\n return hc;\n }\n \n /** Returns whether this list equals the given object, using identity tests on the list elements. */\n public boolean equals(Object obj) {\n if (this == obj) return true;\n if (!(obj instanceof List)) return false;\n List list = (List) obj;\n if (size() != list.size()) return false;\n for (int i = 0; i < size(); i++) {\n if (get(i) != list.get(i)) return false;\n }\n return true; \n }\n}\n" }, { "path": "src/opennlp/ccg/util/CompositeFilter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.util;\n\nimport java.util.Arrays;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.HashSet;\nimport java.util.Set;\n\n/**\n * Composes the effects of several {@linkplain Filter filters} into a single filter.\n * The component filters are interpreted conjunctively, so that they all must allow a\n * given element for the composite filter to allow it.\n * \n * @author Scott Martin\n */\npublic class CompositeFilter implements Filter {\n\n\tSet> filters;\n\t\n\t/**\n\t * Creates a new empty composite filter.\n\t */\n\tpublic CompositeFilter() {\n\t\tthis.filters = new HashSet>();\n\t}\n\t\n\t/**\n\t * Creates a new composite filter made up of the specified filters.\n\t * @see #CompositeFilter(Collection)\n\t */\n\t@SafeVarargs\n\tpublic CompositeFilter(Filter... edgeFilters) {\n\t\tthis(Arrays.asList(edgeFilters));\n\t}\n\t\n\t/**\n\t * Creates a new composite filter made up of the specified filters.\n\t * The specified filters can be filters on an superclass of this filter's\n\t * type parameter.\n\t */\n\tpublic CompositeFilter(Collection> filters) {\n\t\tthis.filters = new HashSet>(filters);\n\t}\n\t\n\t/**\n\t * Gets the set of filters this composite filter is made up of. \n\t * @return An unmodifiable set view of the filters making up this composite filter.\n\t */\n\tpublic Set> filters() {\n\t\treturn Collections.unmodifiableSet(filters);\n\t}\n\t\n\t/**\n\t * Tests whether this filter contains a given filter.\n\t * @return true if the specified filter is one of the ones making up this \n\t * composite filter.\n\t */\n\tpublic boolean containsFilter(Filter filter) {\n\t\treturn filters.contains(filter);\n\t}\n\t\n\t/**\n\t * Adds a filter to this composite filter, if it is not already present.\n\t * @param filter The filter to add.\n\t * @return true if the filter was not already contained.\n\t * @throws IllegalArgumentException If filter is null.\n\t * \n\t * @see Collection#add(Object)\n\t */\n\tpublic boolean addFilter(Filter filter) {\n\t\tif(filter == null) {\n\t\t\tthrow new IllegalArgumentException(\"filter is null\");\n\t\t}\n\t\t\n\t\treturn filters.add(filter);\n\t}\n\t\n\t/**\n\t * Removes the specified filter.\n\t * @param filter The filter to remove.\n\t * @return true if the specified filter was removed from this composite filter.\n\t */\n\tpublic boolean removeFilter(Filter filter) {\n\t\treturn filters.remove(filter);\n\t}\n\n\t/**\n\t * Tests whether the given element is allowed by applying each of this composite filter's components\n\t * to it one by one, calling each of their {@link Filter#allows(Object)} method exactly once wit the\n\t * specified argument.\n\t * \n\t * @return false if one of the filters making up this composite filters returns false from its\n\t * {@link Filter#allows(Object)} method for the argument e, otherwise true. In particular,\n\t * this means that an empty composite filter returns true for every argument.\n\t */\n\t@Override\n\tpublic boolean allows(E e) {\n\t\tfor(Filter f : filters) {\n\t\t\tif(!f.allows(e)) {\n\t\t\t\treturn false;\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn true;\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/util/DelegatedFilter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\n/**\n * Abstract class for filters that delegates to another filter that tests elements that do not\n * necessarily apply to the same type of elements as this filter. This filter's\n * {@link #allows(Object)} method returns the value of the \n * {@linkplain #getDelegateFilter() delegate filter}'s allows(...) method\n * for the value returned by {@link #delegateValueFor(Object)} for the specified\n * element.\n *

    \n * Concrete subclasses need to specify (1) the delegate filter (via their constructor), and\n * (2) a way to determine which element of type D the delegate filter should use\n * based on a specified element of type E by implementing {@link #delegateValueFor(Object)}.\n * A typical example is the case when a filter is desired that compares elements of type E, but\n * the comparison needs to take place on some type-D object somehow derived from instances of\n * E, e.g. by an accessor method.\n * \n * @param The type of elements that this filter applies to.\n * @param The type of elements that the delegated filter applies to.\n * \n * @author Scott Martin\n */\npublic abstract class DelegatedFilter implements Filter {\n\n\tFilter delegateFilter;\n\t\n\t/**\n\t * Creates a delegated filter with the specified filter to delegate to. The delegated filter will be\n\t * used in the test for {@link #allows(Object)}, through the {@link #delegateValueFor(Object)}.\n\t */\n\tprotected DelegatedFilter(Filter delegateFilter) {\n\t\tthis.delegateFilter = delegateFilter;\n\t}\n\n\t/**\n\t * Gets the filter that this filter delegates to.\n\t * @return The filter specified at creation.\n\t * @see #DelegatedFilter(Filter)\n\t */\n\tpublic Filter getDelegateFilter() {\n\t\treturn delegateFilter;\n\t}\n\n\t/**\n\t * Tests whether this filter allows the specified element by testing whether its\n\t * {@linkplain #getDelegateFilter() delegate filter} allows the value of {@link #delegateValueFor(Object)}\n\t * for the argument e.\n\t * @return true if the delegate filter's {@link Filter#allows(Object)} method returns true\n\t * for the element returned by delegateValueFor(e).\n\t * @see #delegateValueFor(Object)\n\t */\n\t@Override\n\tpublic boolean allows(E e) {\n\t\treturn delegateFilter.allows(delegateValueFor(e));\n\t}\n\n\t/**\n\t * Gets the element of type D that the delegated filter should use in its\n\t * {@link Filter#allows(Object)} comparisons, given the specified type-E element.\n\t * @param e The element to obtain a type-D element for.\n\t * @return The element that the delegated filter should use for comparison, based on e.\n\t */\n\tpublic abstract D delegateValueFor(E e);\n}\n" }, { "path": "src/opennlp/ccg/util/DisplayPrefs.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2006 Ben Wing\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\npackage opennlp.ccg.util;\r\n\r\nimport java.util.prefs.Preferences;\r\n\r\nimport opennlp.ccg.TextCCG;\r\n\r\n/**\r\n * Simple class for holding preferences for converting an object to a\r\n * string or other displayable representation.\r\n *\r\n * @author Ben Wing\r\n * @version $Revision: 1.5 $, $Date: 2007/06/22 01:52:21 $\r\n */\r\npublic class DisplayPrefs {\r\n\r\n /* Whether to show feature info along with each nonterminal */\r\n public boolean showFeats = false; \r\n /* Whether to show semantic info (logical forms) */\r\n public boolean showSem = false;\r\n /* Which features to show. */\r\n public String featsToShow = \"\";\r\n \r\n /** Constructor sets initial prefs from current user prefs. */\r\n public DisplayPrefs() {\r\n \tPreferences prefs = Preferences.userNodeForPackage(TextCCG.class);\r\n \tshowFeats = prefs.getBoolean(TextCCG.SHOW_FEATURES, false);\r\n \tshowSem = prefs.getBoolean(TextCCG.SHOW_SEMANTICS, false);\r\n \tfeatsToShow = prefs.get(TextCCG.FEATURES_TO_SHOW, \"\");\r\n }\r\n}" }, { "path": "src/opennlp/ccg/util/Filter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\n/**\n * Interface for filters that characterize a set by some membership criteria.\n * This interface uses a single method, {@link #allows(Object)}, to allow implementing\n * classes to say whether the specified element should be a member of the collection or not.\n *

    \n * Filters can be thought of as characteristic functions for sets. The type parameter is \n * used to signal what kind of elements a filter applies to.\n * \n * @param The type of elements that this filter applies to.\n * \n * @see FilteredSet\n * @see FilteredMap\n * @author Scott Martin\n */\npublic interface Filter {\n\n\t/**\n\t * Tests whether the specified element is allowed.\n\t * @return true if the provided element should be allowed into the collection.\n\t */\n\tboolean allows(E e);\n}\n" }, { "path": "src/opennlp/ccg/util/FilteredMap.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\npackage opennlp.ccg.util;\n\nimport java.util.AbstractMap;\nimport java.util.LinkedHashMap;\nimport java.util.Map;\nimport java.util.Set;\n\n/**\n * A map whose keys must be allowed by a specified {@linkplain #getKeyFilter() key filter}.\n * The filtered mappings are kept in the same order they occurred in in the\n * {@linkplain #getOriginalMap() original map}.\n * \n * @see Filter\n * @author Scott Martin\n */\npublic class FilteredMap extends AbstractMap {\n\n\tMap originalMap;\n\tFilter keyFilter;\n\t\n\tprivate final Map map = new LinkedHashMap();\n\t\n\t/**\n\t * Creates a new filtered map including only the elements in originalMap whose\n\t * keys are {@linkplain Filter#allows(Object) allowed} by the specified\n\t * keyFilter.\n\t *

    \n\t * Filtered maps maintain an {@linkplain #entrySet() entry set} whose entries occur in\n\t * the same order as they occurred in the original map.\n\t * \n\t * @param originalMap The map to draw this map's elements from.\n\t * @param keyFilter The filter that decides which keys from the original map should be\n\t * represented in this filtered map.\n\t * @throws IllegalArgumentException If keyFilter is null.\n\t * @see Map#putAll(Map)\n\t */\n\tpublic FilteredMap(Map originalMap, Filter keyFilter) {\n\t\tif(keyFilter == null) {\n\t\t\tthrow new IllegalArgumentException(\"keyFilter is null\");\n\t\t}\n\t\t\n\t\tthis.originalMap = originalMap;\n\t\tthis.keyFilter = keyFilter;\n\t\t\n\t\tputAll(originalMap);\n\t}\n\n\t/**\n\t * Gets the map that this map draws its elements from.\n\t * @return The map specified at creation\n\t * @see #FilteredMap(Map, Filter)\n\t */\n\tpublic Map getOriginalMap() {\n\t\treturn originalMap;\n\t}\n\t\n\t/**\n\t * Gets this map's key filter.\n\t */\n\tpublic Filter getKeyFilter() {\n\t\treturn keyFilter;\n\t}\t\n\n\t/**\n\t * Gets the entry set for this map. Each entry's key is\n\t * guaranteed to be allowable according to this map's {@linkplain #getKeyFilter() key \n\t * filter}.\n\t * \n\t * @return The subset of the {@linkplain #getOriginalMap() original map}'s entries that\n\t * are allowable by the key filter.\n\t * \n\t * @see Map#entrySet()\n\t */\n\t@Override\n\tpublic Set> entrySet() {\n\t\treturn map.entrySet();\n\t}\n\n\t/**\n\t * Provides the ability to put new mappings into this filtered map, provided the specified\n\t * key is {@linkplain Filter#allows(Object) allowed} by this map's\n\t * {@linkplain #getKeyFilter() key filter}.\n\t * \n\t * @return the element previously associated with key if the specified\n\t * key is allowed by the key filter (and null if none was associated).\n\t * This method always returns null for key/value pairs in which the specified\n\t * key is not allowed by the key filter in effect. \n\t */\n\t@Override\n\tpublic V put(K key, V value) {\n\t\tif(keyFilter.allows(key)) {\n\t\t\treturn map.put(key, value);\n\t\t}\n\t\t\n\t\treturn null;\n\t}\n}\n" }, { "path": "src/opennlp/ccg/util/FilteredSet.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport java.util.AbstractSet;\nimport java.util.Collection;\nimport java.util.Iterator;\nimport java.util.LinkedHashSet;\nimport java.util.Set;\n\n/**\n * A set whose elements must be {@linkplain Filter#allows(Object) allowed} by a specified\n * {@link Filter}. Conceptually, instances of this class represent the sets described by the\n * characteristic function corresponding to their {@linkplain #getFilter() filter}.\n *

    \n * The elements in the filtered set are kept in the same order they occur in according to the\n * {@linkplain #getOriginalCollection() original collection}'s iterator.\n * \n * @see Filter \n * @author Scott Martin \n */\npublic class FilteredSet extends AbstractSet {\n\n\tCollection originalCollection;\n\tFilter filter;\n\t\n\tprivate final Set set = new LinkedHashSet();\n\t\n\t/**\n\t * Creates a new filtered set based on the specified collection and filter. The resulting set will\n\t * contain all the members of the original collection for which the specified filter's\n\t * {@link Filter#allows(Object)} method returns true.\n\t * @param originalCollection The collection from which this filtered set will draw its elements.\n\t * @param filter The filter that decides which of the members of originalCollection are\n\t * allowable. The specified filter can apply to elements of any superclass of this filtered set's\n\t * type parameter.\n\t * @throws IllegalArgumentException If filter is null.\n\t * \n\t * @see #addAll(Collection)\n\t */\n\tpublic FilteredSet(Collection originalCollection, Filter filter) {\n\t\tif(filter == null) {\n\t\t\tthrow new IllegalArgumentException(\"filter is null\");\n\t\t}\n\t\t\n\t\tthis.filter = filter;\n\t\tthis.originalCollection = originalCollection;\n\t\t\n\t\taddAll(originalCollection);\n\t}\n\n\t/**\n\t * Gets the original collection from which this filtered set's elements are drawn.\n\t * @return The collection specified at creation.\n\t * @see #FilteredSet(Collection, Filter)\n\t */\n\tpublic Collection getOriginalCollection() {\n\t\treturn originalCollection;\n\t}\n\n\t/**\n\t * Gets the filter used by this filtered set to determine which elements are allowed in it.\n\t * @return The filter specified at creation.\n\t * @see #FilteredSet(Collection, Filter)\n\t */\n\tpublic Filter getFilter() {\n\t\treturn filter;\n\t}\n\n\t/**\n\t * Gets an iterator over the elements in this filtered set.\n\t */\n\t@Override\n\tpublic Iterator iterator() {\n\t\treturn set.iterator();\n\t}\n\n\t/**\n\t * Gets the size of this filtered set (the number of elements it contains).\n\t */\n\t@Override\n\tpublic int size() {\n\t\treturn set.size();\n\t}\n\n\t/**\n\t * Adds an element if it conforms to the {@linkplain #getFilter() filter in effect}, determined by\n\t * consulting the filter's {@link Filter#allows(Object)} method using the supplied element. \n\t * @param e The element to add, after testing its allowability according to this filtered set's filter.\n\t * @return true if the filter allows e and this set changed as a result of the addition\n\t * (because the specified element e was not already contained).\n\t */\n\t@Override\n\tpublic boolean add(E e) {\n\t\treturn filter.allows(e) && set.add(e);\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/util/GroupMap.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-6 Jason Baldridge, Gann Bierner and \n// Michael White (University of Edinburgh, The Ohio State University)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport gnu.trove.*;\n\nimport java.io.Serializable;\nimport java.util.*;\n\n/**\n * A map where putting a value does not replace an old value but is rather\n * included in a set of values for that key.\n * The map may use identity equals on keys. \n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.9 $, $Date: 2009/07/17 04:23:30 $\n */\npublic class GroupMap implements Serializable {\n \n\tprivate static final long serialVersionUID = -2995356057195571222L;\n\t\n\t// the underlying map\n\tprivate THashMap map;\n\t\n\t/** Default constructor. */\n\tpublic GroupMap() { this(false); }\n\t\n\t/** Constructor with flag for whether to use identity instead of equals on keys. */\n\tpublic GroupMap(boolean useIdentityEquals) {\n\t\tif (useIdentityEquals) map = new THashMap(new TObjectIdentityHashingStrategy());\n\t\telse map = new THashMap();\n\t}\n\t\n /** Adds the given key-value pair to the map, and returns null. */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic Object put(KeyType key, ValType value) {\n // get current val\n Object currentVal = map.get(key);\n // if none, add value to map\n if (currentVal == null) { \n \tmap.put(key, value);\n }\n // if already a set, add value to set\n else if (currentVal instanceof Set) {\n Set set = (Set) currentVal;\n set.add(value);\n }\n // otherwise replace with a set including both values\n else {\n Set set = new THashSet();\n set.add((ValType)currentVal);\n set.add(value);\n map.put(key, set);\n }\n // return null, since we're not really replacing the old val\n return null;\n }\n\n /** Returns the set of values for the given key (or null). */\n @SuppressWarnings(\"unchecked\")\n\tpublic Set get(KeyType key) {\n // get val\n Object val = map.get(key);\n // return if null or already a set\n if (val == null || val instanceof Set) {\n return (Set) val;\n }\n // otherwise replace val with a set and return it\n Set set = new THashSet();\n set.add((ValType)val);\n map.put(key, set);\n return set;\n }\n \n /** Adds a key-value pair to the map for all the given vals. */\n public void putAll(KeyType key, Collection vals) {\n \tfor (ValType val : vals) put(key, val);\n }\n \n \n /** Returns the size of the underlying map. */\n public int size() { return map.size(); }\n\n /** Returns the keys. */\n @SuppressWarnings(\"unchecked\")\n\tpublic Set keySet() {\n \treturn (Set) map.keySet();\n }\n \n /** Returns whether the keys contain the given one. */\n public boolean containsKey(KeyType key) {\n \treturn map.containsKey(key);\n }\n \n /** Removes the given key, returning its previous value (if any). */\n Set remove(KeyType key) {\n \tSet retval = get(key);\n \tmap.remove(key);\n \treturn retval;\n }\n}\n" }, { "path": "src/opennlp/ccg/util/IntHashSetMap.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport gnu.trove.*;\n\n/**\n * A map from ints to sets which allows objects with the same key to be\n * added without overriding previous puts.\n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.6 $, $Date: 2005/10/13 20:33:49 $\n */\npublic class IntHashSetMap extends TIntObjectHashMap {\n\n\tprivate static final long serialVersionUID = 1L;\n\n\t/** Adds the given key-value pair to the map. */\n public Object put(int key, Object value) {\n \tTHashSet val = (THashSet) get(key);\n if (val==null) {\n val = new THashSet();\n val.add(value);\n super.put(key, val); \n } else {\n val.add(value);\n }\n return val;\n }\n}\n" }, { "path": "src/opennlp/ccg/util/Interner.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport java.util.*;\nimport java.lang.ref.*;\n\n/**\n * A utility class for interning (canonicalizing) objects.\n * A WeakHashMap is used as the backing store, so that interned objects can\n * be garbage collected.\n * Normally, it is easier to use the globalIntern method (sharing a global \n * backing store) than to allocate separate interners.\n * Individual interners can be constructed to use soft references to \n * the interned objects, so that they are kept around longer than is the \n * case with weak references (the default). \n *\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2005/10/13 20:33:49 $\n *\n */\npublic class Interner {\n\n // the backing store\n private Map> weakMap = new WeakHashMap>();\n \n // flag for whether to use soft references \n private boolean softRefs = false;\n \n /** Default constructor. */\n public Interner() {}\n\n /** Constructor with soft references flag. */\n public Interner(boolean softRefs) { this.softRefs = softRefs; }\n\n /** \n * Returns a canonical version of the given object.\n * The returned object is .equals() to the given one.\n * If the given object is not equal to one already seen, \n * then the returned object will be == to the given one.\n */\n public T intern(T obj) {\n // check if equivalent key already in map\n if (weakMap.containsKey(obj)) {\n // return existing canonical obj if so \n Reference ref = weakMap.get(obj);\n return ref.get();\n }\n // otherwise add this object to the map, wrapped in a \n // weak/soft reference so that it can still be gc'ed\n Reference ref = (softRefs) \n ? new SoftReference(obj) \n : new WeakReference(obj);\n weakMap.put(obj, ref);\n return obj;\n }\n \n /** \n * Returns the canonical version of the given object, if any, \n * otherwise returns null.\n */\n public T getInterned(T obj) {\n // get weak reference to canonical obj, if any\n Reference ref = weakMap.get(obj);\n // return obj, if any, otherwise null\n return (ref != null) ? ref.get() : null;\n }\n \n /** Returns the number of interned objects. */\n public int size() {\n return weakMap.size();\n }\n \n \n // the global interner\n private static Interner globalInterner = null;\n \n /** \n * Returns a canonical version of the given object using a global interner.\n * The returned object is .equals() to the given one.\n * If the given object is not equal to one already seen, \n * then the returned object will be == to the given one.\n */\n public static Object globalIntern(Object obj) {\n if (globalInterner == null) globalInterner = new Interner();\n return globalInterner.intern(obj);\n }\n \n /** \n * Returns the canonical version of the given object using the global interner, if any, \n * otherwise returns null.\n */\n public static Object getGlobalInterned(Object obj) {\n if (globalInterner == null) return null;\n return globalInterner.getInterned(obj);\n }\n \n /** Returns the number of interned objects in the global interner. */\n public static int globalSize() {\n if (globalInterner == null) return 0;\n return globalInterner.size();\n }\n \n \n /** Tests the implementation. */\n public static void main(String[] args) {\n\n Interner interner = new Interner();\n int SIZE = 100000;\n Integer[] ints = new Integer[SIZE];\n\n System.out.println(\"Adding \" + SIZE + \" ints to interner.\");\n for (int i = 0; i < SIZE; i++) {\n ints[i] = new Integer(i);\n Integer interned = interner.intern(ints[i]);\n if (interned != ints[i]) {\n System.out.println(\"Whoops: ints[\" + i + \"] not == to interned: \" + interned);\n System.exit(-1);\n }\n }\n System.out.println(\"interner.size(): \" + interner.size()); // should be SIZE\n System.out.println(\"Doing gc().\");\n System.gc();\n System.out.println(\"interner.size(): \" + interner.size()); // should be the same\n System.out.println();\n \n System.out.println(\"Now adding \" + SIZE + \" equivalent ints to interner.\");\n for (int i = 0; i < SIZE; i++) {\n Integer intI = new Integer(i);\n Integer interned = interner.intern(intI);\n if (interned == intI) {\n System.out.println(\"Whoops: intI (i=\" + i + \") is == to interned: \" + interned);\n System.exit(-1);\n }\n }\n System.out.println(\"interner.size(): \" + interner.size()); // should be the same\n System.out.println();\n \n System.out.println(\"Next adding \" + SIZE + \" new, unreferenced ints to interner.\");\n for (int i = SIZE; i < SIZE*2; i++) {\n Integer intI = new Integer(i);\n Integer interned = interner.intern(intI);\n if (interned != intI) {\n System.out.println(\"Whoops: intI (i=\" + i + \") not == to interned: \" + interned);\n System.exit(-1);\n }\n }\n System.out.println(\"interner.size(): \" + interner.size()); // should be larger than SIZE\n System.out.println(\"Doing gc().\");\n System.gc();\n System.out.println(\"interner.size(): \" + interner.size()); // should be back to SIZE\n System.out.println();\n }\n}\n" }, { "path": "src/opennlp/ccg/util/InverseFilter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\n/**\n * A filter that wraps another filter an inverts its effects. Specifically, for each argument that\n * the wrapped filter's {@link Filter#allows(Object)} method returns true, this filter\n * returns false, and vice versa.\n * \n * @author Scott Martin\n */\npublic class InverseFilter implements Filter {\n\n\tFilter originalFilter;\n\t\n\t/**\n\t * Creates a new filter based on the specified filter, inverting its effects. The specified filter can\n\t * apply to any superclass of this filter's type parameter.\n\t * @param originalFilter The filter to invert.\n\t * @throws IllegalArgumentException If originalFilter is null.\n\t */\n\tpublic InverseFilter(Filter originalFilter) {\n\t\tif(originalFilter == null) {\n\t\t\tthrow new IllegalArgumentException(\"originalFilter is null\");\n\t\t}\n\t\t\n\t\tthis.originalFilter = originalFilter;\n\t}\n\t\n\t/**\n\t * Gets the original, non-inverted filter that this inverse filter wraps.\n\t * @return The filter specified at creation.\n\t * @see #InverseFilter(Filter)\n\t */\n\tpublic Filter getOriginalFilter() {\n\t\treturn originalFilter;\n\t}\n\n\t/**\n\t * Tests whether this filter allows a specified element by calling the original filter's\n\t * {@link Filter#allows(Object)} method and reversing its boolean value.\n\t * \n\t * @param e The element to test.\n\t * @return A value equivalent to calling !getOriginalFilter().allows(e).\n\t */\n\t@Override\n\tpublic boolean allows(E e) {\n\t\treturn !originalFilter.allows(e);\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/util/JLineReader.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 David Reitter and University of Edinburgh (Michael White) \n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport java.io.*;\nimport java.util.*;\n\nimport jline.*;\n\n/**\n * A command-line reader based on JLine.\n *\n * @author David Reitter\n * @author Michael White\n * @version $Revision: 1.4 $, $Date: 2009/12/21 03:27:18 $\n */\npublic class JLineReader extends LineReader {\n\n\t// reader for console input\n ConsoleReader reader;\n\n // command history, buffer \n History history = null;\n StringWriter histbuf = null;\n \n /** Constructor with completion strings. */\n public JLineReader(String[] completions) throws IOException {\n // init reader\n reader = new ConsoleReader();\n\t\t// store commands for 'tab' argument completion \n \tList completors = new LinkedList();\n \tcompletors.add(new SimpleCompletor(completions));\n reader.addCompletor(new ArgumentCompletor(completors));\n }\n \n /** Sets the command history. */\n public void setCommandHistory(String histStr) throws IOException {\n\t\t// initialize history with max size = 50\n\t\thistory = new History();\n history.setMaxSize(50);\n\t\tif (!histStr.equals(\"\")) {\n histStr = histStr.replaceAll(\"
    \", \"\\n\"); // using
    to get around XML problem in Java 1.4\n\t\t\tStringReader sreader = new StringReader(histStr);\n\t\t\thistory.load(sreader); \n\t\t}\n // set to reader's history\n\t\treader.setHistory(history);\n }\n \n /** Gets the current command history. */\n public String getCommandHistory() throws IOException {\n if (history == null) return \"\";\n StringBuffer retbuf = new StringBuffer();\n List commands = history.getHistoryList();\n for (Iterator it = commands.iterator(); it.hasNext(); ) {\n retbuf.append(it.next().toString());\n if (it.hasNext()) retbuf.append(\"
    \"); // using
    to get around XML problem in Java 1.4\n }\n return retbuf.toString();\n }\n \n /** Returns an input string, using the given prompt. */\n public String readLine(String prompt) throws IOException {\n\t return reader.readLine(prompt);\n }\n}\n" }, { "path": "src/opennlp/ccg/util/LineReader.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White) and David Reitter \n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport java.io.*;\n\n/**\n * Abstract command-line reader.\n *\n * @author Michael White\n * @author David Reitter\n * @version $Revision: 1.2 $, $Date: 2005/10/13 20:33:49 $\n */\nabstract public class LineReader {\n\n /** Creates a default line reader (currently a JLineReader) with the given completion strings. */\n public static LineReader createLineReader(String[] completions) throws IOException {\n return new JLineReader(completions);\n }\n \n /** Sets the command history. */\n abstract public void setCommandHistory(String histStr) throws IOException;\n \n /** Gets the current command history. */\n abstract public String getCommandHistory() throws IOException;\n \n /** Returns an input string, using the given prompt. */\n abstract public String readLine(String prompt) throws IOException;\n}\n" }, { "path": "src/opennlp/ccg/util/ListMap.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2006 Michael White (The Ohio State University)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport gnu.trove.*;\nimport java.util.*;\n\n/**\n * A map where putting a value does not replace an old value \n * but is instead included in a list of values for that key.\n * The map may use identity equals on keys. \n * (NB: A ListMap is essentially a GroupMap that uses lists instead of sets.)\n *\n * @author Michael White\n * @version $Revision: 1.1 $, $Date: 2006/08/15 18:21:31 $\n */\npublic class ListMap {\n \n\t// the underlying map\n\tprivate THashMap map;\n\t\n\t/** Default constructor. */\n\tpublic ListMap() { this(false); }\n\t\n\t/** Constructor with flag for whether to use identity instead of equals on keys. */\n\tpublic ListMap(boolean useIdentityEquals) {\n\t\tif (useIdentityEquals) map = new THashMap(new TObjectIdentityHashingStrategy());\n\t\telse map = new THashMap();\n\t}\n\t\n /** Adds the given key-value pair to the map, and returns null. */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic Object put(KeyType key, ValType value) {\n // get current val\n Object currentVal = map.get(key);\n // if none, add value to map\n if (currentVal == null) { \n \tmap.put(key, value);\n }\n // if already a list, add value to list\n else if (currentVal instanceof List) {\n List list = (List) currentVal;\n list.add(value);\n }\n // otherwise replace with a list including both values\n else {\n List list = new ArrayList(3);\n list.add((ValType)currentVal);\n list.add(value);\n map.put(key, list);\n }\n // return null, since we're not really replacing the old val\n return null;\n }\n\n /** Returns the list of values for the given key (or null). */\n @SuppressWarnings(\"unchecked\")\n\tpublic List get(KeyType key) {\n // get val\n Object val = map.get(key);\n // return if null or already a list\n if (val == null || val instanceof List) {\n return (List) val;\n }\n // otherwise replace val with a list and return it\n List list = new ArrayList(1);\n list.add((ValType)val);\n map.put(key, list);\n return list;\n }\n \n /** Adds a key-value pair to the map for all the given vals. */\n public void putAll(KeyType key, Collection vals) {\n \tfor (ValType val : vals) put(key, val);\n }\n \n \n /** Returns the size of the underlying map. */\n public int size() { return map.size(); }\n\n /** Returns the keys. */\n @SuppressWarnings(\"unchecked\")\n\tpublic Set keySet() {\n \treturn (Set) map.keySet();\n }\n \n /** Returns whether the keys contain the given one. */\n public boolean containsKey(KeyType key) {\n \treturn map.containsKey(key);\n }\n \n /** Removes the given key, returning its previous value (if any). */\n List remove(KeyType key) {\n \tList retval = get(key);\n \tmap.remove(key);\n \treturn retval;\n }\n}\n" }, { "path": "src/opennlp/ccg/util/MembershipFilter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport java.util.Set;\n\n/**\n * A filter that allows elements based on their membership in a set specified at creation. Later modifications\n * to this set will be reflected in the behavior of this filter's {@link #allows(Object)} method because\n * the set is not copied at creation.\n * \n * @author Scott Martin\n */\npublic class MembershipFilter implements Filter {\n\n\t/**\n\t * The set to test for membership.\n\t */\n\tprotected Set members;\n\t\n\t/**\n\t * Creates a new membership filter based on the specified set of members. This set can later be modified\n\t * and have its new membership reflected by this filter's {@link #allows(Object)} method because the\n\t * specified set is not copied by this filter.\n\t * \n\t * @param members The set to test for membership.\n\t * @throws IllegalArgumentException If members is null.\n\t */\n\tpublic MembershipFilter(Set members) {\n\t\tif(members == null) {\n\t\t\tthrow new IllegalArgumentException(\"members is null\");\n\t\t}\n\t\t\n\t\tthis.members = members;\n\t}\n\n\t/**\n\t * Tests whether this membership filter allows the specified element by testing whether the membership\n\t * set contains the element.\n\t * \n\t * @param e The element to test membership for. \n\t * @return true if the set of members specified at creation contains e.\n\t * @see #MembershipFilter(Set)\n\t */\n\t@Override\n\tpublic boolean allows(E e) {\n\t\treturn members.contains(e);\n\t}\n\n}\n" }, { "path": "src/opennlp/ccg/util/Pair.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003 Jason Baldridge, Gann Bierner and \n// University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport java.io.Serializable;\n\n/**\n * Dinky class to package pairs of things.\n *\n * @author Jason Baldridge\n * @author Gann Bierner\n * @author Michael White\n * @version $Revision: 1.7 $, $Date: 2009/12/21 03:27:18 $\n */\npublic final class Pair implements Serializable {\n \n\tprivate static final long serialVersionUID = 3626104184233533389L;\n\n\t/** The first element of the pair. */\n public final TypeA a; \n\n /** The second element of the pair. */\n public final TypeB b;\n\n /** Constructor. */\n public Pair(TypeA a, TypeB b) {\n this.a = a; this.b = b; \n }\n \n /** Returns a hash code constructed from those of a and b. */\n public int hashCode() { return a.hashCode() - b.hashCode(); }\n \n /** Returns true if the given object pairs the same elements. */\n public boolean equals(Object obj) {\n if (this == obj) return true;\n if (!(obj instanceof Pair)) return false;\n Pair p = (Pair) obj;\n return a.equals(p.a) && b.equals(p.b);\n }\n\n /** Returns \"[a/b]\". */\n public String toString() { return \"[\"+a+\"/\"+b+\"]\"; }\n}\n" }, { "path": "src/opennlp/ccg/util/SingletonList.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2004 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport java.io.Serializable;\nimport java.util.*;\n\n/**\n * An unmodifiable list of one element.\n * Equality is checked using == on the list element, which \n * is assumed to be canonical.\n *\n * @author Michael White\n * @version $Revision: 1.5 $, $Date: 2009/12/21 03:27:18 $\n */\npublic class SingletonList extends AbstractList implements Serializable {\n \n\tprivate static final long serialVersionUID = -4340168177098319085L;\n\t\n\t/** The single list element. */\n public final T elt;\n\n /** Constructor. */\n public SingletonList(T elt) {\n this.elt = elt;\n }\n \n /** Returns the size of this list. */\n public int size() { return 1; }\n \n /** Returns the ith element of the list. */\n public T get(int i) {\n if (i == 0) return elt;\n else throw new IndexOutOfBoundsException(\"No element with index: \" + i);\n }\n \n /** Returns a hash code for this list, using the identity hash code of the list element. */\n public int hashCode() {\n return 31 + System.identityHashCode(elt);\n }\n \n /** Returns whether this list equals the given object, using identity tests on the list element. */\n public boolean equals(Object obj) {\n if (this == obj) return true;\n if (!(obj instanceof List)) return false;\n List list = (List) obj;\n if (size() != list.size()) return false;\n return (get(0) == list.get(0));\n }\n}\n" }, { "path": "src/opennlp/ccg/util/StructureSharingList.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2003-4 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport java.io.Serializable;\nimport java.util.*;\n\n/**\n * An unmodifiable list formed by sequencing two sublists. \n * The sublists are assumed to remain unchanged as well.\n * Equality is checked using == on the list elements, which \n * are assumed to be canonical.\n *\n * @author Michael White\n * @version $Revision: 1.7 $, $Date: 2009/12/21 03:27:18 $\n */\npublic class StructureSharingList extends AbstractList implements Serializable {\n \n\tprivate static final long serialVersionUID = 6692080357319326492L;\n\n\t/** The first sublist. */\n public final List first; \n\n /** The second sublist. */\n public final List second;\n \n // cached hashcode\n private int hashcode = -1;\n\n // size\n private final int size;\n \n /** Constructor. */\n public StructureSharingList(List first, List second) {\n this.first = first; this.second = second;\n this.size = first.size() + second.size();\n }\n \n /** Returns the size of this list. */\n public int size() { return size; }\n \n /** Returns the ith element of the list. */\n public T get(int i) {\n if (i < first.size()) { \n return first.get(i); \n }\n else {\n return second.get(i - first.size());\n }\n }\n \n /** Returns a hash code for this list, using identity hash codes of the list elements. */\n public int hashCode() {\n // check whether already cached\n if (hashcode != -1) return hashcode;\n int hc = 1;\n for (int i = 0; i < size(); i++) {\n hc = 31*hc + System.identityHashCode(get(i));\n }\n // cache then return\n hashcode = hc;\n return hc;\n }\n \n /** Returns whether this list equals the given object, using identity tests on the list elements. */\n public boolean equals(Object obj) {\n if (this == obj) return true;\n if (!(obj instanceof List)) return false;\n List list = (List) obj;\n if (list instanceof StructureSharingList) {\n StructureSharingList ssl = (StructureSharingList) list;\n if (first == ssl.first && second == ssl.second) return true;\n }\n if (size() != list.size()) return false;\n for (int i = 0; i < size(); i++) {\n if (get(i) != list.get(i)) return false;\n }\n return true; \n }\n}\n" }, { "path": "src/opennlp/ccg/util/TrieMap.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005 University of Edinburgh (Michael White)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport java.util.*;\n\n/**\n * Implements a trie with a data object at each node. Keys are assumed to be\n * canonical, and thus checked using identity (==) rather than equality. For\n * efficient allocation, all children can be added at once.\n * \n * @author Michael White\n * @version $Revision: 1.6 $, $Date: 2011/03/20 20:11:58 $\n */\npublic class TrieMap {\n\n\t/** Interface for methods returning an interned key. */\n\tpublic interface KeyExtractor {\n\t\t/** Returns an interned key. */\n\t\tpublic KeyType getKey();\n\t}\n\n\t/** The data object. */\n\tpublic DataType data;\n\n\t/**\n\t * The mapping to the children. If there is just one child, it's stored in a\n\t * pair with its key. Otherwise, an IdentityHashMap is used.\n\t */\n\tprivate Object childMap = null;\n\n\t/** The parent node. */\n\tprivate TrieMap parent = null;\n\n\t/** Constructor with data object. */\n\tpublic TrieMap(DataType data) {\n\t\tthis.data = data;\n\t}\n\n\t/** Factory method, for adding empty child nodes. */\n\tprotected TrieMap createNode() {\n\t\treturn new TrieMap(null);\n\t}\n\t\n\t/** Adds the given child with its key. */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic void addChild(KeyType key, TrieMap child) {\n\t\tchild.parent = this;\n\t\tif (childMap == null) {\n\t\t\tchildMap = new Pair>(key, child);\n\t\t\treturn;\n\t\t}\n\t\tMap> map;\n\t\tif (childMap instanceof Pair) {\n\t\t\tPair> pair = (Pair>) childMap;\n\t\t\tmap = new IdentityHashMap>();\n\t\t\tmap.put(pair.a, pair.b);\n\t\t\tchildMap = map;\n\t\t} else {\n\t\t\tmap = (Map>) childMap;\n\t\t}\n\t\tmap.put(key, child);\n\t}\n\n\t/** Adds the given children with their keys. */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic void addChildren(List keys, List> childNodes) {\n\t\tif (childMap == null && keys.size() == 1) {\n\t\t\tTrieMap child = childNodes.get(0);\n\t\t\tchild.parent = this;\n\t\t\tchildMap = new Pair>(keys.get(0), child);\n\t\t\treturn;\n\t\t}\n\t\tMap> map;\n\t\tif (childMap == null) {\n\t\t\tmap = new IdentityHashMap>(keys.size());\n\t\t\tchildMap = map;\n\t\t} else if (childMap instanceof Pair) {\n\t\t\tPair> pair = (Pair>) childMap;\n\t\t\tmap = new IdentityHashMap>(keys.size() + 1);\n\t\t\tmap.put(pair.a, pair.b);\n\t\t\tchildMap = map;\n\t\t} else {\n\t\t\tmap = (Map>) childMap;\n\t\t}\n\t\tfor (int i = 0; i < keys.size(); i++) {\n\t\t\tTrieMap child = childNodes.get(i);\n\t\t\tchild.parent = this;\n\t\t\tmap.put(keys.get(i), child);\n\t\t}\n\t}\n\n\t/** Gets the parent node, or null if none. */\n\tpublic TrieMap getParent() {\n\t\treturn parent;\n\t}\n\n\t/** Gets the child for the given key, or null if none. */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic TrieMap getChild(KeyType key) {\n\t\tif (childMap == null) return null;\n\t\tif (childMap instanceof Pair) {\n\t\t\tPair> pair = (Pair>) childMap;\n\t\t\tif (pair.a == key) return pair.b;\n\t\t\telse return null;\n\t\t}\n\t\tMap> map = (Map>) childMap;\n\t\treturn map.get(key);\n\t}\n\n\t/** Gets the child for the given list of keys, or null if none. */\n\tpublic TrieMap getChildFromList(List keys) {\n\t\tTrieMap next = this;\n\t\tfor (int pos = 0; pos < keys.size(); pos++) {\n\t\t\tnext = next.getChild(keys.get(pos));\n\t\t\tif (next == null) return null;\n\t\t}\n\t\treturn next;\n\t}\n\n\t/** Gets the child for the given list of keys extractors, or null if none. */\n\tpublic TrieMap getChildFromLazyList(List> keyExtractors) {\n\t\tTrieMap next = this;\n\t\tfor (int pos = 0; pos < keyExtractors.size(); pos++) {\n\t\t\tnext = next.getChild(keyExtractors.get(pos).getKey());\n\t\t\tif (next == null) return null;\n\t\t}\n\t\treturn next;\n\t}\n\n\t/**\n\t * Finds the child for the given key, adding one (with a null data object)\n\t * if necessary.\n\t */\n\tpublic TrieMap findChild(KeyType key) {\n\t\tTrieMap child = getChild(key);\n\t\tif (child == null) {\n\t\t\tchild = createNode();\n\t\t\taddChild(key, child);\n\t\t}\n\t\treturn child;\n\t}\n\n\t/**\n\t * Finds the child for the given list of keys, adding one (with a null data\n\t * object) if necessary, along with any necessary intervening parents.\n\t */\n\tpublic TrieMap findChildFromList(List keys) {\n\t\tTrieMap next = this;\n\t\tfor (int pos=0; pos < keys.size(); pos++) {\n\t\t\tKeyType key = keys.get(pos);\n\t\t\tTrieMap child = next.getChild(key);\n\t\t\tif (child == null) {\n\t\t\t\tchild = createNode();\n\t\t\t\tnext.addChild(key, child);\n\t\t\t}\n\t\t\tnext = child;\n\t\t}\n\t\treturn next;\n\t}\n\n\t/**\n\t * Finds the child for the given list of keys, adding one (with a null data\n\t * object) if necessary, along with any necessary intervening parents.\n\t */\n\tpublic TrieMap findChildFromLazyList(List> keyExtractors) {\n\t\tTrieMap next = this;\n\t\tfor (int pos=0; pos < keyExtractors.size(); pos++) {\n\t\t\tKeyType key = keyExtractors.get(pos).getKey();\n\t\t\tTrieMap child = next.getChild(key);\n\t\t\tif (child == null) {\n\t\t\t\tchild = createNode();\n\t\t\t\tnext.addChild(key, child);\n\t\t\t}\n\t\t\tnext = child;\n\t\t}\n\t\treturn next;\n\t}\n\n\t/**\n\t * Gets the keys leading to this node. This requires a linear search at each\n\t * level.\n\t */\n\t@SuppressWarnings(\"unchecked\")\n\tpublic List traceKeys() {\n\t\tArrayList retval = new ArrayList();\n\t\t// collect keys up to root\n\t\tTrieMap currentNode = this;\n\t\tTrieMap currentParent = parent;\n\t\twhile (currentParent != null) {\n\t\t\tif (currentParent.childMap instanceof Pair) {\n\t\t\t\tPair> pair = (Pair>) currentParent.childMap;\n\t\t\t\tretval.add(pair.a);\n\t\t\t} else {\n\t\t\t\tMap> map = (Map>) currentParent.childMap;\n\t\t\t\tfor (Map.Entry> entry : map.entrySet()) {\n\t\t\t\t\tif (entry.getValue() == currentNode) {\n\t\t\t\t\t\tretval.add(entry.getKey());\n\t\t\t\t\t\tbreak;\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\tcurrentNode = currentParent;\n\t\t\tcurrentParent = currentParent.parent;\n\t\t}\n\t\t// reverse and return\n\t\tCollections.reverse(retval);\n\t\treturn retval;\n\t}\n\n\t/** Returns this trie map as a string, with indenting. */\n\tpublic String toString() {\n\t\tStringBuffer sb = new StringBuffer();\n\t\ttoString(sb, \"\");\n\t\treturn sb.toString();\n\t}\n\n\t// appends this trie map as a string, with the given indenting level,\n\t// to the given string buffer\n\t@SuppressWarnings(\"unchecked\")\n\tprivate void toString(StringBuffer sb, String indent) {\n\t\tsb.append(\"node: \" + data);\n\t\tif (childMap == null)\n\t\t\treturn;\n\t\tindent += \" \";\n\t\tif (childMap instanceof Pair) {\n\t\t\tPair> pair = (Pair>) childMap;\n\t\t\ttoString(sb, indent, pair.a, pair.b);\n\t\t} else {\n\t\t\tMap> map = (Map>) childMap;\n\t\t\tList keys = new ArrayList(map.keySet());\n\t\t\tComparator toStringComparator = new Comparator() {\n\t\t\t\tpublic int compare(KeyType o1, KeyType o2) {\n\t\t\t\t\treturn o1.toString().compareTo(o2.toString());\n\t\t\t\t}\n\t\t\t};\n\t\t\tCollections.sort(keys, toStringComparator);\n\t\t\tfor (KeyType key : keys) {\n\t\t\t\ttoString(sb, indent, key, map.get(key));\n\t\t\t}\n\t\t}\n\t}\n\n\t// appends the given key and child\n\tprivate void toString(StringBuffer sb, String indent, Object key, TrieMap child) {\n\t\tsb.append(\"\\n\").append(indent).append('[').append(key).append(\"] \");\n\t\tchild.toString(sb, indent);\n\t}\n}\n" }, { "path": "src/opennlp/ccg/util/VisitedFilter.java", "content": "//////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2012 Scott Martin\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\nimport java.util.HashSet;\n\n/**\n * A filter that tracks which elements have already been visited by some process or iteration, allowing only\n * those that have not yet been visited. An element is considered visited when this filter's \n * {@link #allows(Object)} method has been called with it as an argument.\n *

    \n * Internally, the visited elements are tracked by using a\n * {@link MembershipFilter} based on a {@link HashSet}. \n * \n * @author Scott Martin\n */\npublic class VisitedFilter extends MembershipFilter {\n\n\t/**\n\t * Creates a new visited filter with an empty set of visited elements.\n\t */\n\tpublic VisitedFilter() {\n\t\tsuper(new HashSet());\n\t}\n\t\n\t/**\n\t * Tests whether this filter allows e by testing whether or not it has been visited. An element\n\t * has been visited if this method has been previously called with it as an argument.\n\t * \n\t * @param e The element to test whether it has been visited or not.\n\t * @return true if e has not yet been visited.\n\t * \n\t * @see #hasVisited(Object)\n\t */\n\t@Override\n\tpublic boolean allows(E e) {\n\t\tif(!hasVisited(e)) {\n\t\t\tmembers.add(e);\n\t\t\treturn true;\n\t\t}\n\t\t\n\t\treturn false;\n\t}\n\t\n\t/**\n\t * Tests whether the specified element has been visited or not.\n\t * @param e The element to test for visitation.\n\t * @return true if e is among the elements that have been previously visited.\n\t * \n\t * @see #allows(Object)\n\t * @see MembershipFilter#allows(Object)\n\t */\n\tpublic boolean hasVisited(E e) {\n\t\treturn super.allows(e);\n\t}\n}\n" }, { "path": "src/opennlp/ccg/util/Visualizer.java", "content": "///////////////////////////////////////////////////////////////////////////////\r\n// Copyright (C) 2004 Alexandros Triantafyllidis and \r\n// University of Edinburgh (Michael White)\r\n// \r\n// This library is free software; you can redistribute it and/or\r\n// modify it under the terms of the GNU Lesser General Public\r\n// License as published by the Free Software Foundation; either\r\n// version 2.1 of the License, or (at your option) any later version.\r\n// \r\n// This library is distributed in the hope that it will be useful,\r\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\r\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\r\n// GNU Lesser General Public License for more details.\r\n// \r\n// You should have received a copy of the GNU Lesser General Public\r\n// License along with this program; if not, write to the Free Software\r\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\r\n//////////////////////////////////////////////////////////////////////////////\r\npackage opennlp.ccg.util;\r\n\r\nimport opennlp.ccg.lexicon.*;\r\nimport opennlp.ccg.grammar.*;\r\nimport opennlp.ccg.synsem.*;\r\n\r\nimport java.io.*;\r\nimport java.util.*;\r\n\r\n/**\r\n * Class for visualizing CCG derivations using latex.\r\n *\r\n * @author Alexandros Triantafyllidis\r\n * @author Michael White\r\n * @version $Revision: 1.9 $, $Date: 2009/12/21 03:27:18 $\r\n */\r\npublic class Visualizer {\r\n\r\n private String ruleToTeX(String rule, int indent, int length) {\r\n StringBuffer sb = new StringBuffer();\r\n int i=0;\r\n for (i=0; i < indent; i++) sb.append(\"&\");\r\n sb.append(\" \\\\mc{\" + Integer.toString(length) + \"} \");\r\n if(rule == null || rule.length()==1)\r\n sb.append(\"{\\\\hrulefill_{\"+ rule + \"}} \\\\\\\\\\n\"); \r\n else\r\n sb.append(\"{\\\\hrulefill_{\"+ rule.substring(0,1) + \"\\\\mathbf{\" + rule.substring(1) + \"}}}\\\\\\\\\\n\");\r\n return sb.toString();\r\n }\r\n \r\n // Gets a file name for a temporary file, e.g.: ~tmp0\r\n public String getTempFileName() {\r\n File f1 = null;\r\n File f2 = null;\r\n int i=0;\r\n f1 = new File(\"~tmp\"+Integer.toString(i)+\".tex\");\r\n f2 = new File(\"~tmp\"+Integer.toString(i)+\".div\");\r\n while(f1.exists() || f2.exists() ) {\r\n i++;\r\n f1 = null; f2 = null;\r\n f1 = new File(\"~tmp\"+Integer.toString(i)+\".tex\");\r\n f2 = new File(\"~tmp\"+Integer.toString(i)+\".div\");\r\n }\r\n f1 = null; f2 = null;\r\n return \"~tmp\"+Integer.toString(i);\r\n }\r\n \r\n public String getTempDirName() {\r\n File f1 = null;\r\n int i=0;\r\n f1 = new File(\"tmp\");\r\n while(f1.exists() ) {\r\n i++;\r\n f1 = null; \r\n f1 = new File(\"tmp\"+Integer.toString(i) );\r\n }\r\n f1 = null; \r\n return \"tmp\"+Integer.toString(i);\r\n }\r\n \r\n public int getTreeDepth(Sign sign, int level)\r\n {\r\n int max_depth = 0, depth = 0;\r\n Sign[] children = sign.getDerivationHistory().getInputs();\r\n if (children != null && sign.getWords().size() > 1) \r\n for (int i=0; i < children.length; i++) {\r\n depth = getTreeDepth(children[i], level+1);\r\n if (depth > max_depth) max_depth = depth;\r\n }\r\n else\r\n max_depth = level;\r\n return max_depth;\r\n }\r\n\r\n private int numberOfLeaves(Sign results) {\r\n int totalLeaves = 0; \r\n Sign[] children = results.getDerivationHistory().getInputs();\r\n if (children==null || results.getWords().size()==1) return 1;\r\n for(int i=0;i processSign(Sign results, int level, int identation) {\r\n List signList = new ArrayList();\r\n Sign[] children = results.getDerivationHistory().getInputs();\r\n int depth = getTreeDepth(results, 0);\r\n TeXSign ts = new TeXSign();\r\n int offset=0;\r\n if (children != null && results.getWords().size() > 1)\r\n for (int i=0; i < children.length; i++) {\r\n if (i > 0)\r\n offset += numberOfLeaves(children[i-1]);\r\n signList.addAll( processSign(children[i], level + 1, identation + offset) );\r\n }\r\n ts.identation = identation;\r\n ts.height = depth;\r\n ts.sign = results;\r\n signList.add(ts);\r\n return signList;\r\n }\r\n\r\n public boolean writeFooter(String fileName){\r\n java.io.BufferedWriter bw = null;\r\n try{ \r\n bw = new java.io.BufferedWriter(new FileWriter(fileName,true) ); \r\n bw.write(\"\\\\end{document}\\n\"); \r\n bw.close();\r\n }\r\n catch(Exception e){ return false; }\r\n return true;\r\n }\r\n\r\n public boolean writeHeader(String fileName) {\r\n java.io.BufferedWriter bw = null;\r\n try {\r\n bw = new java.io.BufferedWriter(new FileWriter(fileName) ); \r\n bw.write(\"\\\\documentclass{article}\\n\");\r\n bw.write(\"\\\\usepackage[margin=0.5in]{geometry}\\n\");\r\n bw.write(\"\\\\newcommand{\\\\deriv}[2]\\n\");\r\n bw.write(\"{ \\\\renewcommand{\\\\arraystretch}{.5}\\n\");\r\n bw.write(\"$\\\\begin{array}[t]{*{#1}{c}}\\n\");\r\n bw.write(\" #2\\n\");\r\n bw.write(\" \\\\end{array}$ }\\n\");\r\n bw.write(\"\\\\newcommand{\\\\gf}[1]{\\\\textsf{\\\\textsl{#1}}}\\n\");\r\n bw.write(\"\\\\newcommand{\\\\cf}[1]{\\\\mbox{\\\\ensuremath{\\\\cfont{#1}}}}\\n\");\r\n bw.write(\"\\\\newcommand{\\\\uline}[1]\\n\");\r\n bw.write(\"{\\\\mc{#1}{\\\\hrulefill} }\\n\");\r\n bw.write(\"\\\\newcommand{\\\\mc}[2]\\n\");\r\n bw.write(\" {\\\\multicolumn{#1}{c}{#2}}\\n\");\r\n bw.write(\"\\\\newcommand{\\\\cfont}{\\\\mathsf}\\n\");\r\n bw.write(\"\\\\newcommand{\\\\bs}{\\\\backslash}\\n\");\r\n bw.write(\"\\\\newcommand{\\\\subsa}[1]{\\\\hspace{-0.75mm}_{_{#1}}}\\n\");\r\n bw.write(\"\\\\newcommand{\\\\subsb}[1]{\\\\hspace{-0.10mm}_{_{#1}}}\\n\");\r\n bw.write(\"\\\\newcommand{\\\\subs}[1]{\\\\hspace{-0.40mm}_{#1}}\\n\");\r\n bw.write(\"\\\\newcommand{\\\\subsf}[1]{\\\\hspace{-0.75mm}_{_{#1}}}\\n\");\r\n bw.write(\"\\\\newcommand{\\\\supsa}[1]{\\\\hspace{-1.75mm}^{^{#1}} }\\n\");\r\n bw.write(\"\\\\newcommand{\\\\supsb}[1]{\\\\hspace{-0.80mm}^{^{#1}} }\\n\");\r\n bw.write(\"\\\\newcommand{\\\\sups}[1]{\\\\hspace{-0.40mm}^{#1}}\\n\");\r\n\t bw.write(\"\\\\pagestyle{empty}\\n\");\r\n bw.write(\"\\\\begin{document}\\n\");\r\n bw.close();\r\n }\r\n catch(Exception e){ return false; }\r\n return true;\r\n }\r\n\r\n /** \r\n * Shows the current derivation using YaP or xdvi.\r\n */\r\n public boolean show(String fileName) {\r\n String viewerName = null;\r\n try {\r\n \trunCommand(\"latex \" + fileName + \".tex\");\r\n //Process p =\r\n \tjava.lang.Runtime.getRuntime().exec(\"latex \" + fileName + \".tex\");\r\n if (System.getProperty(\"os.name\").toUpperCase().startsWith(\"WINDOWS\"))\r\n viewerName = \"yap\";\r\n else\r\n viewerName = \"xdvi\";\r\n\r\n System.out.println(\"Close \" + viewerName + \" to continue ...\");\r\n runCommand(viewerName + \" \" + fileName);\r\n // The process will wait indefinitely unless we close each of the related streams:/\r\n //p.getInputStream().close();\r\n //p.getOutputStream().close();\r\n //p.getErrorStream().close();\r\n //p.waitFor();\r\n //p = null;\r\n //p = java.lang.Runtime.getRuntime().exec(viewerName + \" \" + fileName);\r\n //p.getInputStream().close();\r\n //p.getOutputStream().close();\r\n //p.getErrorStream().close();\r\n //System.out.println(\"Close \" + viewerName + \" to continue ...\");\r\n //p.waitFor();\r\n } catch(Exception e) {\r\n System.out.println(\"Error invoking latex/\" + viewerName + \" : \" + e.toString());\r\n return false;\r\n }\r\n return true;\r\n }\r\n \r\n class myFilter implements FileFilter {\r\n String baseFileName=null;\r\n public myFilter(String s) {\r\n baseFileName = s.toUpperCase();\r\n }\r\n public boolean accept(File f) {\r\n System.out.println(\"checking: \" + f.getName());\r\n if(f.getName().toUpperCase().startsWith(baseFileName))\r\n return true;\r\n else\r\n return false;\r\n }\r\n }\r\n \r\n public class myFileNameFilter implements FilenameFilter {\r\n public String fn=null;\r\n public myFileNameFilter(String s) {\r\n fn=s;\r\n }\r\n public boolean accept(File dir, String name) { \r\n return name.startsWith(fn);\r\n }\r\n }\r\n \r\n public boolean cleanFiles(String fileName) {\r\n try {\r\n File dir = new File(System.getProperty(\"user.dir\")); \r\n myFileNameFilter filter = new myFileNameFilter( fileName) ;\r\n File[] allFiles = dir.listFiles(filter);\r\n for(int i=0;i signList = null;\r\n java.io.BufferedWriter bw = null;\r\n try {\r\n int i=0, numDerivs=0;\r\n TeXSign texSign = null;\r\n bw = new java.io.BufferedWriter(new FileWriter( fileName,true) ); \r\n signList = sortList( processSign(results,0, 0 ) );\r\n numDerivs = results.getWords().size();\r\n Tokenizer tokenizer = Grammar.theGrammar.lexicon.tokenizer;\r\n bw.write(\"\\\\deriv{\" + Integer.toString(numDerivs) + \"}{\\n\");\r\n for (i=0; i < results.getWords().size(); i++) {\r\n if (i != 0) bw.write(\" & \");\r\n String orth = tokenizer.getOrthography((Word)results.getWords().get(i), false);\r\n orth = orth.replaceAll(\"_\", \"\\\\\\\\_\");\r\n orth = orth.replaceAll(\"%\", \"\\\\\\\\%\");\r\n bw.write(\"\\\\gf{\" + orth + \"}\");\r\n }\r\n bw.write(\" \\\\\\\\\\n\\\\uline{1}\");\r\n for (i=1; i < results.getWords().size(); i++)\r\n bw.write(\" & \\\\uline{1}\");\r\n bw.write(\" \\\\\\\\\\n\");\r\n texSign = (TeXSign)signList.get(0);\r\n bw.write(\"\\\\cf{\"+ texSign.sign.getCategory().toTeX() + \"}\");\r\n for (i=1; i < numDerivs; i++) {\r\n texSign = (TeXSign)signList.get(i);\r\n bw.write(\" & \\\\cf{\"+ texSign.sign.getCategory().toTeX() + \"}\");\r\n }\r\n bw.write(\" \\\\\\\\\\n\"); \r\n\r\n for (i=numDerivs; i < signList.size(); i++) {\r\n String ruleStr=null;\r\n texSign = (TeXSign)signList.get(i);\r\n ruleStr = ruleToTeX(texSign.sign.getDerivationHistory().getRule().name(), texSign.identation, texSign.sign.getWords().size() );\r\n bw.write(ruleStr);\r\n for (int j=0; j < texSign.identation; j++)\r\n bw.write(\"&\");\r\n bw.write(\" \\\\mc{\" + texSign.sign.getWords().size() + \"}{\\\\cf{\"+ texSign.sign.getCategory().toTeX() +\"}} \\\\\\\\\\n\");\r\n }\r\n\t // Originally 1in, but that's too much when displayed onscreen\r\n bw.write(\"}\\n\\n\\\\vspace{5mm}\\n\\n\");\r\n bw.close();\r\n } catch(Exception e) {\r\n System.out.println(\"Error while saving to TeX: \" + e.toString()); \r\n e.printStackTrace();\r\n return false; \r\n }\r\n \r\n return true;\r\n }\r\n\r\n private List sortList(List signList) {\r\n for (int i=0; i < signList.size(); i++)\r\n for(int j=i; j < signList.size(); j++) {\r\n TeXSign texSign1 = signList.get(i);\r\n TeXSign texSign2 = signList.get(j);\r\n if(texSign1.height > texSign2.height) { \r\n signList.set(i,texSign2); signList.set(j, texSign1); }\r\n if(texSign1.height == texSign2.height)\r\n if(texSign1.identation > texSign2.identation) { \r\n signList.set(i,texSign2); signList.set(j, texSign1); }\r\n }\r\n return signList;\r\n }\r\n\r\n private class TeXSign {\r\n Sign sign = null;\r\n int identation = 0;\r\n int height = 0;\r\n }\r\n\r\n /**\r\n * Calls runCommand/2 assuming that wait=true.\r\n *\r\n * @param cmd The string containing the command to execute\r\n */\r\n public static void runCommand (String cmd) {\r\n\trunCommand(cmd, true);\r\n }\r\n\r\n /**\r\n * Run a command with the option of waiting for it to finish.\r\n *\r\n * @param cmd The string containing the command to execute\r\n * @param wait True if the caller should wait for this thread to \r\n * finish before continuing, false otherwise.\r\n */\r\n public static void runCommand (String cmd, boolean wait) {\r\n\ttry {\r\n //System.out.println(\"Running command: \"+ cmd);\r\n\t Process proc = Runtime.getRuntime().exec(cmd);\r\n\r\n\t // This needs to be done, otherwise some processes fill up\r\n\t // some Java buffer and make it so the spawned process\r\n\t // doesn't complete.\r\n BufferedReader br = \r\n\t\tnew BufferedReader(new InputStreamReader(proc.getInputStream()));\r\n //String line = null;\r\n //while ( (line = br.readLine()) != null) {\r\n while ( (br.readLine()) != null) {\r\n\t\t; // just eat up the inputstream\r\n\r\n\t\t// Use this if you want to see the output from running\r\n\t\t// the command.\r\n\t\t//System.out.println(line);\r\n\t }\r\n\r\n\t if (wait) {\r\n\t\ttry {\r\n\t\t proc.waitFor();\r\n\t\t} catch (InterruptedException e) {\r\n\t\t Thread.currentThread().interrupt();\r\n\t\t}\r\n\t }\r\n\t proc.getInputStream().close();\r\n\t proc.getOutputStream().close();\r\n\t proc.getErrorStream().close();\r\n\t} catch (IOException e) {\r\n\t System.out.println(\"Unable to run command: \"+cmd);\r\n\t}\r\n }\r\n\r\n}\r\n" }, { "path": "src/opennlp/ccg/util/XmlScanner.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2006 Michael White (The Ohio State University)\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccg.util;\n\n//JAXP packages\nimport javax.xml.parsers.*;\nimport org.xml.sax.*;\nimport org.xml.sax.helpers.*;\n\n// jdom\nimport org.jdom.*;\n\nimport java.io.*;\nimport java.net.*;\nimport java.util.*;\n\n/**\n * Quick-and-dirty class for using JDOM elements in parsing XML \n * without building a document for the whole XML file.\n * The parse method parses an XML file with a SAX \n * parser, constructing JDOM elements for root and the top-level elements,\n * without attaching the top-level elements to the root.\n * Along the way, the handleRoot and handleElement \n * methods are invoked, to handle these elements incrementally.\n * To use the class, just implement handleElement, and optionally \n * implement handleRoot, and call parse on the \n * input XML via its URL. Note that with Java's incremental garbage \n * collection, an XmlScanner should be reasonably efficient, but not \n * as efficient as a pure SAX parser (which however requires considerably \n * more work to implement). At present, the parser only handles elements \n * and attributes without namespaces; all attributes with namespaces, text nodes, \n * comments, etc. are ignored. \n *\n * @author Michael White\n * @version $Revision: 1.3 $, $Date: 2009/12/21 03:27:18 $\n */\npublic abstract class XmlScanner extends DefaultHandler {\n \n\t/** \n\t * Method for handling the root element, sans children. \n\t * The default method does nothing. \n\t */\n public void handleRoot(Element e) {}\n \n\t/** Method for handling top-level elements. */\n public abstract void handleElement(Element e);\n \n /** \n * Method for parsing an XML document, handling the childless root element \n * and the top-level elements along the way. \n */\n public void parse(URL url) throws IOException {\n \ttry {\n\t // Create a JAXP SAXParserFactory and configure it\n\t SAXParserFactory spf = SAXParserFactory.newInstance();\n\t\n\t // Create a JAXP SAXParser\n\t SAXParser saxParser = spf.newSAXParser();\n\t\n\t // Get the encapsulated SAX XMLReader\n\t XMLReader xmlReader = saxParser.getXMLReader();\n\t\n\t // Set the ContentHandler of the XMLReader\n\t xmlReader.setContentHandler(this);\n\t\n\t // Tell the XMLReader to parse the XML document\n\t xmlReader.parse(url.toString());\n \t}\n \tcatch (ParserConfigurationException exc) {\n \t\tthrow (IOException) new IOException().initCause(exc);\n \t}\n \tcatch (SAXException exc) {\n \t\tthrow (IOException) new IOException().initCause(exc);\n \t}\n }\n\n \n // flag for whether the root element has been seen yet\n private boolean seenRoot = false;\n // the element that is currently being processed\n private Element current = null;\n\n \n //\n // ContentHandler methods (just elements)\n //\n\n /** \n * For the root, a childless element is created and handled, via handleRoot; \n * for all other elements, a new current element is created and added as a child of the \n * current element, if any. \n */\n public void startElement(String uri, String localName, String qname, Attributes attributes) throws SAXException {\n \n if (!seenRoot) { \n \tseenRoot = true;\n \tElement root = createElement(uri, localName, qname, attributes);\n \thandleRoot(root);\n \treturn; \n \t}\n \n Element parent = current;\n current = createElement(uri, localName, qname, attributes);\n if (parent != null) parent.addContent(current);\n }\n\n /** \n * Constructs and returns a new element from the given info.\n * This implementation ignores the uri and localName, \n * and filters out any attributes whose qname contains a colon.\n */\n protected Element createElement(String uri, String localName, String qname, Attributes attributes) {\n Element retval = new Element(qname);\n if (attributes != null) {\n int length = attributes.getLength();\n for (int i = 0; i < length; i++) {\n String attrQName = attributes.getQName(i);\n if (attrQName.indexOf(':') >= 0) continue;\n String attrValue = attributes.getValue(i);\n retval.setAttribute(attrQName, attrValue);\n }\n }\n return retval;\n }\n \n /** \n * Resets the current element to its parent, after first invoking \n * handleElement on the element if it's a top-level one. \n */\n public void endElement(String uri, String localName, String qname) throws SAXException {\n if (current == null) return; // for root\n Element parent = (Element) current.getParent();\n if (parent == null) handleElement(current);\n current = parent;\n } \n\n \n\n /**\n * Example scanner: prints root, counts top- and second-level elements.\n */\n public static class MyScanner extends XmlScanner {\n int count = 0; int second = 0;\n\t\tpublic void handleRoot(Element e) {\n \tSystem.out.println(\"root name: \" + e.getName());\n \tSystem.out.print(\"root attributes: \");\n @SuppressWarnings(\"unchecked\")\n \tList attrs = (List) e.getAttributes();\n \tfor (Attribute attr : attrs) {\n \t\tSystem.out.print(attr.getName() + \"=\" + attr.getValue() + \" \");\n \t}\n \tSystem.out.println();\n }\n public void handleElement(Element e) { \n count++; second += e.getContentSize();\n }\n }\n \n /**\n * The main method shows an example of using a scanner, by \n * invoking an instance of MyScanner on the file \n * whose name is given by the first arg.\n */\n public static void main(String[] args) throws IOException {\n String filename = args[0];\n MyScanner myScanner = new MyScanner();\n myScanner.parse(new File(filename).toURI().toURL());\n System.out.println(\"top-level elements: \" + myScanner.count);\n System.out.println(\"second-level elements: \" + myScanner.second);\n }\n}\n" }, { "path": "src/opennlp/ccgbank/CCGBankConvert.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n/*\n * $Id: CCGBankConvert.java,v 1.8 2011/11/10 22:18:42 mwhite14850 Exp $ \n */\npackage opennlp.ccgbank;\n\nimport java.io.BufferedReader;\nimport java.io.File;\nimport java.io.FileReader;\nimport java.io.IOException;\nimport java.io.Reader;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.StringTokenizer;\n\nimport javax.xml.transform.Source;\nimport javax.xml.transform.sax.SAXSource;\n\nimport opennlp.ccgbank.convert.InfoHelper;\nimport opennlp.ccgbank.convert.XSLTTrueCaser;\nimport opennlp.ccgbank.convert.MorphLookup;\nimport opennlp.ccgbank.parse.CCGbankDerivation;\nimport opennlp.ccgbank.parse.SimpleNode;\n\nimport org.apache.tools.ant.BuildException;\nimport org.apache.tools.ant.DefaultLogger;\nimport org.apache.tools.ant.Project;\nimport org.apache.tools.ant.ProjectHelper;\nimport org.jdom.Document;\nimport org.jdom.Element;\nimport org.jdom.transform.JDOMSource;\nimport org.xml.sax.InputSource;\n\n\n/**\n * Converts the CCGBank to a modified version for grammar extraction.\n *

    \n * Within this task, a series of FileLists is specified. These\n * files are the lists of xsltProcessors that should be used to transform the \n * CCGBank. These xsltProcessors are processed in the order they occur in the\n * FileList specified within this task. \n * @author Scott Martin\n * @author Rajakrishnan Rajkumar\n * @version $Revision: 1.8 $\n * @see CCGBankExtract\n */\npublic class CCGBankConvert extends CCGBankTask {\n\t\n\t/** Flag for whether to keep case-marking preps in PP categories; defaults to false. */\n\tboolean keepPPHeads = false;\n\n\tTreeWalker treeWalker = new TreeWalker();\n\t\n\tCCGbankDerivation deriv = null;\n\tFile auxFileDirectory, bbnAuxDirectory, wordsFile, stemsFile,\n\t\tcurrentDirectory = null;\n\t\n\t/* (non-Javadoc)\n\t * @see opennlp.ccgbank.CCGBankTask#start()\n\t */\n\t@Override\n\tprotected void start() throws BuildException {\n\t\t\n\t\tInfoHelper.init(auxFileDirectory, bbnAuxDirectory);\n\t\tString trueCaseListPath=auxFileDirectory.getAbsolutePath()+\"/\"+\"truecase-list.gz\";\n\t\tXSLTTrueCaser.init(trueCaseListPath);\n\t\t\n\t\ttry {\n\t\t\tMorphLookup.init(wordsFile, stemsFile);\n\t\t\t\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tthrow new BuildException(\"problem loading words or stems\", io,\n\t\t\t\t\tgetLocation());\n\t\t}\n\t\t\n\t\txsltProcessor = useXMLFilter\n\t\t\t? new XMLFilterProcessor(this, this)\n\t\t\t: new TemplatesProcessor(this);\n\t\t\n\t\t// \"prime\" parser\n\t\t// TODO fix this hack!!\n\t\ttry {\n\t\t\tFile tmp = File.createTempFile(getClass().getName(), \"prime\");\n\t\t\ttmp.deleteOnExit();\n\t\t\t\t\t\t\n\t\t\tderiv = new CCGbankDerivation(new FileReader(tmp));\n\t\t}\n\t\tcatch(IOException e) {\n\t\t\tthrow new BuildException(\"Problem priming parser: \"\n\t\t\t\t+ e.getMessage(), e, getLocation());\n\t\t}\n\t}\n\n\t/** Read aux files for the next WSJ section **/\n\t@Override\n\tprotected void nextDirectory(File section) throws BuildException {\n\t\t\n\t\tcurrentDirectory = section;\n\t\t\n\t\t// only create if a numbered directory\n\t\tFile d = new File(target, currentDirectory.getName());\n\t\tif(!d.exists() && !d.mkdirs()) {\n\t\t\tthrow new BuildException(\"unable to create directory \" + d);\n\t\t}\n\t\t\n\t\t//Read in aux files\n\t\ttry { \n\t\t\tInfoHelper.readBBNAuxfiles(section.getName());\n\t\t\tInfoHelper.readQuoteAuxfiles(section.getName());\n\t\t\tInfoHelper.readPTBAuxfiles(section.getName());\n\t\t\tInfoHelper.readTreeAuxfiles(section.getName());\n\t\t}\n\t\tcatch(NumberFormatException nfe) {\n\t\t\t// not a numbered PTB directory\n\t\t}\n\t}\n\n\t@Override\n\tprotected InputSource nextFile(File file) throws BuildException {\n\t\ttry {\n\t\t\tReader reader = new BufferedReader(new FileReader(file));\n\t\t\t\n\t\t\tif(deriv == null) { \n\t\t\t\tderiv = new CCGbankDerivation(reader);\n\t\t\t}\n\t\t\telse {\n\t\t\t\tCCGbankDerivation.ReInit(reader);\n\t\t\t}\n\t\t\t\n\t\t\tSimpleNode root = CCGbankDerivation.start();\n\t\t\tElement result = new Element(\"Derivation\");\n\t\t\t\n\t\t\tString fileName = file.getName();\n\t\t\tint start = fileName.contains(File.separator)\n\t\t\t\t? fileName.lastIndexOf(File.separatorChar) : 0;\n\t\t\t\n\t\t\tStringBuilder sb = new StringBuilder(\n\t\t\t \tfileName.substring(start, fileName.lastIndexOf('.')));\n\t\t sb.append(\".xml\");\n\t\t \n\t\t File targetDir = new File(target, currentDirectory.getName());\n\t\t File targetFile = new File(targetDir, sb.toString());\t\t\t \n\t\t \n\t\t xsltProcessor.resetSerializer();\n\t\t xsltProcessor.setTarget(targetFile);\n\t\t \t\t \n\t\t Document doc = new Document(treeWalker.eval(root, result));\n\t\t \n\t\t // TODO attempt to get error reporting for file / line !!\n\t\t Source s = new JDOMSource(doc);\n\t\t s.setSystemId(file.toURI().toString());\n\t\t \n\t\t return SAXSource.sourceToInputSource(s);\n\t\t}\n\t\tcatch(Exception e) {\n\t\t\tthrow new BuildException(e, getLocation());\n\t\t}\n\t}\n\t\n\t\n\t/** @param keepPPHeads the keepPPHeads value to set */\n\tpublic void setKeepPPHeads(boolean keepPPHeads) { this.keepPPHeads = keepPPHeads; }\n\t\n\t\n\t/**\n\t * @param stemsFile the stemsFile to set\n\t */\n\tpublic void setStemsFile(File stemsFile) {\n\t\tthis.stemsFile = stemsFile;\n\t}\n\n\t\n\t/**\n\t * @param wordsFile the wordsFile to set\n\t */\n\tpublic void setWordsFile(File wordsFile) {\n\t\tthis.wordsFile = wordsFile;\n\t}\n\n\t/**\n\t * @param auxFileDirectory the auxFileDirectory to set\n\t */\n\tpublic void setAuxFileDirectory(File auxFileDirectory) {\n\t\tthis.auxFileDirectory = auxFileDirectory;\n\t\t\n\t}\n\t\n\t/**\n\t * @param bbnAuxDirectory the bbnAuxDirectory to set\n\t */\n\tpublic void setBbnAuxDirectory(File bbnAuxDirectory) {\n\t\tthis.bbnAuxDirectory = bbnAuxDirectory;\n\t}\n\n\tpublic static void main(String[] args) {\n\t\tFile baseDir = new File(System.getProperty(\"user.dir\"));\n\t\tFile buildFile = new File(baseDir, \"build.xml\");\n\t\t\n\t\tProject project = new Project();\n\t\t\n\t\tproject.init();\n\t\t\n\t\tproject.setBaseDir(baseDir);\n\t\t\n\t\tProjectHelper helper = ProjectHelper.getProjectHelper();\n\t\t\n\t\tproject.setProjectReference(helper);\n\t\t\n\t\thelper.parse(project, buildFile);\n\t\t\n\t\tDefaultLogger logger = new DefaultLogger();\n\t\tlogger.setErrorPrintStream(System.err);\n\t\tlogger.setOutputPrintStream(System.out);\n\t\t\n\t\tproject.addBuildListener(logger);\n\t\t\n\t\tproject.executeTarget(\"convert-base\");\t\t\n\t}\n\t\n\tclass TreeWalker {\n\t\t// General purpose datastructure to store ccgbank indices of categories.\n\t\t// Refreshed after the lifespan of a node is over.\n\t\tpublic List idList = new ArrayList();\n\t\t// flag for whether under a leaf node;\n\t\t// used to control whether to add fs id's\n\t\tprivate boolean underLeaf = false;\n\t\t\n\t\tpublic Element eval(SimpleNode node, Element root) throws Exception {\n\n\t\t\t// No:of children of any given node\n\t\t\tint numC = node.jjtGetNumChildren();\n\n\t\t\t// Loop & flag variables\n\t\t\tint i = 0;\n\t\t\tSimpleNode child;\n\n\t\t\t// Processing the child nodes of the current node.\n\t\t\tfor (i = 0; i < numC; i++) {\n\n\t\t\t\tchild = (SimpleNode) node.jjtGetChild(i);\n\n\t\t\t\t// Cat spec without co-indexation info in the leafnodes\n\t\t\t\tif (child.type.equals(\"Redundant\")) {\n\n\t\t\t\t\tif (node.type.equals(\"Leafnode\")) {\n\t\t\t\t\t\tnode.catRedundant = child.print();\n\t\t\t\t\t\tcontinue;\n\t\t\t\t\t}\n\n\t\t\t\t\t// Processes treenode categories\n\t\t\t\t\tif (node.type.equals(\"Treenode\"))\n\t\t\t\t\t\tchild.type = \"complexcat\";\n\t\t\t\t}\n\n\t\t\t\t// The header node is accessed and the CCGbankId is passed on to the\n\t\t\t\t// treenode root of the sentence which is processed next\n\t\t\t\tif (child.type.equals(\"Header\")) {\n\n\t\t\t\t\ti++;\n\t\t\t\t\tString temp1 = child.getHeader();\n\t\t\t\t\tint spacePos = temp1.indexOf(' ');\n\t\t\t\t\tif (spacePos > 0)\n\t\t\t\t\t\ttemp1 = temp1.substring(0, spacePos);\n\n\t\t\t\t\tchild = (SimpleNode) node.jjtGetChild(i);\n\t\t\t\t\tchild.setHeader(temp1);\n\t\t\t\t\t// System.out.println(temp1);\n\t\t\t\t}\n\n\t\t\t\t// Xml element which is going to be generated.\n\t\t\t\tElement leaf = new Element(child.type);\n\n\t\t\t\tif (child.type.equals(\"complexcat\")\n\t\t\t\t\t\t|| child.type.equals(\"Treenode\")\n\t\t\t\t\t\t|| child.type.equals(\"Leafnode\")) {\n\n\t\t\t\t\t// Atomic categories are represented in the javacc tree as\n\t\t\t\t\t// catSpec-aotmcat. So for such cases the catSpec child is\n\t\t\t\t\t// skipped and the next child is accessed.\n\n\t\t\t\t\tif (child.jjtGetNumChildren() == 1\n\t\t\t\t\t\t\t&& child.type.equals(\"complexcat\")) {\n\n\t\t\t\t\t\tchild = (SimpleNode) child.jjtGetChild(0);\n\n\t\t\t\t\t\t// The element which is to be added to the xml\n\t\t\t\t\t\t// representation\n\t\t\t\t\t\tleaf = new Element(\"atomcat\");\n\n\t\t\t\t\t\t// Extracting the content of the node and storing it.\n\t\t\t\t\t\tnode.cat = child.print();\n\t\t\t\t\t} else {\n\n\t\t\t\t\t\t// A complexcat element is created.\n\t\t\t\t\t\tleaf = ccinserter(child, leaf);\n\n\t\t\t\t\t\t// Extracting the content of the node and storing it.\n\t\t\t\t\t\tif (root.getName().equals(\"Leafnode\")) {\n\t\t\t\t\t\t\tnode.cat = child.print();\n\t\t\t\t\t\t}\n\n\t\t\t\t\t\tif (root.getName().equals(\"Treenode\")\n\t\t\t\t\t\t\t\t&& leaf.getName().equals(\"complexcat\")) {\n\t\t\t\t\t\t\t// Leafnode and treenode cat spec elements created\n\t\t\t\t\t\t\tnode.cat = child.print();\n\t\t\t\t\t\t}\n\n\t\t\t\t\t\t// Recursive processing of the children of the current node\n\t\t\t\t\t\tleaf = eval(child, leaf);\n\n\t\t\t\t\t\t// Sending Leaf,Tree nodes for to a function which inserts\n\t\t\t\t\t\t// the family (ie normalized cat spec) of its contents.\n\t\t\t\t\t\tif (!child.type.equals(\"complexcat\"))\n\t\t\t\t\t\t\tleaf = catInserter(child, leaf);\n\n\t\t\t\t\t\t// Adding the current element to its parent in the xml tree.\n\t\t\t\t\t\troot.addContent(leaf);\n\n\t\t\t\t\t\tcontinue;\n\t\t\t\t\t}\n\n\t\t\t\t}\n\n\t\t\t\t// Slash elements added to the tree\n\t\t\t\tif (child.cat.equals(\"/\") || child.cat.equals(\"\\\\\")) {\n\t\t\t\t\tleaf = opinserter(child);\n\t\t\t\t\troot.addContent(leaf);\n\t\t\t\t\tcontinue;\n\t\t\t\t}\n\n\t\t\t\t// Atomcat elements added to the tree\n\t\t\t\tleaf = atomcatinserter(child);\n\t\t\t\troot.addContent(leaf);\n\n\t\t\t\t// if(!child.type.matches(\"\\\\p{Punct}\"))\n\t\t\t}\n\n\t\t\t// The final result of the above operations returned.\n\t\t\treturn root;\n\n\t\t}\n\t\t\n\t\tpublic Element ccinserter(SimpleNode node, Element leaf) {\n\n\t\t\t// This function produces complexcat/treenode/leafnode elements.\n\t\t\t// mww: the name of this function is not very helpful\n\n\t\t\t// The node can be of any of the above types.\n\t\t\tString name;\n\t\t\tname = node.type;\n\n\t\t\t// Set treebankId,parseNo at root of the sent\n\t\t\tString h = node.getHeader();\n\t\t\tif (h != null)\n\t\t\t\tleaf.setAttribute(\"Header\", h);\n\n\t\t\t// Treenode info ie head,daughter represented\n\t\t\tif (name.equals(\"Treenode\")) {\n\n\t\t\t\tleaf.setAttribute(\"head\", node.head);\n\t\t\t\tleaf.setAttribute(\"dtr\", node.dtr);\n\n\t\t\t\tidList.clear();\n\t\t\t\tunderLeaf = false;\n\t\t\t}\n\n\t\t\t// Leafnode info represented\n\t\t\tif (name.equals(\"Leafnode\")) {\n\t\t\t\tleaf.setAttribute(\"lexeme\", node.lex); // nb: may be truecased later\n\t\t\t\tleaf.setAttribute(\"lexeme0\", node.lex);\n\t\t\t\tleaf.setAttribute(\"pos\", node.pos);\n\t\t\t\tidList.clear();\n\t\t\t\tunderLeaf = true;\n\t\t\t}\n\n\t\t\t// add propbank info here\n\t\t\tif (node.nodeRoles != null) {\n\t\t\t\tString roles = \"\";\n\t\t\t\tfor (SimpleNode.LexSenseRole lexSenseRole : node.nodeRoles) {\n\t\t\t\t\tif (lexSenseRole.role.equals(\"rel\")) {\n\t\t\t\t\t\tleaf.setAttribute(\"rel\", lexSenseRole.lex + \".\" + lexSenseRole.sense);\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tString role = adjustRole(lexSenseRole.role);\n\t\t\t\t\t\troles += lexSenseRole.lex + \".\" + lexSenseRole.sense + \":\" + role + \" \";\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tif (roles.length() > 0) leaf.setAttribute(\"roles\", roles.trim());\n\t\t\t}\n\t\t\tif (node.argRoles != null) {\n\t\t\t\tString args = \"\";\n\t\t\t\tfor (String role : node.argRoles) {\n\t\t\t\t\trole = adjustRole(role);\n\t\t\t\t\targs += role + \" \";\n\t\t\t\t}\n\t\t\t\tleaf.setAttribute(\"argRoles\", args.trim());\n\t\t\t}\n\t\t\t// done\n\t\t\treturn leaf;\n\t\t}\n\n\t\tpublic Element atomcatinserter(SimpleNode node) throws Exception {\n\n\t\t\t// Predicate for atomcat creation.\n\t\t\t// Flag signifies whether the elem is a single atomcat.\n\t\t\t// Relevant as if the present cat is an atomcat LF variable can be set\n\t\t\t// in the syntax here itself.\n\n\t\t\tStringTokenizer lex;\n\t\t\tElement atomcat = new Element(\"atomcat\");\n\t\t\tElement fs = new Element(\"fs\");\n\t\t\tElement feat = new Element(\"feat\");\n\t\t\tString id = \"NotGiven\";\n\t\t\tString form;\n\n\t\t\t// Current element info extracted from the node\n\t\t\tString elem = node.print();\n\n\t\t\t// Index extracted by sensing undescore delimiter\n\t\t\tif (elem.contains(\"_\")) {\n\t\t\t\t// elem=elem.replaceAll(\":[A-Z]\",\"\");\n\t\t\t\tlex = new StringTokenizer(elem, \"_\");\n\t\t\t\telem = lex.nextToken();\n\t\t\t\tid = lex.nextToken();\n\t\t\t\tString x[] = id.split(\":\");\n\t\t\t\t// System.out.println(id);\n\n\t\t\t\tif (x.length == 2) {\n\t\t\t\t\t// mww: moved this to atomcat\n\t\t\t\t\t//feat.setAttribute(\"attr\", \"dep\");\n\t\t\t\t\t//feat.setAttribute(\"val\", x[1]);\n\t\t\t\t\t//fs.addContent(feat);\n\t\t\t\t\tatomcat.setAttribute(\"dep\", x[1]);\n\t\t\t\t}\n\n\t\t\t\tid = id.replaceAll(\":[A-Z]\", \"\");\n\t\t\t} else {\n\t\t\t\t// Default id 1 is alloted otherwise\n\t\t\t\tid = Integer.toString(idList.size() + 1);\n\t\t\t\tidList.add(id);\n\t\t\t}\n\n\t\t\t// Normalizing the id by comparing with previous indices.\n\t\t\tif (!idList.contains(id))\n\t\t\t\tidList.add(id);\n\t\t\tid = Integer.toString(idList.indexOf(id) + 1);\n\n\t\t\t// Normalized index is set (leaves only)\n\t\t\tif (underLeaf)\n\t\t\t\tfs.setAttribute(\"id\", id);\n\n\t\t\t// make lowercase\n\t\t\telem = elem.toLowerCase();\n\t\t\t// remove superfluous [nb] in np[nb]\n\t\t\telem = elem.replaceAll(\"np\\\\[nb\\\\]\", \"np\");\n\t\t\t// also strip PP heads if apropos\n\t\t\telem = stripPPHeads(elem);\n\n\t\t\t// Form attrtibute detected and set.\n\t\t\tif (elem.contains(\"[\")) {\n\t\t\t\tlex = new StringTokenizer(elem, \"[,]\");\n\t\t\t\telem = lex.nextToken();\n\t\t\t\tform = lex.nextToken();\n\t\t\t\tfeat = new Element(\"feat\");\n\t\t\t\tfeat.setAttribute(\"attr\", \"form\");\n\t\t\t\tfeat.setAttribute(\"val\", form);\n\t\t\t\tfs.addContent(feat);\n\t\t\t}\n\n\t\t\t// add fs if non-empty\n\t\t\tif (underLeaf || fs.getContentSize() > 0)\n\t\t\t\tatomcat.addContent(fs);\n\t\t\tatomcat.setAttribute(\"type\", elem);\n\t\t\treturn atomcat;\n\n\t\t}\n\n\t\tpublic Element opinserter(SimpleNode node) {\n\n\t\t\tElement slash = new Element(\"slash\");\n\t\t\tString dir;\n\t\t\tString op = node.cat;\n\n\t\t\t// Slash direction sensed and set.\n\n\t\t\tif (op.equals(\"\\\\\"))\n\t\t\t\tdir = \"<\";\n\t\t\telse\n\t\t\t\tdir = \">\";\n\n\t\t\tslash.setAttribute(\"dir\", op);\n\t\t\tslash.setAttribute(\"mode\", dir);\n\t\t\treturn slash;\n\n\t\t}\n\n\t\tpublic Element catInserter(SimpleNode node, Element leaf) {\n\t\t\tint i;\n\n\t\t\t// The normalization process. Relevant indices replaced by 1,2,3..n\n\n\t\t\tif (idList.size() > 0) {\n\t\t\t\tfor (i = 0; i < idList.size(); i++)\n\t\t\t\t\tnode.cat = node.cat.replaceAll(idList.get(i),\n\t\t\t\t\t\t\tInteger.toString(i + 1));\n\t\t\t}\n\n\t\t\tString l = node.getLeftover();\n\t\t\tif (l != null)\n\t\t\t\tnode.cat = node.cat + l;\n\n\t\t\t// Purging the cat spec of indices outside brackets & colons ie )_2 ,:B\n\t\t\tint ind = node.cat.indexOf(\")_\");\n\n\t\t\twhile (ind != -1) {\n\n\t\t\t\tString str1 = node.cat.substring(0, ind + 1);\n\t\t\t\tString str2 = node.cat.substring(ind + 1, node.cat.length());\n\n\t\t\t\tstr2 = str2.replaceFirst(\"_(\\\\p{Digit})++\", \"\");\n\n\t\t\t\t// System.out.println(str1);\n\t\t\t\t// System.out.println(str2);\n\n\t\t\t\tnode.cat = str1 + str2;\n\n\t\t\t\tind = node.cat.indexOf(\")_\");\n\n\t\t\t}\n\n\t\t\tnode.cat = node.cat.replaceAll(\":[A-Z]\", \"\");\n\n\t\t\t// Add categories with normalized indices, lowercased\n\t\t\tString cat = node.cat.toLowerCase();\n\t\t\tcat = cat.replaceAll(\"np\\\\[nb\\\\]\", \"np\");\n\t\t\t// also strip PP heads if apropos\n\t\t\tcat = stripPPHeads(cat);\n\t\t\tleaf.setAttribute(\"cat\", cat);\n\n\t\t\tString cat0 = \"\";\n\n\t\t\t// Add the same category to the treenodes\n\t\t\tif (node.type.equals(\"Treenode\"))\n\t\t\t\tcat0 = node.cat;\n\t\t\telse\n\t\t\t\tcat0 = node.catRedundant;\n\n\t\t\t// Add the bare category to the leafnodes\n\t\t\tleaf.setAttribute(\"cat0\", cat0);\n\n\t\t\t// Refresh index list.\n\t\t\tidList.clear();\n\n\t\t\treturn leaf;\n\t\t}\n\t}\n\t\n\t// strips PP heads if apropos\n\tprivate String stripPPHeads(String cat) {\n\t\tif (keepPPHeads) return cat;\n\t\treturn cat.replaceAll(\"pp\\\\[[a-z]+\\\\]\", \"pp\");\n\t}\n\t\n\t// adjusts role, stripping PP head if apropos\n\tprivate String adjustRole(String role) {\n\t\trole = role.replaceFirst(\"ARG\", \"Arg\");\n\t\tif (!keepPPHeads) {\n\t\t\tint hyph = role.indexOf('-');\n\t\t\tif (hyph > 0) role = role.substring(0, hyph);\n\t\t}\n\t\treturn role;\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/CCGBankExtract.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n/*\n * $Id: CCGBankExtract.java,v 1.5 2011/11/04 01:49:57 raja-asoka Exp $ \n */\npackage opennlp.ccgbank;\n\nimport java.io.BufferedInputStream;\nimport java.io.File;\nimport java.io.FileInputStream;\nimport java.io.FileNotFoundException;\nimport java.io.FileWriter;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.io.PrintWriter;\nimport java.io.Writer;\nimport java.util.EnumMap;\nimport java.util.Map;\n\nimport javax.xml.transform.Source;\nimport javax.xml.transform.Templates;\nimport javax.xml.transform.TransformerConfigurationException;\nimport javax.xml.transform.TransformerException;\nimport javax.xml.transform.TransformerFactory;\nimport javax.xml.transform.URIResolver;\nimport javax.xml.transform.sax.SAXTransformerFactory;\nimport javax.xml.transform.stream.StreamSource;\n\nimport opennlp.ccgbank.CCGBankTaskTemplates.Type;\nimport opennlp.ccgbank.extract.FreqTally;\nimport opennlp.ccgbank.extract.RulesTally;\nimport opennlp.ccgbank.extract.Testbed;\n\nimport org.apache.tools.ant.BuildException;\nimport org.xml.sax.InputSource;\nimport org.xml.sax.SAXException;\n\n\n/**\n * Extracts a grammar from a converted version of the CCGBank.\n * @author Scott Martin\n * @version $Revision: 1.5 $\n * @see CCGBankConvert\n */\npublic class CCGBankExtract extends CCGBankTask implements URIResolver {\n\t\n\tstatic String pkgPath = null;\n\tstatic final String LEXICON_TEMPLATE = \"lexicon-base.xsl\",\n\t\tRULES_TEMPLATE = \"rules-base.xsl\";\n\t\n\tString grammarName = \"ccgbankextract\";\n\tboolean pPheads = true, skipUnmatched = false;\n\tint catFreqCutoff = 1, lexFreqCutoff = 1, openFreqCutoff = 100,\n\t\truleFreqCutoff = 1;\n\tCCGBankTaskTestbed testbed = null;\n\t\n\tFile lexiconTempFile, rulesTempFile;\n\tTemplatesProcessor ruleProcessor;\n\tMap xsltProcessors\n\t\t= new EnumMap(Type.class);\n\t\n\tpublic CCGBankExtract() {\n\t\tsuper();\n\t\tif(pkgPath == null) {\n\t\t\tpkgPath = getClass().getPackage().getName().replace('.', '/');\n\t\t}\n\t}\n\t\n\t/**\n\t * Sets the name of the generated grammar.\n\t * @param grammarName The name of the generated grammar. This is the string\n\t * that will appear in the \"name\" attribute of the root element of the\n\t * generated grammar's grammar.xml file.\n\t */\n\tpublic void setGrammarName(String grammarName) {\n\t\tthis.grammarName = grammarName;\n\t}\n\n\t\n\t/**\n\t * @param tb the testbed to set\n\t */\n\tpublic void addConfiguredTestbed(CCGBankTaskTestbed tb) {\n\t\tthis.testbed = tb;\n\t}\n\n\t\n\t/**\n\t * @param catFreqCutoff the catFreqCutoff to set\n\t */\n\tpublic void setCatFreqCutoff(int catFreqCutoff) {\n\t\tthis.catFreqCutoff = catFreqCutoff;\n\t}\n\n\t\n\t/**\n\t * @param lexFreqCutoff the lexFreqCutoff to set\n\t */\n\tpublic void setLexFreqCutoff(int lexFreqCutoff) {\n\t\tthis.lexFreqCutoff = lexFreqCutoff;\n\t}\n\n\t\n\t/**\n\t * @param openFreqCutoff the openFreqCutoff to set\n\t */\n\tpublic void setOpenFreqCutoff(int openFreqCutoff) {\n\t\tthis.openFreqCutoff = openFreqCutoff;\n\t}\n\n\t\n\t/**\n\t * @param pPheads the ppheads to set\n\t */\n\tpublic void setPPheads(boolean pPheads) {\n\t\tthis.pPheads = pPheads;\n\t}\n\n\t\n\t/**\n\t * @param ruleFreqCutoff the ruleFreqCutoff to set\n\t */\n\tpublic void setRuleFreqCutoff(int ruleFreqCutoff) {\n\t\tthis.ruleFreqCutoff = ruleFreqCutoff;\n\t}\n\n\t\n\t/**\n\t * @param skipUnmatched the skipUnmatched to set\n\t */\n\tpublic void setSkipUnmatched(boolean skipUnmatched) {\n\t\tthis.skipUnmatched = skipUnmatched;\n\t}\n\t\n\t\n\t/* (non-Javadoc)\n\t * @see javax.xml.transform.URIResolver#resolve(java.lang.String, java.lang.String)\n\t */\n\tpublic Source resolve(String href, String base) {\n\t\tif(href != null && href.length() > 0 && href.startsWith(pkgPath)) {\n\t\t\tString lastChunk = (href.contains(\"/\") && !href.endsWith(\"/\"))\n\t\t\t\t? href.substring(href.lastIndexOf('/') + 1) : href;\n\t\t\tif(lastChunk.endsWith(CCGBankExtract.LEXICON_TEMPLATE)\n\t\t\t\t\t|| lastChunk.endsWith(CCGBankExtract.RULES_TEMPLATE)) {\n\t\t\t\treturn new StreamSource(getResource(href));\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn new StreamSource(new File(href));\n\t}\n\t\n\t\n\t/* (non-Javadoc)\n\t * @see opennlp.ccgbank.CCGBankTask#addConfiguredCCGBankTaskTemplates(opennlp.ccgbank.CCGBankTaskTemplates)\n\t */\n\t@Override\n\tpublic void addConfiguredTemplates(CCGBankTaskTemplates taskTemplates) {\n\t\tif(xsltProcessors.containsKey(taskTemplates.type)) {\n\t\t\tthrow new BuildException(taskTemplates.type \n\t\t\t\t\t+ \" extraction type is multiply defined\");\n\t\t}\n\t\t\n\t\tXSLTProcessor xp = useXMLFilter\n\t\t\t? new XMLFilterProcessor(this, this)\n\t\t\t: new TemplatesProcessor(this);\n\t\t\n\t\txp.addTemplates(taskTemplates);\n\t\txp.transformerFactory.setURIResolver(this);\n\t\t\n\t\txsltProcessors.put(taskTemplates.type, xp);\n\t}\n\n\n\t/* (non-Javadoc)\n\t * @see opennlp.ccgbank.CCGBankTask#start()\n\t */\n\t@Override\n\tprotected void start() throws BuildException {\n\t\txsltProcessor = new TemplatesProcessor(this);\n\t\t((TemplatesProcessor)xsltProcessor).addTemplates(\n\t\t\tloadTemplates(pkgPath + \"/\" + CCGBankExtract.LEXICON_TEMPLATE));\n\t\truleProcessor = new TemplatesProcessor(this);\n\t\truleProcessor.addTemplates(loadTemplates(pkgPath + \"/\"\n\t\t\t\t+ CCGBankExtract.RULES_TEMPLATE));\n\n\t\tFreqTally.reset();\n\t\tFreqTally.CAT_FREQ_CUTOFF = catFreqCutoff;\n\t\tFreqTally.LEX_FREQ_CUTOFF = lexFreqCutoff;\n\t\tFreqTally.OPEN_FREQ_CUTOFF = openFreqCutoff;\n\t\t\n\t\tRulesTally.reset();\n\t\tRulesTally.RULE_FREQ_CUTOFF = ruleFreqCutoff;\n RulesTally.KEEP_UNMATCHED = !skipUnmatched;\n\t\t\n\t\ttry {\n\t\t\tlexiconTempFile = File.createTempFile(grammarName, \".xml\");\n\t\t\tlexiconTempFile.deleteOnExit();\n\t\t\txsltProcessor.setTarget(lexiconTempFile);\n\t\t\t\n\t\t\trulesTempFile = File.createTempFile(grammarName + \"-rules\", \".xml\");\n\t\t\trulesTempFile.deleteOnExit();\n\t\t\truleProcessor.setTarget(rulesTempFile);\n\t\t\t\n\t\t\tWriter w = xsltProcessor.serializer.getWriter();\n\t\t\tw.write(\"\");\n\t\t\tw.flush();\n\t\t\t\n\t\t\tWriter rw = ruleProcessor.serializer.getWriter();\n\t\t\trw.write(\"\");\n\t\t\trw.flush();\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tthrow new BuildException(io, getLocation());\n\t\t}\n\t}\n\t\n\t\t\n\t/* (non-Javadoc)\n\t * @see opennlp.ccgbank.CCGBankTask#nextFile(java.io.File)\n\t */\n\t@Override\n\tprotected InputSource nextFile(File file) throws BuildException {\n\t\ttry {\n\t\t\truleProcessor.process(super.nextFile(file));\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tthrow new BuildException(\"I/O problem processing \" + file + \": \"\n\t\t\t\t+ io.getMessage(), io, getLocation());\n\t\t}\n\t\tcatch(SAXException se) {\n\t\t\tthrow new BuildException(\"Problem processing \" + file + \": \"\n\t\t\t\t+ se.getMessage(), se, getLocation());\n\t\t}\n\t\tcatch(TransformerException te) {\n\t\t\tthrow new BuildException(\"Problem processing \" + file + \": \"\n\t\t\t\t+ te.getMessageAndLocation(), te, getLocation());\n\t\t}\n\t\t\n\t\treturn super.nextFile(file); // TODO is this right?\n\t}\n\n\n\t/* (non-Javadoc)\n\t * @see opennlp.ccgbank.CCGBankTask#finish()\n\t */\n\t@Override\n\tprotected void finish() throws BuildException {\n\t\ttry {\n\t\t\tWriter w = xsltProcessor.serializer.getWriter();\n\t\t\tw.write(\"\");\n\t\t\tw.close();\n\t\t\t\n\t\t\tWriter rw = ruleProcessor.serializer.getWriter();\n\t\t\trw.write(\"\");\n\t\t\trw.close();\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tthrow new BuildException(io, getLocation());\n\t\t}\n\t\t\n\t\t// generate lexicon, morph, rules\n\t\tfor(Type t : xsltProcessors.keySet()) {\n\t\t\tif(t == Type.LEXICON) {\n\t\t\t\ttry {\n\t\t\t\t\tFreqTally.printTally(target);\n\t\t\t\t}\n\t\t\t\tcatch(FileNotFoundException fnfe) {\n\t\t\t\t\tthrow new BuildException(\"problem generating frequencies\",\n\t\t\t\t\t\t\tfnfe, getLocation());\n\t\t\t\t}\n\t\t\t}\n\t\t\telse if(t == Type.RULES) {\n\t\t\t\ttry {\n\t\t\t\t\tRulesTally.printTally(target);\n\t\t\t\t}\n\t\t\t\tcatch(FileNotFoundException fnfe) {\n\t\t\t\t\tthrow new BuildException(\n\t\t\t\t\t\t\"problem generating rule frequencies\", fnfe,\n\t\t\t\t\t\tgetLocation());\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tString fileName = t.fileName();\n\t\t\tlog(\"Generating \" + fileName);\n\t\t\ttry {\n\t\t\t\tXSLTProcessor xp = xsltProcessors.get(t);\n\t\t\t\txp.setTarget(new File(target, fileName));\n\t\t\t\t\n\t\t\t\txp.process(new InputSource(\n\t\t\t\t\tnew BufferedInputStream(new FileInputStream(\t\t\t\t\t\t\n\t\t\t\t\t\t(t == Type.RULES) ? rulesTempFile : lexiconTempFile))));\n\t\t\t}\n\t\t\tcatch(IOException io) {\n\t\t\t\tthrow new BuildException(\"I/O problem writing \" + fileName,\n\t\t\t\t\t\tio, getLocation());\n\t\t\t}\n\t\t\tcatch(TransformerException te) {\n\t\t\t\tthrow new BuildException(\"Problem transforming \" + fileName\n\t\t\t\t\t+ \": \" + te.getMessageAndLocation(), te, getLocation());\n\t\t\t}\n\t\t\tcatch(SAXException se) {\n\t\t\t\tthrow new BuildException(\"Problem transforming \" + fileName\n\t\t\t\t\t+ \": \" + se.getMessage(), se, getLocation());\n\t\t\t}\n\t\t}\n\t\t\n\t\t// generate grammar.xml, if it doesn't already exist\n\t\t// nb: should eventually make schema refs relative to OPENCCG_HOME\t\t\n\t\ttry {\n\t\t\tFile gramFile = new File(target, \"grammar.xml\");\n\t\t\tif (!gramFile.exists()) {\n\t\t\t\tlog(\"Generating grammar.xml\");\n\t\t\t\tPrintWriter gramOut = new PrintWriter(new FileWriter(gramFile));\n\t\t\t\tgramOut.println(\"\");\n\t\t\t\tgramOut.println(\"\");\n\t\t\t\tgramOut.println(\" \");\n\t\t\t\tgramOut.println(\" \");\n\t\t\t\tgramOut.println(\" \");\n\t\t\t\tgramOut.println(\"\");\n\t\t\t\tgramOut.println(\"\");\n\t\t\t\tgramOut.println(\"\");\n\t\t\t\tgramOut.println(\"\");\n\t\t\t\tgramOut.println(\"\");\n\t\t\t\tgramOut.println(\"\");\n\t\t\t\tgramOut.println(\"\");\n\t\t\t\tgramOut.println(\"\");\n\t\t\t\tgramOut.println(\"\");\n\t\t\t\tgramOut.println(\"\");\n\t\t\t\tgramOut.close();\n\t\t\t}\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tthrow new BuildException(\"problem generating grammar.xml\",\n\t\t\t\t\tio, getLocation());\n\t\t}\n\t\t\n\t\tif(testbed != null) {\n\t\t\tlog(\"Creating testbed ...\");\n\t\t\t\n\t\t\ttry {\n\t\t\t\tTestbed ct = new Testbed(ccgBankTaskSources,\n\t\t\t\t\t\ttarget, testbed);\n\t\t\t\tct.createTestFiles();\n\t\t\t}\n\t\t\tcatch(Exception e) {e.printStackTrace();\n\t\t\t\tthrow new BuildException(\"problem generating testbed: \"\n\t\t\t\t\t\t+ e.getMessage(), e, getLocation());\n\t\t\t}\n\t\t}\n\t}\n\t\n\tTemplates loadTemplates(String resourceName) throws BuildException {\n\t\ttry {\n\t\t\t// XXX nb: no xsltc option this way\n\t\t\t//TransformerFactory tf = XSLTProcessor.newTransformerFactory(); \n\t\t\tSAXTransformerFactory tf = (SAXTransformerFactory)TransformerFactory.newInstance();\n\t\t\t\n\t\t\treturn tf.newTemplates(new StreamSource(new BufferedInputStream(\n\t\t\t\tgetResource(resourceName))));\n\t\t}\n\t\tcatch(TransformerConfigurationException e) {\n\t\t\tthrow new BuildException(\"Problem loading template \"\n\t\t\t\t+ resourceName + \": \" + e.getMessage(), e, getLocation());\n\t\t}\n\t}\n\t\n\t/**\n\t * Loads a resource using the fully qualified name with the current\n\t * class loader\n\t */\n\tInputStream getResource(String resourceName) {\n\t\treturn getClass().getClassLoader().getResourceAsStream(resourceName);\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/CCGBankTask.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n/*\n * $Id: CCGBankTask.java,v 1.5 2010/11/30 18:44:32 mwhite14850 Exp $ \n */\npackage opennlp.ccgbank;\n\nimport java.io.BufferedInputStream;\nimport java.io.File;\nimport java.io.FileInputStream;\nimport java.io.FileNotFoundException;\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Set;\n\nimport javax.xml.transform.ErrorListener;\nimport javax.xml.transform.TransformerException;\n\nimport opennlp.ccgbank.parse.TokenMgrError;\n\nimport org.apache.tools.ant.BuildException;\nimport org.apache.tools.ant.Task;\nimport org.apache.tools.ant.TaskContainer;\nimport org.xml.sax.ErrorHandler;\nimport org.xml.sax.InputSource;\nimport org.xml.sax.SAXException;\nimport org.xml.sax.SAXParseException;\n\n\n/**\n * Abstract class to provide functionality for applying XSLT to an XML stream.\n *

    \n * This class is designed to be run as a task from within an Ant build file.\n * There is one concrete implementer for each of the separate tasks of \n * converting the CCGBank and extracting a grammar for the converted corpus.\n * @author Scott Martin\n * @version $Revision: 1.5 $\n * @see CCGBankConvert\n * @see CCGBankExtract\n * @see Ant home page\n */\npublic abstract class CCGBankTask extends Task\n\t\timplements TaskContainer,ErrorHandler,ErrorListener {\n\t\n\tFile target;\n\tSet ccgBankTaskSources\n\t\t= new HashSet();\n\tList ccgBankTaskTemplates\n\t\t= new ArrayList();\n\tXSLTProcessor xsltProcessor = null;\n\t\n\tboolean useXMLFilter = true;\n\tboolean terminateOnError = true, terminateOnWarning = false;\n\t\t\n\t/**\n\t * Counters\n\t */\n\tint directoriesProcessed, filesProcessed, warnings, errors;\n\t\n\t\n\t/**\n\t * Adds a sub-task, but included here only for binary compatibility.\n\t * @throws BuildException Always throws an exception, only\n\t * sourcesSet or xsltProcessors can be added to this task.\n\t */\n\tpublic void addTask(Task task) {\n\t\tthrow new BuildException(\"nested task \\\"\" + task.getTaskName()\n\t\t\t\t+ \"\\\" not supported, only sourcesSet or xsltProcessors\");\n\t}\n\n\t\n\t/**\n\t * Sets the target directory.\n\t * @param target The location of the result of the XSLT conversion (the \n\t * converted corpus, extracted grammar, etc.).\n\t * @throws BuildException If the target is null\n\t * or not a directory.\n\t */\n\tpublic void setTarget(File target) {\n\t\tthis.target = target;\n\t\t\n\t\tif(target == null) {\n\t\t\tthrow new BuildException(\"no target specified\");\n\t\t}\n\t\tif(!target.exists()) {\n\t\t\ttarget.mkdirs();\n\t\t}\n\t\telse if(!target.isDirectory()) {\n\t\t\tthrow new BuildException(\"specified target is not a directory\");\n\t\t}\n\t}\n\t\n\t/**\n\t * @param terminateOnError the terminateOnError to set\n\t */\n\tpublic void setTerminateOnError(boolean terminateOnError) {\n\t\tthis.terminateOnError = terminateOnError;\n\t}\n\n\n\t\n\t/**\n\t * @param terminateOnWarning the terminateOnWarning to set\n\t */\n\tpublic void setTerminateOnWarning(boolean terminateOnWarning) {\n\t\tthis.terminateOnWarning = terminateOnWarning;\n\t}\n\n\t\n\t/**\n\t * @param useXMLFilter the useXMLFilter to set\n\t */\n\tpublic void setUseXMLFilter(boolean useXMLFilter) {\n\t\tthis.useXMLFilter = useXMLFilter;\n\t}\n\n\n\t/**\n\t * Adds a file set of source files.\n\t */\n\tpublic void addConfiguredSources(CCGBankTaskSources sources) {\n\t\tccgBankTaskSources.add(sources);\n\t}\n\t\n\t\n\t/**\n\t * Adds a series of xsltProcessors for XSLT transformation.\n\t */\n\tpublic void addConfiguredTemplates(CCGBankTaskTemplates templates) {\n\t\tccgBankTaskTemplates.add(templates);\n\t}\n\n\t\n\t/**\n\t * Hook to be overridden by subclasses that want notification of the start\n\t * of the transformation process.\n\t */\n\tprotected void start() throws BuildException {\t\t\n\t\t// to be overridden\n\t}\n\t\n\t\n\t/**\n\t * Hook to be overridden by subclasses that want notification of the end\n\t * of the transformation process.\n\t */\n\tprotected void finish() throws BuildException {\t\t\n\t\t// to be overridden\n\t}\n\n\t\n\t/**\n\t * Hook that lets subclasses be notified when processing starts on a new\n\t * directory.\n\t * @param section The file (directory) on which processing is starting.\n\t */\n\tprotected void nextDirectory(File section)\n\t\t\tthrows BuildException {\t\t\n\t\t// to be overridden\n\t} \n\t\n\n\t/**\n\t * Hook that lets implementing subclasses know when processing starts on\n\t * a new file.\n\t * @param file The file on which processing is about to start.\n\t * @return The input source to process.\n\t */\n\tprotected InputSource nextFile(File file) throws BuildException {\n\t\ttry {\n\t\t\treturn new InputSource(\n\t\t\t\t\tnew BufferedInputStream(new FileInputStream(file)));\n\t\t}\n\t\tcatch(FileNotFoundException fnfe) {\n\t\t\tthrow new BuildException(\"Unable to find file \" + file,\n\t\t\t\t\tfnfe, getLocation());\n\t\t}\n\t}\n\t\n\t\n\t/**\n\t * Required by {@link ErrorHandler}. Reports the specified error using the\n\t * Ant task {@link Task#log(String)} method.\n\t */\n\tpublic void error(SAXParseException exception) {\n\t\terrors++;\n\t\thandleError(\"Error\", exception, terminateOnError);\n\t}\n\n\t\n\t/**\n\t * Required by {@link ErrorHandler}. Reports the specified error using the\n\t * Ant task {@link Task#log(String)} method.\n\t */\n\tpublic void fatalError(SAXParseException exception) {\n\t\terrors++;\n\t\thandleError(\"Fatal error\", exception, terminateOnError);\n\t}\n\n\t\n\t/**\n\t * Required by {@link ErrorHandler}. Reports the specified error using the\n\t * Ant task {@link Task#log(String)} method.\n\t */\n\tpublic void warning(SAXParseException exception) {\n\t\twarnings++;\n\t\thandleError(\"Warning\", exception, terminateOnWarning);\n\t}\n\t\n\t\n\t/**\n\t * Required by {@link ErrorListener}. Reports the specified error using the\n\t * Ant task {@link Task#log(String)} method.\n\t */\n\tpublic void error(TransformerException exception) {\n\t\terrors++;\n\t\thandleError(\"Error\", exception, terminateOnError);\n\t}\n\n\t\n\t/**\n\t * Required by {@link ErrorListener}. Reports the specified error using the\n\t * Ant task {@link Task#log(String)} method.\n\t */\n\tpublic void fatalError(TransformerException exception) {\n\t\terrors++;\n\t\thandleError(\"Fatal error\", exception, terminateOnError);\n\t}\n\n\t\n\t/**\n\t * Required by {@link ErrorListener}. Reports the specified error using the\n\t * Ant task {@link Task#log(String)} method.\n\t */\n\tpublic void warning(TransformerException exception) {\n\t\twarnings++;\n\t\thandleError(\"Warning\", exception, terminateOnWarning);\n\t}\n\n\t\n\t/**\n\t * Helper method for the methods required by {@link ErrorHandler}.\n\t */\n\tvoid handleError(String prefix, SAXParseException spe, boolean terminate) {\n\t\tStringBuilder sb = new StringBuilder(prefix);\n\t\tsb.append(\": problem in parse: \");\n\t\tsb.append(spe.getSystemId());\n\t\tsb.append(\" on line \");\n\t\tsb.append(spe.getLineNumber());\n\t\tsb.append(\", column \");\n\t\tsb.append(spe.getColumnNumber());\n\t\tsb.append(\": \");\n\t\tsb.append(spe.getMessage());\n\t\t\n\t\tif(!terminate) {\n\t\t\tlog(sb.toString());\n\t\t}\n\t\telse {\n\t\t\tthrow new BuildException(sb.toString(), spe, getLocation());\n\t\t}\n\t}\n\t\n\t\n\t/**\n\t * Helper method for the methods required by {@link ErrorListener}.\n\t */\n\tvoid handleError(String prefix, TransformerException te,\n\t\t\tboolean terminate) {\n\t\tStringBuilder sb = new StringBuilder(prefix);\n\t\tsb.append(\": problem in transform: \");\n\t\tsb.append(te.getMessageAndLocation());\n\t\t\n\t\tif(!terminate) {\n\t\t\tlog(sb.toString());\n\t\t}\n\t\telse {\n\t\t\tthrow new BuildException(sb.toString(), te, getLocation());\n\t\t}\n\t}\n\t\n\t\n\t/**\n\t * Does the work of transforming the CCGBank and extracting grammars.\n\t * @throws BuildException In case no sourcesSet have been specified or an \n\t * error occurs during the transformation process.\n\t *

    \n\t * This method calls {@link #start()}, {@link #finish()},\n\t * {@link #nextDirectory(File)}, and {@link #nextFile(File)} as required.\n\t */\n\t@Override\n\tpublic void execute() throws BuildException {\n\t\tif(ccgBankTaskSources.isEmpty()) {\n\t\t\tthrow new BuildException(\"no sourcesSet specified\");\n\t\t}\n\t\t\t\t\n\t\tfilesProcessed = directoriesProcessed = warnings = errors = 0;\n\t\t\n\t\tstart();\n\t\tlog(\"Target: \" + target);\n\t\t\n\t\tif(xsltProcessor == null) { // should have been configured\n\t\t\tthrow new BuildException(\"null XSLT processor\");\n\t\t}\n\t\t\n\t\txsltProcessor.addAllTemplates(ccgBankTaskTemplates);\n\t\t\n\t\ttry {\n\t\t\tfor(CCGBankTaskSources sources : ccgBankTaskSources) {\n\t\t\t\tFile prevDir = null;\n\t\t\t\tFile currentDir = null;\n\t\t\t\t\n\t\t\t\tfor(File file : sources) {\n\t\t\t\t\tcurrentDir = file.getParentFile();\n\t\t\t\t\tif(!currentDir.equals(prevDir)) {\n\t\t\t\t\t\tlog(\"Processing \" + currentDir + \" ...\");\n\t\t\t\t\t\tdirectoriesProcessed++;\n\t\t\t\t\t\t\n\t\t\t\t\t\tnextDirectory(currentDir);\n\t\t\t\t\t}\n\t\t\t\t\t\n\t\t\t\t\tprevDir = currentDir;\n\t\t\t\t\t\n\t\t\t\t\tlog(\"Processing \" + file);\n\t\t\t\t\tfilesProcessed++;\n\t\t\t\t\t\n\t\t\t\t\txsltProcessor.process(nextFile(file));\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tthrow new BuildException(\"I/O problem during processing: \" + \n\t\t\t\tio.getMessage(), io, getLocation());\n\t\t}\n\t\tcatch(SAXException se) {\n\t\t\tthrow new BuildException(\"Problem during processing: \" + \n\t\t\t\t\tse.getMessage(), se, getLocation());\n\t\t}\n\t\tcatch(TransformerException te) {\n\t\t\tthrow new BuildException(\"I/O problem during processing: \" + \n\t\t\t\t\tte.getMessageAndLocation(), te, getLocation());\n\t\t}\n\t\tcatch (TokenMgrError te) {\n\t\t\tthrow new BuildException(\"I/O problem during processing: \" + \n\t\t\t\t\tte.getMessage(), te, getLocation());\n\t\t}\n\t\tfinally {\n\t\t\tfinish();\n\t\t\t\n\t\t\tStringBuilder sb = new StringBuilder(\"Processed \");\n\t\t\tsb.append(filesProcessed);\n\t\t\tsb.append(\" files in \");\n\t\t\tsb.append(directoriesProcessed);\n\t\t\tsb.append(\" directories with \");\n\t\t\tsb.append(errors);\n\t\t\tsb.append(\" error\");\n\t\t\tif(errors != 1) {\n\t\t\t\tsb.append('s');\n\t\t\t}\n\t\t\tsb.append(\" and \");\n\t\t\tsb.append(warnings);\n\t\t\tsb.append(\" warning\");\n\t\t\tif(warnings != 1) {\n\t\t\t\tsb.append('s');\n\t\t\t}\n\t\t\t\n\t\t\tlog(sb.toString());\n\t\t}\n\t}\n\n}\n" }, { "path": "src/opennlp/ccgbank/CCGBankTaskFileGroup.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n/*\n * $Id: CCGBankTaskFileGroup.java,v 1.1 2009/11/09 19:21:50 mwhite14850 Exp $ \n */\npackage opennlp.ccgbank;\n\nimport java.io.File;\nimport java.util.Collection;\nimport java.util.Iterator;\nimport java.util.NoSuchElementException;\n\nimport org.apache.tools.ant.BuildException;\nimport org.apache.tools.ant.Task;\nimport org.apache.tools.ant.TaskContainer;\nimport org.apache.tools.ant.types.FileList;\nimport org.apache.tools.ant.types.FileSet;\n\n\n/**\n * Abstract class providing generalized functionality for file groups used by\n * {@link CCGBankTask}s.\n *

    \n * For convenience, this class implements {@link Iterable} over {@link File}s.\n * This allows instances to be used in standard iteration constructs while\n * abstracting away from implementation details such as multiple\n * {@link FileSet}s or {@link FileList}s. \n * @author Scott Martin\n * @version $Revision: 1.1 $\n */\npublic abstract class CCGBankTaskFileGroup extends Task implements\n\t\tTaskContainer,Iterable {\n\n\tprotected Collection subGroups;\n\t\n\t/**\n\t * Creates a file group over the specified collection of subgroups.\n\t * @param subGroups The collection over which this class is an abstracted\n\t * view.\n\t */\n\tprotected CCGBankTaskFileGroup(Collection subGroups) {\n\t\tthis.subGroups = subGroups;\n\t}\n\t\n\t\n\t/**\n\t * Adds a subgroup to the collection of subgroups this class abstracts over.\n\t */\n\tprotected void addGroup(G group) {\n\t\tsubGroups.add(group);\n\t}\n\t\n\t\n\t/**\n\t * Gets all the files in a group as an array. To be implemented by extending\n\t * classes, as {@link FileSet} and {@link FileList} represent files \n\t * differently.\n\t * @return The collection of files in group, as an array.\n\t */\n\tprotected abstract File[] getFiles(G group); \n\t\n\t\n\t/**\n\t * Creates an array of files given a directory and an array of file names\n\t * (specified relative to that directory).\n\t * @param directory The directory that the specified file names are relative\n\t * to.\n\t * @param fileNames The file names, relative to the specified directory.\n\t * @return An array containing all the files as specified relative to the\n\t * specified directory.\n\t */\n\tprotected File[] makeFiles(File directory, String[] fileNames) {\n\t\tFile[] files = new File[fileNames.length];\n\t\t\n\t\tfor(int i = 0; i < fileNames.length; i++) {\n\t\t\tfiles[i] = new File(directory, fileNames[i]);\n\t\t}\n\t\t\n\t\treturn files;\n\t}\n\t\n\t\n\t/**\n\t * Included for binary compatibility with {@link TaskContainer}.\n\t * @throws BuildException Always throws a build exception, as only the\n\t * parameterized type of this class's subgroups can be contained by this\n\t * task.\n\t */\n\tpublic void addTask(Task task) {\n\t\tthrow new BuildException(\"nested task \\\"\" + task\n\t\t\t\t+ \"\\\" not supported, only \"\n\t\t\t\t+ subGroups.getClass().getTypeParameters()[0]\n\t\t\t\t .getGenericDeclaration());\n\t}\n\t\n\t\n\t/**\n\t * Provides an\n\t * iterator over all the files in the collection of subgroups contained by\n\t * this instance. The iterator returned will iterate through files in all\n\t * the subgroups returned in the same order as the order returned by the\n\t * subgroups collection. \n\t */\n\tpublic Iterator iterator() {\n\t\treturn new AllFileView();\n\t}\n\n\t\n\t/**\n\t * Implements an iterator over the files contained in the subgroups\n\t * collection. This class iterates over all the files contained in the\n\t * groups in the subgroups collection, in the order that they\n\t * are returned by the subgroups collection.\n\t * @author Scott Martin\n\t * @version $Revision: 1.1 $\n\t */\n\tclass AllFileView implements Iterator {\n\n\t\tIterator groupIterator = subGroups.iterator();\n\t\tIterator currentIterator;\n\t\t\n\t\t/**\n\t\t * Tests whether there is a next file.\n\t\t * @return true If the current subgroup contains a next file, or if\n\t\t * there is a next subgroup that is non-empty.\n\t\t */\n\t\tpublic boolean hasNext() {\n\t\t\twhile((currentIterator == null || !currentIterator.hasNext())\n\t\t\t\t\t&& groupIterator.hasNext()) {\n\t\t\t\tcurrentIterator = new FileArrayIterator(\n\t\t\t\t\t\tgetFiles(groupIterator.next()));\n\t\t\t}\n\t\t\t\n\t\t\t// current may be empty\n\t\t\treturn (currentIterator != null && currentIterator.hasNext()); \n\t\t}\n\t\t\n\t\t\n\t\t/**\n\t\t * Gets the next file in the series, as returned in order by the \n\t\t * subgroups collection.\n\t\t * @throws NoSuchElementException If the collection of subgroups is\n\t\t * exhausted.\n\t\t */\n\t\tpublic File next() {\n\t\t\tif(!hasNext()) {\n\t\t\t\tthrow new NoSuchElementException(\"elements exhausted\");\n\t\t\t}\n\t\t\t\n\t\t\treturn currentIterator.next();\n\t\t}\n\t\t\n\t\t/**\n\t\t * Included only for binary compatibility with {@link Iterator}.\n\t\t * @throws UnsupportedOperationException Always, as this operation is\n\t\t * not supported.\n\t\t */\n\t\tpublic void remove() {\n\t\t\tthrow new UnsupportedOperationException(\"removed not supported\");\n\t\t}\n\t\t\n\t}\n\t\n\t\n\t/**\n\t * Implements an iterator view of an array of {@link File} objects.\n\t * @author Scott Martin\n\t * @version $Revision: 1.1 $\n\t */\n\tclass FileArrayIterator implements Iterator {\n\n\t\tFile[] array;\n\t\tint index = 0;\n\t\t\n\t\t\n\t\t/**\n\t\t * Creates a new iterator view over the specified array of files.\n\t\t * @param array The file array backing this iterator view.\n\t\t */\n\t\tFileArrayIterator(File[] array) {\n\t\t\tthis.array = array;\n\t\t}\n\t\t\n\t\t/**\n\t\t * Tests whether the array of files is exhausted.\n\t\t * @return true If the current index is less than the array length.\n\t\t */\n\t\tpublic boolean hasNext() {\n\t\t\treturn (index < array.length);\n\t\t}\n\n\t\t/**\n\t\t * Gets the next file in series, as specified by the array backing this\n\t\t * iterator view.\n\t\t * @throws NoSuchElementException If the array of files is exhausted.\n\t\t * @see #hasNext()\n\t\t */\n\t\tpublic File next() {\n\t\t\tif(!hasNext()) {\n\t\t\t\tthrow new NoSuchElementException(\"elements exhausted\");\n\t\t\t}\n\t\t\t\n\t\t\treturn array[index++];\n\t\t}\n\n\t\t/**\n\t\t * Included only for binary compatibility with {@link Iterator}.\n\t\t * @throws UnsupportedOperationException Always, as this operation is\n\t\t * not supported.\n\t\t */\n\t\tpublic void remove() {\n\t\t\tthrow new UnsupportedOperationException(\"remove not supported\");\n\t\t}\n\t\t\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/CCGBankTaskSources.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n/*\n * $Id: CCGBankTaskSources.java,v 1.1 2009/11/09 19:21:50 mwhite14850 Exp $ \n */\npackage opennlp.ccgbank;\n\nimport java.io.File;\nimport java.util.HashSet;\n\nimport org.apache.tools.ant.BuildException;\nimport org.apache.tools.ant.DirectoryScanner;\nimport org.apache.tools.ant.Project;\nimport org.apache.tools.ant.types.FileSet;\n\n\n/**\n * Represents a set of source files. This class encapsulates a set of \n * {@link FileSet}s.\n * @author Scott Martin\n * @version $Revision: 1.1 $\n * @see Ant home page\n * @see FileSet\n */\npublic class CCGBankTaskSources extends CCGBankTaskFileGroup {\n\n\t/**\n\t * Creates a new sources object (required by Ant).\n\t */\n\tpublic CCGBankTaskSources() {\n\t\tsuper(new HashSet());\n\t}\n\t\n\t\n\t/**\n\t * Adds a file set to this sources object.\n\t */\n\tpublic void addConfiguredFileSet(FileSet fileSet) {\n\t\taddGroup(fileSet);\n\t}\n\n\n\t/**\n\t * Gets the files in the specified group as an array of files. The files in\n\t * the returned array are in the order returned by group's\n\t * {@link FileSet#getDirectoryScanner(Project) directory scanner}.\n\t */\n\t@Override\n\tprotected File[] getFiles(FileSet group) {\n\t\tProject proj = getProject();\n\t\tDirectoryScanner scanner = group.getDirectoryScanner(proj);\n\t\tscanner.scan();\n\t\t\n\t\tString[] fileNames = scanner.getIncludedFiles();\n\t\tif(fileNames.length == 0) {\n\t\t\tthrow new BuildException(\"no source files included\");\n\t\t}\n\t\t\n\t\treturn makeFiles(group.getDir(proj), fileNames);\n\t}\n\n}\n" }, { "path": "src/opennlp/ccgbank/CCGBankTaskTemplates.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n/*\n * $Id: CCGBankTaskTemplates.java,v 1.1 2009/11/09 19:21:50 mwhite14850 Exp $ \n */\npackage opennlp.ccgbank;\n\nimport java.io.File;\nimport java.util.ArrayList;\n\nimport org.apache.tools.ant.Project;\nimport org.apache.tools.ant.types.FileList;\n\n\n/**\n * Represents a series of templates. This class encapsulates a list of lists of\n * {@link FileList}s.\n * @author Scott Martin\n * @version $Revision: 1.1 $\n * @see Ant home page\n * @see FileList\n */\npublic class CCGBankTaskTemplates extends CCGBankTaskFileGroup {\n\n\t/**\n\t * File types/names for the generated OpenCCG-format grammar files.\n\t * @author Scott Martin\n\t * @version $Revision: 1.1 $\n\t */\n\tenum Type {\n\t\t/**\n\t\t * The lexicon file.\n\t\t */\n\t\tLEXICON, \n\t\t\n\t\t/**\n\t\t * The file containing morphological information.\n\t\t */\n\t\tMORPH, \n\t\t\n\t\t/**\n\t\t * The file where the grammar rules are stored.\n\t\t */\n\t\tRULES;\n\t\t\n\t\t/**\n\t\t * Gets a filename corresponding to a given file type.\n\t\t * @return The file type's name, lowercased, with the string \n\t\t * \".xml\" appended. Example: for LEXICON,\n\t\t * returns the string lexicon.xml.\n\t\t */\n\t\tString fileName() {\n\t\t\tStringBuilder sb = new StringBuilder(name().toLowerCase());\n\t\t\tsb.append(\".xml\");\n\t\t\treturn sb.toString();\n\t\t}\n\t}\n\t\n\tType type = null;\n\t\n\t\n\t/**\n\t * Creates a new xsltProcessors object (no-arg constructor required by Ant).\n\t */\n\tpublic CCGBankTaskTemplates() {\n\t\tsuper(new ArrayList());\n\t}\n\t\n\t/**\n\t * Adds a file list to the list of transforms.\n\t * @param fileList The FileList object to add.\n\t */\n\tpublic void addConfiguredFilelist(FileList fileList) {\n\t\taddGroup(fileList);\n\t}\n\n\t\n\t/**\n\t * Gets the list of files contained in group as an array.\n\t * The order of files in the returned array is the same as the order \n\t * of group's {@link FileList#getFiles(Project) files}.\n\t */\n\t@Override\n\tprotected File[] getFiles(FileList group) {\n\t\tProject proj = getProject();\n\t\treturn makeFiles(group.getDir(proj), group.getFiles(proj));\n\t}\n\n\t/**\n\t * Sets the {@link CCGBankTaskTemplates#type file type}.\n\t * @param typeName The name of the type to set. The actual type is\n\t * coerced using {@link Enum#valueOf(Class, String)} using\n\t * typeName as an argument.\n\t */\n\tpublic void setType(String typeName) {\n\t\tthis.type = Type.valueOf(typeName.toUpperCase());\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/CCGBankTaskTestbed.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n/*\n * $Id: CCGBankTaskTestbed.java,v 1.3 2010/12/09 04:58:12 mwhite14850 Exp $ \n */\npackage opennlp.ccgbank;\n\nimport java.io.File;\n\nimport org.apache.tools.ant.Task;\n\n\n/**\n * @author Scott Martin\n * @version $Revision: 1.3 $\n *\n */\npublic class CCGBankTaskTestbed extends Task {\n\t\n\tboolean debugDerivations = false, showSem = false;\n\tFile text, factors, combos, preds, treeAuxFile;\n\t\n\t\n\t/**\n\t * @return the combos\n\t */\n\tpublic File getCombos() {\n\t\treturn combos;\n\t}\n\n\t\n\t/**\n\t * @return the debugDerivations\n\t */\n\tpublic boolean isDebugDerivations() {\n\t\treturn debugDerivations;\n\t}\n\n\t\n\t/**\n\t * @return the showSem\n\t */\n\tpublic boolean isShowsSem() {\n\t\treturn showSem;\n\t}\n\n\t\n\t/**\n\t * @return the factors\n\t */\n\tpublic File getFactors() {\n\t\treturn factors;\n\t}\n\n\t\n\t/**\n\t * @return the preds\n\t */\n\tpublic File getPreds() {\n\t\treturn preds;\n\t}\n\n\t\n\t/**\n\t * @return the text\n\t */\n\tpublic File getText() {\n\t\treturn text;\n\t}\n\n\t//Get the file which stores info about the id info of treenodes\n\tpublic File getTree() {\n\t\treturn treeAuxFile;\n\t}\n\t\n\t/**\n\t * @param combos the combos to set\n\t */\n\tpublic void setCombos(File combos) {\n\t\tthis.combos = combos;\n\t}\n\t\n\t/**\n\t * @param debugDerivations the debugDerivations to set\n\t */\n\tpublic void setDebugDerivations(boolean debugDerivations) {\n\t\tthis.debugDerivations = debugDerivations;\n\t}\n\t\n\t/**\n\t * @param showSem the showSem to set\n\t */\n\tpublic void setShowSem(boolean showSem) {\n\t\tthis.showSem = showSem;\n\t}\n\t\n\t/**\n\t * @param factors the factors to set\n\t */\n\tpublic void setFactors(File factors) {\n\t\tthis.factors = factors;\n\t}\n\t\n\t/**\n\t * @param preds the preds to set\n\t */\n\tpublic void setPreds(File preds) {\n\t\tthis.preds = preds;\n\t}\n\t\n\t/**\n\t * @param text the text to set\n\t */\n\tpublic void setText(File text) {\n\t\tthis.text = text;\n\t}\n\t\n\tpublic void setTree(File treeAuxFile) {\n\t\tthis.treeAuxFile = treeAuxFile;\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/InputSourceAdapter.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n/*\n * $Id: InputSourceAdapter.java,v 1.1 2009/11/09 19:21:50 mwhite14850 Exp $\n * Copyright (C) 2009 Scott Martin (http://www.coffeeblack.org/contact/)\n */\npackage opennlp.ccgbank;\n\nimport java.io.File;\nimport java.io.InputStream;\nimport java.io.Reader;\n\nimport javax.xml.transform.Source;\nimport javax.xml.transform.stream.StreamSource;\n\nimport org.xml.sax.InputSource;\n\n\n/**\n * Turns an {@link InputSource} into a {@link StreamSource}. This class wraps\n * an input source for XSLT transformation routines that expect {@link Source}\n * objects.\n * @author Scott Martin\n * @version $Revision: 1.1 $\n */\nclass InputSourceAdapter extends StreamSource {\n\tInputSource inputSource;\n\t\n\tInputSourceAdapter(InputSource inputSource) {\n\t\tthis.inputSource = inputSource;\n\t}\n\n\t/**\n\t * @return\n\t * @see org.xml.sax.InputSource#getPublicId()\n\t */\n\t@Override\n\tpublic String getPublicId() {\n\t\treturn inputSource.getPublicId();\n\t}\n\n\t/**\n\t * @return\n\t * @see org.xml.sax.InputSource#getSystemId()\n\t */\n\t@Override\n\tpublic String getSystemId() {\n\t\treturn inputSource.getSystemId();\n\t}\n\n\t/**\n\t * @param publicId\n\t * @see org.xml.sax.InputSource#setPublicId(java.lang.String)\n\t */\n\t@Override\n\tpublic void setPublicId(String publicId) {\n\t\tinputSource.setPublicId(publicId);\n\t}\n\n\t/**\n\t * @param systemId\n\t * @see org.xml.sax.InputSource#setSystemId(java.lang.String)\n\t */\n\t@Override\n\tpublic void setSystemId(String systemId) {\n\t\tinputSource.setSystemId(systemId);\n\t}\n\n\t/* (non-Javadoc)\n\t * @see javax.xml.transform.stream.StreamSource#getInputStream()\n\t */\n\t@Override\n\tpublic InputStream getInputStream() {\n\t\treturn inputSource.getByteStream();\n\t}\n\n\t/* (non-Javadoc)\n\t * @see javax.xml.transform.stream.StreamSource#getReader()\n\t */\n\t@Override\n\tpublic Reader getReader() {\n\t\treturn inputSource.getCharacterStream();\n\t}\n\n\t/* (non-Javadoc)\n\t * @see javax.xml.transform.stream.StreamSource#setInputStream(java.io.InputStream)\n\t */\n\t@Override\n\tpublic void setInputStream(InputStream inputStream) {\n\t\tinputSource.setByteStream(inputStream);\n\t}\n\n\t/* (non-Javadoc)\n\t * @see javax.xml.transform.stream.StreamSource#setReader(java.io.Reader)\n\t */\n\t@Override\n\tpublic void setReader(Reader reader) {\n\t\tinputSource.setCharacterStream(reader);\n\t}\n\n\t/* (non-Javadoc)\n\t * @see javax.xml.transform.stream.StreamSource#setSystemId(java.io.File)\n\t */\n\t@Override\n\tpublic void setSystemId(File f) {\n\t\tsuper.setSystemId(f);\n\t\tinputSource.setSystemId(super.getSystemId());\n\t}\n\t\n}\n" }, { "path": "src/opennlp/ccgbank/TemplatesProcessor.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n/*\n * $Id: TemplatesProcessor.java,v 1.2 2010/09/04 16:24:36 mwhite14850 Exp $\n * Copyright (C) 2009 Scott Martin (http://www.coffeeblack.org/contact/)\n */\npackage opennlp.ccgbank;\n\nimport java.io.BufferedOutputStream;\nimport java.io.ByteArrayInputStream;\nimport java.io.ByteArrayOutputStream;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.util.ArrayList;\nimport java.util.Iterator;\nimport java.util.List;\n\nimport javax.xml.transform.ErrorListener;\nimport javax.xml.transform.Result;\nimport javax.xml.transform.Source;\nimport javax.xml.transform.Templates;\nimport javax.xml.transform.Transformer;\nimport javax.xml.transform.TransformerException;\nimport javax.xml.transform.stream.StreamResult;\nimport javax.xml.transform.stream.StreamSource;\n\nimport org.xml.sax.InputSource;\nimport org.xml.sax.SAXException;\n\n\n/**\n * Implements an XSLT processor using {@link Templates}. This class processes\n * XSLT template objects successively with a given input, writing the output\n * of each successive transformation into memory, then feeding that output to\n * the next template in the chain.\n * @author Scott Martin\n * @version $Revision: 1.2 $\n */\nclass TemplatesProcessor extends XSLTProcessor {\n\tList templates = null;\n\t\n\tTemplatesProcessor(ErrorListener errorListener) {\n\t\tsuper(errorListener);\n\t}\n\t\n\tvoid addTemplates(Templates t) {\n\t\tif(templates == null) {\n\t\t\ttemplates = new ArrayList();\n\t\t}\n\t\t\n\t\ttemplates.add(t);\n\t}\n\t\n\t/* (non-Javadoc)\n\t * @see opennlp.ccgbank.XSLTProcessor#process(java.io.File)\n\t */\n\t@Override\n\tvoid process(InputSource inputSource) throws IOException,SAXException,\n\t\t\tTransformerException {\n\t\tif(templates == null) {\n\t\t\ttemplates = makeTemplates(taskTemplatesList);\n\t\t}\n\t\t\n\t\tStreamSource input = new InputSourceAdapter(inputSource);\n\t\t\n\t\tByteArrayOutputStream buffer = new ByteArrayOutputStream();\n\t\tbyte[] bytesIn = null;\n\t\t\n\t\ttry {\n\t\t\t// transform input with each template successively,\n\t\t\t// writing the output of each to a memory buffer\n\t\t\tIterator i = templates.iterator();\n\t\t\tSource source;\n\t\t\tStreamSource memorySource = null;\n\t\t\twhile(i.hasNext()) {\n\t\t\t\tif(bytesIn == null) { // first pass?\n\t\t\t\t\tsource = input; // use source\n\t\t\t\t}\n\t\t\t\telse { // use buffer otherwise\n\t\t\t\t\tInputStream in = new ByteArrayInputStream(bytesIn);\n\t\t\t\t\tif(memorySource == null) {\n\t\t\t\t\t\tmemorySource = new StreamSource(in);\t\t\t\t\t\t\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tmemorySource.setInputStream(in);\n\t\t\t\t\t}\n\t\t\t\t\t\n\t\t\t\t\tsource = memorySource;\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\t// get and configure transformer for this template\n\t\t\t\tTemplates template = i.next();\n\t\t\t\tTransformer transformer = template.newTransformer();\n\t\t\t\ttransformer.setOutputProperties(xmlProperties);\n\t\t\t\ttransformer.setErrorListener(errorListener);\n\t\t\t\t\n\t\t\t\tboolean ihn = i.hasNext(); // reuse\n\t\t\t\t\n\t\t\t\tResult result = ihn // last template?\n\t\t\t\t\t? new StreamResult(buffer)\n\t\t\t\t\t// if it's the last, write output to file\n\t\t\t\t\t: new StreamResult(new BufferedOutputStream(\n\t\t\t\t\t\t\tserializer.getOutputStream()));\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\ttransformer.transform(source, result);\n\t\t\t\t\n\t\t\t\tif(ihn) {\n\t\t\t\t\tbytesIn = buffer.toByteArray();\n\t\t\t\t\tbuffer.reset();\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tfinally {\n\t\t\tbytesIn = null;\n\t\t\ttry {\n\t\t\t\tbuffer.close();\n\t\t\t}\n\t\t\tcatch(IOException e) {\n\t\t\t\t// do nothing\n\t\t\t}\n\t\t}\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/XMLFilterProcessor.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n/*\n * $Id: XMLFilterProcessor.java,v 1.4 2010/09/05 15:54:43 mwhite14850 Exp $\n * Copyright (C) 2009 Scott Martin (http://www.coffeeblack.org/contact/)\n */\npackage opennlp.ccgbank;\n\nimport java.io.FileNotFoundException;\nimport java.io.IOException;\nimport java.util.*;\n\nimport javax.xml.transform.ErrorListener;\nimport javax.xml.transform.Templates;\nimport javax.xml.transform.TransformerConfigurationException;\nimport javax.xml.transform.TransformerException;\nimport javax.xml.transform.sax.SAXTransformerFactory;\n\nimport org.apache.tools.ant.BuildException;\nimport org.xml.sax.ErrorHandler;\nimport org.xml.sax.InputSource;\nimport org.xml.sax.SAXException;\nimport org.xml.sax.XMLFilter;\nimport org.xml.sax.helpers.XMLReaderFactory;\n\n\n/**\n * An implementation of {@link XSLTProcessor} that performs transformations\n * using an {@link XMLFilter}. This particular implementation uses the XSLTC\n * compiler distributed with Apache's Xalan in order to avoid the known \n * problems with re-using {@link XMLFilter}.\n * @see Apache Xalan\n * @author Scott Martin\n * @version $Revision: 1.4 $\n */\nclass XMLFilterProcessor extends XSLTProcessor {\n\t\n\t// nb: this could be in the super class\n\tList templates = null;\n\t\n\tXMLFilter filter;\n\tErrorHandler errorHandler;\n\t\n\tstatic final String\n\t\tXSLT_KEY = \"javax.xml.transform.TransformerFactory\",\n\t\tXSLTC_VALUE = \"org.apache.xalan.xsltc.trax.TransformerFactoryImpl\";\n\t\n\tXMLFilterProcessor(ErrorListener errorListener, ErrorHandler errorHandler) {\n\t\tsuper(errorListener);\n\t\tthis.errorHandler = errorHandler;\n\t}\n\t\n\tSAXTransformerFactory newTransformerFactory() {\n\t\t// TODO try using xsltc (seems to yield hard-to-trace bugs at the moment)\n\t\t//System.setProperty(XSLT_KEY, XSLTC_VALUE);\n\t\treturn super.newTransformerFactory();\n\t}\n\t\n\t/* (non-Javadoc)\n\t * @see opennlp.ccgbank.XSLTProcessor#process(java.io.File)\n\t */\n\t@Override\n\tvoid process(InputSource inputSource) throws IOException,SAXException,\n\t\t\tTransformerException {\n\t\t// TODO figure out how to re-use filter without breaking :(\n\t\t// make new filter each time\n\t\tfilter = makeFilter(taskTemplatesList);\n\t\tfilter.setContentHandler(serializer.asContentHandler());\n\t\tfilter.parse(inputSource);\n\t}\n\t\n\t/**\n\t * Makes a filter from a single xsltProcessors object.\n\t * @see #makeFilter(List)\n\t */\n\tXMLFilter makeFilter(CCGBankTaskTemplates templates)\n\t\tthrows FileNotFoundException,SAXException,\n\t\t\tTransformerConfigurationException {\n\t\treturn makeFilter(Collections.singletonList(templates));\n\t}\n\t\n\t\n\t/**\n\t * Makes a filter from a series of xsltProcessors that applies those \n\t * templates in order.\n\t * @param templateList The series of xsltProcessors used to construct the\n\t * filter.\n\t * @throws BuildException If no xsltProcessors are specified.\n\t */\n\tXMLFilter makeFilter(List templateList)\n\t\tthrows FileNotFoundException,SAXException,\n\t\t\t TransformerConfigurationException {\n\n\t\t// make templates\n\t\tif(templates == null) {\n\t\t\ttemplates = makeTemplates(taskTemplatesList);\n\t\t}\n\t\t\n\t\t// assemble list of xslt templates into a filter\n\t\tXMLFilter currentFilter = null, previousFilter = null;\n\t\tfor (Templates t : templates) {\n\t\t\tcurrentFilter = transformerFactory.newXMLFilter(t);\n\t\t\tcurrentFilter.setErrorHandler(errorHandler);\n\t\t\t\n\t\t\tif(previousFilter == null) { // it's the first one\n\t\t\t\tcurrentFilter.setParent(\n\t\t\t\t\t\tXMLReaderFactory.createXMLReader());\t\t\t\t\t\n\t\t\t}\n\t\t\telse {\n\t\t\t\tcurrentFilter.setParent(previousFilter);\n\t\t\t}\n\t\t\t\n\t\t\tpreviousFilter = currentFilter;\n\t\t}\n\t\t\n\t\tif(currentFilter == null ) {\n\t\t\tthrow new IllegalArgumentException(\"no templates specified\");\n\t\t}\n\t\t\n\t\tcurrentFilter.setErrorHandler(errorHandler);\n\t\tcurrentFilter.setFeature(\"http://xml.org/sax/features/namespace-prefixes\", true);\n\t\t\n\t\treturn currentFilter;\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/XSLTProcessor.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n/*\n * $Id: XSLTProcessor.java,v 1.4 2010/09/05 15:54:43 mwhite14850 Exp $\n * Copyright (C) 2009 Scott Martin (http://www.coffeeblack.org/contact/)\n */\npackage opennlp.ccgbank;\n\nimport java.io.BufferedInputStream;\nimport java.io.BufferedOutputStream;\nimport java.io.File;\nimport java.io.FileInputStream;\nimport java.io.FileNotFoundException;\nimport java.io.FileOutputStream;\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Properties;\n\nimport javax.xml.transform.ErrorListener;\nimport javax.xml.transform.OutputKeys;\nimport javax.xml.transform.Templates;\nimport javax.xml.transform.TransformerConfigurationException;\nimport javax.xml.transform.TransformerException;\nimport javax.xml.transform.TransformerFactory;\nimport javax.xml.transform.sax.SAXResult;\nimport javax.xml.transform.sax.SAXSource;\nimport javax.xml.transform.sax.SAXTransformerFactory;\nimport javax.xml.transform.stream.StreamSource;\n\nimport org.apache.tools.ant.BuildException;\nimport org.apache.xml.serializer.OutputPropertiesFactory;\nimport org.apache.xml.serializer.Serializer;\nimport org.apache.xml.serializer.SerializerFactory;\nimport org.xml.sax.InputSource;\nimport org.xml.sax.SAXException;\n\n\n/**\n * Abstract base class for XSLT processing. Templates are added to a processor,\n * then {@link #process(InputSource)} is called for each input source. \n * Subclasses will implement different processing strategies for transforming\n * XML using a series of XSL templates.\n * @author Scott Martin\n * @version $Revision: 1.4 $\n */\nabstract class XSLTProcessor {\n\t\n\tSAXTransformerFactory transformerFactory = newTransformerFactory();\n\t\n\tstatic final Properties xmlProperties\n\t\t= OutputPropertiesFactory.getDefaultMethodProperties(\"xml\");\n\t\n\tstatic {\n\t\txmlProperties.setProperty(OutputKeys.INDENT, \"yes\");\n\t\txmlProperties.setProperty(\n\t\t\t\t\"{http://xml.apache.org/xalan}indent-amount\", \"2\");\n\t}\n\t\n\tList taskTemplatesList\n\t\t= new ArrayList();\n\tSerializer serializer = SerializerFactory.getSerializer(\n\t\t\tXSLTProcessor.xmlProperties);\n\tErrorListener errorListener;\t\n\t\n\tXSLTProcessor(ErrorListener errorListener) {\n\t\tthis.errorListener = errorListener;\n\t\ttransformerFactory.setErrorListener(errorListener);\n\t}\n\t\n\tboolean addAllTemplates(List templateList) {\n\t\tboolean b = false;\n\t\t\n\t\tfor(CCGBankTaskTemplates t : templateList) {\n\t\t\tb |= addTemplates(t);\n\t\t}\n\t\t\n\t\treturn b;\n\t}\n\t\n\tboolean addTemplates(CCGBankTaskTemplates taskTemplates) {\n\t\treturn taskTemplatesList.add(taskTemplates);\n\t}\n\t\n\t\n\t/**\n\t * Processes an input source, applying each of the templates specified \n\t * using {@link #addTemplates(CCGBankTaskTemplates)} or\n\t * {@link #addAllTemplates(List)}. Subclasses will actually implement this\n\t * method.\n\t * @param inputSource The input source to which the templates will be\n\t * applied.\n\t * @throws IOException If a problem reading or writing occurs.\n\t * @throws SAXException If a subclass uses a SAX processor and there is a \n\t * problem with it.\n\t * @throws TransformerException If a subclass uses a processor that causes\n\t * a transformer problem.\n\t */\n\tabstract void process(InputSource inputSource)\n\t\tthrows IOException,SAXException,TransformerException;\n\t\n\tvoid setTarget(File file) throws FileNotFoundException {\n\t\tserializer.setOutputStream(\n\t\t\tnew BufferedOutputStream(new FileOutputStream(file)));\n\t\t// ensure output properties set (shouldn't really be nec!)\n\t\tserializer.setOutputFormat(xmlProperties);\n\t}\n\t\n\t/**\n\t * Resets the serializer, if resetting is possible. If not, re-creates the\n\t * serializer.\n\t */\n\tvoid resetSerializer() {\n\t\tif(!serializer.reset()) {\n\t \tserializer // create new unless re-useable\n\t \t\t= SerializerFactory.getSerializer(xmlProperties);\n\t }\n\t}\n\t\n\tSAXTransformerFactory newTransformerFactory() {\n\t\tSAXTransformerFactory tf\n\t\t\t= (SAXTransformerFactory)TransformerFactory.newInstance();\n\t\t\n\t\tif(!tf.getFeature(SAXSource.FEATURE)) {\n\t\t\tthrow new IllegalStateException(\n\t\t\t\t\"SAX transformer factory does not support SAXSource\");\n\t\t}\n\t\tif(!tf.getFeature(SAXResult.FEATURE)) {\n\t\t\tthrow new IllegalStateException(\n\t\t\t\t\"SAX transformer factory does not support SAXResult\");\n\t\t}\n\t\t\n\t\treturn tf;\n\t}\n\t\n\t/**\n\t * Makes a list of templates from a single xsltProcessors object.\n\t * @see #makeTemplates(List)\n\t */\n\tList makeTemplates(CCGBankTaskTemplates taskTemplates) \n\t\t\tthrows FileNotFoundException,TransformerConfigurationException {\n\t\treturn makeTemplates(Collections.singletonList(taskTemplates));\n\t}\n\t\n\t\n\t/**\n\t * Makes a list of templates from a series of xsltProcessors that applies those xsltProcessors\n\t * in order.\n\t * @param templateList The series of xsltProcessors used to construct the\n\t * filter.\n\t * @throws BuildException If no xsltProcessors are specified.\n\t */\n\tList makeTemplates(List templateList)\n\t\t\tthrows FileNotFoundException,TransformerConfigurationException {\n\t\tList l = new ArrayList();\n\t\t\n\t\tfor(CCGBankTaskTemplates taskTemplates : templateList) {\n\t\t\tfor(File f : taskTemplates) {\n\t\t\t\tStreamSource ss = new StreamSource(\n\t\t\t\t\tnew BufferedInputStream(new FileInputStream(f)));\n\t\t\t\tss.setSystemId(f);\n\t\t\t\t\n\t\t\t\tl.add(transformerFactory.newTemplates(ss));\t\t\t\t\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn Collections.unmodifiableList(l);\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/ccgbank.properties", "content": "ccgbanktask=opennlp.ccgbank.CCGBankTask\nsources=opennlp.ccgbank.CCGBankTaskSources\ntemplates=opennlp.ccgbank.CCGBankTaskTemplates\nconvert=opennlp.ccgbank.CCGBankConvert\nextract=opennlp.ccgbank.CCGBankExtract\ntestbed=opennlp.ccgbank.CCGBankTaskTestbed" }, { "path": "src/opennlp/ccgbank/convert/ApposTally.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Class which stores info about punctuations\npackage opennlp.ccgbank.convert;\n\nimport java.io.BufferedReader;\nimport java.io.FileReader;\nimport java.io.IOException;\nimport java.util.ArrayList;\n\npublic class ApposTally {\n\n\t//Operation performed\n\t//private String oper = \"\";\n\n\t//Dest dir\n\t//private String destDir = \"\";\n\n\t//Sentence id\n\tprivate String id = \"\";\n\n\t//Context of a comma \n\t//private String cont = \"\";\n\n\t//Status whether comma is balanced or not\n\tprivate String balStatus = \"\";\n\n\t//Intervening lexical mtl\n\tprivate String lexMtl = \"\";\n\n\t//Cue analysis result\n\t//private String cueResult = \"\";\n\n\t//Features for appos vs conj identification\n\tprivate ArrayList featInfo = new ArrayList();\n\n\t//Heads of np\n\tprivate ArrayList headInfo1 = new ArrayList();\n\n\t//Heads of appositive\n\tprivate ArrayList headInfo2 = new ArrayList();\n\n\t//Cues\n\tprivate static ArrayList cueList = new ArrayList();\n\n\t//Proc which opens the cue file\n\tpublic void openCueFile(String fileName) throws IOException {\n\n\t\tBufferedReader br = new BufferedReader(new FileReader(fileName));\n\t\tString line = \"\";\n\n\t\twhile ((line = br.readLine()) != null) {\n\t\t\tif (!cueList.contains(line))\n\t\t\t\tcueList.add(line);\n\t\t\t//System.out.println(line);\n\t\t}\n\t\tbr.close();\n\t}\n\n\t//Proc which traps, stores id of each sentence in a global var \n\tpublic String storeId(String x) {\n\t\tid = \"\";\n\t\tid = x;\n\t\t//System.out.println(id);\n\t\treturn null;\n\t}\n\n\t//Proc which reinitializes all vars when a new comma is encountered\n\tpublic void flushVars() {\n\t\tbalStatus = \"conj\";\n\t\tlexMtl = \"\";\n\t\theadInfo1.clear();\n\t\theadInfo2.clear();\n\t\t//cueResult = \"\";\n\t\tfeatInfo.clear();\n\t}\n\n\t//Proc which stores whether comma is balanced\n\tpublic void storeBalance(String status) {\n\n\t\tbalStatus = status;\n\n\t}\n\n\t//Proc which returns balance status\n\tpublic String getBalance() {\n\n\t\treturn balStatus;\n\n\t}\n\n\t//Proc which returns capitalized string of balance status\n\tpublic String getCaps(String x) {\n\n\t\treturn x.toUpperCase();\n\n\t}\n\n\t//Proc which stores intervening lexical mtl\n\tpublic void storeLex(String word, String pos) {\n\n\t\tString info = \"\";\n\n\t\tif (pos.equals(\"X\"))\n\t\t\tinfo = word;\n\t\telse\n\t\t\tinfo = word + \"/\" + pos;\n\n\t\tlexMtl = lexMtl + \" \" + info;\n\n\t}\n\n\t//Proc which prints intervening lexical mtl\n\tpublic void printLex() {\n\n\t\tlexMtl = id + \" \" + lexMtl;\n\t\t//System.out.println(lexMtl+'\\n');\n\t\tlexMtl = \"\";\n\t}\n\n\t//Proc which stores head of np1\n\tpublic void storeHead(String word, String pos, int npNo) {\n\n\t\tString info = \"\";\n\n\t\t//Eliding the distinction b/w sing&plural nouns\n\t\tif (pos.equals(\"NNS\"))\n\t\t\tpos = \"NN\";\n\t\tif (pos.equals(\"NNPS\"))\n\t\t\tpos = \"NNP\";\n\n\t\tinfo = word + \"/\" + pos;\n\n\t\tif (npNo == 1)\n\t\t\theadInfo1.add(info);\n\t\telse\n\t\t\theadInfo2.add(info);\n\n\t}\n\n\t//Heuristic2: Cue based analysis\n\tpublic String cueAnalysis() {\n\n\t\t//String[] np = lexMtl.split(\"X\");\n\t\tString[] sent = lexMtl.split(\"X\");\n\n\t\t//Sift out possessed in genitive constr\n\t\tif (sent[0].contains(\"'s/POS\") && headInfo1.size() > 0)\n\t\t\theadInfo1.remove(0);\n\t\tif (sent[1].contains(\"'s/POS\") && headInfo2.size() > 0)\n\t\t\theadInfo2.remove(0);\n\n\t\tString res = \"\";\n\n\t\tint flag = 0;\n\n\t\tString np1head[] = new String[2];\n\n\t\tif (headInfo1.size() == 0 || headInfo2.size() == 0) {\n\t\t\t//System.out.println(id+\": \"+lexMtl+'\\n');\n\t\t\theadInfo1.add(\"X1/ECK\");\n\t\t\tnp1head = (headInfo1.get(headInfo1.size() - 1)).split(\"/\");\n\n\t\t} else\n\t\t\tnp1head = (headInfo1.get(headInfo1.size() - 1)).split(\"/\");\n\n\t\t//Comparing the heads of np1 & np2\n\t\tfor (String x : headInfo2) {\n\t\t\tString[] np2head = x.split(\"/\");\n\t\t\tif (np2head[1].equals(np1head[1]))\n\t\t\t\tflag = 1;\n\t\t\telse {\n\t\t\t\tflag = 0;\n\t\t\t\tbreak;\n\t\t\t}\n\t\t}\n\n\t\t//Avoid place names: ie like c\n\t\tif (flag == 1 && headInfo1.size() == 1 && headInfo2.size() == 1)\n\t\t\tflag = 0;\n\n\t\tif (flag == 1 && sent[0].contains(\"/CC\") && !sent[0].contains(\",\"))\n\t\t\tflag = 0;\n\n\t\t//Like nps together smacks of a conjunction\n\t\tif (flag == 0 && sent[1].contains(\"/CC\")) {\n\t\t\theadInfo2.add(\"X2/ECK\");\n\t\t\tString np2head[] = (headInfo2.get(0)).split(\"/\");\n\t\t\tif (np2head[1].equals(np1head[1]))\n\t\t\t\tflag = 1;\n\t\t}\n\n\t\t//if(flag==0 && sent[1].contains(\",/, and/CC\"))flag=1;\n\n\t\t/*Stub to print a particular sentence\n\t\t if(id.equals(\"ID=wsj_0012.3\")){\n\t\t System.out.println(\"Flag: \"+flag);\n\t\t System.out.println(headInfo1);\n\t\t System.out.println(headInfo2);\n\t\t }*/\n\n\t\tif (flag == 1)\n\t\t\tres = \"conj\";\n\t\telse\n\t\t\tres = \"appos\";\n\n\t\t//if(unit.contains(\"/CD\") || unit.contains(\"/POS\") || unit.contains(\"/IN\") || unit.contains(\"/DT\")|| unit.contains(\"PRP$\") || head1==true || cue==true)featInfo.add(\"appos\");\n\n\t\treturn res;\n\n\t}\n\n}\n" }, { "path": "src/opennlp/ccgbank/convert/DiscrCheck.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Class which stores the context of punctuations\npackage opennlp.ccgbank.convert;\n\n\n\npublic class DiscrCheck{\n\n\t\t//Sentence id\n\t\tprivate String id=\"\";\n\n\t //Proc which traps, stores id of each sentence in a global var\n public String storeId(String x){\n\n\t\t\t\tid=\"\";\n\t\t\t\tid=x;\n\t\t\t\treturn null;\n }\n\n\t\t public void checkCatDiscr(String cat,String cat0,String lex){\n\n\t\t\t\t //Comparing discrepancies between cat0 & cat1\n\t\t\t\t //Just a check. Should be commented out in the final version\n\t\t\t\t \n\t\t\t\t String catA=cat0;\n\t\t\t\t String catB=cat;\n\t\t\t\t \n\t\t\t\t catA=catA.toLowerCase();\n\t\t\t\t catB=catB.replaceAll(\"[0-9]\",\"\");\n\t\t\t\t catB=catB.replaceAll(\"_\",\"\");\n\t\t\t\t \n\t\t\t\t if(!catA.equals(catB) && !cat0.contains(\"nb\"))\n\t\t\t\t\t\t System.out.println(id+\": \"+cat0+\"***\"+cat+\" - \"+lex);\n\t\t\t\t \n\t\t }\n\t\t\n}\n\n\n\n" }, { "path": "src/opennlp/ccgbank/convert/GenChal11Adjuster.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2011 Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccgbank.convert;\r\n\nimport java.util.*;\n\r\n/**\r\n * Utility class for adjusting Generation Challenges 2011 outputs.\n * Strings are lowercased, named entities and hyphenated words are split, \n * and dollar sign and numbers are transposed. \r\n */ \r\npublic class GenChal11Adjuster {\r\n\r\n\t/** Returns the adjusted text string . */\n\tpublic String getAdjustedString(String text) {\n\t\t// lowercase and split\n\t\tString[] tokens = text.toLowerCase().split(\"\\\\s+\");\n\t\t// swap dollar signs\n\t\tfor (int i=0; i < tokens.length-1; i++) {\n\t\t\tif (tokens[i+1].equals(\"$\")) {\n\t\t\t\ttry {\n\t\t\t\t\t// check for preceding number token\n\t\t\t\t\tDouble.parseDouble(tokens[i]);\n\t\t\t\t\t// swap, skip\n\t\t\t\t\tString num = tokens[i];\n\t\t\t\t\ttokens[i] = tokens[i+1];\n\t\t\t\t\ttokens[i+1] = num;\n\t\t\t\t\ti++;\n\t\t\t\t}\n\t\t\t\tcatch (NumberFormatException e) {}\n\t\t\t}\n\t\t}\n\t\t// split NEs and hyphenated words\n\t\tList splitTokens = new ArrayList(tokens.length*2);\n\t\tfor (String token : tokens) {\n\t\t\tString[] tokenSplits = token.replace(\"-\",\" - \").split(\"[_ ]\");\n\t\t\tfor (String s : tokenSplits)\n\t\t\t\tsplitTokens.add(s);\n\t\t}\n\t\t// join\n\t\tStringBuffer retval = new StringBuffer();\n\t\tfor (int i=0; i < splitTokens.size()-1; i++) {\n\t\t\tretval.append(splitTokens.get(i));\n\t\t\tretval.append(' ');\n\t\t}\n\t\tretval.append(splitTokens.get(splitTokens.size()-1));\n\t\t// done\n\t\treturn retval.toString();\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccgbank/convert/GenConjRule.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Java class which adds brackets, stores arg1 position for inferConjRules.xsl,normConjRulesId, normTreenodeId.xsl\n\npackage opennlp.ccgbank.convert;\n\nimport java.util.ArrayList;\nimport java.util.Hashtable;\n//import java.util.StringTokenizer;\n\npublic class GenConjRule {\n\n\t//The largest current id\n\tprivate static int globalId = 0;\n\n\t//The store of ids\n\tprivate static Hashtable idTally = new Hashtable();\n\n\t//Dollar status of res, arg1 & arg2\n\tprivate static ArrayList dollarStatus = new ArrayList();\n\n\t//Final result of dollar status calcs\n\tprivate static String ds = \"No_Dollar\";\n\n\t//Add brackets to complex categories\n\tpublic String addParen(String str) {\n\n\t\tif (str.contains(\"\\\\\") || str.contains(\"/\"))\n\t\t\tstr = \"(\" + str + \")\";\n\t\treturn str;\n\n\t}\n\n\t//Add _conj to the result category\n\tpublic String modRes(String str) {\n\t\t//StringTokenizer st = new StringTokenizer(str, \"[]\");\n\t\tstr = str + \"_conj\";\n\t\treturn str;\n\t}\n\n\t//Procedure which cleans the unary rule result\n\tpublic String getConjRes(String x) {\n\t\tx = x.replaceAll(\"_conj\", \"\");\n\t\tx = x.replaceAll(\"_[0-9]\", \"\");\n\t\t//System.out.println(x);\n\t\treturn x;\n\t}\n\n\t//Normalizes id of input category\n\tpublic String normId(String oldId, String oldInhId, String cat) {\n\n\t\tString newId = \"\";\n\t\t//int choice = 0;\n\n\t\t//Switch for usage between Treenode & Leafnode Id normalization \n\t\tif (oldId.length() > 0)\n\t\t\tcat = cat + \"_\" + oldId;\n\t\tif (oldInhId.length() > 0)\n\t\t\tcat = cat + \"_\" + oldInhId;\n\n\t\tcat = cat.trim();\n\n\t\t//Normalization condition\n\t\tif (!idTally.containsKey(cat)) {\n\t\t\tglobalId++;\n\t\t\tnewId = Integer.toString(globalId);\n\t\t\tidTally.put(cat, newId);\n\t\t}\n\n\t\tnewId = (String) idTally.get(cat);\n\n\t\treturn newId;\n\t}\n\n\t//Initialization of idTally & globalId before start of a new conj rule \n\tpublic String globalInit() {\n\t\tglobalId = 0;\n\t\tidTally.clear();\n\t\treturn null;\n\t}\n\n\t//Initialization of idTally before each of Result,arg1 & arg2 is added \n\tpublic String localInit() {\n\t\tidTally.clear();\n\t\treturn null;\n\t}\n\n\t//Calculation of dollarStatus before start of a new conj rule \n\n\t//Initialization of dollarStatus before start of a new conj rule \n\tpublic String dsInit() {\n\t\t//System.out.println(dollarStatus);\n\t\tdollarStatus.clear();\n\t\tds = \"No_Dollar\";\n\t\treturn null;\n\t}\n\n\t//Store dollar status of res, arg1 & arg2\n\tpublic String storeDollarStatus(String type) {\n\t\ttype = type.trim();\n\t\tdollarStatus.add(type);\n\t\t/*System.out.println('\\n');\n\t\t System.out.println(\"Insertion of: \"+type);\n\t\t System.out.println(dollarStatus);\n\t\t System.out.println('\\n');*/\n\t\treturn \"null\";\n\t}\n\n\tpublic String dsCalc() {\n\n\t\t//System.out.println(dollarStatus);\n\t\tif (dollarStatus.size() == 3)\n\t\t\tds = \"Dollar\";\n\n\t\t/*System.out.println('\\n');\n\t\t System.out.println(\"Retrieval\");\n\t\t System.out.println(dollarStatus);\n\t\t System.out.println('\\n');*/\n\n\t\treturn null;\n\t}\n\n\t//Get dollar status of conjunct\n\tpublic String getDollarStatus() {\n\t\treturn ds;\n\t}\n\n\t//Function invoked by invertedDirSpComma.xsl\n\tpublic String getglobalId() {\n\t\tglobalId++;\n\t\treturn Integer.toString(globalId);\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/convert/InfoHelper.java", "content": "///////////////////////////////////////////////////////////////////////////////\n//Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n//\n//This library is free software; you can redistribute it and/or\n//modify it under the terms of the GNU Lesser General Public\n//License as published by the Free Software Foundation; either\n//version 2.1 of the License, or (at your option) any later version.\n//\n//This library is distributed in the hope that it will be useful,\n//but WITHOUT ANY WARRANTY; without even the implied warranty of\n//MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n//GNU Lesser General Public License for more details.\n//\n//You should have received a copy of the GNU Lesser General Public\n//License along with this program; if not, write to the Free Software\n//Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Java class which helps info extr from PTB\n\npackage opennlp.ccgbank.convert;\n\n//import opennlp.ccg.lexicon.*;\n\nimport java.util.*;\nimport java.io.*;\n\npublic class InfoHelper{\n\t\n\t/** CCG terminal & non-terminal nos.. */\n\tprivate int termNo=0;private int ntNo=500;\n\t\n\t/** BBN-info. */\n\tprivate static Hashtable bbnInfo=new Hashtable();\n\tprivate static Hashtable> bbnSpans=new Hashtable>();\n\tprivate static ArrayListbbnClasses=new ArrayList();\n\t\n\t/** Quote info. */\n\tprivate static Hashtable quoteInfo=new Hashtable();\n\t\n\t/** PTB aux info viz. SBJ, FN_Tag & TPC annotation **/\n\tprivate static Hashtable sbjInfo=new Hashtable();\n\tprivate static Hashtable fntagInfo=new Hashtable();\n\tprivate static Hashtable tpcInfo=new Hashtable();\n\t\n\t/** Treenode info. */\n\tprivate static Hashtable treeInfo=new Hashtable();\n\t\n\t/** Directory where aux file and BBN NE info is stored. */\n\tstatic File auxFileDirectory = null, bbnAuxDirectory = null;\n\t\n\t/** Store aux file directories. */\n\tpublic static void init(File auxDir, File bbnAuxDir) {\n\t\tInfoHelper.auxFileDirectory = auxDir;\n\t\tInfoHelper.bbnAuxDirectory = bbnAuxDir;\n\t}\n\t\n\t/** Read BBN NE aux file corresponding to the WSJ Section provided as argument. */\n\tpublic static void readBBNAuxfiles(String sect){\n\t\t\n\t\tString bbnAuxFile=bbnAuxDirectory+\"/\"+\"bbn-ccg\"+sect+\".aux\";\n\t\t\n\t\ttry{\n\t\t\t\n\t\t\tif(new File(bbnAuxFile).exists()){\n\t\t\t\tBufferedReader inp= new BufferedReader(new FileReader(bbnAuxFile));\n\t\t\t\tSystem.out.println(\"Reading in BBN aux file: \"+bbnAuxFile);\n\t\t\t\tString line=\"\";\n\t\t\t\twhile((line=inp.readLine())!=null ){\n\t\t\t\t\tif(line.length()==0)continue;\n\t\t\t\t\tString bbn[]=line.trim().split(\" \");\n\t\t\t\t\tString sentId=bbn[0];\n\t\t\t\t\tString span=bbn[1]+\",\"+bbn[2];\n\t\t\t\t\tString key=sentId+\" \"+span;\n\t\t\t\t\tString bbnData=\"\";\n\t\t\t\t\t\n\t\t\t\t\tfor(int i=3;i spanList=new ArrayList();\n\t\t\t\t\tif(!bbnSpans.containsKey(sentId))\n\t\t\t\t\t\tbbnSpans.put(sentId,spanList);\n\t\t\t\t\tspanList=bbnSpans.get(sentId);\n\t\t\t\t\tspanList.add(span);\n\t\t\t\t\tbbnSpans.put(sentId,spanList);\n\t\t\t\t\t\n\t\t\t\t}\n\t\t\t\tinp.close();\n\t\t\t}\n\t\t}\n\t\tcatch(IOException e){\n\t\t\tSystem.out.println(\"Error reading: \"+bbnAuxFile);\n\t\t}\n\t}\n\t\n\t/** Read quotes aux file corresponding to the WSJ Section provided as argument. */\n\tpublic static void readQuoteAuxfiles(String sect){\n\t\t\n\t\tString quoteAuxFile=auxFileDirectory+\"/\"+\"aux-quotes-\"+sect+\".txt\";\n\t\t\n\t\ttry{\n\t\t\t\n\t\t\tif(new File(quoteAuxFile).exists()){\n\t\t\t\tSystem.out.println(\"Reading in quotes aux File: \"+quoteAuxFile);\n\t\t\t\tString line=\"\";\n\t\t\t\tBufferedReader inp= new BufferedReader(new FileReader(quoteAuxFile));\n\t\t\t\twhile((line=inp.readLine())!=null ){\n\t\t\t\t\t\n\t\t\t\t\tif(line.length()==0)continue;\n\t\t\t\t\tString quoteData[]=line.trim().split(\" \");\n\t\t\t\t\tString sentId=quoteData[0];\n\t\t\t\t\tString span=quoteData[1]+\",\"+quoteData[2];\n\t\t\t\t\tString key=sentId+\" \"+span;\n\t\t\t\t\tString quotedText=\"\";\n\t\t\t\t\t\n\t\t\t\t\tfor(int i=3;i treeCont=new ArrayList();\n\tpublic boolean checkTreeInfo(String header,String ntId,int numCats){\n\t\t\n\t\tString key=header+\" \"+ntId;\n\t\t\n\t\tif(treeInfo.containsKey(key)){\n\t\t\tString treeCats=treeInfo.get(key);\n\t\t\tString[] x=treeCats.split(\",\");\n\t\t\t\n\t\t\tif(numCats==x.length)\n\t\t\t\treturn true;\n\t\t\telse return false;\n\t\t}\n\t\telse return false;\n\t}\n\t\n\t/** Retrieve treenode info. */\n\tpublic String getTreeInfo(String key){\n\t\t\n\t\tString retVal=\"\";\n\t\ttreeCont=new ArrayList();\n\t\t\n\t\tif(treeInfo.containsKey(key)){\n\t\t\tretVal=treeInfo.get(key);\n\t\t\tString[]temp=retVal.split(\",\");\n\t\t\t\n\t\t\tfor(String x:temp)\n\t\t\t\ttreeCont.add(x);\n\t\t}\n\t\t\n\t\treturn retVal;\n\t}\n\t\n\t/** Procedure which gives back id of a particular cat. */\n\tpublic String getTreeId(){\n\t\t\n\t\tString retVal=\"\";\n\t\t\n\t\tif(treeCont.size()>0){\n\t\t\tretVal=treeCont.get(0);\n\t\t\ttreeCont.remove(0);\n\t\t\tString x[]=retVal.split(\"_\");\n\t\t\tretVal=x[1];\n\t\t\tif(x.length==3)\n\t\t\t\tretVal=\"M_\"+retVal;\n\t\t}\n\t\t\n\t\treturn retVal;\n\t}\n\t\n\t/** Procedure which gives slash of combination. */\n\tpublic String getTreeSlash(){\n\t\t\n\t\tString retVal=\"\";\n\t\tif(treeCont.size()>0){\n\t\t\tString slash=treeCont.get(0);\n\t\t\tString mode=\"\";\n\t\t\t\n\t\t\tif(slash.length()>1)\n\t\t\t\tmode=Character.toString(slash.charAt(1));\n\t\t\telse if(slash.startsWith(\"/\"))\n\t\t\t\tmode=\">\";\n\t\t\telse if(slash.startsWith(\"\\\\\"))\n\t\t\t\tmode=\"<\";\n\t\t\ttreeCont.remove(0);\n\t\t\tretVal=Character.toString(slash.charAt(0));\n\t\t\tretVal=retVal+\"_\"+mode;\n\t\t\t\n\t\t}\n\t\t\n\t\treturn retVal;\n\t}\n\t\n\t/** Retrieve BBN class for lexical items for use in the Leafnodes. */\n\tpublic String getBBNClass(String header,String lex,String pos,String cat,int nodeInd){\n\t\t\n\t\tString retVal=\"\";\n\t\tString semClass=\"\";\n\t\t\n\t\t//Check and exit if the pos is not relevant\n\t\tboolean relFlag=false;\n\t\t\n\t\tif(cat.matches(\"pp\\\\[[a-z]+\\\\]_~2/np_2\"))\n\t\t\treturn \"\";\n\t\t\n\t\tif (pos.startsWith(\"NN\") || pos.startsWith(\"RB\") || pos.startsWith(\"JJ\") || pos.startsWith(\"VB\") ||pos.equals(\"CD\") || lex.equals(\"%\") || pos.equals(\"$\"))\n\t\t\trelFlag=true;\n\t\t\n\t\tif(!relFlag)\n\t\t\treturn \"\";\n\t\t\n\t\tString sentId=header.replaceFirst(\"ID=\",\"\");\n\t\tArrayList spanList=new ArrayList();\n\t\t\n\t\tString key=sentId+\" \"+Integer.toString(nodeInd)+\",\"+Integer.toString(nodeInd);\n\t\tif(bbnInfo.containsKey(key)){\n\t\t\t\n\t\t\tString bbnData=bbnInfo.get(key);\n\t\t\tString info[]=bbnData.split(\" \");\n\t\t\t\n\t\t\tif(info.length>=2){\n\t\t\t\t\n\t\t\t\tsemClass=getCleanClass(info[0]);\n\t\t\t\tretVal=(classReplace(semClass,lex,info[1])).trim();\n\t\t\t\tif(retVal.length()>0 && !bbnClasses.contains(retVal))\n\t\t\t\t\tbbnClasses.add(retVal);\n\t\t\t}\n\t\t\t\n\t\t\treturn retVal;\n\t\t}\n\t\telse if(bbnSpans.containsKey(sentId))\n\t\t\tspanList=bbnSpans.get(sentId);\n\t\t\n\t\tfor(String span: spanList){\n\t\t\t\n\t\t\tString inds[]=span.split(\",\");\n\t\t\tif(inds.length!=2)continue;\n\t\t\t\n\t\t\tif(!inds[0].matches(\"[0-9]+\") || !inds[1].matches(\"[0-9]+\") || inds[0].equals(\"NA\") || inds[1].equals(\"NA\"))\n\t\t\t\tcontinue;\n\t\t\t\n\t\t\tint ind1=Integer.parseInt(inds[0]);\n\t\t\tint ind2=Integer.parseInt(inds[1]);\n\t\t\tif(nodeInd >=ind1 && nodeInd <=ind2){\n\t\t\t\t\n\t\t\t\tkey=sentId+\" \"+span;\n\t\t\t\t\n\t\t\t\tString bbnData=bbnInfo.get(key);\n\t\t\t\tString info[]=bbnData.split(\" \");\n\t\t\t\t\n\t\t\t\tif(info.length==0)\n\t\t\t\t\tcontinue;\n\t\t\t\t\n\t\t\t\tsemClass=getCleanClass(info[0]);\n\t\t\t\t\n\t\t\t\t//Compile a list of acceptable classes\n\t\t\t\tboolean accClasses=false;\n\t\t\t\t\n\t\t\t\tif (!semClass.startsWith(\"DATE\") && !semClass.startsWith(\"TIME\") && !semClass.startsWith(\"ORDINAL\") && !semClass.startsWith(\"QUANTITY\") && !semClass.startsWith(\"PERCENT\") && !semClass.startsWith(\"MONEY\")) \n\t\t\t\t\taccClasses=true;\n\t\t\t\t\n\t\t\t\tif (accClasses || pos.startsWith(\"NN\") || pos.equals(\"CD\") || lex.equals(\"%\") || pos.equals(\"$\")){\n\t\t\t\t\tString wordBit=\"\";\n\t\t\t\t\tint relInd=nodeInd-ind1+1;\n\t\t\t\t\t\n\t\t\t\t\tif(relInd=0)\n\t\t\t\t\t\twordBit=info[(nodeInd-ind1+1)];\n\t\t\t\t\t\n\t\t\t\t\tretVal=classReplace(semClass,lex,wordBit);\n\t\t\t\t\tif(retVal.length()>0 && !bbnClasses.contains(retVal))\n\t\t\t\t\t\tbbnClasses.add(retVal);\n\t\t\t\t\tbreak;\n\t\t\t\t}\n\t\t\t\t\n\t\t\t}\n\t\t}\n\t\t\n\t\treturn retVal;\n\t\t\n\t}\n\t\n\t/** Perform semantic replacement over relevant parts of the part. */\n\tpublic String classReplace(String semClass,String lex,String wordBit){\n\t\t\n\t\tString retVal=\"\";\n\t\t\n\t\t/*CITY-based classes\n\t\tif(wordBit.equals(lex))\n\t\t\tretVal=semClass;\n\t\telse if(lex.contains(wordBit))\n\t\t\tretVal=lex.replaceFirst(wordBit,semClass);*/\n\t\t\n\t\t//Ignore CITY-based classes\n\t\tif(wordBit.equals(lex))\n\t\t\tretVal=semClass;\n\t\t\n\t\treturn retVal;\n\t\t\n\t}\n\t\n\t/** Strip off label like ENAMEX,TIMEX,NUMEX. */\n\tpublic String getCleanClass(String semClass){\n\t\t\n\t\tString retVal=\"\";\n\t\tString x[]=semClass.split(\"=\");\n\t\tif(x.length>=2)\n\t\t\tretVal=x[1];\n\t\t\n\t\treturn retVal;\n\t}\n\t\n\t/** Retrieve stored bbn-info for use in the Treenodes. */\n\tpublic String getBBNInfo(String header,String span,String words){\n\t\tString sentId=header.replaceFirst(\"ID=\",\"\");\n\t\tString bbnData=\"\";\n\t\tString key=sentId+\" \"+span;\n\t\t\n\t\tboolean legitPhr=false;\n\t\t\n\t\t//Checking stored BBN-data with actual words\n\t\tif(bbnInfo.containsKey(key)){\n\t\t\tString[] ccgWords=words.split(\"_\");\n\t\t\tbbnData=bbnInfo.get(key);\n\t\t\tString[] bbnWords=bbnData.split(\" \");\n\t\t\t\n\t\t\tif(ccgWords.length>0 && bbnWords.length>1 && ccgWords.length==bbnWords.length-1){\n\t\t\t\tfor(int i=0;i0)\n\t\t\t\tretVal=span1+\" \"+qInfo[0];\n\t\t}\n\t\telse if(quoteInfo.containsKey(key2)){\n\t\t\tquotedText=quoteInfo.get(key2);\n\t\t\tString []qInfo=quotedText.split(\" \");\n\t\t\t\n\t\t\tif(qInfo.length>0 && qInfo[qInfo.length-1].matches(\"\\\\p{Punct}\")){\n\t\t\t\tretVal=span2+\" \"+qInfo[0]+\" \"+qInfo[qInfo.length-1];\n\t\t\t}\n\t\t\t\n\t\t}\n\t\t\n\t\treturn retVal;\n\t}\n\t\n\t/** Store result cat. */\n\tString res=\"\";\n\tpublic void storeRes(String str){\n\t\tthis.res=str.replaceAll(\"\\\\[.*\",\"\");\n\t}\n\t\n\t/** Retrieve result cat. */\n\tpublic String getRes(){\n\t\tString retVal=this.res;\n\t\treturn retVal;\n\t}\n\t\n\t/** The store of ids. */\n\tprivate Hashtable idTally=new Hashtable();\n\tpublic String id(String cat){\n\t\t\n\t\tString retVal=\"\";\n\t\t\n\t\tint idNum=idTally.size()+2;\n\t\tif(!idTally.containsKey(cat))\n\t\t\tidTally.put(cat,idNum);\n\t\telse \n\t\t\tidNum=idTally.get(cat);\n\t\t\n\t\tif(idTally.size()==1)\n\t\t\tretVal=\"first\"+\"_\"+Integer.toString(idNum);\n\t\telse retVal=\"later\"+\"_\"+Integer.toString(idNum);\n\t\t\n\t\treturn retVal;\n\t\t\n\t}\n\t\n\tpublic void id(){\n\t\tidTally=new Hashtable();\n\t}\n\t\n\t/** Retrieve terminal no. */\n\tpublic String getTermNo(){\n\t\tString tn=Integer.toString(this.termNo);\n\t\tthis.termNo++;\n\t\treturn tn;\n\t}\n\t\n\t/** Retrieve non-terminal no. */\n\tpublic String getNonTermNo(){\n\t\tString ntNo=Integer.toString(this.ntNo);\n\t\tthis.ntNo++;\n\t\treturn ntNo;\n\t}\n\t\n\t/** Get punctless index. */\n\tint plessInd=0;\n\tpublic String getPunctlessIndex(String word){\n\t\tboolean isCCGWord=this.isCCGWord(word);\n\t\tint retval=-1;\n\t\tif(isCCGWord){\n\t\t\tretval=this.plessInd;\n\t\t\tthis.plessInd++;\n\t\t}\n\t\treturn Integer.toString(retval);\n\t}\n\t\n\t/** Given a lexical item, ascertain whether it is a legit original CCGbank word **/\n\tpublic boolean isCCGWord(String word){\n\t\t\n\t\tboolean retval=true;\n\t\tif(word.matches(\"\\\\p{Punct}|[\\\\.]+|(-lrb-)|(-rrb-)|(-lcb-)|(-rcb-)|(--)|(`)|(')|(``)|('')\") && !word.equals(\"$\") && !word.equals(\"%\")){\n\t\t\tretval=false;\n\t\t}\n\t\treturn retval;\n\t}\n\t\n\t/** Init terminal nos. */\n\tpublic String initId(){\n\t\tthis.termNo=0;\n\t\tthis.ntNo=500;\n\t\tthis.plessInd=0;\n\t\treturn null;\n\t}\n\t\n\t/** Extract PTB SBJ,FN-TAG & TPC annotation. */\n\tpublic String getPTBInfo(String label,String sentId,String head,String lexInd){\n\t\t\n\t\tString retval=\"\";\n\t\tsentId=sentId.replaceFirst(\"ID=\",\"\");\n\t\tString key=sentId+\" \"+head+\"_\"+lexInd;\n\t\tString rel=null;\n\t\t\n\t\tif(label.equals(\"SBJ\"))rel=sbjInfo.get(key);\n\t\telse if(label.equals(\"FNT\"))rel=fntagInfo.get(key);\n\t\telse if(label.equals(\"TPC\"))rel=tpcInfo.get(key);\n\t\t\n\t\tif(rel!=null){\n\t\t\tretval=rel;\n\t\t}\n\t\treturn retval;\n\t}\n\t\n\t/** Print out BBN classes (for use in grammar.xml). */\n\tpublic void printBBNClasses(){\n\t\t\n\t\ttry{\n\t\t\t\n\t\t\tSystem.out.println(\"Printing BBN classes used in the corpus to bbn-types.txt (for use in grammar.xml)\");\n\t\t\t// Create a FileWriter stream to the file\n\t\t\tFileWriter file_writer = new FileWriter (\"bbn-types.txt\");\n\t\t\tBufferedWriter buf_writer = new BufferedWriter (file_writer);\n\t\t\tPrintWriter print_writer = new PrintWriter (buf_writer,true);\n\t\t\tprint_writer.print(\"\");\n\t\t\t\tprint_writer.flush();\n\t\t\t}\n\t\t\tprint_writer.close();\n\t\t}\n\t\tcatch (Exception e){\n\t\t\tSystem.err.println (\"Error writing info to file\");\n\t\t}\n\t}\n\t\n\t/** Input a string which contains a ':' and replace it by '|'. */\n\tpublic String replaceColon(String str) {\n\t\t\n\t\tString replacedStr=str.replace(\":\",\"|\");\n\t\treturn replacedStr;\n\t}\t\n\t\n}\n" }, { "path": "src/opennlp/ccgbank/convert/Javafns.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Java Program invoked by pp-head extraction routines\n\npackage opennlp.ccgbank.convert;\n\nimport java.util.ArrayList;\nimport java.util.List;\n\n// NB: addRes and changeCat probably made superfluous by computeCats.xsl\npublic class Javafns{\n\t\n\t//Prep heads storage\n\tpublic List heads = new ArrayList();\n\t\n\t//Temp id of pp heads with args\n\tString tempId=\"\";\t\n\n\t//Insert the prep-head into a result category by string replacement\n\tpublic String addRes(String cat,String ppHead){\n\t\t\n\t\t//Xsl spl char detected and escaped\n\t\tif(ppHead.equals(\"$\"))\n\t\t\tppHead=\"\\\\$\";\n\t\t\n\t\t//Head inserted into first PP category\n\t\tcat=cat.replaceFirst(\"pp\",\"pp[\"+ppHead+\"]\");\n\t\treturn cat;\n\t}\n\t\n\t//A safety hatch to elide extra heads detected - For leaf nodes\n\tpublic String elimRedun(int headCount){\n\t\t\n\t\t//Calculating redundant Headcount\n\t\tint j=0;\n\t\t\n\t\t//All heads following the last PP-head in a cat spec are redundant\n\t\tint redun=heads.size()-headCount;\n\t\t\n\t\t//Redundant heads removed\n\t\tfor(j=0;j-1;i--){\n\t\t\t\n\t\t\tString ppHead = heads.get(i);\n\t\t\t\n\t\t\t//Escaping dollar signs for xsl\n\t\t\tif(ppHead.equals(\"$\"))\n\t\t\t\tppHead=\"\\\\$\";\n\t\t\t\n\t\t\t//Simple head insertion by replacement on the string\n\t\t\tcat=cat.replaceFirst(\"pp_\",\"pp[\"+ppHead+\"]_\");\n\t\t\t\n\t\t}\n\n\t\treturn cat;\n\t}\n\t\n\tpublic String flush(){\n\t\theads.clear();tempId=\"\";\t\t\n\t\treturn null;\n\t}\n\t\n\t\n\tpublic String setHead(String head){\n\t\theads.add(head);\n\t\treturn null;\n\t}\n\t\n\t\n\tpublic String getHead(){\n\t\t\n\t\tString head=\"\";\n\t\t\n\t\tif(heads.size()==0)\n\t\t\thead=\"WrongHead\";\n\t\telse{\n\t\t\thead = heads.get(heads.size()-1);\n\t\t\theads.remove(heads.size()-1);\n\t\t} \n\t\t\n\t\treturn head;\n\t}\n\t\n\t\n\tpublic String peekHead(){\n\t\t\n\t\tString head=\"WrongHead\";\n\t\t\n\t\tif(heads.size()>0)\n\t\t\thead = heads.get(heads.size()-1);\n\t\t\n\t\treturn head;\n\t}\n\n\tpublic String printCat(String cat){\n\t\tSystem.out.println(cat);\n\t\treturn cat;\n\t}\n\n}\n" }, { "path": "src/opennlp/ccgbank/convert/MWHelper.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Bkgrnd java class which helps with operations reltd to multi-word units\n//As a first stab, I combine multi-word conjns like \"As well as\"\n\npackage opennlp.ccgbank.convert;\n\nimport java.util.*;\n\npublic class MWHelper {\n\n\t// The largest current id\n\tprivate String lex = \"\";\n\n\tprivate String pos = \"\";\n\n\tprivate String term_no = \"\";\n\n\t// A list of the particle terminal nos\n\tprivate ArrayList prtTally = new ArrayList();\n\n\t// Index of terminal nos and the lexical items they correspond to\n\tprivate Hashtable prtIndex = new Hashtable();\n\n\tpublic void initSettings() {\n\t\tprtTally = new ArrayList();\n\t\tprtIndex = new Hashtable();\n\t}\n\n\t// Concat lex,pos & term_nos of multi-word units\n\tpublic void concatWords(String lex, String pos, String term_no) {\n\n\t\t// System.out.println(lex);\n\t\tthis.lex = this.lex + \"_\" + lex;\n\t\tthis.pos = this.pos + \" \" + pos;\n\t\tthis.term_no = this.term_no + \" \" + term_no;\n\t}\n\n\t// Retrieve stored info\n\tpublic String getInfo(int choice) {\n\n\t\tString retVal = \"\";\n\n\t\tswitch (choice) {\n\n\t\tcase 1:\n\t\t\tretVal = lex.trim().replaceFirst(\"_\", \"\");\n\t\t\tthis.lex = \"\";\n\t\t\tbreak;\n\t\tcase 2:\n\t\t\tretVal = pos.trim();\n\t\t\tthis.pos = \"\";\n\t\t\tbreak;\n\t\tcase 3:\n\t\t\tretVal = term_no.trim();\n\t\t\tthis.term_no = \"\";\n\t\t\tbreak;\n\n\t\t}\n\n\t\treturn retVal;\n\t}\n\n\t// Store particle ids\n\tpublic void storePrt(String prt_term_no, String prt) {\n\t\tprtTally.add(Integer.parseInt(prt_term_no));\n\t\tprtIndex.put(Integer.parseInt(prt_term_no), prt);\n\t}\n\n\tpublic String peekPrt(int nextPrnNo) {\n\n\t\tString retVal = \"\";\n\t\tif (prtIndex.containsKey(nextPrnNo)) {\n\t\t\tretVal = prtIndex.get(nextPrnNo);\n\t\t}\n\n\t\treturn retVal;\n\t}\n\n\tpublic String getPrt() {\n\t\t// System.out.println(prtTally);\n\t\t// prtTally=new ArrayList();\n\t\tCollections.sort(prtTally);\n\t\tString retVal = \"\";\n\t\tif (prtTally.size() > 0) {\n\n\t\t\tretVal = (prtTally.get(prtTally.size() - 1)).toString();\n\t\t\t// System.out.println(retVal);\n\t\t\tprtTally.remove(prtTally.size() - 1);\n\t\t}\n\n\t\treturn retVal;\n\t}\n\n}\n" }, { "path": "src/opennlp/ccgbank/convert/MorphLookup.java", "content": "package opennlp.ccgbank.convert;\r\n\r\nimport java.io.BufferedReader;\r\nimport java.io.File;\r\nimport java.io.FileReader;\r\nimport java.io.IOException;\r\nimport java.util.HashMap;\r\nimport java.util.Map;\r\n\r\n/**\r\n * Utility class for looking up stems as determined by the morpha utility. \r\n */ \r\npublic class MorphLookup {\r\n\r\n\t// map from word_pos to stem\r\n\tprivate static Map stemMap = null;\r\n\tstatic File words, stems;\r\n\t\r\n\tpublic static void init(File wordsFile, File stemsFile) throws IOException {\r\n\t\tMorphLookup.words = wordsFile;\r\n\t\tMorphLookup.stems = stemsFile;\r\n\t\t\r\n\t\tif(MorphLookup.words == null) {\r\n\t\t\tthrow new IllegalArgumentException(\"words file not specified\");\r\n\t\t}\r\n\t\tif(MorphLookup.stems == null) {\r\n\t\t\tthrow new IllegalArgumentException(\"stems file not specified\");\r\n\t\t}\r\n\t\t\r\n\t\tstemMap = new HashMap();\r\n\t\t\r\n\t\tBufferedReader wordsReader = new BufferedReader(\r\n\t\t\t\tnew FileReader(words));\r\n\t\tBufferedReader stemsReader = new BufferedReader(\r\n\t\t\t\tnew FileReader(stems));\r\n\t\tString wordsLine, stemsLine;\r\n\t\t// read lines in parallel\r\n\t\twhile ( (wordsLine = wordsReader.readLine()) != null ) {\r\n\t\t\tstemsLine = stemsReader.readLine();\r\n\t\t\t//System.out.println(stemsLine);\r\n\t\t\t// wordsLine has a word and a POS\r\n\t\t\tString[] tokens = wordsLine.split(\"\\\\s+\");\r\n\t\t\tString word = tokens[0]; String pos = tokens[1];\r\n\t\t\t// stemsLine just has a stem; lowercase it, for good measure\r\n\t\t\tString stem = stemsLine.trim().toLowerCase();\r\n\t\t\t// add word_POS -> stem to map, also with word lowercased\r\n\t\t\tString key = word + \"_\" + pos;\r\n\t\t\tString key2 = word.toLowerCase() + \"_\" + pos;\r\n\t\t\tstemMap.put(key, stem);\r\n\t\t\tstemMap.put(key2, stem);\r\n\t\t}\r\n\t\twordsReader.close();\r\n\t\tstemsReader.close();\r\n\t}\r\n\t\r\n\t/** Returns the stem for the given word and pos, or the empty string if none. */\r\n\tpublic String getStem(String word, String pos) {\r\n\t\tString retval=\"\";String key=word + \"_\" + pos;\r\n\t\tif (MorphLookup.stemMap.containsKey(key))\r\n\t\t\tretval = stemMap.get(word + \"_\" + pos);\r\n\t\t\t/*if (retval == null) retval = \"\";\r\n\t\t//System.out.println(key+\" \"+retval);*/\r\n\t\t\r\n\t\tif(retval.length()==0)\r\n\t\t\tSystem.out.println(\"addStems: No stem for: \"+key);\r\n\t\treturn retval;\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccgbank/convert/OrigPunctRules.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Class which stores the context of punctuations\n\npackage opennlp.ccgbank.convert;\n\nimport java.util.Hashtable;\n\npublic class OrigPunctRules {\n\n\t//Sentence id\n\tprivate String id = \"\";\n\n\t//The largest current id\n\tprivate static int globalId = 0;\n\n\t//ccgbank section\n\tprivate String sect=\"\";\n\n\t//Label for a punct mark\n\tprivate String label = \"\";\n\n\t//The store of ids\n\tprivate static Hashtable idTally = new Hashtable();\n\n\t//Proc which traps, stores id of each sentence in a global var\n\tpublic String storeId(String x) {\n\t\tid = \"\";\n\t\tsect = \"\";\n\t\t//System.out.println(x+\" \"+\"Raja\");\n\t\tString expId[] = x.split(\"_\");\n\t\tid = expId[1];\n\t\tsect = id.substring(0, 2);\n\t\tif (sect.length() < 0) sect += \"just avoiding a warning here\"; \n\t\t//System.out.println(sect+\" \"+\"Raja\");\n\t\treturn null;\n\t}\n\n\t//Initialization before start of a new binary rule \n\tpublic String initId() {\n\t\tglobalId = 0;\n\t\tidTally.clear();\n\t\treturn null;\n\t}\n\n\t//Id allotment\n\tpublic String allotId(String cat) {\n\t\tString newId = \"\";\n\t\t//Id allotment\n\t\tif (!idTally.containsKey(cat)) {\n\t\t\tglobalId++;\n\t\t\tnewId = Integer.toString(globalId);\n\t\t\tidTally.put(cat, newId);\n\t\t}\n\t\tnewId = (String) idTally.get(cat);\n\t\treturn newId;\n\t}\n\n\tpublic String storeLabel(String x) {\n\t\tlabel = x;\n\t\treturn null;\n\t}\n\n\tpublic String getLabel() {\n\t\treturn label;\n\t}\n\n\tpublic String initLabel() {\n\t\tlabel = \"\";\n\t\treturn null;\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/convert/PunctHelper.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Java class which adds brackets, stores arg1 position for inferConjRules.xsl,normConjRulesId, normTreenodeId.xsl\n\npackage opennlp.ccgbank.convert;\n\nimport java.util.ArrayList;\nimport java.util.Hashtable;\n\npublic class PunctHelper {\n\n\t//The largest current id\n\tprivate int globalId = 0;\n\n\t//The store of ids\n\t//private Hashtable idTally = new Hashtable();\n\n\tprivate String feat = \"\";\n\n\tprivate String pos = \"\";\n\n\tprivate String balCom = \"\";\n\n\t//The pos-indexRel tally\n\tprivate static Hashtable> indexRel = new Hashtable>();\n\n\t//Calculate & store the indexRel\n\tpublic String calcIndexRel(String cat, String pos) {\n\n\t\tArrayList temp = new ArrayList();\n\t\tpos = pos.replaceAll(\"[0-9]\", \"\");\n\n\t\t//System.out.println(cat);\n\n\t\tif (!pos.equals(\"PUNCT_LPAREN\")) {\n\n\t\t\tcat = cat.replaceAll(\"/\\\\*punct\\\\[,\\\\]_[0-9]\", \"\");\n\t\t\tcat = cat.replaceAll(\"/\\\\*punct\\\\[--\\\\]_[0-9]\", \"\");\n\t\t\tcat = cat.replaceAll(\"/\\\\*punct\\\\[-rrb-\\\\]_[0-9]\", \"\");\n\t\t\tcat = cat.replaceAll(\"/\\\\*punct\\\\[-rcb-\\\\]_[0-9]\", \"\");\n\n\t\t}\n\n\t\t//System.out.println(cat);\n\n\t\tif (!indexRel.containsKey(pos)) {\n\t\t\ttemp.add(cat);\n\t\t\tindexRel.put(pos, temp);\n\t\t}\n\n\t\ttemp = indexRel.get(pos);\n\t\tif (!temp.contains(cat))\n\t\t\ttemp.add(cat);\n\n\t\tString ind = Integer.toString(temp.indexOf(cat) + 1);\n\n\t\treturn ind;\n\n\t}\n\n\t//Initialization before start of a new conj rule \n\tpublic String globalInit() {\n\t\tglobalId = 0;\n\t\t//idTally.clear();\n\t\treturn null;\n\t}\n\n\t//Function invoked by invertedDirSpComma.xsl\n\tpublic String getglobalId() {\n\t\tglobalId++;\n\t\treturn Integer.toString(globalId);\n\t}\n\n\tpublic String setglobalId(int x) {\n\t\tglobalId = x;\n\t\treturn null;\n\t}\n\n\tpublic String storePOS(String x) {\n\t\tpos = x;\n\t\treturn null;\n\t}\n\n\tpublic String getPOS() {\n\t\tString retVal = pos;\n\t\treturn retVal;\n\t}\n\n\tpublic String initPOS() {\n\t\tpos = \"\";\n\t\treturn null;\n\t}\n\n\tpublic String balInit() {\n\t\tbalCom = \"\";\n\t\treturn null;\n\t}\n\n\tpublic String storeBal(String x) {\n\t\tbalCom = x;\n\t\treturn null;\n\t}\n\n\tpublic String getBal() {\n\t\treturn balCom;\n\t}\n\n\tpublic String storeFeat(String x) {\n\t\tfeat = x;\n\t\treturn null;\n\t}\n\n\tpublic String getFeat() {\n\t\treturn feat;\n\t}\n\n\tpublic String featInit() {\n\t\tfeat = \"\";\n\t\treturn null;\n\t}\n\n\tpublic String debugPrint(String x, String y) {\n\t\tSystem.out.println(\"Debug: \" + x + \" at \" + y);\n\t\treturn null;\n\t}\n\n\tpublic String removeFeats(String cat) {\n\t\tcat = cat.replaceAll(\"\\\\[[a-zA-Z]+\\\\]\", \"\");\n\t\t//System.out.println(\"Debug: \"+cat);\n\t\treturn cat;\n\t}\n\n\tpublic String purgeCat(String cat) {\n\t\tcat = cat.replaceAll(\"\\\\[[a-zA-Z]+\\\\]\", \"\");\n\t\tcat = cat.replaceAll(\"~\", \"\");\n\t\tcat = cat.replaceAll(\"_[0-9]+\", \"\");\n\t\t//System.out.println(\"Debug: \"+cat);\n\t\treturn cat;\n\t}\n\n\tpublic String purgeCat1(String cat) {\n\t\tcat = cat.toLowerCase();\n\t\tcat = cat.replaceAll(\"\\\\[[a-zA-Z]+\\\\]\", \"\");\n\t\tcat = cat.replaceAll(\"~\", \"\");\n\t\tcat = cat.replaceAll(\"_[0-9]+\", \"\");\n\t\t//System.out.println(\"Debug: \"+cat);\n\t\treturn cat;\n\t}\n\n\t//Replace pp[] by pp\n\tpublic String cleanPP(String cat) {\n\t\tcat = cat.replaceAll(\"pp\\\\[\\\\]\", \"pp\");\n\t\treturn cat;\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/convert/RoleAdjuster.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccgbank.convert;\r\n\r\n/**\r\n * Utility class for adjusting roles and augmenting cat names with roles. \r\n */ \r\npublic class RoleAdjuster {\r\n\r\n\t/** Returns the adjusted argument roles, making guesses at the missing roles. */\r\n\tpublic String getAdjustedRoles(String cat, String roles) {\r\n\t\t// aux things like \"have to\"\r\n\t\tif (roles.equals(\"null e\") || roles.equals(\"e e\")) {\r\n\t\t\tif (cat.indexOf(\"np[thr]\") < 0)\r\n\t\t\t\treturn \"Arg0 Arg1\";\r\n\t\t\telse \r\n\t\t\t\treturn \"e Arg1\";\r\n\t\t}\r\n\t\t// vp mods\r\n\t\tif (roles.startsWith(\"null e\")) {\r\n\t\t\treturn \"null ArgM\" + roles.substring(\"null e\".length());\r\n\t\t}\r\n\t\t// missing subjects, mostly\r\n\t\tif (roles.startsWith(\"null\")) {\r\n\t\t\tString rest = roles.substring(\"null\".length());\r\n\t\t\treturn addMissingArg(cat, rest);\r\n\t\t}\r\n\t\t// various\r\n\t\tif (roles.startsWith(\"e\")) {\r\n\t\t\tString rest = roles.substring(\"e\".length());\r\n\t\t\t// mods\r\n\t\t\tif (cat.indexOf(\"_~\") > 0)\r\n\t\t\t\treturn \"ArgM\" + rest;\r\n\t\t\t// leave expletives unchanged\r\n\t\t\tif (cat.indexOf(\"np[expl]\") > 0 || cat.indexOf(\"np[thr]\") > 0)\r\n\t\t\t\treturn roles;\r\n\t\t\t// otherwise add standard guess\r\n\t\t\treturn addMissingArg(cat, rest);\r\n\t\t}\r\n\t\t// otherwise unchanged\r\n\t\treturn roles;\r\n\t}\r\n\r\n\t// add guess at missing arg\r\n\tprivate String addMissingArg(String cat, String rest) {\r\n\t\t// distinguish passive\r\n\t\tif (cat.startsWith(\"s[pss]\")) {\r\n\t\t\tif (rest.indexOf(\"Arg1\") < 0)\r\n\t\t\t\treturn \"Arg1\" + rest;\r\n\t\t\telse \r\n\t\t\t\treturn \"Arg2\" + rest;\r\n\t\t}\r\n\t\t// otherwise Arg0 or Arg1\r\n\t\tif (rest.indexOf(\"Arg0\") < 0)\r\n\t\t\treturn \"Arg0\" + rest;\r\n\t\telse \r\n\t\t\treturn \"Arg1\" + rest;\r\n\t}\r\n\t\r\n\t/** Returns the cat name augmented with the given argument roles. */\r\n\tpublic String getCatPlusRoles(String cat, String roles) {\r\n\t\treturn cat + \":\" + roles.replaceAll(\" \", \"+\");\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccgbank/convert/XSLTTrueCaser.java", "content": "///////////////////////////////////////////////////////////////////////////////\n//Copyright (C) 2011 Dennis N. Mehay\n//\n//This library is free software; you can redistribute it and/or\n//modify it under the terms of the GNU Lesser General Public\n//License as published by the Free Software Foundation; either\n//version 2.1 of the License, or (at your option) any later version.\n//\n//This library is distributed in the hope that it will be useful,\n//but WITHOUT ANY WARRANTY; without even the implied warranty of\n//MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n//GNU Lesser General Public License for more details.\n//\n//You should have received a copy of the GNU Lesser General Public\n//License along with this program; if not, write to the Free Software\n//Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccgbank.convert;\n\n/**\n * A class that has a static constructor to create a TrueCaser so that XSLT (which requires such a\n * set-up) can call the TrueCaser.\n *\n * @author Dennis N. Mehay\n *\n */\n\nimport opennlp.ccg.lexicon.TrueCaser;\n\npublic class XSLTTrueCaser {\n\t\n\tstatic TrueCaser tc = null;\n\t\n\t/**\n\t * Static constructor that creates a true-caser. See the TrueCaser.java doc's for more info.\n\t */\n\tpublic static void init(String pathToTrueCaseList) {\n\t\tXSLTTrueCaser.tc = new TrueCaser(pathToTrueCaseList, 0.5);\n\t}\n\t\n\t/** Function invoked from the XSLT transform trueCaser.xsl to true case words in a derivation .*/\n\tpublic String trueCase(String theWord, String neClass, String pos,String wordPosition) {\n\t\treturn tc.trueCase(theWord, true, true);\n\t}\n}" }, { "path": "src/opennlp/ccgbank/extract/CatNode.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Java class to store all the info associated with a category\n\npackage opennlp.ccgbank.extract;\n\nimport java.io.PrintWriter;\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.StringTokenizer;\n\npublic class CatNode {\n\t\n\t// Category name, POS\n\tpublic final String cat;\n\tpublic final String pos;\n\t\n\t//Lexeme frequency\n\tprivate Map lexFreq = new HashMap();\n\t\n\t//Sentence id of lexemes\n\tprivate Map> lexOccur = new HashMap>();\n\t\n\t//Constructor to initialize the cat name\n\tpublic CatNode(String cat, String pos){\n\t\tthis.cat=cat; this.pos = pos;\n\t}\n\t\n\t//Procedure to insert info associated with all lexemes of a category\n\tpublic void lexInsert(String lex, String id){\n\t\t//When first occurences of lexemes detected, entries opened\n\t\tif (!lexFreq.containsKey(lex)){\n\t\t\tlexFreq.put(lex, 1);\n\t\t\tList idList = new ArrayList(4);\n\t\t\tidList.add(id);\n\t\t\tlexOccur.put(lex, idList);\n\t\t}\n\t\telse { //Subsequent occurences of lexemes updated\n\t\t\tlexFreq.put(lex, lexFreq.get(lex)+1);\n\t\t\tList idList = lexOccur.get(lex);\n\t\t\t//Ids of up to four sentences stored\n\t\t\tif(!idList.contains(id) && idList.size() < 4){\n\t\t\t\tidList.add(id);\n\t\t\t}\n\t\t}\n\t}\n\t\n\t// returns frequence of lex with this cat and pos\n\tpublic int getLexFreq(String lex) {\n\t\tInteger retval = lexFreq.get(lex);\n\t\tif (retval == null) return 0;\n\t\telse return retval;\n\t}\n\t\n\t//Proc which prints out the lexical info of a category\n\tpublic void printTally(PrintWriter output) {\n\t\t\n\t\t//Lexemes sorted in descending order of freq\n\t\tList sortedLex = FreqTally.sortTally(lexFreq);\n\t\t\n\t\tString ccgbankHome = System.getProperty(\"CCGBANK_HOME\", \"/home/corpora/EN/ccgbank\");\n\t\t\n\t\t//Sorted list processed\n\t\tfor (int i=0; i<3 && i\");\n\t\t\toutput.println(\"

  • \");\n\t\t\toutput.println(lex+\" \"+freq);\n\t\t\t\n\t\t\t//Sentence ids also printed\n\t\t\tList temp = lexOccur.get(lex);\n\t\t\tfor (String id: temp) {\n\t\t\t\t//System.out.println(id);\n\t\t\t\tString[]idInfo=id.split(\"\\\\.\");\n\t\t\t\t//System.out.println(idInfo[0]);\n\n\t\t\t\tStringTokenizer st=new StringTokenizer(id,\".\");\n\t\t\t\toutput.println(\"
      \");\n\n\t\t\t\tString idLink=\"\";\n\t\t\t\tString sentNo=\"\";\n\t\t\t\tString dir=\"\";\n\n\t\t\t\t//2 courses of action depending on whether input is gold std .auto parses or C&C .auto parses\n\t\t\t\tif(idInfo.length==2){\n\t\t\t\t\t\tidLink=st.nextToken()+\".html\";\n\t\t\t\t\t\tsentNo=\"#Sentence \"+st.nextToken();\n\t\t\t\t\t\tdir=id.substring(4,6);\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\t\tidLink=idInfo[0];\n\t\t\t\t\t\tsentNo=\"#Sentence \"+idInfo[0];\n\t\t\t\t\t\tdir=idInfo[0];\n\t\t\t\t}\n\t\t\t\toutput.println(\"
    • \");\n\t\t\t\toutput.println(id);\n\t\t\t\toutput.println(\" \");\n\t\t\t\toutput.println(idLink);\n\t\t\t\toutput.println(\"\"); \n\t\t\t\toutput.println(\"
      • \"); \n\t\t\t\toutput.println(\"
    \");\n\t\t\t}\n\n\t\t\t\n\t\t\toutput.println(\"
    • \");\n\t\t\toutput.println(\"\");\n\t\t\toutput.println(\"\");\n\t\t\toutput.flush();\n\t\t}\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/extract/DebugHelper.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Bkgrnd java class which helps with operations for debugging LFs\n\npackage opennlp.ccgbank.extract;\n\nimport java.util.*;\nimport java.io.*;\n\npublic class DebugHelper {\n\n\tpublic static boolean init = true;\n\n\tpublic static ArrayList unmatCats = new ArrayList();\n\n\tpublic static int nsrCount = 0;\n\n\tpublic static int unmatCount = 0;\n\n\tpublic boolean getInit() {\n\n\t\treturn init;\n\n\t}\n\n\t//Read in bkgrnd info\n\tpublic void readInfo() {\n\n\t\ttry {\n\n\t\t\tBufferedReader inp = new BufferedReader(new FileReader(\n\t\t\t\t\t\"/scratch/propgrammar/unmat.txt\"));\n\t\t\tString line = \"\";\n\n\t\t\twhile ((line = inp.readLine()) != null) {\n\n\t\t\t\tif (line.length() == 0)\n\t\t\t\t\tcontinue;\n\n\t\t\t\tString parts[] = line.split(\" \");\n\t\t\t\tString name = parts[parts.length - 2];\n\t\t\t\tString pos = parts[parts.length - 1];\n\t\t\t\tString x[] = name.split(\"=\");\n\t\t\t\tString y[] = pos.split(\"=\");\n\t\t\t\tname = purgeCat(x[1]);\n\t\t\t\tpos = purgeCat(y[1]);\n\t\t\t\tString unmat = name + \" \" + pos;\n\t\t\t\tunmatCats.add(unmat);\n\t\t\t\t//System.out.println(name+\" \"+pos);\t\t\n\n\t\t\t}\n\t\t\t//System.out.println(tagInfo);\n\t\t\t//System.out.println(tagInfo.size());\n\t\t\tinit = false;\n\t\t\tinp.close();\n\t\t}\n\n\t\tcatch (IOException e) {\n\t\t\tSystem.out.println(\"Error reading input file\");\n\t\t}\n\t}\n\n\tpublic String purgeCat(String cat) {\n\n\t\tcat = cat.replaceAll(\"\\\"\", \"\");\n\t\tcat = cat.replaceAll(\"~\", \"\");\n\t\tcat = cat.replaceAll(\"_[0-9]+\", \"\");\n\t\tcat = cat.replaceAll(\">\", \"\");\n\t\t//System.out.println(\"Debug: \"+cat);\n\t\treturn cat;\n\n\t}\n\n\tpublic void recordInfo(String sentId, String pred, String misc) {\n\n\t\tnsrCount++;\n\t\tif (unmatCats.contains(misc))\n\t\t\tunmatCount++;\n\n\t}\n\n\tpublic void printInfo() {\n\t\tSystem.out.println(unmatCats);\n\t\tSystem.out.println(\"No:of nsr LFs: \" + nsrCount);\n\t\tSystem.out.println(\"No:of unmatched that a NSR LF contains: \"\n\t\t\t\t+ unmatCount);\n\t}\n\n}\n" }, { "path": "src/opennlp/ccgbank/extract/DefaultLFHelper.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Bkgrnd java class which helps with operations for debugging LFs\n\npackage opennlp.ccgbank.extract;\n\nimport java.util.*;\n//import java.io.*;\n\npublic class DefaultLFHelper {\n\n\tprivate ArrayList idTally = new ArrayList();\n\n\tprivate Hashtable freqTable = new Hashtable();\n\n\tprivate String lfType = \"ord\";\n\n\tpublic void init() {\n\n\t\tlfType = \"ord\";\n\t\tidTally = new ArrayList();\n\t\tfreqTable = new Hashtable();\n\t\targCounter = 0;\n\t}\n\n\tpublic void storeCat(String cat, String id, String idType) {\n\n\t\tint freq = 1;\n\n\t\tif (id.length() > 0) {\n\n\t\t\t//System.out.println(cat+\" \"+id+\" \"+idType);\n\n\t\t\tif (!freqTable.containsKey(id))\n\t\t\t\tfreqTable.put(id, freq);\n\t\t\telse {\n\t\t\t\tfreq = freqTable.get(id) + 1;\n\t\t\t\tfreqTable.put(id, freq);\n\t\t\t}\n\n\t\t\tif (id.equals(\"1\") && idType.equals(\"inherits\"))\n\t\t\t\tlfType = \"mod-mod\";\n\t\t\telse if (!lfType.equals(\"mod-mod\") && id.equals(\"1\")\n\t\t\t\t\t&& idTally.size() > 0 && idTally.get(0).equals(\"1\"))\n\t\t\t\tlfType = \"mod\";\n\n\t\t\tidTally.add(id);\n\t\t}\n\t}\n\n\tpublic String getType() {\n\t\tString retVal = lfType;\n\t\tlfType = \"\";\n\t\targCounter = 0;\n\t\treturn retVal;\n\t}\n\n\tprivate int argCounter = 0;\n\n\tpublic String getArgNo(int argCount) {\n\t\tint argNo = argCount - argCounter;\n\t\targCounter++;\n\t\treturn Integer.toString(argNo);\n\t}\n\n\tpublic boolean isArg(String id) {\n\n\t\t//System.out.println(id);\n\t\t//System.out.println(freqTable);\n\n\t\tint freq = 0;\n\t\tif (freqTable.containsKey(id))\n\t\t\tfreq = freqTable.get(id);\n\n\t\tboolean retVal = false;\n\t\tif (freq == 1)\n\t\t\tretVal = true;\n\t\treturn retVal;\n\t}\n\n\tpublic String purgeCat(String cat) {\n\t\tcat = cat.replaceAll(\"\\\"\", \"\");\n\t\tcat = cat.replaceAll(\"~\", \"\");\n\t\tcat = cat.replaceAll(\"_[0-9]+\", \"\");\n\t\tcat = cat.replaceAll(\">\", \"\");\n\t\t//System.out.println(\"Debug: \"+cat);\n\t\treturn cat;\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/extract/ExtractGrammar.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\npackage opennlp.ccgbank.extract;\n\nimport java.io.File;\nimport java.io.FileWriter;\nimport java.io.PrintWriter;\nimport java.util.Arrays;\nimport java.util.Iterator;\nimport java.util.List;\n\nimport javax.xml.transform.stream.StreamSource;\n\npublic class ExtractGrammar {\n\n\t/** Class for holding properties of desired grammar extraction. */\n\tpublic static class ExtractionProperties {\n\n\t\t/** Whether to use the PP head augmented corpus. */\n\t\tpublic boolean ppHeads = true;\n\n\t\t/** Source directory. */\n\t\tpublic String srcDir = \"/scratch/ccgbank/converted\";\n\n\t\t/** Destination directory. */\n\t\tpublic String destDir = \"/scratch/grammars/protogrammar\";\n\n\t\t/** Temp directory. */\n\t\tpublic String tempDir = \"/tmp/ccgbankextract\";\n\n\t\t/** Start section. */\n\t\tpublic int startSection = 0;\n\n\t\t/** End section. */\n\t\tpublic int endSection = 24;\n\n\t\t/** Selected file (-1 if none). */\n\t\tpublic int fileNum = -1;\n\n\t\t/** Frequency cutoff for including an extracted cat. */\n\t\tpublic int catFreqCutoff = 1;\n\n\t\t/** Frequency cutoff for including an extracted lex, cat, pos triple. */\n\t\tpublic int lexFreqCutoff = 1;\n\n\t\t/** Frequency cutoff for making a family (ie, cat & pos) open. */\n\t\tpublic int openFreqCutoff = 100;\n\n\t\t/** Rule frequency cutoff. */\n\t\tpublic int ruleFreqCutoff = 1;\n\n\t\t/** Flag for whether to skip unmatched rules. */\n\t\tpublic boolean skipUnmatched = false;\n\n\t\t/** Whether to show debug info for failed derivations. */\n\t\tpublic boolean debugDerivs = false;\n\n\t\t/** File name for text only output. */\n\t\tpublic String textfile = null;\n\n\t\t/** File name for text factors output. */\n\t\tpublic String factorsfile = null;\n\n\t\t/** File name for observed supertag-rule combos.. */\n\t\tpublic String combosfile = null;\n\n\t\t// Flag to add feats excl to the lexicon\n\t\tpublic boolean lexF = false;\n\n\t\t// Flag to adjust lfs of orig puncts ie those corrs to extant corp\n\t\t// binary rules\n\t\tpublic boolean origPuncts = false;\n\n\t\t// String taking names of macros as input. Expects a dash separated list\n\t\tpublic String macroSpecs = \"\";\n\n\t\t// String taking names of LF specificity condtions as input. Expects a\n\t\t// dash separated list\n\t\t// Overt wh pronouns: wh\n\t\tpublic String lfSpecs = \"\";\n\n\t}\n\n\t/** Processes args and invokes extraction steps. */\n\tpublic static void main(String args[]) throws Exception {\n\n\t\tList arguments = Arrays.asList(args);\n\t\tExtractionProperties extractProps = new ExtractionProperties();\n\n\t\t// flags for each extraction step\n\t\tboolean doLex = true;\n\t\tboolean doMorph = true;\n\t\tboolean doRules = true;\n\t\tboolean doTestbed = true;\n\n\t\tif (arguments.contains(\"-h\") || arguments.contains(\"--help\")) {\n\t\t\tSystem.out.println(\"usage: extractGrammar \\n\"\n\t\t\t\t\t\t\t+ \"\\t[-noPPs|--noPPHeads] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[-lexF] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[-origPuncts] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[-s|--section sectnum] [-ss|--startSection sectnum] [-es|endSection sectnum] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[-f|--file filenum] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[--lexOnly|--morpOnly|--rulesOnly|--testbedOnly] [--skipLex] [--skipMorph] [--skipRules] [--skipTestbed]\\n\"\n\t\t\t\t\t\t\t+ \"\\t[-tmp|--tempDir tempDir] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[-cfc|--catFreqCutoff num] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[-lfc|--lexFreqCutoff num] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[-ofc|--openFreqCutoff num] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[-rfc|--ruleFreqCutoff num] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[--skipUnmatched] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[-dd|--debugDerivs] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[--text textfile] [--textf factorsfile] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[--combos combosfile] \\n\"\n\t\t\t\t\t\t\t+ \"\\t[srcDir] [destDir]\");\n\t\t\tSystem.exit(0);\n\t\t}\n\n\t\tif (arguments.contains(\"-noPPs\") || arguments.contains(\"--noPPHeads\")) {\n\t\t\textractProps.ppHeads = false;\n\t\t}\n\n\t\t// process args\n\t\tIterator it = arguments.iterator();\n\n\t\tString s;\n\t\tboolean seenSrc = false;\n\t\twhile (it.hasNext()) {\n\t\t\ts = it.next();\n\t\t\tif (s.equals(\"-s\") || s.equals(\"--section\") || s.equals(\"-ss\")\n\t\t\t\t\t|| s.equals(\"--startSection\") || s.equals(\"-es\")\n\t\t\t\t\t|| s.equals(\"--endSection\")) {\n\t\t\t\tif (!it.hasNext()) {\n\t\t\t\t\tthrow new IllegalArgumentException(\"encountered flag \" + s\n\t\t\t\t\t\t\t+ \", but no sectnum specified\");\n\t\t\t\t}\n\t\t\t\tint sectNum = Integer.parseInt(it.next());\n\t\t\t\tif (s.equals(\"-s\") || s.equals(\"--section\") || s.equals(\"-ss\")\n\t\t\t\t\t\t|| s.equals(\"--startSection\"))\n\t\t\t\t\textractProps.startSection = sectNum;\n\t\t\t\tif (s.equals(\"-s\") || s.equals(\"--section\") || s.equals(\"-es\")\n\t\t\t\t\t\t|| s.equals(\"--endSection\"))\n\t\t\t\t\textractProps.endSection = sectNum;\n\t\t\t} else if (s.equals(\"-f\") || s.equals(\"--filenum\")) {\n\t\t\t\tif (!it.hasNext()) {\n\t\t\t\t\tthrow new IllegalArgumentException(\"encountered flag \" + s\n\t\t\t\t\t\t\t+ \", but no filenum specified\");\n\t\t\t\t}\n\t\t\t\textractProps.fileNum = Integer.parseInt(it.next());\n\t\t\t} else if (s.equals(\"-lexF\")) {\n\t\t\t\tSystem.out\n\t\t\t\t\t\t.println(\"Inserting lexicon specific feats - Punct filter placeholder feats now\");\n\t\t\t\textractProps.lexF = true;\n\n\t\t\t} else if (s.equals(\"-origPuncts\")) {\n\t\t\t\textractProps.origPuncts = true;\n\n\t\t\t}\n\n\t\t\telse if (s.equals(\"--lexOnly\")) {\n\t\t\t\tdoMorph = false;\n\t\t\t\tdoRules = false;\n\t\t\t\tdoTestbed = false;\n\t\t\t} else if (s.equals(\"--morphOnly\")) {\n\t\t\t\tdoLex = false;\n\t\t\t\tdoRules = false;\n\t\t\t\tdoTestbed = false;\n\t\t\t} else if (s.equals(\"--rulesOnly\")) {\n\t\t\t\tdoLex = false;\n\t\t\t\tdoMorph = false;\n\t\t\t\tdoTestbed = false;\n\t\t\t} else if (s.equals(\"--testbedOnly\")) {\n\t\t\t\tdoLex = false;\n\t\t\t\tdoMorph = false;\n\t\t\t\tdoRules = false;\n\t\t\t} else if (s.equals(\"--skipLex\")) {\n\t\t\t\tdoLex = false;\n\t\t\t} else if (s.equals(\"--skipMorph\")) {\n\t\t\t\tdoMorph = false;\n\t\t\t} else if (s.equals(\"--skipRules\")) {\n\t\t\t\tdoRules = false;\n\t\t\t} else if (s.equals(\"--skipTestbed\")) {\n\t\t\t\tdoTestbed = false;\n\t\t\t} else if (s.equals(\"-tmp\") || s.equals(\"--tempDir\")) {\n\t\t\t\tif (!it.hasNext()) {\n\t\t\t\t\tthrow new IllegalArgumentException(\"encountered flag \" + s\n\t\t\t\t\t\t\t+ \", but no temp dir specified\");\n\t\t\t\t}\n\t\t\t\textractProps.tempDir = it.next();\n\t\t\t} else if (s.equals(\"-cfc\") || s.equals(\"--catFreqCutoff\")) {\n\t\t\t\tif (!it.hasNext()) {\n\t\t\t\t\tthrow new IllegalArgumentException(\"encountered flag \" + s\n\t\t\t\t\t\t\t+ \", but no num specified\");\n\t\t\t\t}\n\t\t\t\tint num = Integer.parseInt(it.next());\n\t\t\t\textractProps.catFreqCutoff = num;\n\t\t\t} else if (s.equals(\"-lfc\") || s.equals(\"--lexFreqCutoff\")) {\n\t\t\t\tif (!it.hasNext()) {\n\t\t\t\t\tthrow new IllegalArgumentException(\"encountered flag \" + s\n\t\t\t\t\t\t\t+ \", but no num specified\");\n\t\t\t\t}\n\t\t\t\tint num = Integer.parseInt(it.next());\n\t\t\t\textractProps.lexFreqCutoff = num;\n\t\t\t} else if (s.equals(\"-ofc\") || s.equals(\"--openFreqCutoff\")) {\n\t\t\t\tif (!it.hasNext()) {\n\t\t\t\t\tthrow new IllegalArgumentException(\"encountered flag \" + s\n\t\t\t\t\t\t\t+ \", but no num specified\");\n\t\t\t\t}\n\t\t\t\tint num = Integer.parseInt(it.next());\n\t\t\t\textractProps.openFreqCutoff = num;\n\t\t\t} else if (s.equals(\"-rfc\") || s.equals(\"--ruleFreqCutoff\")) {\n\t\t\t\tif (!it.hasNext()) {\n\t\t\t\t\tthrow new IllegalArgumentException(\"encountered flag \" + s\n\t\t\t\t\t\t\t+ \", but no num specified\");\n\t\t\t\t}\n\t\t\t\tint num = Integer.parseInt(it.next());\n\t\t\t\textractProps.ruleFreqCutoff = num;\n\t\t\t} else if (s.equals(\"--skipUnmatched\"))\n\t\t\t\textractProps.skipUnmatched = true;\n\t\t\telse if (s.equals(\"-dd\") || s.equals(\"--debugDerivs\"))\n\t\t\t\textractProps.debugDerivs = true;\n\t\t\telse if (s.equals(\"--text\")) {\n\t\t\t\tif (!it.hasNext()) {\n\t\t\t\t\tthrow new IllegalArgumentException(\"encountered flag \" + s\n\t\t\t\t\t\t\t+ \", but no file name specified\");\n\t\t\t\t}\n\t\t\t\textractProps.textfile = it.next();\n\t\t\t} else if (s.equals(\"--textf\")) {\n\t\t\t\tif (!it.hasNext()) {\n\t\t\t\t\tthrow new IllegalArgumentException(\"encountered flag \" + s\n\t\t\t\t\t\t\t+ \", but no file name specified\");\n\t\t\t\t}\n\t\t\t\textractProps.factorsfile = it.next();\n\t\t\t} else if (s.equals(\"--combos\")) {\n\t\t\t\tif (!it.hasNext()) {\n\t\t\t\t\tthrow new IllegalArgumentException(\"encountered flag \" + s\n\t\t\t\t\t\t\t+ \", but no file name specified\");\n\t\t\t\t}\n\t\t\t\textractProps.combosfile = it.next();\n\t\t\t} else if (!seenSrc) {\n\t\t\t\textractProps.srcDir = s;\n\t\t\t\tseenSrc = true;\n\t\t\t} else {\n\t\t\t\textractProps.destDir = s;\n\t\t\t}\n\t\t}\n\n\t\t// ensure directories exist or can be made\n\t\tFile tempDir = new File(extractProps.tempDir);\n\t\tif (!tempDir.exists() && !tempDir.mkdirs())\n\t\t\tthrow new IllegalArgumentException(\n\t\t\t\t\t\"could not create temp directory: \" + extractProps.tempDir);\n\t\tFile srcDir = new File(extractProps.srcDir);\n\t\tif (!srcDir.exists() || !srcDir.isDirectory())\n\t\t\tthrow new IllegalArgumentException(\n\t\t\t\t\t\"source directory does not exist: \" + extractProps.srcDir);\n\t\tFile destDir = new File(extractProps.destDir);\n\t\tif (!destDir.exists() && !destDir.mkdirs())\n\t\t\tthrow new IllegalArgumentException(\n\t\t\t\t\t\"could not create destination directory: \"\n\t\t\t\t\t\t\t+ extractProps.destDir);\n\n\t\t// log params\n\t\tSystem.out.println(\"Extracting Grammar\");\n\t\tSystem.out.println(\"Reading from: \" + srcDir);\n\t\tSystem.out.println(\"Writing to: \" + destDir);\n\t\tSystem.out.println(\"Temp dir: \" + tempDir);\n\t\tSystem.out.println(\"Start section: \" + extractProps.startSection);\n\t\tSystem.out.println(\"End section: \" + extractProps.endSection);\n\t\tif (extractProps.fileNum >= 0)\n\t\t\tSystem.out.println(\"File: \" + extractProps.fileNum);\n\n\t\t// do extraction steps\n\t\tif (doLex)\n\t\t\tLexExtract.extractLex(extractProps);\n\t\tif (doMorph)\n\t\t\tMorphExtract.extractMorph(extractProps);\n\t\tif (doRules)\n\t\t\tRulesExtract.extractRules(extractProps);\n\n\t\t// generate grammar.xml, if it doesn't already exist\n\t\t// nb: should eventually make schema refs relative to OPENCCG_HOME\n\t\tFile gramFile = new File(destDir, \"grammar.xml\");\n\t\tif (!gramFile.exists()) {\n\t\t\tSystem.out.println(\"Generating grammar.xml\");\n\t\t\tPrintWriter gramOut = new PrintWriter(new FileWriter(gramFile));\n\t\t\tgramOut.println(\"\");\n\t\t\tgramOut.println(\"\");\n\t\t\tgramOut.println(\" \");\n\t\t\tgramOut.println(\" \");\n\t\t\tgramOut.println(\" \");\n\t\t\tgramOut.println(\"\");\n\t\t\tgramOut.close();\n\t\t}\n\n\t\t// do testbed\n\t\tif (doTestbed && !doTestbed) ; // nb: just avoiding a warning here\n\t\t// TODO if (doTestbed) Testbed.createTestFiles(extractProps);\n\t}\n\n\t/* Returns a stream source for the given resource from the class loader. */\n\tpublic static StreamSource getSource(String resourceName) {\n\t\tClassLoader cl = ExtractGrammar.class.getClassLoader();\n\t\treturn new StreamSource(cl.getResourceAsStream(resourceName));\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/extract/FreqTally.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Class which stores lexical info and associated frequencies. \n\n//This class is invoked by LexExtr.xsl and StemInsert.xsl transforms\n\npackage opennlp.ccgbank.extract;\n\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileOutputStream;\nimport java.io.PrintWriter;\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\n\nimport opennlp.ccgbank.extract.ExtractGrammar.ExtractionProperties;\n\n//import javax.xml.transform.TransformerConfigurationException;\n//import javax.xml.transform.TransformerException;\n\n//import org.jdom.JDOMException;\n//import org.xml.sax.SAXException;\n\n\npublic class FreqTally{\n\n\t// Frequency cutoff for including an extracted cat \n\tpublic static int CAT_FREQ_CUTOFF = 1; \n\n\t// Frequency cutoff for including an extracted lex, cat, pos triple \n\tpublic static int LEX_FREQ_CUTOFF = 1; \n\n\t// Frequency cutoff for making a family (ie, cat & pos) open\n\tpublic static int OPEN_FREQ_CUTOFF = 100; \n\n\n\t//The object where lexical info of each category has been stored\n\tprivate static Map catInfo = new HashMap();\n\n\t//Freq of cat specs\n\tprivate static Map catFreq = new HashMap();\n\n\t//Sentence id\n\tprivate static String id=\"\";\n\n\t// Observed lex combos\n\tprivate static Set observedLexCombos = new HashSet();\n\n\t/** Resets the statically held tallies. */\n\tpublic static void reset() {\n\t\tcatInfo = new HashMap();\n\t\tcatFreq = new HashMap();\n\t\tid=\"\";\n\t\tobservedLexCombos = new HashSet();\n\t}\n\t\n\t//Proc which traps and stores id of each sentence\n\tpublic String storeId(String x) {\n\t\tif(x.length()>0){\n\t\t\tid = x;\n\t\t\tint posEquals = x.indexOf('=');\n\t\t\tif (posEquals > 0) {\n\t\t\t\tid = x.substring(posEquals+1);\n\t\t\t}\n\t\t}\n\t\treturn id;\n\t}\n\n\t//Changes case of proper nouns\n\tpublic String changeCase(String lex,String pos){\n\n\t\t//if(!pos.equals(\"NNP\") && !pos.equals(\"NNPS\") && !lex.equals(\"I\"))lex=lex.toLowerCase();\n\t\treturn lex;\n\n\t}\n\n\t// Creating a freq tally using hashtables. Invoked by LexExtr.xsl.\n\t// Returns whether cat+pos is seen for the first time.\n\tpublic boolean loadTally(String lex, String cat, String pos) {\n\n\t\tString key = catPosKey(cat, pos);\n\t\tCatNode cn;\n\t\tboolean retval;\n\n\t\t//First occurence of cat+pos. Entry made\n\t\tif(!catFreq.containsKey(key)){\n\t\t\tcn = new CatNode(cat, pos);\n\t\t\tcatInfo.put(key, cn);\n\t\t\tcatFreq.put(key,1);\n\t\t\tretval = true;\n\t\t}\n\t\telse { // otherwise inc count\n\t\t\tcn = catInfo.get(key);\n\t\t\tcatFreq.put(key, catFreq.get(key)+1);\n\t\t\tretval = false;\n\t\t}\n\n\t\t// store lex info \n\t\tcn.lexInsert(lex,id);\n\n\t\treturn retval;\n\t}\n\n\n\t/** Returns a string key for a cat and pos. */\n\tpublic static String catPosKey(String cat, String pos) {\n\t\treturn cat+\"-\"+pos;\n\t}\n\n\n\t//Proc which outputs list of map keys in descending order of frequencies\n\tpublic static List sortTally(Map x) { \n\n\t\t// retval\n\t\tList sortedList = new ArrayList();\n\n\t\t//Sorting by freq\n\t\tArrayList vals1 = new ArrayList(x.values());\n\t\tCollections.sort(vals1);\n\n\t\t//Removing unique frequencies to a new arraylist\n\t\tArrayList vals = new ArrayList(vals1.size());\n\t\tint prev = -1;\n\t\tfor (Integer freq : vals1) {\n\t\t\tif (freq != prev) vals.add(freq);\n\t\t\tprev = freq;\n\t\t}\n\n\t\t//Finding all the keys corresponding to a particular freq\n\t\tfor (int i=vals.size()-1; i >=0; i--) {\n\t\t\tint sortedFreq = vals.get(i);\n\t\t\tfor (String key : x.keySet()) {\n\t\t\t\tint freq = x.get(key);\n\t\t\t\tif(freq==sortedFreq) sortedList.add(key);\n\t\t\t}\n\t\t}\n\n\t\treturn sortedList;\n\t}\n\n\tpublic static void printTally(ExtractionProperties extractProps) throws FileNotFoundException {\n\t\tFreqTally.printTally(new File(extractProps.tempDir));\n\t}\n\n\n\tpublic static void printTally(File directory) throws FileNotFoundException {\n\n\t\tSystem.out.println(\"Generating CorpFreq.html\");\n\n\t\t//Freq Output file\n\t\tFile freqFile = new File(directory, \"CorpFreq.html\");\n\t\tPrintWriter output=new PrintWriter(new FileOutputStream(freqFile));\n\t\tList sortedCatKeys = sortTally(catFreq);\n\n\t\t//Printing the final ouput in html form\n\t\toutput.println(\"\");\n\t\toutput.println(\"\");\n\t\toutput.println(\"\");\n\t\toutput.println(\"Lexical Info\");\n\t\toutput.println(\"\");\n\t\toutput.println(\"\");\n\t\toutput.println(\"\");\n\t\toutput.flush();\n\n\t\tfor (int i=0; i < sortedCatKeys.size(); i++) {\n\n\t\t\tString key = sortedCatKeys.get(i);\n\t\t\tCatNode cn = catInfo.get(key);\n\t\t\tString cat = cn.cat;\n\t\t\tString pos = cn.pos;\n\t\t\tint freq = catFreq.get(key);\n\n\t\t\toutput.println(\"

    \");\n\t\t\toutput.println(i+1+\" Category: \"+cat+\" POS: \"+pos+\" Freq: \"+freq);\n\t\t\toutput.println(\"

    \");\n\n\t\t\toutput.println();\n\t\t\tcn.printTally(output);\n\t\t\toutput.flush();\n\t\t}\n\n\t\toutput.println(\"\");\n\t\toutput.println(\"\");\n\t}\n\n\n\t/** Returns whether this lex combo has been seen for the first time. */\n\tpublic boolean firstLexCombo(String lex, String stem, String rel, String cat, String pos,String semClass) {\n\t\tString key = lex + \"_\" + stem + \"_\" + rel + \"_\" + cat + \"_\" + pos + \"_\" + semClass;\n\t\t//String key = lex + \"_\" + stem + \"_\" + rel + \"_\" + cat + \"_\" + pos;\n\n\t\tif (observedLexCombos.contains(key)) return false;\n\t\tobservedLexCombos.add(key);\n\t\treturn true;\n\t}\n\n\n\t// returns the freq for the given key, or 0 if not present\n\tprivate int getFreq(String key) {\n\t\tInteger freq = catFreq.get(key);\n\t\treturn (freq != null) ? freq : 0;\n\t}\n\n\t/** Returns the frequency of the cat and pos. */\n\tpublic int getFreq(String cat, String pos) {\n\t\tString key = catPosKey(cat, pos);\n\t\treturn getFreq(key); \n\t}\n\n\t/** Returns whether the cat and pos pass the frequency cutoff. */\n\tpublic boolean checkFreqStatus(String cat, String pos) {\n\n\t\t/*if(cat.contains(\"Arg\") || cat.startsWith(\"pp[\"))\n\t\t\t\treturn true;*/\n\n\t\t/*if(id.contains(\"wsj_00\"))\n\t\t\t\treturn true;*/\n\n\t\treturn getFreq(cat, pos) >= CAT_FREQ_CUTOFF; \n\t}\n\n\t/** Returns whether the lex, cat and pos pass the frequency cutoffs. */\n\tpublic boolean checkFreqStatus(String lex, String cat, String pos) {\n\t\tString key = catPosKey(cat, pos);\n\t\t//System.out.println(cat);\n\t\tif(cat.contains(\"pp[\"))\n\t\t\treturn true;\n\n\t\t/*if(id.contains(\"wsj_00\"))\n\t\t\treturn true;*/\n\n\t\tif (getFreq(key) < CAT_FREQ_CUTOFF) return false; \n\t\tCatNode cn = catInfo.get(key);\n\t\treturn cn.getLexFreq(lex) >= LEX_FREQ_CUTOFF;\n\t}\n\n\t/** Returns whether the cat and pos are for an open family. */\n\tpublic boolean isOpen(String cat, String pos) {\n\n\t\tif (getFreq(cat, pos) < OPEN_FREQ_CUTOFF) return false;\n\t\tif (pos.startsWith(\"NN\") || pos.equals(\"CD\")) return true;\n\t\telse if (pos.startsWith(\"JJ\") && (cat.equals(\"n_~1/n_1\") || cat.equals(\"s[adj]_1\\np_2\"))) return true;\n\t\telse return false;\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/extract/InsertLFHelper.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Helper Class for insertLF.xsl\n\n//This class is invoked by MorphExtr.xsl \n\npackage opennlp.ccgbank.extract;\n\nimport java.util.ArrayList;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Set;\n\npublic class InsertLFHelper{\n\t\n\tprivate List featTally=new ArrayList();\n\t\n\t//Flush feat tally\n\tpublic String initFeat(){\n\t\tfeatTally.clear();\n\t\treturn null;\n\t}\n\t\n\tpublic String putFeat(String feat){\n\t\tfeatTally.add(feat);\n\t\treturn null;\n\t}\n\t\n\tpublic String getFeat(){\n\t\tString feat=\"\";\n\t\t\n\t\tif (featTally.size() > 0) {\n\t\t\tfeat = featTally.get(0);\n\t\t\tfeatTally.remove(0);\n\t\t}\n\t\telse feat=\"xxx\";\n\t\t\n\t\treturn feat;\n\t}\n\n\t// for ensuring uniqueness of stem/rel pairs\n\tprivate Set stemRelPairs = new HashSet();\n\t\n\t// reset\n\tpublic String resetStemRelPairs() { stemRelPairs.clear(); return null; }\n\t\n\t// contains, updating\n\tpublic boolean containsStemRelPair(String stem, String rel) {\n\t\tString key = stem + \"_\" + rel;\n\t\tif (stemRelPairs.contains(key)) return true;\n\t\tstemRelPairs.add(key);\n\t\treturn false;\n\t}\n\t\n\tprivate String[] rolesArray = {};\n\t\n\t// sets the roles\n\tpublic boolean setRoles(String roles) {\n\t\trolesArray = roles.split(\"\\\\s+\");\n\t\treturn true;\n\t}\n\t\n\t// returns the nth role\n\tpublic String getRole(int n) {\n\t\treturn (n < rolesArray.length) ? rolesArray[n] : \"null\";\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/extract/LexExtract.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Program which creates a temp.xml file from the bareparse. temp.xml serves are the input for creating lexicon.xml & morph.xml\r\n\r\npackage opennlp.ccgbank.extract;\r\n\r\nimport java.io.File;\r\nimport java.io.FileOutputStream;\r\nimport java.io.IOException;\r\nimport java.io.PrintWriter;\r\nimport java.util.Arrays;\r\n\r\nimport javax.xml.transform.Transformer;\r\nimport javax.xml.transform.TransformerConfigurationException;\r\nimport javax.xml.transform.TransformerException;\r\nimport javax.xml.transform.TransformerFactory;\r\nimport javax.xml.transform.sax.SAXResult;\r\nimport javax.xml.transform.sax.SAXSource;\r\nimport javax.xml.transform.sax.SAXTransformerFactory;\r\nimport javax.xml.transform.stream.StreamResult;\r\nimport javax.xml.transform.stream.StreamSource;\r\n\r\nimport opennlp.ccgbank.extract.ExtractGrammar.ExtractionProperties;\r\n\r\nimport org.apache.xml.serializer.OutputPropertiesFactory;\r\nimport org.apache.xml.serializer.Serializer;\r\nimport org.apache.xml.serializer.SerializerFactory;\r\nimport org.jdom.JDOMException;\r\nimport org.xml.sax.InputSource;\r\nimport org.xml.sax.SAXException;\r\nimport org.xml.sax.XMLFilter;\r\nimport org.xml.sax.XMLReader;\r\nimport org.xml.sax.helpers.XMLReaderFactory;\r\n\r\n\r\n/**\r\n * Program which reads in each file of the bare parse xml rep and generates a lexicon, \r\n * a freq tally of the lexical info and a list of ccgbank sentences.\r\n */\r\npublic class LexExtract{\r\n\t\r\n\tpublic static void extractLex(ExtractionProperties extractProps) throws TransformerException,TransformerConfigurationException,SAXException,IOException,JDOMException {\r\n\t\t\r\n\t\tSystem.out.println(\"Extracting lexicon info:\");\r\n\t\t\r\n\t\tFile lexFile = new File(new File(extractProps.destDir), \"lexicon.xml\");\r\n\t\tFile tempFile = new File(new File(extractProps.tempDir), \"temp.xml\");\r\n\t\tPrintWriter tempOut = new PrintWriter(new FileOutputStream(tempFile),true);\r\n\t\t\r\n\t\tFile ccgbankDir = new File(extractProps.srcDir);\r\n\t\tFile[] ccgbankSections=ccgbankDir.listFiles();\r\n\t\tArrays.sort(ccgbankSections);\r\n\t\t\r\n\t\tFreqTally.CAT_FREQ_CUTOFF = extractProps.catFreqCutoff;\r\n\t\tFreqTally.LEX_FREQ_CUTOFF = extractProps.lexFreqCutoff;\r\n\t\tFreqTally.OPEN_FREQ_CUTOFF = extractProps.openFreqCutoff;\r\n\t\t\r\n\t\t//temp.xml creation\r\n\r\n\t\tTransformerFactory tFactory = TransformerFactory.newInstance();\r\n\t\tTransformer lexExtrTransformer = tFactory.newTransformer(ExtractGrammar.getSource(\"opennlp.ccgbank/transform/lexExtr.xsl\"));\r\n\t\t\r\n\t\t// add root \r\n\t\ttempOut.println(\"\");\r\n\t\t\r\n\t\tfor (int i=extractProps.startSection; i<=extractProps.endSection; i++){\r\n\t\t\t\r\n\t\t\tSystem.out.println(\"Section \" + ccgbankSections[i].getName());\r\n\t\t\tFile[] files=ccgbankSections[i].listFiles();\r\n\t\t\tArrays.sort(files);\r\n\t\t\t\r\n\t\t\tint fileStart = 0; int fileLimit = files.length;\r\n\t\t\tif (extractProps.fileNum >= 0) {\r\n\t\t\t\tfileStart = extractProps.fileNum;\r\n\t\t\t\tfileLimit = extractProps.fileNum + 1;\r\n\t\t\t}\r\n\t\t\t\r\n\t\t\tfor (int j=fileStart; j\");\r\n\t\ttempOut.flush();\r\n\t\ttempOut.close();\r\n\t\t\r\n\t\t//Generating a freq tally from static datastructures\r\n\t\tFreqTally.printTally(extractProps);\r\n\t\t\r\n\t\tSystem.out.println(\"Generating lexicon.xml\");\r\n\t\t\r\n\t\tif (tFactory.getFeature(SAXSource.FEATURE) && tFactory.getFeature(SAXResult.FEATURE)) {\r\n\t\t\t\r\n\t\t\tSAXTransformerFactory saxTFactory = ((SAXTransformerFactory) tFactory);\r\n\t\t\t\r\n\t\t\t// Create an XMLFilter for each stylesheet.\r\n\t\t\t\r\n\t\t\t// Extract lexicon from temp.xml\r\n\t\t\tXMLFilter xmlFilter0 = saxTFactory.newXMLFilter(ExtractGrammar.getSource(\"opennlp.ccgbank/transform/filterLex.xsl\"));\r\n\t\t\t\r\n\t\t\tXMLFilter xmlFilter1 = saxTFactory.newXMLFilter(ExtractGrammar.getSource(\"opennlp.ccgbank/transform/closedCatInsert.xsl\"));\r\n\t\t\t\r\n\t\t\tXMLFilter xmlFilter2 = saxTFactory.newXMLFilter(ExtractGrammar.getSource(\"opennlp.ccgbank/transform/insertLF.xsl\"));\r\n\r\n\t\t\tXMLFilter xmlFilter3 = saxTFactory.newXMLFilter(ExtractGrammar.getSource(\"opennlp.ccgbank/transform/insertPunctLF.xsl\"));\r\n\r\n\t\t\tXMLFilter xmlFilter4 = saxTFactory.newXMLFilter(ExtractGrammar.getSource(\"opennlp.ccgbank/transform/insertOrigPunctsLF.xsl\"));\r\n\r\n\t\t\tXMLFilter xmlFilter5 = saxTFactory.newXMLFilter(ExtractGrammar.getSource(\"opennlp.ccgbank/transform/addFilterLexFeats.xsl\"));\r\n\t\t\t\r\n\t\t\tXMLFilter xmlFilter6 = saxTFactory.newXMLFilter(ExtractGrammar.getSource(\"opennlp.ccgbank/transform/insertSemFeats.xsl\"));\r\n\t\t\t\r\n\t\t\tXMLFilter xmlFilter7 = saxTFactory.newXMLFilter(ExtractGrammar.getSource(\"opennlp.ccgbank/transform/markUnmatched.xsl\"));\r\n\t\t\t\r\n\t\t\t// Create an XMLReader.\r\n\t\t\tXMLReader reader = XMLReaderFactory.createXMLReader();\r\n\t\t\t\r\n\t\t\t// xmlFilter0 uses the XMLReader as its reader.\r\n\t\t\txmlFilter0.setParent(reader);\r\n\t\t\txmlFilter1.setParent(xmlFilter0);\r\n\r\n\t\t\txmlFilter2.setParent(xmlFilter1);\r\n\t\t\txmlFilter3.setParent(xmlFilter2);\r\n\r\n\t\t\tif (extractProps.lexF) {\r\n\t\t\t\txmlFilter5.setParent(xmlFilter3);\r\n\t\t\t\txmlFilter6.setParent(xmlFilter5);\r\n\t\t\t} else if (extractProps.origPuncts) {\r\n\t\t\t\txmlFilter4.setParent(xmlFilter2);\r\n\t\t\t\txmlFilter6.setParent(xmlFilter4);\r\n\t\t\t}\r\n\r\n\t\t\telse xmlFilter6.setParent(xmlFilter3);\r\n\r\n\t\t\txmlFilter7.setParent(xmlFilter6);\r\n\t\t\tXMLFilter xmlFilter = xmlFilter7;\r\n\t\t\t\r\n\t\t\tjava.util.Properties xmlProps = OutputPropertiesFactory.getDefaultMethodProperties(\"xml\");\r\n\t\t\txmlProps.setProperty(\"indent\", \"yes\");\r\n\t\t\txmlProps.setProperty(\"standalone\", \"no\"); \r\n \t\txmlProps.setProperty(\"{http://xml.apache.org/xalan}indent-amount\", \"2\");\r\n\t\t\tSerializer serializer = SerializerFactory.getSerializer(xmlProps); \r\n\t\t\tserializer.setOutputStream(new FileOutputStream(lexFile));\r\n\t\t\txmlFilter.setContentHandler(serializer.asContentHandler());\r\n\t\t\txmlFilter.parse(new InputSource(tempFile.getPath()));\r\n\t\t}\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccgbank/extract/MorphExtrHelper.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//This class is invoked by MorphExtr.xsl \n\npackage opennlp.ccgbank.extract;\n\nimport java.util.HashSet;\nimport java.util.Set;\n\npublic class MorphExtrHelper {\n\t\n\tprivate FreqTally aFreqTally = new FreqTally(); \n\tprivate Set seenLexPos = new HashSet();\n\t\n\t/** Returns whether the lex, cat and pos pass the frequency cutoffs, \n\t * and the lex-stem-pos combo is new. */\n\tpublic boolean checkFreqAndNoveltyStatus(String lex, String stem, String cat, String pos,String semClass) {\n\t\tif (!aFreqTally.checkFreqStatus(lex, cat, pos)) return false;\n\t\tString key = lex + \"_\" + stem + \"_\" + \"_\" + pos + \"_\"+ semClass;\n\t\t//String key = lex + \"_\" + stem + \"_\" + \"_\" + pos;\n\t\tif (seenLexPos.contains(key)) return false;\n\t\tseenLexPos.add(key);\n\t\treturn true;\n\t}\n\n\t//Applies rules to discern whether noun is animate or not\n\tpublic String macroNamer(String macro, String semClass,String pos,String lex){\n\n\t\t\tif((semClass.startsWith(\"PER\") && pos.startsWith(\"N\"))||pos.equals(\"DT\")){\n\t\t\t\t\tmacro=macro+\" \"+\"@anim-nom\";\n\t\t\t}\n\t\t\telse if(pos.startsWith(\"PP\") || lex.equals(\"those\") || (pos.startsWith(\"PRP\") && !lex.startsWith(\"it\")))\n\t\t\t\t\tmacro=macro+\" \"+\"@anim-nom\";\n\t\t\telse macro=macro+\" \"+\"@non-anim-nom\";\n\n\t\t\tmacro=macro.trim();\n\n\t\t\t//Skip date time entities from animacy classification\n\t\t\tif(semClass.contains(\"STATE\") || semClass.contains(\"NATION\") || semClass.startsWith(\"ORG_DESC\") || semClass.contains(\"DATE\")||semClass.contains(\"TIME\")||semClass.contains(\"QUANTITY\")||semClass.contains(\"CARDINAL\") || semClass.contains(\"PERCENT\"))\n\t\t\t\t\tmacro=\"\";\n\t\t\t\t\t\t\n\t\t\t//Eliminate collective nouns\n\t\t\tif(lex.equals(\"audience\") || lex.equals(\"band\") || lex.equals(\"group\") || lex.equals(\"team\") || lex.equals(\"club\") || lex.equals(\"congregation\"))\n\t\t\t\t\tmacro=\"\";\n\t\t\t\t\t\n\n\t\t\treturn macro;\n\n\t}\n\n\t//Applies rules to discern whether noun should have number agreement for the copula macro\n\tpublic String agrMacroDecider(String macro,String semClass,String pos,String lex){\n\n\t\t\tif(pos.equals(\"NN\")){\n\t\t\t\t\t\n\t\t\t\t\t//if(lex.equals(\"couple\") || lex.equals(\"following\") ||lex.equals(\"rest\") || semClass.contains(\"STATE\") || semClass.contains(\"NATION\") || semClass.startsWith(\"ORG_DESC\") || semClass.contains(\"DATE\")||semClass.contains(\"TIME\")||semClass.contains(\"QUANTITY\")||semClass.contains(\"CARDINAL\") ||semClass.endsWith(\"'S\"))\n\t\t\t\t\t\n\t\t\t\t\tif(lex.equals(\"couple\") || semClass.startsWith(\"ORG_DESC\") || lex.equals(\"following\") ||lex.equals(\"rest\") || semClass.contains(\"STATE\") || semClass.contains(\"NATION\") || semClass.contains(\"DATE\")||semClass.contains(\"TIME\")||semClass.contains(\"QUANTITY\")||semClass.contains(\"CARDINAL\") || semClass.contains(\"PERCENT\") || semClass.endsWith(\"'S\"))\n\t\t\t\t\t\t\tmacro=\"\";\n\n\t\t\t\t\tif(semClass.length()==0)\n\t\t\t\t\t\t\tmacro=\"\";\n\t\t\t\t\t\t\t\t\t\n\t\t\t}\n\t\t\telse{\n\t\t\t\t\t\n\t\t\t}\n\n\t\t\t//System.out.println(macro);\t\t\n\n\t\t\treturn macro;\n\n\t}\n\n\t\tpublic String whLex=\"\";\n\tpublic void storeWHLex(String whLex){\n\t\t\t//System.out.println(\"Raja: \"+whLex);\n\t\t\tthis.whLex=whLex;\n\t}\t\n\tpublic String getWHLex(){\n\t\t\treturn this.whLex;\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/extract/MorphExtract.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Program which takes in the /tmp/temp.xml file generated and forms a morph.xml file\n\npackage opennlp.ccgbank.extract;\n\nimport java.io.File;\nimport java.io.FileOutputStream;\nimport java.io.IOException;\nimport java.util.ArrayList;\n\nimport javax.xml.transform.TransformerConfigurationException;\nimport javax.xml.transform.TransformerException;\nimport javax.xml.transform.TransformerFactory;\nimport javax.xml.transform.sax.SAXResult;\nimport javax.xml.transform.sax.SAXSource;\nimport javax.xml.transform.sax.SAXTransformerFactory;\n\nimport opennlp.ccgbank.extract.ExtractGrammar.ExtractionProperties;\n\nimport org.apache.xml.serializer.OutputPropertiesFactory;\nimport org.apache.xml.serializer.Serializer;\nimport org.apache.xml.serializer.SerializerFactory;\nimport org.jdom.JDOMException;\nimport org.xml.sax.InputSource;\nimport org.xml.sax.SAXException;\nimport org.xml.sax.XMLFilter;\nimport org.xml.sax.XMLReader;\nimport org.xml.sax.helpers.XMLReaderFactory;\n\npublic class MorphExtract {\n\n\tpublic static void extractMorph(ExtractionProperties extractProps)\n\t\t\tthrows TransformerException, TransformerConfigurationException,\n\t\t\tSAXException, IOException, JDOMException {\n\n\t\tSystem.out.println(\"Extracting morph:\");\n\t\tSystem.out.println(\"Generating morph.xml\");\n\n\t\tTransformerFactory tFactory = TransformerFactory.newInstance();\n\n\t\tFile morphFile = new File(new File(extractProps.destDir), \"morph.xml\");\n\t\tFile tempFile = new File(new File(extractProps.tempDir), \"temp.xml\");\n\n\t\tif (tFactory.getFeature(SAXSource.FEATURE)\n\t\t\t\t&& tFactory.getFeature(SAXResult.FEATURE)) {\n\n\t\t\tSAXTransformerFactory saxTFactory = ((SAXTransformerFactory) tFactory);\n\n\t\t\tArrayList filterChain = new ArrayList();\n\t\t\tArrayList xslChain = new ArrayList();\n\n\t\t\tif (extractProps.macroSpecs.length() > 0) {\n\n\t\t\t}\n\n\t\t\taddTransforms(xslChain, extractProps.macroSpecs);\n\n\t\t\tfor (String xslFile : xslChain)\n\t\t\t\tfilterChain.add(saxTFactory.newXMLFilter(ExtractGrammar\n\t\t\t\t\t\t.getSource(xslFile)));\n\t\t\t// Create an XMLReader and set first xsl transform to that.\n\t\t\tXMLReader reader = XMLReaderFactory.createXMLReader();\n\t\t\tXMLFilter xmlFilter0 = filterChain.get(0);\n\t\t\txmlFilter0.setParent(reader);\n\n\t\t\t//Create chain of xsl transforms\n\t\t\t// Create an XMLFilter for each stylesheet.\n\t\t\tfor (int i = 1; i < filterChain.size(); i++) {\n\t\t\t\tXMLFilter xmlFilterPrev = filterChain.get(i - 1);\n\t\t\t\tXMLFilter xmlFilterCurr = filterChain.get(i);\n\t\t\t\txmlFilterCurr.setParent(xmlFilterPrev);\n\t\t\t}\n\n\t\t\tXMLFilter xmlFilter = filterChain.get(filterChain.size() - 1);\n\n\t\t\tjava.util.Properties xmlProps = OutputPropertiesFactory\n\t\t\t\t\t.getDefaultMethodProperties(\"xml\");\n\t\t\txmlProps.setProperty(\"indent\", \"yes\");\n\t\t\txmlProps.setProperty(\"standalone\", \"no\");\n\t\t\txmlProps.setProperty(\"{http://xml.apache.org/xalan}indent-amount\",\n\t\t\t\t\t\"2\");\n\t\t\tSerializer serializer = SerializerFactory.getSerializer(xmlProps);\n\t\t\tserializer.setOutputStream(new FileOutputStream(morphFile));\n\t\t\t//XMLFilter xmlFilter = xmlFilter2;\n\t\t\t//XMLFilter xmlFilter = xmlFilter3;\n\n\t\t\txmlFilter.setContentHandler(serializer.asContentHandler());\n\t\t\txmlFilter.parse(new InputSource(tempFile.getPath()));\n\t\t}\n\n\t\t//Deleting the temporary lex file\n\t\t//tempFile.delete();\n\t}\n\n\tpublic static void addTransforms(ArrayList xslChain, String macroSpecs) {\n\n\t\txslChain.add(\"opennlp.ccgbank/transform/morphExtr.xsl\");\n\n\t\tif (macroSpecs.length() == 0)\n\t\t\txslChain.add(\"opennlp.ccgbank/transform/macroInsert.xsl\");\n\n\t\tif (macroSpecs.contains(\"agr\")) {\n\t\t\tSystem.out\n\t\t\t\t\t.println(\"Inserting a macro to check agreement in the copula\");\n\t\t\txslChain.add(\"opennlp.ccgbank/transform/agr-macroInsert.xsl\");\n\t\t}\n\n\t\tif (macroSpecs.contains(\"anim\")) {\n\t\t\tSystem.out\n\t\t\t\t\t.println(\"Inserting a macro to check animacy constraints\");\n\t\t\txslChain.add(\"opennlp.ccgbank/transform/anim-macroInsert.xsl\");\n\t\t}\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/extract/RulesExtract.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Program which extracts unary rules and their frequencies and finally outputs the rules.xml file\r\n\r\npackage opennlp.ccgbank.extract;\r\n\r\nimport java.io.File;\r\nimport java.io.FileOutputStream;\r\nimport java.io.IOException;\r\nimport java.io.PrintWriter;\r\nimport java.util.Arrays;\r\n\r\nimport javax.xml.transform.Transformer;\r\nimport javax.xml.transform.TransformerConfigurationException;\r\nimport javax.xml.transform.TransformerException;\r\nimport javax.xml.transform.TransformerFactory;\r\nimport javax.xml.transform.sax.SAXResult;\r\nimport javax.xml.transform.sax.SAXSource;\r\nimport javax.xml.transform.sax.SAXTransformerFactory;\r\nimport javax.xml.transform.stream.StreamResult;\r\nimport javax.xml.transform.stream.StreamSource;\r\n\r\nimport opennlp.ccgbank.extract.ExtractGrammar.ExtractionProperties;\r\n\r\nimport org.apache.xml.serializer.OutputPropertiesFactory;\r\nimport org.apache.xml.serializer.Serializer;\r\nimport org.apache.xml.serializer.SerializerFactory;\r\nimport org.jdom.JDOMException;\r\nimport org.xml.sax.InputSource;\r\nimport org.xml.sax.SAXException;\r\nimport org.xml.sax.XMLFilter;\r\nimport org.xml.sax.XMLReader;\r\nimport org.xml.sax.helpers.XMLReaderFactory;\r\n\r\npublic class RulesExtract {\r\n\t\r\n\tpublic static void extractRules(ExtractionProperties extractProps) throws TransformerException, TransformerConfigurationException,SAXException, IOException,JDOMException{\r\n\t\t\r\n\t\tSystem.out.println(\"Extracting rule info:\");\r\n\t\t\r\n\t\tFile rulesFile = new File(new File(extractProps.destDir), \"rules.xml\");\r\n\t\tFile tempFile = new File(new File(extractProps.tempDir), \"temp-rules.xml\");\r\n\t\tPrintWriter tempOut=new PrintWriter(new FileOutputStream(tempFile),true);\r\n\t\t\r\n\t\tFile ccgbankDir = new File(extractProps.srcDir);\r\n\t\tFile[] ccgbankSections=ccgbankDir.listFiles();\r\n\t\tArrays.sort(ccgbankSections);\r\n\t\t\r\n\t\tRulesTally.RULE_FREQ_CUTOFF = extractProps.ruleFreqCutoff;\r\n RulesTally.KEEP_UNMATCHED = !extractProps.skipUnmatched;\r\n\t\t\r\n\t\t// add root\r\n\t\ttempOut.println(\"\");\r\n\t\t\r\n\t\tTransformerFactory tFactory = TransformerFactory.newInstance();\r\n\t\tTransformer transformer = tFactory.newTransformer(ExtractGrammar.getSource(\"opennlp.ccgbank/transform/rulesExtr.xsl\"));\r\n\t\t\r\n\t\tfor (int i=extractProps.startSection; i<=extractProps.endSection; i++){\r\n\t\t\t\r\n\t\t\tFile[] files=ccgbankSections[i].listFiles();\r\n\t\t\tArrays.sort(files);\r\n\t\t\t\r\n\t\t\tint fileStart = 0; int fileLimit = files.length;\r\n\t\t\tif (extractProps.fileNum >= 0) {\r\n\t\t\t\tfileStart = extractProps.fileNum;\r\n\t\t\t\tfileLimit = extractProps.fileNum + 1;\r\n\t\t\t}\r\n\t\t\t\r\n\t\t\tfor (int j=fileStart; j\");\r\n\t\ttempOut.close();\r\n\t\t\r\n\t\tRulesTally.printTally(extractProps);\r\n\t\t\r\n\t\tSystem.out.println(\"Generating rules.xml\");\r\n\t\t\r\n\t\tif (tFactory.getFeature(SAXSource.FEATURE) && tFactory.getFeature(SAXResult.FEATURE)){\r\n\t\t\t\r\n\t\t\tSAXTransformerFactory saxTFactory = ((SAXTransformerFactory) tFactory);\r\n\t\t\t\r\n\t\t\t// Create an XMLFilter for each stylesheet.\r\n\t\t\tXMLFilter xmlFilter1 = saxTFactory.newXMLFilter(ExtractGrammar.getSource(\"opennlp.ccgbank/transform/ccgRules.xsl\"));\r\n\t\t\t\r\n\r\n\t\t\t//XMLFilter xmlFilter3 = saxTFactory.newXMLFilter(new StreamSource(\"foo3.xsl\"));\r\n\t\t\t\r\n\t\t\t// Create an XMLReader.\r\n\t\t\tXMLReader reader = XMLReaderFactory.createXMLReader();\r\n\t\t\t\r\n\t\t\t// xmlFilter1 uses the XMLReader as its reader.\r\n\t\t\txmlFilter1.setParent(reader);\r\n\t\t\t\r\n\t\t\tjava.util.Properties xmlProps = OutputPropertiesFactory.getDefaultMethodProperties(\"xml\");\r\n\t\t\txmlProps.setProperty(\"indent\", \"yes\");\r\n\t\t\txmlProps.setProperty(\"standalone\", \"no\"); \r\n\t\t\txmlProps.setProperty(\"{http://xml.apache.org/xalan}indent-amount\", \"2\");\r\n\t\t\tSerializer serializer = SerializerFactory.getSerializer(xmlProps);\r\n\t\t\tserializer.setOutputStream(new FileOutputStream(rulesFile));\r\n\r\n\r\n\t\t\tXMLFilter xmlFilter = xmlFilter1;\r\n\t\t\txmlFilter.setContentHandler(serializer.asContentHandler());\r\n\t\t\txmlFilter.parse(new InputSource(tempFile.getPath()));\r\n\t\t}\r\n\t\t\r\n\t\t//Deleting the temporory lex file\r\n\t\t//lexiconTempFile.delete();\r\n\t}\r\n}\r\n" }, { "path": "src/opennlp/ccgbank/extract/RulesTally.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Class which stores unary rule freqs\n\n//This class is invoked by the RulesExtr.xsl transform\n\npackage opennlp.ccgbank.extract;\n\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.FileOutputStream;\nimport java.io.PrintWriter;\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.StringTokenizer;\n\nimport opennlp.ccgbank.extract.ExtractGrammar.ExtractionProperties;\n\npublic class RulesTally {\n\t\n\t// Frequency cutoff for including an extracted rule \n\tpublic static int RULE_FREQ_CUTOFF = 1;\n \n // Flag for whether to keep unmatched rules in extracted grammar\n public static boolean KEEP_UNMATCHED = true;\n\t\n\t//Unary Rule Freq\n\tprivate static Map ruleFreq = new HashMap();\n\t\n\t//Rule Occurrence\n\tprivate static Map> ruleOccur = new HashMap>();\n\t\n\t//Sentence id\n\tprivate static String id=\"\";\n\n\t/** Resets the statically held tallies. */\n\tpublic static void reset() {\n\t\truleFreq = new HashMap();\n\t\truleOccur = new HashMap>();\n\t\tid=\"\";\n\t}\n\n\t\n\t//Proc which traps and stores id of each sentence\n\tpublic String storeId(String x){\n\t\tif(x.length()>0){\n\t\t\tid = x;\n\t\t\tint posEquals = x.indexOf('=');\n\t\t\tif (posEquals > 0) {\n\t\t\t\tid = x.substring(posEquals+1);\n\t\t\t}\n\t\t}\n\t\treturn id;\n\t}\n\t\n\t//Loads freq tables, returns rule name\n\tpublic String loadTally(String res, String arg) throws FileNotFoundException{\n\t\t\n\t\targ = arg.replaceAll(\"_\\\\d\", \"\");\n\t\tres = res.replaceAll(\"_\\\\d\", \"\");\n\t\tString rule = arg+\"_to_\"+res;\n\t\tList temp;\n\t\t\n\t\t//Freq table entry opened at first instance of rule\n\t\tif (!ruleFreq.containsKey(rule)) {\n\t\t\truleFreq.put(rule,1);\n\t\t\ttemp = new ArrayList(4);\n\t\t\ttemp.add(id);\n\t\t\truleOccur.put(rule,temp);\n\t\t}\n\t\telse {\n\t\t\tint freq = ruleFreq.get(rule)+1;\n\t\t\truleFreq.put(rule,freq);\n\t\t\ttemp = ruleOccur.get(rule);\n\t\t}\n\t\t\n\t\t//First 4 instances of rules stored\n\t\tif(temp.size()<4 && !temp.contains(id)){\n\t\t\ttemp.add(id);\n\t\t\truleOccur.put(rule,temp);\n\t\t}\n\t\t\n\t\treturn rule;\n\t}\n\t\n\tpublic static void printTally(ExtractionProperties extractProps) throws FileNotFoundException {\n\t\tRulesTally.printTally(new File(extractProps.tempDir));\n\t}\n\t\n\t//Rule Frequencies printed to file\n\tpublic static void printTally(File directory) throws FileNotFoundException{\n\t\t\n\t\tSystem.out.println(\"Generating RuleFreq.html\");\n\t\t\n\t\t//Freq Output file\n\t\tFile freqFile = new File(directory, \"RuleFreq.html\");\n\t\tPrintWriter output=new PrintWriter(new FileOutputStream(freqFile));\n\t\t\n\t\tList ruleList = FreqTally.sortTally(ruleFreq);\n\t\t\n\t\t//Printing the final ouput in html form\n\t\toutput.flush();\n\t\toutput.println(\"\");\n\t\toutput.println(\"\");\n\t\toutput.println(\"\");output.println(\"Unary Rule Info\");output.println(\"\");\n\t\toutput.println(\"\");\n\t\toutput.println(\"\");\n\t\toutput.flush();\n\t\t\n\t\tString ccgbankHome = System.getProperty(\"CCGBANK_HOME\", \"/home/corpora/EN/ccgbank\");\n\t\t\n\t\tfor (int i=0; i\");\n\t\t\toutput.println(i+1+\" Rule: \"+rule+\" Freq: \"+freq);\n\t\t\toutput.println(\"

    \");\n\t\t\toutput.flush();\n\t\t\t\n\t\t\tList rules = ruleOccur.get(rule);\n\t\t\toutput.flush();\n\t\t\t\n\t\t\toutput.println(\"

      \");\n\t\t\toutput.println(\"
    • \");\n\t\t\toutput.flush();\n\t\t\t\n\t\t\tfor (int j=0; j\");\n\t\t\t\tid=rules.get(j);\n\n\t\t\t\tString[]idInfo=id.split(\"\\\\.\");\n\n\t\t\t\tStringTokenizer st=new StringTokenizer(id,\".\");\n\n\t\t\t\tString idLink=\"\";\n\t\t\t\tString sentNo=\"\";\n\t\t\t\tString dir=\"\";\n\n\t\t\t\t//2 courses of action depending on whether input is gold std .auto parses or C&C .auto parses\n\t\t\t\tif(idInfo.length==2){\n\t\t\t\t\t\tidLink=st.nextToken()+\".html\";\n\t\t\t\t\t\tsentNo=\"#Sentence \"+st.nextToken();\n\t\t\t\t\t\tdir=id.substring(4,6);\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\t\tidLink=idInfo[0];\n\t\t\t\t\t\tsentNo=\"#Sentence \"+idInfo[0];\n\t\t\t\t\t\tdir=idInfo[0];\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\t//System.out.println(idLink);\n\t\t\t\t\n\t\t\t\toutput.println(\"
    • \");\n\t\t\t\toutput.println(id);\n\t\t\t\toutput.println(\" \");\n\t\t\t\toutput.println(idLink);\n\t\t\t\toutput.println(\"\");\n\t\t\t\toutput.println(\"
      • \");\n\t\t\t\toutput.println(\"
    \");\n\t\t\t}\n\t\t\t\t\t\t\n\t\t\toutput.flush();\n\t\t\toutput.println(\"\");\n\t\t\toutput.println(\"\");\n\t\t\toutput.println(\"\");\n\t\t\toutput.flush();\n\t\t}\n\t\t\n\t\toutput.flush();\n\t\toutput.println(\"\");\n\t\toutput.println(\"\");\n\t\toutput.flush();\n\t\toutput.close();\n\t}\n\t\n\t\n\t//Invoked by RulesExtr.xsl to check repetition of categories\n\tpublic boolean checkRuleStatus(String rule) {\n\t\tInteger freq = ruleFreq.get(rule);\n\t\treturn (freq != null && freq == 1);\n\t}\n\t\n\t//Checks the freq of a rule\n\tpublic boolean checkRuleFreqStatus(String rule){\n\t\tint freq = ruleFreq.get(rule);\n\t\t//Freqs >= cutoff accepted\n\t\treturn (freq >= RULE_FREQ_CUTOFF);\n\t}\n \n // returns flag\n public boolean keepUnmatched() {\n return KEEP_UNMATCHED;\n }\n}\n" }, { "path": "src/opennlp/ccgbank/extract/Testbed.java", "content": "///////////////////////////////////////////////////////////////////////////////\n// Copyright (C) 2005-2009 Scott Martin, Rajakrishan Rajkumar and Michael White\n// \n// This library is free software; you can redistribute it and/or\n// modify it under the terms of the GNU Lesser General Public\n// License as published by the Free Software Foundation; either\n// version 2.1 of the License, or (at your option) any later version.\n// \n// This library is distributed in the hope that it will be useful,\n// but WITHOUT ANY WARRANTY; without even the implied warranty of\n// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n// GNU Lesser General Public License for more details.\n// \n// You should have received a copy of the GNU Lesser General Public\n// License along with this program; if not, write to the Free Software\n// Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.\n//////////////////////////////////////////////////////////////////////////////\n\n//Program which creates a temp.xml file from the bareparse. temp.xml serves are the input for creating lexicon.xml & morph.xml\n\npackage opennlp.ccgbank.extract;\n\nimport java.io.*;\nimport java.util.*;\n\nimport opennlp.ccgbank.CCGBankTaskSources;\nimport opennlp.ccgbank.CCGBankTaskTestbed;\nimport opennlp.ccg.grammar.Grammar;\nimport opennlp.ccg.grammar.RuleGroup;\nimport opennlp.ccg.hylo.*;\nimport opennlp.ccg.lexicon.*;\nimport opennlp.ccg.parse.ParseException;\nimport opennlp.ccg.synsem.*;\nimport opennlp.ccg.test.*;\nimport opennlp.ccg.unify.*;\n\nimport org.jdom.Document;\nimport org.jdom.Element;\nimport org.jdom.JDOMException;\nimport org.jdom.input.SAXBuilder;\nimport org.jdom.output.Format;\nimport org.jdom.output.XMLOutputter;\n\n/**\n * Creates test files under in 'test' dir under extracted grammar.\n */\npublic class Testbed {\n\n\t// the grammar\n\tprivate Grammar grammar;\n\tprivate Lexicon lexicon;\n\tprivate RuleGroup rules;\n\n\t// supertagger stand-in\n\tprivate SupertaggerStandIn supertaggerStandIn = new SupertaggerStandIn();\n\t\n\t// results of following deriv\n\tprivate Sign sign = null;\n\n\tprivate LF lf = null;\n\n\tprivate String str = \"\";\n\n\tprivate int numParses = 0;\n\n\tprivate String header = \"\";\n\n\t// Store info related to treenodes in the xml deriv\n\tprivate static ArrayList treeInfo = new ArrayList();\n\n\tprivate static boolean treeInfoFlag = false;\n\n\t// Store details of preds (nomId key: pos,stag, pos and pred name)\n\tprivate static Hashtable predInfo = new Hashtable();\n\n\t// supertag-rule combos\n\tprivate Set combos = null;\n\n\tSet sourcesSet;\n\n\tCCGBankTaskTestbed ccgBankTaskTestbed;\n\n\tFile grammarFile, targetDirectory;\n\n\t// constructor\n\tpublic Testbed(Set sourcesSet, File targetDirectory,\n\t\t\tCCGBankTaskTestbed testbed) throws IOException {\n\t\tgrammarFile = new File(targetDirectory, \"grammar.xml\");\n\n\t\tthis.grammar = new Grammar(grammarFile.toURI().toURL(), true);\n\t\tthis.lexicon = grammar.lexicon;\n\t\tthis.rules = grammar.rules;\n\n\t\tthis.sourcesSet = sourcesSet;\n\t\tthis.targetDirectory = targetDirectory;\n\t\tthis.ccgBankTaskTestbed = testbed;\n\t}\n\n\t// main method for creating test files\n\t@SuppressWarnings(\"rawtypes\")\n\tpublic void createTestFiles() throws IOException, JDOMException {\n\n\t\tccgBankTaskTestbed.log(\"Creating test files:\");\n\n\t\t// config grammar\n\t\tTokenizer tokenizer = grammar.lexicon.tokenizer;\n\t\tgrammar.prefs.showFeats = true;\n\t\tgrammar.prefs.showSem = ccgBankTaskTestbed.isShowsSem();\n\n\t\t// ensure test dir exists\n\t\tFile testDir = new File(targetDirectory, \"test\");\n\t\ttestDir.mkdirs();\n\t\tccgBankTaskTestbed.log(\"Writing test files to: \" + testDir.getPath());\n\n\t\t// text, class-replaced text factors etc. output\n\t\tPrintWriter textPW = null;\n\t\tPrintWriter textscPW = null;\n\t\tPrintWriter factorsPW = null;\n\t\tPrintWriter combosPW = null;\n\t\tPrintWriter predsPW = null;\n\t\tPrintWriter treePW = null;\n\t\tFile textFile = ccgBankTaskTestbed.getText();\n\t\tFile factorsFile = ccgBankTaskTestbed.getFactors();\n\t\tFile combosFile = ccgBankTaskTestbed.getCombos(); \n\t\tFile predsFile = ccgBankTaskTestbed.getPreds(); \n\t\tFile treeFile = ccgBankTaskTestbed.getTree();\n\t\t\n\t\tif (textFile != null) {\n\t\t\tFile textscFile=new File(textFile.getParent()+\"/\"+textFile.getName().replaceFirst(\"text-\",\"textsc-\"));\n\t\t\tccgBankTaskTestbed.log(\"Writing text to: \" + textFile);\n\t\t\tccgBankTaskTestbed.log(\"Writing class-replaced text to: \" + textscFile);\n textFile.getParentFile().mkdirs(); \n\t\t\ttextPW = new PrintWriter(new BufferedWriter(new FileWriter(textFile)));\n\t\t\ttextscPW = new PrintWriter(new BufferedWriter(new FileWriter(textscFile)));\n\t\t}\n\t\tif (factorsFile != null) {\n\t\t\tccgBankTaskTestbed.log(\"Writing factors to: \" + factorsFile);\n factorsFile.getParentFile().mkdirs();\n\t\t\tfactorsPW = new PrintWriter(new BufferedWriter(new FileWriter(factorsFile)));\n\t\t}\n\t\tif (combosFile != null) {\n\t\t\tccgBankTaskTestbed.log(\"Writing supertag-rule combos to: \" + combosFile);\n combosFile.getParentFile().mkdirs(); \n\t\t\tcombos = new HashSet();\n\t\t\tcombosPW = new PrintWriter(new BufferedWriter(new FileWriter(combosFile)));\n\t\t}\n\t\tif (predsFile != null) {\n\t\t\tccgBankTaskTestbed.log(\"Writing preds to: \" + predsFile);\n\t\t\tpredsFile.getParentFile().mkdirs();\n\t\t\tpredsPW = new PrintWriter(new BufferedWriter(new FileWriter(predsFile)));\n\t\t}\n\t\tif (treeFile != null) {\n\t\t\tccgBankTaskTestbed.log(\"Writing tree node info to: \" + treeFile);\n treeFile.getParentFile().mkdirs();\n\t\t\ttreePW = new PrintWriter(new BufferedWriter(new FileWriter(treeFile)));\n\t\t\ttreeInfoFlag = true;\n\t\t}\n\n\t\t// jdom stuff\n\t\tSAXBuilder builder = new SAXBuilder();\n\t\tXMLOutputter outputter = new XMLOutputter(Format.getPrettyFormat());\n\n\t\t// counters\n\t\tint numWithLFs = 0;\n\t\tint numSingleRootLFs = 0;\n\t\tint numWithoutLFs = 0;\n\n\t\tfor (CCGBankTaskSources sources : sourcesSet) {\n\t\t\tfor (File file : sources) {\n\t\t\t\tFile testSectDir = new File(testDir, file.getParentFile().getName());\n\t\t\t\ttestSectDir.mkdir();\n\n\t\t\t\tccgBankTaskTestbed.log(\"Debug Print: \" + testSectDir.getAbsolutePath());\n\n\t\t\t\t// parse derivations\n\t\t\t\tDocument inDoc = builder.build(file);\n\t\t\t\tElement inRoot = inDoc.getRootElement();\n\n\t\t\t\t// make test doc, sign map\n\t\t\t\tDocument outDoc = new Document();\n\t\t\t\tElement outRoot = new Element(\"regression\");\n\t\t\t\toutDoc.setRootElement(outRoot);\n\t\t\t\tMap signMap = new HashMap();\n\n\t\t\t\t// loop through derivations, making test items\n\t\t\t\tList derivElts = inRoot.getChildren();\n\t\t\t\tfor (Object derivObj : derivElts) {\n\t\t\t\t\tElement derivElt = (Element) derivObj;\n\t\t\t\t\tfollowDeriv(derivElt);\n\t\t\t\t\tif (lf != null) {\n\t\t\t\t\t\tnumWithLFs++;\n\t\t\t\t\t\t// check for single root\n\t\t\t\t\t\tif (lf instanceof SatOp)\n\t\t\t\t\t\t\tnumSingleRootLFs++;\n\t\t\t\t\t\t// add test item, sign\n\t\t\t\t\t\tElement item = RegressionInfo.makeTestItem(grammar, str, numParses, lf);\n\t\t\t\t\t\tif (header == null) {\n\t\t\t\t\t\t\theader = \"missing\";\n\t\t\t\t\t\t\tccgBankTaskTestbed.log(\"Warning: missing header in \" + file);\n\t\t\t\t\t\t}\n\t\t\t\t\t\titem.setAttribute(\"info\", header);\n\t\t\t\t\t\tif (header != null) signMap.put(header, sign);\n\n\t\t\t\t\t\t// Add parsed words as a separate LF element\n\t\t\t\t\t\tElement fullWordsElt = new Element(\"full-words\");\n\t\t\t\t\t\tfullWordsElt.addContent(tokenizer.format(sign.getWords()));\n\n\t\t\t\t\t\t// Add info about LF lexical preds as a separate element\n\t\t\t\t\t\tElement predInfoElt = new Element(\"pred-info\");\n\t\t\t\t\t\tString predInfoText = collectPredInfo(header);\n\t\t\t\t\t\tpredInfoElt.setAttribute(\"data\", predInfoText);\n\n\t\t\t\t\t\titem.addContent(fullWordsElt);\n\t\t\t\t\t\titem.addContent(predInfoElt);\n\t\t\t\t\t\toutRoot.addContent(item);\n\n\t\t\t\t\t\t// append to text, factors files\n\t\t\t\t\t\tif (textPW != null)\n\t\t\t\t\t\t\ttextPW.println(str);\n\t\t\t\t\t\t\n\t\t\t\t\t\t if (textscPW != null) {\n textscPW.flush();\n String textsc=\"\";\n //Note sem class replacement works only for NE classes spec in the grammar file\n textsc=tokenizer.getOrthography((List)sign.getWords(),true);\n textscPW.println(textsc);\n textscPW.flush();\n\t\t\t\t\t\t }\n\n\t\t\t\t\t\t\n\t\t\t\t\t\tif (factorsPW != null)\n\t\t\t\t\t\t\tfactorsPW.println(tokenizer.format(sign.getWords()));\n\t\t\t\t\t\t// append new combos to combos file\n\t\t\t\t\t\tif (combosPW != null) {\n\t\t\t\t\t\t\tfor (String combo : newCombos()) combosPW.println(combo);\n\t\t\t\t\t\t}\n\t\t\t\t\t\t// also to preds\n\t\t\t\t\t\tif (predsPW != null)\n\t\t\t\t\t\t\tpredsPW.println(predInfoText);\n\n\t\t\t\t\t\tif (treePW != null) {\n\t\t\t\t\t\t\tfor (String info : treeInfo) {\n\t\t\t\t\t\t\t\ttreePW.println(info);\n\t\t\t\t\t\t\t\ttreePW.flush();\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t\ttreeInfo = new ArrayList();\n\t\t\t\t\t} else\n\t\t\t\t\t\tnumWithoutLFs++;\n\t\t\t\t}\n\n\t\t\t\t// write test doc, saved signs\n\t\t\t\tFile regressionFile = new File(testSectDir, file.getName());\n\t\t\t\toutputter.output(outDoc, new FileOutputStream(regressionFile));\n\t\t\t\tRegressionInfo.writeSerFile(signMap, regressionFile);\n\t\t\t}\n\t\t}\n\n\t\t// flush text, factors, combos, preds, tree files\n\t\tif (textPW != null) {\n\t\t\ttextPW.flush();\n\t\t\ttextPW.close();\n\t\t}\n\t\tif (factorsPW != null) {\n\t\t\tfactorsPW.flush();\n\t\t\tfactorsPW.close();\n\t\t}\n\t\tif (combosPW != null) {\n\t\t\tcombosPW.flush();\n\t\t\tcombosPW.close();\n\t\t}\n\t\tif (predsPW != null) {\n\t\t\tpredsPW.flush();\n\t\t\tpredsPW.close();\n\t\t}\n\t\tif (treePW != null) {\n\t\t\ttreePW.flush();\n\t\t\ttreePW.close();\n\t\t}\n\n\t\t// summary\n\t\tccgBankTaskTestbed.log(\"numWithLFs: \" + numWithLFs);\n\t\tccgBankTaskTestbed.log(\"numSingleRootLFs: \" + numSingleRootLFs);\n\t\tccgBankTaskTestbed.log(\"numWithoutLFs: \" + numWithoutLFs);\n\t\tccgBankTaskTestbed.log(\"total: \" + (numWithLFs + numWithoutLFs));\n\t}\n\n\tprivate void followDeriv(Element derivElt) {\n\t\t\n\t\t// reset\n\t\tsign = null;\n\t\tlf = null;\n\t\tstr = \"\";\n\t\theader = derivElt.getAttributeValue(\"Header\");\n\t\t\n\t\t// bookkeeping\n\t\tUnifyControl.startUnifySequence();\n\t\t\n\t\ttry {\n\n\t\t\tCategory cat = null;\n\t\t\tNominal index = null;\n\t\t\tLF flatLF = null;\n\n\t\t\t// recurse through deriv\n\t\t\tSignHash signs = followDerivR(derivElt);\n\t\t\t// set results, using first available sign (ie some arbitrary one)\n\t\t\tif (!signs.isEmpty()) {\n\t\t\t\tIterator iter = signs.asSignSet().iterator();\n\t\t\t\t// System.out.println(\"Processing file no: \"+header);\n\t\t\t\t// Count of single rooted LFs produced by the constrained parser\n\t\t\t\tint matchSRLF = 0;\n\t\t\t\t// Check whether any of the signs have a single rooted LF\n\t\t\t\twhile (iter.hasNext()) {\n\t\t\t\t\t// System.out.println(\"Found LF\");\n\t\t\t\t\tsign = iter.next();\n\t\t\t\t\tcat = sign.getCategory();\n\t\t\t\t\tindex = cat.getIndexNominal();\n\t\t\t\t\tflatLF = cat.getLF();\n\t\t\t\t\tif (flatLF != null) {\n\t\t\t\t\t\tlf = HyloHelper.compactAndConvertNominals(flatLF, index, sign);\n\t\t\t\t\t\t// Break when the first single rooted LF is encountered\n\t\t\t\t\t\tif (lf instanceof SatOp) {\n\t\t\t\t\t\t\tmatchSRLF++;\n\t\t\t\t\t\t\t// System.out.println(\"Single root LF found\");\n\t\t\t\t\t\t\tbreak;\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\n\t\t\t\t// If no single rooted LF is there, using first available sign\n\t\t\t\t// (ie some arbitrary one)\n\t\t\t\tif (matchSRLF == 0) {\n\t\t\t\t\tsign = signs.asSignSet().iterator().next();\n\t\t\t\t\tcat = sign.getCategory();\n\t\t\t\t\tindex = cat.getIndexNominal();\n\t\t\t\t\tflatLF = cat.getLF();\n\t\t\t\t\tif (flatLF != null) lf = HyloHelper.compactAndConvertNominals(flatLF, index, sign);\n\t\t\t\t}\n\n\t\t\t\tif (flatLF != null) {\n\t\t\t\t\textrPredInfo(flatLF, \"\");\n\t\t\t\t}\n\n\t\t\t\tnumParses = signs.size();\n\t\t\t\tstr = str.trim();\n\t\t\t}\n\t\t\t\n\t\t} catch (ParseException exc) {\n\t\t\tccgBankTaskTestbed.log(\"Warning for \" + header + \": \" + exc.toString());\n\t\t}\n\t}\n\n\t// recurse through deriv, returning signs\n\t@SuppressWarnings({ \"rawtypes\", \"unchecked\" })\n\tprivate SignHash followDerivR(Element derivElt) throws ParseException {\n\t\tString eltName = derivElt.getName();\n\t\t// follow deriv, applying combinatory rules\n\t\t// nb: no checks made for intended deriv!\n\t\tif (eltName.equals(\"Treenode\")) {\n\t\t\tString cat = derivElt.getAttributeValue(\"cat\");\n\t\t\tString ntId = derivElt.getAttributeValue(\"nt_id\");\n\t\t\tString simpleCat = derivElt.getAttributeValue(\"stag\");\n\t\t\tList childElts = derivElt.getChildren();\n\t\t\tint numChildren = childElts.size();\n\t\t\tif (numChildren == 0)\n\t\t\t\tthrow new ParseException(header\n\t\t\t\t\t\t+ \": no child elements for TreeNode for cat: \" + cat);\n\t\t\t// if no cat element present, adjust list with an initial dummy node, \n\t\t\t// to avoid code changes in what follows\n\t\t\tElement elt0 = (Element) childElts.get(0);\n\t\t\tString elt0name = elt0.getName();\n\t\t\tif (elt0name.equals(\"Treenode\") || elt0name.equals(\"Leafnode\")) {\n\t\t\t\tchildElts.add(0, new Element(\"dummy\"));\n\t\t\t\tnumChildren++;\n\t\t\t}\n\t\t\tif (numChildren != 2 && numChildren != 3)\n\t\t\t\tthrow new ParseException(header\n\t\t\t\t\t\t+ \": wrong number of child elements: \" + numChildren\n\t\t\t\t\t\t+ \" for cat: \" + cat);\n\t\t\tElement firstInputElt = (Element) childElts.get(1);\n\t\t\tSignHash firstSigns = followDerivR(firstInputElt);\n\t\t\tSignHash retval = new SignHash();\n\t\t\t// unary case\n\t\t\tif (numChildren == 2) {\n\t\t\t\t// apply rules\n\t\t\t\tfor (Sign s : firstSigns.asSignSet()) {\n\t\t\t\t\tList results = rules.applyUnaryRules(s);\n\t\t\t\t\tfor (Sign rSign : results)\n\t\t\t\t\t\tretval.insert(rSign);\n\t\t\t\t}\n\t\t\t\t// caution/warn upon failure\n\t\t\t\tif (!containsCat(retval, simpleCat)) {\n\t\t\t\t\tboolean noResults = retval.isEmpty();\n\t\t\t\t\tString inCat = firstInputElt.getAttributeValue(\"cat\");\n\t\t\t\t\tString msg = \"Unable to derive: \" + cat + \" from: \" + inCat;\n\t\t\t\t\tif (!noResults)\n\t\t\t\t\t\tccgBankTaskTestbed.log(\"Caution for \" + header + \": \" + msg);\n\t\t\t\t\tif (ccgBankTaskTestbed.isDebugDerivations()) {\n\t\t\t\t\t\tccgBankTaskTestbed.log(header + \": derivation stymied; inputs: \");\n\t\t\t\t\t\tfor (Sign s : firstSigns.asSignSet()) {\n\t\t\t\t\t\t\tccgBankTaskTestbed.log(s.toString());\n\t\t\t\t\t\t}\n\t\t\t\t\t\tif (!noResults) {\n\t\t\t\t\t\t\tccgBankTaskTestbed.log(\"Outputs: \");\n\t\t\t\t\t\t\tfor (Sign s : retval.asSignSet())\n\t\t\t\t\t\t\t\tccgBankTaskTestbed.log(s.toString());\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\tif (noResults)\n\t\t\t\t\t\tthrow new ParseException(\"Derivation blocked: \" + msg);\n\t\t\t\t}\n\t\t\t}\n\t\t\t// binary case\n\t\t\telse if (numChildren == 3) {\n\t\t\t\tElement secondInputElt = (Element) childElts.get(2);\n\t\t\t\tSignHash secondSigns = followDerivR(secondInputElt);\n\t\t\t\t// apply rules\n\t\t\t\tfor (Sign sign1 : firstSigns.asSignSet()) {\n\t\t\t\t\tfor (Sign sign2 : secondSigns.asSignSet()) {\n\t\t\t\t\t\tList results = rules.applyBinaryRules(sign1, sign2);\n\t\t\t\t\t\tfor (Sign rSign : results)\n\t\t\t\t\t\t\tretval.insert(rSign);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\t// if no results, propagate one input if the other is\n\t\t\t\t// internal punct\n\t\t\t\tif (retval.isEmpty()) {\n\t\t\t\t\tif (isPunct(secondInputElt))\n\t\t\t\t\t\treturn firstSigns;\n\t\t\t\t\telse if (isPunct(firstInputElt))\n\t\t\t\t\t\treturn secondSigns;\n\t\t\t\t}\n\t\t\t\t// caution/warn upon failure\n\t\t\t\tif (!containsCat(retval, simpleCat)) {\n\t\t\t\t\tboolean noResults = retval.isEmpty();\n\t\t\t\t\tString inCat1 = firstInputElt.getAttributeValue(\"cat\");\n\t\t\t\t\tString inCat2 = secondInputElt.getAttributeValue(\"cat\");\n\t\t\t\t\tString msg = \"Unable to derive: \" + cat + \" from: \" + inCat1 + \" and: \" + inCat2;\n\t\t\t\t\tif (!noResults)\n\t\t\t\t\t\tccgBankTaskTestbed.log(\"Caution for \" + header + \": \" + msg);\n\t\t\t\t\tif (ccgBankTaskTestbed.isDebugDerivations()) {\n\t\t\t\t\t\tccgBankTaskTestbed.log(header + \": derivation stymied; first inputs: \");\n\t\t\t\t\t\tfor (Sign sign1 : firstSigns.asSignSet()) {\n\t\t\t\t\t\t\tccgBankTaskTestbed.log(sign1.toString());\n\t\t\t\t\t\t}\n\t\t\t\t\t\tccgBankTaskTestbed.log(\"Second inputs: \");\n\t\t\t\t\t\tfor (Sign sign2 : secondSigns.asSignSet()) {\n\t\t\t\t\t\t\tccgBankTaskTestbed.log(sign2.toString());\n\t\t\t\t\t\t}\n\t\t\t\t\t\tif (!noResults) {\n\t\t\t\t\t\t\tccgBankTaskTestbed.log(\"Outputs: \");\n\t\t\t\t\t\t\tfor (Sign s : retval.asSignSet())\n\t\t\t\t\t\t\t\tccgBankTaskTestbed.log(s.toString());\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\tif (noResults)\n\t\t\t\t\t\tthrow new ParseException(\"Derivation blocked: \" + msg);\n\t\t\t\t}\n\t\t\t}\n\n\t\t\t// Store cat ids of tree nodes for printing to aux files\n\t\t\tif (treeInfoFlag) {\n\n\t\t\t\tfor (Sign s : retval.asSignSet()) {\n\n\t\t\t\t\tHashtable idConvTally = new Hashtable();\n\t\t\t\t\tHashtable freqTally = new Hashtable();\n\t\t\t\t\tArrayList fullCat = new ArrayList();\n\t\t\t\t\tString catId = \"\";\n\n\t\t\t\t\tCategory treeCat = s.getCategory();\n\t\t\t\t\t// System.out.println(header+\" \"+ntId+\" \"+treeCat);\n\t\t\t\t\trecurseCat(treeCat, fullCat, idConvTally, freqTally);\n\t\t\t\t\t/*\n\t\t\t\t\t * System.out.println(freqTally);\n\t\t\t\t\t * System.out.println(fullCat); System.out.println('\\n');\n\t\t\t\t\t */\n\n\t\t\t\t\tif (fullCat.size() > 1) {\n\n\t\t\t\t\t\tfor (String x : fullCat) {\n\n\t\t\t\t\t\t\tString y[] = x.split(\"_\");\n\t\t\t\t\t\t\tif (y.length == 1) {\n\t\t\t\t\t\t\t\tcatId = catId + \",\" + y[0];\n\t\t\t\t\t\t\t\tcontinue;\n\t\t\t\t\t\t\t}\n\n\t\t\t\t\t\t\tint freq = freqTally.get(y[1]);\n\t\t\t\t\t\t\tfreqTally.put(y[1], freq - 1);\n\n\t\t\t\t\t\t\tif (x.endsWith(\"_M\") && freq <= 1)\n\t\t\t\t\t\t\t\tx = x.replaceFirst(\"_M\", \"\");\n\n\t\t\t\t\t\t\tcatId = catId + \",\" + x;\n\t\t\t\t\t\t}\n\t\t\t\t\t\tcatId = catId.replaceFirst(\",\", \"\");\n\t\t\t\t\t\ttreeInfo.add(header + \" \" + ntId + \" \" + catId);\n\t\t\t\t\t}\n\t\t\t\t\t/*\n\t\t\t\t\t * System.out.println(idConvTally);\n\t\t\t\t\t * System.out.println(fullCat); System.out.println('\\n');\n\t\t\t\t\t */\n\t\t\t\t}\n\n\t\t\t}\n\n\t\t\t// done\n\t\t\treturn retval;\n\t\t}\n\t\t// lex lookup\n\t\t// nb: not always insisting on right POS, b/c hashing strategy uses\n\t\t// surface words,\n\t\t// thus doesn't distinguish lex signs based solely on POS\n\t\t// nb: might make sense to warn on lex cats with missing semantics\n\t\telse if (eltName.equals(\"Leafnode\")) {\n\t\t\ttry {\n\t\t\t\tString lex = derivElt.getAttributeValue(\"lexeme\");\n\t\t\t\tWord w = lexicon.tokenizer.parseToken(lex);\n\t\t\t\tstr += w.getForm() + \" \";\n\t\t\t\tString cat = derivElt.getAttributeValue(\"cat\");\n\t\t\t\tString simpleCat = derivElt.getAttributeValue(\"stag\");\n\t\t\t\tString rel = derivElt.getAttributeValue(\"rel\");\n\t\t\t\tString indexRel = derivElt.getAttributeValue(\"indexRel\");\n\t\t\t\tString semClass = \"\";\n\t\t\t\tsemClass = derivElt.getAttributeValue(\"class\");\n\n\t\t\t\tString roles = derivElt.getAttributeValue(\"argRoles\");\n\t\t\t\tString pos = derivElt.getAttributeValue(\"pos\");\n\t\t\t\t// nb: for now, need to ignore rel for non-VB pos\n\t\t\t\tif (!pos.startsWith(\"VB\"))\n\t\t\t\t\trel = null;\n\t\t\t\t// lex lookup with required supertag\n\t\t\t\t// NB: there's no guarantee of getting the right arg roles if the word-cat pair is observed \n\t\t\t\tlexicon.setSupertagger(supertaggerStandIn);\n\t\t\t\tsupertaggerStandIn.setTag(simpleCat); \n\t\t\t\tSignHash lexSigns = lexicon.getSignsFromWord(w);\n\n\t\t\t\tif (semClass == null || semClass.length() == 0)\n\t\t\t\t\tsemClass = \"NoClass\";\n\n\t\t\t\t// add lex signs, filtered by rel, reindexed\n\t\t\t\t// also check number with matching pos, match on no class\n\t\t\t\tint matchPOS = 0;\n\t\t\t\tboolean matchNoClass = false;\n\t\t\t\tfor (Iterator it = lexSigns.asSignSet().iterator(); it.hasNext();) {\n\t\t\t\t\tSign s = it.next();\n\n\t\t\t\t\tWord wTemp = s.getWords().get(0);\n\t\t\t\t\tString morphClass = wTemp.getSemClass();\n\t\t\t\t\tif (morphClass == null || morphClass.length() == 0)\n\t\t\t\t\t\tmorphClass = \"NoClass\";\n\n\t\t\t\t\tCategory lexcat = s.getCategory();\n\t\t\t\t\tLF lexLF = lexcat.getLF();\n\n\t\t\t\t\t// allow any class if no sem class given\n\t\t\t\t\tif (!(semClass.equals(\"NoClass\") || semClass.equals(morphClass))\n\t\t\t\t\t\t\t|| !containsPred(lexLF, rel)\n\t\t\t\t\t\t\t|| !containsRoles(lexLF, roles)\n\t\t\t\t\t\t\t|| !containsRel(lexLF, indexRel, s)) {\n\t\t\t\t\t\tit.remove();\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tUnifyControl.reindex(lexcat);\n\t\t\t\t\t\tif (wTemp.getPOS().equals(pos)) {\n\t\t\t\t\t\t\tmatchPOS++;\n\t\t\t\t\t\t\tif (semClass.equals(\"NoClass\") && morphClass.equals(\"NoClass\"))\n\t\t\t\t\t\t\t\tmatchNoClass = true;\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\t// filter by pos unless none match\n\t\t\t\tif (matchPOS > 0) {\n\t\t\t\t\tfor (Iterator it = lexSigns.asSignSet().iterator(); it.hasNext();) {\n\t\t\t\t\t\tSign s = it.next();\n\t\t\t\t\t\tWord wTemp = s.getWords().get(0);\n\t\t\t\t\t\tif (!wTemp.getPOS().equals(pos)) {\n\t\t\t\t\t\t\tit.remove(); continue;\n\t\t\t\t\t\t}\n\t\t\t\t\t\t// filter by mismatched class if apropos\n\t\t\t\t\t\tif (matchNoClass) {\n\t\t\t\t\t\t\tString morphClass = wTemp.getSemClass();\n\t\t\t\t\t\t\tif (morphClass != null && morphClass.length() != 0)\n\t\t\t\t\t\t\t\tit.remove();\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tif (lexSigns.isEmpty())\n\t\t\t\t\tthrow new LexException(\"No matching category \" + cat + \" for: \" + w);\n\t\t\t\treturn lexSigns;\n\t\t\t} catch (LexException exc) {\n\t\t\t\t// try continuing derivations without lex signs for punctuation,\n\t\t\t\t// otherwise throw parse exception\n\t\t\t\tif (isPunct(derivElt)) {\n\t\t\t\t\tif (ccgBankTaskTestbed.isDebugDerivations()) {\n\t\t\t\t\t\tccgBankTaskTestbed.log(header + \": \" + exc.toString());\n\t\t\t\t\t}\n\t\t\t\t\treturn new SignHash();\n\t\t\t\t}\n\t\t\t\tthrow new ParseException(exc.toString());\n\t\t\t} catch (RuntimeException exc) {\n\t\t\t\t// for other exceptions, throw parse exception\n\t\t\t\tthrow new ParseException(exc.toString());\n\t\t\t}\n\t\t} else\n\t\t\tthrow new RuntimeException(header + \": unrecognized element in derivation: \" + eltName);\n\t}\n\n\t// Recurse through a CCG cat and print out the atomcats and their ids\n\tprivate static void recurseCat(Category cat, ArrayList fullCat,\n\t\t\tHashtable idConvTally,\n\t\t\tHashtable freqTally) {\n\n\t\tif (cat instanceof ComplexCat) {\n\n\t\t\tComplexCat cc = (ComplexCat) cat.copy();\n\t\t\tCategory resCat = cc.getResult();\n\n\t\t\trecurseCat(resCat, fullCat, idConvTally, freqTally);\n\t\t\tint argStart = 0;\n\n\t\t\tif (resCat instanceof ComplexCat) {\n\t\t\t\tComplexCat temp = (ComplexCat) resCat.copy();\n\t\t\t\targStart = temp.getArgStack().size();\n\t\t\t}\n\n\t\t\tArgStack argStack = cc.getArgStack(argStart);\n\n\t\t\tfor (int i = 0; i < argStack.size(); i++) {\n\t\t\t\tif (argStack.get(i) instanceof BasicArg) {\n\t\t\t\t\tBasicArg bArg = (BasicArg) argStack.get(i);\n\t\t\t\t\tCategory argCat = (Category) bArg.getCat();\n\t\t\t\t\tSlash argSlash = (Slash) bArg.getSlash();\n\t\t\t\t\t// System.out.println(argSlash.toString()+'\\n');\n\t\t\t\t\tfullCat.add(argSlash.toString());\n\t\t\t\t\trecurseCat(argCat, fullCat, idConvTally, freqTally);\n\t\t\t\t}\n\t\t\t}\n\n\t\t} else if (cat instanceof AtomCat) {\n\n\t\t\tAtomCat ac = (AtomCat) cat.copy();\n\t\t\tFeatureStructure fs = ac.getFeatureStructure();\n\n\t\t\tif (fs.hasAttribute(\"index\")) {\n\t\t\t\tString index = fs.getValue(\"index\").toString();\n\t\t\t\t// System.out.println(index);\n\t\t\t\tString id[] = index.split(\":\");\n\n\t\t\t\tif (!idConvTally.containsKey(id[0]))\n\t\t\t\t\tidConvTally.put(id[0], Integer.toString(idConvTally.size() + 1));\n\t\t\t\tString numId = idConvTally.get(id[0]);\n\t\t\t\tString catId = ac.getType() + \"_\" + numId;\n\n\t\t\t\tif (!freqTally.containsKey(numId))\n\t\t\t\t\tfreqTally.put(numId, 0);\n\n\t\t\t\tint freq = freqTally.get(numId);\n\t\t\t\tfreqTally.put(numId, freq + 1);\n\n\t\t\t\tif (fs.hasAttribute(\"mod-index\"))\n\t\t\t\t\tcatId = catId + \"_\" + \"M\";\n\n\t\t\t\t// System.out.println('\\n');\n\n\t\t\t\tfullCat.add(catId);\n\t\t\t}\n\t\t}\n\t}\n\n\t// returns whether the given LF contains the given the lexical predicate\n\tprivate static boolean containsPred(LF lf, String pred) {\n\t\tif (pred == null)\n\t\t\treturn true;\n\t\tif (lf == null)\n\t\t\treturn false;\n\t\tfor (SatOp satOp : HyloHelper.getPreds(lf)) {\n\t\t\tif (HyloHelper.isLexPred(satOp)) {\n\t\t\t\tif (HyloHelper.getLexPred(satOp).equals(pred))\n\t\t\t\t\treturn true;\n\t\t\t}\n\t\t}\n\t\treturn false;\n\t}\n\n\t// roles in a given LF\n\tprivate static Set rolesSet = new HashSet();\n\n\t// returns whether the given LF contains the given the lexical predicate\n\tprivate static boolean containsRoles(LF lf, String roles) {\n\t\tif (roles == null)\n\t\t\treturn true;\n\t\tif (lf == null)\n\t\t\treturn false;\n\t\tString[] rolesArray = roles.split(\"\\\\s+\");\n\t\t// get roles in LF\n\t\trolesSet.clear();\n\t\tfor (SatOp satOp : HyloHelper.getPreds(lf)) {\n\t\t\tif (HyloHelper.isRelPred(satOp)) {\n\t\t\t\trolesSet.add(HyloHelper.getRel(satOp));\n\t\t\t}\n\t\t}\n\t\t// check presence of roles in LF\n\t\tfor (String role : rolesArray) {\n\t\t\tif (role.equals(\"null\") || role.equals(\"e\"))\n\t\t\t\tcontinue;\n\t\t\tif (!rolesSet.contains(role))\n\t\t\t\treturn false;\n\t\t}\n\t\treturn true;\n\t}\n\n\t// returns whether the given LF contains the given indexRel\n\tprivate static boolean containsRel(LF lf, String indexRel, Sign sign) {\n\n\t\tif (indexRel == null)\n\t\t\treturn true;\n\t\tif (lf == null)\n\t\t\treturn false;\n\n\t\tindexRel = \"<\" + indexRel + \">\";\n\n\t\t/*\n\t\t * System.out.println(sign.getSupertag()+\" \"+sign.getPOS());\n\t\t * System.out.println(indexRel); System.out.println(rolesSet);\n\t\t * System.out.println(lf); System.out.println('\\n');\n\t\t */\n\n\t\t// check presence of that rel/feat in LF\n\t\tif (!lf.toString().contains(indexRel))\n\t\t\treturn false;\n\t\telse\n\t\t\treturn true;\n\t}\n\n\t// identifies punctuation\n\tprivate static boolean isPunct(Element elt) {\n\t\tString pos = elt.getAttributeValue(\"pos\");\n\t\tif (pos == null)\n\t\t\treturn false;\n\n\t\treturn (pos.equals(\"|\") || pos.equals(\".\") || pos.equals(\",\")\n\t\t\t\t|| pos.equals(\";\") || pos.equals(\":\") || pos.equals(\"LRB\")\n\t\t\t\t|| pos.equals(\"RRB\") || pos.equals(\"``\") || pos.equals(\"''\"));\n\t}\n\n\t// return whether signs contains cat; filter if so\n\tprivate static boolean containsCat(SignHash signs, String cat) {\n\t\t// special case: give free pass to cats with dollars\n\t\tif (!signs.isEmpty() && cat.indexOf('$') >= 0)\n\t\t\treturn true;\n\t\t// check for cat\n\t\tboolean retval = false;\n\t\tfor (Sign sign : signs.asSignSet()) {\n\t\t\tString supertag = sign.getCategory().getSupertag();\n\t\t\t// again, give free pass to cats with dollars\n\t\t\tif (supertag.indexOf('$') >= 0 || cat.equals(supertag)) {\n\t\t\t\tretval = true;\n\t\t\t\tbreak;\n\t\t\t}\n\t\t}\n\t\t// filter if found\n\t\tif (retval) {\n\t\t\tfor (Iterator it = signs.asSignSet().iterator(); it.hasNext();) {\n\t\t\t\tSign sign = it.next();\n\t\t\t\tString supertag = sign.getCategory().getSupertag();\n\t\t\t\tif (supertag.indexOf('$') >= 0 || cat.equals(supertag))\n\t\t\t\t\tcontinue;\n\t\t\t\telse\n\t\t\t\t\tit.remove();\n\t\t\t}\n\t\t}\n\t\treturn retval;\n\t}\n\n\t// returns new combos for current sign\n\tprivate List newCombos() {\n\t\tList retval = new ArrayList();\n\t\tnewCombos(sign, retval);\n\t\treturn retval;\n\t}\n\n\t// recursively adds new combos for given sign\n\tprivate void newCombos(Sign s, List retval) {\n\t\tSign[] inputs = s.getDerivationHistory().getInputs();\n\t\tif (inputs != null) {\n\t\t\tStringBuffer sb = new StringBuffer();\n\t\t\tfor (int i = 0; i < inputs.length; i++) {\n\t\t\t\tsb.append(inputs[i].getCategory().getSupertag()).append(' ');\n\t\t\t}\n\t\t\tsb.append(s.getDerivationHistory().getRule().name());\n\t\t\tString combo = sb.toString();\n\t\t\tif (!combos.contains(combo)) {\n\t\t\t\tretval.add(combo);\n\t\t\t\tcombos.add(combo);\n\t\t\t}\n\t\t\tfor (int i = 0; i < inputs.length; i++) {\n\t\t\t\tnewCombos(inputs[i], retval);\n\t\t\t}\n\t\t}\n\t}\n\n\t// Extracts nom-id,pos,supertag info related to LF lexical preds\n\tprivate void extrPredInfo(LF lf, String sentId) {\n\t\t// System.out.println(sentId);\n\t\textractPredInfo(lf, predInfo);\n\t}\n\n\t/**\n\t * Extracts the nom id, pos, and supertag info related to LF lexical preds, \n\t * and puts it in the given map keyed off the nom id.\n\t * Note that the map should be cleared for each new LF.\n\t */\n\tpublic static void extractPredInfo(LF lf, Map predInfoMap) {\n\n\t\tString predData = \"\";\n\t\tList preds = HyloHelper.getPreds(lf);\n\n\t\tfor (SatOp pred : preds) {\n\n\t\t\tString lexPred = HyloHelper.getLexPred(pred);\n\t\t\tif (lexPred == null)\n\t\t\t\tcontinue;\n\n\t\t\tif (!(pred.getArg() instanceof Proposition))\n\t\t\t\tcontinue;\n\n\t\t\tProposition p = (Proposition) pred.getArg();\n\n\t\t\tString lex = (p.getName()).toString();\n\n\t\t\t// Get supertag & pos tag info and store that\n\t\t\tString stag = pred.getOrigin().getSupertag();\n\t\t\tString pos = pred.getOrigin().getPOS();\n\t\t\tNominal nom = pred.getNominal();\n\t\t\tString nomInd = nom.toString();\n\t\t\tString nomIndParts[] = nomInd.split(\":\");\n\n\t\t\tif (stag == null || pos == null || lex == null)\n\t\t\t\tcontinue;\n\n\t\t\tpredData = escape(stag) + \":\" + escape(pos) + \":\" + escape(lex);\n\t\t\tpredInfoMap.put(nomIndParts[0], predData);\n\t\t}\n\t}\n\n\t// Collects nom-id,pos,supertag info related to LF lexical preds for this\n\t// particular LF\n\tprivate static String collectPredInfo(String sentId) {\n\n\t\tString predData = \"\";\n\n\t\tfor (Enumeration e = predInfo.keys(); e.hasMoreElements();) {\n\n\t\t\tString nomId = e.nextElement();\n\t\t\tpredData = predData + \" \" + nomId + \":\" + predInfo.get(nomId);\n\t\t}\n\n\t\tpredInfo = new Hashtable();\n\n\t\treturn predData.trim();\n\t}\n\n\t/**\n\t * Returns the pred info string for the given pred info map (see extractPredInfo).\n\t */\n\tpublic static String getPredInfo(Map predInfoMap) {\n\t\tString predData = \"\";\n\t\tfor (String nomId : predInfoMap.keySet()) {\n\t\t\tpredData = predData + \" \" + nomId + \":\" + predInfoMap.get(nomId);\n\t\t}\n\t\treturn predData.trim();\n\t}\n\t\n\t// escapes a string using DefaultTokenizer\n\tprivate static String escape(String s) { return DefaultTokenizer.escape(s); }\n\t\n\t// stands in for a supertagger during lex lookup\n\tprivate static class SupertaggerStandIn implements SupertaggerAdapter {\n\t\t// map for a single key\n\t\tprivate Map map = new HashMap(2);\n\t\tpublic Map getSupertags() { return map; }\n\t\t\n\t\t// set tag\n\t\tvoid setTag(String tag) { map.clear(); map.put(tag, 1.0); }\n\t\t\n\t\t// dummy implementations\n\t\tpublic void setIncludeGold(boolean includeGold) {}\n\t\tpublic void resetBeta() {}\n\t\tpublic void resetBetaToMax() {}\n\t\tpublic void nextBeta() {}\n\t\tpublic void previousBeta() {}\n\t\tpublic boolean hasMoreBetas() { return false; }\n\t\tpublic boolean hasLessBetas() { return false; }\n\t\tpublic double[] getBetas() { return new double[]{1.0}; }\n\t\tpublic void setBetas(double[] betas) {}\n\t\tpublic double getCurrentBetaValue() { return 1.0; }\n\t}\n}\n" }, { "path": "src/opennlp/ccgbank/lexicon-base.xsl", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \t\n \n \n \n \n \n \n \n \n \n \n \n\n \n\t\t\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\t\t\t\t\n\t\t\t\t\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "src/opennlp/ccgbank/parse/CCGbankDerivation.jjt", "content": "// Grammar to parse all the ccgbank derivations in a given ccgbank file\n\noptions {\n MULTI=true;\n NODE_DEFAULT_VOID=false;\n NODE_SCOPE_HOOK=true; \n STATIC=true;\n USER_TOKEN_MANAGER=false;\n NODE_PREFIX=\"\"; \n NODE_USES_PARSER=true;\n}\n\nPARSER_BEGIN(CCGbankDerivation)\n\npackage opennlp.ccgbank.parse;\n\nimport java.util.*;\nimport org.apache.tools.ant.Task;\n//import java.io.*;\n\npublic class CCGbankDerivation {\n\n // The java code to operate the node scope hook\n static void jjtreeOpenNodeScope(Node n) {\n ((SimpleNode)n).first_token = getToken(1);\n }\n\n static void jjtreeCloseNodeScope(Node n) {\n ((SimpleNode)n).last_token = getToken(0);\n }\n}\n\nPARSER_END(CCGbankDerivation)\n\n\n// enter PROPINFO state (for extra Propbank-derived info) on \"{\"\n// nb: this is a workaround for ATOMCAT being perhaps too broadly defined\nTOKEN: { : PROPINFO }\n\n// return to normal on \">\" at end of node\n<*> TOKEN: { \"> : DEFAULT }\n\n// tokens in PROPINFO state\n TOKEN: { }\n TOKEN: { }\n TOKEN: { }\n TOKEN: { } // for stems and sem roles\n TOKEN: { }\n TOKEN: { }\n TOKEN: { }\n\n\n// regular tokens\n// mww: ATOMCAT is really too broad, b/c it's used for words too ...\nTOKEN:\n{\n \n |\n \n |\n (\"\\\\/\")*)+|(\"\\\\*\")+>\n //The SPL token is for treebank words like \"1\\/2\" or \"bassoonist\\/pianist\\/composer\" or \"\\*\" where the middle tokens stand elsewhere for ccg operations. \n //So the SPL prevents such words from interfering with the tokenization\n}\n\n<*> SKIP: { \" \" | \"\\t\" | \"\\n\" | \"\\r\" }\n\n\nSimpleNode start():\n{}\n{\n // Every file consists of one or more treebank entries\n // Each treebank entry consists of 1 header and 1 or more treenodes or leafnodes\n ( header() ( LOOKAHEAD(2) leafnode() | treenode()) )+\n { return jjtThis; }\n}\n\nvoid header():\n{\n Token t1 = new Token();\n Token t2 = new Token();\n Token t3 = new Token();\n jjtThis.type=\"Header\";\n}\n{\n // Header, eg: ID=wsj_2300.1 PARSER=GOLD NUMPARSE=1\n t1= [t2= t3=]\n {\n jjtThis.header = t1.image + \" \" + t2.image + \" \" + t3.image;\n }\n}\n\n\nvoid treenode():\n{\n Token leftover = new Token();\n Token head = new Token();\n Token dtr = new Token();\n Token lex = new Token();\n Token sense = new Token();\n Token role = new Token();\n jjtThis.type=\"Treenode\";\n}\n{\n // Storing the ccgID. (mww: huh?)\n // Sample Treenode: \n // With Propbank roles: \n \"(\" \n \" head= dtr=\n { jjtThis.leftover=leftover.image; } \n |\n head=\n dtr=\n )\n \n // Headedness, number of dtrs info stored in the SimpleNode data structure\n { jjtThis.head=head.image; jjtThis.dtr=dtr.image; }\n \n // optional propbank info\n // nb: need to use eg instead of \".\" once in PROPINFO lex state \n [\n \"{\"\n lex= sense= role= \n { \n \tjjtThis.nodeRoles = new ArrayList(3);\n \tjjtThis.nodeRoles.add(new SimpleNode.LexSenseRole(lex.image, sense.image, role.image));\n \t}\n (\n \n\t lex= sense= role= \n\t { jjtThis.nodeRoles.add(new SimpleNode.LexSenseRole(lex.image, sense.image, role.image)); }\n )*\n \n ]\n \n // closure of one the root of a treenode\n \n // A treenode consists of 1 or more treenodes or leafnodes. \n ( LOOKAHEAD(2) leafnode() | treenode() )+\n \n \")\" // Close of a treenode\n}\n\n\nvoid leafnode():\n{\n Token t = new Token();\n Token lex = new Token();\n Token sense = new Token();\n Token role = new Token();\n jjtThis.type=\"Leafnode\";\n}\n{\n /* Sample Leafnode structures:\n ()\n ()\n ()\n */\n \"(\" \n \" { jjtThis.pos = t.image; }\n \n // The pos tag has been repeated in the ccgbank. \n // For numerals (null) is an entry. \n // Hence the optional brackets in the bnf below.\n ( [\"(\"] [\")\"])\n \n ( t= | t= )\n { jjtThis.lex = t.image; }\n \n catSpec()\n \n // optional propbank info\n [\n \"{\"\n lex= sense= role= // role or 'rel'\n { \n \tjjtThis.nodeRoles = new ArrayList(3);\n \tjjtThis.nodeRoles.add(new SimpleNode.LexSenseRole(lex.image, sense.image, role.image));\n \t}\n (\n \n\t lex= sense= role= \n\t { jjtThis.nodeRoles.add(new SimpleNode.LexSenseRole(lex.image, sense.image, role.image)); }\n )*\n \n ]\n [\n \n role= \n { \n \tjjtThis.argRoles = new ArrayList(4);\n \tjjtThis.argRoles.add(role.image);\n \t}\n (\n \n role= \n { jjtThis.argRoles.add(role.image); }\n )*\n \n ]\n //\">\" \n \")\"\n}\n\nvoid catSpec():\n{ jjtThis.type=\"complexcat\"; }\n{\n // The result category (atomic or complex)\n ( \n ( \"(\" catSpec() \")\" [] ) // mww: the extra \"atomcat\" is really for an index on the complex cat\n |\n atomcat()\n )\n \n // The argument of the function (atomic or complex). \n // Note: The following could be processed as a separate production \"Embedded category\" , but this would produce a node of that name. \n // To avoid that, the preceding BNF notation has been repeated here. \n [\n op()\n ( \n ( \"(\" catSpec() \")\" [] )\n |\n atomcat()\n )\n ]\n}\n\n\nvoid catSpecRedundant():\n{ jjtThis.type=\"Redundant\"; } \n{\n ( \n ( \"(\" catSpec() \")\" )\n |\n atomcat()\n )\n [\n op()\n ( \n ( \"(\" catSpec() \")\" )\n |\n atomcat()\n )\n ]\n}\n\n\nvoid atomcat():\n{ jjtThis.type=\"atomcat\"; }\n{\n \n}\n\nvoid op():\n{ Token t = new Token(); jjtThis.type=\"op\"; }\n{\n t= { jjtThis.cat=t.image; }\n}\n\n\n" }, { "path": "src/opennlp/ccgbank/parse/SimpleNode.java", "content": "/* Generated By:JJTree: Do not edit this line. SimpleNode.java */\n\npackage opennlp.ccgbank.parse;\n\nimport java.util.List;\n\nimport opennlp.ccgbank.parse.CCGbankDerivation;\nimport opennlp.ccgbank.parse.CCGbankDerivationTreeConstants;\n\npublic class SimpleNode implements Node {\n //Javacc generated variables\n protected Node parent;\n protected Node[] children;\n protected int id;\n protected CCGbankDerivation parser;\n\n //User defined variables\n\n // lex, sense, role triples\n public static class LexSenseRole {\n public String lex, sense, role;\n\n public LexSenseRole(String lex, String sense, String role) {\n this.lex = lex;\n this.sense = sense;\n this.role = role;\n }\n }\n\n //CCGbank id\n public String header;\n\n //The serial no of the gold standard parse\n String parseNo = \"\";\n\n //Node type eg:-Treenode,Leafnode,atomcat etc\n public String type = \"\";\n\n //Traps any feature which is leftover\n public String leftover;\n\n //Headedness info 0 or 1\n public String head = \"\";\n\n //No:of daughters of a node\n public String dtr = \"\";\n\n //Category Specification\n public String cat = \"\";\n\n //Category Specification without co-indexation info in leafnodes\n public String catRedundant = \"\";\n\n //Lexical information\n public String lex = \"\";\n\n //Part of speech info. eg: RB, IN etc\n public String pos = \"\";\n\n // The roles (or rel) that the node plays\n public List nodeRoles = null;\n\n // The arg roles of a verbal cat\n public List argRoles = null;\n\n //First token in the node scope\n Token first_token;\n\n //Final token in the node scope\n Token last_token;\n\n //Function which produces the content of the node.\n public String print() throws Exception {\n\n Token p = first_token;\n\n while (p != last_token) {\n cat = cat + p.image;\n p = p.next;\n }\n\n return cat + last_token.image;\n }\n\n\n //The remaining part incl comments is Javacc generated.\n\n public SimpleNode(int i) {\n id = i;\n }\n\n public SimpleNode(CCGbankDerivation p, int i) {\n this(i);\n parser = p;\n }\n\n /**\n * @return the header\n */\n public String getHeader() {\n return header;\n }\n\n /**\n * @param header the header to set\n */\n public void setHeader(String header) {\n this.header = header;\n }\n\n /**\n * @return the leftover\n */\n public String getLeftover() {\n return leftover;\n }\n\n public int getId(){\n return id;\n }\n\n\n public void jjtOpen() {\n }\n\n public void jjtClose() {\n }\n\n\n public void jjtSetParent(Node n) {\n parent = n;\n }\n\n public Node jjtGetParent() {\n return parent;\n }\n\n public void jjtAddChild(Node n, int i) {\n if (children == null) {\n children = new Node[i + 1];\n } else if (i >= children.length) {\n Node c[] = new Node[i + 1];\n System.arraycopy(children, 0, c, 0, children.length);\n children = c;\n }\n children[i] = n;\n }\n\n public Node jjtGetChild(int i) {\n return children[i];\n }\n\n public int jjtGetNumChildren() {\n return (children == null) ? 0 : children.length;\n }\n\n /* You can override these two methods in subclasses of SimpleNode to\n customize the way the node appears when the tree is dumped. If\n your output uses more than one line you should override\n toString(String), otherwise overriding toString() is probably all\n you need to do. */\n @Override\n public String toString() {\n return CCGbankDerivationTreeConstants.jjtNodeName[id];\n }\n\n public String toString(String prefix) {\n return prefix + toString();\n }\n\n /* Override this method if you want to customize how the node dumps\n out its children. */\n\n public void dump(String prefix) {\n System.out.println(toString(prefix));\n if (children != null) {\n for (int i = 0; i < children.length; ++i) {\n SimpleNode n = (SimpleNode) children[i];\n if (n != null) {\n n.dump(prefix + \" \");\n }\n }\n }\n }\n}\n\n" }, { "path": "src/opennlp/ccgbank/parse/grammarInsert", "content": "static void jjtreeOpenNodeScope(Node n)\n {\n ((SimpleNode)n).first_token = getToken(1);\n }\n\nstatic void jjtreeCloseNodeScope(Node n)\n {\n ((SimpleNode)n).last_token = getToken(0);\n }\n\n" }, { "path": "src/opennlp/ccgbank/rules-base.xsl", "content": "\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \t\n \n \n \n \n \n \n \n \t\t \n \n\n \n \t\t \n \n\n \t\n \t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t \n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n \t\t\t \n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t \n \n \n \t \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n" }, { "path": "src/pom.xml", "content": "\n\n 4.0.0\n opennlp\n openccg\n 0.10.0\n \n \n 1.8\n 1.8\n ${project.build.directory}/generated-sources\n \n\n \n \n \n jdom\n jdom\n 1.1\n \n \n \n trove\n trove\n 1.0.2\n \n\n \n org.apache.ant\n ant\n 1.9.0\n \n\n \n net.sf.jgrapht\n jgrapht\n 0.8.3\n \n\n \n net.sf.jopt-simple\n jopt-simple\n 3.1\n \n \n \n junit\n junit\n 4.12\n \n \n\n \n jline\n jline\n 1.0\n \n\n \n org.eclipse.birt.runtime.3_7_1\n org.apache.xml.serializer\n 2.7.1\n \n\n\n \n\n \n openccg\n .\n \n \n maven-compiler-plugin\n 3.7.0\n \n \n **/.backup.orig/**\n srilmbridge/\n kenlm/\n \n \n \n \n org.codehaus.mojo\n javacc-maven-plugin\n 2.6\n \n \n jjt\n generate-sources\n \n jjtree-javacc\n \n \n ${basedir}/opennlp/ccgbank/parse/\n ${project.build.gen}/jjtree/\n \n \n \n jj\n generate-sources\n \n javacc\n \n \n ${project.build.gen}/jjtree/opennlp/ccgbank/parse/\n ${project.build.gen}/jjtree/\n \n \n \n \n \n org.codehaus.mojo\n build-helper-maven-plugin\n \n \n generate-sources\n \n add-source\n \n \n \n ${project.build.gen}\n \n \n \n \n \n \n \n \n \n" }, { "path": "src/srilmbridge/Makefile", "content": "\ncompile:\n\tjavah -d . -classpath ${OPENCCG_HOME}/classes \\\n\t\topennlp.ccg.ngrams.SRILMNgramModel\n\tg++ -o ${OPENCCG_HOME}/lib/libsrilmbridge.so \\\n\t\t-Wl,-soname,srilmbridge.so \\\n\t\t-I${JAVA_HOME}/include \\\n\t\t-I${JAVA_HOME}/include/linux \\\n\t\t-I${SRILM}/include \\\n\t\t-L${SRILM}/lib/i686 \\\n\t\tsrilmbridge.cpp ${SRILM}/lm/src/LM.cc \\\n\t\t-shared -lc -ldstruct -lflm -llattice -lmisc -loolm \\\n\t\t-Wno-deprecated\t\t\n\nclean:\n\trm ${OPENCCG_HOME}/lib/libsrilmbridge.so\n\trm opennlp_ccg_ngrams_SRILMNgramModel.h\n" }, { "path": "src/srilmbridge/srilmbridge.cpp", "content": "/* $Id: srilmbridge.cpp,v 1.13 2007/06/16 22:26:28 coffeeblack Exp $ */\n#include \n#include \n#include \n#include \n#include \"opennlp_ccg_ngrams_SRILMNgramModel.h\"\n\n/*\n * Bridge from Java to the SRILM toolkit library. Loads a language model based\n * on specified parameters, then calculates the probability of a word within a\n * given context.\n *\n * Author: Scott Martin (http://www.ling.osu.edu/~scott/)\n * Version: $Revision: 1.13 $\n */\n\nconst static unsigned STANDARD = 0, COUNT = 1;\n\n/*\n * The language model we will use to calculate word probabilities.\n */\nLM *lm = NULL;\n\n/*\n * The type of language model in effect, as specified in loadLM.\n */\nunsigned nativeLMType = STANDARD;\n\n/*\n * Loads a language model from a specified file with the specified ngram order.\n * The parameter lmType specifies what type (format) of language model to\n * expect.\n *\n * Throws:\n *\tjava.io.IOException If fileName is null or empty or if a problem is\n * \t\tencountered reading the language model file.\n *\tjava.lang.IllegalStateException If an LM has already been loaded.\n *\tjava.lang.IllegalArgumentException If the specified LM type is not\n *\t\tsupported. Currently supports STANDARD (type 0) and COUNT (type 1).\n */\nJNIEXPORT void JNICALL Java_opennlp_ccg_ngrams_SRILMNgramModel_loadLM\n \t(JNIEnv *env, jobject obj, jint order, jstring fileName, jint lmType) {\n if(lm != NULL) { // already loaded\n \tenv->ThrowNew(env->FindClass(\"java/lang/IllegalStateException\"),\n \t\t\"LM already loaded\");\n \treturn;\n }\n \n nativeLMType = lmType;\n \n if(nativeLMType < STANDARD || nativeLMType > COUNT) {\n \t// only STANDARD and COUNT are allowed\n \tenv->ThrowNew(env->FindClass(\"java/lang/IllegalArgumentException\"),\n \t\t\"LM type not supported\");\n \treturn;\n }\n if(fileName == 0 || env->GetStringLength(fileName) == 0) {\n\t\tenv->ThrowNew(env->FindClass(\"java/io/IOException\"),\n\t\t\t\"problem reading LM: empty file name\");\n\t\treturn;\n }\n\t\n Vocab *vocab = new Vocab;\n lm = (nativeLMType == COUNT)\n \t? (LM *)new NgramCountLM(*vocab, order)\n \t:\t(LM *)new Ngram(*vocab, order);\n \n const char* nativeFileName = env->GetStringUTFChars(fileName, NULL);\n \n File file(nativeFileName, \"r\", 0);\n unsigned lmError = 0;\n \n if(((File *)&file)->error() > 0) {\n \tlmError = 1;\n \tenv->ThrowNew(env->FindClass(\"java/io/IOException\"),\n \t\t\"problem with LM file\");\n }\n else if(!lm->read(file)) {\n \tlmError = 1;\n\t\tenv->ThrowNew(env->FindClass(\"java/io/IOException\"), \"problem reading LM\");\n }\n //TODO the following just repeats finalize(), should be in reusable function\n if(lmError > 0) { // destroy lm so this can be called again\n \tvocab->~Vocab();\n \t\n \tif(lm != NULL) {\n \t\tif(nativeLMType == STANDARD) {\n\t \t\t((Ngram *)lm)->~Ngram();\n\t \t}\n\t \telse if(nativeLMType == COUNT) {\n\t \t\t((NgramCountLM *)lm)->~NgramCountLM();\n\t \t}\n\t \telse { // as a failsafe, call the abstract destructor\n\t\t \tlm->~LM();\n\t\t }\n \t\n\t \tdelete lm;\n \t}\n \t\n\t\tlm = NULL; //TODO clean up file somehow?\n }\n \n ((File *)&file)->close();\n env->ReleaseStringUTFChars(fileName, nativeFileName);\n}\n\n/*\n * Uses the SRILM toolkit library to calculate the log prob of a word in a\n * specified context. The context is a history of tokens preceeding the\n * specified word specified in reverse order. For example, to find the\n * probability of \"rain\" in the context \"in the rain\", this method should be\n * called with \"rain\" as the parameter `word' and the array {\"the\", \"in\"} in\n * the parameter `context'.\n *\n * To calculate the log probability of a single word with no context, call this\n * method with either (1) NULL, or (2) a zero-length array as the value of\n * the parameter `context'.\n *\n * Throws:\n *\tjava.lang.IllegalStateException If an error happened while loading\n * \t\tthe LM and word probabilities can not be computed.\n *\tjava.lang.IllegalArgumentException If the specified word is null or\n *\t\tzero-length or if the specified context contains a null or zero-length\n *\t\tstring.\n */\nJNIEXPORT jfloat JNICALL Java_opennlp_ccg_ngrams_SRILMNgramModel_doLogProb\n \t(JNIEnv *env, jobject obj, jstring word, jobjectArray context) {\n // make sure LM is ok to use\n if(lm == NULL) {\n \tenv->ThrowNew(env->FindClass(\"java/lang/IllegalStateException\"),\n \t\t\"LM not loaded\");\n \treturn 0;\n }\n \n // sanity checks must throw Java exceptions\n if(word == NULL || env->GetStringLength(word) == 0) {\n \tenv->ThrowNew(env->FindClass(\"java/lang/IllegalArgumentException\"),\n\t\t\t\"word is null or zero-length\");\n \treturn 0;\n }\n \n int contextLength = (context == NULL) ? 0 : env->GetArrayLength(context);\n\n\tVocabString nativeWord = (VocabString)env->GetStringUTFChars(word, NULL);\n VocabString nativeContext[contextLength + 1];\n nativeContext[contextLength] = NULL; // context must be terminated by NULL\n \n // build context, converting each Java string to a VocabString\n jstring jstr = NULL;\n for(unsigned i = 0; i < contextLength; i++) {\n \tjstr = (jstring)env->GetObjectArrayElement(context, i);\n \t\n \tif(jstr == NULL || env->GetStringLength(jstr) == 0) {\n \t\tenv->ThrowNew(env->FindClass(\"java/lang/IllegalArgumentException\"),\n \t\t\t\"context contains null or zero-length string\");\n \t\t\n \t\tenv->DeleteLocalRef(jstr);\n \t\tenv->ReleaseStringUTFChars(word, nativeWord); // release word\n \t\t// release already translated strings\n \t\tfor(unsigned j = (i - 1); j >= 0; j--) {\n \t\t\tenv->ReleaseStringUTFChars(\n \t\t\t\t(jstring)env->GetObjectArrayElement(context, j), nativeContext[j]);\n \t\t}\n \t\t\n \t\treturn 0;\n \t}\n \t\n \tnativeContext[i] = (VocabString)env->GetStringUTFChars(jstr, NULL);\n }\n \n if(jstr != NULL) {\n\t env->DeleteLocalRef(jstr);\n\t}\n \n LogP prob = lm->wordProb(nativeWord, nativeContext);\n \n // clean up\n env->ReleaseStringUTFChars(word, nativeWord); // release word\n \n // release context strings if any\n for(unsigned k = 0; k < contextLength; k++) {\n \tenv->ReleaseStringUTFChars((jstring)env->GetObjectArrayElement(context, k),\n \t\tnativeContext[k]);\n\t}\n \n return prob;\n}\n\n/*\n * Should be called by a finalize() method from within Java. Calls the\n * destructor method on the language model object we are using.\n */\nJNIEXPORT void JNICALL Java_opennlp_ccg_ngrams_SRILMNgramModel_finalize\n (JNIEnv *env, jobject obj) {\n if(lm != NULL) { // call local destructors if type was specified\n \tif(nativeLMType == STANDARD) {\n \t\t((Ngram *)lm)->~Ngram();\n \t}\n \telse if(nativeLMType == COUNT) {\n \t\t((NgramCountLM *)lm)->~NgramCountLM();\n \t}\n \telse { // as a failsafe, call the abstract destructor\n\t \tlm->~LM();\n\t }\n \t\n \tdelete lm;\n }\n}\n\n" }, { "path": "test/grammar.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n\n" }, { "path": "test/lexicon.xml", "content": "\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "test/morph.xml", "content": "\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "test/opennlp/ccg/alignment/AlignmentTest.java", "content": "package opennlp.ccg.alignment;\n\nimport static opennlp.ccg.alignment.PhrasePosition.A;\nimport static opennlp.ccg.alignment.PhrasePosition.B;\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertFalse;\nimport static org.junit.Assert.assertNotSame;\nimport static org.junit.Assert.assertNull;\nimport static org.junit.Assert.assertTrue;\nimport static org.junit.Assert.fail;\n\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.HashSet;\nimport java.util.Iterator;\nimport java.util.Map;\nimport java.util.Set;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class AlignmentTest {\n\n\tInteger id;\n\tPhrase one, two;\n\tSet mappings;\n\tAlignment alignment;\n\t\n\tMap> map, pam;\n\t\n\tSet twoAVals;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tid = new Integer(37);\n\t\tone = new Phrase(id, Alignments.tokenize(\"This is it .\"));\n\t\ttwo = new Phrase(id, Alignments.tokenize(\"A test this is .\"));\n\t\t\n\t\tmappings = new HashSet();\n\t\tmappings.add(new Mapping(id, 0, 2));\n\t\tmappings.add(new Mapping(id, 1, 3));\n\t\tmappings.add(new Mapping(id, 3, 4));\n\t\tmappings.add(new Mapping(id, 2, 0));\n\t\tmappings.add(new Mapping(id, 2, 1));\n\t\t\n\t\tmap = new HashMap>();\n\t\tpam = new HashMap>();\n\t\t\n\t\tmap.put(0, new HashSet(Collections.singleton(2)));\n\t\tmap.put(1, new HashSet(Collections.singleton(3)));\n\t\tmap.put(3, new HashSet(Collections.singleton(4)));\n\t\t\n\t\ttwoAVals = new HashSet();\n\t\ttwoAVals.add(0);\n\t\ttwoAVals.add(1);\n\t\ttwoAVals = Collections.unmodifiableSet(twoAVals);\n\t\t\n\t\tmap.put(2, twoAVals);\n\t\t\n\t\tpam.put(2, new HashSet(Collections.singleton(0)));\n\t\tpam.put(3, new HashSet(Collections.singleton(1)));\n\t\tpam.put(4, new HashSet(Collections.singleton(3)));\n\t\tpam.put(0, new HashSet(Collections.singleton(2)));\n\t\tpam.put(1, new HashSet(Collections.singleton(2)));\n\t\t\n\t\talignment = new Alignment(one, two, mappings);\n\t\tassertEquals(mappings, alignment);\n\t}\n\t\n\t@Test\n\tpublic void testCompare() {\n\t\tPhrase o = new Phrase(43, one), t = new Phrase(43, two);\n\t\t\n\t\tSet ms = new HashSet();\n\t\tfor(Mapping m : mappings) {\n\t\t\tms.add(m.copyWithPhraseNumber(43));\n\t\t}\n\t\t\n\t\tAlignment a = new Alignment(o, t, ms);\n\t\t\n\t\tassertEquals(-1, alignment.compareTo(a));\n\t\tassertEquals(1, a.compareTo(alignment));\n\t\tassertEquals(0, a.compareTo(a));\n\t\tassertEquals(0, alignment.compareTo(alignment));\n\t}\n\n\t@Test\n\tpublic void testAlignment() {\n\t\ttry {\n\t\t\tnew Alignment(null, two, mappings);\n\t\t\tfail(\"able to create alignment with null phrase\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew Alignment(one, null, mappings);\n\t\t\tfail(\"able to create alignment with null phrase\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew Alignment(one, two, null);\n\t\t\tfail(\"able to create alignment with null mappings\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testSize() {\n\t\tassertEquals(mappings.size(), alignment.size());\n\t}\n\n\t@Test\n\tpublic void testGet() {\n\t\tassertEquals(one, alignment.getA());\n\t\tassertEquals(two, alignment.getB());\n\t\tassertNotSame(one, alignment.getB());\n\t}\n\n\t@Test\n\tpublic void testAddMapping() {\n\t\tMapping m = new Mapping(id, 0, 4);\n\t\t\n\t\tassertTrue(alignment.add(m));\n\t\tassertFalse(alignment.add(new Mapping(id, 3, 4)));\n\t\t\n\t\talignment.remove(m);\n\t\t\n\t\ttry {\n\t\t\talignment.add(new Mapping(id + 1, 0, 4));\n\t\t\tfail(\"able to add mapping with non-matching ID\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\talignment.add(new Mapping(id, null, 5));\n\t\t\tfail(\"able to add mapping with null index\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\talignment.add(new Mapping(id, 0, 5));\n\t\t\tfail(\"able to add mapping with out of bounds index\");\n\t\t}\n\t\tcatch(IndexOutOfBoundsException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\talignment.add(new Mapping(id, -2, 3));\n\t\t\tfail(\"able to add mapping with out of bounds index\");\n\t\t}\n\t\tcatch(IndexOutOfBoundsException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\talignment.add(new Mapping(id, 5, 3));\n\t\t\tfail(\"able to add mapping with out of bounds index\");\n\t\t}\n\t\tcatch(IndexOutOfBoundsException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\talignment.add(new Mapping(id, 3, -3));\n\t\t\tfail(\"able to add mapping with out of bounds index\");\n\t\t}\n\t\tcatch(IndexOutOfBoundsException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testGetTargets() {\n\t\tSet ts = alignment.getTargets(2, A);\n\t\tSet s = new HashSet();\n\t\ts.add(0);\n\t\ts.add(1);\n\t\t\n\t\tassertFalse(s.retainAll(ts));\n\t\tassertEquals(s.size(), ts.size());\n\t\t\n\t\tts.add(4);\n\t\tassertTrue(ts.contains(4));\n\t\t\n\t\ttry {\n\t\t\tts.remove(4);\n\t\t\tassertFalse(ts.contains(4));\n\t\t}\n\t\tcatch(UnsupportedOperationException e) {\n\t\t\tfail(\"unable to call remove()\");\t\t\t\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tIterator i = ts.iterator();\n\t\t\ti.next();\n\t\t\ti.remove();\n\t\t}\n\t\tcatch(UnsupportedOperationException e) {\n\t\t\tfail(\"unable to call iterator().remove()\");\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testMapEntrySet() {\n\t\tMap> amap = alignment.asMap(A), bmap = alignment.asMap(B);\n\t\tSet s = new HashSet(twoAVals);\n\t\t\n\t\tfor(Map.Entry> e : amap.entrySet()) {\n\t\t\tif(e.getKey().equals(2)) {\n\t\t\t\tassertEquals(s, e.getValue());\n\t\t\t}\n\t\t}\n\t\t\n\t\tIterator>> i = bmap.entrySet().iterator();\n\t\twhile(i.hasNext()) {\n\t\t\tMap.Entry> e = i.next();\n\t\t\tif(e.getKey().equals(1)) {\n\t\t\t\tassertEquals(Collections.singleton(2), e.getValue());\n\t\t\t}\n\t\t\telse if(e.getKey().equals(4)) {\n\t\t\t\tassertEquals(Collections.singleton(3), e.getValue());\n\t\t\t}\n\t\t\telse {\n\t\t\t\ttry {\n\t\t\t\t\ti.remove();\n\t\t\t\t}\n\t\t\t\tcatch(UnsupportedOperationException ex) {\n\t\t\t\t\tfail(\"unable to call Iterator.remove()\");\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\ttry {\n\t\t\t\t\tassertTrue(e.getValue().add(3));\n\t\t\t\t\tassertTrue(e.getValue().contains(3));\n\t\t\t\t\tassertFalse(e.getValue().add(3));\n\t\t\t\t}\n\t\t\t\tcatch(UnsupportedOperationException ex) {\n\t\t\t\t\tfail(\"unable to add to entry value\");\t\t\t\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\ttry {\n\t\t\t\t\tassertTrue(e.getValue().remove(3));\n\t\t\t\t\tassertFalse(e.getValue().contains(3));\n\t\t\t\t\tassertFalse(e.getValue().remove(3));\n\t\t\t\t}\n\t\t\t\tcatch(UnsupportedOperationException ex) {\n\t\t\t\t\tfail(\"unable to remove from entry value\");\t\t\t\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\ttry {\n\t\t\t\t\tif(!e.getValue().isEmpty()) {\n\t\t\t\t\t\te.getValue().remove(e.getValue().iterator().next());\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tcatch(UnsupportedOperationException ex) {\n\t\t\t\t\tfail(\"unable to call remove() for entry value\");\t\t\t\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\ttry {\n\t\t\t\t\tif(!e.getValue().isEmpty()) {\n\t\t\t\t\t\tIterator it = e.getValue().iterator();\n\t\t\t\t\t\tit.next();\n\t\t\t\t\t\tit.remove();\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tcatch(UnsupportedOperationException ex) {\n\t\t\t\t\tfail(\"unable to call remove() for entry value iterator\");\t\t\t\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\ttry {\n\t\t\t\t\te.setValue(new HashSet(Collections.singleton(0)));\n\t\t\t\t}\n\t\t\t\tcatch(UnsupportedOperationException ex) {\n\t\t\t\t\tfail(\"unable to set entry value\");\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testMapValues() {\n\t\tSet> as = new HashSet>();\n\t\tas.add(Collections.singleton(2));\n\t\tas.add(Collections.singleton(3));\n\t\tas.add(Collections.singleton(4));\n\t\t\n\t\tSet s = new HashSet(twoAVals);\n\t\tas.add(s);\n\t\t\n\t\tMap> amap = alignment.asMap(A), bmap = alignment.asMap(B);\n\t\t\n\t\tassertTrue(amap.values().size() == as.size() && amap.values().containsAll(as));\n\t\t\n\t\tSet> bvals = new HashSet>(); // avoid doubling\n\t\tbvals.addAll(bmap.values());\n\t\t\n\t\tas.remove(s);\n\t\tas.add(Collections.singleton(0));\n\t\tas.add(Collections.singleton(1));\n\t\tas.remove(Collections.singleton(4));\n\t\t\n\t\tassertEquals(as, bvals);\n\t\tassertTrue(bmap.values().contains(Collections.singleton(2)));\n\t\t\n\t\ttry {\n\t\t\tamap.values().add(Collections.singleton(1));\n\t\t\tfail(\"able to add value\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// do nothing\t\t\t\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tif(!amap.values().isEmpty()) {\n\t\t\t\tamap.values().remove(amap.values().iterator().next());\n\t\t\t}\n\t\t}\n\t\tcatch(UnsupportedOperationException ex) {\n\t\t\tfail(\"unable to remove value\");\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testMapKeySet() {\n\t\tMap> amap = alignment.asMap(A), bmap = alignment.asMap(B);\n\t\t\n\t\tassertTrue(amap.keySet().contains(1));\n\t\tassertTrue(bmap.keySet().contains(3));\n\t\t\n\t\tassertFalse(amap.keySet().contains(4));\n\t\tassertFalse(amap.keySet().contains(null));\n\t\tassertFalse(bmap.keySet().contains(5));\n\t\t\t\t\n\t\ttry {\n\t\t\tamap.keySet().add(4);\n\t\t\tfail(\"able to add key to key set\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// do nothing, expected\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tbmap.keySet().add(4);\n\t\t\tfail(\"able to add key to key set\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// do nothing, expected\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tamap.keySet().remove(1);\n\t\t\tassertFalse(amap.keySet().contains(1));\n\t\t}\n\t\tcatch(UnsupportedOperationException ex) {\n\t\t\tfail(\"unable to remove key from key set\");\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tbmap.keySet().remove(3);\n\t\t\tassertFalse(bmap.keySet().contains(3));\n\t\t}\n\t\tcatch(UnsupportedOperationException ex) {\n\t\t\tfail(\"unable to remove key from key set\");\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tamap.keySet().clear();\n\t\t\tassertTrue(amap.keySet().isEmpty());\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\tfail(\"unable to clear key set\");\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tbmap.keySet().clear();\n\t\t\tassertTrue(bmap.keySet().isEmpty());\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\tfail(\"able to clear key set\");\n\t\t}\n\t\t\n\t\ttry{\n\t\t\tamap.keySet().add(5);\n\t\t\tfail(\"able to add to key set\");\n\t\t}\n\t\tcatch(UnsupportedOperationException ex) {\n\t\t\t// expected\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testMapContains() {\n\t\tMap> amap = alignment.asMap(A), bmap = alignment.asMap(B);\n\t\t\n\t\tfor(int i = 0; i < 4; i++) {\n\t\t\tassertTrue(amap.containsKey(i));\n\t\t\tassertTrue(bmap.containsKey(i));\n\t\t}\n\t\t\n\t\tassertTrue(bmap.containsKey(4));\n\t\tassertFalse(amap.containsKey(4));\n\t\t\n\t\tassertTrue(bmap.containsValue(Collections.singleton(2)));\n\t\t\n\t\tSet s = new HashSet(twoAVals);\n\t\tassertTrue(amap.containsValue(s));\n\t}\n\t\n\t@Test\n\tpublic void testMapGet() {\n\t\tMap> amap = alignment.asMap(A), bmap = alignment.asMap(B);\n\t\t\n\t\tSet s = new HashSet(twoAVals);\n\t\t\n\t\tassertEquals(s, amap.get(2));\n\t\tassertEquals(Collections.singleton(3), bmap.get(4));\n\t\tassertNull(amap.get(4));\n\t\t\n\t\ttry {\n\t\t\tassertTrue(amap.get(2).contains(1));\n\t\t\tamap.get(2).remove(1);\n\t\t\tassertFalse(amap.get(2).contains(1));\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\tfail(\"unable remove from value set\");\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tassertFalse(bmap.get(4).contains(2));\n\t\t\tbmap.get(4).add(2);\n\t\t\tassertTrue(bmap.get(4).contains(2));\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\tfail(\"unable add to value set\");\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testMapPut() {\n\t\tMap> amap = alignment.asMap(A), bmap = alignment.asMap(B);\n\t\t\n\t\ttry {\n\t\t\tassertEquals(Collections.singleton(3), amap.put(1, Collections.singleton(2)));\n\t\t\tassertEquals(Collections.singleton(2), amap.get(1));\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\tfail(\"unable to put\");\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tassertEquals(Collections.singleton(0), bmap.put(2, Collections.singleton(3)));\n\t\t\tassertEquals(Collections.singleton(3), bmap.get(2));\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\tfail(\"unable to put\");\n\t\t}\n\t\t\n\t\tamap.remove(1);\n\t\tassertNull(amap.put(1, Collections.singleton(0)));\n\t\t\n\t\t\n\t}\n\t\n\t@Test\n\tpublic void testMapRemove() {\n\t\tMap> amap = alignment.asMap(A), bmap = alignment.asMap(B);\n\t\t\n\t\ttry {\n\t\t\tassertTrue(amap.containsKey(1));\n\t\t\tamap.remove(1);\n\t\t\tassertFalse(amap.containsKey(1));\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\tfail(\"unable to remove\");\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tassertTrue(bmap.containsKey(2));\n\t\t\tbmap.remove(2);\n\t\t\tassertFalse(bmap.containsKey(2));\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\tfail(\"unable to remove\");\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testMapAdd() {\n\t\tMap> amap = alignment.asMap(A), bmap = alignment.asMap(B);\n\t\t\n\t\ttry {\n\t\t\tamap.get(1).add(0);\n\t\t\tassertTrue(amap.get(1).contains(0));\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\tfail(\"unable to add\");\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tbmap.get(2).add(3);\n\t\t\tassertTrue(bmap.get(2).contains(3));\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\tfail(\"unable to add\");\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testMapClear() {\n\t\tMap> amap = alignment.asMap(A), bmap = alignment.asMap(B);\n\t\t\n\t\ttry {\n\t\t\tamap.clear();\n\t\t\tassertTrue(amap.isEmpty());\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\tfail(\"unable to clear\");\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tbmap.clear();\n\t\t\tassertTrue(bmap.isEmpty());\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\tfail(\"unable to clear\");\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testAsMap() {\n\t\tMap> amap = alignment.asMap(A), bmap = alignment.asMap(B);\n\t\t\n\t\tassertEquals(map, amap);\n\t\tassertEquals(pam, bmap);\n\t\t\n\t\tassertTrue(map.keySet().containsAll(amap.keySet()));\n\t\tassertTrue(pam.keySet().containsAll(bmap.keySet()));\n\t\t\n\t\tassertTrue(map.values().containsAll(amap.values()));\n\t\tassertTrue(pam.values().containsAll(bmap.values()));\n\t\t\n\t\tassertEquals(4, amap.size());\n\t\tassertEquals(5, bmap.size());\n\t\t\n\t\tassertFalse(amap.isEmpty());\n\t\tassertFalse(bmap.isEmpty());\n\t\t\n\t\talignment.add(new Mapping(id, 2, 2));\n\t\tassertTrue(alignment.asMap(A).get(2).contains(2));\n\t}\n\n\t@Test\n\tpublic void testFromMap() {\n\t\tassertEquals(alignment, Alignment.fromMap(one, two, map));\n\t\tassertEquals(alignment.reverse(), Alignment.fromMap(two, one, pam));\n\t\t\n\t\tassertEquals(alignment, Alignment.fromMap(one, two, alignment.asMap(A)));\n\t\tassertEquals(alignment.reverse(), Alignment.fromMap(two, one, alignment.asMap(B)));\n\t}\n\t\n\t@Test\n\tpublic void testReverse() {\n\t\tfor(PhrasePosition pos : PhrasePosition.values()) {\n\t\t\tassertEquals(alignment.get(pos), alignment.reverse().get(pos.opposite()));\n\t\t}\n\t\t\n\t\tfor(Mapping r : alignment.reverse()) {\n\t\t\tassertTrue(alignment.contains(r.reverse()));\n\t\t}\n\t\t\n\t\tassertEquals(alignment, alignment.reverse().reverse());\n\t}\n\t\n\t@Test\n\tpublic void testGetIndices() {\n\t\tSet is = new HashSet();\n\t\tfor(int i = 0; i < 4; i++) {\n\t\t\tis.add(i);\n\t\t}\n\t\t\n\t\tassertEquals(is, alignment.getIndices(A));\n\t\t\n\t\tis.add(4);\n\t\t\n\t\tassertEquals(is, alignment.getIndices(B));\n\t}\n}\n" }, { "path": "test/opennlp/ccg/alignment/IdentifiedPhraseReaderWriterTest.java", "content": "package opennlp.ccg.alignment;\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertNull;\nimport static org.junit.Assert.fail;\n\nimport java.io.IOException;\nimport java.io.StringReader;\nimport java.io.StringWriter;\nimport java.util.ArrayList;\nimport java.util.Iterator;\nimport java.util.List;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\n\npublic class IdentifiedPhraseReaderWriterTest {\n\n\tString lineSep = System.getProperty(\"line.separator\");\n\t\n\tString input = \"First phrase.\\n Second phrase\\r\\n Third phrase . \",\n\t\toutput = \"First phrase.\" + lineSep + \"Second phrase\"\n\t\t\t+ lineSep + \"Third phrase .\" + lineSep,\n\t\tpaddedOutput = \" First phrase. \" + lineSep + \" Second phrase \"\n\t\t\t\t+ lineSep + \" Third phrase . \" + lineSep;\n\t\n\tList phrases;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tphrases = new ArrayList();\n\t\tphrases.add(new Phrase(\"157\", 0, Alignments.tokenize(\"First phrase.\")));\n\t\tphrases.add(new Phrase(\"387b\", 1, Alignments.tokenize(\"Second phrase\")));\n\t\tphrases.add(new Phrase(\"55\", 2, Alignments.tokenize(\"Third phrase .\")));\n\t}\n\t\n\t@Test\n\tpublic void testIdentifiedPhraseReader() {\n\t\ttry {\n\t\t\tnew IdentifiedPhraseReader(new StringReader(\"\"), null);\n\t\t\tfail(\"able to specify null number base\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew IdentifiedPhraseReader(new StringReader(\"\"), IndexBase.ZERO, null, \"\");\n\t\t\tfail(\"able to specify null string\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew IdentifiedPhraseReader(new StringReader(\"\"), IndexBase.ZERO, \"\", null);\n\t\t\tfail(\"able to specify null string\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testBoth() {\n\t\tStringWriter sw = new StringWriter();\n\t\tIdentifiedPhraseWriter writer = new IdentifiedPhraseWriter(sw);\n\t\t\n\t\ttry {\n\t\t\tfor(Phrase p : phrases) {\n\t\t\t\twriter.writePhrase(p);\n\t\t\t}\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(io.getMessage());\n\t\t}\n\t\t\n\t\tIdentifiedPhraseReader reader = new IdentifiedPhraseReader(new StringReader(sw.getBuffer().toString()));\n\t\tIterator i = phrases.iterator();\n\t\t\n\t\ttry {\n\t\t\tPhrase p;\n\t\t\twhile((p = reader.readPhrase()) != null) {\n\t\t\t\tassertEquals(i.next(), p);\n\t\t\t}\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(io.getMessage());\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testReadPhrase() {\n\t\tIdentifiedPhraseReader reader = new IdentifiedPhraseReader(new StringReader(input));\n\t\t\n\t\ttry {\n\t\t\tIterator i = phrases.iterator();\n\t\t\tPhrase p;\n\t\t\twhile((p = reader.readPhrase()) != null) {\n\t\t\t\tassertEquals(i.next(), p);\n\t\t\t}\n\t\t\t\n\t\t\treader = new IdentifiedPhraseReader(new StringReader(\"\"));\n\t\t\tassertNull(reader.readPhrase());\n\t\t\t\n\t\t\treader.close();\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(io.getMessage());\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testWritePhrase() {\n\t\tStringWriter sw = new StringWriter();\n\t\tIdentifiedPhraseWriter writer = new IdentifiedPhraseWriter(sw);\n\t\t\n\t\ttry {\n\t\t\tfor(Phrase p : phrases) {\n\t\t\t\twriter.writePhrase(p);\n\t\t\t}\n\t\t\t\n\t\t\tassertEquals(output, sw.getBuffer().toString());\n\t\t\t\n\t\t\twriter.close();\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(io.getMessage());\n\t\t}\n\t\t\n\t\t// test padded version\n\t\tsw = new StringWriter();\n\t\twriter = new IdentifiedPhraseWriter(sw, writer.getWordSeparator(), writer.getPhraseTag(),\n\t\t\t\twriter.getPhraseIdentifierAttribute(), true);\n\t\t\n\t\ttry {\n\t\t\tfor(Phrase p : phrases) {\n\t\t\t\twriter.writePhrase(p);\n\t\t\t}\n\t\t\t\n\t\t\tassertEquals(paddedOutput, sw.getBuffer().toString());\n\t\t\t\n\t\t\twriter.close();\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(io.getMessage());\n\t\t}\n\t}\n}\n" }, { "path": "test/opennlp/ccg/alignment/IndexBaseTest.java", "content": "package opennlp.ccg.alignment;\n\nimport static opennlp.ccg.alignment.IndexBase.ONE;\nimport static opennlp.ccg.alignment.IndexBase.ZERO;\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertFalse;\nimport static org.junit.Assert.assertTrue;\nimport static org.junit.Assert.fail;\n\nimport org.junit.Test;\n\npublic class IndexBaseTest {\n\n\tInteger zero = new Integer(0), one = new Integer(1),\n\t\t\ttwo = new Integer(2), negOne = new Integer(-1),\n\t\t\tnegTwo = new Integer(-2);\n\t\n\t@Test\n\tpublic void testGetStart() {\n\t\tassertEquals(zero, ZERO.start);\n\t\tassertEquals(one, ONE.start);\n\t}\n\n\t@Test\n\tpublic void testGetNullValue() {\n\t\tassertEquals(negOne, ZERO.nullValue);\n\t\tassertEquals(zero, ONE.nullValue);\n\t}\n\n\t@Test\n\tpublic void testIsValidIndex() {\n\t\tassertTrue(ZERO.isValidIndex(zero));\n\t\tassertTrue(ZERO.isValidIndex(one));\n\t\tassertTrue(ZERO.isValidIndex(negOne));\n\t\tassertFalse(ZERO.isValidIndex(negTwo));\n\t\t\n\t\tassertTrue(ONE.isValidIndex(zero));\n\t\tassertTrue(ONE.isValidIndex(one));\n\t\tassertFalse(ONE.isValidIndex(negOne));\n\t\tassertFalse(ONE.isValidIndex(negTwo));\n\t\t\n\t\tassertTrue(ZERO.isValidIndex(two));\n\t\tassertTrue(ONE.isValidIndex(two));\n\t}\n\n\t@Test\n\tpublic void testTranslate() {\n\t\ttry {\n\t\t\tZERO.translate(negTwo, ONE);\n\t\t\tfail(\"ZERO able to translate \" + negTwo);\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tONE.translate(negOne, ZERO);\n\t\t\tfail(\"ONE able to translate \" + negOne);\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\t\t\n\t\t// identity tests\n\t\t\n\t\tassertEquals(one, ZERO.translate(one, ZERO));\n\t\tassertEquals(one, ONE.translate(one, ONE));\n\t\t\n\t\tassertEquals(zero, ZERO.translate(zero, ZERO));\n\t\tassertEquals(zero, ONE.translate(zero, ONE));\n\t\t\n\t\t// actual translations\n\t\t\n\t\tassertEquals(zero, ZERO.translate(negOne, ONE));\n\t\tassertEquals(one, ZERO.translate(zero, ONE));\n\t\tassertEquals(two, ZERO.translate(one, ONE));\n\t\t\n\t\tassertEquals(negOne, ONE.translate(zero, ZERO));\n\t\tassertEquals(zero, ONE.translate(one, ZERO));\n\t\tassertEquals(one, ONE.translate(two, ZERO));\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/alignment/MappingFormatTest.java", "content": "package opennlp.ccg.alignment;\n\nimport static org.junit.Assert.*;\n\nimport java.text.ParseException;\nimport java.util.HashSet;\nimport java.util.Set;\n\nimport opennlp.ccg.alignment.MappingFormat.Field;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\nimport static opennlp.ccg.alignment.Alignments.*;\n\npublic class MappingFormatTest {\n\n\tSet formats;\n\tMapping vanilla, chocolate, nullId, nullValue;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tMappingFormat moses = MappingFormat.getInstance(MOSES_ENCODING_SCHEME),\n\t\t\tmosesShort = MappingFormat.getInstance(MOSES_ENCODING_SCHEME,\n\t\t\t\tAlignments.MOSES_SHORT_FIELDS),\n\t\t\tmosesShortStrict = MappingFormat.getInstance(MOSES_ENCODING_SCHEME,\n\t\t\t\tAlignments.MOSES_SHORT_FIELDS, true),\n\t\t\tnaacl = MappingFormat.getInstance(NAACL_ENCODING_SCHEME),\n\t\t\tnaaclShort = MappingFormat.getInstance(NAACL_ENCODING_SCHEME, Alignments.NAACL_SHORT_FIELDS),\n\t\t\tnaaclShortStrict = MappingFormat.getInstance(NAACL_ENCODING_SCHEME, Alignments.NAACL_SHORT_FIELDS,\n\t\t\t\t\ttrue),\n\t\t\tnaaclVeryShort = MappingFormat.getInstance(NAACL_ENCODING_SCHEME, Alignments.NAACL_VERY_SHORT_FIELDS);\n\t\t\n\t\tformats = new HashSet();\n\t\tformats.add(moses);\n\t\tformats.add(mosesShort);\n\t\tformats.add(mosesShortStrict);\n\t\tformats.add(naacl);\n\t\tformats.add(naaclShort);\n\t\tformats.add(naaclShortStrict);\n\t\tformats.add(naaclVeryShort);\n\t\t\n\t\tvanilla = new Mapping(31, 4, 9);\n\t\tchocolate = new Mapping(31, 13, 5, Status.POSSIBLE, 0.75);\n\t\tnullId = new Mapping(null, 2, 2);\n\t\tnullValue = new Mapping(17, -1, 5);\n\t}\n\t\n\t@Test\n\tpublic void testMappingFormat() {\n\t\tSet fields = new HashSet();\n\t\t\n\t\tfields.add(Field.PHRASE_NUMBER_FIELD);\n\t\t\n\t\ttry {\n\t\t\tMappingFormat.getInstance(MOSES_ENCODING_SCHEME, fields, true);\n\t\t\tfail(\"able to create Moses formatter with ID field\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// should happen\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tMappingFormat.getInstance(NAACL_ENCODING_SCHEME, fields, false);\n\t\t\tfail(\"able to create NAACL formatter with only ID field\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// should happen\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tMappingFormat.getInstance(MOSES_ENCODING_SCHEME, null, true);\n\t\t\tfail(\"able to create Moses formatter with null fields\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// should happen\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tMappingFormat.getInstance(NAACL_ENCODING_SCHEME, null, false);\n\t\t\tfail(\"able to create NAACL formatter with null fields\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// should happen\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tMappingFormat.getInstance(null, fields, true);\n\t\t\tfail(\"able to create formatter null scheme\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// should happen\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testFormatMapping() {\n\t\tfor(MappingFormat mf : formats) {\n\t\t\tString v = mf.format(vanilla), c = mf.format(chocolate), ni = null;\n\t\t\tEncodingScheme es = mf.encodingScheme;\n\t\t\t\n\t\t\ttry {\n\t\t\t\tni = mf.formatMapping(nullId);\n\t\t\t}\n\t\t\tcatch(IllegalArgumentException e) {\n\t\t\t\tif(!es.getRequired().contains(MappingFormat.Field.PHRASE_NUMBER_FIELD)) {\n\t\t\t\t\tfail(\"unexpected exception: \" + e.getMessage());\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\ttry {\n\t\t\t\tmf.format(nullValue);\n\t\t\t\tfail(\"able to format mapping with null index\");\n\t\t\t}\n\t\t\tcatch(IllegalArgumentException expected) {\n\t\t\t\t// should happen\n\t\t\t}\n\t\t\t\t\n\t\t\tif(es.equals(MOSES_ENCODING_SCHEME)) {\n\t\t\t\tif(mf.fields.contains(MappingFormat.Field.STATUS_FIELD)) {\n\t\t\t\t\tassertEquals(\"13-5-P\", c);\n\t\t\t\t\tif(mf.isStrict()) {\n\t\t\t\t\t\tassertEquals(\"4-9-S\", v);\n\t\t\t\t\t\tassertEquals(\"2-2-S\", ni);\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tassertEquals(\"4-9\", v);\n\t\t\t\t\t\tassertEquals(\"2-2\", ni);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tassertEquals(\"4-9\", v);\n\t\t\t\t\tassertEquals(\"13-5\", c);\n\t\t\t\t\tassertEquals(\"2-2\", ni);\n\t\t\t\t}\n\t\t\t}\n\t\t\telse if(es.equals(NAACL_ENCODING_SCHEME)) {\n\t\t\t\ttry {\n\t\t\t\t\tni = mf.formatMapping(nullId);\n\t\t\t\t\tfail(\"able to format mapping with null id\");\n\t\t\t\t}\n\t\t\t\tcatch(IllegalArgumentException expected) {\n\t\t\t\t\t// should happen\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tif(mf.fields.contains(MappingFormat.Field.STATUS_FIELD)) {\n\t\t\t\t\tif(mf.fields.contains(MappingFormat.Field.CONFIDENCE_FIELD)) {\n\t\t\t\t\t\tif(mf.isStrict()) {\n\t\t\t\t\t\t\tassertEquals(\"31 5 10 S 1.0\", v);\n\t\t\t\t\t\t\tassertEquals(\"31 14 6 P 0.75\", c);\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse {\n\t\t\t\t\t\t\tassertEquals(\"31 5 10\", v);\n\t\t\t\t\t\t\tassertEquals(\"31 14 6 P 0.75\", c);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tassertEquals(\"31 14 6 P\", c);\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\tif(mf.isStrict()) {\n\t\t\t\t\t\t\tassertEquals(\"31 5 10 S\", v);\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse {\n\t\t\t\t\t\t\tassertEquals(\"31 5 10\", v);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tassertEquals(\"31 5 10\", v);\n\t\t\t\t\tassertEquals(\"31 14 6\", c);\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testParseMapping() {\n\t\tfor(MappingFormat mf : formats) {\n\t\t\tEncodingScheme es = mf.encodingScheme;\n\t\t\tMapping v, c, ni;\n\t\t\t\n\t\t\tif(es.equals(MOSES_ENCODING_SCHEME)) {\n\t\t\t\ttry {\n\t\t\t\t\tmf.parseMapping(\"-1-5\");\n\t\t\t\t\tfail(\"able to parse mapping with negative index\");\n\t\t\t\t}\n\t\t\t\tcatch(ParseException expected) {\n\t\t\t\t\t// should\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tif(mf.fields.contains(MappingFormat.Field.STATUS_FIELD)) {\n\t\t\t\t\ttry {\n\t\t\t\t\t\tv = mf.parseMapping(\"4-9-S\").copyWithPhraseNumber(chocolate.phraseNumber);\n\t\t\t\t\t\t\n\t\t\t\t\t\tassertEquals(vanilla, v);\n\t\t\t\t\t\t\n\t\t\t\t\t\tc = mf.parseMapping(\"13-5-P\").copyWithPhraseNumber(chocolate.phraseNumber);\n\t\t\t\t\t\tc.setConfidence(chocolate.confidence);\n\t\t\t\t\t\tassertEquals(chocolate, c);\n\t\t\t\t\t\t\n\t\t\t\t\t\tni = mf.parseMapping(\"2-2-S\");\n\t\t\t\t\t\tassertEquals(nullId, ni);\n\t\t\t\t\t\t\n\t\t\t\t\t\tif(mf.isStrict()) {\n\t\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\t\tmf.parseMapping(\"4-9\");\n\t\t\t\t\t\t\t\tfail(\"strict format able to parse loose input\");\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\tcatch(ParseException expected) {\n\t\t\t\t\t\t\t\tassertEquals(3, expected.getErrorOffset());\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse {\n\t\t\t\t\t\t\tv = mf.parseMapping(\"4-9\").copyWithPhraseNumber(chocolate.phraseNumber);\n\t\t\t\t\t\t\tassertEquals(vanilla, v);\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\tni = mf.parseMapping(\"2-2\");\n\t\t\t\t\t\t\tassertEquals(nullId, ni);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\tcatch(ParseException p) {\n\t\t\t\t\t\tfail(\"parse exception: \" + p.getMessage());\n\t\t\t\t\t}\t\t\t\t\t\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\ttry {\n\t\t\t\t\t\tv = mf.parseMapping(\"4-9\").copyWithPhraseNumber(chocolate.phraseNumber);\n\t\t\t\t\t\tassertEquals(vanilla, v);\n\t\t\t\t\t\t\n\t\t\t\t\t\tc = mf.parseMapping(\"13-5\").copyWithPhraseNumber(chocolate.phraseNumber);\n\t\t\t\t\t\tc.setStatus(Status.POSSIBLE);\n\t\t\t\t\t\tc.setConfidence(chocolate.confidence);\n\t\t\t\t\t\tassertEquals(chocolate, c);\n\t\t\t\t\t\t\n\t\t\t\t\t\tni = mf.parseMapping(\"2-2\");\n\t\t\t\t\t\tassertEquals(nullId, ni);\n\t\t\t\t\t}\n\t\t\t\t\tcatch(ParseException p) {\n\t\t\t\t\t\tfail(\"parse exception: \" + p.getMessage());\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\telse if(es.equals(NAACL_ENCODING_SCHEME)) {\n\t\t\t\ttry {\n\t\t\t\t\tmf.parseMapping(\"31 0 6 S 1.0\");\n\t\t\t\t\tfail(\"able to parse mapping with 0 index, but index base is 1\");\n\t\t\t\t}\n\t\t\t\tcatch(ParseException expected) {\n\t\t\t\t\t// should\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tif(mf.fields.contains(MappingFormat.Field.STATUS_FIELD)) {\n\t\t\t\t\tif(mf.fields.contains(MappingFormat.Field.CONFIDENCE_FIELD)) {\n\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\tif(mf.isStrict()) {\n\t\t\t\t\t\t\t\tv = mf.parseMapping(\"31 5 10 S 1.0\");\n\t\t\t\t\t\t\t\tassertEquals(vanilla, v);\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\t\t\tmf.parseMapping(\"31 5 10\");\n\t\t\t\t\t\t\t\t\tfail(\"able to parse loose input with strict format\");\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\tcatch(ParseException expected) {\n\t\t\t\t\t\t\t\t\tassertEquals(7, expected.getErrorOffset());\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\telse {\n\t\t\t\t\t\t\t\tv = mf.parseMapping(\"31 5 10\");\n\t\t\t\t\t\t\t\tassertEquals(vanilla, v);\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\tc = mf.parseMapping(\"31 14 6 P 0.75\");\n\t\t\t\t\t\t\tassertEquals(chocolate, c);\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\t\tni = mf.parseMapping(\"3 3 S\");\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\tcatch(ParseException should) {\n\t\t\t\t\t\t\t\t// expected\n\t\t\t\t\t\t\t\tassertEquals(4, should.getErrorOffset());\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t\tcatch(ParseException p) {\n\t\t\t\t\t\t\tfail(\"parse exception: \" + p.getMessage());\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\tif(mf.isStrict()) {\n\t\t\t\t\t\t\t\tv = mf.parseMapping(\"31 5 10 S\");\n\t\t\t\t\t\t\t\tassertEquals(vanilla, v);\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\t\t\tmf.parseMapping(\"31 5 10\");\n\t\t\t\t\t\t\t\t\tfail(\"able to parse loose input with strict format\");\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\tcatch(ParseException expected) {\n\t\t\t\t\t\t\t\t\tassertEquals(7, expected.getErrorOffset());\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\telse {\n\t\t\t\t\t\t\t\tv = mf.parseMapping(\"31 5 10\");\n\t\t\t\t\t\t\t\tassertEquals(vanilla, v);\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\tc = mf.parseMapping(\"31 14 6 P\");\n\t\t\t\t\t\t\tc.setConfidence(chocolate.confidence);\n\t\t\t\t\t\t\tassertEquals(chocolate, c);\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\t\tni = mf.parseMapping(\"3 3 S\");\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\tcatch(ParseException should) {\n\t\t\t\t\t\t\t\t// expected\n\t\t\t\t\t\t\t\tassertEquals(4, should.getErrorOffset());\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t\tcatch(ParseException p) {\n\t\t\t\t\t\t\tfail(\"parse exception: \" + p.getMessage());\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\ttry {\n\t\t\t\t\t\tv = mf.parseMapping(\"31 5 10\");\n\t\t\t\t\t\tassertEquals(vanilla, v);\n\t\t\t\t\t\t\n\t\t\t\t\t\tc = mf.parseMapping(\"31 14 6\");\n\t\t\t\t\t\tc.setStatus(Status.POSSIBLE);\n\t\t\t\t\t\tc.setConfidence(chocolate.confidence);\n\t\t\t\t\t\tassertEquals(chocolate, c);\n\t\t\t\t\t\t\n\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\tni = mf.parseMapping(\"3 3 S\");\n\t\t\t\t\t\t\tfail(\"able to parse mapping without ID\");\n\t\t\t\t\t\t}\n\t\t\t\t\t\tcatch(ParseException should) {\n\t\t\t\t\t\t\t// expected\n\t\t\t\t\t\t\tassertEquals(4, should.getErrorOffset());\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\tcatch(ParseException p) {\n\t\t\t\t\t\tfail(\"parse exception: \" + p.getMessage());\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\t\t\t\n\t\t}\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/alignment/MappingGroupTest.java", "content": "package opennlp.ccg.alignment;\n\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertNotSame;\nimport static org.junit.Assert.fail;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class MappingGroupTest {\n\n\tMappingGroup one, two;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tone = new MappingGroup(37, 12);\n\t\ttwo = new MappingGroup(1, 8);\n\t}\n\t\n\t@Test\n\tpublic void testMappingGroup() {\n\t\ttry {\n\t\t\tnew MappingGroup(null, 1);\n\t\t\tfail(\"able to specify null number\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew MappingGroup(37, -1);\n\t\t\tfail(\"able to specify negative length\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testEqualsObject() {\n\t\tassertNotSame(one, two);\n\t\tassertNotSame(two, null);\n\t\tassertEquals(one, new MappingGroup(one.phraseNumber, one.length));\n\t}\n\n\t@Test\n\tpublic void testCompareTo() {\n\t\tassertEquals(0, one.compareTo(one));\n\t\tassertEquals(1, one.compareTo(two));\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/alignment/MappingReaderWriterTest.java", "content": "package opennlp.ccg.alignment;\n\nimport static opennlp.ccg.alignment.Status.POSSIBLE;\nimport static opennlp.ccg.alignment.Status.SURE;\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertFalse;\nimport static org.junit.Assert.assertNull;\nimport static org.junit.Assert.fail;\n\nimport java.io.IOException;\nimport java.io.StringReader;\nimport java.io.StringWriter;\nimport java.util.Iterator;\nimport java.util.LinkedHashSet;\nimport java.util.Set;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\nimport static opennlp.ccg.alignment.Alignments.*;\n\npublic class MappingReaderWriterTest {\n\n\tMappingReader mosesReader, naaclReader;\n\tMappingWriter mosesWriter, naaclWriter;\n\tStringWriter mosesStringWriter, naaclStringWriter;\n\tMappingFormat mosesFormat = MappingFormat.getInstance(MOSES_ENCODING_SCHEME),\n\t\tnaaclFormat = MappingFormat.getInstance(NAACL_ENCODING_SCHEME, Alignments.NAACL_SHORT_FIELDS, false);\n\t\n\tSet mosesMappings = new LinkedHashSet(), naaclMappings = new LinkedHashSet();\n\tSet mosesGroups = new LinkedHashSet(),\n\t\t\tnaaclGroups = new LinkedHashSet();\n\t\n\tString lineSep = System.getProperty(\"line.separator\");\n\t\n\tString mosesInput = \"0-1-S 0-0 2-1-P 3-3 4-8 21-23\\r\\n3-4 34-55-P 1-4 23-1-S\\n\",\n\t\t\tmosesOutput = \"0-1 0-0 2-1-P 3-3 4-8 21-23\" + lineSep + \"3-4 34-55-P 1-4 23-1\",\n\t\t\tnaaclInput = \"17 1 1 S\\r17 2 3 P\\n17 5 5\\r\\n17 4 4 S 0.75\\n37 3 2 P\",\n\t\t\tnaaclOutput = \"17 1 1\" + lineSep + \"17 2 3 P\" + lineSep + \"17 5 5\"\n\t\t\t\t+ lineSep + \"17 4 4\" + lineSep + \"37 3 2 P\",\n\t\t\temptyInput = \"\",\n\t\t\tlineEndingOnly = \"\\n\";\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tmosesReader = new MappingReader(new StringReader(mosesInput), mosesFormat);\n\t\tnaaclReader = new MappingReader(new StringReader(naaclInput), naaclFormat);\n\t\t\n\t\tmosesStringWriter = new StringWriter();\n\t\tmosesWriter = new MappingWriter(mosesStringWriter, mosesFormat);\n\t\tnaaclStringWriter = new StringWriter();\n\t\tnaaclWriter = new MappingWriter(naaclStringWriter, naaclFormat);\n\t\t\n\t\tmosesMappings.add(new Mapping(0, 0, 1, SURE));\n\t\tmosesMappings.add(new Mapping(0, 0, 0));\n\t\tmosesMappings.add(new Mapping(0, 2, 1, POSSIBLE));\n\t\tmosesMappings.add(new Mapping(0, 3, 3));\n\t\tmosesMappings.add(new Mapping(0, 4, 8));\n\t\tmosesMappings.add(new Mapping(0, 21, 23));\n\t\tmosesMappings.add(new Mapping(1, 3, 4));\n\t\tmosesMappings.add(new Mapping(1, 34, 55, POSSIBLE));\n\t\tmosesMappings.add(new Mapping(1, 1, 4));\n\t\tmosesMappings.add(new Mapping(1, 23, 1, SURE));\n\t\t\n\t\tnaaclMappings.add(new Mapping(17, 0, 0, SURE));\n\t\tnaaclMappings.add(new Mapping(17, 1, 2, POSSIBLE));\n\t\tnaaclMappings.add(new Mapping(17, 4, 4));\n\t\tnaaclMappings.add(new Mapping(17, 3, 3, SURE, new Double(0.75d)));\n\t\tnaaclMappings.add(new Mapping(37, 2, 1, POSSIBLE));\n\t\t\n\t\tmosesGroups.add(new MappingGroup(0, 6));\n\t\tmosesGroups.add(new MappingGroup(1, 4));\n\t\t\n\t\tnaaclGroups.add(new MappingGroup(17, 4));\n\t\tnaaclGroups.add(new MappingGroup(37, 1));\n\t}\n\t\n\t@Test\n\tpublic void testConstructors() {\n\t\ttry {\n\t\t\tnew MappingReader(new StringReader(\"\"), null);\n\t\t\tfail(\"able to specify null format\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew MappingWriter(new StringWriter(), null);\n\t\t\tfail(\"able to specify null format\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testBoth() {\n\t\tIterator mi = mosesMappings.iterator();\n\t\tIterator gi = mosesGroups.iterator();\n\t\t\n\t\tStringWriter sw = new StringWriter();\n\t\tMappingWriter mw = new MappingWriter(sw, mosesFormat);\n\t\t\n\t\ttry {\n\t\t\twhile(gi.hasNext()) {\n\t\t\t\tmw.startGroup(gi.next());\n\t\t\t\twhile(mw.canWrite() && mi.hasNext()) {\n\t\t\t\t\tmw.writeMapping(mi.next());\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tgi = mosesGroups.iterator();\n\t\t\tmi = mosesMappings.iterator();\n\t\t\tMappingReader mr = new MappingReader(new StringReader(sw.getBuffer().toString()), mosesFormat);\n\t\t\t\n\t\t\twhile(gi.hasNext()) {\n\t\t\t\tMappingGroup g = gi.next();\n\t\t\t\tassertEquals(g, mr.nextGroup());\n\t\t\t\twhile(mr.canRead()) {\n\t\t\t\t\tassertEquals(mi.next(), mr.readMapping());\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tmi = naaclMappings.iterator();\n\t\t\tgi = naaclGroups.iterator();\n\t\t\tsw = new StringWriter();\n\t\t\t\n\t\t\tmw.close();\n\t\t\tmw = new MappingWriter(sw, naaclFormat);\n\t\t\t\n\t\t\twhile(gi.hasNext()) {\n\t\t\t\tmw.startGroup(gi.next());\n\t\t\t\twhile(mw.canWrite() && mi.hasNext()) {\n\t\t\t\t\tmw.writeMapping(mi.next());\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tmi = naaclMappings.iterator();\n\t\t\tgi = naaclGroups.iterator();\n\t\t\t\n\t\t\tmr.close();\n\t\t\tmr = new MappingReader(new StringReader(sw.getBuffer().toString()), naaclFormat);\n\t\t\t\n\t\t\twhile(gi.hasNext()) {\n\t\t\t\tMappingGroup g = gi.next();\n\t\t\t\tassertEquals(g, mr.nextGroup());\n\t\t\t\twhile(mr.canRead()) {\n\t\t\t\t\tMapping m = mi.next();\n\t\t\t\t\tm.setConfidence(Alignments.DEFAULT_CONFIDENCE);\n\t\t\t\t\tassertEquals(m, mr.readMapping());\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tmosesWriter.close();\n\t\t\tnaaclWriter.close();\n\t\t\tmr.close();\n\t\t\tmw.close();\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(io.getMessage());\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testMappingWriter() {\n\t\tIterator mi = mosesMappings.iterator();\n\t\tIterator gi = mosesGroups.iterator();\n\t\t\n\t\ttry {\n\t\t\twhile(gi.hasNext()) {\n\t\t\t\tmosesWriter.startGroup(gi.next());\n\t\t\t\twhile(mosesWriter.canWrite()) {\n\t\t\t\t\tmosesWriter.writeMapping(mi.next());\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tmosesWriter.close();\n\t\t\t\n\t\t\tassertEquals(mosesOutput, mosesStringWriter.getBuffer().toString());\n\t\t\t\n\t\t\tmi = naaclMappings.iterator();\n\t\t\tgi = naaclGroups.iterator();\n\t\t\t\t\t\t\n\t\t\twhile(gi.hasNext()) {\n\t\t\t\tnaaclWriter.startGroup(gi.next());\n\t\t\t\twhile(naaclWriter.canWrite()) {\n\t\t\t\t\tnaaclWriter.writeMapping(mi.next());\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tnaaclWriter.close();\n\t\t\t\n\t\t\tassertEquals(naaclOutput, naaclStringWriter.getBuffer().toString());\n\t\t\t\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(io.getMessage());\n\t\t}\n\t\t\n\t\tMappingWriter mw = new MappingWriter(new StringWriter(), naaclFormat);\n\t\ttry {\n\t\t\tmw.writeMapping(new Mapping(1, 0));\n\t\t\tfail(\"able to write mapping without starting group\");\n\t\t}\n\t\tcatch(IOException expected) {\n\t\t\t//should happen\n\t\t}\n\t\t\n\t\tmw = new MappingWriter(new StringWriter(), mosesFormat);\n\t\t\n\t\ttry {\n\t\t\tmw.startGroup(new MappingGroup(0, 1));\n\t\t\ttry {\n\t\t\t\tmw.writeMapping(new Mapping(1, 1, 1));\n\t\t\t\tfail(\"able to write mapping from different group\");\n\t\t\t}\n\t\t\tcatch(IOException expected) {\n\t\t\t\t//should happen\n\t\t\t}\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(\"problem testing: \" + io.getMessage());\n\t\t}\n\t\t\n\t\tmw = new MappingWriter(new StringWriter(), naaclFormat);\n\t\t\n\t\ttry {\n\t\t\tmw.startGroup(new MappingGroup(0, 1));\n\t\t\tmw.writeMapping(new Mapping(0, 1, 1));\n\t\t\ttry {\n\t\t\t\tmw.writeMapping(new Mapping(0, 1, 2));\n\t\t\t\tfail(\"able to write too many mappings\");\n\t\t\t}\n\t\t\tcatch(IOException expected) {\n\t\t\t\t//should happen\n\t\t\t}\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(\"problem testing: \" + io.getMessage());\n\t\t}\n\t\t\n\t\tmw = new MappingWriter(new StringWriter(), mosesFormat);\n\t\t\n\t\ttry {\n\t\t\tmw.startGroup(new MappingGroup(0, 2));\n\t\t\tmw.writeMapping(new Mapping(0, 1, 1));\n\t\t\ttry {\n\t\t\t\tmw.close();\n\t\t\t\tfail(\"able to write too few mappings\");\n\t\t\t}\n\t\t\tcatch(IOException expected) {\n\t\t\t\t//should happen\n\t\t\t}\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(\"problem testing: \" + io.getMessage());\n\t\t}\n\t\t\n\t\tStringWriter sw = new StringWriter();\t\t\n\t\tmw = new MappingWriter(sw, naaclFormat);\n\t\t\n\t\ttry {\n\t\t\tmw.startGroup(new MappingGroup(0, 0));\n\t\t\tmw.endGroup();\n\t\t\tmw.close();\n\t\t\tassertEquals(\"\", sw.getBuffer().toString());\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(\"problem testing: \" + io.getMessage());\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testMappingReader() {\n\t\tIterator mi = mosesMappings.iterator();\n\t\tIterator gi = mosesGroups.iterator();\n\t\t\n\t\ttry {\n\t\t\tMappingGroup g;\n\t\t\twhile((g = mosesReader.nextGroup()) != null) {\n\t\t\t\tassertEquals(gi.next(), g);\n\t\t\t\twhile(mosesReader.canRead()) {\n\t\t\t\t\tassertEquals(mi.next(), mosesReader.readMapping());\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tmosesReader.close();\n\t\t\t\n\t\t\tmi = naaclMappings.iterator();\n\t\t\tgi = naaclGroups.iterator();\n\t\t\t\n\t\t\twhile((g = naaclReader.nextGroup()) != null) {\n\t\t\t\tassertEquals(gi.next(), g);\n\t\t\t\twhile(naaclReader.canRead()) {\n\t\t\t\t\tassertEquals(mi.next(), naaclReader.readMapping());\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tnaaclReader.close();\n\t\t\t\n\t\t\tmosesReader = new MappingReader(new StringReader(emptyInput), mosesFormat);\n\t\t\tnaaclReader = new MappingReader(new StringReader(emptyInput), naaclFormat);\n\t\t\t\n\t\t\ttry {\n\t\t\t\tassertFalse(mosesReader.ready());\n\t\t\t\tassertFalse(mosesReader.canRead());\n\t\t\t\tassertNull(mosesReader.nextGroup());\n\t\t\t}\n\t\t\tcatch(IOException io) {\n\t\t\t\tfail(\"problem testing: \" + io.getMessage());\n\t\t\t}\n\t\t\t\n\t\t\ttry {\n\t\t\t\tassertFalse(naaclReader.ready());\n\t\t\t\tassertFalse(naaclReader.canRead());\n\t\t\t\tassertNull(naaclReader.nextGroup());\n\t\t\t}\n\t\t\tcatch(IOException expected) {\n\t\t\t\t// should happen\n\t\t\t}\n\t\t\t\n\t\t\tmosesReader = new MappingReader(new StringReader(lineEndingOnly), mosesFormat);\n\t\t\tnaaclReader = new MappingReader(new StringReader(lineEndingOnly), naaclFormat);\n\t\t\t\n\t\t\ttry {\n\t\t\t\tassertFalse(mosesReader.ready());\n\t\t\t\tassertFalse(mosesReader.canRead());\n\t\t\t\tassertNull(mosesReader.nextGroup());\n\t\t\t}\n\t\t\tcatch(IOException io) {\n\t\t\t\tfail(\"problem testing: \" + io.getMessage());\n\t\t\t}\n\t\t\t\n\t\t\ttry {\n\t\t\t\tassertFalse(naaclReader.ready());\n\t\t\t\tassertFalse(naaclReader.canRead());\n\t\t\t\tassertNull(naaclReader.nextGroup());\n\t\t\t}\n\t\t\tcatch(IOException io) {\n\t\t\t\tfail(\"problem testing: \" + io.getMessage());\n\t\t\t}\n\t\t\t\n\t\t\tmosesReader = new MappingReader(new StringReader(\"5-4-\"), mosesFormat);\n\t\t\tnaaclReader = new MappingReader(new StringReader(\"0 S\\n\"), naaclFormat);\n\t\t\t\n\t\t\ttry {\n\t\t\t\tmosesReader.nextGroup();\n\t\t\t\tfail(\"able to get next group from garbage input\");\n\t\t\t}\n\t\t\tcatch(IOException expected) {\n\t\t\t\t// should happen\n\t\t\t}\n\t\t\t\n\t\t\ttry {\n\t\t\t\tnaaclReader.nextGroup();\n\t\t\t\tfail(\"able to get next group from garbage input\");\n\t\t\t}\n\t\t\tcatch(IOException expected) {\n\t\t\t\t// should happen\n\t\t\t}\n\t\t\t\n\t\t\tmosesReader = new MappingReader(new StringReader(mosesInput), mosesFormat);\n\t\t\tnaaclReader = new MappingReader(new StringReader(naaclInput), naaclFormat);\n\t\t\t\n\t\t\ttry {\n\t\t\t\tmosesReader.readMapping();\n\t\t\t\tfail(\"able to read mapping without group\");\n\t\t\t}\n\t\t\tcatch(IOException expected) {\n\t\t\t\t// should happen\n\t\t\t}\n\t\t\t\n\t\t\ttry {\n\t\t\t\tnaaclReader.readMapping();\n\t\t\t\tfail(\"able to read mapping without group\");\n\t\t\t}\n\t\t\tcatch(IOException expected) {\n\t\t\t\t// should happen\n\t\t\t}\n\t\t\t\n\t\t\tmosesReader = new MappingReader(new StringReader(mosesInput), mosesFormat);\n\t\t\tnaaclReader = new MappingReader(new StringReader(naaclInput), naaclFormat);\n\t\t\t\n\t\t\tMappingGroup mg = mosesReader.nextGroup();\n\t\t\tfor(int i = 0; i < mg.length - 1; i++) {\n\t\t\t\tmosesReader.readMapping();\n\t\t\t}\n\t\t\t\n\t\t\ttry {\n\t\t\t\tmosesReader.close();\n\t\t\t\tfail(\"able to read too few mappings\");\n\t\t\t}\n\t\t\tcatch(IOException expected) {\n\t\t\t\t// should happen\n\t\t\t}\n\t\t\t\n\t\t\tmg = naaclReader.nextGroup();\n\t\t\tfor(int i = 0; i < mg.length - 1; i++) {\n\t\t\t\tnaaclReader.readMapping();\n\t\t\t}\n\t\t\t\n\t\t\ttry {\n\t\t\t\tnaaclReader.close();\n\t\t\t\tfail(\"able to read too few mappings\");\n\t\t\t}\n\t\t\tcatch(IOException expected) {\n\t\t\t\t// should happen\n\t\t\t}\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(io.getMessage());\n\t\t}\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/alignment/MappingTest.java", "content": "package opennlp.ccg.alignment;\n\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertNotSame;\nimport static org.junit.Assert.fail;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class MappingTest {\n\n\tInteger id, first, second;\n\tMapping mapping;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tid = new Integer(37);\n\t\tfirst = new Integer(4);\n\t\tsecond = new Integer(7);\n\t\tmapping = new Mapping(id, first, second);\n\t}\n\n\t@Test\n\tpublic void testMapping() {\n\t\ttry {\n\t\t\tnew Mapping(null, 3);\n\t\t\tfail(\"able to specify null index\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew Mapping(3, null);\n\t\t\tfail(\"able to specify null index\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew Mapping(1, 2, 3, null);\n\t\t\tfail(\"able to specify null status\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew Mapping(1, 2, 3, Status.SURE, null);\n\t\t\tfail(\"able to specify null confidence\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testGet() {\n\t\tassertEquals(id, mapping.getPhraseNumber());\n\t\t\n\t\tassertEquals(first, mapping.getA());\n\t\tassertEquals(first, mapping.get(PhrasePosition.A));\n\t\t\n\t\tassertEquals(second, mapping.getB());\n\t\tassertEquals(second, mapping.get(PhrasePosition.B));\n\t}\n\n\t@Test\n\tpublic void testCompareTo() {\n\t\tMapping m = new Mapping(id, first, second),\n\t\t\t\tn = new Mapping(id, first - 1, second + 1),\n\t\t\t\to = new Mapping(id, first, second - 1),\n\t\t\t\tp = new Mapping(id, first + 1, second);\n\t\t\n\t\tassertEquals(0, mapping.compareTo(m));\n\t\tassertEquals(0, m.compareTo(mapping));\n\t\t\n\t\tassertEquals(1, mapping.compareTo(n));\n\t\tassertEquals(-1, n.compareTo(mapping));\n\t\t\n\t\tassertEquals(1, mapping.compareTo(o));\n\t\tassertEquals(-1, o.compareTo(mapping));\n\t\t\n\t\tassertEquals(-1, mapping.compareTo(p));\n\t\tassertEquals(1, p.compareTo(mapping));\n\t}\n\n\t@Test\n\tpublic void testEqualsObject() {\n\t\tMapping m = new Mapping(id, first, second),\n\t\t\t\tn = new Mapping(id, first - 1, second + 1),\n\t\t\t\to = new Mapping(id, first, second - 1),\n\t\t\t\tp = new Mapping(id, first + 1, second);\n\t\t\n\t\tassertEquals(mapping, m);\n\t\tassertNotSame(mapping, n);\n\t\tassertNotSame(mapping, o);\n\t\tassertNotSame(mapping, p);\n\t}\n\t\n\t@Test\n\tpublic void testReverse() {\n\t\tassertEquals(new Mapping(id, second, first), mapping.reverse());\n\t}\n\n\t@Test\n\tpublic void testSet() {\n\t\tassertEquals(Alignments.DEFAULT_STATUS, mapping.getStatus());\n\t\tmapping.setStatus(Status.POSSIBLE);\n\t\tassertEquals(Status.POSSIBLE, mapping.getStatus());\n\t\t\n\t\tassertEquals(Alignments.DEFAULT_CONFIDENCE, mapping.getConfidence());\n\t\tmapping.setConfidence(0.5);\n\t\tassertEquals(Double.valueOf(0.5d), mapping.getConfidence());\n\t}\n\t\n\t@Test\n\tpublic void testHashCode() {\n\t\tint hash = mapping.hashCode();\n\t\t\n\t\tmapping.setStatus(Status.POSSIBLE);\n\t\tassertEquals(hash, mapping.hashCode());\n\t\tmapping.setConfidence(0.5);\n\t\tassertEquals(hash, mapping.hashCode());\n\t}\n}\n" }, { "path": "test/opennlp/ccg/alignment/PhraseReaderWriterTest.java", "content": "package opennlp.ccg.alignment;\n\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertNull;\nimport static org.junit.Assert.fail;\n\nimport java.io.IOException;\nimport java.io.StringReader;\nimport java.io.StringWriter;\nimport java.util.ArrayList;\nimport java.util.Iterator;\nimport java.util.List;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class PhraseReaderWriterTest {\n\n\tString lineSep = System.getProperty(\"line.separator\");\n\t\n\tString input = \"Phrase one.\\nPhrase two\\r\\nPhrase three .\",\n\t\t\toutput = \"Phrase one.\" + lineSep + \"Phrase two\" + lineSep + \"Phrase three .\" + lineSep;\n\t\n\tPhraseReader reader;\n\tPhraseWriter writer;\n\tList phrases;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tphrases = new ArrayList();\n\t\tphrases.add(new Phrase(0, Alignments.tokenize(\"Phrase one.\")));\n\t\tphrases.add(new Phrase(1, Alignments.tokenize(\"Phrase two\")));\n\t\tphrases.add(new Phrase(2, Alignments.tokenize(\"Phrase three .\")));\n\t}\n\n\t@Test\n\tpublic void testConstructors() {\n\t\ttry {\n\t\t\tnew PhraseReader(new StringReader(\"\"), null);\n\t\t\tfail(\"able to specify null number base\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew PhraseWriter(new StringWriter(), null);\n\t\t\tfail(\"able to specify null number base\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testBoth() {\n\t\tStringWriter sw = new StringWriter();\n\t\twriter = new PhraseWriter(sw);\n\t\t\n\t\ttry {\n\t\t\tfor(Phrase p : phrases) {\n\t\t\t\twriter.writePhrase(p);\n\t\t\t}\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(io.getMessage());\n\t\t}\n\t\t\n\t\treader = new PhraseReader(new StringReader(sw.getBuffer().toString()));\n\t\tIterator i = phrases.iterator();\n\t\t\n\t\ttry {\n\t\t\tPhrase p;\n\t\t\twhile((p = reader.readPhrase()) != null) {\n\t\t\t\tassertEquals(i.next(), p);\n\t\t\t}\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(io.getMessage());\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testReadPhrase() {\n\t\treader = new PhraseReader(new StringReader(input));\n\t\t\n\t\ttry {\n\t\t\tIterator i = phrases.iterator();\n\t\t\tPhrase p;\n\t\t\twhile((p = reader.readPhrase()) != null) {\n\t\t\t\tassertEquals(i.next(), p);\n\t\t\t}\n\t\t\t\n\t\t\treader = new PhraseReader(new StringReader(\"\"), reader.getNumberBase());\n\t\t\tassertNull(reader.readPhrase());\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(io.getMessage());\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testWritePhrase() {\n\t\tStringWriter sw = new StringWriter();\n\t\twriter = new PhraseWriter(sw);\n\t\t\n\t\ttry {\n\t\t\tfor(Phrase p : phrases) {\n\t\t\t\twriter.writePhrase(p);\n\t\t\t}\n\t\t\t\n\t\t\tassertEquals(output, sw.getBuffer().toString());\n\t\t}\n\t\tcatch(IOException io) {\n\t\t\tfail(io.getMessage());\n\t\t}\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/alignment/PhraseTest.java", "content": "package opennlp.ccg.alignment;\n\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertNotSame;\nimport static org.junit.Assert.fail;\n\nimport java.util.ArrayList;\nimport java.util.Iterator;\nimport java.util.List;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class PhraseTest {\n\n\tPhrase phrase;\n\t\n\tString[] wordList = Alignments.tokenize(\"This is a test .\");\n\t\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tphrase = new Phrase(37, wordList);\n\t}\n\n\t@Test\n\tpublic void testCompare() {\n\t\tPhrase o = new Phrase(43, phrase), t = new Phrase(43, phrase);\n\t\t\n\t\tassertEquals(-1, phrase.compareTo(o));\n\t\tassertEquals(1, t.compareTo(phrase));\n\t\tassertEquals(0, o.compareTo(t));\n\t\tassertEquals(0, phrase.compareTo(phrase));\n\t}\n\t\n\t@Test\n\tpublic void testSize() {\n\t\tassertEquals(5, phrase.size());\n\t}\n\n\t@Test\n\tpublic void testPhraseComparableOfIListOfString() {\n\t\tassertEquals(phrase, new Phrase(phrase.getNumber(), wordList));\n\t\t\n\t\tString[] str = null;\n\t\t\n\t\ttry {\n\t\t\tnew Phrase(phrase.getNumber(), str);\n\t\t\tfail(\"able to create phrase with null word list\");\n\t\t}\n\t\tcatch(IllegalArgumentException ex) {\n\t\t\t// expected\n\t\t}\n\t\t\n\t\tstr = new String[]{\"blah\", null, \"blah\"};\n\t\t\n\t\ttry {\n\t\t\tnew Phrase(phrase.getNumber(), str);\n\t\t\tfail(\"able to create phrase with null word in list\");\n\t\t}\n\t\tcatch(IllegalArgumentException ex) {\n\t\t\t// expected\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testGetNumber() {\n\t\tassertEquals(new Integer(37), phrase.getNumber());\n\t}\n\n\t@Test\n\tpublic void testGetInt() {\n\t\tassertEquals(\"is\", phrase.get(1));\n\t\tassertEquals(\"a\", phrase.get(2));\n\t\tassertEquals(\".\", phrase.get(4));\n\t\t\n\t\ttry {\n\t\t\tphrase.get(phrase.size());\n\t\t\tfail(\"able to access word in phrase after end\");\n\t\t}\n\t\tcatch(IndexOutOfBoundsException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testSetIntString() {\n\t\ttry {\n\t\t\tphrase.set(2, \"sdfskjdlkjflksjdlkj\");\n\t\t\tfail(\"able to set\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testAdd() {\n\t\ttry {\n\t\t\tphrase.add(\"blah\");\n\t\t\tfail(\"able to add\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t}\n\t\n\tpublic void testRemove() {\n\t\ttry {\n\t\t\tphrase.remove(\"is\");\n\t\t\tfail(\"able to remove\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t}\n\t\n\tpublic void testIteratorRemove() {\n\t\ttry {\n\t\t\tIterator i = phrase.iterator();\n\t\t\ti.next();\n\t\t\ti.remove();\n\t\t\tfail(\"able to remove via iterator\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testEqualsObject() {\n\t\tList l = new ArrayList();\n\t\t\n\t\tfor(int i = 0; i < 3; i++) {\n\t\t\tl.add(\"blah\");\n\t\t}\n\t\t\n\t\tPhrase same = new Phrase(phrase.getNumber(), phrase), diff = new Phrase(17, l);\n\t\t\n\t\tassertEquals(phrase, same);\n\t\tassertNotSame(phrase, diff);\n\t\t\n\t\tif(!phrase.equals(same)) {\n\t\t\tfail(\"not equal\");\n\t\t}\n\t\tif(phrase.equals(diff)) {\n\t\t\tfail(\"equal\");\n\t\t}\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/disjunctivizer/AlignedEdgeFilterTest.java", "content": "package opennlp.ccg.disjunctivizer;\n\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertFalse;\nimport static org.junit.Assert.assertTrue;\nimport static org.junit.Assert.fail;\n\nimport java.util.Collections;\nimport java.util.HashSet;\nimport java.util.Set;\n\nimport opennlp.ccg.disjunctivizer.AlignedEdgeFilter;\nimport opennlp.ccg.disjunctivizer.MatchType;\nimport opennlp.ccg.hylo.ModeLabel;\nimport opennlp.ccg.hylo.NominalAtom;\nimport opennlp.ccg.hylo.Proposition;\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.hylo.graph.LFEdgeLabel;\nimport opennlp.ccg.hylo.graph.LFBaseTest;\nimport opennlp.ccg.hylo.graph.LFVertex;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class AlignedEdgeFilterTest extends LFBaseTest {\n\n\tAlignedEdgeFilter filter;\n\tSet indices;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tsuper.setUp();\n\t\t\n\t\tindices = new HashSet();\n\t\tindices.add(0);\n\t\tindices.add(2);\n\t\tindices.add(3);\n\t\t\n\t\tfilter = new AlignedEdgeFilter(indices, MatchType.SOURCE_ALIGNED, MatchType.TARGET_UNALIGNED);\n\t}\n\t\n\t@Test\n\tpublic void testAlignedEdgeFilter() {\n\t\ttry {\n\t\t\tnew AlignedEdgeFilter(null, MatchType.SOURCE_ALIGNED);\n\t\t\tfail(\"able to specify null alignment indices\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testGetAlignmentIndices() {\n\t\tassertEquals(indices, filter.getAlignmentIndices());\n\t}\n\n\t@SuppressWarnings(\"unchecked\")\n\t@Test\n\tpublic void testSetAlignmentIndices() {\n\t\tfilter.setAlignmentIndices(Collections.EMPTY_SET);\n\t\tassertTrue(filter.getAlignmentIndices().isEmpty());\n\t\t\n\t\ttry {\n\t\t\tfilter.setAlignmentIndices(null);\n\t\t\tfail(\"able to specify null alignment indices\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\n\t@SuppressWarnings(\"unchecked\")\n\t@Test\n\tpublic void testAllows() {\n\t\tLFEdge one = new LFEdge(new LFVertex(new NominalAtom(\"w0\"), new Proposition(\"blah\")),\n\t\t\t\tnew LFVertex(new NominalAtom(\"w1\"), new Proposition(\"blah blah\")),\n\t\t\t\tnew LFEdgeLabel(new ModeLabel(\"blase\"))),\n\t\t\ttwo = new LFEdge(new LFVertex(new NominalAtom(\"w2\"), new Proposition(\"blah\")),\n\t\t\t\tnew LFVertex(new NominalAtom(\"w3\"), new Proposition(\"blah blah\")),\n\t\t\t\tnew LFEdgeLabel(new ModeLabel(\"blase\")));\n\t\t\n\t\tassertTrue(filter.allows(one));\n\t\tassertFalse(filter.allows(two));\n\t\t\n\t\tfilter.setAlignmentIndices(Collections.EMPTY_SET);\n\t\tassertFalse(filter.allows(one));\n\t\tassertFalse(filter.allows(two));\n\t\t\n\t\t// make self-contradictory filter\n\t\tfilter = new AlignedEdgeFilter(indices, MatchType.SOURCE_ALIGNED, MatchType.SOURCE_UNALIGNED);\n\t\tassertFalse(filter.allows(one));\n\t\tassertFalse(filter.allows(two));\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/disjunctivizer/DisjunctivizerTest.java", "content": "package opennlp.ccg.disjunctivizer;\n\nimport static org.junit.Assert.*;\n\nimport java.io.BufferedWriter;\nimport java.io.File;\nimport java.io.FileWriter;\nimport java.util.LinkedHashSet;\nimport java.util.Properties;\nimport java.util.Set;\n\nimport javax.xml.parsers.DocumentBuilder;\nimport javax.xml.parsers.DocumentBuilderFactory;\nimport javax.xml.parsers.ParserConfigurationException;\n\nimport opennlp.ccg.alignment.Alignment;\nimport opennlp.ccg.alignment.Alignments;\nimport opennlp.ccg.alignment.Mapping;\nimport opennlp.ccg.alignment.Phrase;\nimport opennlp.ccg.alignment.PhrasePosition;\nimport opennlp.ccg.alignment.Status;\nimport opennlp.ccg.hylo.graph.LFGraphFactory;\nimport opennlp.ccg.hylo.graph.LFBaseTest;\n\nimport org.apache.xml.serializer.OutputPropertiesFactory;\nimport org.apache.xml.serializer.Serializer;\nimport org.apache.xml.serializer.SerializerFactory;\nimport org.jdom.input.DOMBuilder;\nimport org.junit.Before;\nimport org.junit.Test;\nimport org.w3c.dom.Document;\nimport org.w3c.dom.Element;\nimport org.w3c.dom.NodeList;\n\npublic class DisjunctivizerTest extends LFBaseTest {\n\n\tstatic Properties OUTPUT_PROPERTIES = OutputPropertiesFactory.getDefaultMethodProperties(\"xml\");\n\t\n\tstatic {\n\t\tOUTPUT_PROPERTIES.setProperty(\"indent\", \"yes\");\n\t\tOUTPUT_PROPERTIES.setProperty(\"media-type\", \"text/xml\");\n\t\tOUTPUT_PROPERTIES.setProperty(OutputPropertiesFactory.S_KEY_INDENT_AMOUNT, \"2\");\n\t\tOUTPUT_PROPERTIES.setProperty(\"{http\\u003a//xml.apache.org/xalan}indent-amount\", \"2\");\n\t}\n\t\n\tDocumentBuilder documentBuilder;\n\tDOMBuilder domBuilder;\n\tFile alignmentsFile, paraphrasesFile, outputFile;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tsuper.setUp();\n\t\t\n\t\ttry {\n\t\t\tdocumentBuilder = DocumentBuilderFactory.newInstance().newDocumentBuilder();\n\t\t}\n\t\tcatch(ParserConfigurationException e) {\n\t\t\tthrow new Exception(\"problem with parser configuration: \" + e.getLocalizedMessage(), e);\n\t\t}\n\t\t\n\t\tdomBuilder = new DOMBuilder();\n\t\t\n\t\tFile testDir = new File(System.getProperty(\"user.dir\"), \"test\");\n\t\t\n\t\tparaphrasesFile = new File(testDir, \"paraphrases.xml\");\n\t\toutputFile = new File(testDir, \"output.xml\");\n\t}\n\t\n\t@Test\n\tpublic void testDisjunctivizer() {\n\t\ttry {\n\t\t\tnew Disjunctivizer(null);\n\t\t\tfail(\"able to create disjunctivizer with null document\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testBuildDisjunctiveLF() throws Exception {\n\t\tDocument paraphrases = documentBuilder.parse(paraphrasesFile);\n\t\t\n\t\tSerializer s = SerializerFactory.getSerializer(OUTPUT_PROPERTIES);\n\t\ts.setOutputFormat(OUTPUT_PROPERTIES);\n\t\ts.setWriter(new BufferedWriter(new FileWriter(outputFile)));\n\t\t\n\t\tDisjunctivizer disj = null;\n\t\tNodeList paras = paraphrases.getElementsByTagName(\"paraphrase\");\n\t\tDocument out = documentBuilder.newDocument();\n\t\tElement dlfsElement = out.createElement(\"dlfs\");\n\t\tout.appendChild(dlfsElement);\n\t\t\n\t\tfor(int i = 0; i < paras.getLength(); i++) {\n\t\t\tElement para = (Element)paras.item(i);\n\t\t\tInteger id = Integer.parseInt(para.getAttribute(\"id\"));\n\t\t\t\n\t\t\tElement first = (Element)para.getElementsByTagName(\"first\").item(0), \n\t\t\t\t\tsecond = (Element)para.getElementsByTagName(\"second\").item(0);\n\t\t\t\n\t\t\tSet ms = new LinkedHashSet();\n\t\t\tNodeList als = para.getElementsByTagName(\"alignments\");\n\t\t\tfor(int j = 0; j < als.getLength(); j++) {\n\t\t\t\tElement al = (Element)als.item(j);\n\t\t\t\tif(al.getAttribute(\"source\").equals(\"ANNOTATOR\")) {\n\t\t\t\t\tNodeList as = al.getElementsByTagName(\"alignment\");\n\t\t\t\t\tfor(int k = 0; k < as.getLength(); k++) {\n\t\t\t\t\t\tElement a = (Element)as.item(k);\n\t\t\t\t\t\tms.add(new Mapping(id, Integer.parseInt(a.getAttribute(\"first\")),\n\t\t\t\t\t\t\t\tInteger.parseInt(a.getAttribute(\"second\")),\n\t\t\t\t\t\t\t\t\t\tStatus.forAbbreviation(a.getAttribute(\"status\"))));\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\t\n\t\t\tAlignment a = new Alignment(new Phrase(id,\n\t\t\t\t\t\tAlignments.tokenize(first.getElementsByTagName(\"string\").item(0).getTextContent())),\n\t\t\t\t\t\t\tnew Phrase(id,\n\t\t\t\t\t\t\t\tAlignments.tokenize(second.getElementsByTagName(\"string\").item(0).getTextContent())),\n\t\t\t\t\t\tms);\n\t\t\t\n\t\t\tElement firstLF = (Element)first.getElementsByTagName(\"lf\").item(0),\n\t\t\t\t\tsecondLF = (Element)second.getElementsByTagName(\"lf\").item(0);\n\t\t\t\n\t\t\tLFGraphDifference diff = (firstLF != null && secondLF != null)\n\t\t\t\t\t? new LFGraphDifference(LFGraphFactory.newGraphFrom(firstLF),\n\t\t\t\t\t\t\tLFGraphFactory.newGraphFrom(secondLF), a)\n\t\t\t\t\t: null;\n\t\t\t\n\t\t\tfor(PhrasePosition pos : PhrasePosition.values()) {\n\t\t\t\tElement str = out.createElement(\"string\");\n\t\t\t\tstr.setAttribute(\"number\", Integer.toString(id));\n\t\t\t\tstr.setAttribute(\"position\", pos.name());\n\t\t\t\tstr.setTextContent(Alignments.untokenize(a.get(pos)));\n\t\t\t\t\n\t\t\t\tdlfsElement.appendChild(str);\n\t\t\t}\n\t\t\t\n\t\t\tElement msEl = out.createElement(\"mappings\");\n\t\t\tmsEl.appendChild(out.createCDATASection(ms.toString()));\n\t\t\tdlfsElement.appendChild(msEl);\n\t\t\t\n\t\t\tif(diff == null) {\n\t\t\t\tdlfsElement.appendChild(out.createComment(\"missing LF!\"));\n\t\t\t}\n\t\t\telse {\n\t\t\t\tif(disj == null) {\n\t\t\t\t\tdisj = new Disjunctivizer(out);\n\t\t\t\t}\n\t\t\t\t\n\t\t\t\tElement dlf = disj.buildDisjunctiveLFFor(diff);\n\t\t\t\tdlfsElement.appendChild(dlf);\n\t\t\t\t\n\t\t\t\tassertEquals(dlf, disj.buildDisjunctiveLFFor(diff));\n\t\t\t\t\n\t\t\t\tdlfsElement.appendChild(disj.buildDisjunctiveLFFor(diff.reverse()));\n\t\t\t\t\n\t\t\t\tassertNotSame(dlf, disj.buildDisjunctiveLFFor(diff.reverse()));\n\t\t\t}\n\t\t}\n\t\t\n\t\ts.asDOMSerializer().serialize(out);\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/disjunctivizer/EdgeMatchFilterTest.java", "content": "package opennlp.ccg.disjunctivizer;\n\nimport static org.junit.Assert.assertFalse;\nimport static org.junit.Assert.assertTrue;\nimport static org.junit.Assert.fail;\nimport opennlp.ccg.disjunctivizer.EdgeMatchFilter;\nimport opennlp.ccg.disjunctivizer.MatchType;\nimport opennlp.ccg.hylo.ModeLabel;\nimport opennlp.ccg.hylo.NominalAtom;\nimport opennlp.ccg.hylo.Proposition;\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.hylo.graph.LFEdgeLabel;\nimport opennlp.ccg.hylo.graph.LFBaseTest;\nimport opennlp.ccg.hylo.graph.LFVertex;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class EdgeMatchFilterTest extends LFBaseTest {\n\n\tEdgeMatchFilter filter;\n\tLFEdge edge;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tsuper.setUp();\n\t\t\n\t\tedge = new LFEdge(new LFVertex(new NominalAtom(\"w0\"), new Proposition(\"blah\")),\n\t\t\t\tnew LFVertex(new NominalAtom(\"w1\"), new Proposition(\"blah blah\")),\n\t\t\t\tnew LFEdgeLabel(new ModeLabel(\"blase\")));\n\t\t\n\t\tfilter = new EdgeMatchFilter(edge, MatchType.LABEL_MISMATCH, MatchType.SOURCE_PREDICATE_MISMATCH,\n\t\t\t\tMatchType.TARGET_PREDICATE_MATCH);\n\t}\n\n\t@Test\n\tpublic void testEdgeMatchFilter() {\n\t\ttry {\n\t\t\tnew EdgeMatchFilter(null, MatchType.LABEL_MATCH);\n\t\t\tfail(\"able to specify null edge\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testAllows() {\n\t\tLFEdge test = new LFEdge(new LFVertex(new NominalAtom(\"w0\"), new Proposition(\"blah blah\")),\n\t\t\t\tnew LFVertex(new NominalAtom(\"w1\"), new Proposition(\"blah blah\")),\n\t\t\t\tnew LFEdgeLabel(new ModeLabel(\"boring\")));\n\t\t\n\t\tassertTrue(filter.allows(test));\n\t\t\n\t\ttest = new LFEdge(test.getSource(), test.getTarget(), edge.getLabel());\n\t\tassertFalse(filter.allows(test));\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/disjunctivizer/FilteredLFEdgeSetTest.java", "content": "package opennlp.ccg.disjunctivizer;\n\nimport static org.junit.Assert.*;\n\nimport java.util.HashSet;\nimport java.util.LinkedHashSet;\nimport java.util.Set;\n\nimport opennlp.ccg.disjunctivizer.AlignedEdgeFilter;\nimport opennlp.ccg.disjunctivizer.EdgeMatchFilter;\nimport opennlp.ccg.disjunctivizer.FilteredLFEdgeSet;\nimport opennlp.ccg.disjunctivizer.MatchType;\nimport opennlp.ccg.hylo.ModeLabel;\nimport opennlp.ccg.hylo.NominalAtom;\nimport opennlp.ccg.hylo.Proposition;\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.hylo.graph.LFEdgeLabel;\nimport opennlp.ccg.hylo.graph.LFBaseTest;\nimport opennlp.ccg.hylo.graph.LFVertex;\nimport opennlp.ccg.util.CompositeFilter;\nimport opennlp.ccg.util.Filter;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class FilteredLFEdgeSetTest extends LFBaseTest {\n\n\tFilteredLFEdgeSet set;\n\tSet edges;\n\tFilter edgeFilter;\n\t\n\tLFEdge one, two, three;\n\t\n\t@SuppressWarnings(\"unchecked\")\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tsuper.setUp();\n\t\t\n\t\tone = new LFEdge(new LFVertex(new NominalAtom(\"w0\"), new Proposition(\"blah\")),\n\t\t\t\tnew LFVertex(new NominalAtom(\"w1\"), new Proposition(\"blah blah\")),\n\t\t\t\tnew LFEdgeLabel(new ModeLabel(\"blase\")));\n\t\ttwo = new LFEdge(new LFVertex(new NominalAtom(\"w2\"), new Proposition(\"blah\")),\n\t\t\t\tnew LFVertex(new NominalAtom(\"w3\"), new Proposition(\"blah blah\")),\n\t\t\t\tnew LFEdgeLabel(new ModeLabel(\"bored\")));\n\t\tthree = new LFEdge(new LFVertex(new NominalAtom(\"w0\"), new Proposition(\"zzz\")),\n\t\t\t\tnew LFVertex(new NominalAtom(\"w2\"), new Proposition(\"snooze\")),\n\t\t\t\tnew LFEdgeLabel(new ModeLabel(\"blase\")));\n\t\t\n\t\tedges = new LinkedHashSet();\n\t\tedges.add(one);\n\t\tedges.add(two);\n\t\tedges.add(three);\n\t\t\n\t\tSet indices = new HashSet();\n\t\tindices.add(0);\n\t\tindices.add(2);\n\t\tindices.add(3);\n\t\t\n\t\tedgeFilter = new CompositeFilter(new EdgeMatchFilter(one, MatchType.SOURCE_PREDICATE_MATCH,\n\t\t\t\tMatchType.LABEL_MISMATCH), new AlignedEdgeFilter(indices,\n\t\t\t\t\t\tMatchType.TARGET_ALIGNED, MatchType.SOURCE_ALIGNED));\n\t\t\n\t\tset = new FilteredLFEdgeSet(edges, edgeFilter);\n\t}\n\n\t@Test\n\tpublic void testSourceView() {\n\t\tassertTrue(set.sourceView().contains(two.getSource()));\n\t\tassertEquals(1, set.sourceView().size());\n\t}\n\n\t@Test\n\tpublic void testTargetView() {\n\t\tassertTrue(set.targetView().contains(two.getTarget()));\n\t\tassertEquals(1, set.targetView().size());\n\t}\n\n\t@Test\n\tpublic void testLabelView() {\n\t\tassertTrue(set.labelView().contains(two.getLabel()));\n\t\tassertEquals(1, set.labelView().size());\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/disjunctivizer/LFGraphDifferenceTest.java", "content": "package opennlp.ccg.disjunctivizer;\n\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertFalse;\nimport static org.junit.Assert.assertTrue;\nimport static org.junit.Assert.fail;\n\nimport java.util.Collections;\nimport java.util.Iterator;\nimport java.util.Map;\nimport java.util.Set;\n\nimport opennlp.ccg.alignment.Alignment;\nimport opennlp.ccg.alignment.Mapping;\nimport opennlp.ccg.alignment.Phrase;\nimport opennlp.ccg.alignment.PhrasePosition;\nimport opennlp.ccg.hylo.ModeLabel;\nimport opennlp.ccg.hylo.NominalAtom;\nimport opennlp.ccg.hylo.Proposition;\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.hylo.graph.LFEdgeLabel;\nimport opennlp.ccg.hylo.graph.LFGraph;\nimport opennlp.ccg.hylo.graph.LFBaseTest;\nimport opennlp.ccg.hylo.graph.LFVertex;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class LFGraphDifferenceTest extends LFBaseTest {\n\n\tLFGraph aGraph, bGraph;\n\tAlignment alignment;\n\tLFGraphDifference diff;\n\t\n\tLFEdge aDet, aArg0, aMod, bArg0;\n\t\n\t@SuppressWarnings(\"unchecked\")\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tsuper.setUp();\n\t\t\n\t\talignment = new Alignment(new Phrase(337, \"A\", \"boy\", \"walks\", \"quickly\"),\n\t\t\t\tnew Phrase(337, \"He\", \"moves\"),\n\t\t\t\tCollections.EMPTY_SET);\n\t\t\n\t\talignment.add(new Mapping(0, 0));\n\t\talignment.add(new Mapping(1, 0));\n\t\talignment.add(new Mapping(2, 1));\n\t\t\n\t\taGraph = new LFGraph();\n\t\t\n\t\tLFVertex aw0 = new LFVertex(new NominalAtom(\"w0\"), new Proposition(\"a\")),\n\t\t\taw1 = new LFVertex(new NominalAtom(\"w1\"), new Proposition(\"boy\")),\n\t\t\taw2 = new LFVertex(new NominalAtom(\"w2\"), new Proposition(\"walk\")),\n\t\t\taw3 = new LFVertex(new NominalAtom(\"w3\"), new Proposition(\"quickly\"));\n\t\t\n\t\taGraph.addVertex(aw0);\n\t\taGraph.addVertex(aw1);\n\t\taGraph.addVertex(aw2);\n\t\taGraph.addVertex(aw3);\n\t\t\t\t\n\t\taDet = aGraph.addLabeledEdge(aw1, aw0, LFEdgeLabel.forMode(new ModeLabel(\"Det\")));\n\t\taArg0 = aGraph.addLabeledEdge(aw2, aw1, LFEdgeLabel.forMode(new ModeLabel(\"Arg0\")));\n\t\taMod = aGraph.addLabeledEdge(aw2, aw3, LFEdgeLabel.forMode(new ModeLabel(\"Mod\")));\n\t\t\n\t\tbGraph = new LFGraph();\n\t\t\n\t\tLFVertex bw0 = new LFVertex(new NominalAtom(\"w0\"), new Proposition(\"he\")),\n\t\t\tbw1 = new LFVertex(new NominalAtom(\"w1\"), new Proposition(\"move\"));\n\t\t\n\t\tbGraph.addVertex(bw0);\n\t\tbGraph.addVertex(bw1);\n\t\t\t\t\n\t\tbArg0 = bGraph.addLabeledEdge(bw1, bw0, LFEdgeLabel.forMode(new ModeLabel(\"Arg0\")));\n\t\t\n\t\tdiff = new LFGraphDifference(aGraph, bGraph, alignment);\n\t}\n\t\n\t@Test\n\tpublic void testLFGraphDifference() {\n\t\ttry {\n\t\t\tnew LFGraphDifference(null, bGraph, alignment);\n\t\t\tfail(\"able to create LF graph difference with null graph\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew LFGraphDifference(aGraph, null, alignment);\n\t\t\tfail(\"able to create LF graph difference with null graph\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew LFGraphDifference(aGraph, bGraph, null);\n\t\t\tfail(\"able to create LF graph difference with null graph\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testReverse() {\n\t\tLFGraphDifference ffid = diff.reverse();\n\t\t\n\t\tassertEquals(diff.a.vertexSet(), ffid.b.vertexSet());\n\t\tassertEquals(diff.a.edgeSet(), ffid.b.edgeSet());\n\t\t\n\t\tfor(PhrasePosition pos : PhrasePosition.values()) {\n\t\t\tassertEquals(diff.alignment.get(pos), ffid.alignment.get(pos.opposite()));\n\t\t}\n\t\t\n\t\tMap> m = ffid.alignment.asMap();\n\t\t\n\t\tassertTrue(m.get(0).contains(0));\n\t\tassertTrue(m.get(0).contains(1));\n\t\tassertTrue(m.get(1).contains(2));\n\t}\n\n\t@Test\n\tpublic void testDeletes() {\n\t\tSet dels = diff.deletes();\n\t\t\n\t\ttry {\n\t\t\tdels.add(bArg0);\n\t\t\tfail(\"able to add edge\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tdels.remove(bArg0);\n\t\t\tfail(\"able to remove edge\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tIterator i = dels.iterator();\n\t\t\ti.next();\n\t\t\ti.remove();\n\t\t\tfail(\"able to remove edge\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t\t\n\t\tassertEquals(Collections.singleton(aMod), diff.deletes());\n\t\tassertEquals(Collections.emptySet(), diff.reverse().deletes());\n\t}\n\n\t@Test\n\tpublic void testInserts() {\n\t\tSet ins = diff.inserts();\n\t\t\n\t\ttry {\n\t\t\tins.add(bArg0);\n\t\t\tfail(\"able to add edge\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tins.remove(bArg0);\n\t\t\tfail(\"able to remove edge\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t\t\n\t\tassertEquals(Collections.emptySet(), diff.inserts());\n\t\tassertEquals(Collections.singleton(aMod), diff.reverse().inserts());\n\t}\n\n\t@Test\n\tpublic void testSubstitutions() {\n\t\tSet subs = diff.substitutions();\n\t\t\n\t\ttry {\n\t\t\tsubs.add(bArg0);\n\t\t\tfail(\"able to add edge\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tsubs.remove(bArg0);\n\t\t\tfail(\"able to remove edge\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tIterator i = subs.iterator();\n\t\t\ti.next();\n\t\t\ti.remove();\n\t\t\tfail(\"able to remove edge\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t\t\n\t\tassertTrue(subs.contains(bArg0));\n\t\tassertFalse(subs.contains(aArg0));\n\t\tassertFalse(subs.contains(aDet));\n\t\tassertFalse(subs.contains(aMod));\n\t\t\n\t\tassertEquals(Collections.singleton(bArg0), diff.substitutionsFor(aArg0));\n\t\tassertEquals(Collections.singleton(aArg0), diff.reverse().substitutionsFor(bArg0));\n\t}\n\n\t@Test\n\tpublic void testSubstitutionsBySource() {\n\t\tMap> map = diff.substitutionsBySource();\n\t\tassertTrue(map.keySet().contains(bArg0.getSource()));\n\t\tassertTrue(map.get(bArg0.getSource()).contains(bArg0));\n\t\tassertEquals(1, map.size());\n\t\t\n\t\ttry {\n\t\t\tmap.remove(bArg0.getSource());\n\t\t\tfail(\"able to remove edge\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tmap.put(aArg0.getSource(), Collections.singleton(aArg0));\n\t\t\tfail(\"able to put edge\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tIterator>> i = map.entrySet().iterator();\n\t\t\ti.next();\n\t\t\ti.remove();\n\t\t\tfail(\"able to remove entry\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testSubstitutionsBySourceFor() {\n\t\tMap> map = diff.substitutionsBySourceFor(aArg0);\n\t\tassertTrue(map.keySet().contains(bArg0.getSource()));\n\t\tassertTrue(map.get(bArg0.getSource()).contains(bArg0));\n\t\tassertEquals(1, map.size());\n\t\t\n\t\ttry {\n\t\t\tmap.remove(bArg0.getSource());\n\t\t\tfail(\"able to remove edge\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tmap.put(aArg0.getSource(), Collections.singleton(aArg0));\n\t\t\tfail(\"able to put edge\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tIterator>> i = map.entrySet().iterator();\n\t\t\ti.next();\n\t\t\ti.remove();\n\t\t\tfail(\"able to remove entry\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// noop\n\t\t}\n\t\t\n\t\tmap = diff.substitutionsBySourceFor(aDet);\n\t\tassertTrue(map.isEmpty());\n\t}\n}\n" }, { "path": "test/opennlp/ccg/disjunctivizer/LabelMatchFilterTest.java", "content": "package opennlp.ccg.disjunctivizer;\n\nimport static org.junit.Assert.*;\n\nimport opennlp.ccg.disjunctivizer.LabelMatchFilter;\nimport opennlp.ccg.hylo.ModeLabel;\nimport opennlp.ccg.hylo.NominalAtom;\nimport opennlp.ccg.hylo.Proposition;\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.hylo.graph.LFEdgeLabel;\nimport opennlp.ccg.hylo.graph.LFBaseTest;\nimport opennlp.ccg.hylo.graph.LFVertex;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class LabelMatchFilterTest extends LFBaseTest {\n\n\tLabelMatchFilter filter;\n\tLFEdgeLabel label;\n\tLFEdge one, two;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tsuper.setUp();\n\t\t\n\t\tone = new LFEdge(new LFVertex(new NominalAtom(\"w0\"), new Proposition(\"blah\")),\n\t\t\t\tnew LFVertex(new NominalAtom(\"w1\"), new Proposition(\"blah blah\")),\n\t\t\t\tnew LFEdgeLabel(new ModeLabel(\"blase\")));\n\t\ttwo = new LFEdge(new LFVertex(new NominalAtom(\"w2\"), new Proposition(\"blah\")),\n\t\t\t\tnew LFVertex(new NominalAtom(\"w3\"), new Proposition(\"blah blah\")),\n\t\t\t\tnew LFEdgeLabel(new ModeLabel(\"blurg\")));\n\t\t\n\t\tlabel = new LFEdgeLabel(new ModeLabel(\"blase\"));\n\t\t\n\t\tfilter = new LabelMatchFilter(label);\n\t}\n\t\n\t@Test\n\tpublic void testLabelMatchFilter() {\n\t\ttry {\n\t\t\tnew LabelMatchFilter(null);\n\t\t\tfail(\"able to specify null label\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testSetBasis() {\n\t\tfilter.setBasis(new LFEdgeLabel(new ModeLabel(\"boo\")));\n\t\t\n\t\tassertFalse(filter.allows(one));\n\t\tassertFalse(filter.allows(two));\n\t\t\n\t\ttry {\n\t\t\tfilter.setBasis(null);\n\t\t\tfail(\"able to specify null label\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testAllows() {\n\t\tassertTrue(filter.allows(one));\n\t\tassertFalse(filter.allows(two));\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/disjunctivizer/VertexMatchFilterTest.java", "content": "package opennlp.ccg.disjunctivizer;\n\nimport static org.junit.Assert.*;\n\nimport opennlp.ccg.disjunctivizer.MatchType;\nimport opennlp.ccg.disjunctivizer.VertexMatchFilter;\nimport opennlp.ccg.hylo.ModeLabel;\nimport opennlp.ccg.hylo.NominalAtom;\nimport opennlp.ccg.hylo.Proposition;\nimport opennlp.ccg.hylo.graph.LFEdge;\nimport opennlp.ccg.hylo.graph.LFEdgeLabel;\nimport opennlp.ccg.hylo.graph.LFBaseTest;\nimport opennlp.ccg.hylo.graph.LFVertex;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class VertexMatchFilterTest extends LFBaseTest {\n\t\n\tVertexMatchFilter filter;\n\tLFVertex one, two;\n\tLFEdge edge;\n\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tsuper.setUp();\n\t\t\n\t\tone = new LFVertex(new NominalAtom(\"w0\"), new Proposition(\"blah\"));\n\t\ttwo = new LFVertex(new NominalAtom(\"w1\"), new Proposition(\"blah\"));\n\t\t\n\t\tedge = new LFEdge(two,\n\t\t\t\tnew LFVertex(new NominalAtom(\"w2\"), new Proposition(\"blah blah\")),\n\t\t\t\tnew LFEdgeLabel(new ModeLabel(\"blase\")));\n\t\t\n\t\tfilter = new VertexMatchFilter(two, MatchType.SOURCE_MATCH);\n\t}\n\n\t@Test\n\tpublic void testVertexMatchFilter() {\n\t\ttry {\n\t\t\tnew VertexMatchFilter(null, MatchType.SOURCE_PREDICATE_MATCH);\n\t\t\tfail(\"able to specify null basis\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testSetBasis() {\n\t\tfilter.setBasis(one);\n\t\tassertFalse(filter.allows(edge));\n\t\t\n\t\tedge = new LFEdge(one, edge.getTarget(), edge.getLabel());\n\t\tassertTrue(filter.allows(edge));\n\t\t\n\t\ttry {\n\t\t\tfilter.setBasis(null);\n\t\t\tfail(\"able to specify null basis\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testAllows() {\n\t\tassertTrue(filter.allows(edge));\n\t\t\n\t\tedge = new LFEdge(one, edge.getTarget(), edge.getLabel());\n\t\tassertFalse(filter.allows(edge));\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/hylo/graph/LFBaseTest.java", "content": "package opennlp.ccg.hylo.graph;\n\nimport java.io.File;\n\nimport opennlp.ccg.grammar.Grammar;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class LFBaseTest {\n\n\tstatic Grammar grammar = null;\n\t\n\t@Before\n\t@SuppressWarnings(\"deprecation\")\n\tpublic void setUp() throws Exception {\n\t\tif(grammar == null) {\n\t\t\tgrammar = new Grammar(new File(new File(\n\t\t\t\t\tnew File(System.getProperty(\"user.dir\")), \"test\"),\n\t\t\t\t\t\"grammar.xml\").toURL());\n\t\t}\n\t}\n\n\t@Test\n\tpublic void dummy() {}\n\n}\n" }, { "path": "test/opennlp/ccg/hylo/graph/LFEdgeFactoryTest.java", "content": "package opennlp.ccg.hylo.graph;\n\nimport static org.junit.Assert.*;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class LFEdgeFactoryTest extends LFEdgeTest {\n\n\tLFEdgeFactory factory;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tsuper.setUp();\n\t\t\n\t\tfactory = new DefaultLFEdgeFactory();\n\t}\n\n\t@Test\n\tpublic void testCreateEdge() {\n\t\tassertNotSame(edge, factory.createEdge(edge.source, edge.target));\n\t\t\n\t\tedge = new LFEdge(edge.source, edge.target, null);\n\t\tassertEquals(edge, factory.createLabeledEdge(edge.source, edge.target, null));\n\t}\n\n\t@Test\n\tpublic void testCreateLabeledEdge() {\n\t\tassertEquals(edge, factory.createLabeledEdge(edge.source, edge.target, edge.label));\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/hylo/graph/LFEdgeTest.java", "content": "package opennlp.ccg.hylo.graph;\n\nimport static org.junit.Assert.*;\n\nimport opennlp.ccg.hylo.ModeLabel;\nimport opennlp.ccg.hylo.NominalAtom;\nimport opennlp.ccg.hylo.Proposition;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class LFEdgeTest extends LFBaseTest {\n\n\tLFVertex source, target;\n\tLFEdgeLabel label;\n\tLFEdge edge;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tsuper.setUp();\n\t\t\n\t\tsource = new LFVertex(new NominalAtom(\"w3\"), new Proposition(\"prop1\"));\n\t\ttarget = new LFVertex(new NominalAtom(\"w9\"), new Proposition(\"prop2\"));\n\t\tlabel = new LFEdgeLabel(new ModeLabel(\"Arg0\"));\n\t\t\n\t\tedge = new LFEdge(source, target, label);\n\t}\n\n\t@Test\n\tpublic void testLFEdge() {\n\t\ttry {\n\t\t\tnew LFEdge(null, target);\n\t\t\tfail(\"able to specify null target\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew LFEdge(source, null);\n\t\t\tfail(\"able to specify null target\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew LFEdge(source, target, null);\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\tfail(\"unable to specify null label\");\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testEqualsObject() {\n\t\tassertEquals(edge, new LFEdge(edge.source, edge.target, edge.label));\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/hylo/graph/LFGraphTest.java", "content": "package opennlp.ccg.hylo.graph;\n\nimport static org.junit.Assert.*;\n\nimport java.io.File;\nimport java.util.HashSet;\n\nimport javax.xml.parsers.DocumentBuilder;\nimport javax.xml.parsers.DocumentBuilderFactory;\nimport javax.xml.parsers.ParserConfigurationException;\n\nimport opennlp.ccg.hylo.ModeLabel;\nimport opennlp.ccg.hylo.NominalAtom;\nimport opennlp.ccg.hylo.Proposition;\nimport opennlp.ccg.realize.Realizer;\nimport opennlp.ccg.synsem.LF;\n\nimport org.jdom.input.DOMBuilder;\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class LFGraphTest extends LFBaseTest {\n\n\tLF testLF;\n\tLFGraph graph, expected;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tsuper.setUp();\n\t\t\n\t\tDocumentBuilder db;\n\t\ttry {\n\t\t\tdb = DocumentBuilderFactory.newInstance().newDocumentBuilder();\n\t\t}\n\t\tcatch(ParserConfigurationException e) {\n\t\t\tthrow new Exception(\"problem with parser configuration: \" + e.getLocalizedMessage(), e);\n\t\t}\n\t\t\n\t\tFile testFile = new File(new File(new File(System.getProperty(\"user.dir\")), \"test\"), \"testlf.xml\");\n\t\ttestLF = Realizer.getLfFromElt(new DOMBuilder().build(db.parse(testFile).getDocumentElement()));\n\t\t\n\t\tgraph = LFGraphFactory.newGraphFrom(testLF);\n\t\t\n\t\texpected = new LFGraph(LFGraphFactory.DEFAULT_EDGE_FACTORY);\n\t\tLFVertex w7 = new LFVertex(new NominalAtom(\"w7\"), new Proposition(\"be\"));\n\t\tw7.setAttribute(new ModeLabel(\"mood\"), new Proposition(\"dcl\"));\n\t\tw7.setAttribute(new ModeLabel(\"tense\"), new Proposition(\"past\"));\n\t\texpected.addVertex(w7);\n\t\t\n\t\tLFVertex w0 = new LFVertex(new NominalAtom(\"w0\"), new Proposition(\"bank\"));\n\t\tw0.setAttribute(new ModeLabel(\"det\"), new Proposition(\"nil\"));\n\t\texpected.addVertex(w0);\n\t\t\n\t\tLFVertex w1 = new LFVertex(new NominalAtom(\"w1\"), new Proposition(\"of\"));\n\t\texpected.addVertex(w1);\n\t\t\n\t\tLFVertex w2 = new LFVertex(new NominalAtom(\"w2\"), new Proposition(\"holland\"));\n\t\tw2.setAttribute(new ModeLabel(\"det\"), new Proposition(\"nil\"));\n\t\tw2.setAttribute(new ModeLabel(\"num\"), new Proposition(\"sg\"));\n\t\texpected.addVertex(w2);\n\t\t\n\t\tLFVertex w5 = new LFVertex(new NominalAtom(\"w5\"), new Proposition(\"office\"));\n\t\tw5.setAttribute(new ModeLabel(\"det\"), new Proposition(\"nil\"));\n\t\tw5.setAttribute(new ModeLabel(\"num\"), new Proposition(\"sg\"));\n\t\texpected.addVertex(w5);\n\t\t\n\t\tLFVertex w4 = new LFVertex(new NominalAtom(\"w4\"), new Proposition(\"wuhan\"));\n\t\tw4.setAttribute(new ModeLabel(\"num\"), new Proposition(\"sg\"));\n\t\texpected.addVertex(w4);\n\t\t\n\t\tLFVertex w9 = new LFVertex(new NominalAtom(\"w9\"), new Proposition(\"officially\"));\n\t\texpected.addVertex(w9);\n\t\t\n\t\tLFVertex w8 = new LFVertex(new NominalAtom(\"w8\"), new Proposition(\"also\"));\n\t\texpected.addVertex(w8);\n\t\t\n\t\tLFVertex w10 = new LFVertex(new NominalAtom(\"w10\"), new Proposition(\"establish\"));\n\t\tw10.setAttribute(new ModeLabel(\"tense\"), new Proposition(\"past\"));\n\t\texpected.addVertex(w10);\n\t\t\n\t\tLFVertex w11 = new LFVertex(new NominalAtom(\"w11\"), new Proposition(\"just\"));\n\t\texpected.addVertex(w11);\n\t\t\n\t\tLFVertex w12 = new LFVertex(new NominalAtom(\"w12\"), new Proposition(\"recently\"));\n\t\texpected.addVertex(w12);\n\t\t\n\t\texpected.addLabeledEdge(w7, w0, LFEdgeLabel.forMode(new ModeLabel(\"Arg0\")));\n\t\texpected.addLabeledEdge(w0, w1, LFEdgeLabel.forMode(new ModeLabel(\"Mod\")));\n\t\texpected.addLabeledEdge(w1, w2, LFEdgeLabel.forMode(new ModeLabel(\"Arg1\")));\n\t\texpected.addLabeledEdge(w2, w5, LFEdgeLabel.forMode(new ModeLabel(\"ApposRel\")));\n\t\texpected.addLabeledEdge(w5, w4, LFEdgeLabel.forMode(new ModeLabel(\"Mod\")));\n\t\t\n\t\texpected.addLabeledEdge(w7, w9, LFEdgeLabel.forMode(new ModeLabel(\"Arg1\")));\n\t\texpected.addLabeledEdge(w9, w0, LFEdgeLabel.forMode(new ModeLabel(\"Arg0\")));\n\t\t\n\t\texpected.addLabeledEdge(w7, w8, LFEdgeLabel.forMode(new ModeLabel(\"Mod\")));\n\t\texpected.addLabeledEdge(w7, w10, LFEdgeLabel.forMode(new ModeLabel(\"GenRel\")));\n\t\texpected.addLabeledEdge(w10, w0, LFEdgeLabel.forMode(new ModeLabel(\"Arg1\")));\n\t\texpected.addLabeledEdge(w10, w11, LFEdgeLabel.forMode(new ModeLabel(\"Mod\")));\n\t\texpected.addLabeledEdge(w10, w12, LFEdgeLabel.forMode(new ModeLabel(\"Mod\")));\t\t\n\t}\n\n\t@Test\n\tpublic void testLFGraph() {\n\t\tassertEquals(expected.vertexSet(), graph.vertexSet());\n\t\tassertEquals(expected.edgeSet(), graph.edgeSet());\n\t}\n\n\t@Test\n\tpublic void testRemoveVertex() {\n\t\tfor(LFVertex v : new HashSet(graph.vertexSet())) {\n\t\t\tgraph.removeVertex(v);\n\t\t\tassertNull(graph.findVertexByNominal(v.nominal));\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testFindVertexByNominal() {\n\t\tfor(LFVertex vertex : expected.vertexSet()) {\n\t\t\tassertEquals(vertex, graph.findVertexByNominal(vertex.nominal));\n\t\t}\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/hylo/graph/LFVertexTest.java", "content": "package opennlp.ccg.hylo.graph;\n\nimport static org.junit.Assert.*;\n\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.Iterator;\nimport java.util.Map;\n\nimport opennlp.ccg.hylo.Mode;\nimport opennlp.ccg.hylo.ModeLabel;\nimport opennlp.ccg.hylo.Nominal;\nimport opennlp.ccg.hylo.NominalAtom;\nimport opennlp.ccg.hylo.Proposition;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class LFVertexTest extends LFBaseTest {\n\n\tNominal wordNominal, nonwordNominal;\n\tProposition proposition;\n\t\n\tInteger wordIndex, nonwordIndex;\n\tLFVertex word, nonword;\n\t\n\tMap attrs;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tsuper.setUp();\n\t\t\n\t\twordNominal = new NominalAtom(\"w7\");\n\t\tnonwordNominal = new NominalAtom(\"x1\");\n\t\tproposition = new Proposition(\"prop\");\n\t\t\n\t\twordIndex = Integer.valueOf(7);\n\t\tnonwordIndex = Integer.valueOf(1);\n\t\t\n\t\tattrs = new HashMap();\n\t\tattrs.put(new ModeLabel(\"num\"), new Proposition(\"sg\"));\n\t\tattrs.put(new ModeLabel(\"det\"), new Proposition(\"nil\"));\n\t\tattrs.put(new ModeLabel(\"tense\"), new Proposition(\"past\"));\n\t\t\n\t\tword = new LFVertex(wordNominal, proposition, attrs);\n\t\tnonword = new LFVertex(nonwordNominal, proposition);\n\t}\n\n\t@Test\n\tpublic void testLFVertex() {\n\t\ttry {\n\t\t\tnew LFVertex((Nominal)null);\n\t\t\tfail(\"able to specify null nominal\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tnew LFVertex(wordNominal, null);\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\tfail(\"unable to specify null proposition\");\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testGetType() {\n\t\tassertEquals(LFVertexType.WORD, word.getType());\n\t\tassertEquals(LFVertexType.NONWORD, nonword.getType());\n\t\t\n\t\tassertNotSame(LFVertexType.WORD, nonword.getType());\n\t\tassertNotSame(LFVertexType.NONWORD, word.getType());\n\t}\n\n\t@Test\n\tpublic void testGetIndex() {\n\t\tassertEquals(wordIndex, word.getIndex());\n\t\tassertEquals(nonwordIndex, nonword.getIndex());\n\t}\n\n\t@Test\n\tpublic void testAttributeNames() {\n\t\tassertEquals(attrs.keySet(), word.attributeNames());\n\t\tassertEquals(Collections.emptySet(), nonword.attributeNames());\n\t\t\n\t\ttry {\n\t\t\tIterator i = word.attributeNames().iterator();\n\t\t\ti.next();\n\t\t\ti.remove();\n\t\t\tfail(\"able to remove attribute name\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testContainsAttribute() {\n\t\tMode num = new ModeLabel(\"num\");\n\t\tassertTrue(word.containsAttribute(num));\n\t\tword.removeAttribute(num);\n\t\tassertFalse(word.containsAttribute(num));\n\t}\n\n\t@Test\n\tpublic void testGetAttribute() {\n\t\tfor(Mode m : attrs.keySet()) {\n\t\t\tassertEquals(attrs.get(m), word.getAttributeValue(m));\n\t\t\tassertNull(nonword.getAttributeValue(m));\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testAddAttribute() {\n\t\tMode num = new ModeLabel(\"num\");\n\t\tProposition prop = new Proposition(\"pl\");\n\t\t\n\t\tassertFalse(word.addAttribute(num, new Proposition(\"sg\")));\n\t\tassertTrue(word.addAttribute(num, prop));\n\t\tassertFalse(word.addAttribute(num, prop));\n\t\tassertTrue(word.containsAttribute(num));\n\t\t\n\t\tassertTrue(nonword.addAttribute(num, new Proposition(\"sg\")));\n\t\tassertFalse(nonword.addAttribute(num, new Proposition(\"sg\")));\n\t\tassertTrue(nonword.containsAttribute(num));\n\t\tassertTrue(nonword.addAttribute(num, prop));\n\t\tassertFalse(nonword.addAttribute(num, prop));\n\t\tassertTrue(nonword.containsAttribute(num));\n\t}\n\t\n\t@Test\n\tpublic void testSetAttribute() {\n\t\tMode num = new ModeLabel(\"num\");\n\t\tProposition prop = new Proposition(\"pl\");\n\t\t\n\t\tassertEquals(attrs.get(num), word.setAttribute(num, prop));\n\t\tassertEquals(prop, word.getAttributeValue(num));\n\t\t\n\t\tassertNull(nonword.setAttribute(num, prop));\n\t\tassertEquals(prop, nonword.getAttributeValue(num));\n\t}\n\t\n\t@Test\n\tpublic void testRemoveAttribute() {\n\t\tMode num = new ModeLabel(\"num\");\n\t\t\n\t\tassertEquals(attrs.get(num), word.removeAttribute(num));\n\t\tassertNull(nonword.removeAttribute(num));\n\t}\n\n\t@Test\n\tpublic void testEqualsObject() {\n\t\tLFVertex v = new LFVertex(wordNominal, proposition);\n\t\tfor(Mode m : attrs.keySet()) {\n\t\t\tv.setAttribute(m, attrs.get(m));\n\t\t}\n\t\t\n\t\tassertEquals(v, word);\n\t\tassertNotSame(v, nonword);\n\t}\n\t\n\t@Test\n\tpublic void testGetAttributeMap() {\n\t\tMap m = word.getAttributeMap();\n\t\tassertEquals(attrs, m);\n\t\t\n\t\ttry {\n\t\t\tIterator> i = m.entrySet().iterator();\n\t\t\ti.next();\n\t\t\ti.remove();\n\t\t\tfail(\"able to remove from attribute map\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// do nothing\n\t\t}\n\t\t\n\t\ttry {\n\t\t\tm.put(new ModeLabel(\"foo\"), new Proposition(\"bar\"));\n\t\t\tfail(\"able to put into attribute map\");\n\t\t}\n\t\tcatch(UnsupportedOperationException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n}\n" }, { "path": "test/opennlp/ccg/util/CompositeFilterTest.java", "content": "package opennlp.ccg.util;\n\nimport static org.junit.Assert.*;\n\nimport java.util.HashSet;\nimport java.util.Set;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class CompositeFilterTest {\n\n\tVisitedFilter visited;\n\tInteger target = 37;\n\tFilter lessThanFilter, greaterThanFilter;\n\tCompositeFilter bothFilter, equalToFilter;\n\t\n\t@SuppressWarnings(\"unchecked\")\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tvisited = new VisitedFilter();\n\t\t\n\t\tlessThanFilter = new Filter() {\n\t\t\t@Override\n\t\t\tpublic boolean allows(Integer e) {\n\t\t\t\treturn e < target;\n\t\t\t}\n\t\t};\n\t\t\n\t\tgreaterThanFilter = new Filter() {\n\t\t\t@Override\n\t\t\tpublic boolean allows(Integer e) {\n\t\t\t\treturn e > target;\n\t\t\t}\n\t\t};\n\t\t\n\t\tSet> s = new HashSet>();\n\t\ts.add(lessThanFilter);\n\t\ts.add(greaterThanFilter);\n\t\t\n\t\tbothFilter = new CompositeFilter(s);\n\t\tequalToFilter = new CompositeFilter(new InverseFilter(bothFilter));\n\t}\n\n\t@Test\n\tpublic void testContainsFilter() {\n\t\tassertTrue(bothFilter.containsFilter(lessThanFilter));\n\t\tassertTrue(bothFilter.containsFilter(greaterThanFilter));\n\t\tassertFalse(bothFilter.containsFilter(bothFilter));\n\t}\n\n\t@Test\n\tpublic void testAddFilter() {\n\t\tFilter f = new VisitedFilter();\n\t\tequalToFilter.addFilter(f);\n\t\t\n\t\tassertTrue(equalToFilter.allows(37));\n\t\tassertFalse(equalToFilter.allows(37));\n\t\t\n\t\ttry {\n\t\t\tequalToFilter.addFilter(null);\n\t\t\tfail(\"able to add null filter\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testRemoveFilter() {\n\t\tFilter f = new VisitedFilter();\n\t\tequalToFilter.addFilter(f);\n\t\tassertTrue(equalToFilter.allows(37));\n\t\t\n\t\tequalToFilter.removeFilter(f);\n\t\t\n\t\tassertTrue(equalToFilter.allows(37));\n\t\tassertTrue(equalToFilter.allows(37));\n\t}\n\n\t@Test\n\tpublic void testAllows() {\n\t\tassertTrue(lessThanFilter.allows(17));\n\t\tassertFalse(lessThanFilter.allows(38));\n\t\t\n\t\tassertFalse(greaterThanFilter.allows(17));\n\t\tassertTrue(greaterThanFilter.allows(38));\n\t\t\n\t\tassertFalse(bothFilter.allows(37));\n\t\tassertTrue(equalToFilter.allows(37));\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/util/DelegatedFilterTest.java", "content": "package opennlp.ccg.util;\n\nimport static org.junit.Assert.*;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class DelegatedFilterTest {\n\n\tFilter lengthFilter;\n\tDelegatedFilter stringFilter;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tlengthFilter = new Filter() {\n\t\t\t@Override\n\t\t\tpublic boolean allows(Integer i) {\n\t\t\t\treturn i <= 5;\n\t\t\t}\n\t\t};\n\t\t\n\t\tstringFilter = new DelegatedFilter(lengthFilter) {\n\t\t\t@Override\n\t\t\tpublic Integer delegateValueFor(String e) {\n\t\t\t\treturn e.length();\n\t\t\t}\n\t\t};\n\t}\n\n\t@Test\n\tpublic void testAllows() {\n\t\tassertTrue(stringFilter.allows(\"Scott\"));\n\t\tassertTrue(stringFilter.allows(\"Mike\"));\n\t\tassertTrue(stringFilter.allows(\"Jason\"));\n\t\tassertFalse(stringFilter.allows(\"Dominic\"));\n\t\tassertFalse(stringFilter.allows(\"Dennis\"));\n\t}\n\n\t@Test\n\tpublic void testDelegateValueFor() {\n\t\tassertEquals(Integer.valueOf(5), stringFilter.delegateValueFor(\"Scott\"));\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/util/FilteredMapTest.java", "content": "package opennlp.ccg.util;\n\nimport static org.junit.Assert.assertEquals;\nimport static org.junit.Assert.assertFalse;\nimport static org.junit.Assert.assertNull;\nimport static org.junit.Assert.assertTrue;\nimport static org.junit.Assert.fail;\n\nimport java.util.HashMap;\nimport java.util.Map;\nimport java.util.Map.Entry;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class FilteredMapTest {\n\n\tMap map;\n\tFilteredMap filteredMap;\n\tFilter keyFilter;\n\t\n\tInteger target = 37;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tmap = new HashMap();\n\t\tmap.put(17, \"seventeen\");\n\t\tmap.put(31, \"thirty-one\");\n\t\tmap.put(37, \"thirty-seven\");\n\t\tmap.put(43, \"forty-three\");\n\t\t\n\t\tkeyFilter = new Filter() {\n\t\t\t\n\t\t\t@Override\n\t\t\tpublic boolean allows(Integer e) {\n\t\t\t\treturn target >= e;\n\t\t\t}\n\t\t};\n\t\t\n\t\tfilteredMap = new FilteredMap(map, keyFilter);\n\t}\n\n\t@Test\n\tpublic void testFilteredMap() {\n\t\ttry {\n\t\t\tnew FilteredMap(map, null);\n\t\t\tfail(\"able to specify null key filter\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testGetOriginalMap() {\n\t\tassertEquals(map, filteredMap.getOriginalMap());\n\t}\n\n\t@Test\n\tpublic void testGetKeyFilter() {\n\t\tassertEquals(keyFilter, filteredMap.getKeyFilter());\n\t}\n\n\t@Test\n\tpublic void testEntrySet() {\n\t\tfor(Entry e : filteredMap.entrySet()) {\n\t\t\tif(e.getKey() > target) {\n\t\t\t\tfail(\"filtered map contains bad key\");\n\t\t\t}\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testPut() {\n\t\tassertNull(filteredMap.put(47, \"blah\"));\n\t\tassertNull(filteredMap.put(29, \"twenty-nine\"));\n\t\t\n\t\tassertFalse(filteredMap.containsKey(47));\n\t\tassertTrue(filteredMap.containsKey(29));\n\t\t\n\t\tassertEquals(\"twenty-nine\", filteredMap.put(29, \"blah\"));\n\t\tassertEquals(\"blah\", filteredMap.get(29));\n\t}\n\n\t@Test\n\tpublic void testContainsValue() {\n\t\tassertTrue(filteredMap.containsValue(\"seventeen\"));\n\t\tassertFalse(filteredMap.containsValue(\"forty-three\"));\n\t}\n\n\t@Test\n\tpublic void testContainsKey() {\n\t\tassertTrue(filteredMap.containsKey(31));\n\t\tassertFalse(filteredMap.containsKey(43));\n\t}\n\n\t@Test\n\tpublic void testKeySet() {\n\t\tfor(Integer k : filteredMap.keySet()) {\n\t\t\tif(k.equals(43)) {\n\t\t\t\tfail(\"filtered map contains bad key\");\n\t\t\t}\n\t\t}\n\t}\n\n\t@Test\n\tpublic void testValues() {\n\t\tassertTrue(filteredMap.values().contains(\"seventeen\"));\n\t\tassertFalse(filteredMap.values().contains(\"forty-three\"));\n\t\t\n\t\tfilteredMap.remove(17);\n\t\tassertFalse(filteredMap.values().contains(\"seventeen\"));\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/util/FilteredSetTest.java", "content": "package opennlp.ccg.util;\n\nimport static org.junit.Assert.*;\n\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.Iterator;\nimport java.util.List;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class FilteredSetTest {\n\n\tFilteredSet testSet, sameSet;\n\tList values;\n\t\n\tFilter testFilter, sameFilter;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tvalues = new ArrayList(Arrays.asList(\"test\", \"test\", \"other\", \"different\"));\n\t\t\n\t\ttestFilter = new Filter() {\n\t\t\t\n\t\t\t@Override\n\t\t\tpublic boolean allows(String e) {\n\t\t\t\treturn e.equals(\"test\");\n\t\t\t}\n\t\t};\n\t\t\n\t\tsameFilter = new VisitedFilter();\n\t\t\n\t\ttestSet = new FilteredSet(values, testFilter);\n\t\tsameSet = new FilteredSet(values, sameFilter);\n\t}\n\n\t@Test\n\tpublic void testFilteredSet() {\n\t\ttry {\n\t\t\tnew FilteredSet(testSet, null);\n\t\t\tfail(\"able to specify null filter\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testSize() {\n\t\tassertEquals(1, testSet.size());\n\t\tassertEquals(values.size() - 1, sameSet.size());\n\t}\n\n\t@Test\n\tpublic void testAdd() {\n\t\tint sz = testSet.size();\n\t\tassertFalse(testSet.add(\"test\"));\n\t\tassertTrue(testSet.remove(\"test\"));\n\t\tassertEquals(sz - 1, testSet.size());\n\t\t\n\t\tassertFalse(sameSet.add(\"test\"));\n\t\tassertTrue(sameSet.add(\"blah\"));\n\t\tassertTrue(sameSet.add(\"xyxyx\"));\n\t\tassertFalse(sameSet.add(\"xyxyx\"));\n\t\tassertEquals(5, sameSet.size());\n\t}\n\n\t@Test\n\tpublic void testIterator() {\n\t\tIterator i = testSet.iterator();\n\t\tassertTrue(i.hasNext());\n\t\tassertEquals(\"test\", i.next());\n\t\tassertFalse(i.hasNext());\n\t\t\n\t\ti = sameSet.iterator();\n\t\tassertTrue(i.hasNext());\n\t\tassertEquals(\"test\", i.next());\n\t\tassertEquals(\"other\", i.next());\n\t\tassertEquals(\"different\", i.next());\n\t\t\n\t\ti = sameSet.iterator();\n\t\tassertEquals(\"test\", i.next());\n\t\ti.remove();\n\t\tassertFalse(sameSet.contains(\"test\"));\n\t}\n\n\t@Test\n\tpublic void testRemove() {\n\t\ttestSet.remove(\"test\");\n\t\tassertFalse(testSet.contains(\"test\"));\n\t\t\n\t\tsameSet.remove(\"test\");\n\t\tassertFalse(sameSet.contains(\"test\"));\n\t}\n\n\t@Test\n\tpublic void testClear() {\n\t\ttestSet.clear();\n\t\tassertEquals(0, testSet.size());\n\t\tsameSet.clear();\n\t\tassertEquals(0, sameSet.size());\n\t}\n\t\n\t@Test\n\tpublic void testGetOriginalCollection() {\n\t\tassertEquals(values, testSet.getOriginalCollection());\n\t\tassertEquals(values, sameSet.getOriginalCollection());\n\t}\n}\n" }, { "path": "test/opennlp/ccg/util/InverseFilterTest.java", "content": "package opennlp.ccg.util;\n\nimport static org.junit.Assert.*;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class InverseFilterTest {\n\n\tVisitedFilter visited;\n\tInverseFilter inverse;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tvisited = new VisitedFilter();\n\t\tinverse = new InverseFilter(visited);\n\t}\n\n\t@Test\n\tpublic void testInverseFilter() {\n\t\ttry {\n\t\t\tnew InverseFilter(null);\n\t\t\tfail(\"able to specify null filter\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testGetOriginalFilter() {\n\t\tassertEquals(visited, inverse.getOriginalFilter());\n\t}\n\n\t@Test\n\tpublic void testAllows() {\n\t\tassertTrue(visited.allows(37));\n\t\tassertFalse(visited.allows(37));\n\t\t\n\t\tassertTrue(inverse.allows(37));\n\t\tassertFalse(inverse.allows(17));\n\t\tassertFalse(visited.allows(17));\n\t\tassertTrue(inverse.allows(17));\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/util/MembershipFilterTest.java", "content": "package opennlp.ccg.util;\n\nimport static org.junit.Assert.*;\n\nimport java.util.HashSet;\nimport java.util.Set;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class MembershipFilterTest {\n\n\tFilter filter;\n\tSet strings;\n\t\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tstrings = new HashSet();\n\t\tstrings.add(\"test\");\n\t\tstrings.add(\"one\");\n\t\tstrings.add(\"two\");\n\t\tstrings.add(\"three\");\n\t\t\n\t\tfilter = new MembershipFilter(strings);\n\t}\n\n\t@Test\n\tpublic void testMembershipFilter() {\n\t\ttry {\n\t\t\tnew MembershipFilter(null);\n\t\t\tfail(\"able to specify null members\");\n\t\t}\n\t\tcatch(IllegalArgumentException expected) {\n\t\t\t// do nothing\n\t\t}\n\t}\n\t\n\t@Test\n\tpublic void testAllows() {\n\t\tfor(String s : strings) {\n\t\t\tassertTrue(filter.allows(s));\n\t\t}\n\t\t\n\t\tassertFalse(filter.allows(\"blah\"));\n\t\tassertFalse(filter.allows(\"\"));\n\t\tassertFalse(filter.allows(null));\n\t}\n\n}\n" }, { "path": "test/opennlp/ccg/util/VisitedFilterTest.java", "content": "package opennlp.ccg.util;\n\nimport static org.junit.Assert.*;\n\nimport org.junit.Before;\nimport org.junit.Test;\n\npublic class VisitedFilterTest {\n\t\n\tVisitedFilter filter;\n\n\t@Before\n\tpublic void setUp() throws Exception {\n\t\tfilter = new VisitedFilter();\n\t}\n\n\t@Test\n\tpublic void testAllows() {\n\t\tassertTrue(filter.allows(1));\n\t\tassertFalse(filter.allows(1));\n\t\tassertTrue(filter.allows(0));\n\t}\n\n\t@Test\n\tpublic void testHasVisited() {\n\t\tassertTrue(filter.allows(1));\n\t\tassertTrue(filter.hasVisited(1));\n\t\tassertFalse(filter.hasVisited(13));\n\t}\n\n}\n" }, { "path": "test/output.xml", "content": "\n chinese officials have repeatedly indicated that taiwan is a province of china and that china is a domestic chinese issue .\n the chinese government has always been signifying that taiwan is a province of china and taiwan issue is the internal issue of china .\n 1, 1 <-> 2, 2 <-> 3, 2 <-> 5, 2 <-> 6, 3 <-> 4, 4 <-> 3, 4 <-> 5, 4 <-> 6, 5 <-> 7, 6 <-> 8, 6 <-> 15, 7 <-> 9, 8 <-> 10, 9 <-> 11, 10 <-> 12, 11 <-> 13, 12 <-> 14, 15 <-> 17, 16 <-> 18, 17 <-> 19, 18 <-> 21, 18 <-> 22, 19 <-> 20, 20 <-> 23]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n the government of qinghai province made the best use of the situation , coming up with an idea to commercialize the urban infrastructure construction , and ratified , at the beginning of this year , provisions for xining city to encourage and guide foreign investments .\n the government of qinghai province put forward the idea of commercialization of city infrastructure , and approved some regulations on encouraging and attracting foreign investment in xining city early this year .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 3, 4 <-> 4, 13 <-> 5, 13 <-> 6, 14 <-> 5, 14 <-> 6, 15 <-> 5, 15 <-> 6, 16 <-> 7, 17 <-> 8, 18 <-> 9, 18 <-> 10, 19 <-> 9, 19 <-> 10, 21 <-> 12, 22 <-> 13, 24 <-> 14, 25 <-> 15, 26 <-> 16, 28 <-> 28, 29 <-> 28, 30 <-> 28, 32 <-> 29, 33 <-> 30, 35 <-> 18, 36 <-> 19, 36 <-> 25, 37 <-> 26, 38 <-> 27, 39 <-> 19, 40 <-> 20, 41 <-> 21, 42 <-> 22, 43 <-> 23, 44 <-> 24, 45 <-> 31]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n relevant sections of the henan government released their foreign cooperation projects concerning economy and technology . representatives attending the meeting consulted on information of some projects they interested in.\n the competent agencies of the henan provincial government released [ a list of ] foreign economic technological cooperation projects of the province at the meeting and held discussions regarding the purpose of cooperation in the related projects with representatives at the meeting .\n 1, 1 <-> 2, 2 <-> 3, 3 <-> 4, 4 <-> 5, 4 <-> 21, 5 <-> 7, 6 <-> 8, 8 <-> 14, 9 <-> 17, 9 <-> 32, 10 <-> 18, 12 <-> 15, 14 <-> 16, 16 <-> 38, 17 <-> 39, 18 <-> 40, 19 <-> 24, 19 <-> 41, 20 <-> 26, 20 <-> 27, 21 <-> 28, 21 <-> 29, 21 <-> 30, 21 <-> 31, 23 <-> 33, 24 <-> 34, 24 <-> 35, 25 <-> 36]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n foreign minister duma expressed his thoughts that the un was born after world war ii , and that france and other permanent member countries were all the victors of that war .\n the foreign minister feels that the united nations was born out of the second world war , and that france and the other permanent members of the council were victors in that war .\n 1, 1 <-> 2, 3 <-> 3, 5 <-> 3, 6 <-> 4, 7 <-> 5, 8 <-> 6, 8 <-> 7, 9 <-> 8, 10 <-> 9, 11 <-> 10, 11 <-> 11, 12 <-> 14, 13 <-> 15, 14 <-> 13, 15 <-> 16, 16 <-> 17, 17 <-> 18, 18 <-> 19, 19 <-> 20, 20 <-> 22, 21 <-> 23, 22 <-> 24, 23 <-> 24, 24 <-> 28, 27 <-> 29, 28 <-> 30, 29 <-> 31, 30 <-> 32, 31 <-> 33]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n moore said the australian military attache to jakarta will conduct free investigations after he arrives in east timor tomorrow .\n moore said , after the australian military attache to jakarta arrives in east timor tomorrow , he can conduct investigations freely in the area .\n 0, 1 <-> 1, 2 <-> 4, 3 <-> 5, 4 <-> 6, 5 <-> 7, 5 <-> 16, 6 <-> 8, 7 <-> 9, 8 <-> 17, 8 <-> 18, 9 <-> 17, 9 <-> 18, 10 <-> 20, 11 <-> 19, 14 <-> 10, 15 <-> 11, 16 <-> 12, 16 <-> 23, 17 <-> 13, 17 <-> 23, 18 <-> 14, 19 <-> 24]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n at that time , jiang zeming indicated that china would consider joining the missile technology control regime . the indication has been taken as the principle result of the meeting between the two state heads .\n jiang zemin promised at that time that china would consider joining the missile technology control agreement , which was seen as one of the major achievements in the china-us summit meeting .\n 3, 1 <-> 4, 2 <-> 5, 4 <-> 0, 5 <-> 1, 6 <-> 2, 7 <-> 6, 8 <-> 7, 9 <-> 8, 10 <-> 9, 11 <-> 10, 12 <-> 11, 13 <-> 12, 14 <-> 13, 15 <-> 14, 16 <-> 15, 17 <-> 16, 18 <-> 17, 19 <-> 17, 20 <-> 18, 20 <-> 19, 21 <-> 18, 21 <-> 19, 22 <-> 18, 22 <-> 19, 23 <-> 20, 24 <-> 21, 24 <-> 23, 25 <-> 24, 26 <-> 25, 27 <-> 26, 28 <-> 27, 29 <-> 29, 29 <-> 30, 30 <-> 28, 31 <-> 28, 32 <-> 28, 33 <-> 28, 34 <-> 29, 35 <-> 31]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n more than 90% of china 's people centralize on one third of its territory .\n more than 90 percent of the population is compacted into one-third of the country 's territory .\n 0, 1 <-> 1, 2 <-> 2, 2 <-> 3, 3 <-> 4, 4 <-> 13, 6 <-> 6, 7 <-> 7, 7 <-> 8, 8 <-> 9, 9 <-> 10, 10 <-> 10, 11 <-> 11, 12 <-> 13, 12 <-> 14, 13 <-> 15, 14 <-> 16]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n what 's more , next year la nina will be just as bad as el nino .\n and la nina will put on a frightening display next year no less devastating than that of el nino .\n 0, 1 <-> 0, 2 <-> 0, 4 <-> 9, 5 <-> 10, 6 <-> 1, 7 <-> 2, 8 <-> 3, 8 <-> 4, 8 <-> 5, 8 <-> 6, 8 <-> 7, 8 <-> 8, 8 <-> 11, 8 <-> 12, 8 <-> 13, 8 <-> 14, 9 <-> 3, 9 <-> 4, 9 <-> 5, 9 <-> 6, 9 <-> 7, 9 <-> 8, 9 <-> 11, 9 <-> 12, 9 <-> 13, 9 <-> 14, 10 <-> 3, 10 <-> 4, 10 <-> 6, 10 <-> 8, 10 <-> 11, 10 <-> 12, 10 <-> 13, 10 <-> 14, 11 <-> 3, 11 <-> 4, 11 <-> 6, 11 <-> 8, 11 <-> 11, 11 <-> 12, 11 <-> 13, 11 <-> 14, 12 <-> 3, 12 <-> 4, 12 <-> 6, 12 <-> 8, 12 <-> 11, 12 <-> 12, 12 <-> 13, 12 <-> 14, 13 <-> 3, 13 <-> 4, 13 <-> 6, 13 <-> 8, 13 <-> 11, 13 <-> 12, 13 <-> 13, 13 <-> 14, 14 <-> 17, 15 <-> 18, 16 <-> 19]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n russian prime minister on domestic economy\n russian prime minister talked about russian economic situation .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 4, 4 <-> 5, 5 <-> 6, 5 <-> 7]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n nowadays , there appeared a lot of enterprise groups with high technology and strong stamina .\n nowadays , there appeared a lot of enterprise groups with high technology and strong stamina .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 3, 4 <-> 4, 5 <-> 5, 6 <-> 6, 7 <-> 7, 8 <-> 8, 9 <-> 9, 10 <-> 10, 11 <-> 11, 12 <-> 12, 13 <-> 13, 14 <-> 14, 15 <-> 15]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n the gnp created by the village and township enterprises occupied about 1\\/3 of the total gnp in fujian province .\n the gross national product of the industry of villages and towns made up 1\\/3 of the gnp of fujian province .\n 0, 1 <-> 1, 1 <-> 2, 1 <-> 3, 3 <-> 4, 4 <-> 5, 5 <-> 8, 6 <-> 9, 7 <-> 10, 8 <-> 6, 9 <-> 11, 9 <-> 12, 11 <-> 13, 12 <-> 14, 13 <-> 15, 15 <-> 16, 16 <-> 17, 17 <-> 18, 18 <-> 19, 19 <-> 20]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n both parties expressed satisfaction for the friendly cooperation between the two armies in the past 30 years .\n both parties expressed satisfaction for the friendly cooperation between the two armies in the past 30 years .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 3, 4 <-> 4, 5 <-> 5, 6 <-> 6, 7 <-> 7, 8 <-> 8, 9 <-> 9, 10 <-> 10, 11 <-> 11, 12 <-> 12, 13 <-> 13, 14 <-> 14, 15 <-> 15, 16 <-> 16, 17 <-> 17]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n also present at the seminar will be long yongtu , vice minister of china 's ministry of foreign trade and economic cooperation who will accompany rugerro to shanghai , and shen jueren , former chief negotiator of china in gatt negotiations .\n attendees will also include the vice minister of foreign trade & economic cooperation , yongtu long , who accompanied ruggiero to shanghai , and former chief representative for trade negotiations of the moftec .\n 2, 1 <-> 0, 1 <-> 3, 5 <-> 1, 6 <-> 3, 7 <-> 15, 8 <-> 14, 9 <-> 16, 10 <-> 5, 11 <-> 6, 16 <-> 7, 17 <-> 8, 18 <-> 9, 19 <-> 10, 20 <-> 11, 21 <-> 12, 22 <-> 17, 23 <-> 18, 24 <-> 18, 25 <-> 19, 26 <-> 20, 27 <-> 21, 28 <-> 22, 29 <-> 23, 33 <-> 24, 34 <-> 25, 35 <-> 26, 35 <-> 27, 35 <-> 29, 36 <-> 30, 37 <-> 32, 38 <-> 27, 39 <-> 28, 40 <-> 29, 41 <-> 33]]]>\n \n xinhua news agency , nanjing , 16 december , by zhoufang the investment of foreign capital in agriculture of jiangsu increased .\n xinhua news agency , nanjing , dec. 16 . ( reporter zhou fang ) an increasing number of investments from foreign capital are being made on agriculture in jiangsu .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 3, 4 <-> 4, 5 <-> 5, 6 <-> 7, 7 <-> 6, 8 <-> 8, 10 <-> 11, 10 <-> 12, 12 <-> 18, 13 <-> 19, 14 <-> 20, 15 <-> 21, 16 <-> 25, 17 <-> 26, 18 <-> 27, 19 <-> 28, 20 <-> 15, 21 <-> 29]]]>\n \n france ministry of foreign affairs said that leaders of iraq refused to co-operate with united nations and have caused this deeply regretted outcome .\n the french foreign ministry said that because iraqi leader refuses to cooperate with the un caused this sad events .\n 1, 1 <-> 3, 2 <-> 2, 3 <-> 2, 4 <-> 2, 5 <-> 4, 6 <-> 5, 7 <-> 8, 8 <-> 7, 9 <-> 7, 10 <-> 9, 11 <-> 10, 12 <-> 11, 13 <-> 12, 14 <-> 14, 15 <-> 14, 17 <-> 15, 18 <-> 15, 19 <-> 16, 20 <-> 17, 21 <-> 17, 22 <-> 18, 23 <-> 19]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n during the preliminaries , lan wei , a highly-skilled twenty-six year-old native of guangdong , scored 355.35 for first place .\n in the qualifiers , 26-year-old lan wei from guangdong displayed excellent skill , and ranked no. 1 with a total score of 355.3 .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 3, 4 <-> 5, 5 <-> 6, 8 <-> 9, 8 <-> 10, 8 <-> 11, 9 <-> 4, 10 <-> 4, 11 <-> 7, 12 <-> 7, 13 <-> 8, 14 <-> 12, 15 <-> 20, 16 <-> 22, 18 <-> 15, 18 <-> 16, 19 <-> 14, 19 <-> 15, 20 <-> 23]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n nanjing , december 16 ( xinhua ) foreign fund put into jiangsu 's agricultural sector is increasing\n xinhua news agency , nanjing , dec. 16 ( reporter zhou fang ) - foreign investment in jiangsu 's agriculture is on the rise .\n 4, 1 <-> 5, 2 <-> 6, 3 <-> 7, 4 <-> 8, 5 <-> 0, 7 <-> 14, 8 <-> 15, 8 <-> 16, 9 <-> 15, 9 <-> 16, 10 <-> 15, 10 <-> 16, 11 <-> 17, 12 <-> 18, 13 <-> 19, 14 <-> 19, 15 <-> 20, 16 <-> 21, 16 <-> 22, 16 <-> 23]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n someone connected global warming to the phenomenon of el nino .\n some linked global warming with el nino .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 3, 4 <-> 4, 8 <-> 5, 9 <-> 6, 10 <-> 7]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n xinhua news agency , beijing , january 16th , by xintang xu and yuhong qian- the president of the bank of china , xianglong dai said that , china will continue implementing the financial opening up policy .\n xinhua news agency , beijing , january 16th ( reporter : xu xingtang , qian yuhong ) . dai xianglong , the president of people 's bank of china , said that china will carry on with the open financial policy .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 3, 4 <-> 4, 5 <-> 5, 6 <-> 6, 7 <-> 7, 10 <-> 10, 11 <-> 9, 12 <-> 11, 13 <-> 13, 14 <-> 12, 15 <-> 19, 16 <-> 20, 17 <-> 21, 18 <-> 23, 19 <-> 24, 20 <-> 25, 21 <-> 26, 22 <-> 27, 23 <-> 17, 24 <-> 16, 25 <-> 28, 26 <-> 29, 28 <-> 30, 29 <-> 31, 30 <-> 32, 30 <-> 33, 31 <-> 34, 32 <-> 35, 33 <-> 37, 34 <-> 36, 35 <-> 36, 36 <-> 38, 37 <-> 39]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n this kind of measure restrained the economic growth , leading to the rise of unemployment .\n such measures checked economic growth and caused unemployment to rise .\n 0, 1 <-> 0, 2 <-> 0, 3 <-> 1, 4 <-> 2, 6 <-> 3, 7 <-> 4, 8 <-> 5, 9 <-> 6, 10 <-> 6, 12 <-> 9, 14 <-> 7, 15 <-> 10]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n siemens germany is currently cooperating with a local thai company , and constructing another 23km long electric railway project in bangkok .\n the german shermans company is currently working together with a local company constructing a 23 kilometer electronic train engineering project .\n 2, 1 <-> 1, 2 <-> 4, 3 <-> 5, 4 <-> 6, 4 <-> 7, 5 <-> 8, 6 <-> 9, 7 <-> 10, 9 <-> 11, 12 <-> 12, 13 <-> 13, 14 <-> 14, 14 <-> 15, 16 <-> 16, 17 <-> 17, 17 <-> 18, 18 <-> 19, 21 <-> 20]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n he said that the same resolve would lead to the success of the kyoto protocol .\n he said that the same resolve would lead to the success of the kyoto protocol .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 3, 4 <-> 4, 5 <-> 5, 6 <-> 6, 7 <-> 7, 8 <-> 8, 9 <-> 9, 10 <-> 10, 11 <-> 11, 12 <-> 12, 13 <-> 13, 14 <-> 14, 15 <-> 15]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n however , the official who disclosed the above declined to elaborate what kind of missile technology is being exported and when china provided such technology to pakistan and iran .\n however , the official was not willing to explain what kind of missile technology that china provided to pakistan and iran , and when this technology was provided .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 3, 8 <-> 4, 8 <-> 5, 8 <-> 6, 9 <-> 7, 10 <-> 8, 11 <-> 9, 12 <-> 10, 13 <-> 11, 14 <-> 12, 15 <-> 13, 19 <-> 22, 20 <-> 23, 21 <-> 15, 22 <-> 16, 22 <-> 26, 22 <-> 27, 23 <-> 24, 24 <-> 13, 24 <-> 25, 25 <-> 17, 26 <-> 18, 27 <-> 19, 28 <-> 20, 29 <-> 28]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n yunnan gardens was developed by fujian company , there are 313 apartments in this residential project .\n the yunnan gardens project developed by the fujian society is a 313-unit housing project with land .\n 1, 1 <-> 2, 2 <-> 4, 3 <-> 4, 4 <-> 5, 5 <-> 7, 6 <-> 8, 9 <-> 9, 10 <-> 11, 11 <-> 11, 13 <-> 10, 14 <-> 12, 15 <-> 13, 16 <-> 16]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n in the last two years , some foreign businessmen showed their interest in the construction of xiling city .\n during the past two years , a batch of foreign businessmen expressed their wishes to get involved in xining 's city construction one after another .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 3, 4 <-> 4, 5 <-> 5, 6 <-> 6, 6 <-> 7, 6 <-> 8, 7 <-> 9, 8 <-> 10, 9 <-> 11, 10 <-> 12, 11 <-> 13, 12 <-> 17, 13 <-> 19, 14 <-> 21, 15 <-> 19, 16 <-> 18, 17 <-> 20, 18 <-> 25]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n the dai yu xiang industrial consulting company said that spacious front door area is the biggest characteristic of yunnan gardens with land ownership .\n dai yuxiang real estate consulting firm said that the most special characteristic of the landed residences at yunnan gardens is their spacious appearance .\n 0, 2 <-> 1, 3 <-> 1, 4 <-> 2, 4 <-> 3, 5 <-> 4, 6 <-> 5, 7 <-> 6, 8 <-> 7, 9 <-> 21, 10 <-> 22, 11 <-> 22, 12 <-> 22, 13 <-> 19, 14 <-> 8, 15 <-> 9, 15 <-> 10, 16 <-> 11, 17 <-> 12, 18 <-> 17, 19 <-> 18, 20 <-> 14, 21 <-> 14, 22 <-> 15, 22 <-> 22, 23 <-> 23]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n this measure restrained economic growth and raised the rate of unemployment .\n this curbs economic growth and increases unemployment rates .\n 0, 2 <-> 1, 3 <-> 2, 4 <-> 3, 5 <-> 4, 6 <-> 5, 8 <-> 7, 10 <-> 6, 11 <-> 8]]]>\n \n according to xinhua news agency , london , on february 3rd , bbc reporter bowen stayed in baghdad . the capital of iraq stated that there was no evidence to indicate the bomb-shelter destroyed by a us bomber was a military target .\n according to xinhua news report from london on february 13 , bonne , a reporter of bbc in iraqi capital baghdad , confirmed on 13th that no any sign showed the bomb shelter in baghdad destroyed by american bomber was a military blindage .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 3, 6 <-> 6, 8 <-> 7, 9 <-> 8, 10 <-> 9, 11 <-> 10, 12 <-> 15, 12 <-> 16, 13 <-> 14, 14 <-> 11, 16 <-> 17, 17 <-> 20, 17 <-> 34, 20 <-> 19, 21 <-> 18, 22 <-> 18, 23 <-> 22, 24 <-> 25, 26 <-> 26, 27 <-> 26, 28 <-> 28, 30 <-> 29, 31 <-> 30, 32 <-> 31, 32 <-> 32, 33 <-> 35, 34 <-> 36, 36 <-> 37, 37 <-> 38, 38 <-> 39, 39 <-> 40, 40 <-> 41, 41 <-> 42, 42 <-> 43]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n mr. siazon said , someone used to say , have n't we been burned once before ? when we were ready to accept them , it did n't take long before they fought agaion .\n siazon said : someone says , were we deceived before ? when we prepared to accept them but just after several days they began to fight again .\n 0, 1 <-> 0, 2 <-> 1, 3 <-> 2, 4 <-> 3, 5 <-> 4, 6 <-> 4, 7 <-> 4, 8 <-> 5, 9 <-> 6, 9 <-> 8, 11 <-> 7, 12 <-> 6, 12 <-> 8, 13 <-> 6, 13 <-> 8, 15 <-> 9, 16 <-> 10, 17 <-> 11, 18 <-> 12, 19 <-> 13, 20 <-> 13, 21 <-> 14, 22 <-> 15, 23 <-> 16, 24 <-> 17, 25 <-> 18, 25 <-> 19, 25 <-> 20, 25 <-> 21, 26 <-> 18, 26 <-> 19, 26 <-> 20, 26 <-> 21, 27 <-> 18, 27 <-> 19, 27 <-> 20, 27 <-> 21, 28 <-> 18, 28 <-> 19, 28 <-> 20, 28 <-> 21, 29 <-> 18, 29 <-> 19, 29 <-> 20, 29 <-> 21, 30 <-> 18, 30 <-> 19, 30 <-> 20, 30 <-> 21, 31 <-> 22, 32 <-> 23, 32 <-> 24, 32 <-> 25, 33 <-> 26, 34 <-> 27]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n railroad officials blamed the bus passengers for the accident , the indian news agency india press trust report said .\n report of pti said that railway official charged the bus passengers upon the accident .\n 5, 1 <-> 6, 2 <-> 7, 3 <-> 8, 4 <-> 9, 5 <-> 10, 6 <-> 11, 7 <-> 12, 8 <-> 13, 14 <-> 2, 15 <-> 2, 16 <-> 2, 17 <-> 0, 18 <-> 3, 19 <-> 14]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n we mentioned the problems australia is concerned with , such as human rights and dispatching army officers to east timor .\n we mentioned the problems that have attracted australia 's attention , such as the humam rights in east timor and the issue of sending military officers to east timor .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 3, 4 <-> 7, 5 <-> 5, 5 <-> 6, 5 <-> 8, 5 <-> 9, 6 <-> 5, 6 <-> 6, 6 <-> 8, 6 <-> 9, 7 <-> 5, 7 <-> 6, 7 <-> 8, 7 <-> 9, 8 <-> 10, 9 <-> 11, 10 <-> 12, 11 <-> 14, 12 <-> 15, 13 <-> 19, 14 <-> 23, 15 <-> 24, 16 <-> 25, 17 <-> 26, 18 <-> 27, 19 <-> 17, 19 <-> 28, 20 <-> 18, 20 <-> 29]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n the two prime ministers requested that un secretary-general prolong the un representative 's stay in cambodia to 6 months or longer .\n the two prime ministers of cambodia asked the un secretary general to extend the un representative 's term of stay in cambodia to six months or longer .\n 0, 1 <-> 1, 2 <-> 2, 3 <-> 3, 4 <-> 6, 6 <-> 8, 7 <-> 9, 7 <-> 10, 8 <-> 12, 9 <-> 13, 10 <-> 14, 11 <-> 15, 12 <-> 16, 13 <-> 19, 14 <-> 20, 15 <-> 5, 15 <-> 21, 16 <-> 22, 17 <-> 23, 18 <-> 24, 19 <-> 25, 20 <-> 26, 21 <-> 27]]]>\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "test/paraphrases.xml", "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nchinese officials have repeatedly indicated that taiwan is a province of china and that china is a domestic chinese issue .\nw19:n:NN:issue w9:n:NN:province w18:n/n:NN:chinese w8:np/n:DT:a w17:n/n:JJ:domestic w7:s[dcl]\\np/np:VBZ:be w16:np/n:DT:a w6:n:NN:taiwan w15:s[dcl]\\np/np:VBZ:be w14:n:NN:china w4:s[pt]\\np/s[em]:VBN:indicate w3:s\\np\\(s\\np):RB:repeatedly w12:s[em]$\\(s[em]$)/(s[em]$):CC:and w2:s[dcl]\\np/(s[pt]\\np):VBP:PERF w11:n:NN:china w1:n:NNS:official w10:np\\np/np:IN:of w0:n/n:NN:chinese\nchinese:S-chinese:P-NN:T-n/n officials:S-official:P-NNS:T-n have:S-have:P-VBP:T-s[dcl]\\np/(s[pt]\\np) repeatedly:S-repeatedly:P-RB:T-s\\np\\(s\\np) indicated:S-indicate:P-VBN:T-s[pt]\\np/s[em] that:S-that:P-IN:T-s[em]/s[dcl] taiwan:S-taiwan:P-NN:T-n is:S-be:P-VBZ:T-s[dcl]\\np/np a:S-a:P-DT:T-np/n province:S-province:P-NN:T-n of:S-of:P-IN:T-np\\np/np china:S-china:P-NN:T-n and:S-and:P-CC:T-s[em]$\\(s[em]$)/(s[em]$) that:S-that:P-DT:T-s[em]/s[dcl] china:S-china:P-NN:T-n is:S-be:P-VBZ:T-s[dcl]\\np/np a:S-a:P-DT:T-np/n domestic:S-domestic:P-JJ:T-n/n chinese:S-chinese:P-NN:T-n/n issue:S-issue:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nthe chinese government has always been signifying that taiwan is a province of china and taiwan issue is the internal issue of china .\nw9:s[dcl]\\np/np:VBZ:be w19:n/n:JJ:internal w18:np/n:DT:the w8:n:NN:taiwan w17:s[dcl]\\np/np:VBZ:be w16:n:NN:issue w6:s[ng]\\np/s[em]:VBG:signify w15:n/n:JJ:taiwan w5:s[pt]\\np/(s[ng]\\np):VBN:PROG w14:s[dcl]$\\(s[dcl]$)/(s[dcl]$):CC:and w4:s\\np\\(s\\np):RB:always w13:n:NN:china w3:s[dcl]\\np/(s[pt]\\np):VBZ:PERF w12:np\\np/np:IN:of w2:n:NN:government w11:n:NN:province w1:n/n:JJ:chinese w10:np/n:DT:a w0:np/n:DT:the w22:n:NN:china w21:np\\np/np:IN:of w20:n:NN:issue\nthe:S-the:P-DT:T-np/n chinese:S-chinese:P-JJ:T-n/n government:S-government:P-NN:T-n has:S-have:P-VBZ:T-s[dcl]\\np/(s[pt]\\np) always:S-always:P-RB:T-s\\np\\(s\\np) been:S-be:P-VBN:T-s[pt]\\np/(s[ng]\\np) signifying:S-signify:P-VBG:T-s[ng]\\np/s[em] that:S-that:P-DT:T-s[em]/s[dcl] taiwan:S-taiwan:P-NN:T-n is:S-be:P-VBZ:T-s[dcl]\\np/np a:S-a:P-DT:T-np/n province:S-province:P-NN:T-n of:S-of:P-IN:T-np\\np/np china:S-china:P-NN:T-n and:S-and:P-CC:T-s[dcl]$\\(s[dcl]$)/(s[dcl]$) taiwan:S-taiwan:P-JJ:T-n/n issue:S-issue:P-NN:T-n is:S-be:P-VBZ:T-s[dcl]\\np/np the:S-the:P-DT:T-np/n internal:S-internal:P-JJ:T-n/n issue:S-issue:P-NN:T-n of:S-of:P-IN:T-np\\np/np china:S-china:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nthe government of qinghai province made the best use of the situation , coming up with an idea to commercialize the urban infrastructure construction , and ratified , at the beginning of this year , provisions for xining city to encourage and guide foreign investments .\nw36:s[ng]\\np/np:VBG:xining w35:np\\np/(s[ng]\\np):IN:for w34:n:NNS:provision w33:np\\np/np:,:, w32:n:NN:year w31:np/n:DT:this w30:np\\np/np:IN:of w29:n:NN:beginning w28:np/n:DT:the w27:s\\np\\(s\\np)/np:IN:at w26:s[dcl]\\np:VBD:ratify w25:s[dcl]$\\(s[dcl]$)/(s[dcl]$):CC:and w23:n:NN:construction w22:n/n:NN:infrastructure w21:n/n:JJ:urban w20:np/n:DT:the w9:np\\np/np:IN:of w8:n:NN:use w7:n/n:JJS:best w6:np/n:DT:the w5:s[dcl]\\np/np:VBD:make w4:n:NN:province w3:n/n:NNP:qinghai w19:s[b]\\np/np:VB:commercialize w2:np\\np/np:IN:of w1:n:NN:government w17:n:NN:idea w0:np/n:DT:the w16:np/n:DT:an w15:pp/np:IN:with w14:s\\np\\(s\\np):RP:up w13:s[ng]\\np/pp:VBG:come w11:n:NN:situation w43:n:NNS:investment w10:np/n:DT:the w42:n/n:JJ:foreign w41:s[b]\\np/np:VB:guide w40:s[b]$\\(s[b]$)/(s[b]$):CC:and w39:s[b]\\np/np:VB:encourage w37:n:NN:city\nthe:S-the:P-DT:T-np/n government:S-government:P-NN:T-n of:S-of:P-IN:T-np\\np/np qinghai:S-qinghai:P-NNP:T-n/n province:S-province:P-NN:T-n made:S-make:P-VBD:T-s[dcl]\\np/np the:S-the:P-DT:T-np/n best:S-best:P-JJS:T-n/n use:S-use:P-NN:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n situation:S-situation:P-NN:T-n ,:S-,:P-,:T-s\\np\\(s\\np)/punct[,]/(s\\np) coming:S-come:P-VBG:T-s[ng]\\np/pp up:S-up:P-RP:T-s\\np\\(s\\np) with:S-with:P-IN:T-pp/np an:S-an:P-DT:T-np/n idea:S-idea:P-NN:T-n to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) commercialize:S-commercialize:P-VB:T-s[b]\\np/np the:S-the:P-DT:T-np/n urban:S-urban:P-JJ:T-n/n infrastructure:S-infrastructure:P-NN:T-n/n construction:S-construction:P-NN:T-n ,:S-,:P-,:T-punct[,] and:S-and:P-CC:T-s[dcl]$\\(s[dcl]$)/(s[dcl]$) ratified:S-ratify:P-VBD:T-s[dcl]\\np ,:S-,:P-,:T-punct[,] at:S-at:P-IN:T-s\\np\\(s\\np)/np the:S-the:P-DT:T-np/n beginning:S-beginning:P-NN:T-n of:S-of:P-IN:T-np\\np/np this:S-this:P-DT:T-np/n year:S-year:P-NN:T-n ,:S-,:P-,:T-np\\np/np provisions:S-provision:P-NNS:T-n for:S-for:P-IN:T-np\\np/(s[ng]\\np) xining:S-xining:P-VBG:T-s[ng]\\np/np city:S-city:P-NN:T-n to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) encourage:S-encourage:P-VB:T-s[b]\\np/np and:S-and:P-CC:T-s[b]$\\(s[b]$)/(s[b]$) guide:S-guide:P-VB:T-s[b]\\np/np foreign:S-foreign:P-JJ:T-n/n investments:S-investment:P-NNS:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nthe government of qinghai province put forward the idea of commercialization of city infrastructure , and approved some regulations on encouraging and attracting foreign investment in xining city early this year .\nw30:n:NN:year w19:np\\np/(s[ng]\\np):IN:on w9:np\\np/np:IN:of w18:n:NNS:regulation w8:n:NN:idea w17:np/n:DT:some w7:np/n:DT:the w16:s[dcl]\\np/np:VBD:approve w6:s\\np\\(s\\np):RP:forward w15:s[dcl]$\\(s[dcl]$)\\punct[,]/(s[dcl]$):CC:and w5:s[dcl]\\np/np:VBD:put w4:n:NN:province w13:n:NN:infrastructure w3:n/n:NNP:qinghai w12:n/n:NN:city w2:np\\np/np:IN:of w1:n:NN:government w11:np\\np/np:IN:of w10:n:NN:commercialization w0:np/n:DT:the w29:s\\np\\(s\\np)/n:DT:this w28:s\\np\\(s\\np)/(s\\np\\(s\\np)):RB:early w27:n:NN:city w26:s[ng]\\np/np:VBG:xining w25:np\\np/(s[ng]\\np):IN:in w24:n:NN:investment w23:n/n:JJ:foreign w22:s[ng]\\np/np:VBG:attract w21:s[ng]$\\(s[ng]$)/(s[ng]$):CC:and w20:s[ng]\\np/np:VBG:encourage\nthe:S-the:P-DT:T-np/n government:S-government:P-NN:T-n of:S-of:P-IN:T-np\\np/np qinghai:S-qinghai:P-NNP:T-n/n province:S-province:P-NN:T-n put:S-put:P-VBD:T-s[dcl]\\np/np forward:S-forward:P-RP:T-s\\np\\(s\\np) the:S-the:P-DT:T-np/n idea:S-idea:P-NN:T-n of:S-of:P-IN:T-np\\np/np commercialization:S-commercialization:P-NN:T-n of:S-of:P-IN:T-np\\np/np city:S-city:P-NN:T-n/n infrastructure:S-infrastructure:P-NN:T-n ,:S-,:P-,:T-punct[,] and:S-and:P-CC:T-s[dcl]$\\(s[dcl]$)\\punct[,]/(s[dcl]$) approved:S-approve:P-VBD:T-s[dcl]\\np/np some:S-some:P-DT:T-np/n regulations:S-regulation:P-NNS:T-n on:S-on:P-IN:T-np\\np/(s[ng]\\np) encouraging:S-encourage:P-VBG:T-s[ng]\\np/np and:S-and:P-CC:T-s[ng]$\\(s[ng]$)/(s[ng]$) attracting:S-attract:P-VBG:T-s[ng]\\np/np foreign:S-foreign:P-JJ:T-n/n investment:S-investment:P-NN:T-n in:S-in:P-IN:T-np\\np/(s[ng]\\np) xining:S-xining:P-VBG:T-s[ng]\\np/np city:S-city:P-NN:T-n early:S-early:P-RB:T-s\\np\\(s\\np)/(s\\np\\(s\\np)) this:S-this:P-DT:T-s\\np\\(s\\np)/n year:S-year:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nrelevant sections of the henan government released their foreign cooperation projects concerning economy and technology . representatives attending the meeting consulted on information of some projects they interested in.\nw19:s[dcl]\\np/np/pp:VBD:consulted w9:n/n:NN:cooperation w18:n:NN:meeting w8:n/n:JJ:foreign w17:np/n:DT:the w7:np/n:PRP$:their w6:s[dcl]\\np/np:VBD:release w16:s[ng]\\np/np:VBG:attend w15:n:NNS:representative w5:n:NN:government w14:n:NN:technology w4:n/n:JJ:henan w13:n\\n/n:CC:and w3:np/n:DT:the w12:n:NN:economy w2:np\\np/np:IN:of w11:n/n:VBG:concern w1:n:NNS:section w10:n/n:NNS:project w0:n/n:JJ:relevant w27:s[adj]\\np\\(s[adj]\\np):NN:in. w26:s[adj]\\np:JJ:interested w25:np:PRP:they w24:n:NNS:project w23:np/n:DT:some w22:np\\np/np:IN:of w21:n:NN:information w20:pp/np:IN:on\nrelevant:S-relevant:P-JJ:T-n/n sections:S-section:P-NNS:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n henan:S-henan:P-JJ:T-n/n government:S-government:P-NN:T-n released:S-release:P-VBD:T-s[dcl]\\np/np their:S-their:P-PRP$:T-np/n foreign:S-foreign:P-JJ:T-n/n cooperation:S-cooperation:P-NN:T-n/n projects:S-project:P-NNS:T-n/n concerning:S-concern:P-VBG:T-n/n economy:S-economy:P-NN:T-n and:S-and:P-CC:T-n\\n/n technology:S-technology:P-NN:T-n .:S-.:P-.:T-punct[.] representatives:S-representative:P-NNS:T-n attending:S-attend:P-VBG:T-s[ng]\\np/np the:S-the:P-DT:T-np/n meeting:S-meeting:P-NN:T-n consulted:S-consulted:P-VBD:T-s[dcl]\\np/np/pp on:S-on:P-IN:T-pp/np information:S-information:P-NN:T-n of:S-of:P-IN:T-np\\np/np some:S-some:P-DT:T-np/n projects:S-project:P-NNS:T-n they:S-they:P-PRP:T-np interested:S-interested:P-JJ:T-s[adj]\\np in.:S-in.:P-NN:T-s[adj]\\np\\(s[adj]\\np)\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nthe competent agencies of the henan provincial government released [ a list of ] foreign economic technological cooperation projects of the province at the meeting and held discussions regarding the purpose of cooperation in the related projects with representatives at the meeting .\nw36:n:NNS:project w35:n/n:JJ:related w34:np/n:DT:the w33:np\\np/np:IN:in w32:n:NN:cooperation w31:np\\np/np:IN:of w30:n:NN:purpose w29:np/n:DT:the w28:s[ng]\\np/np:VBG:regard w27:n:NNS:discussion w26:s[dcl]\\np/np:VBD:hold w25:s[dcl]$\\(s[dcl]$)/(s[dcl]$):CC:and w24:n:NN:meeting w23:np/n:DT:the w22:s\\np\\(s\\np)/np:IN:at w21:n:NN:province w20:np/n:DT:the w9:s\\np\\(s\\np)/np:IN:lsb w8:s[dcl]\\np/np:VBD:release w7:n:NN:government w6:n/n:JJ:provincial w5:n/n:JJ:henan w4:np/n:DT:the w3:np\\np/np:IN:of w19:np\\np/np:IN:of w2:n:NNS:agency w18:n:NNS:project w1:n/n:JJ:competent w17:n/n:NN:cooperation w0:np/n:DT:the w16:n/n:JJ:technological w15:n/n:JJ:economic w14:n/n:JJ:foreign w13:n/n:JJ:rsb w12:np\\np/np:IN:of w11:n:NN:list w10:np/n:DT:a w41:n:NN:meeting w40:np/n:DT:the w39:s\\np\\(s\\np)/np:IN:at w38:n:NNS:representative w37:np\\np/np:IN:with\nthe:S-the:P-DT:T-np/n competent:S-competent:P-JJ:T-n/n agencies:S-agency:P-NNS:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n henan:S-henan:P-JJ:T-n/n provincial:S-provincial:P-JJ:T-n/n government:S-government:P-NN:T-n released:S-release:P-VBD:T-s[dcl]\\np/np [:S-[:P-IN:T-s\\np\\(s\\np)/np a:S-a:P-DT:T-np/n list:S-list:P-NN:T-n of:S-of:P-IN:T-np\\np/np ]:S-]:P-JJ:T-n/n foreign:S-foreign:P-JJ:T-n/n economic:S-economic:P-JJ:T-n/n technological:S-technological:P-JJ:T-n/n cooperation:S-cooperation:P-NN:T-n/n projects:S-project:P-NNS:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n province:S-province:P-NN:T-n at:S-at:P-IN:T-s\\np\\(s\\np)/np the:S-the:P-DT:T-np/n meeting:S-meeting:P-NN:T-n and:S-and:P-CC:T-s[dcl]$\\(s[dcl]$)/(s[dcl]$) held:S-hold:P-VBD:T-s[dcl]\\np/np discussions:S-discussion:P-NNS:T-n regarding:S-regard:P-VBG:T-s[ng]\\np/np the:S-the:P-DT:T-np/n purpose:S-purpose:P-NN:T-n of:S-of:P-IN:T-np\\np/np cooperation:S-cooperation:P-NN:T-n in:S-in:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n related:S-related:P-JJ:T-n/n projects:S-project:P-NNS:T-n with:S-with:P-IN:T-np\\np/np representatives:S-representative:P-NNS:T-n at:S-at:P-IN:T-s\\np\\(s\\np)/np the:S-the:P-DT:T-np/n meeting:S-meeting:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nforeign minister duma expressed his thoughts that the un was born after world war ii , and that france and other permanent member countries were all the victors of that war .\nw30:n:NN:war w9:s[dcl]\\np/(s[pss]\\np):VBD:PASS w19:n\\(n/n)/n:CC:and w8:n:NN:un w18:n/n:NN:france w7:np/n:DT:the h1:n\\(n/n)/n:CC:has-rel w16:s[em]$\\(s[em]$)\\punct[,]/(s[em]$):CC:and w5:n/s[em]:NNS:thought w14:n:NNS:ii w4:np/n:PRP$:his w3:s[dcl]\\np/np:VBD:express w13:n/n:NN:war w12:n/n:NN:world w2:n:NN:duma w11:s\\np\\(s\\np)/np:IN:after w1:n/n:NN:minister w10:s[pss]\\np:VBN:bear w0:n/n:JJ:foreign w29:np/n:DT:that w28:np\\np/np:IN:of w27:n:NNS:victors w26:np/n:DT:the w25:np/np:PDT:all w24:s[dcl]\\np/np:VBD:be w23:n:NNS:country w22:n/n:NN:member w21:n/n:JJ:permanent w20:n/n:JJ:other\nforeign:S-foreign:P-JJ:T-n/n minister:S-minister:P-NN:T-n/n duma:S-duma:P-NN:T-n expressed:S-express:P-VBD:T-s[dcl]\\np/np his:S-his:P-PRP$:T-np/n thoughts:S-thought:P-NNS:T-n/s[em] that:S-that:P-IN:T-s[em]/s[dcl] the:S-the:P-DT:T-np/n un:S-un:P-NN:T-n was:S-be:P-VBD:T-s[dcl]\\np/(s[pss]\\np) born:S-bear:P-VBN:T-s[pss]\\np after:S-after:P-IN:T-s\\np\\(s\\np)/np world:S-world:P-NN:T-n/n war:S-war:P-NN:T-n/n ii:S-ii:P-NNS:T-n ,:S-,:P-,:T-punct[,] and:S-and:P-CC:T-s[em]$\\(s[em]$)\\punct[,]/(s[em]$) that:S-that:P-DT:T-s[em]/s[dcl] france:S-france:P-NN:T-n/n and:S-and:P-CC:T-n\\(n/n)/n other:S-other:P-JJ:T-n/n permanent:S-permanent:P-JJ:T-n/n member:S-member:P-NN:T-n/n countries:S-country:P-NNS:T-n were:S-be:P-VBD:T-s[dcl]\\np/np all:S-all:P-PDT:T-np/np the:S-the:P-DT:T-np/n victors:S-victors:P-NNS:T-n of:S-of:P-IN:T-np\\np/np that:S-that:P-DT:T-np/n war:S-war:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nthe foreign minister feels that the united nations was born out of the second world war , and that france and the other permanent members of the council were victors in that war .\nw32:n:NN:war w31:np/n:DT:that w30:s\\np\\(s\\np)/np:IN:in w9:s[pss]\\np:VBN:bear w19:n:NN:france w8:s[dcl]\\np/(s[pss]\\np):VBD:PASS w17:s[em]$\\(s[em]$)\\punct[,]/(s[em]$):CC:and w7:n:NNS:nation w6:n/n:JJ:united w15:n:NN:war w5:np/n:DT:the w14:n/n:NN:world w3:s[dcl]\\np/s[em]:VBZ:feel w13:n/n:JJ:second w2:n:NN:minister w12:np/n:DT:the w11:pp/np:IN:of w1:n/n:JJ:foreign w10:s\\np\\(s\\np)/pp:IN:out w0:np/n:DT:the w29:n:NNS:victors w28:s[dcl]\\np/np:VBD:be w27:n:NN:council w26:np/n:DT:the w25:np\\np/np:IN:of w24:n:NNS:member w23:n/n:JJ:permanent w22:n/n:JJ:other w21:np/n:DT:the w20:np\\np/np:CC:and\nthe:S-the:P-DT:T-np/n foreign:S-foreign:P-JJ:T-n/n minister:S-minister:P-NN:T-n feels:S-feel:P-VBZ:T-s[dcl]\\np/s[em] that:S-that:P-IN:T-s[em]/s[dcl] the:S-the:P-DT:T-np/n united:S-united:P-JJ:T-n/n nations:S-nation:P-NNS:T-n was:S-be:P-VBD:T-s[dcl]\\np/(s[pss]\\np) born:S-bear:P-VBN:T-s[pss]\\np out:S-out:P-IN:T-s\\np\\(s\\np)/pp of:S-of:P-IN:T-pp/np the:S-the:P-DT:T-np/n second:S-second:P-JJ:T-n/n world:S-world:P-NN:T-n/n war:S-war:P-NN:T-n ,:S-,:P-,:T-punct[,] and:S-and:P-CC:T-s[em]$\\(s[em]$)\\punct[,]/(s[em]$) that:S-that:P-DT:T-s[em]/s[dcl] france:S-france:P-NN:T-n and:S-and:P-CC:T-np\\np/np the:S-the:P-DT:T-np/n other:S-other:P-JJ:T-n/n permanent:S-permanent:P-JJ:T-n/n members:S-member:P-NNS:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n council:S-council:P-NN:T-n were:S-be:P-VBD:T-s[dcl]\\np/np victors:S-victors:P-NNS:T-n in:S-in:P-IN:T-s\\np\\(s\\np)/np that:S-that:P-DT:T-np/n war:S-war:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nmoore said the australian military attache to jakarta will conduct free investigations after he arrives in east timor tomorrow .\nw9:s[b]\\np/np:VB:conduct w18:s\\np\\(s\\np):NN:tomorrow w8:s[dcl]\\np/(s[b]\\np):MD:will w17:n:NN:timor w7:n:NNP:jakarta w16:n/n:JJ:east w6:np\\np/np:TO:to w15:s\\np\\(s\\np)/np:IN:in w5:n:NN:attache w14:s[dcl]\\np:VBZ:arrive w4:n/n:JJ:military w13:np:PRP:he w3:n/n:JJ:australian w12:s\\np\\(s\\np)/s[dcl]:IN:after w2:np/n:DT:the w11:n:NNS:investigation w1:s[dcl]\\np/s[dcl]:VBD:say w10:n/n:JJ:free w0:n:NN:moore\nmoore:S-moore:P-NN:T-n said:S-say:P-VBD:T-s[dcl]\\np/s[dcl] the:S-the:P-DT:T-np/n australian:S-australian:P-JJ:T-n/n military:S-military:P-JJ:T-n/n attache:S-attache:P-NN:T-n to:S-to:P-TO:T-np\\np/np jakarta:S-jakarta:P-NNP:T-n will:S-will:P-MD:T-s[dcl]\\np/(s[b]\\np) conduct:S-conduct:P-VB:T-s[b]\\np/np free:S-free:P-JJ:T-n/n investigations:S-investigation:P-NNS:T-n after:S-after:P-IN:T-s\\np\\(s\\np)/s[dcl] he:S-he:P-PRP:T-np arrives:S-arrive:P-VBZ:T-s[dcl]\\np in:S-in:P-IN:T-s\\np\\(s\\np)/np east:S-east:P-JJ:T-n/n timor:S-timor:P-NN:T-n tomorrow:S-tomorrow:P-NN:T-s\\np\\(s\\np) .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nmoore said , after the australian military attache to jakarta arrives in east timor tomorrow , he can conduct investigations freely in the area .\nw19:n:NNS:investigation w9:n:NN:jakarta w18:s[b]\\np/np:VB:conduct w8:np\\np/np:TO:to w7:n:NN:attache w17:s[dcl]\\np/(s[b]\\np):MD:can w16:np:PRP:he w6:n/n:JJ:military w5:n/n:JJ:australian w14:s\\np\\(s\\np):NN:tomorrow w4:np/n:DT:the w13:n:NN:timor w3:s/s/s[dcl]:IN:after w12:n/n:JJ:east w11:s\\np\\(s\\np)/np:IN:in w1:s[dcl]\\np/s[dcl]/punct[,]:VBD:say w10:s[dcl]\\np:VBZ:arrive w0:n:NN:moore w23:n:NN:area w22:np/n:DT:the w21:s\\np\\(s\\np)/np:IN:in w20:s\\np\\(s\\np):RB:freely\nmoore:S-moore:P-NN:T-n said:S-say:P-VBD:T-s[dcl]\\np/s[dcl]/punct[,] ,:S-,:P-,:T-punct[,] after:S-after:P-IN:T-s/s/s[dcl] the:S-the:P-DT:T-np/n australian:S-australian:P-JJ:T-n/n military:S-military:P-JJ:T-n/n attache:S-attache:P-NN:T-n to:S-to:P-TO:T-np\\np/np jakarta:S-jakarta:P-NN:T-n arrives:S-arrive:P-VBZ:T-s[dcl]\\np in:S-in:P-IN:T-s\\np\\(s\\np)/np east:S-east:P-JJ:T-n/n timor:S-timor:P-NN:T-n tomorrow:S-tomorrow:P-NN:T-s\\np\\(s\\np) ,:S-,:P-,:T-s/s\\(s/s) he:S-he:P-PRP:T-np can:S-can:P-MD:T-s[dcl]\\np/(s[b]\\np) conduct:S-conduct:P-VB:T-s[b]\\np/np investigations:S-investigation:P-NNS:T-n freely:S-freely:P-RB:T-s\\np\\(s\\np) in:S-in:P-IN:T-s\\np\\(s\\np)/np the:S-the:P-DT:T-np/n area:S-area:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nat that time , jiang zeming indicated that china would consider joining the missile technology control regime . the indication has been taken as the principle result of the meeting between the two state heads .\nw33:n:NNS:head w32:n/n:NN:state w31:n/n:CD:two w30:np/n:DT:the w9:s[dcl]\\np/(s[b]\\np):MD:would w19:s[dcl]\\np/(s[pt]\\np):VBZ:PERF w18:n:NN:indication w8:n:NN:china w17:np/n:DT:the w6:s[dcl]\\np/s[em]:VBD:indicate w16:n:NN:regime w5:n:VBG:zeming w15:n/n:NN:control w14:n/n:NN:technology w4:n/n:VBG:jiang w13:n/n:NN:missile w2:n:NN:time w12:np/n:DT:the w11:s[ng]\\np/np:VBG:join w1:np/n:DT:that w10:s[b]\\np/np:VB:consider w0:s/s/np:IN:at w29:np\\np/np:IN:between w28:n:NN:meeting w27:np/n:DT:the w26:np\\np/np:IN:of w25:n:NN:result w24:n:NN:principle w23:np/n:DT:the w22:pp/np:IN:as w21:s[pss]\\np/pp:VBN:take w20:s[pt]\\np/(s[pss]\\np):VBN:PASS\nat:S-at:P-IN:T-s/s/np that:S-that:P-DT:T-np/n time:S-time:P-NN:T-n ,:S-,:P-,:T-s/s\\(s/s) jiang:S-jiang:P-VBG:T-n/n zeming:S-zeming:P-VBG:T-n indicated:S-indicate:P-VBD:T-s[dcl]\\np/s[em] that:S-that:P-IN:T-s[em]/s[dcl] china:S-china:P-NN:T-n would:S-would:P-MD:T-s[dcl]\\np/(s[b]\\np) consider:S-consider:P-VB:T-s[b]\\np/np joining:S-join:P-VBG:T-s[ng]\\np/np the:S-the:P-DT:T-np/n missile:S-missile:P-NN:T-n/n technology:S-technology:P-NN:T-n/n control:S-control:P-NN:T-n/n regime:S-regime:P-NN:T-n .:S-.:P-.:T-punct[.] the:S-the:P-DT:T-np/n indication:S-indication:P-NN:T-n has:S-have:P-VBZ:T-s[dcl]\\np/(s[pt]\\np) been:S-be:P-VBN:T-s[pt]\\np/(s[pss]\\np) taken:S-take:P-VBN:T-s[pss]\\np/pp as:S-as:P-IN:T-pp/np the:S-the:P-DT:T-np/n principle:S-principle:P-NN:T-n result:S-result:P-NN:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n meeting:S-meeting:P-NN:T-n between:S-between:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n two:S-two:P-CD:T-n/n state:S-state:P-NN:T-n/n heads:S-head:P-NNS:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\njiang zemin promised at that time that china would consider joining the missile technology control agreement , which was seen as one of the major achievements in the china-us summit meeting .\nw30:n:NN:meeting w19:s[pss]\\np/pp:VBN:see w9:s[b]\\np/(s[ng]\\np):VB:consider w18:s[dcl]\\np/(s[pss]\\np):VBD:PASS w8:s[dcl]\\np/(s[b]\\np):MD:would w7:n:NN:china w15:n:NN:agreement w5:n:NN:time w14:n/n:NN:control w4:np/n:DT:that w13:n/n:NN:technology w3:s\\np\\(s\\np)/np:IN:at w2:s[dcl]\\np/np:VBD:promise w12:n/n:NN:missile w11:np/n:DT:the w1:n:NNP:zemin w10:s[ng]\\np/np:VBG:join w0:n/n:NNP:jiang w29:n/n:NN:summit w28:n/n:JJ:china-us w27:np/n:DT:the w26:np\\np/np:IN:in w25:n:NNS:achievement w24:n/n:JJ:major w23:np/n:DT:the w22:np\\np/np:IN:of w21:n:CD:one w20:pp/np:IN:as\njiang:S-jiang:P-NNP:T-n/n zemin:S-zemin:P-NNP:T-n promised:S-promise:P-VBD:T-s[dcl]\\np/np at:S-at:P-IN:T-s\\np\\(s\\np)/np that:S-that:P-DT:T-np/n time:S-time:P-NN:T-n that:S-that:P-IN:T-np\\np/(s[dcl]/np) china:S-china:P-NN:T-n would:S-would:P-MD:T-s[dcl]\\np/(s[b]\\np) consider:S-consider:P-VB:T-s[b]\\np/(s[ng]\\np) joining:S-join:P-VBG:T-s[ng]\\np/np the:S-the:P-DT:T-np/n missile:S-missile:P-NN:T-n/n technology:S-technology:P-NN:T-n/n control:S-control:P-NN:T-n/n agreement:S-agreement:P-NN:T-n ,:S-,:P-,:T-punct[,] which:S-which:P-WDT:T-np\\np\\punct[,]/(s[dcl]\\np) was:S-be:P-VBD:T-s[dcl]\\np/(s[pss]\\np) seen:S-see:P-VBN:T-s[pss]\\np/pp as:S-as:P-IN:T-pp/np one:S-one:P-CD:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n major:S-major:P-JJ:T-n/n achievements:S-achievement:P-NNS:T-n in:S-in:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n china-us:S-china-us:P-JJ:T-n/n summit:S-summit:P-NN:T-n/n meeting:S-meeting:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nmore than 90% of china 's people centralize on one third of its territory .\nw9:n/n:CD:one w8:pp/np:IN:on w7:s[dcl]\\np/pp:VBP:centralize w6:n:NNS:people w4:n:NN:china w3:np\\np/np:IN:of w2:n:CD:90% w1:n/n\\(s[adj]\\np):IN:than w0:s[adj]\\np:JJR:more w13:n:NN:territory w12:np/n:PRP$:its w11:np\\np/np:IN:of w10:n:NN:third\nmore:S-more:P-JJR:T-s[adj]\\np than:S-than:P-IN:T-n/n\\(s[adj]\\np) 90%:S-90%:P-CD:T-n of:S-of:P-IN:T-np\\np/np china:S-china:P-NN:T-n 's:S-'s:P-POS:T-np/n\\np people:S-people:P-NNS:T-n centralize:S-centralize:P-VBP:T-s[dcl]\\np/pp on:S-on:P-IN:T-pp/np one:S-one:P-CD:T-n/n third:S-third:P-NN:T-n of:S-of:P-IN:T-np\\np/np its:S-its:P-PRP$:T-np/n territory:S-territory:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nmore than 90 percent of the population is compacted into one-third of the country 's territory .\nw9:pp/np:IN:into w8:s[pss]\\np/pp:VBN:compact w7:s[dcl]\\np/(s[pss]\\np):VBZ:PASS w6:n:NN:population w5:np/n:DT:the w4:np\\np/np:IN:of w3:n:NN:percent w2:n/n:CD:90 w1:n/n/(n/n)\\(s[adj]\\np):IN:than w15:n:NN:territory w0:s[adj]\\np:JJR:more w13:n:NN:country w12:np/n:DT:the w11:np\\np/np:IN:of w10:n:NN:one-third\nmore:S-more:P-JJR:T-s[adj]\\np than:S-than:P-IN:T-n/n/(n/n)\\(s[adj]\\np) 90:S-90:P-CD:T-n/n percent:S-percent:P-NN:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n population:S-population:P-NN:T-n is:S-be:P-VBZ:T-s[dcl]\\np/(s[pss]\\np) compacted:S-compact:P-VBN:T-s[pss]\\np/pp into:S-into:P-IN:T-pp/np one-third:S-one-third:P-NN:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n country:S-country:P-NN:T-n 's:S-'s:P-POS:T-np/n\\np territory:S-territory:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nwhat 's more , next year la nina will be just as bad as el nino .\nw9:s[b]\\np/(s[adj]\\np):VB:be w8:s[dcl]\\np/(s[b]\\np):MD:will w7:n:NN:nina w6:np/n:DT:la w5:s/s:NN:year w4:s/s/(s/s):JJ:next w2:n:JJR:more w1:s[dcl]\\np/np:VBZ:be w15:n:NN:nino w14:n/n:JJ:el w13:s[adj]\\np\\(s[adj]\\np)/np:IN:as w12:s[adj]\\np:JJ:bad w11:s[adj]\\np/(s[adj]\\np):RB:as w10:s[adj]\\np/(s[adj]\\np):RB:just\nwhat:S-what:P-WP:T-np/(s[dcl]\\np) 's:S-be:P-VBZ:T-s[dcl]\\np/np more:S-more:P-JJR:T-n ,:S-,:P-,:T-s/s\\np next:S-next:P-JJ:T-s/s/(s/s) year:S-year:P-NN:T-s/s la:S-la:P-DT:T-np/n nina:S-nina:P-NN:T-n will:S-will:P-MD:T-s[dcl]\\np/(s[b]\\np) be:S-be:P-VB:T-s[b]\\np/(s[adj]\\np) just:S-just:P-RB:T-s[adj]\\np/(s[adj]\\np) as:S-as:P-RB:T-s[adj]\\np/(s[adj]\\np) bad:S-bad:P-JJ:T-s[adj]\\np as:S-as:P-IN:T-s[adj]\\np\\(s[adj]\\np)/np el:S-el:P-JJ:T-n/n nino:S-nino:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nand la nina will put on a frightening display next year no less devastating than that of el nino .\nw9:np\\np/(np\\np):JJ:next w18:n:NN:nino w8:n:NN:display w17:n/n:JJ:el w7:n/n:JJ:frightening w16:np\\np/np:IN:of w6:np/n:DT:a w15:np:DT:that w5:pp/np:IN:on w14:s[adj]\\np\\(s[adj]\\np)/np:IN:than w4:s[b]\\np/pp:VB:put w13:s[adj]\\np:JJ:devastating w3:s[dcl]\\np/(s[b]\\np):MD:will w12:s[adj]\\np/(s[adj]\\np):RBR:less w2:n:NN:nina w11:s[adj]\\np/(s[adj]\\np):RB:no w1:np/n:DT:la w10:np\\np:NN:year w0:s/s:CC:and\nand:S-and:P-CC:T-s/s la:S-la:P-DT:T-np/n nina:S-nina:P-NN:T-n will:S-will:P-MD:T-s[dcl]\\np/(s[b]\\np) put:S-put:P-VB:T-s[b]\\np/pp on:S-on:P-IN:T-pp/np a:S-a:P-DT:T-np/n frightening:S-frightening:P-JJ:T-n/n display:S-display:P-NN:T-n next:S-next:P-JJ:T-np\\np/(np\\np) year:S-year:P-NN:T-np\\np no:S-no:P-RB:T-s[adj]\\np/(s[adj]\\np) less:S-less:P-RBR:T-s[adj]\\np/(s[adj]\\np) devastating:S-devastating:P-JJ:T-s[adj]\\np than:S-than:P-IN:T-s[adj]\\np\\(s[adj]\\np)/np that:S-that:P-DT:T-np of:S-of:P-IN:T-np\\np/np el:S-el:P-JJ:T-n/n nino:S-nino:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nrussian prime minister on domestic economy\nw2:n:NN:minister w1:n/n:JJ:prime w0:n/n:JJ:russian w5:n:NN:economy w4:n/n:JJ:domestic w3:np\\np/np:IN:on\nrussian:S-russian:P-JJ:T-n/n prime:S-prime:P-JJ:T-n/n minister:S-minister:P-NN:T-n on:S-on:P-IN:T-np\\np/np domestic:S-domestic:P-JJ:T-n/n economy:S-economy:P-NN:T-n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nrussian prime minister talked about russian economic situation .\nw2:n:NN:minister w1:n/n:JJ:prime w0:n/n:JJ:russian w7:n:NN:situation w6:n/n:JJ:economic w5:n/n:JJ:russian w4:pp/np:IN:about w3:s[dcl]\\np/pp:VBD:talk\nrussian:S-russian:P-JJ:T-n/n prime:S-prime:P-JJ:T-n/n minister:S-minister:P-NN:T-n talked:S-talk:P-VBD:T-s[dcl]\\np/pp about:S-about:P-IN:T-pp/np russian:S-russian:P-JJ:T-n/n economic:S-economic:P-JJ:T-n/n situation:S-situation:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nnowadays , there appeared a lot of enterprise groups with high technology and strong stamina .\nw9:np\\np/np:IN:with w8:n:NNS:group w7:n/n:NN:enterprise w6:s\\np\\(s\\np)\\(s\\np\\(s\\np))/np:IN:of w5:n:NN:lot w4:s\\np\\(s\\np)/n:DT:a w3:s[dcl]\\np:VBD:appear w0:s/s:RB:nowadays w14:n:NN:stamina w13:n/n:JJ:strong w12:n\\n/n:CC:and w11:n:NN:technology w10:n/n:JJ:high\nnowadays:S-nowadays:P-RB:T-s/s ,:S-,:P-,:T-s/s\\(s/s) there:S-there:P-EX:T-np[thr] appeared:S-appear:P-VBD:T-s[dcl]\\np a:S-a:P-DT:T-s\\np\\(s\\np)/n lot:S-lot:P-NN:T-n of:S-of:P-IN:T-s\\np\\(s\\np)\\(s\\np\\(s\\np))/np enterprise:S-enterprise:P-NN:T-n/n groups:S-group:P-NNS:T-n with:S-with:P-IN:T-np\\np/np high:S-high:P-JJ:T-n/n technology:S-technology:P-NN:T-n and:S-and:P-CC:T-n\\n/n strong:S-strong:P-JJ:T-n/n stamina:S-stamina:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nnowadays , there appeared a lot of enterprise groups with high technology and strong stamina .\nw9:np\\np/np:IN:with w8:n:NNS:group w7:n/n:NN:enterprise w6:s\\np\\(s\\np)\\(s\\np\\(s\\np))/np:IN:of w5:n:NN:lot w4:s\\np\\(s\\np)/n:DT:a w3:s[dcl]\\np:VBD:appear w0:s/s:RB:nowadays w14:n:NN:stamina w13:n/n:JJ:strong w12:n\\n/n:CC:and w11:n:NN:technology w10:n/n:JJ:high\nnowadays:S-nowadays:P-RB:T-s/s ,:S-,:P-,:T-s/s\\(s/s) there:S-there:P-EX:T-np[thr] appeared:S-appear:P-VBD:T-s[dcl]\\np a:S-a:P-DT:T-s\\np\\(s\\np)/n lot:S-lot:P-NN:T-n of:S-of:P-IN:T-s\\np\\(s\\np)\\(s\\np\\(s\\np))/np enterprise:S-enterprise:P-NN:T-n/n groups:S-group:P-NNS:T-n with:S-with:P-IN:T-np\\np/np high:S-high:P-JJ:T-n/n technology:S-technology:P-NN:T-n and:S-and:P-CC:T-n\\n/n strong:S-strong:P-JJ:T-n/n stamina:S-stamina:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nthe gnp created by the village and township enterprises occupied about 1\\/3 of the total gnp in fujian province .\nw9:s[dcl]\\np/np:VBD:occupy w8:n:NNS:enterprise w18:n:NN:province w17:n/n:JJ:fujian w7:n/n:NN:township h1:n\\(n/n)/n:CC:has-rel w16:np\\np/np:IN:in w6:n\\(n/n)/n:CC:and w15:n:NN:gnp w5:n/n:NN:village w14:n/n:JJ:total w4:np/n:DT:the w13:np/n:DT:the w3:s\\np\\(s\\np)/np:IN:by w12:np\\np/np:IN:of w2:s[pss]\\np:VBN:create w11:n:CD:1\\/3 w1:n:NN:gnp w10:n/n:IN:about w0:np/n:DT:the\nthe:S-the:P-DT:T-np/n gnp:S-gnp:P-NN:T-n created:S-create:P-VBN:T-s[pss]\\np by:S-by:P-IN:T-s\\np\\(s\\np)/np the:S-the:P-DT:T-np/n village:S-village:P-NN:T-n/n and:S-and:P-CC:T-n\\(n/n)/n township:S-township:P-NN:T-n/n enterprises:S-enterprise:P-NNS:T-n occupied:S-occupy:P-VBD:T-s[dcl]\\np/np about:S-about:P-IN:T-n/n 1\\/3:S-1\\/3:P-CD:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n total:S-total:P-JJ:T-n/n gnp:S-gnp:P-NN:T-n in:S-in:P-IN:T-np\\np/np fujian:S-fujian:P-JJ:T-n/n province:S-province:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nthe gross national product of the industry of villages and towns made up 1\\/3 of the gnp of fujian province .\nw9:np\\np/np:CC:and w19:n:NN:province w18:n/n:JJ:fujian w8:n:NNS:village w17:np\\np/np:IN:of w7:np\\np/np:IN:of w16:n:NN:gnp w6:n:NN:industry w15:np/n:DT:the w5:np/n:DT:the w14:np\\np/np:IN:of w4:np\\np/np:IN:of w13:n:CD:1\\/3 w3:n:NN:product w12:s\\np\\(s\\np):RP:up w2:n/n:JJ:national w11:s[dcl]\\np/np:VBD:make w1:n/n:JJ:gross w10:n:NNS:town w0:np/n:DT:the\nthe:S-the:P-DT:T-np/n gross:S-gross:P-JJ:T-n/n national:S-national:P-JJ:T-n/n product:S-product:P-NN:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n industry:S-industry:P-NN:T-n of:S-of:P-IN:T-np\\np/np villages:S-village:P-NNS:T-n and:S-and:P-CC:T-np\\np/np towns:S-town:P-NNS:T-n made:S-make:P-VBD:T-s[dcl]\\np/np up:S-up:P-RP:T-s\\np\\(s\\np) 1\\/3:S-1\\/3:P-CD:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n gnp:S-gnp:P-NN:T-n of:S-of:P-IN:T-np\\np/np fujian:S-fujian:P-JJ:T-n/n province:S-province:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nboth parties expressed satisfaction for the friendly cooperation between the two armies in the past 30 years .\nw9:np/n:DT:the w8:np\\np/np:IN:between w7:n:NN:cooperation w6:n/n:JJ:friendly w5:np/n:DT:the w4:np\\np/np:IN:for w3:n:NN:satisfaction w2:s[dcl]\\np/np:VBD:express w16:n:NNS:year w1:n:NNS:party w15:n/n:CD:30 w0:np/n:DT:both w14:n/n:JJ:past w13:np/n:DT:the w12:np\\np/np:IN:in w11:n:NNS:army w10:n/n:CD:two\nboth:S-both:P-DT:T-np/n parties:S-party:P-NNS:T-n expressed:S-express:P-VBD:T-s[dcl]\\np/np satisfaction:S-satisfaction:P-NN:T-n for:S-for:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n friendly:S-friendly:P-JJ:T-n/n cooperation:S-cooperation:P-NN:T-n between:S-between:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n two:S-two:P-CD:T-n/n armies:S-army:P-NNS:T-n in:S-in:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n past:S-past:P-JJ:T-n/n 30:S-30:P-CD:T-n/n years:S-year:P-NNS:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nboth parties expressed satisfaction for the friendly cooperation between the two armies in the past 30 years .\nw9:np/n:DT:the w8:np\\np/np:IN:between w7:n:NN:cooperation w6:n/n:JJ:friendly w5:np/n:DT:the w4:np\\np/np:IN:for w3:n:NN:satisfaction w2:s[dcl]\\np/np:VBD:express w16:n:NNS:year w1:n:NNS:party w15:n/n:CD:30 w0:np/n:DT:both w14:n/n:JJ:past w13:np/n:DT:the w12:np\\np/np:IN:in w11:n:NNS:army w10:n/n:CD:two\nboth:S-both:P-DT:T-np/n parties:S-party:P-NNS:T-n expressed:S-express:P-VBD:T-s[dcl]\\np/np satisfaction:S-satisfaction:P-NN:T-n for:S-for:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n friendly:S-friendly:P-JJ:T-n/n cooperation:S-cooperation:P-NN:T-n between:S-between:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n two:S-two:P-CD:T-n/n armies:S-army:P-NNS:T-n in:S-in:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n past:S-past:P-JJ:T-n/n 30:S-30:P-CD:T-n/n years:S-year:P-NNS:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nalso present at the seminar will be long yongtu , vice minister of china 's ministry of foreign trade and economic cooperation who will accompany rugerro to shanghai , and shen jueren , former chief negotiator of china in gatt negotiations .\n\n\nattendees will also include the vice minister of foreign trade & economic cooperation , yongtu long , who accompanied ruggiero to shanghai , and former chief representative for trade negotiations of the moftec .\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nxinhua news agency , nanjing , 16 december , by zhoufang the investment of foreign capital in agriculture of jiangsu increased .\n\n\nxinhua news agency , nanjing , dec. 16 . ( reporter zhou fang ) an increasing number of investments from foreign capital are being made on agriculture in jiangsu .\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nfrance ministry of foreign affairs said that leaders of iraq refused to co-operate with united nations and have caused this deeply regretted outcome .\nw19:np/n:DT:this w9:n:NNS:iraq w18:s[pt]\\np/np:VBN:cause w8:np\\np/np:IN:of w17:s[dcl]\\np/(s[pt]\\np):VBP:PERF w7:n:NNS:leader w16:s[dcl]$\\(s[dcl]$)/(s[dcl]$):CC:and w6:np/n:DT:that w15:n:NNS:nation w5:s[dcl]\\np/s[dcl]:VBD:say w14:n/n:JJ:united w4:n:NNS:affair w13:pp/np:IN:with w3:n/n:JJ:foreign w12:s[b]\\np/pp:VB:co-operate w2:np\\np/np:IN:of w1:n:NN:ministry w10:s[dcl]\\np/(s[to]\\np):VBD:refuse w0:n/n:NN:france w22:n:NN:outcome w21:n/n:JJ:regretted w20:n/n/(n/n):RB:deeply\nfrance:S-france:P-NN:T-n/n ministry:S-ministry:P-NN:T-n of:S-of:P-IN:T-np\\np/np foreign:S-foreign:P-JJ:T-n/n affairs:S-affair:P-NNS:T-n said:S-say:P-VBD:T-s[dcl]\\np/s[dcl] that:S-that:P-DT:T-np/n leaders:S-leader:P-NNS:T-n of:S-of:P-IN:T-np\\np/np iraq:S-iraq:P-NNS:T-n refused:S-refuse:P-VBD:T-s[dcl]\\np/(s[to]\\np) to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) co-operate:S-co-operate:P-VB:T-s[b]\\np/pp with:S-with:P-IN:T-pp/np united:S-united:P-JJ:T-n/n nations:S-nation:P-NNS:T-n and:S-and:P-CC:T-s[dcl]$\\(s[dcl]$)/(s[dcl]$) have:S-have:P-VBP:T-s[dcl]\\np/(s[pt]\\np) caused:S-cause:P-VBN:T-s[pt]\\np/np this:S-this:P-DT:T-np/n deeply:S-deeply:P-RB:T-n/n/(n/n) regretted:S-regretted:P-JJ:T-n/n outcome:S-outcome:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nthe french foreign ministry said that because iraqi leader refuses to cooperate with the un caused this sad events .\nw9:s[dcl]\\np/(s[to]\\np):VBZ:refuse w8:n:NN:leader w7:n/n:JJ:iraqi w6:s/s/s[dcl]:IN:because w4:s[dcl]\\np/s[em]:VBD:say w18:n:NNS:event w3:n:NN:ministry w17:n/n:JJ:sad w2:n/n:JJ:foreign w16:np/n:DT:this w1:n/n:JJ:french w15:s[dcl]\\pp/np:VBD:cause w0:np/n:DT:the w14:n:NN:un w13:np/n:DT:the w12:pp/np:IN:with w11:s[b]\\np:VB:cooperate\nthe:S-the:P-DT:T-np/n french:S-french:P-JJ:T-n/n foreign:S-foreign:P-JJ:T-n/n ministry:S-ministry:P-NN:T-n said:S-say:P-VBD:T-s[dcl]\\np/s[em] that:S-that:P-IN:T-s[em]/s[dcl] because:S-because:P-IN:T-s/s/s[dcl] iraqi:S-iraqi:P-JJ:T-n/n leader:S-leader:P-NN:T-n refuses:S-refuse:P-VBZ:T-s[dcl]\\np/(s[to]\\np) to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) cooperate:S-cooperate:P-VB:T-s[b]\\np with:S-with:P-IN:T-pp/np the:S-the:P-DT:T-np/n un:S-un:P-NN:T-n caused:S-cause:P-VBD:T-s[dcl]\\pp/np this:S-this:P-DT:T-np/n sad:S-sad:P-JJ:T-n/n events:S-event:P-NNS:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nduring the preliminaries , lan wei , a highly-skilled twenty-six year-old native of guangdong , scored 355.35 for first place .\nw9:n/n:JJ:twenty-six w8:n/n:JJ:highly-skilled w7:np/n:DT:a w5:n:NNS:wei w19:n:NN:place w4:n/n:JJ:lan w18:n/n:JJ:first w2:n:NNS:preliminaries w17:s\\np\\(s\\np)/np:IN:for w16:s\\np\\(s\\np):CD:355.35 w1:np/n:DT:the w15:s[dcl]\\np:VBD:score w0:s/s/np:IN:during w13:n:NN:guangdong w12:np\\np/np:IN:of w11:n:NN:native w10:n/n:JJ:year-old\nduring:S-during:P-IN:T-s/s/np the:S-the:P-DT:T-np/n preliminaries:S-preliminaries:P-NNS:T-n ,:S-,:P-,:T-s/s\\(s/s) lan:S-lan:P-JJ:T-n/n wei:S-wei:P-NNS:T-n ,:S-,:P-,:T-np\\np/punct[,]/np a:S-a:P-DT:T-np/n highly-skilled:S-highly-skilled:P-JJ:T-n/n twenty-six:S-twenty-six:P-JJ:T-n/n year-old:S-year-old:P-JJ:T-n/n native:S-native:P-NN:T-n of:S-of:P-IN:T-np\\np/np guangdong:S-guangdong:P-NN:T-n ,:S-,:P-,:T-punct[,] scored:S-score:P-VBD:T-s[dcl]\\np 355.35:S-355.35:P-CD:T-s\\np\\(s\\np) for:S-for:P-IN:T-s\\np\\(s\\np)/np first:S-first:P-JJ:T-n/n place:S-place:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nin the qualifiers , 26-year-old lan wei from guangdong displayed excellent skill , and ranked no. 1 with a total score of 355.3 .\nw9:s[pss]\\np:VBN:display w19:n/n:JJ:total w18:np/n:DT:a w8:s[ng]\\np/(s[pss]\\np):VBG:guangdong w17:s\\np\\(s\\np)/np:IN:with w7:np\\np/(s[ng]\\np):IN:from w16:n[num]:CD:1 w6:n:NNS:wei w15:s\\np\\(s\\np)/n[num]:NNP:no. w5:n/n:JJ:lan w14:s[pss]\\np:VBD:rank w4:n/n:JJ:26-year-old w13:s[pss]$\\(s[pss]$)\\punct[,]/(s[pss]$):CC:and w3:np\\np/np:,:, w2:n:NNS:qualifiers w11:s\\np\\(s\\np):NN:skill w1:np/n:DT:the w10:s\\np\\(s\\np)/(s\\np\\(s\\np)):JJ:excellent w0:s/s/np:IN:in w22:n:CD:355.3 w21:np\\np/np:IN:of w20:n:NN:score\nin:S-in:P-IN:T-s/s/np the:S-the:P-DT:T-np/n qualifiers:S-qualifiers:P-NNS:T-n ,:S-,:P-,:T-np\\np/np 26-year-old:S-26-year-old:P-JJ:T-n/n lan:S-lan:P-JJ:T-n/n wei:S-wei:P-NNS:T-n from:S-from:P-IN:T-np\\np/(s[ng]\\np) guangdong:S-guangdong:P-VBG:T-s[ng]\\np/(s[pss]\\np) displayed:S-display:P-VBN:T-s[pss]\\np excellent:S-excellent:P-JJ:T-s\\np\\(s\\np)/(s\\np\\(s\\np)) skill:S-skill:P-NN:T-s\\np\\(s\\np) ,:S-,:P-,:T-punct[,] and:S-and:P-CC:T-s[pss]$\\(s[pss]$)\\punct[,]/(s[pss]$) ranked:S-rank:P-VBD:T-s[pss]\\np no.:S-no.:P-NNP:T-s\\np\\(s\\np)/n[num] 1:S-1:P-CD:T-n[num] with:S-with:P-IN:T-s\\np\\(s\\np)/np a:S-a:P-DT:T-np/n total:S-total:P-JJ:T-n/n score:S-score:P-NN:T-n of:S-of:P-IN:T-np\\np/np 355.3:S-355.3:P-CD:T-n .:S-.:P-.:T-sent\\(s/s)\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nnanjing , december 16 ( xinhua ) foreign fund put into jiangsu 's agricultural sector is increasing\nw9:s[pss]\\np/pp:VBD:put w8:n:NN:fund w7:n/n:JJ:foreign w6:np/n:DT:rrb w5:np\\np\\(np\\np)/np:IN:xinhua w4:np\\np:NNS:lrb w3:np\\np/(np\\np):CD:16 w2:n:NN:december w16:s[ng]\\np:VBG:increase w15:s[dcl]\\np/(s[ng]\\np):VBZ:PROG w0:s/s:NN:nanjing w14:n:NN:sector w13:n/n:JJ:agricultural w11:n:NNP:jiangsu w10:pp/np:IN:into\nnanjing:S-nanjing:P-NN:T-s/s ,:S-,:P-,:T-s/s\\(s/s) december:S-december:P-NN:T-n 16:S-16:P-CD:T-np\\np/(np\\np) (:S-(:P-NNS:T-np\\np xinhua:S-xinhua:P-IN:T-np\\np\\(np\\np)/np ):S-):P-DT:T-np/n foreign:S-foreign:P-JJ:T-n/n fund:S-fund:P-NN:T-n put:S-put:P-VBD:T-s[pss]\\np/pp into:S-into:P-IN:T-pp/np jiangsu:S-jiangsu:P-NNP:T-n 's:S-'s:P-POS:T-np/n\\np agricultural:S-agricultural:P-JJ:T-n/n sector:S-sector:P-NN:T-n is:S-be:P-VBZ:T-s[dcl]\\np/(s[ng]\\np) increasing:S-increase:P-VBG:T-s[ng]\\np\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nxinhua news agency , nanjing , dec. 16 ( reporter zhou fang ) - foreign investment in jiangsu 's agriculture is on the rise .\nw19:s[dcl]\\np/pp:VBZ:be w9:n/n:NN:reporter w18:n:NN:agriculture w8:n/n:NN:lrb w7:n/n:CD:16 w16:n:NNP:jiangsu w6:s[dcl]\\np/np:VBZ:dec. w15:np\\np/np:IN:in w14:n:NN:investment w4:n:NN:nanjing w13:n/n:JJ:foreign w12:n:SYM:rrb w2:n:NN:agency w11:n/n:VBG:fang w1:n/n:NN:news w10:n/n:NN:zhou w0:np/n:DT:xinhua w22:n:NN:rise w21:np/n:DT:the w20:pp/np:IN:on\nxinhua:S-xinhua:P-DT:T-np/n news:S-news:P-NN:T-n/n agency:S-agency:P-NN:T-n ,:S-,:P-,:T-np\\np/punct[,]/np nanjing:S-nanjing:P-NN:T-n ,:S-,:P-,:T-punct[,] dec.:S-dec.:P-VBZ:T-s[dcl]\\np/np 16:S-16:P-CD:T-n/n (:S-(:P-NN:T-n/n reporter:S-reporter:P-NN:T-n/n zhou:S-zhou:P-NN:T-n/n fang:S-fang:P-VBG:T-n/n ):S-):P-SYM:T-n -:S--:P--:T-punct[-] foreign:S-foreign:P-JJ:T-n/n investment:S-investment:P-NN:T-n in:S-in:P-IN:T-np\\np/np jiangsu:S-jiangsu:P-NNP:T-n 's:S-'s:P-POS:T-np/n\\np agriculture:S-agriculture:P-NN:T-n is:S-be:P-VBZ:T-s[dcl]\\np/pp on:S-on:P-IN:T-pp/np the:S-the:P-DT:T-np/n rise:S-rise:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nsomeone connected global warming to the phenomenon of el nino .\nw9:n:NN:nino w8:n/n:JJ:el w7:np\\np/np:IN:of w6:n:NN:phenomenon w5:np/n:DT:the w4:np\\np/np:TO:to w3:n:NN:warming w2:n/n:JJ:global w1:n/n:JJ:connected w0:np/n:DT:someone\nsomeone:S-someone:P-DT:T-np/n connected:S-connected:P-JJ:T-n/n global:S-global:P-JJ:T-n/n warming:S-warming:P-NN:T-n to:S-to:P-TO:T-np\\np/np the:S-the:P-DT:T-np/n phenomenon:S-phenomenon:P-NN:T-n of:S-of:P-IN:T-np\\np/np el:S-el:P-JJ:T-n/n nino:S-nino:P-NN:T-n .:S-.:P-.:T-sent\\np\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nsome linked global warming with el nino .\nw2:n/n:JJ:global w1:n/n:VBN:link w0:np/n:DT:some w6:n:NN:nino w5:n/n:JJ:el w4:np\\np/np:IN:with w3:n:NN:warming\nsome:S-some:P-DT:T-np/n linked:S-link:P-VBN:T-n/n global:S-global:P-JJ:T-n/n warming:S-warming:P-NN:T-n with:S-with:P-IN:T-np\\np/np el:S-el:P-JJ:T-n/n nino:S-nino:P-NN:T-n .:S-.:P-.:T-sent\\np\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nxinhua news agency , beijing , january 16th , by xintang xu and yuhong qian- the president of the bank of china , xianglong dai said that , china will continue implementing the financial opening up policy .\nw34:n:NN:policy w33:np\\np/np:IN:up w32:n:NN:opening w31:n/n:JJ:financial w30:np/n:DT:the w9:s[ng]\\np/np:VBG:xintang w19:np\\np/np:IN:of w18:n:NN:bank w8:s\\np\\(s\\np)/(s[ng]\\np):IN:by w17:np/n:DT:the w16:np\\np/np:IN:of w6:s\\np\\(s\\np):JJ:16th w15:n:NN:president w5:s\\np\\(s\\np)/(s\\np\\(s\\np)):JJ:january w4:s[ng]\\np:VBG:beijing w14:np/n:DT:the w13:s\\np\\(s\\np)/np:IN:qian- w12:n:NN:yuhong w2:n:NN:agency w11:n\\n/n:CC:and w1:n/n:NN:news w10:n:NN:xu w0:np/n:DT:xinhua w29:s[ng]\\np/np:VBG:implement w28:s[b]\\np/(s[ng]\\np):VB:continue w27:s[dcl]\\np/(s[b]\\np):MD:will w26:n:NN:china w24:s[dcl]\\np/s[em]:VBD:say w23:s\\np/(s\\np):NN:dai w22:s\\np/(s\\np)/(s\\np/(s\\np)):JJ:xianglong w20:n:NN:china\nxinhua:S-xinhua:P-DT:T-np/n news:S-news:P-NN:T-n/n agency:S-agency:P-NN:T-n ,:S-,:P-,:T-s\\np/(s\\np)/punct[,]/(s\\np) beijing:S-beijing:P-VBG:T-s[ng]\\np ,:S-,:P-,:T-punct[,] january:S-january:P-JJ:T-s\\np\\(s\\np)/(s\\np\\(s\\np)) 16th:S-16th:P-JJ:T-s\\np\\(s\\np) ,:S-,:P-,:T-s\\np\\(s\\np)/(s\\np\\(s\\np)) by:S-by:P-IN:T-s\\np\\(s\\np)/(s[ng]\\np) xintang:S-xintang:P-VBG:T-s[ng]\\np/np xu:S-xu:P-NN:T-n and:S-and:P-CC:T-n\\n/n yuhong:S-yuhong:P-NN:T-n qian-:S-qian-:P-IN:T-s\\np\\(s\\np)/np the:S-the:P-DT:T-np/n president:S-president:P-NN:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n bank:S-bank:P-NN:T-n of:S-of:P-IN:T-np\\np/np china:S-china:P-NN:T-n ,:S-,:P-,:T-punct[,] xianglong:S-xianglong:P-JJ:T-s\\np/(s\\np)/(s\\np/(s\\np)) dai:S-dai:P-NN:T-s\\np/(s\\np) said:S-say:P-VBD:T-s[dcl]\\np/s[em] that:S-that:P-IN:T-s[em]/s[dcl] ,:S-,:P-,:T-punct[,] china:S-china:P-NN:T-n will:S-will:P-MD:T-s[dcl]\\np/(s[b]\\np) continue:S-continue:P-VB:T-s[b]\\np/(s[ng]\\np) implementing:S-implement:P-VBG:T-s[ng]\\np/np the:S-the:P-DT:T-np/n financial:S-financial:P-JJ:T-n/n opening:S-opening:P-NN:T-n up:S-up:P-IN:T-np\\np/np policy:S-policy:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nxinhua news agency , beijing , january 16th ( reporter : xu xingtang , qian yuhong ) . dai xianglong , the president of people 's bank of china , said that china will carry on with the open financial policy .\nw37:n/n:JJ:open w36:np/n:DT:the w35:s\\np\\(s\\np)/np:IN:with w34:s\\np\\(s\\np):RP:on w33:s[b]\\np:VB:carry w32:s[dcl]\\np/(s[b]\\np):MD:will w31:n:NN:china w9:n:NN:reporter w18:n:NNP:xianglong w8:n/n:NN:lrb w17:n/n:NNP:dai w7:n/n:JJ:16th w16:n:NN:rrb w6:n/n:JJ:january w15:n/n:JJ:yuhong w14:n/n:JJ:qian w4:s[ng]\\np/np:VBG:beijing w13:np\\np/np:,:, w2:n:NN:agency w12:n:NNP:xingtang w11:n/n:NNP:xu w1:n/n:NN:news w10:np\\np/np:IN:| w0:np/n:DT:xinhua w29:s[dcl]\\np/s[em]:VBD:say w27:n:NN:china w26:np\\np/np:IN:of w25:n:NN:bank w23:n:NNS:people w22:np\\np/np:IN:of w21:n:NN:president w20:np/n:DT:the w39:n:NN:policy w38:n/n:JJ:financial\nxinhua:S-xinhua:P-DT:T-np/n news:S-news:P-NN:T-n/n agency:S-agency:P-NN:T-n ,:S-,:P-,:T-s[dcl]\\np/(s[dcl]\\np) beijing:S-beijing:P-VBG:T-s[ng]\\np/np ,:S-,:P-,:T-s\\np\\(s\\np)/np january:S-january:P-JJ:T-n/n 16th:S-16th:P-JJ:T-n/n (:S-(:P-NN:T-n/n reporter:S-reporter:P-NN:T-n &#58;:S-&#58;:P-IN:T-np\\np/np xu:S-xu:P-NNP:T-n/n xingtang:S-xingtang:P-NNP:T-n ,:S-,:P-,:T-np\\np/np qian:S-qian:P-JJ:T-n/n yuhong:S-yuhong:P-JJ:T-n/n ):S-):P-NN:T-n .:S-.:P-.:T-punct[.] dai:S-dai:P-NNP:T-n/n xianglong:S-xianglong:P-NNP:T-n ,:S-,:P-,:T-np\\np/punct[,]/np the:S-the:P-DT:T-np/n president:S-president:P-NN:T-n of:S-of:P-IN:T-np\\np/np people:S-people:P-NNS:T-n 's:S-'s:P-POS:T-np/n\\np bank:S-bank:P-NN:T-n of:S-of:P-IN:T-np\\np/np china:S-china:P-NN:T-n ,:S-,:P-,:T-punct[,] said:S-say:P-VBD:T-s[dcl]\\np/s[em] that:S-that:P-DT:T-s[em]/s[dcl] china:S-china:P-NN:T-n will:S-will:P-MD:T-s[dcl]\\np/(s[b]\\np) carry:S-carry:P-VB:T-s[b]\\np on:S-on:P-RP:T-s\\np\\(s\\np) with:S-with:P-IN:T-s\\np\\(s\\np)/np the:S-the:P-DT:T-np/n open:S-open:P-JJ:T-n/n financial:S-financial:P-JJ:T-n/n policy:S-policy:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nthis kind of measure restrained the economic growth , leading to the rise of unemployment .\nw9:s[ng]\\np:VBG:lead w7:n:NN:growth w6:n/n:JJ:economic w5:np/n:DT:the w4:s[dcl]\\np/np:VBD:restrain w3:n:NN:measure w2:np\\np/np:IN:of w1:n:NN:kind w0:np/n:DT:this w14:n:NN:unemployment w13:np\\np/np:IN:of w12:n:NN:rise w11:np/n:DT:the w10:s\\np\\(s\\np)/np:TO:to\nthis:S-this:P-DT:T-np/n kind:S-kind:P-NN:T-n of:S-of:P-IN:T-np\\np/np measure:S-measure:P-NN:T-n restrained:S-restrain:P-VBD:T-s[dcl]\\np/np the:S-the:P-DT:T-np/n economic:S-economic:P-JJ:T-n/n growth:S-growth:P-NN:T-n ,:S-,:P-,:T-s\\np\\(s\\np)/(s\\np) leading:S-lead:P-VBG:T-s[ng]\\np to:S-to:P-TO:T-s\\np\\(s\\np)/np the:S-the:P-DT:T-np/n rise:S-rise:P-NN:T-n of:S-of:P-IN:T-np\\np/np unemployment:S-unemployment:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nsuch measures checked economic growth and caused unemployment to rise .\nw9:s[b]\\np:VB:rise w7:n:NN:unemployment w6:s[dcl]\\np/(s[to]\\np)/np:VBD:cause w5:s[dcl]$\\(s[dcl]$)/(s[dcl]$):CC:and w4:n:NN:growth w3:n/n:JJ:economic w2:s[dcl]\\np/np:VBD:check w1:n:NNS:measure w0:np/np:JJ:such\nsuch:S-such:P-JJ:T-np/np measures:S-measure:P-NNS:T-n checked:S-check:P-VBD:T-s[dcl]\\np/np economic:S-economic:P-JJ:T-n/n growth:S-growth:P-NN:T-n and:S-and:P-CC:T-s[dcl]$\\(s[dcl]$)/(s[dcl]$) caused:S-cause:P-VBD:T-s[dcl]\\np/(s[to]\\np)/np unemployment:S-unemployment:P-NN:T-n to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) rise:S-rise:P-VB:T-s[b]\\np .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nsiemens germany is currently cooperating with a local thai company , and constructing another 23km long electric railway project in bangkok .\nw19:s\\np\\(s\\np)/np:IN:in w9:n:NN:company w18:n:NN:project w8:n/n:NN:thai w17:n/n:NN:railway w7:n/n:JJ:local w16:n/n:JJ:electric w6:np/n:DT:a w15:n/n:JJ:long w5:pp/np:IN:with w14:n/n:JJ:23km w4:s[ng]\\np/pp:VBG:cooperate w13:np/n:DT:another w3:s\\np\\(s\\np):RB:currently w12:s[ng]\\np/np:VBG:construct w2:s[dcl]\\np/(s[ng]\\np):VBZ:PROG w11:s[ng]$\\(s[ng]$)\\punct[,]/(s[ng]$):CC:and w1:n:NN:germany w0:n/n:NNS:siemens w20:n:NN:bangkok\nsiemens:S-siemens:P-NNS:T-n/n germany:S-germany:P-NN:T-n is:S-be:P-VBZ:T-s[dcl]\\np/(s[ng]\\np) currently:S-currently:P-RB:T-s\\np\\(s\\np) cooperating:S-cooperate:P-VBG:T-s[ng]\\np/pp with:S-with:P-IN:T-pp/np a:S-a:P-DT:T-np/n local:S-local:P-JJ:T-n/n thai:S-thai:P-NN:T-n/n company:S-company:P-NN:T-n ,:S-,:P-,:T-punct[,] and:S-and:P-CC:T-s[ng]$\\(s[ng]$)\\punct[,]/(s[ng]$) constructing:S-construct:P-VBG:T-s[ng]\\np/np another:S-another:P-DT:T-np/n 23km:S-23km:P-JJ:T-n/n long:S-long:P-JJ:T-n/n electric:S-electric:P-JJ:T-n/n railway:S-railway:P-NN:T-n/n project:S-project:P-NN:T-n in:S-in:P-IN:T-s\\np\\(s\\np)/np bangkok:S-bangkok:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nthe german shermans company is currently working together with a local company constructing a 23 kilometer electronic train engineering project .\nw19:n:NN:project w9:np/n:DT:a w18:n/n:NN:engineering w8:s\\np\\(s\\np)/np:IN:with w17:n/n:NN:train w7:s\\np\\(s\\np):RB:together w6:s[ng]\\np:VBG:work w16:n/n:JJ:electronic w15:n/n:NN:kilometer w5:s\\np\\(s\\np):RB:currently w4:s[dcl]\\np/(s[ng]\\np):VBZ:PROG w14:n/n:CD:23 w13:np/n:DT:a w3:n:NN:company w12:s[ng]\\np/np:VBG:construct w2:n/n:NNS:shermans w11:n:NN:company w1:n/n:NN:german w10:n/n:JJ:local w0:np/n:DT:the\nthe:S-the:P-DT:T-np/n german:S-german:P-NN:T-n/n shermans:S-shermans:P-NNS:T-n/n company:S-company:P-NN:T-n is:S-be:P-VBZ:T-s[dcl]\\np/(s[ng]\\np) currently:S-currently:P-RB:T-s\\np\\(s\\np) working:S-work:P-VBG:T-s[ng]\\np together:S-together:P-RB:T-s\\np\\(s\\np) with:S-with:P-IN:T-s\\np\\(s\\np)/np a:S-a:P-DT:T-np/n local:S-local:P-JJ:T-n/n company:S-company:P-NN:T-n constructing:S-construct:P-VBG:T-s[ng]\\np/np a:S-a:P-DT:T-np/n 23:S-23:P-CD:T-n/n kilometer:S-kilometer:P-NN:T-n/n electronic:S-electronic:P-JJ:T-n/n train:S-train:P-NN:T-n/n engineering:S-engineering:P-NN:T-n/n project:S-project:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nhe said that the same resolve would lead to the success of the kyoto protocol .\nw9:np/n:DT:the w8:pp/np:TO:to w7:s[b]\\np/pp:VB:lead w6:s[dcl]\\np/(s[b]\\np):MD:would w5:n:NN:resolve w4:n/n:JJ:same w3:np/n:DT:the w1:s[dcl]\\np/s[em]:VBD:say w0:np:PRP:he w14:n:NN:protocol w13:n/n:NN:kyoto w12:np/n:DT:the w11:np\\np/np:IN:of w10:n:NN:success\nhe:S-he:P-PRP:T-np said:S-say:P-VBD:T-s[dcl]\\np/s[em] that:S-that:P-IN:T-s[em]/s[dcl] the:S-the:P-DT:T-np/n same:S-same:P-JJ:T-n/n resolve:S-resolve:P-NN:T-n would:S-would:P-MD:T-s[dcl]\\np/(s[b]\\np) lead:S-lead:P-VB:T-s[b]\\np/pp to:S-to:P-TO:T-pp/np the:S-the:P-DT:T-np/n success:S-success:P-NN:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n kyoto:S-kyoto:P-NN:T-n/n protocol:S-protocol:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nhe said that the same resolve would lead to the success of the kyoto protocol .\nw9:np/n:DT:the w8:pp/np:TO:to w7:s[b]\\np/pp:VB:lead w6:s[dcl]\\np/(s[b]\\np):MD:would w5:n:NN:resolve w4:n/n:JJ:same w3:np/n:DT:the w1:s[dcl]\\np/s[em]:VBD:say w0:np:PRP:he w14:n:NN:protocol w13:n/n:NN:kyoto w12:np/n:DT:the w11:np\\np/np:IN:of w10:n:NN:success\nhe:S-he:P-PRP:T-np said:S-say:P-VBD:T-s[dcl]\\np/s[em] that:S-that:P-IN:T-s[em]/s[dcl] the:S-the:P-DT:T-np/n same:S-same:P-JJ:T-n/n resolve:S-resolve:P-NN:T-n would:S-would:P-MD:T-s[dcl]\\np/(s[b]\\np) lead:S-lead:P-VB:T-s[b]\\np/pp to:S-to:P-TO:T-pp/np the:S-the:P-DT:T-np/n success:S-success:P-NN:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n kyoto:S-kyoto:P-NN:T-n/n protocol:S-protocol:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nhowever , the official who disclosed the above declined to elaborate what kind of missile technology is being exported and when china provided such technology to pakistan and iran .\nw19:np\\np\\(np\\np)/(np\\np):CC:and w18:s[pss]\\np:VBN:export w8:s[dcl]\\np/(s[to]\\np):VBD:decline w17:s[ng]\\np/(s[pss]\\np):VBG:PASS w7:n:NN:above w16:s[dcl]\\np/(s[ng]\\np):VBZ:PROG w6:np/n:DT:the w15:n:NN:technology w5:s[dcl]\\np/np:VBD:disclose w14:n/n:NN:missile w13:np\\np/np:IN:of w3:n:NN:official w12:n:NN:kind w2:np/n:DT:the w11:np/n:WP:what w10:s[b]\\np:VB:elaborate w0:s/s:RB:however w28:s[b]\\np:VB:iran w27:s[b]$\\(s[b]$)/(s[b]$):CC:and w26:s[b]\\np:VB:pakistan w24:n:NN:technology w23:np/np:JJ:such w22:s[dcl]\\np/np:VBD:provide w21:n:NN:china w20:np\\np/s[dcl]:WRB:when\nhowever:S-however:P-RB:T-s/s ,:S-,:P-,:T-s/s\\(s/s) the:S-the:P-DT:T-np/n official:S-official:P-NN:T-n who:S-who:P-WP:T-np\\np/(s[dcl]\\np) disclosed:S-disclose:P-VBD:T-s[dcl]\\np/np the:S-the:P-DT:T-np/n above:S-above:P-NN:T-n declined:S-decline:P-VBD:T-s[dcl]\\np/(s[to]\\np) to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) elaborate:S-elaborate:P-VB:T-s[b]\\np what:S-what:P-WP:T-np/n kind:S-kind:P-NN:T-n of:S-of:P-IN:T-np\\np/np missile:S-missile:P-NN:T-n/n technology:S-technology:P-NN:T-n is:S-be:P-VBZ:T-s[dcl]\\np/(s[ng]\\np) being:S-be:P-VBG:T-s[ng]\\np/(s[pss]\\np) exported:S-export:P-VBN:T-s[pss]\\np and:S-and:P-CC:T-np\\np\\(np\\np)/(np\\np) when:S-when:P-WRB:T-np\\np/s[dcl] china:S-china:P-NN:T-n provided:S-provide:P-VBD:T-s[dcl]\\np/np such:S-such:P-JJ:T-np/np technology:S-technology:P-NN:T-n to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) pakistan:S-pakistan:P-VB:T-s[b]\\np and:S-and:P-CC:T-s[b]$\\(s[b]$)/(s[b]$) iran:S-iran:P-VB:T-s[b]\\np .:S-.:P-.:T-sent\\(np\\np)\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nhowever , the official was not willing to explain what kind of missile technology that china provided to pakistan and iran , and when this technology was provided .\nw19:n:NN:iran w9:n:NN:kind w18:n\\n/n:CC:and w8:np/n:WP:what w17:n:NN:pakistan w7:s[b]\\np/np:VB:explain w16:pp/np:TO:to w15:s[pss]\\np/pp:VBN:provide w5:s[adj]\\np/(s[to]\\np):JJ:willing w14:s[dcl]\\np/(s[pss]\\np):VBZ:china w4:s\\np\\(s\\np):RB:not w3:s[dcl]\\np/(s[adj]\\np):VBD:be w12:n:NN:technology w2:n:NN:official w11:n/n:NN:missile w1:np/n:DT:the w10:np\\np/np:IN:of w0:s/s:RB:however w26:s[pss]\\np:VBN:provide w25:s[dcl]\\np/(s[pss]\\np):VBD:PASS w24:n:NN:technology w23:np/n:DT:this w22:s/s/s[dcl]:WRB:when w21:s$\\(s$)/(s$):CC:and\nhowever:S-however:P-RB:T-s/s ,:S-,:P-,:T-punct[,] the:S-the:P-DT:T-np/n official:S-official:P-NN:T-n was:S-be:P-VBD:T-s[dcl]\\np/(s[adj]\\np) not:S-not:P-RB:T-s\\np\\(s\\np) willing:S-willing:P-JJ:T-s[adj]\\np/(s[to]\\np) to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) explain:S-explain:P-VB:T-s[b]\\np/np what:S-what:P-WP:T-np/n kind:S-kind:P-NN:T-n of:S-of:P-IN:T-np\\np/np missile:S-missile:P-NN:T-n/n technology:S-technology:P-NN:T-n that:S-that:P-WDT:T-np\\np/(s[dcl]\\np) china:S-china:P-VBZ:T-s[dcl]\\np/(s[pss]\\np) provided:S-provide:P-VBN:T-s[pss]\\np/pp to:S-to:P-TO:T-pp/np pakistan:S-pakistan:P-NN:T-n and:S-and:P-CC:T-n\\n/n iran:S-iran:P-NN:T-n ,:S-,:P-,:T-s/s\\np and:S-and:P-CC:T-s$\\(s$)/(s$) when:S-when:P-WRB:T-s/s/s[dcl] this:S-this:P-DT:T-np/n technology:S-technology:P-NN:T-n was:S-be:P-VBD:T-s[dcl]\\np/(s[pss]\\np) provided:S-provide:P-VBN:T-s[pss]\\np .:S-.:P-.:T-sent\\(s/s)\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nyunnan gardens was developed by fujian company , there are 313 apartments in this residential project .\nw9:n/n:CD:313 w8:s[dcl]\\np[thr]/np:VBP:be w6:n:NN:company w5:n/n:JJ:fujian w4:s\\np\\(s\\np)/np:IN:by w3:s[pss]\\np:VBN:develop w2:s[dcl]\\np/(s[pss]\\np):VBD:PASS w1:n:NNS:garden w0:n/n:JJ:yunnan w14:n:NN:project w13:n/n:JJ:residential w12:np/n:DT:this w11:np\\np/np:IN:in w10:n:NNS:apartment\nyunnan:S-yunnan:P-JJ:T-n/n gardens:S-garden:P-NNS:T-n was:S-be:P-VBD:T-s[dcl]\\np/(s[pss]\\np) developed:S-develop:P-VBN:T-s[pss]\\np by:S-by:P-IN:T-s\\np\\(s\\np)/np fujian:S-fujian:P-JJ:T-n/n company:S-company:P-NN:T-n ,:S-,:P-,:T-punct[,] there:S-there:P-EX:T-np[thr] are:S-be:P-VBP:T-s[dcl]\\np[thr]/np 313:S-313:P-CD:T-n/n apartments:S-apartment:P-NNS:T-n in:S-in:P-IN:T-np\\np/np this:S-this:P-DT:T-np/n residential:S-residential:P-JJ:T-n/n project:S-project:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nthe yunnan gardens project developed by the fujian society is a 313-unit housing project with land .\nw9:s[dcl]\\np/np:VBZ:be w8:n:NN:society w7:n/n:JJ:fujian w6:np/n:DT:the w5:s\\np\\(s\\np)/np:IN:by w4:s[pss]\\np:VBN:develop w3:n:NN:project w2:n/n:NNS:garden w1:n/n:JJ:yunnan w15:n:NN:land w0:np/n:DT:the w14:np\\np/np:IN:with w13:n:NN:project w12:n/n:NN:housing w11:n/n:JJ:313-unit w10:np/n:DT:a\nthe:S-the:P-DT:T-np/n yunnan:S-yunnan:P-JJ:T-n/n gardens:S-garden:P-NNS:T-n/n project:S-project:P-NN:T-n developed:S-develop:P-VBN:T-s[pss]\\np by:S-by:P-IN:T-s\\np\\(s\\np)/np the:S-the:P-DT:T-np/n fujian:S-fujian:P-JJ:T-n/n society:S-society:P-NN:T-n is:S-be:P-VBZ:T-s[dcl]\\np/np a:S-a:P-DT:T-np/n 313-unit:S-313-unit:P-JJ:T-n/n housing:S-housing:P-NN:T-n/n project:S-project:P-NN:T-n with:S-with:P-IN:T-np\\np/np land:S-land:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nin the last two years , some foreign businessmen showed their interest in the construction of xiling city .\nw9:s[dcl]\\np/np:VBD:show w8:n:NNS:businessman w7:n/n:JJ:foreign w6:np/n:DT:some w4:n:NNS:year w3:n/n:CD:two w17:n:NN:city w2:n/n:JJ:last w16:s[ng]\\np/np:VBG:xiling w1:np/n:DT:the w15:np\\np/(s[ng]\\np):IN:of w0:s/s/np:IN:in w14:n:NN:construction w13:np/n:DT:the w12:np\\np/np:IN:in w11:n:NN:interest w10:np/n:PRP$:their\nin:S-in:P-IN:T-s/s/np the:S-the:P-DT:T-np/n last:S-last:P-JJ:T-n/n two:S-two:P-CD:T-n/n years:S-year:P-NNS:T-n ,:S-,:P-,:T-s/s\\(s/s) some:S-some:P-DT:T-np/n foreign:S-foreign:P-JJ:T-n/n businessmen:S-businessman:P-NNS:T-n showed:S-show:P-VBD:T-s[dcl]\\np/np their:S-their:P-PRP$:T-np/n interest:S-interest:P-NN:T-n in:S-in:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n construction:S-construction:P-NN:T-n of:S-of:P-IN:T-np\\np/(s[ng]\\np) xiling:S-xiling:P-VBG:T-s[ng]\\np/np city:S-city:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nduring the past two years , a batch of foreign businessmen expressed their wishes to get involved in xining 's city construction one after another .\nw9:n/n:JJ:foreign w18:n:NN:xining w8:np\\np/np:IN:of w17:pp/np:IN:in w7:n:NN:batch w16:s[adj]\\np/pp:VBN:involve w6:np/n:DT:a w15:s[b]\\np/(s[adj]\\np):VB:get w4:n:NNS:year w13:n:NNS:wish w3:n/n:CD:two w12:np/n:PRP$:their w2:n/n:JJ:past w11:s[dcl]\\np/np:VBD:express w1:np/n:DT:the w10:n:NNS:businessman w0:s/s/np:IN:during w24:np:DT:another w23:np\\np/np:IN:after w22:n:NN:one w21:n/n:NN:construction w20:n/n:NN:city\nduring:S-during:P-IN:T-s/s/np the:S-the:P-DT:T-np/n past:S-past:P-JJ:T-n/n two:S-two:P-CD:T-n/n years:S-year:P-NNS:T-n ,:S-,:P-,:T-s/s\\(s/s) a:S-a:P-DT:T-np/n batch:S-batch:P-NN:T-n of:S-of:P-IN:T-np\\np/np foreign:S-foreign:P-JJ:T-n/n businessmen:S-businessman:P-NNS:T-n expressed:S-express:P-VBD:T-s[dcl]\\np/np their:S-their:P-PRP$:T-np/n wishes:S-wish:P-NNS:T-n to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) get:S-get:P-VB:T-s[b]\\np/(s[adj]\\np) involved:S-involve:P-VBN:T-s[adj]\\np/pp in:S-in:P-IN:T-pp/np xining:S-xining:P-NN:T-n 's:S-'s:P-POS:T-np/n\\np city:S-city:P-NN:T-n/n construction:S-construction:P-NN:T-n/n one:S-one:P-NN:T-n after:S-after:P-IN:T-np\\np/np another:S-another:P-DT:T-np .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nthe dai yu xiang industrial consulting company said that spacious front door area is the biggest characteristic of yunnan gardens with land ownership .\nw19:n:NNS:garden w9:n/n:JJ:spacious w18:n/n:JJ:yunnan w7:s[dcl]\\np/s[em]:VBD:say w17:np\\np/np:IN:of w16:n:NN:characteristic w6:n:NN:company w15:n/n:JJS:biggest w5:n/n:NN:consulting w14:np/n:DT:the w4:n/n:JJ:industrial w13:s[dcl]\\np/np:VBZ:be w3:n/n:VBG:xiang w12:n:NN:area w2:n/n:NN:yu w11:n/n:NN:door w1:n/n:JJ:dai w10:n/n:JJ:front w0:np/n:DT:the w22:n:NN:ownership w21:n/n:NN:land w20:np\\np/np:IN:with\nthe:S-the:P-DT:T-np/n dai:S-dai:P-JJ:T-n/n yu:S-yu:P-NN:T-n/n xiang:S-xiang:P-VBG:T-n/n industrial:S-industrial:P-JJ:T-n/n consulting:S-consulting:P-NN:T-n/n company:S-company:P-NN:T-n said:S-say:P-VBD:T-s[dcl]\\np/s[em] that:S-that:P-IN:T-s[em]/s[dcl] spacious:S-spacious:P-JJ:T-n/n front:S-front:P-JJ:T-n/n door:S-door:P-NN:T-n/n area:S-area:P-NN:T-n is:S-be:P-VBZ:T-s[dcl]\\np/np the:S-the:P-DT:T-np/n biggest:S-biggest:P-JJS:T-n/n characteristic:S-characteristic:P-NN:T-n of:S-of:P-IN:T-np\\np/np yunnan:S-yunnan:P-JJ:T-n/n gardens:S-garden:P-NNS:T-n with:S-with:P-IN:T-np\\np/np land:S-land:P-NN:T-n/n ownership:S-ownership:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\ndai yuxiang real estate consulting firm said that the most special characteristic of the landed residences at yunnan gardens is their spacious appearance .\nw19:s[dcl]\\np/np:VBZ:be w9:n/n/(n/n):RBS:most w18:n:NNS:garden w8:np/n:DT:the w17:n/n:JJ:yunnan w6:s[dcl]\\np/s[em]:VBD:say w16:np\\np/np:IN:at w15:n:NNS:residence w5:n:NN:firm w14:n/n:VBN:land w4:n/n:NN:consulting w13:np/n:DT:the w3:n/n:NN:estate w12:np\\np/np:IN:of w2:n/n:JJ:real w11:n:NN:characteristic w1:n/n:NNP:yuxiang w10:n/n:JJ:special w0:n/n:NNP:dai w22:n:NN:appearance w21:n/n:JJ:spacious w20:np/n:PRP$:their\ndai:S-dai:P-NNP:T-n/n yuxiang:S-yuxiang:P-NNP:T-n/n real:S-real:P-JJ:T-n/n estate:S-estate:P-NN:T-n/n consulting:S-consulting:P-NN:T-n/n firm:S-firm:P-NN:T-n said:S-say:P-VBD:T-s[dcl]\\np/s[em] that:S-that:P-IN:T-s[em]/s[dcl] the:S-the:P-DT:T-np/n most:S-most:P-RBS:T-n/n/(n/n) special:S-special:P-JJ:T-n/n characteristic:S-characteristic:P-NN:T-n of:S-of:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n landed:S-land:P-VBN:T-n/n residences:S-residence:P-NNS:T-n at:S-at:P-IN:T-np\\np/np yunnan:S-yunnan:P-JJ:T-n/n gardens:S-garden:P-NNS:T-n is:S-be:P-VBZ:T-s[dcl]\\np/np their:S-their:P-PRP$:T-np/n spacious:S-spacious:P-JJ:T-n/n appearance:S-appearance:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nthis measure restrained economic growth and raised the rate of unemployment .\nw9:np\\np/np:IN:of w8:n:NN:rate w7:np/n:DT:the w6:s[dcl]\\np/np:VBD:raise w5:s[dcl]$\\(s[dcl]$)/(s[dcl]$):CC:and w4:n:NN:growth w3:n/n:JJ:economic w2:s[dcl]\\np/np:VBD:restrain w1:n:NN:measure w0:np/n:DT:this w10:n:NN:unemployment\nthis:S-this:P-DT:T-np/n measure:S-measure:P-NN:T-n restrained:S-restrain:P-VBD:T-s[dcl]\\np/np economic:S-economic:P-JJ:T-n/n growth:S-growth:P-NN:T-n and:S-and:P-CC:T-s[dcl]$\\(s[dcl]$)/(s[dcl]$) raised:S-raise:P-VBD:T-s[dcl]\\np/np the:S-the:P-DT:T-np/n rate:S-rate:P-NN:T-n of:S-of:P-IN:T-np\\np/np unemployment:S-unemployment:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nthis curbs economic growth and increases unemployment rates .\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\naccording to xinhua news agency , london , on february 3rd , bbc reporter bowen stayed in baghdad . the capital of iraq stated that there was no evidence to indicate the bomb-shelter destroyed by a us bomber was a military target .\nw36:s[dcl]\\np/np:VBD:be w35:s\\np/(s\\np):NN:bomber w34:np:PRP:us w33:np:DT:a w32:s\\np\\(s\\np)/np:IN:by w31:s[pss]\\np:VBN:destroy w30:n:NN:bomb-shelter w29:np/n:DT:the w28:s[b]\\np/np:VB:indicate w26:n:NN:evidence w25:np/n:DT:no w24:s[dcl]\\np[thr]/np:VBD:be w21:s[dcl]\\np/s[em]:VBD:state w20:n:NNP:iraq h1:n\\(n/n)/n:,:has-rel w9:n/n:JJ:3rd w8:n/n:JJ:february w7:s\\np/(s\\np)/np:IN:on w5:n:NN:london w4:n:NN:agency w3:n/n:NN:news w19:np\\np/np:IN:of w2:n/n:NNP:xinhua w18:n:NN:capital w1:pp/np:TO:to w17:np/n:DT:the w0:s/s/pp:VBG:accord w16:n:NN:baghdad w15:pp/np:IN:in w14:s[dcl]\\np/pp:VBD:stay w13:n:NN:bowen w12:n/n:NN:reporter w11:n/n:JJ:bbc w10:n\\(n/n)/n:,:, w39:n:NN:target w38:n/n:JJ:military w37:np/n:DT:a\naccording:S-accord:P-VBG:T-s/s/pp to:S-to:P-TO:T-pp/np xinhua:S-xinhua:P-NNP:T-n/n news:S-news:P-NN:T-n/n agency:S-agency:P-NN:T-n ,:S-,:P-,:T-punct[,] london:S-london:P-NN:T-n ,:S-,:P-,:T-s[dcl]\\np/(s[dcl]\\np) on:S-on:P-IN:T-s\\np/(s\\np)/np february:S-february:P-JJ:T-n/n 3rd:S-3rd:P-JJ:T-n/n ,:S-,:P-,:T-n\\(n/n)/n bbc:S-bbc:P-JJ:T-n/n reporter:S-reporter:P-NN:T-n/n bowen:S-bowen:P-NN:T-n stayed:S-stay:P-VBD:T-s[dcl]\\np/pp in:S-in:P-IN:T-pp/np baghdad:S-baghdad:P-NN:T-n .:S-.:P-.:T-punct[.] the:S-the:P-DT:T-np/n capital:S-capital:P-NN:T-n of:S-of:P-IN:T-np\\np/np iraq:S-iraq:P-NNP:T-n stated:S-state:P-VBD:T-s[dcl]\\np/s[em] that:S-that:P-IN:T-s[em]/s[dcl] there:S-there:P-EX:T-np[thr] was:S-be:P-VBD:T-s[dcl]\\np[thr]/np no:S-no:P-DT:T-np/n evidence:S-evidence:P-NN:T-n to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) indicate:S-indicate:P-VB:T-s[b]\\np/np the:S-the:P-DT:T-np/n bomb-shelter:S-bomb-shelter:P-NN:T-n destroyed:S-destroy:P-VBN:T-s[pss]\\np by:S-by:P-IN:T-s\\np\\(s\\np)/np a:S-a:P-DT:T-np us:S-us:P-PRP:T-np bomber:S-bomber:P-NN:T-s\\np/(s\\np) was:S-be:P-VBD:T-s[dcl]\\np/np a:S-a:P-DT:T-np/n military:S-military:P-JJ:T-n/n target:S-target:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\naccording to xinhua news report from london on february 13 , bonne , a reporter of bbc in iraqi capital baghdad , confirmed on 13th that no any sign showed the bomb shelter in baghdad destroyed by american bomber was a military blindage .\nw36:s\\np\\(s\\np)/np:IN:by w35:s[pss]\\np:VBN:destroy w34:n:NN:baghdad w33:np\\np/np:IN:in w32:n:NN:shelter w31:n/n:NN:bomb w30:np/n:DT:the w29:s[dcl]\\np/s[dcl]:VBD:show w28:n:NN:sign w27:n/n:DT:any w26:np/n:DT:no w24:n:NN:13th w23:s\\np\\(s\\np)/np:IN:on w22:s[dcl]\\np/np:VBD:confirm w20:n:NN:baghdad w9:n:CD:13 w8:n/n:JJ:february w7:np\\np/np:IN:on w6:n:NN:london w5:np\\np/np:IN:from w4:n:NN:report w3:n/n:NN:news w19:n/n:NN:capital w2:np/n:DT:xinhua w18:n/n:JJ:iraqi w1:pp/np:TO:to w17:np\\np/np:IN:in w0:s/s/pp:VBG:accord w16:n:NN:bbc w15:np\\np/np:IN:of w14:n:NN:reporter w13:np/n:DT:a w11:n:NN:bonne w42:n:NN:blindage w41:n/n:JJ:military w40:np/n:DT:a w39:s[dcl]\\np/np:VBD:be w38:n:NN:bomber w37:n/n:JJ:american\naccording:S-accord:P-VBG:T-s/s/pp to:S-to:P-TO:T-pp/np xinhua:S-xinhua:P-DT:T-np/n news:S-news:P-NN:T-n/n report:S-report:P-NN:T-n from:S-from:P-IN:T-np\\np/np london:S-london:P-NN:T-n on:S-on:P-IN:T-np\\np/np february:S-february:P-JJ:T-n/n 13:S-13:P-CD:T-n ,:S-,:P-,:T-np\\np/punct[,]/np bonne:S-bonne:P-NN:T-n ,:S-,:P-,:T-punct[,] a:S-a:P-DT:T-np/n reporter:S-reporter:P-NN:T-n of:S-of:P-IN:T-np\\np/np bbc:S-bbc:P-NN:T-n in:S-in:P-IN:T-np\\np/np iraqi:S-iraqi:P-JJ:T-n/n capital:S-capital:P-NN:T-n/n baghdad:S-baghdad:P-NN:T-n ,:S-,:P-,:T-s[dcl]\\np/(s[dcl]\\np) confirmed:S-confirm:P-VBD:T-s[dcl]\\np/np on:S-on:P-IN:T-s\\np\\(s\\np)/np 13th:S-13th:P-NN:T-n that:S-that:P-IN:T-np\\np/(s[dcl]/np) no:S-no:P-DT:T-np/n any:S-any:P-DT:T-n/n sign:S-sign:P-NN:T-n showed:S-show:P-VBD:T-s[dcl]\\np/s[dcl] the:S-the:P-DT:T-np/n bomb:S-bomb:P-NN:T-n/n shelter:S-shelter:P-NN:T-n in:S-in:P-IN:T-np\\np/np baghdad:S-baghdad:P-NN:T-n destroyed:S-destroy:P-VBN:T-s[pss]\\np by:S-by:P-IN:T-s\\np\\(s\\np)/np american:S-american:P-JJ:T-n/n bomber:S-bomber:P-NN:T-n was:S-be:P-VBD:T-s[dcl]\\np/np a:S-a:P-DT:T-np/n military:S-military:P-JJ:T-n/n blindage:S-blindage:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nmr. siazon said , someone used to say , have n't we been burned once before ? when we were ready to accept them , it did n't take long before they fought agaion .\nw32:s\\np\\(s\\np):NN:agaion w31:s[dcl]\\np:VBD:fight w30:np:PRP:they w19:s[adj]\\np/(s[to]\\np):JJ:ready w9:s[dcl]\\np/np:VBP:have w18:s[dcl]\\np/(s[adj]\\np):VBD:be w17:np:PRP:we w7:s[b]\\np:VB:say w16:s/s/s[dcl]:WRB:when w15:s\\np\\(s\\np):RB:before w5:s[pss]\\np/(s[to]\\np):VBD:use w14:s\\np\\(s\\np)/(s\\np\\(s\\np)):RB:once w4:np:DT:someone w13:s[pss]\\np:VBN:burn w12:s[dcl]\\np/(s[pss]\\np):VBD:been w2:s[dcl]\\np/s[dcl]/punct[,]:VBD:say w11:np:PRP:we w1:n:NN:siazon w10:s\\np\\(s\\np):RB:n't w0:np/n:DT:mr. w29:s\\np\\(s\\np)/s[dcl]:IN:before w28:s\\np\\(s\\np):RB:long w27:s[b]\\np:VB:take w26:s\\np\\(s\\np):RB:n't w25:s[dcl]\\np/(s[b]\\np):VBD:do w24:np:PRP:it w22:np:PRP:them w21:s[b]\\np/np:VB:accept\nmr.:S-mr.:P-DT:T-np/n siazon:S-siazon:P-NN:T-n said:S-say:P-VBD:T-s[dcl]\\np/s[dcl]/punct[,] ,:S-,:P-,:T-punct[,] someone:S-someone:P-DT:T-np used:S-use:P-VBD:T-s[pss]\\np/(s[to]\\np) to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) say:S-say:P-VB:T-s[b]\\np ,:S-,:P-,:T-s[pss]\\np\\(s[pss]\\np) have:S-have:P-VBP:T-s[dcl]\\np/np n't:S-n't:P-RB:T-s\\np\\(s\\np) we:S-we:P-PRP:T-np been:S-been:P-VBD:T-s[dcl]\\np/(s[pss]\\np) burned:S-burn:P-VBN:T-s[pss]\\np once:S-once:P-RB:T-s\\np\\(s\\np)/(s\\np\\(s\\np)) before:S-before:P-RB:T-s\\np\\(s\\np) ?:S-?:P-?:T-punct[?] when:S-when:P-WRB:T-s/s/s[dcl] we:S-we:P-PRP:T-np were:S-be:P-VBD:T-s[dcl]\\np/(s[adj]\\np) ready:S-ready:P-JJ:T-s[adj]\\np/(s[to]\\np) to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) accept:S-accept:P-VB:T-s[b]\\np/np them:S-them:P-PRP:T-np ,:S-,:P-,:T-s/s\\(s/s) it:S-it:P-PRP:T-np did:S-do:P-VBD:T-s[dcl]\\np/(s[b]\\np) n't:S-n't:P-RB:T-s\\np\\(s\\np) take:S-take:P-VB:T-s[b]\\np long:S-long:P-RB:T-s\\np\\(s\\np) before:S-before:P-IN:T-s\\np\\(s\\np)/s[dcl] they:S-they:P-PRP:T-np fought:S-fight:P-VBD:T-s[dcl]\\np agaion:S-agaion:P-NN:T-s\\np\\(s\\np) .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nsiazon said : someone says , were we deceived before ? when we prepared to accept them but just after several days they began to fight again .\nw19:n/n:JJ:several w9:s\\np\\(s\\np):IN:before w18:s/s/np:IN:after w8:s[dcl]\\np:VBD:deceive w17:s/s/(s/s):RB:just w7:np:PRP:we w16:s$\\(s$)/(s$):CC:but w6:s[dcl]\\np/np:VBD:be w15:np:PRP:them w5:s[dcl]$\\(s[dcl]$)/(s[dcl]$):,:, w14:s[b]\\np/np:VB:accept w4:s[dcl]\\np:VBZ:say w3:np:DT:someone w12:s[dcl]\\np/(s[to]\\np):VBD:prepare w2:s\\s/s[dcl]:IN:| w11:np:PRP:we w1:s[dcl]\\np/s[dcl]:VBD:say w10:s/s/s[dcl]:WRB:when w0:n:NN:siazon w25:s\\np\\(s\\np):RB:again w24:s[b]\\np:VB:fight w22:s[dcl]\\np/(s[to]\\np):VBD:begin w21:np:PRP:they w20:n:NNS:day\nsiazon:S-siazon:P-NN:T-n said:S-say:P-VBD:T-s[dcl]\\np/s[dcl] &#58;:S-&#58;:P-IN:T-s\\s/s[dcl] someone:S-someone:P-DT:T-np says:S-say:P-VBZ:T-s[dcl]\\np ,:S-,:P-,:T-s[dcl]$\\(s[dcl]$)/(s[dcl]$) were:S-be:P-VBD:T-s[dcl]\\np/np we:S-we:P-PRP:T-np deceived:S-deceive:P-VBD:T-s[dcl]\\np before:S-before:P-IN:T-s\\np\\(s\\np) ?:S-?:P-?:T-punct[?] when:S-when:P-WRB:T-s/s/s[dcl] we:S-we:P-PRP:T-np prepared:S-prepare:P-VBD:T-s[dcl]\\np/(s[to]\\np) to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) accept:S-accept:P-VB:T-s[b]\\np/np them:S-them:P-PRP:T-np but:S-but:P-CC:T-s$\\(s$)/(s$) just:S-just:P-RB:T-s/s/(s/s) after:S-after:P-IN:T-s/s/np several:S-several:P-JJ:T-n/n days:S-day:P-NNS:T-n they:S-they:P-PRP:T-np began:S-begin:P-VBD:T-s[dcl]\\np/(s[to]\\np) to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) fight:S-fight:P-VB:T-s[b]\\np again:S-again:P-RB:T-s\\np\\(s\\np) .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nrailroad officials blamed the bus passengers for the accident , the indian news agency india press trust report said .\nw18:s[dcl]\\s[dcl]\\np:VBD:say w8:n:NN:accident w17:n:NN:report w7:np/n:DT:the w16:n/n:NN:trust w6:pp/np:IN:for w15:n/n:NN:press w5:n:NNS:passenger w14:n/n:NN:india w4:n/n:NN:bus w13:n/n:NN:agency w3:np/n:DT:the w12:n/n:NN:news w2:s[dcl]\\np/pp/np:VBD:blame w11:n/n:JJ:indian w1:n:NNS:official w10:np/n:DT:the w0:n/n:NN:railroad\nrailroad:S-railroad:P-NN:T-n/n officials:S-official:P-NNS:T-n blamed:S-blame:P-VBD:T-s[dcl]\\np/pp/np the:S-the:P-DT:T-np/n bus:S-bus:P-NN:T-n/n passengers:S-passenger:P-NNS:T-n for:S-for:P-IN:T-pp/np the:S-the:P-DT:T-np/n accident:S-accident:P-NN:T-n ,:S-,:P-,:T-s\\s/(s\\s) the:S-the:P-DT:T-np/n indian:S-indian:P-JJ:T-n/n news:S-news:P-NN:T-n/n agency:S-agency:P-NN:T-n/n india:S-india:P-NN:T-n/n press:S-press:P-NN:T-n/n trust:S-trust:P-NN:T-n/n report:S-report:P-NN:T-n said:S-say:P-VBD:T-s[dcl]\\s[dcl]\\np .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nreport of pti said that railway official charged the bus passengers upon the accident .\nw9:n/n:NN:bus w8:np/n:DT:the w7:s[dcl]\\np/pp/np:VBD:charge w6:n:NN:official w5:n/n:NN:railway w3:s[dcl]\\np/s[em]:VBD:say w2:n:NNS:pti w1:np\\np/np:IN:of w0:n:NN:report w13:n:NN:accident w12:np/n:DT:the w11:pp/np:IN:upon w10:n:NNS:passenger\nreport:S-report:P-NN:T-n of:S-of:P-IN:T-np\\np/np pti:S-pti:P-NNS:T-n said:S-say:P-VBD:T-s[dcl]\\np/s[em] that:S-that:P-DT:T-s[em]/s[dcl] railway:S-railway:P-NN:T-n/n official:S-official:P-NN:T-n charged:S-charge:P-VBD:T-s[dcl]\\np/pp/np the:S-the:P-DT:T-np/n bus:S-bus:P-NN:T-n/n passengers:S-passenger:P-NNS:T-n upon:S-upon:P-IN:T-pp/np the:S-the:P-DT:T-np/n accident:S-accident:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nwe mentioned the problems australia is concerned with , such as human rights and dispatching army officers to east timor .\nw9:np\\np/np:IN:as w18:n:NN:timor w8:np\\np/(np\\np):JJ:such w7:pp/np:IN:with w17:n/n:JJ:east w16:np\\np/np:TO:to w6:s[pss]\\np/pp:VBN:concern w15:n:NNS:officer w5:s[dcl]\\np/(s[pss]\\np):VBZ:PASS w14:n/n:NN:army w4:n:NN:australia w13:n/n:VBG:dispatch w3:n/n:NNS:problem w12:np\\np/np:CC:and w2:np/n:DT:the w11:n:NNS:rights w1:s[dcl]\\np/np:VBD:mention w10:n/n:JJ:human w0:np:PRP:we\nwe:S-we:P-PRP:T-np mentioned:S-mention:P-VBD:T-s[dcl]\\np/np the:S-the:P-DT:T-np/n problems:S-problem:P-NNS:T-n/n australia:S-australia:P-NN:T-n is:S-be:P-VBZ:T-s[dcl]\\np/(s[pss]\\np) concerned:S-concern:P-VBN:T-s[pss]\\np/pp with:S-with:P-IN:T-pp/np ,:S-,:P-,:T-punct[,] such:S-such:P-JJ:T-np\\np/(np\\np) as:S-as:P-IN:T-np\\np/np human:S-human:P-JJ:T-n/n rights:S-rights:P-NNS:T-n and:S-and:P-CC:T-np\\np/np dispatching:S-dispatch:P-VBG:T-n/n army:S-army:P-NN:T-n/n officers:S-officer:P-NNS:T-n to:S-to:P-TO:T-np\\np/np east:S-east:P-JJ:T-n/n timor:S-timor:P-NN:T-n .:S-.:P-.:T-sent\\(np\\np)\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nwe mentioned the problems that have attracted australia 's attention , such as the humam rights in east timor and the issue of sending military officers to east timor .\nw19:np\\np/np:CC:and w9:n:NN:attention w18:n:NN:timor w17:n/n:JJ:east w7:n:NN:australia w6:s[pt]\\np/np:VBN:attract w16:np\\np/np:IN:in w15:n:NNS:rights w5:s[dcl]\\np/(s[pt]\\np):VBP:PERF w14:n/n:NN:humam w3:n:NNS:problem w13:np/n:DT:the w12:np\\np/np:IN:as w2:np/n:DT:the w11:np\\np/(np\\np):JJ:such w1:s[dcl]\\np/np:VBD:mention w0:np:PRP:we w28:n:NN:timor w27:n/n:JJ:east w26:s\\np\\(s\\np)/np:TO:to w25:n:NNS:officer w24:n/n:JJ:military w23:s[ng]\\np/np:VBG:send w22:np\\np/(s[ng]\\np):IN:of w21:n:NN:issue w20:np/n:DT:the\nwe:S-we:P-PRP:T-np mentioned:S-mention:P-VBD:T-s[dcl]\\np/np the:S-the:P-DT:T-np/n problems:S-problem:P-NNS:T-n that:S-that:P-WDT:T-np\\np/(s[dcl]\\np) have:S-have:P-VBP:T-s[dcl]\\np/(s[pt]\\np) attracted:S-attract:P-VBN:T-s[pt]\\np/np australia:S-australia:P-NN:T-n 's:S-'s:P-POS:T-np/n\\np attention:S-attention:P-NN:T-n ,:S-,:P-,:T-s[dcl]\\np\\(s[dcl]\\np) such:S-such:P-JJ:T-np\\np/(np\\np) as:S-as:P-IN:T-np\\np/np the:S-the:P-DT:T-np/n humam:S-humam:P-NN:T-n/n rights:S-rights:P-NNS:T-n in:S-in:P-IN:T-np\\np/np east:S-east:P-JJ:T-n/n timor:S-timor:P-NN:T-n and:S-and:P-CC:T-np\\np/np the:S-the:P-DT:T-np/n issue:S-issue:P-NN:T-n of:S-of:P-IN:T-np\\np/(s[ng]\\np) sending:S-send:P-VBG:T-s[ng]\\np/np military:S-military:P-JJ:T-n/n officers:S-officer:P-NNS:T-n to:S-to:P-TO:T-s\\np\\(s\\np)/np east:S-east:P-JJ:T-n/n timor:S-timor:P-NN:T-n .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nthe two prime ministers requested that un secretary-general prolong the un representative 's stay in cambodia to 6 months or longer .\nw19:np\\np\\(np\\np)/(np\\np):CC:or w9:np/n:DT:the w18:n:NNS:month w8:s[ng]\\np/np:VBG:prolong w7:n:NN:secretary-general w17:n/n:CD:6 w16:np\\np/np:TO:to w6:n/n:JJ:un w15:n:NN:cambodia w5:np/n:DT:that w14:np\\np/np:IN:in w4:s[dcl]\\np/np:VBD:request w13:n:NN:stay w3:n:NNS:minister w2:n/n:JJ:prime w11:n:NN:representative w1:n/n:CD:two w10:n/n:JJ:un w0:np/n:DT:the w20:np\\np:JJR:longer\nthe:S-the:P-DT:T-np/n two:S-two:P-CD:T-n/n prime:S-prime:P-JJ:T-n/n ministers:S-minister:P-NNS:T-n requested:S-request:P-VBD:T-s[dcl]\\np/np that:S-that:P-DT:T-np/n un:S-un:P-JJ:T-n/n secretary-general:S-secretary-general:P-NN:T-n prolong:S-prolong:P-VBG:T-s[ng]\\np/np the:S-the:P-DT:T-np/n un:S-un:P-JJ:T-n/n representative:S-representative:P-NN:T-n 's:S-'s:P-POS:T-np/n\\np stay:S-stay:P-NN:T-n in:S-in:P-IN:T-np\\np/np cambodia:S-cambodia:P-NN:T-n to:S-to:P-TO:T-np\\np/np 6:S-6:P-CD:T-n/n months:S-month:P-NNS:T-n or:S-or:P-CC:T-np\\np\\(np\\np)/(np\\np) longer:S-longer:P-JJR:T-np\\np .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\nthe two prime ministers of cambodia asked the un secretary general to extend the un representative 's term of stay in cambodia to six months or longer .\nw19:n:NN:stay w9:n:NN:secretary w18:np\\np/np:IN:of w8:n/n:NN:un w17:n:NN:term w7:np/n:DT:the w6:s[dcl]\\np/(s[to]\\np)/np:VBD:ask w15:n:NN:representative w5:n:NN:cambodia w14:n/n:JJ:un w4:np\\np/np:IN:of w3:n:NNS:minister w13:np/n:DT:the w12:s[b]\\np/pp/np:VB:extend w2:n/n:JJ:prime w1:n/n:CD:two w10:s[adj]\\np:JJ:general w0:np/n:DT:the w26:np\\np:JJR:longer w25:np\\np/(np\\np):CC:or w24:n:NNS:month w23:n/n:CD:six w22:pp/np:TO:to w21:n:NN:cambodia w20:np\\np/np:IN:in\nthe:S-the:P-DT:T-np/n two:S-two:P-CD:T-n/n prime:S-prime:P-JJ:T-n/n ministers:S-minister:P-NNS:T-n of:S-of:P-IN:T-np\\np/np cambodia:S-cambodia:P-NN:T-n asked:S-ask:P-VBD:T-s[dcl]\\np/(s[to]\\np)/np the:S-the:P-DT:T-np/n un:S-un:P-NN:T-n/n secretary:S-secretary:P-NN:T-n general:S-general:P-JJ:T-s[adj]\\np to:S-to:P-TO:T-s[to]\\np/(s[b]\\np) extend:S-extend:P-VB:T-s[b]\\np/pp/np the:S-the:P-DT:T-np/n un:S-un:P-JJ:T-n/n representative:S-representative:P-NN:T-n 's:S-'s:P-POS:T-np/n\\np term:S-term:P-NN:T-n of:S-of:P-IN:T-np\\np/np stay:S-stay:P-NN:T-n in:S-in:P-IN:T-np\\np/np cambodia:S-cambodia:P-NN:T-n to:S-to:P-TO:T-pp/np six:S-six:P-CD:T-n/n months:S-month:P-NNS:T-n or:S-or:P-CC:T-np\\np/(np\\np) longer:S-longer:P-JJR:T-np\\np .:S-.:P-.:T-sent\\s[dcl]\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n" }, { "path": "test/rules.xml", "content": "\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "test/testlf.xml", "content": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" } ]