[
  {
    "path": ".gitignore",
    "content": "# dependencies\nvendor"
  },
  {
    "path": ".travis.yml",
    "content": "language: go\n\ngo:\n  - 1.8.x\n  - 1.9.x\n  - tip\n\nbefore_install:\n  - go get -u github.com/golang/dep/cmd/dep\n  - dep ensure\n\nscript:\n  - go test -v -race -coverprofile=coverage.txt -covermode=atomic\n\nafter_success:\n  - bash <(curl -s https://codecov.io/bash)"
  },
  {
    "path": "Gopkg.toml",
    "content": "\nrequired = [\n    \"github.com/mna/pigeon\"\n]"
  },
  {
    "path": "LICENSE",
    "content": "The MIT License (MIT)\n\nCopyright (c) 2017 Juan Álvarez / @Shixzie\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n"
  },
  {
    "path": "Makefile",
    "content": "help:\n\t@echo \"deps   -> Get all dependencies\"\n\t@echo \"parser -> Generates the sample parser\"\n\t@echo \"tests  -> Run all tests\"\n\ndeps:\n\t@go get -u github.com/golang/dep/cmd/dep\n\t@dep ensure\n\nparser:\n\t@pigeon -o \"./parser/parser.go\" \"./parser/nlp.peg\"\n\ntests:\n\t@go test -v -race ./..."
  },
  {
    "path": "README.md",
    "content": "[![GoDoc](https://godoc.org/github.com/shixzie/nlp?status.svg)](https://godoc.org/github.com/shixzie/nlp) \n[![Go Report Card](https://goreportcard.com/badge/github.com/shixzie/nlp)](https://goreportcard.com/report/github.com/shixzie/nlp)\n[![Build Status](https://travis-ci.org/shixzie/nlp.svg?branch=master)](https://travis-ci.org/shixzie/nlp)\n[![codecov](https://codecov.io/gh/shixzie/nlp/branch/master/graph/badge.svg)](https://codecov.io/gh/shixzie/nlp)\n\n\n# nlp\n\n> `nlp` is a general purpose any-lang Natural Language Processor that parses the data inside a text and returns a filled model\n\n## Supported types\n```go\nint  int8  int16  int32  int64\nuint uint8 uint16 uint32 uint64\nfloat32 float64\nstring\ntime.Time\ntime.Duration\n```\n\n## Installation\n```\n// go1.8+ is required\ngo get -u github.com/shixzie/nlp\n```\n\n\n**Feel free to create PR's and open Issues :)**\n\n## How it works\n\nYou will always begin by creating a NL type calling nlp.New(), the NL type is a \nNatural Language Processor that owns 3 funcs, RegisterModel(), Learn() and P().\n\n### RegisterModel(i interface{}, samples []string, ops ...ModelOption) error\n\nRegisterModel takes 3 parameters, an empty struct, a set of samples and some options for the model.\n\nThe empty struct lets nlp know all possible values inside the text, for example:\n```go\ntype Song struct {\n\tName        string // fields must be exported\n\tArtist      string\n\tReleasedAt  time.Time\n}\nerr := nl.RegisterModel(Song{}, someSamples, nlp.WithTimeFormat(\"2006\"))\nif err != nil {\n\tpanic(err)\n}\n// ...\n```\n\ntells nlp that inside the text may be a Song.Name, a Song.Artist and a Song.ReleasedAt.\n\nThe samples are the key part about nlp, not just because they set the *limits*\nbetween *keywords* but also because they will be used to choose which model \nuse to handle an expression.\n\nSamples must have a special syntax to set those *limits* and *keywords*.\n```go\nsongSamples := []string{\n\t\"play {Name} by {Artist}\",\n\t\"play {Name} from {Artist}\",\n\t\"play {Name}\",\n\t\"from {Artist} play {Name}\",\n\t\"play something from {ReleasedAt}\",\n}\n```\n\nIn the example below, you can see we're reffering to the Name and Artist fields\nof the `Song` type declared above, both `{Name}` and `{Artist}` are our *keywords* \nand yes! you guessed it! Everything between `play` and `by` will be treated as a\n`{Name}`, and everything that's after `by` will be treated as an `{Artist}` meaning \nthat `play` and `by` are our *limits*.\n```\n     limits\n ┌─────┴─────┐\n┌┴─┐        ┌┴┐\nplay {Name} by  {Artist}\n     └─┬──┘     └───┬──┘\n       └──────┬─────┘\n           keywords\n```\n\nAny character can be a *limit*, a `,` for example can be used as a limit.\n\n*keywords* as well as *limits* are `CaseSensitive` so be sure to type them right.\n\n**Note that putting 2 *keywords* together will cause that only 1 or none of them will be detected**\n\n> *limits are important* - Me :3\n\n\n### Learn() error\n\nLearn maps all models samples to their respective models using the NaiveBayes \nalgorithm based on those samples. `Learn()` also trains all registered models\nso they're able to fit expressions in the future.\n\n```go\n// must call after all models are registrated and before calling nl.P()\nerr := nl.Learn() \nif err != nil {\n\tpanic(err)\n}\n// ...\n```\n\nOnce the algorithm has finished learning, we're now ready to start Processing \nthose texts.\n\n**Note that you must call NL.Learn() after all models are registrated and before calling NL.P()**\n\n### P(expr string) interface{}\n\nP first asks the trained algorithm which model should be used, once we get\nthe right *and already trained* model, we just make it fit the expression.\n\n**Note that everything in the expression must be separated by a _space_ or _tab_**\n\nWhen processing an expression, nlp searches for the *limits* inside that \nexpression and evaluates which sample fits better the expression, it doesn't\nmatter if the text has `trash`. In this example:\n```\n     limits\n ┌─────┴─────┐\n┌┴─┐        ┌┴┐\nplay {Name} by  {Artist}\n     └─┬──┘     └───┬──┘\n       └──────┬─────┘\n           keywords\n```\n\nwe have 2 *limits*, `play` and `by`, it doesn't matter if we had an expression \n*hello sir can you pleeeeeease play King by Lauren Aquilina*, since:\n```\n                                  limits\n            trash              ┌────┴────┐\n┌─────────────┴─────────────┐ ┌┴─┐      ┌┴┐\nhello sir can you pleeeeeease play King by  Lauren Aquilina\n                                   └┬─┘     └─────┬───────┘\n                                 {Name}       {Artist}\n                                 └─┬──┘       └───┬──┘\n                                   └──────┬───────┘\n                                       keywords\n```\n\n`{Name}` would be replaced with `King`, \n`{Artist}` would be replaced with `Lauren Aquilina`, \n`trash` would be ignored as well as the *limits* `play` and `by`, \nand then **a pointer to a filled struct with the type used to register the model** (`Song`) \n( `Song.Name` being `{Name}` and `Song.Artist` beign `{Artist}` ) \n**will be returned**.\n\n## Usage\n\n```go\ntype Song struct {\n\tName       string\n\tArtist     string\n\tReleasedAt time.Time\n}\n\nsongSamples := []string{\n\t\"play {Name} by {Artist}\",\n\t\"play {Name} from {Artist}\",\n\t\"play {Name}\",\n\t\"from {Artist} play {Name}\",\n\t\"play something from {ReleasedAt}\",\n}\n\nnl := nlp.New()\nerr := nl.RegisterModel(Song{}, songSamples, nlp.WithTimeFormat(\"2006\"))\nif err != nil {\n\tpanic(err)\n}\n\nerr = nl.Learn() // you must call Learn after all models are registered and before calling P\nif err != nil {\n\tpanic(err)\n}\n\n// after learning you can call P the times you want\ns := nl.P(\"hello sir can you pleeeeeease play King by Lauren Aquilina\") \nif song, ok := s.(*Song); ok {\n\tfmt.Println(\"Success\")\n\tfmt.Printf(\"%#v\\n\", song)\n} else {\n\tfmt.Println(\"Failed\")\n}\n\n// Prints\n//\n// Success\n// &main.Song{Name: \"King\", Artist: \"Lauren Aquilina\"}\n```\n"
  },
  {
    "path": "benchmark_test.go",
    "content": "package nlp\n\nimport (\n\t\"testing\"\n\t\"time\"\n)\n\nfunc BenchmarkNL_P(b *testing.B) {\n\ttype T struct {\n\t\tString string\n\t\tInt    int\n\t\tUint   uint\n\t\tFloat  float32\n\t\tTime   time.Time\n\t\tDur    time.Duration\n\t}\n\n\ttSamples := []string{\n\t\t\"string {String}\",\n\t\t\"int {Int}\",\n\t\t\"uint {Uint}\",\n\t\t\"float {Float}\",\n\t\t\"time {Time}\",\n\t\t\"dur {Dur}\",\n\t\t\"string {String} int {Int}\",\n\t\t\"string {String} time {Time}\",\n\t}\n\n\tnl := New()\n\n\tnl.RegisterModel(T{}, tSamples)\n\n\terr := nl.RegisterModel(T{}, tSamples)\n\tif err != nil {\n\t\tb.Error(err)\n\t}\n\n\terr = nl.Learn()\n\tif err != nil {\n\t\tb.Error(err)\n\t}\n\n\ttim, err := time.ParseInLocation(\"01-02-2006_3:04pm\", \"05-18-1999_6:42pm\", time.Local)\n\tif err != nil {\n\t\tb.Error(err)\n\t}\n\n\tdur, err := time.ParseDuration(\"4h2m\")\n\tif err != nil {\n\t\tb.Error(err)\n\t}\n\n\tcases := []struct {\n\t\tname       string\n\t\texpression string\n\t\twant       interface{}\n\t}{\n\t\t{\n\t\t\t\"string\",\n\t\t\t\"string Hello World\",\n\t\t\t\"Hello World\",\n\t\t},\n\t\t{\n\t\t\t\"int\",\n\t\t\t\"int 42\",\n\t\t\tint(42),\n\t\t},\n\t\t{\n\t\t\t\"uint\",\n\t\t\t\"uint 43\",\n\t\t\tuint(43),\n\t\t},\n\t\t{\n\t\t\t\"float\",\n\t\t\t\"float 44\",\n\t\t\tfloat32(44),\n\t\t},\n\t\t{\n\t\t\t\"time\",\n\t\t\t\"time 05-18-1999_6:42pm\",\n\t\t\ttim,\n\t\t},\n\t\t{\n\t\t\t\"duration\",\n\t\t\t\"dur 4h2m\",\n\t\t\tdur,\n\t\t},\n\t\t{\n\t\t\t\"string int\",\n\t\t\t\"string Lmao int 42\",\n\t\t\t&T{\n\t\t\t\tString: \"Lmao\",\n\t\t\t\tInt:    42,\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\t\"string time\",\n\t\t\t\"string What's Up Boy time 05-18-1999_6:42pm\",\n\t\t\t&T{\n\t\t\t\tString: \"What's Up Boy\",\n\t\t\t\tTime:   tim,\n\t\t\t},\n\t\t},\n\t}\n\n\tfor _, c := range cases {\n\t\tb.Run(c.name, func(b *testing.B) {\n\t\t\tnl.P(c.expression)\n\t\t})\n\t}\n}\n"
  },
  {
    "path": "nlp.go",
    "content": "// Package nlp provides general purpose Natural Language Processing.\npackage nlp\n\nimport (\n\t\"bytes\"\n\t\"errors\"\n\t\"fmt\"\n\t\"reflect\"\n\t\"strconv\"\n\t\"time\"\n\t\"unicode\"\n\n\t\"github.com/cdipaolo/goml/base\"\n\t\"github.com/cdipaolo/goml/text\"\n\t\"github.com/shixzie/nlp/parser\"\n)\n\n// NL is a Natural Language Processor\ntype NL struct {\n\tmodels []*model\n\tnaive  *text.NaiveBayes\n\t// Output contains the training output for the\n\t// NaiveBayes algorithm\n\tOutput *bytes.Buffer\n}\n\n// New returns a *NL\nfunc New() *NL { return &NL{Output: bytes.NewBufferString(\"\")} }\n\n// P proccesses the expr and returns one of\n// the types passed as the i parameter to the RegistryModel\n// func filled with the data inside expr\nfunc (nl *NL) P(expr string) interface{} { return nl.models[nl.naive.Predict(expr)].fit(expr) }\n\n// Learn maps the models samples to the models themselves and\n// returns an error if something occurred while learning\nfunc (nl *NL) Learn() error {\n\tif len(nl.models) > 0 {\n\t\tstream := make(chan base.TextDatapoint)\n\t\terrors := make(chan error)\n\t\tnl.naive = text.NewNaiveBayes(stream, uint8(len(nl.models)), base.OnlyWordsAndNumbers)\n\t\tnl.naive.Output = nl.Output\n\t\tgo nl.naive.OnlineLearn(errors)\n\t\tfor i := range nl.models {\n\t\t\terr := nl.models[i].learn()\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"model#%d %v\", i, err)\n\t\t\t}\n\t\t\tfor _, s := range nl.models[i].samples {\n\t\t\t\tstream <- base.TextDatapoint{\n\t\t\t\t\tX: string(s),\n\t\t\t\t\tY: uint8(i),\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tclose(stream)\n\t\tfor {\n\t\t\terr := <-errors\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"error occurred while learning: %s\", err)\n\t\t\t}\n\t\t\t// training is done!\n\t\t\tbreak\n\t\t}\n\t\treturn nil\n\t}\n\treturn fmt.Errorf(\"register at least one model before learning\")\n}\n\ntype model struct {\n\ttpy          reflect.Type\n\tfields       []field\n\texpected     [][]item\n\tsamples      [][]byte\n\ttimeFormat   string\n\ttimeLocation *time.Location\n}\n\ntype item struct {\n\tlimit bool\n\tvalue []byte\n\tfield field\n}\n\ntype field struct {\n\tindex int\n\tname  string\n\tkind  interface{}\n}\n\n// ModelOption is an option for a specific model\ntype ModelOption func(*model) error\n\n// WithTimeFormat sets the format used in time.Parse(format, val),\n// note that format can't contain any spaces, the default is 01-02-2006_3:04pm\nfunc WithTimeFormat(format string) ModelOption {\n\treturn func(m *model) error {\n\t\tfor _, v := range format {\n\t\t\tif unicode.IsSpace(v) {\n\t\t\t\treturn errors.New(\"time format can't contain any spaces\")\n\t\t\t}\n\t\t}\n\t\tm.timeFormat = format\n\t\treturn nil\n\t}\n}\n\n// WithTimeLocation sets the location used in time.ParseInLocation(format, value, loc),\n// the default is time.Local\nfunc WithTimeLocation(loc *time.Location) ModelOption {\n\treturn func(m *model) error {\n\t\tif loc == nil {\n\t\t\treturn errors.New(\"time location can't be nil\")\n\t\t}\n\t\tm.timeLocation = loc\n\t\treturn nil\n\t}\n}\n\n// RegisterModel registers a model i and creates possible patterns\n// from samples, the default layout when parsing time is 01-02-2006_3:04pm\n// and the default location is time.Local.\n// Samples must have special formatting:\n//\n//\t\"play {Name} by {Artist}\"\nfunc (nl *NL) RegisterModel(i interface{}, samples []string, ops ...ModelOption) error {\n\tif i == nil {\n\t\treturn fmt.Errorf(\"can't create model from nil value\")\n\t}\n\tif len(samples) == 0 {\n\t\treturn fmt.Errorf(\"samples can't be nil or empty\")\n\t}\n\ttpy, val := reflect.TypeOf(i), reflect.ValueOf(i)\n\tif tpy.Kind() == reflect.Struct {\n\t\tmod := &model{\n\t\t\ttpy:          tpy,\n\t\t\texpected:     make([][]item, len(samples)),\n\t\t\ttimeFormat:   \"01-02-2006_3:04pm\",\n\t\t\ttimeLocation: time.Local,\n\t\t}\n\t\tmod.setSamples(samples)\n\t\tfor _, op := range ops {\n\t\t\terr := op(mod)\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t}\n\tNextField:\n\t\tfor i := 0; i < tpy.NumField(); i++ {\n\t\t\tif tpy.Field(i).Anonymous || tpy.Field(i).PkgPath != \"\" {\n\t\t\t\tcontinue NextField\n\t\t\t}\n\t\t\tif v, ok := val.Field(i).Interface().(time.Time); ok {\n\t\t\t\tmod.fields = append(mod.fields, field{i, tpy.Field(i).Name, v})\n\t\t\t\tcontinue NextField\n\t\t\t} else if v, ok := val.Field(i).Interface().(time.Duration); ok {\n\t\t\t\tmod.fields = append(mod.fields, field{i, tpy.Field(i).Name, v})\n\t\t\t\tcontinue NextField\n\t\t\t}\n\t\t\tswitch val.Field(i).Kind() {\n\t\t\tcase reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Float32, reflect.Float64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.String:\n\t\t\t\tmod.fields = append(mod.fields, field{i, tpy.Field(i).Name, val.Field(i).Kind()})\n\t\t\t}\n\t\t}\n\t\tnl.models = append(nl.models, mod)\n\t\treturn nil\n\t}\n\treturn fmt.Errorf(\"can't create model from non-struct type\")\n}\n\nfunc (m *model) learn() error {\n\tfor sid, s := range m.samples {\n\t\ttokens, err := parser.ParseSample(sid, s)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tvar exps []item\n\t\tvar hasAtLeastOneKey bool\n\t\tl := len(tokens)\n\t\tfor i, tk := range tokens {\n\t\t\tif tk.Kw {\n\t\t\t\thasAtLeastOneKey = true\n\t\t\t\tmistypedField := true\n\t\t\t\tfor _, f := range m.fields {\n\t\t\t\t\tif string(tk.Val) == f.name {\n\t\t\t\t\t\tmistypedField = false\n\t\t\t\t\t\texps = append(exps, item{field: f, value: tk.Val})\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tif mistypedField {\n\t\t\t\t\treturn fmt.Errorf(\"sample#%d: mistyped field %q\", sid, tk.Val)\n\t\t\t\t}\n\t\t\t} else {\n\t\t\t\tif i+1 < l {\n\t\t\t\t\tif tokens[i+1].Kw {\n\t\t\t\t\t\texps = append(exps, item{limit: true, value: tk.Val})\n\t\t\t\t\t\tcontinue\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tif !hasAtLeastOneKey {\n\t\t\treturn fmt.Errorf(\"sample#%d: need at least one keyword\", sid)\n\t\t}\n\t\tm.expected[sid] = exps\n\t}\n\treturn nil\n}\n\nfunc (m *model) selectBestSample(expr []byte) []item {\n\t// slice [sample_id]score\n\tscores := make([]int, len(m.samples))\n\n\ttokens, _ := parser.ParseSample(0, expr)\n\n\tmapping := make([][]item, len(m.samples))\n\tlimitsOrder := make([][][]byte, len(m.samples)+1)\n\n\tfor sid, exps := range m.expected {\n\t\tvar currentVal [][]byte\n\t\tvar reading bool\n\t\tvar lastToken int\n\texpecteds:\n\t\tfor _, e := range exps {\n\t\t\t// fmt.Printf(\"expecting: %s - limit: %v\\n\", e.value, e.limit)\n\t\t\tif e.limit {\n\t\t\t\treading = false\n\t\t\t\tlimitsOrder[sid+1] = append(limitsOrder[sid+1], e.value)\n\t\t\t} else {\n\t\t\t\treading = true\n\t\t\t}\n\t\t\t// fmt.Printf(\"reading: %v\\n\", reading)\n\t\t\tfor i := lastToken; i < len(tokens); i++ {\n\t\t\t\tt := tokens[i]\n\t\t\t\t// fmt.Printf(\"token: %s - isLimit: %v\\n\", t.Val, m.isLimit(t.Val, sid))\n\t\t\t\tif m.isLimit(t.Val, sid) {\n\t\t\t\t\tif sid == 0 {\n\t\t\t\t\t\tlimitsOrder[0] = append(limitsOrder[0], t.Val)\n\t\t\t\t\t}\n\t\t\t\t\tscores[sid]++\n\t\t\t\t\tif len(currentVal) > 0 {\n\t\t\t\t\t\t// fmt.Printf(\"appending: %s {%v}\\n\", bytes.Join(currentVal, []byte{' '}), e.field.name)\n\t\t\t\t\t\tmapping[sid] = append(mapping[sid], item{field: e.field, value: bytes.Join(currentVal, []byte{' '})})\n\t\t\t\t\t\tcurrentVal = currentVal[:0]\n\t\t\t\t\t\tlastToken = i\n\t\t\t\t\t\tcontinue expecteds\n\t\t\t\t\t}\n\t\t\t\t\tlastToken = i + 1\n\t\t\t\t\tcontinue expecteds\n\t\t\t\t} else {\n\t\t\t\t\tif reading {\n\t\t\t\t\t\t// fmt.Printf(\"adding: %s\\n\", t.Val)\n\t\t\t\t\t\tcurrentVal = append(currentVal, t.Val)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\tif len(currentVal) > 0 {\n\t\t\t\t// fmt.Printf(\"appending: %s {%v}\\n\", bytes.Join(currentVal, []byte{' '}), e.field.name)\n\t\t\t\tmapping[sid] = append(mapping[sid], item{field: e.field, value: bytes.Join(currentVal, []byte{' '})})\n\t\t\t}\n\t\t}\n\t\t// fmt.Printf(\"\\n\\n\")\n\t}\norder:\n\tfor i := 1; i < len(limitsOrder); i++ {\n\t\tif len(limitsOrder[0]) < len(limitsOrder[i]) {\n\t\t\tcontinue order\n\t\t}\n\t\tfor j := range limitsOrder[i] {\n\t\t\tif !bytes.Equal(limitsOrder[i][j], limitsOrder[0][j]) {\n\t\t\t\tcontinue order\n\t\t\t}\n\t\t}\n\t\tscores[i-1]++\n\t}\n\n\t// fmt.Printf(\"orders: %s\\n\\n\", limitsOrder)\n\t// fmt.Printf(\"scores: %v\\n\", scores)\n\n\tbestMapping := selectBestMapping(scores)\n\tif bestMapping == -1 {\n\t\treturn nil\n\t}\n\treturn mapping[bestMapping]\n}\n\nfunc selectBestMapping(scores []int) int {\n\tbestScore, bestMapping := -1, -1\n\tfor id, score := range scores {\n\t\tif score > bestScore {\n\t\t\tbestScore = score\n\t\t\tbestMapping = id\n\t\t}\n\t}\n\treturn bestMapping\n}\n\nfunc (m *model) fit(expr string) interface{} {\n\tval := reflect.New(m.tpy)\n\tif len(expr) == 0 {\n\t\treturn val.Interface()\n\t}\n\texps := m.selectBestSample([]byte(expr))\n\tif len(exps) > 0 {\n\t\tfor _, e := range exps {\n\t\t\tswitch t := e.field.kind.(type) {\n\t\t\tcase reflect.Kind:\n\t\t\t\tswitch t {\n\t\t\t\tcase reflect.String:\n\t\t\t\t\tval.Elem().Field(e.field.index).SetString(string(e.value))\n\t\t\t\tcase reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:\n\t\t\t\t\tv, _ := strconv.ParseUint(string(e.value), 10, 0)\n\t\t\t\t\tval.Elem().Field(e.field.index).SetUint(v)\n\t\t\t\tcase reflect.Float32, reflect.Float64:\n\t\t\t\t\tv, _ := strconv.ParseFloat(string(e.value), 64)\n\t\t\t\t\tval.Elem().Field(e.field.index).SetFloat(v)\n\t\t\t\tcase reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:\n\t\t\t\t\tv, _ := strconv.ParseInt(string(e.value), 10, 0)\n\t\t\t\t\tval.Elem().Field(e.field.index).SetInt(v)\n\t\t\t\t}\n\t\t\tcase time.Time:\n\t\t\t\tv, _ := time.ParseInLocation(m.timeFormat, string(e.value), m.timeLocation)\n\t\t\t\tval.Elem().Field(e.field.index).Set(reflect.ValueOf(v))\n\t\t\tcase time.Duration:\n\t\t\t\tv, _ := time.ParseDuration(string(e.value))\n\t\t\t\tval.Elem().Field(e.field.index).Set(reflect.ValueOf(v))\n\t\t\t}\n\t\t}\n\t}\n\treturn val.Interface()\n}\n\n// isLimit returns true if s is a limit on expected[id]\nfunc (m *model) isLimit(s []byte, id int) bool {\n\tfor _, e := range m.expected[id] {\n\t\tif bytes.Equal(e.value, s) {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\n// setSample converts the []string samples to [][]byte\nfunc (m *model) setSamples(samples []string) {\n\tfor _, s := range samples {\n\t\tm.samples = append(m.samples, []byte(s))\n\t}\n}\n"
  },
  {
    "path": "nlp_test.go",
    "content": "package nlp\n\nimport (\n\t\"bytes\"\n\t\"reflect\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/cdipaolo/goml/text\"\n)\n\nfunc failTest(t *testing.T, err error) {\n\tif err != nil {\n\t\tt.Error(err)\n\t}\n}\n\nfunc TestNL_P(t *testing.T) {\n\ttype T struct {\n\t\tString string\n\t\tInt    int\n\t\tUint   uint\n\t\tFloat  float32\n\t\tTime   time.Time\n\t\tDur    time.Duration\n\t}\n\n\ttSamples := []string{\n\t\t\"string {String}\",\n\t\t\"int {Int}\",\n\t\t\"uint {Uint}\",\n\t\t\"float {Float}\",\n\t\t\"time {Time}\",\n\t\t\"dur {Dur}\",\n\t\t\"string {String} int {Int}\",\n\t\t\"string {String} time {Time}\",\n\t\t\"need {String} since {Time}\",\n\t}\n\n\tnl := New()\n\n\terr := nl.RegisterModel(T{}, tSamples)\n\tfailTest(t, err)\n\n\terr = nl.Learn()\n\tfailTest(t, err)\n\n\ttim, err := time.ParseInLocation(\"01-02-2006_3:04pm\", \"05-18-1999_6:42pm\", time.Local)\n\tfailTest(t, err)\n\n\tdur, err := time.ParseDuration(\"4h2m\")\n\tfailTest(t, err)\n\n\tcases := []struct {\n\t\tname       string\n\t\texpression string\n\t\twant       *T\n\t}{\n\t\t0: {\n\t\t\t\"string\",\n\t\t\t\"string Hello World\",\n\t\t\t&T{String: \"Hello World\"},\n\t\t},\n\t\t1: {\n\t\t\t\"int\",\n\t\t\t\"int 42\",\n\t\t\t&T{Int: 42},\n\t\t},\n\t\t2: {\n\t\t\t\"uint\",\n\t\t\t\"uint 43\",\n\t\t\t&T{Uint: 43},\n\t\t},\n\t\t3: {\n\t\t\t\"float\",\n\t\t\t\"float 44\",\n\t\t\t&T{Float: 44},\n\t\t},\n\t\t4: {\n\t\t\t\"time\",\n\t\t\t\"time 05-18-1999_6:42pm\",\n\t\t\t&T{Time: tim},\n\t\t},\n\t\t5: {\n\t\t\t\"duration\",\n\t\t\t\"dur 4h2m\",\n\t\t\t&T{Dur: dur},\n\t\t},\n\t\t6: {\n\t\t\t\"string int\",\n\t\t\t\"string Lmao int 42\",\n\t\t\t&T{\n\t\t\t\tString: \"Lmao\",\n\t\t\t\tInt:    42,\n\t\t\t},\n\t\t},\n\t\t7: {\n\t\t\t\"string time\",\n\t\t\t\"string What's Up Boy time 05-18-1999_6:42pm\",\n\t\t\t&T{\n\t\t\t\tString: \"What's Up Boy\",\n\t\t\t\tTime:   tim,\n\t\t\t},\n\t\t},\n\t\t8: {\n\t\t\t\"word string time\",\n\t\t\t\"Hi, I am Patrice, I need Issue#4 since 05-18-1999_6:42pm\",\n\t\t\t&T{\n\t\t\t\tString: \"Issue#4\",\n\t\t\t\tTime:   tim,\n\t\t\t},\n\t\t},\n\t}\n\tfor i, tt := range cases {\n\t\tt.Run(tt.name, func(t *testing.T) {\n\t\t\tif res := nl.P(tt.expression); !reflect.DeepEqual(res, tt.want) {\n\t\t\t\tt.Errorf(\"test#%d: got %v want %v\", i, res, tt.want)\n\t\t\t}\n\t\t})\n\t}\n}\n\nfunc TestNL_RegisterModel(t *testing.T) {\n\ttype fields struct {\n\t\tmodels []*model\n\t\tnaive  *text.NaiveBayes\n\t\tOutput *bytes.Buffer\n\t}\n\ttype args struct {\n\t\ti       interface{}\n\t\tsamples []string\n\t\tops     []ModelOption\n\t}\n\ttype T struct {\n\t\tunexported int\n\t\tTime       time.Time\n\t}\n\ttests := []struct {\n\t\tname    string\n\t\tfields  fields\n\t\targs    args\n\t\twantErr bool\n\t}{\n\t\t{\n\t\t\t\"nil struct\",\n\t\t\tfields{},\n\t\t\targs{nil, nil, nil},\n\t\t\ttrue,\n\t\t},\n\t\t{\n\t\t\t\"nil samples\",\n\t\t\tfields{},\n\t\t\targs{args{}, nil, nil},\n\t\t\ttrue,\n\t\t},\n\t\t{\n\t\t\t\"non-struct\",\n\t\t\tfields{},\n\t\t\targs{[]int{}, []string{\"\"}, nil},\n\t\t\ttrue,\n\t\t},\n\t\t{\n\t\t\t\"unexported & time.Time\",\n\t\t\tfields{},\n\t\t\targs{T{}, []string{\"\"}, nil},\n\t\t\tfalse,\n\t\t},\n\t\t{\n\t\t\t\"options\",\n\t\t\tfields{},\n\t\t\targs{T{}, []string{\"\"}, []ModelOption{\n\t\t\t\tWithTimeFormat(\"02-01-2006\"),\n\t\t\t\tWithTimeLocation(time.Local),\n\t\t\t}},\n\t\t\tfalse,\n\t\t},\n\t}\n\tfor i, tt := range tests {\n\t\tt.Run(tt.name, func(t *testing.T) {\n\t\t\tnl := &NL{\n\t\t\t\tmodels: tt.fields.models,\n\t\t\t\tnaive:  tt.fields.naive,\n\t\t\t\tOutput: tt.fields.Output,\n\t\t\t}\n\t\t\tif err := nl.RegisterModel(tt.args.i, tt.args.samples, tt.args.ops...); (err != nil) != tt.wantErr {\n\t\t\t\tt.Errorf(\"[%d] NL.RegisterModel() error = %v, wantErr %v\", i, err, tt.wantErr)\n\t\t\t}\n\t\t})\n\t}\n}\n\nfunc TestNL_Learn(t *testing.T) {\n\ttype fields struct {\n\t\tmodels []*model\n\t\tnaive  *text.NaiveBayes\n\t\tOutput *bytes.Buffer\n\t}\n\ttype T struct {\n\t\tName string\n\t}\n\ttests := []struct {\n\t\tname    string\n\t\tfields  fields\n\t\twantErr bool\n\t}{\n\t\t{\n\t\t\t\"no models\",\n\t\t\tfields{},\n\t\t\ttrue,\n\t\t},\n\t\t{\n\t\t\t\"empty model sample\",\n\t\t\tfields{\n\t\t\t\tmodels: []*model{\n\t\t\t\t\t{\n\t\t\t\t\t\tsamples: [][]byte{{}},\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t\tOutput: bytes.NewBufferString(\"\"),\n\t\t\t},\n\t\t\ttrue,\n\t\t},\n\t\t{\n\t\t\t\"mistyped field\",\n\t\t\tfields{\n\t\t\t\tmodels: []*model{\n\t\t\t\t\t{\n\t\t\t\t\t\tsamples: [][]byte{[]byte(\"Hello {Namee}\")},\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t\tOutput: bytes.NewBufferString(\"\"),\n\t\t\t},\n\t\t\ttrue,\n\t\t},\n\t\t{\n\t\t\t\"sample with no keys\",\n\t\t\tfields{\n\t\t\t\tmodels: []*model{\n\t\t\t\t\t{\n\t\t\t\t\t\tsamples: [][]byte{[]byte(\"Hello\")},\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t\tOutput: bytes.NewBufferString(\"\"),\n\t\t\t},\n\t\t\ttrue,\n\t\t},\n\t}\n\tfor i, tt := range tests {\n\t\tt.Run(tt.name, func(t *testing.T) {\n\t\t\tnl := &NL{\n\t\t\t\tmodels: tt.fields.models,\n\t\t\t\tnaive:  tt.fields.naive,\n\t\t\t\tOutput: tt.fields.Output,\n\t\t\t}\n\t\t\tif err := nl.Learn(); (err != nil) != tt.wantErr {\n\t\t\t\tt.Errorf(\"[%d] NL.Learn() error = %v, wantErr %v\", i, err, tt.wantErr)\n\t\t\t}\n\t\t})\n\t}\n}\n\nfunc TestWithTimeFormat(t *testing.T) {\n\ttype args struct {\n\t\tformat string\n\t\tm      *model\n\t}\n\ttests := []struct {\n\t\tname    string\n\t\targs    args\n\t\twantErr bool\n\t}{\n\t\t{\n\t\t\t\"invalid format\",\n\t\t\targs{\"2006 01 02\", &model{}},\n\t\t\ttrue,\n\t\t},\n\t\t{\n\t\t\t\"valid format\",\n\t\t\targs{\"2006\", &model{}},\n\t\t\tfalse,\n\t\t},\n\t}\n\tfor i, tt := range tests {\n\t\tt.Run(tt.name, func(t *testing.T) {\n\t\t\top := WithTimeFormat(tt.args.format)\n\t\t\tif err := op(tt.args.m); (err != nil) != tt.wantErr {\n\t\t\t\tt.Errorf(\"[%d] WithTimeFormat() error = %v, wantErr %v\", i, err, tt.wantErr)\n\t\t\t}\n\t\t})\n\t}\n}\n\nfunc TestWithTimeLocation(t *testing.T) {\n\ttype args struct {\n\t\tloc *time.Location\n\t\tm   *model\n\t}\n\ttests := []struct {\n\t\tname    string\n\t\targs    args\n\t\twantErr bool\n\t}{\n\t\t{\n\t\t\t\"invalid location\",\n\t\t\targs{nil, &model{}},\n\t\t\ttrue,\n\t\t},\n\t\t{\n\t\t\t\"valid format\",\n\t\t\targs{time.Local, &model{}},\n\t\t\tfalse,\n\t\t},\n\t}\n\tfor i, tt := range tests {\n\t\tt.Run(tt.name, func(t *testing.T) {\n\t\t\top := WithTimeLocation(tt.args.loc)\n\t\t\tif err := op(tt.args.m); (err != nil) != tt.wantErr {\n\t\t\t\tt.Errorf(\"[%d] WithTimeFormat() error = %v, wantErr %v\", i, err, tt.wantErr)\n\t\t\t}\n\t\t})\n\t}\n}\n"
  },
  {
    "path": "parser/nlp.peg",
    "content": "{\n// Package parser contains the sample parser for nlp\npackage parser\n\nimport \"fmt\"\nimport \"errors\"\n\n// Token is a sample token\ntype Token struct {\n    Kw bool\n    Val []byte\n}\n\n// ParseSample will return the tokens within the sample\nfunc ParseSample(sampleID int, sample []byte) ([]Token, error) {\n    samplename := fmt.Sprintf(\"sample#%d\", sampleID)\n    tokens, err := Parse(samplename, sample)\n    var errs errList\n    if err != nil {\n        list := err.(errList)\n        for _, err := range list {\n            pe := err.(*parserError)\n            errs.add(fmt.Errorf(\"%s: %v\", samplename, pe.Inner))\n        }\n        return nil, errs\n    }\n    return tokens.([]Token), nil\n}\n\n}\n\nSample \"sample\"\n= vs:(Identifier / Keyword / Spacing)* {\n    if len(vs.([]interface{})) == 0 {\n        return nil, errors.New(\"empty sample\")\n    }\n    var tokens []Token\n    for _, v := range vs.([]interface{}) {\n        switch tk := v.(type) {\n        case Token:\n            tokens = append(tokens, tk)\n        default:\n        }\n    }\n    return tokens, nil\n}\n\nKeyword \"keyword\"\n= '{' Spacing+ v:Identifier '}' {\n    return Token{Kw: true, Val: v.(Token).Val}, nil\n}\n/ '{' v:Identifier Spacing+ '}' {\n    return Token{Kw: true, Val: v.(Token).Val}, nil\n}\n/ '{' Spacing+ v:Identifier Spacing+ '}' {\n    return Token{Kw: true, Val: v.(Token).Val}, nil\n}\n/ '{' v:Identifier '}' {\n    return Token{Kw: true, Val: v.(Token).Val}, nil\n}\n\n\nPunct \"punct\"\n= [^a-zA-Z0-9{} ]+ {\n    return Token{Val: c.text}, nil\n}\n\n\nIdentifier \"identifier\"\n= Punct / [^{} \\t\\r\\n]+ {\n    return Token{Val: c.text}, nil\n}\n\nSpacing \"spacing\"\n= Space+ / _+\n\nSpace \"Space\"\n= ' '\n\n_ \"whitespace\"\n= [\\t\\r\\n]\n"
  },
  {
    "path": "parser/parser.go",
    "content": "// Package parser contains the sample parser for nlp\npackage parser\n\nimport (\n\t\"bytes\"\n\t\"errors\"\n\t\"fmt\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"math\"\n\t\"os\"\n\t\"sort\"\n\t\"strings\"\n\t\"unicode\"\n\t\"unicode/utf8\"\n)\n\n// Token is a sample token\ntype Token struct {\n\tKw  bool\n\tVal []byte\n}\n\n// ParseSample will return the tokens within the sample\nfunc ParseSample(sampleID int, sample []byte) ([]Token, error) {\n\tsamplename := fmt.Sprintf(\"sample#%d\", sampleID)\n\ttokens, err := Parse(samplename, sample)\n\tvar errs errList\n\tif err != nil {\n\t\tlist := err.(errList)\n\t\tfor _, err := range list {\n\t\t\tpe := err.(*parserError)\n\t\t\terrs.add(fmt.Errorf(\"%s: %v\", samplename, pe.Inner))\n\t\t}\n\t\treturn nil, errs\n\t}\n\treturn tokens.([]Token), nil\n}\n\nvar g = &grammar{\n\trules: []*rule{\n\t\t{\n\t\t\tname:        \"Sample\",\n\t\t\tdisplayName: \"\\\"sample\\\"\",\n\t\t\tpos:         position{line: 32, col: 1, offset: 685},\n\t\t\texpr: &actionExpr{\n\t\t\t\tpos: position{line: 33, col: 3, offset: 703},\n\t\t\t\trun: (*parser).callonSample1,\n\t\t\t\texpr: &labeledExpr{\n\t\t\t\t\tpos:   position{line: 33, col: 3, offset: 703},\n\t\t\t\t\tlabel: \"vs\",\n\t\t\t\t\texpr: &zeroOrMoreExpr{\n\t\t\t\t\t\tpos: position{line: 33, col: 6, offset: 706},\n\t\t\t\t\t\texpr: &choiceExpr{\n\t\t\t\t\t\t\tpos: position{line: 33, col: 7, offset: 707},\n\t\t\t\t\t\t\talternatives: []interface{}{\n\t\t\t\t\t\t\t\t&ruleRefExpr{\n\t\t\t\t\t\t\t\t\tpos:  position{line: 33, col: 7, offset: 707},\n\t\t\t\t\t\t\t\t\tname: \"Identifier\",\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&ruleRefExpr{\n\t\t\t\t\t\t\t\t\tpos:  position{line: 33, col: 20, offset: 720},\n\t\t\t\t\t\t\t\t\tname: \"Keyword\",\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&ruleRefExpr{\n\t\t\t\t\t\t\t\t\tpos:  position{line: 33, col: 30, offset: 730},\n\t\t\t\t\t\t\t\t\tname: \"Spacing\",\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tname:        \"Keyword\",\n\t\t\tdisplayName: \"\\\"keyword\\\"\",\n\t\t\tpos:         position{line: 48, col: 1, offset: 1050},\n\t\t\texpr: &choiceExpr{\n\t\t\t\tpos: position{line: 49, col: 3, offset: 1070},\n\t\t\t\talternatives: []interface{}{\n\t\t\t\t\t&actionExpr{\n\t\t\t\t\t\tpos: position{line: 49, col: 3, offset: 1070},\n\t\t\t\t\t\trun: (*parser).callonKeyword2,\n\t\t\t\t\t\texpr: &seqExpr{\n\t\t\t\t\t\t\tpos: position{line: 49, col: 3, offset: 1070},\n\t\t\t\t\t\t\texprs: []interface{}{\n\t\t\t\t\t\t\t\t&litMatcher{\n\t\t\t\t\t\t\t\t\tpos:        position{line: 49, col: 3, offset: 1070},\n\t\t\t\t\t\t\t\t\tval:        \"{\",\n\t\t\t\t\t\t\t\t\tignoreCase: false,\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&oneOrMoreExpr{\n\t\t\t\t\t\t\t\t\tpos: position{line: 49, col: 7, offset: 1074},\n\t\t\t\t\t\t\t\t\texpr: &ruleRefExpr{\n\t\t\t\t\t\t\t\t\t\tpos:  position{line: 49, col: 7, offset: 1074},\n\t\t\t\t\t\t\t\t\t\tname: \"Spacing\",\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&labeledExpr{\n\t\t\t\t\t\t\t\t\tpos:   position{line: 49, col: 16, offset: 1083},\n\t\t\t\t\t\t\t\t\tlabel: \"v\",\n\t\t\t\t\t\t\t\t\texpr: &ruleRefExpr{\n\t\t\t\t\t\t\t\t\t\tpos:  position{line: 49, col: 18, offset: 1085},\n\t\t\t\t\t\t\t\t\t\tname: \"Identifier\",\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&litMatcher{\n\t\t\t\t\t\t\t\t\tpos:        position{line: 49, col: 29, offset: 1096},\n\t\t\t\t\t\t\t\t\tval:        \"}\",\n\t\t\t\t\t\t\t\t\tignoreCase: false,\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t\t&actionExpr{\n\t\t\t\t\t\tpos: position{line: 52, col: 3, offset: 1158},\n\t\t\t\t\t\trun: (*parser).callonKeyword10,\n\t\t\t\t\t\texpr: &seqExpr{\n\t\t\t\t\t\t\tpos: position{line: 52, col: 3, offset: 1158},\n\t\t\t\t\t\t\texprs: []interface{}{\n\t\t\t\t\t\t\t\t&litMatcher{\n\t\t\t\t\t\t\t\t\tpos:        position{line: 52, col: 3, offset: 1158},\n\t\t\t\t\t\t\t\t\tval:        \"{\",\n\t\t\t\t\t\t\t\t\tignoreCase: false,\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&labeledExpr{\n\t\t\t\t\t\t\t\t\tpos:   position{line: 52, col: 7, offset: 1162},\n\t\t\t\t\t\t\t\t\tlabel: \"v\",\n\t\t\t\t\t\t\t\t\texpr: &ruleRefExpr{\n\t\t\t\t\t\t\t\t\t\tpos:  position{line: 52, col: 9, offset: 1164},\n\t\t\t\t\t\t\t\t\t\tname: \"Identifier\",\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&oneOrMoreExpr{\n\t\t\t\t\t\t\t\t\tpos: position{line: 52, col: 20, offset: 1175},\n\t\t\t\t\t\t\t\t\texpr: &ruleRefExpr{\n\t\t\t\t\t\t\t\t\t\tpos:  position{line: 52, col: 20, offset: 1175},\n\t\t\t\t\t\t\t\t\t\tname: \"Spacing\",\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&litMatcher{\n\t\t\t\t\t\t\t\t\tpos:        position{line: 52, col: 29, offset: 1184},\n\t\t\t\t\t\t\t\t\tval:        \"}\",\n\t\t\t\t\t\t\t\t\tignoreCase: false,\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t\t&actionExpr{\n\t\t\t\t\t\tpos: position{line: 55, col: 3, offset: 1246},\n\t\t\t\t\t\trun: (*parser).callonKeyword18,\n\t\t\t\t\t\texpr: &seqExpr{\n\t\t\t\t\t\t\tpos: position{line: 55, col: 3, offset: 1246},\n\t\t\t\t\t\t\texprs: []interface{}{\n\t\t\t\t\t\t\t\t&litMatcher{\n\t\t\t\t\t\t\t\t\tpos:        position{line: 55, col: 3, offset: 1246},\n\t\t\t\t\t\t\t\t\tval:        \"{\",\n\t\t\t\t\t\t\t\t\tignoreCase: false,\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&oneOrMoreExpr{\n\t\t\t\t\t\t\t\t\tpos: position{line: 55, col: 7, offset: 1250},\n\t\t\t\t\t\t\t\t\texpr: &ruleRefExpr{\n\t\t\t\t\t\t\t\t\t\tpos:  position{line: 55, col: 7, offset: 1250},\n\t\t\t\t\t\t\t\t\t\tname: \"Spacing\",\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&labeledExpr{\n\t\t\t\t\t\t\t\t\tpos:   position{line: 55, col: 16, offset: 1259},\n\t\t\t\t\t\t\t\t\tlabel: \"v\",\n\t\t\t\t\t\t\t\t\texpr: &ruleRefExpr{\n\t\t\t\t\t\t\t\t\t\tpos:  position{line: 55, col: 18, offset: 1261},\n\t\t\t\t\t\t\t\t\t\tname: \"Identifier\",\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&oneOrMoreExpr{\n\t\t\t\t\t\t\t\t\tpos: position{line: 55, col: 29, offset: 1272},\n\t\t\t\t\t\t\t\t\texpr: &ruleRefExpr{\n\t\t\t\t\t\t\t\t\t\tpos:  position{line: 55, col: 29, offset: 1272},\n\t\t\t\t\t\t\t\t\t\tname: \"Spacing\",\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&litMatcher{\n\t\t\t\t\t\t\t\t\tpos:        position{line: 55, col: 38, offset: 1281},\n\t\t\t\t\t\t\t\t\tval:        \"}\",\n\t\t\t\t\t\t\t\t\tignoreCase: false,\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t\t&actionExpr{\n\t\t\t\t\t\tpos: position{line: 58, col: 3, offset: 1343},\n\t\t\t\t\t\trun: (*parser).callonKeyword28,\n\t\t\t\t\t\texpr: &seqExpr{\n\t\t\t\t\t\t\tpos: position{line: 58, col: 3, offset: 1343},\n\t\t\t\t\t\t\texprs: []interface{}{\n\t\t\t\t\t\t\t\t&litMatcher{\n\t\t\t\t\t\t\t\t\tpos:        position{line: 58, col: 3, offset: 1343},\n\t\t\t\t\t\t\t\t\tval:        \"{\",\n\t\t\t\t\t\t\t\t\tignoreCase: false,\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&labeledExpr{\n\t\t\t\t\t\t\t\t\tpos:   position{line: 58, col: 7, offset: 1347},\n\t\t\t\t\t\t\t\t\tlabel: \"v\",\n\t\t\t\t\t\t\t\t\texpr: &ruleRefExpr{\n\t\t\t\t\t\t\t\t\t\tpos:  position{line: 58, col: 9, offset: 1349},\n\t\t\t\t\t\t\t\t\t\tname: \"Identifier\",\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t&litMatcher{\n\t\t\t\t\t\t\t\t\tpos:        position{line: 58, col: 20, offset: 1360},\n\t\t\t\t\t\t\t\t\tval:        \"}\",\n\t\t\t\t\t\t\t\t\tignoreCase: false,\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tname:        \"Punct\",\n\t\t\tdisplayName: \"\\\"punct\\\"\",\n\t\t\tpos:         position{line: 63, col: 1, offset: 1422},\n\t\t\texpr: &actionExpr{\n\t\t\t\tpos: position{line: 64, col: 3, offset: 1438},\n\t\t\t\trun: (*parser).callonPunct1,\n\t\t\t\texpr: &oneOrMoreExpr{\n\t\t\t\t\tpos: position{line: 64, col: 3, offset: 1438},\n\t\t\t\t\texpr: &charClassMatcher{\n\t\t\t\t\t\tpos:        position{line: 64, col: 3, offset: 1438},\n\t\t\t\t\t\tval:        \"[^a-zA-Z0-9{} ]\",\n\t\t\t\t\t\tchars:      []rune{'{', '}', ' '},\n\t\t\t\t\t\tranges:     []rune{'a', 'z', 'A', 'Z', '0', '9'},\n\t\t\t\t\t\tignoreCase: false,\n\t\t\t\t\t\tinverted:   true,\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tname:        \"Identifier\",\n\t\t\tdisplayName: \"\\\"identifier\\\"\",\n\t\t\tpos:         position{line: 69, col: 1, offset: 1496},\n\t\t\texpr: &choiceExpr{\n\t\t\t\tpos: position{line: 70, col: 3, offset: 1522},\n\t\t\t\talternatives: []interface{}{\n\t\t\t\t\t&ruleRefExpr{\n\t\t\t\t\t\tpos:  position{line: 70, col: 3, offset: 1522},\n\t\t\t\t\t\tname: \"Punct\",\n\t\t\t\t\t},\n\t\t\t\t\t&actionExpr{\n\t\t\t\t\t\tpos: position{line: 70, col: 11, offset: 1530},\n\t\t\t\t\t\trun: (*parser).callonIdentifier3,\n\t\t\t\t\t\texpr: &oneOrMoreExpr{\n\t\t\t\t\t\t\tpos: position{line: 70, col: 11, offset: 1530},\n\t\t\t\t\t\t\texpr: &charClassMatcher{\n\t\t\t\t\t\t\t\tpos:        position{line: 70, col: 11, offset: 1530},\n\t\t\t\t\t\t\t\tval:        \"[^{} \\\\t\\\\r\\\\n]\",\n\t\t\t\t\t\t\t\tchars:      []rune{'{', '}', ' ', '\\t', '\\r', '\\n'},\n\t\t\t\t\t\t\t\tignoreCase: false,\n\t\t\t\t\t\t\t\tinverted:   true,\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tname:        \"Spacing\",\n\t\t\tdisplayName: \"\\\"spacing\\\"\",\n\t\t\tpos:         position{line: 74, col: 1, offset: 1584},\n\t\t\texpr: &choiceExpr{\n\t\t\t\tpos: position{line: 75, col: 3, offset: 1604},\n\t\t\t\talternatives: []interface{}{\n\t\t\t\t\t&oneOrMoreExpr{\n\t\t\t\t\t\tpos: position{line: 75, col: 3, offset: 1604},\n\t\t\t\t\t\texpr: &ruleRefExpr{\n\t\t\t\t\t\t\tpos:  position{line: 75, col: 3, offset: 1604},\n\t\t\t\t\t\t\tname: \"Space\",\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t\t&oneOrMoreExpr{\n\t\t\t\t\t\tpos: position{line: 75, col: 12, offset: 1613},\n\t\t\t\t\t\texpr: &ruleRefExpr{\n\t\t\t\t\t\t\tpos:  position{line: 75, col: 12, offset: 1613},\n\t\t\t\t\t\t\tname: \"_\",\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tname:        \"Space\",\n\t\t\tdisplayName: \"\\\"Space\\\"\",\n\t\t\tpos:         position{line: 77, col: 1, offset: 1617},\n\t\t\texpr: &litMatcher{\n\t\t\t\tpos:        position{line: 78, col: 3, offset: 1633},\n\t\t\t\tval:        \" \",\n\t\t\t\tignoreCase: false,\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tname:        \"_\",\n\t\t\tdisplayName: \"\\\"whitespace\\\"\",\n\t\t\tpos:         position{line: 80, col: 1, offset: 1638},\n\t\t\texpr: &charClassMatcher{\n\t\t\t\tpos:        position{line: 81, col: 3, offset: 1655},\n\t\t\t\tval:        \"[\\\\t\\\\r\\\\n]\",\n\t\t\t\tchars:      []rune{'\\t', '\\r', '\\n'},\n\t\t\t\tignoreCase: false,\n\t\t\t\tinverted:   false,\n\t\t\t},\n\t\t},\n\t},\n}\n\nfunc (c *current) onSample1(vs interface{}) (interface{}, error) {\n\tif len(vs.([]interface{})) == 0 {\n\t\treturn nil, errors.New(\"empty sample\")\n\t}\n\tvar tokens []Token\n\tfor _, v := range vs.([]interface{}) {\n\t\tswitch tk := v.(type) {\n\t\tcase Token:\n\t\t\ttokens = append(tokens, tk)\n\t\tdefault:\n\t\t}\n\t}\n\treturn tokens, nil\n}\n\nfunc (p *parser) callonSample1() (interface{}, error) {\n\tstack := p.vstack[len(p.vstack)-1]\n\t_ = stack\n\treturn p.cur.onSample1(stack[\"vs\"])\n}\n\nfunc (c *current) onKeyword2(v interface{}) (interface{}, error) {\n\treturn Token{Kw: true, Val: v.(Token).Val}, nil\n}\n\nfunc (p *parser) callonKeyword2() (interface{}, error) {\n\tstack := p.vstack[len(p.vstack)-1]\n\t_ = stack\n\treturn p.cur.onKeyword2(stack[\"v\"])\n}\n\nfunc (c *current) onKeyword10(v interface{}) (interface{}, error) {\n\treturn Token{Kw: true, Val: v.(Token).Val}, nil\n}\n\nfunc (p *parser) callonKeyword10() (interface{}, error) {\n\tstack := p.vstack[len(p.vstack)-1]\n\t_ = stack\n\treturn p.cur.onKeyword10(stack[\"v\"])\n}\n\nfunc (c *current) onKeyword18(v interface{}) (interface{}, error) {\n\treturn Token{Kw: true, Val: v.(Token).Val}, nil\n}\n\nfunc (p *parser) callonKeyword18() (interface{}, error) {\n\tstack := p.vstack[len(p.vstack)-1]\n\t_ = stack\n\treturn p.cur.onKeyword18(stack[\"v\"])\n}\n\nfunc (c *current) onKeyword28(v interface{}) (interface{}, error) {\n\treturn Token{Kw: true, Val: v.(Token).Val}, nil\n}\n\nfunc (p *parser) callonKeyword28() (interface{}, error) {\n\tstack := p.vstack[len(p.vstack)-1]\n\t_ = stack\n\treturn p.cur.onKeyword28(stack[\"v\"])\n}\n\nfunc (c *current) onPunct1() (interface{}, error) {\n\treturn Token{Val: c.text}, nil\n}\n\nfunc (p *parser) callonPunct1() (interface{}, error) {\n\tstack := p.vstack[len(p.vstack)-1]\n\t_ = stack\n\treturn p.cur.onPunct1()\n}\n\nfunc (c *current) onIdentifier3() (interface{}, error) {\n\treturn Token{Val: c.text}, nil\n}\n\nfunc (p *parser) callonIdentifier3() (interface{}, error) {\n\tstack := p.vstack[len(p.vstack)-1]\n\t_ = stack\n\treturn p.cur.onIdentifier3()\n}\n\nvar (\n\t// errNoRule is returned when the grammar to parse has no rule.\n\terrNoRule = errors.New(\"grammar has no rule\")\n\n\t// errInvalidEncoding is returned when the source is not properly\n\t// utf8-encoded.\n\terrInvalidEncoding = errors.New(\"invalid encoding\")\n\n\t// errMaxExprCnt is used to signal that the maximum number of\n\t// expressions have been parsed.\n\terrMaxExprCnt = errors.New(\"max number of expresssions parsed\")\n)\n\n// Option is a function that can set an option on the parser. It returns\n// the previous setting as an Option.\ntype Option func(*parser) Option\n\n// MaxExpressions creates an Option to stop parsing after the provided\n// number of expressions have been parsed, if the value is 0 then the parser will\n// parse for as many steps as needed (possibly an infinite number).\n//\n// The default for maxExprCnt is 0.\nfunc MaxExpressions(maxExprCnt uint64) Option {\n\treturn func(p *parser) Option {\n\t\toldMaxExprCnt := p.maxExprCnt\n\t\tp.maxExprCnt = maxExprCnt\n\t\treturn MaxExpressions(oldMaxExprCnt)\n\t}\n}\n\n// Debug creates an Option to set the debug flag to b. When set to true,\n// debugging information is printed to stdout while parsing.\n//\n// The default is false.\nfunc Debug(b bool) Option {\n\treturn func(p *parser) Option {\n\t\told := p.debug\n\t\tp.debug = b\n\t\treturn Debug(old)\n\t}\n}\n\n// Memoize creates an Option to set the memoize flag to b. When set to true,\n// the parser will cache all results so each expression is evaluated only\n// once. This guarantees linear parsing time even for pathological cases,\n// at the expense of more memory and slower times for typical cases.\n//\n// The default is false.\nfunc Memoize(b bool) Option {\n\treturn func(p *parser) Option {\n\t\told := p.memoize\n\t\tp.memoize = b\n\t\treturn Memoize(old)\n\t}\n}\n\n// Recover creates an Option to set the recover flag to b. When set to\n// true, this causes the parser to recover from panics and convert it\n// to an error. Setting it to false can be useful while debugging to\n// access the full stack trace.\n//\n// The default is true.\nfunc Recover(b bool) Option {\n\treturn func(p *parser) Option {\n\t\told := p.recover\n\t\tp.recover = b\n\t\treturn Recover(old)\n\t}\n}\n\n// GlobalStore creates an Option to set a key to a certain value in\n// the globalStore.\nfunc GlobalStore(key string, value interface{}) Option {\n\treturn func(p *parser) Option {\n\t\told := p.cur.globalStore[key]\n\t\tp.cur.globalStore[key] = value\n\t\treturn GlobalStore(key, old)\n\t}\n}\n\n// ParseFile parses the file identified by filename.\nfunc ParseFile(filename string, opts ...Option) (i interface{}, err error) {\n\tf, err := os.Open(filename)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tdefer func() {\n\t\tif closeErr := f.Close(); closeErr != nil {\n\t\t\terr = closeErr\n\t\t}\n\t}()\n\treturn ParseReader(filename, f, opts...)\n}\n\n// ParseReader parses the data from r using filename as information in the\n// error messages.\nfunc ParseReader(filename string, r io.Reader, opts ...Option) (interface{}, error) {\n\tb, err := ioutil.ReadAll(r)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn Parse(filename, b, opts...)\n}\n\n// Parse parses the data from b using filename as information in the\n// error messages.\nfunc Parse(filename string, b []byte, opts ...Option) (interface{}, error) {\n\treturn newParser(filename, b, opts...).parse(g)\n}\n\n// position records a position in the text.\ntype position struct {\n\tline, col, offset int\n}\n\nfunc (p position) String() string {\n\treturn fmt.Sprintf(\"%d:%d [%d]\", p.line, p.col, p.offset)\n}\n\n// savepoint stores all state required to go back to this point in the\n// parser.\ntype savepoint struct {\n\tposition\n\trn rune\n\tw  int\n}\n\ntype current struct {\n\tpos  position // start position of the match\n\ttext []byte   // raw text of the match\n\n\t// the globalStore allows the parser to store arbitrary values\n\tglobalStore map[string]interface{}\n}\n\n// the AST types...\n\ntype grammar struct {\n\tpos   position\n\trules []*rule\n}\n\ntype rule struct {\n\tpos         position\n\tname        string\n\tdisplayName string\n\texpr        interface{}\n}\n\ntype choiceExpr struct {\n\tpos          position\n\talternatives []interface{}\n}\n\ntype actionExpr struct {\n\tpos  position\n\texpr interface{}\n\trun  func(*parser) (interface{}, error)\n}\n\ntype seqExpr struct {\n\tpos   position\n\texprs []interface{}\n}\n\ntype labeledExpr struct {\n\tpos   position\n\tlabel string\n\texpr  interface{}\n}\n\ntype expr struct {\n\tpos  position\n\texpr interface{}\n}\n\ntype andExpr expr\ntype notExpr expr\ntype zeroOrOneExpr expr\ntype zeroOrMoreExpr expr\ntype oneOrMoreExpr expr\n\ntype ruleRefExpr struct {\n\tpos  position\n\tname string\n}\n\ntype andCodeExpr struct {\n\tpos position\n\trun func(*parser) (bool, error)\n}\n\ntype notCodeExpr struct {\n\tpos position\n\trun func(*parser) (bool, error)\n}\n\ntype litMatcher struct {\n\tpos        position\n\tval        string\n\tignoreCase bool\n}\n\ntype charClassMatcher struct {\n\tpos             position\n\tval             string\n\tbasicLatinChars [128]bool\n\tchars           []rune\n\tranges          []rune\n\tclasses         []*unicode.RangeTable\n\tignoreCase      bool\n\tinverted        bool\n}\n\ntype anyMatcher position\n\n// errList cumulates the errors found by the parser.\ntype errList []error\n\nfunc (e *errList) add(err error) {\n\t*e = append(*e, err)\n}\n\nfunc (e errList) err() error {\n\tif len(e) == 0 {\n\t\treturn nil\n\t}\n\te.dedupe()\n\treturn e\n}\n\nfunc (e *errList) dedupe() {\n\tvar cleaned []error\n\tset := make(map[string]bool)\n\tfor _, err := range *e {\n\t\tif msg := err.Error(); !set[msg] {\n\t\t\tset[msg] = true\n\t\t\tcleaned = append(cleaned, err)\n\t\t}\n\t}\n\t*e = cleaned\n}\n\nfunc (e errList) Error() string {\n\tswitch len(e) {\n\tcase 0:\n\t\treturn \"\"\n\tcase 1:\n\t\treturn e[0].Error()\n\tdefault:\n\t\tvar buf bytes.Buffer\n\n\t\tfor i, err := range e {\n\t\t\tif i > 0 {\n\t\t\t\tbuf.WriteRune('\\n')\n\t\t\t}\n\t\t\tbuf.WriteString(err.Error())\n\t\t}\n\t\treturn buf.String()\n\t}\n}\n\n// parserError wraps an error with a prefix indicating the rule in which\n// the error occurred. The original error is stored in the Inner field.\ntype parserError struct {\n\tInner    error\n\tpos      position\n\tprefix   string\n\texpected []string\n}\n\n// Error returns the error message.\nfunc (p *parserError) Error() string {\n\treturn p.prefix + \": \" + p.Inner.Error()\n}\n\n// newParser creates a parser with the specified input source and options.\nfunc newParser(filename string, b []byte, opts ...Option) *parser {\n\tp := &parser{\n\t\tfilename: filename,\n\t\terrs:     new(errList),\n\t\tdata:     b,\n\t\tpt:       savepoint{position: position{line: 1}},\n\t\trecover:  true,\n\t\tcur: current{\n\t\t\tglobalStore: make(map[string]interface{}),\n\t\t},\n\t\tmaxFailPos:      position{col: 1, line: 1},\n\t\tmaxFailExpected: make([]string, 0, 20),\n\t}\n\tp.setOptions(opts)\n\n\tif p.maxExprCnt == 0 {\n\t\tp.maxExprCnt = math.MaxUint64\n\t}\n\n\treturn p\n}\n\n// setOptions applies the options to the parser.\nfunc (p *parser) setOptions(opts []Option) {\n\tfor _, opt := range opts {\n\t\topt(p)\n\t}\n}\n\ntype resultTuple struct {\n\tv   interface{}\n\tb   bool\n\tend savepoint\n}\n\ntype parser struct {\n\tfilename string\n\tpt       savepoint\n\tcur      current\n\n\tdata []byte\n\terrs *errList\n\n\tdepth   int\n\trecover bool\n\tdebug   bool\n\n\tmemoize bool\n\t// memoization table for the packrat algorithm:\n\t// map[offset in source] map[expression or rule] {value, match}\n\tmemo map[int]map[interface{}]resultTuple\n\n\t// rules table, maps the rule identifier to the rule node\n\trules map[string]*rule\n\t// variables stack, map of label to value\n\tvstack []map[string]interface{}\n\t// rule stack, allows identification of the current rule in errors\n\trstack []*rule\n\n\t// parse fail\n\tmaxFailPos            position\n\tmaxFailExpected       []string\n\tmaxFailInvertExpected bool\n\n\t// stats and used for stopping the parser\n\t// after a maximum number of expressions are parsed\n\texprCnt uint64\n\n\t// max number of expressions to be parsed\n\tmaxExprCnt uint64\n}\n\n// push a variable set on the vstack.\nfunc (p *parser) pushV() {\n\tif cap(p.vstack) == len(p.vstack) {\n\t\t// create new empty slot in the stack\n\t\tp.vstack = append(p.vstack, nil)\n\t} else {\n\t\t// slice to 1 more\n\t\tp.vstack = p.vstack[:len(p.vstack)+1]\n\t}\n\n\t// get the last args set\n\tm := p.vstack[len(p.vstack)-1]\n\tif m != nil && len(m) == 0 {\n\t\t// empty map, all good\n\t\treturn\n\t}\n\n\tm = make(map[string]interface{})\n\tp.vstack[len(p.vstack)-1] = m\n}\n\n// pop a variable set from the vstack.\nfunc (p *parser) popV() {\n\t// if the map is not empty, clear it\n\tm := p.vstack[len(p.vstack)-1]\n\tif len(m) > 0 {\n\t\t// GC that map\n\t\tp.vstack[len(p.vstack)-1] = nil\n\t}\n\tp.vstack = p.vstack[:len(p.vstack)-1]\n}\n\nfunc (p *parser) print(prefix, s string) string {\n\tif !p.debug {\n\t\treturn s\n\t}\n\n\tfmt.Printf(\"%s %d:%d:%d: %s [%#U]\\n\",\n\t\tprefix, p.pt.line, p.pt.col, p.pt.offset, s, p.pt.rn)\n\treturn s\n}\n\nfunc (p *parser) in(s string) string {\n\tp.depth++\n\treturn p.print(strings.Repeat(\" \", p.depth)+\">\", s)\n}\n\nfunc (p *parser) out(s string) string {\n\tp.depth--\n\treturn p.print(strings.Repeat(\" \", p.depth)+\"<\", s)\n}\n\nfunc (p *parser) addErr(err error) {\n\tp.addErrAt(err, p.pt.position, []string{})\n}\n\nfunc (p *parser) addErrAt(err error, pos position, expected []string) {\n\tvar buf bytes.Buffer\n\tif p.filename != \"\" {\n\t\tbuf.WriteString(p.filename)\n\t}\n\tif buf.Len() > 0 {\n\t\tbuf.WriteString(\":\")\n\t}\n\tbuf.WriteString(fmt.Sprintf(\"%d:%d (%d)\", pos.line, pos.col, pos.offset))\n\tif len(p.rstack) > 0 {\n\t\tif buf.Len() > 0 {\n\t\t\tbuf.WriteString(\": \")\n\t\t}\n\t\trule := p.rstack[len(p.rstack)-1]\n\t\tif rule.displayName != \"\" {\n\t\t\tbuf.WriteString(\"rule \" + rule.displayName)\n\t\t} else {\n\t\t\tbuf.WriteString(\"rule \" + rule.name)\n\t\t}\n\t}\n\tpe := &parserError{Inner: err, pos: pos, prefix: buf.String(), expected: expected}\n\tp.errs.add(pe)\n}\n\nfunc (p *parser) failAt(fail bool, pos position, want string) {\n\t// process fail if parsing fails and not inverted or parsing succeeds and invert is set\n\tif fail == p.maxFailInvertExpected {\n\t\tif pos.offset < p.maxFailPos.offset {\n\t\t\treturn\n\t\t}\n\n\t\tif pos.offset > p.maxFailPos.offset {\n\t\t\tp.maxFailPos = pos\n\t\t\tp.maxFailExpected = p.maxFailExpected[:0]\n\t\t}\n\n\t\tif p.maxFailInvertExpected {\n\t\t\twant = \"!\" + want\n\t\t}\n\t\tp.maxFailExpected = append(p.maxFailExpected, want)\n\t}\n}\n\n// read advances the parser to the next rune.\nfunc (p *parser) read() {\n\tp.pt.offset += p.pt.w\n\trn, n := utf8.DecodeRune(p.data[p.pt.offset:])\n\tp.pt.rn = rn\n\tp.pt.w = n\n\tp.pt.col++\n\tif rn == '\\n' {\n\t\tp.pt.line++\n\t\tp.pt.col = 0\n\t}\n\n\tif rn == utf8.RuneError {\n\t\tif n == 1 {\n\t\t\tp.addErr(errInvalidEncoding)\n\t\t}\n\t}\n}\n\n// restore parser position to the savepoint pt.\nfunc (p *parser) restore(pt savepoint) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"restore\"))\n\t}\n\tif pt.offset == p.pt.offset {\n\t\treturn\n\t}\n\tp.pt = pt\n}\n\n// get the slice of bytes from the savepoint start to the current position.\nfunc (p *parser) sliceFrom(start savepoint) []byte {\n\treturn p.data[start.position.offset:p.pt.position.offset]\n}\n\nfunc (p *parser) getMemoized(node interface{}) (resultTuple, bool) {\n\tif len(p.memo) == 0 {\n\t\treturn resultTuple{}, false\n\t}\n\tm := p.memo[p.pt.offset]\n\tif len(m) == 0 {\n\t\treturn resultTuple{}, false\n\t}\n\tres, ok := m[node]\n\treturn res, ok\n}\n\nfunc (p *parser) setMemoized(pt savepoint, node interface{}, tuple resultTuple) {\n\tif p.memo == nil {\n\t\tp.memo = make(map[int]map[interface{}]resultTuple)\n\t}\n\tm := p.memo[pt.offset]\n\tif m == nil {\n\t\tm = make(map[interface{}]resultTuple)\n\t\tp.memo[pt.offset] = m\n\t}\n\tm[node] = tuple\n}\n\nfunc (p *parser) buildRulesTable(g *grammar) {\n\tp.rules = make(map[string]*rule, len(g.rules))\n\tfor _, r := range g.rules {\n\t\tp.rules[r.name] = r\n\t}\n}\n\nfunc (p *parser) parse(g *grammar) (val interface{}, err error) {\n\tif len(g.rules) == 0 {\n\t\tp.addErr(errNoRule)\n\t\treturn nil, p.errs.err()\n\t}\n\n\t// TODO : not super critical but this could be generated\n\tp.buildRulesTable(g)\n\n\tif p.recover {\n\t\t// panic can be used in action code to stop parsing immediately\n\t\t// and return the panic as an error.\n\t\tdefer func() {\n\t\t\tif e := recover(); e != nil {\n\t\t\t\tif p.debug {\n\t\t\t\t\tdefer p.out(p.in(\"panic handler\"))\n\t\t\t\t}\n\t\t\t\tval = nil\n\t\t\t\tswitch e := e.(type) {\n\t\t\t\tcase error:\n\t\t\t\t\tp.addErr(e)\n\t\t\t\tdefault:\n\t\t\t\t\tp.addErr(fmt.Errorf(\"%v\", e))\n\t\t\t\t}\n\t\t\t\terr = p.errs.err()\n\t\t\t}\n\t\t}()\n\t}\n\n\t// start rule is rule [0]\n\tp.read() // advance to first rune\n\tval, ok := p.parseRule(g.rules[0])\n\tif !ok {\n\t\tif len(*p.errs) == 0 {\n\t\t\t// If parsing fails, but no errors have been recorded, the expected values\n\t\t\t// for the farthest parser position are returned as error.\n\t\t\tmaxFailExpectedMap := make(map[string]struct{}, len(p.maxFailExpected))\n\t\t\tfor _, v := range p.maxFailExpected {\n\t\t\t\tmaxFailExpectedMap[v] = struct{}{}\n\t\t\t}\n\t\t\texpected := make([]string, 0, len(maxFailExpectedMap))\n\t\t\teof := false\n\t\t\tif _, ok := maxFailExpectedMap[\"!.\"]; ok {\n\t\t\t\tdelete(maxFailExpectedMap, \"!.\")\n\t\t\t\teof = true\n\t\t\t}\n\t\t\tfor k := range maxFailExpectedMap {\n\t\t\t\texpected = append(expected, k)\n\t\t\t}\n\t\t\tsort.Strings(expected)\n\t\t\tif eof {\n\t\t\t\texpected = append(expected, \"EOF\")\n\t\t\t}\n\t\t\tp.addErrAt(errors.New(\"no match found, expected: \"+listJoin(expected, \", \", \"or\")), p.maxFailPos, expected)\n\t\t}\n\n\t\treturn nil, p.errs.err()\n\t}\n\treturn val, p.errs.err()\n}\n\nfunc listJoin(list []string, sep string, lastSep string) string {\n\tswitch len(list) {\n\tcase 0:\n\t\treturn \"\"\n\tcase 1:\n\t\treturn list[0]\n\tdefault:\n\t\treturn fmt.Sprintf(\"%s %s %s\", strings.Join(list[:len(list)-1], sep), lastSep, list[len(list)-1])\n\t}\n}\n\nfunc (p *parser) parseRule(rule *rule) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseRule \" + rule.name))\n\t}\n\n\tif p.memoize {\n\t\tres, ok := p.getMemoized(rule)\n\t\tif ok {\n\t\t\tp.restore(res.end)\n\t\t\treturn res.v, res.b\n\t\t}\n\t}\n\n\tstart := p.pt\n\tp.rstack = append(p.rstack, rule)\n\tp.pushV()\n\tval, ok := p.parseExpr(rule.expr)\n\tp.popV()\n\tp.rstack = p.rstack[:len(p.rstack)-1]\n\tif ok && p.debug {\n\t\tp.print(strings.Repeat(\" \", p.depth)+\"MATCH\", string(p.sliceFrom(start)))\n\t}\n\n\tif p.memoize {\n\t\tp.setMemoized(start, rule, resultTuple{val, ok, p.pt})\n\t}\n\treturn val, ok\n}\n\nfunc (p *parser) parseExpr(expr interface{}) (interface{}, bool) {\n\tvar pt savepoint\n\n\tif p.memoize {\n\t\tres, ok := p.getMemoized(expr)\n\t\tif ok {\n\t\t\tp.restore(res.end)\n\t\t\treturn res.v, res.b\n\t\t}\n\t\tpt = p.pt\n\t}\n\n\tp.exprCnt++\n\tif p.exprCnt > p.maxExprCnt {\n\t\tpanic(errMaxExprCnt)\n\t}\n\n\tvar val interface{}\n\tvar ok bool\n\tswitch expr := expr.(type) {\n\tcase *actionExpr:\n\t\tval, ok = p.parseActionExpr(expr)\n\tcase *andCodeExpr:\n\t\tval, ok = p.parseAndCodeExpr(expr)\n\tcase *andExpr:\n\t\tval, ok = p.parseAndExpr(expr)\n\tcase *anyMatcher:\n\t\tval, ok = p.parseAnyMatcher(expr)\n\tcase *charClassMatcher:\n\t\tval, ok = p.parseCharClassMatcher(expr)\n\tcase *choiceExpr:\n\t\tval, ok = p.parseChoiceExpr(expr)\n\tcase *labeledExpr:\n\t\tval, ok = p.parseLabeledExpr(expr)\n\tcase *litMatcher:\n\t\tval, ok = p.parseLitMatcher(expr)\n\tcase *notCodeExpr:\n\t\tval, ok = p.parseNotCodeExpr(expr)\n\tcase *notExpr:\n\t\tval, ok = p.parseNotExpr(expr)\n\tcase *oneOrMoreExpr:\n\t\tval, ok = p.parseOneOrMoreExpr(expr)\n\tcase *ruleRefExpr:\n\t\tval, ok = p.parseRuleRefExpr(expr)\n\tcase *seqExpr:\n\t\tval, ok = p.parseSeqExpr(expr)\n\tcase *zeroOrMoreExpr:\n\t\tval, ok = p.parseZeroOrMoreExpr(expr)\n\tcase *zeroOrOneExpr:\n\t\tval, ok = p.parseZeroOrOneExpr(expr)\n\tdefault:\n\t\tpanic(fmt.Sprintf(\"unknown expression type %T\", expr))\n\t}\n\tif p.memoize {\n\t\tp.setMemoized(pt, expr, resultTuple{val, ok, p.pt})\n\t}\n\treturn val, ok\n}\n\nfunc (p *parser) parseActionExpr(act *actionExpr) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseActionExpr\"))\n\t}\n\n\tstart := p.pt\n\tval, ok := p.parseExpr(act.expr)\n\tif ok {\n\t\tp.cur.pos = start.position\n\t\tp.cur.text = p.sliceFrom(start)\n\t\tactVal, err := act.run(p)\n\t\tif err != nil {\n\t\t\tp.addErrAt(err, start.position, []string{})\n\t\t}\n\t\tval = actVal\n\t}\n\tif ok && p.debug {\n\t\tp.print(strings.Repeat(\" \", p.depth)+\"MATCH\", string(p.sliceFrom(start)))\n\t}\n\treturn val, ok\n}\n\nfunc (p *parser) parseAndCodeExpr(and *andCodeExpr) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseAndCodeExpr\"))\n\t}\n\n\tok, err := and.run(p)\n\tif err != nil {\n\t\tp.addErr(err)\n\t}\n\treturn nil, ok\n}\n\nfunc (p *parser) parseAndExpr(and *andExpr) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseAndExpr\"))\n\t}\n\n\tpt := p.pt\n\tp.pushV()\n\t_, ok := p.parseExpr(and.expr)\n\tp.popV()\n\tp.restore(pt)\n\treturn nil, ok\n}\n\nfunc (p *parser) parseAnyMatcher(any *anyMatcher) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseAnyMatcher\"))\n\t}\n\n\tif p.pt.rn != utf8.RuneError {\n\t\tstart := p.pt\n\t\tp.read()\n\t\tp.failAt(true, start.position, \".\")\n\t\treturn p.sliceFrom(start), true\n\t}\n\tp.failAt(false, p.pt.position, \".\")\n\treturn nil, false\n}\n\nfunc (p *parser) parseCharClassMatcher(chr *charClassMatcher) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseCharClassMatcher\"))\n\t}\n\n\tcur := p.pt.rn\n\tstart := p.pt\n\n\t// can't match EOF\n\tif cur == utf8.RuneError {\n\t\tp.failAt(false, start.position, chr.val)\n\t\treturn nil, false\n\t}\n\n\tif chr.ignoreCase {\n\t\tcur = unicode.ToLower(cur)\n\t}\n\n\t// try to match in the list of available chars\n\tfor _, rn := range chr.chars {\n\t\tif rn == cur {\n\t\t\tif chr.inverted {\n\t\t\t\tp.failAt(false, start.position, chr.val)\n\t\t\t\treturn nil, false\n\t\t\t}\n\t\t\tp.read()\n\t\t\tp.failAt(true, start.position, chr.val)\n\t\t\treturn p.sliceFrom(start), true\n\t\t}\n\t}\n\n\t// try to match in the list of ranges\n\tfor i := 0; i < len(chr.ranges); i += 2 {\n\t\tif cur >= chr.ranges[i] && cur <= chr.ranges[i+1] {\n\t\t\tif chr.inverted {\n\t\t\t\tp.failAt(false, start.position, chr.val)\n\t\t\t\treturn nil, false\n\t\t\t}\n\t\t\tp.read()\n\t\t\tp.failAt(true, start.position, chr.val)\n\t\t\treturn p.sliceFrom(start), true\n\t\t}\n\t}\n\n\t// try to match in the list of Unicode classes\n\tfor _, cl := range chr.classes {\n\t\tif unicode.Is(cl, cur) {\n\t\t\tif chr.inverted {\n\t\t\t\tp.failAt(false, start.position, chr.val)\n\t\t\t\treturn nil, false\n\t\t\t}\n\t\t\tp.read()\n\t\t\tp.failAt(true, start.position, chr.val)\n\t\t\treturn p.sliceFrom(start), true\n\t\t}\n\t}\n\n\tif chr.inverted {\n\t\tp.read()\n\t\tp.failAt(true, start.position, chr.val)\n\t\treturn p.sliceFrom(start), true\n\t}\n\tp.failAt(false, start.position, chr.val)\n\treturn nil, false\n}\n\nfunc (p *parser) parseChoiceExpr(ch *choiceExpr) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseChoiceExpr\"))\n\t}\n\n\tfor _, alt := range ch.alternatives {\n\t\tp.pushV()\n\t\tval, ok := p.parseExpr(alt)\n\t\tp.popV()\n\t\tif ok {\n\t\t\treturn val, ok\n\t\t}\n\t}\n\treturn nil, false\n}\n\nfunc (p *parser) parseLabeledExpr(lab *labeledExpr) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseLabeledExpr\"))\n\t}\n\n\tp.pushV()\n\tval, ok := p.parseExpr(lab.expr)\n\tp.popV()\n\tif ok && lab.label != \"\" {\n\t\tm := p.vstack[len(p.vstack)-1]\n\t\tm[lab.label] = val\n\t}\n\treturn val, ok\n}\n\nfunc (p *parser) parseLitMatcher(lit *litMatcher) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseLitMatcher\"))\n\t}\n\n\tignoreCase := \"\"\n\tif lit.ignoreCase {\n\t\tignoreCase = \"i\"\n\t}\n\tval := fmt.Sprintf(\"%q%s\", lit.val, ignoreCase)\n\tstart := p.pt\n\tfor _, want := range lit.val {\n\t\tcur := p.pt.rn\n\t\tif lit.ignoreCase {\n\t\t\tcur = unicode.ToLower(cur)\n\t\t}\n\t\tif cur != want {\n\t\t\tp.failAt(false, start.position, val)\n\t\t\tp.restore(start)\n\t\t\treturn nil, false\n\t\t}\n\t\tp.read()\n\t}\n\tp.failAt(true, start.position, val)\n\treturn p.sliceFrom(start), true\n}\n\nfunc (p *parser) parseNotCodeExpr(not *notCodeExpr) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseNotCodeExpr\"))\n\t}\n\n\tok, err := not.run(p)\n\tif err != nil {\n\t\tp.addErr(err)\n\t}\n\treturn nil, !ok\n}\n\nfunc (p *parser) parseNotExpr(not *notExpr) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseNotExpr\"))\n\t}\n\n\tpt := p.pt\n\tp.pushV()\n\tp.maxFailInvertExpected = !p.maxFailInvertExpected\n\t_, ok := p.parseExpr(not.expr)\n\tp.maxFailInvertExpected = !p.maxFailInvertExpected\n\tp.popV()\n\tp.restore(pt)\n\treturn nil, !ok\n}\n\nfunc (p *parser) parseOneOrMoreExpr(expr *oneOrMoreExpr) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseOneOrMoreExpr\"))\n\t}\n\n\tvar vals []interface{}\n\n\tfor {\n\t\tp.pushV()\n\t\tval, ok := p.parseExpr(expr.expr)\n\t\tp.popV()\n\t\tif !ok {\n\t\t\tif len(vals) == 0 {\n\t\t\t\t// did not match once, no match\n\t\t\t\treturn nil, false\n\t\t\t}\n\t\t\treturn vals, true\n\t\t}\n\t\tvals = append(vals, val)\n\t}\n}\n\nfunc (p *parser) parseRuleRefExpr(ref *ruleRefExpr) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseRuleRefExpr \" + ref.name))\n\t}\n\n\tif ref.name == \"\" {\n\t\tpanic(fmt.Sprintf(\"%s: invalid rule: missing name\", ref.pos))\n\t}\n\n\trule := p.rules[ref.name]\n\tif rule == nil {\n\t\tp.addErr(fmt.Errorf(\"undefined rule: %s\", ref.name))\n\t\treturn nil, false\n\t}\n\treturn p.parseRule(rule)\n}\n\nfunc (p *parser) parseSeqExpr(seq *seqExpr) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseSeqExpr\"))\n\t}\n\n\tvals := make([]interface{}, 0, len(seq.exprs))\n\n\tpt := p.pt\n\tfor _, expr := range seq.exprs {\n\t\tval, ok := p.parseExpr(expr)\n\t\tif !ok {\n\t\t\tp.restore(pt)\n\t\t\treturn nil, false\n\t\t}\n\t\tvals = append(vals, val)\n\t}\n\treturn vals, true\n}\n\nfunc (p *parser) parseZeroOrMoreExpr(expr *zeroOrMoreExpr) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseZeroOrMoreExpr\"))\n\t}\n\n\tvar vals []interface{}\n\n\tfor {\n\t\tp.pushV()\n\t\tval, ok := p.parseExpr(expr.expr)\n\t\tp.popV()\n\t\tif !ok {\n\t\t\treturn vals, true\n\t\t}\n\t\tvals = append(vals, val)\n\t}\n}\n\nfunc (p *parser) parseZeroOrOneExpr(expr *zeroOrOneExpr) (interface{}, bool) {\n\tif p.debug {\n\t\tdefer p.out(p.in(\"parseZeroOrOneExpr\"))\n\t}\n\n\tp.pushV()\n\tval, _ := p.parseExpr(expr.expr)\n\tp.popV()\n\t// whether it matched or not, consider it a match\n\treturn val, true\n}\n"
  },
  {
    "path": "parser/parser_test.go",
    "content": "package parser\n\nimport (\n\t\"reflect\"\n\t\"testing\"\n)\n\nfunc TestParseSample(t *testing.T) {\n\ttype args struct {\n\t\tsampleID int\n\t\tsample   []byte\n\t}\n\ttests := []struct {\n\t\tname    string\n\t\targs    args\n\t\twant    []Token\n\t\twantErr bool\n\t}{\n\t\t0: {\n\t\t\t\"err: empty sample\",\n\t\t\targs{0, nil},\n\t\t\tnil,\n\t\t\ttrue,\n\t\t},\n\t\t1: {\n\t\t\t\"normal sample\",\n\t\t\targs{1, []byte(\"play {Name} from {Artist}\")},\n\t\t\t[]Token{\n\t\t\t\t{Val: []byte(\"play\")},\n\t\t\t\t{Kw: true, Val: []byte(\"Name\")},\n\t\t\t\t{Val: []byte(\"from\")},\n\t\t\t\t{Kw: true, Val: []byte(\"Artist\")},\n\t\t\t},\n\t\t\tfalse,\n\t\t},\n\t\t2: {\n\t\t\t\"spacing inside keys\",\n\t\t\targs{1, []byte(\"play { \tName} from {\tArtist\t\t}\")},\n\t\t\t[]Token{\n\t\t\t\t{Val: []byte(\"play\")},\n\t\t\t\t{Kw: true, Val: []byte(\"Name\")},\n\t\t\t\t{Val: []byte(\"from\")},\n\t\t\t\t{Kw: true, Val: []byte(\"Artist\")},\n\t\t\t},\n\t\t\tfalse,\n\t\t},\n\t\t3: {\n\t\t\t\"multi word\",\n\t\t\targs{1, []byte(\"I need {Name} since {Since}\")},\n\t\t\t[]Token{\n\t\t\t\t{Val: []byte(\"I\")},\n\t\t\t\t{Val: []byte(\"need\")},\n\t\t\t\t{Kw: true, Val: []byte(\"Name\")},\n\t\t\t\t{Val: []byte(\"since\")},\n\t\t\t\t{Kw: true, Val: []byte(\"Since\")},\n\t\t\t},\n\t\t\tfalse,\n\t\t},\n\t}\n\tfor i, tt := range tests {\n\t\tt.Run(tt.name, func(t *testing.T) {\n\t\t\tgot, err := ParseSample(tt.args.sampleID, tt.args.sample)\n\t\t\tif (err != nil) != tt.wantErr {\n\t\t\t\tt.Errorf(\"Test#%d: ParseSample() error = %v, wantErr %v\", i, err, tt.wantErr)\n\t\t\t\treturn\n\t\t\t}\n\t\t\tif !reflect.DeepEqual(got, tt.want) {\n\t\t\t\tt.Errorf(\"Test#%d: ParseSample() = %v, want %v\", i, got, tt.want)\n\t\t\t}\n\t\t})\n\t}\n}\n"
  }
]