Showing preview only (551K chars total). Download the full file or copy to clipboard to get everything.
Repository: couchbase/vellum
Branch: master
Commit: abd0418dd6c7
Files: 121
Total size: 517.3 KB
Directory structure:
gitextract_4hy_b8n1/
├── .github/
│ └── workflows/
│ └── tests.yml
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── automaton.go
├── builder.go
├── builder_test.go
├── cmd/
│ └── vellum/
│ ├── cmd/
│ │ ├── dot.go
│ │ ├── dump.go
│ │ ├── fuzzy.go
│ │ ├── grep.go
│ │ ├── info.go
│ │ ├── map.go
│ │ ├── range.go
│ │ ├── root.go
│ │ ├── set.go
│ │ └── svg.go
│ └── main.go
├── common.go
├── common_test.go
├── data/
│ └── words-1000.txt
├── decoder_v1.go
├── decoder_v1_test.go
├── docs/
│ └── format.md
├── encoder_v1.go
├── encoder_v1_test.go
├── encoding.go
├── example_test.go
├── fst.go
├── fst_iterator.go
├── fst_iterator_test.go
├── go.mod
├── go.sum
├── levenshtein/
│ ├── LICENSE
│ ├── README.md
│ ├── alphabet.go
│ ├── alphabet_test.go
│ ├── benchmark_test.go
│ ├── dfa.go
│ ├── dfa_test.go
│ ├── levenshtein.go
│ ├── levenshtein_nfa.go
│ ├── levenshtein_test.go
│ └── parametric_dfa.go
├── merge_iterator.go
├── merge_iterator_test.go
├── pack.go
├── pack_test.go
├── regexp/
│ ├── compile.go
│ ├── compile_test.go
│ ├── dfa.go
│ ├── inst.go
│ ├── regexp.go
│ ├── regexp_test.go
│ ├── sparse.go
│ └── sparse_test.go
├── registry.go
├── registry_test.go
├── transducer.go
├── utf8/
│ ├── utf8.go
│ └── utf8_test.go
├── vellum.go
├── vellum_mmap.go
├── vellum_nommap.go
├── vellum_test.go
├── vendor/
│ ├── github.com/
│ │ ├── edsrzf/
│ │ │ └── mmap-go/
│ │ │ ├── LICENSE
│ │ │ ├── mmap.go
│ │ │ ├── mmap_unix.go
│ │ │ ├── mmap_windows.go
│ │ │ ├── msync_netbsd.go
│ │ │ └── msync_unix.go
│ │ ├── inconshreveable/
│ │ │ └── mousetrap/
│ │ │ ├── LICENSE
│ │ │ ├── trap_others.go
│ │ │ ├── trap_windows.go
│ │ │ └── trap_windows_1.4.go
│ │ ├── spf13/
│ │ │ ├── cobra/
│ │ │ │ ├── LICENSE.txt
│ │ │ │ ├── bash_completions.go
│ │ │ │ ├── cobra.go
│ │ │ │ ├── command.go
│ │ │ │ ├── command_notwin.go
│ │ │ │ ├── command_win.go
│ │ │ │ └── doc/
│ │ │ │ ├── man_docs.go
│ │ │ │ ├── md_docs.go
│ │ │ │ ├── util.go
│ │ │ │ └── yaml_docs.go
│ │ │ └── pflag/
│ │ │ ├── LICENSE
│ │ │ ├── bool.go
│ │ │ ├── bool_slice.go
│ │ │ ├── count.go
│ │ │ ├── duration.go
│ │ │ ├── flag.go
│ │ │ ├── float32.go
│ │ │ ├── float64.go
│ │ │ ├── golangflag.go
│ │ │ ├── int.go
│ │ │ ├── int32.go
│ │ │ ├── int64.go
│ │ │ ├── int8.go
│ │ │ ├── int_slice.go
│ │ │ ├── ip.go
│ │ │ ├── ip_slice.go
│ │ │ ├── ipmask.go
│ │ │ ├── ipnet.go
│ │ │ ├── string.go
│ │ │ ├── string_array.go
│ │ │ ├── string_slice.go
│ │ │ ├── uint.go
│ │ │ ├── uint16.go
│ │ │ ├── uint32.go
│ │ │ ├── uint64.go
│ │ │ ├── uint8.go
│ │ │ └── uint_slice.go
│ │ └── willf/
│ │ └── bitset/
│ │ ├── LICENSE
│ │ ├── bitset.go
│ │ ├── popcnt.go
│ │ ├── popcnt_amd64.go
│ │ ├── popcnt_amd64.s
│ │ └── popcnt_generic.go
│ └── manifest
├── writer.go
└── writer_test.go
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/workflows/tests.yml
================================================
on:
push:
branches:
- master
pull_request:
name: Tests
jobs:
test:
strategy:
matrix:
go-version: [1.13.x, 1.14.x]
platform: [ubuntu-latest, macos-latest, windows-latest]
runs-on: ${{ matrix.platform }}
steps:
- name: Install Go
uses: actions/setup-go@v1
with:
go-version: ${{ matrix.go-version }}
- name: Checkout code
uses: actions/checkout@v2
- name: Test
run: |
go version
go test -race ./...
================================================
FILE: CONTRIBUTING.md
================================================
# Contributing to Vellum
We look forward to your contributions, but ask that you first review these guidelines.
### Sign the CLA
As Vellum is a Couchbase project we require contributors accept the [Couchbase Contributor License Agreement](http://review.couchbase.org/static/individual_agreement.html). To sign this agreement log into the Couchbase [code review tool](http://review.couchbase.org/). The Vellum project does not use this code review tool but it is still used to track acceptance of the contributor license agreements.
### Submitting a Pull Request
All types of contributions are welcome, but please keep the following in mind:
- If you're planning a large change, you should really discuss it in a github issue first. This helps avoid duplicate effort and spending time on something that may not be merged.
- Existing tests should continue to pass, new tests for the contribution are nice to have.
- All code should have gone through `go fmt`
- All code should pass `go vet`
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: README.md
================================================
#  vellum
# NOTE: active development of the vellum library has moved to [https://github.com/blevesearch/vellum](https://github.com/blevesearch/vellum)
This repository will remain as is to support previous Couchbase builds.
[](https://github.com/couchbase/vellum/actions?query=workflow%3ATests+event%3Apush+branch%3Amaster)
[](https://coveralls.io/github/couchbase/vellum?branch=master)
[](https://godoc.org/github.com/couchbase/vellum)
[](https://goreportcard.com/report/github.com/couchbase/vellum)
[](https://opensource.org/licenses/Apache-2.0)
A Go library implementing an FST (finite state transducer) capable of:
- mapping between keys ([]byte) and a value (uint64)
- enumerating keys in lexicographic order
Some additional goals of this implementation:
- bounded memory use while building the FST
- streaming out FST data while building
- mmap FST runtime to support very large FTSs (optional)
## Usage
### Building an FST
To build an FST, create a new builder using the `New()` method. This method takes an `io.Writer` as an argument. As the FST is being built, data will be streamed to the writer as soon as possible. With this builder you **MUST** insert keys in lexicographic order. Inserting keys out of order will result in an error. After inserting the last key into the builder, you **MUST** call `Close()` on the builder. This will flush all remaining data to the underlying writer.
In memory:
```go
var buf bytes.Buffer
builder, err := vellum.New(&buf, nil)
if err != nil {
log.Fatal(err)
}
```
To disk:
```go
f, err := os.Create("/tmp/vellum.fst")
if err != nil {
log.Fatal(err)
}
builder, err := vellum.New(f, nil)
if err != nil {
log.Fatal(err)
}
```
**MUST** insert keys in lexicographic order:
```go
err = builder.Insert([]byte("cat"), 1)
if err != nil {
log.Fatal(err)
}
err = builder.Insert([]byte("dog"), 2)
if err != nil {
log.Fatal(err)
}
err = builder.Insert([]byte("fish"), 3)
if err != nil {
log.Fatal(err)
}
err = builder.Close()
if err != nil {
log.Fatal(err)
}
```
### Using an FST
After closing the builder, the data can be used to instantiate an FST. If the data was written to disk, you can use the `Open()` method to mmap the file. If the data is already in memory, or you wish to load/mmap the data yourself, you can instantiate the FST with the `Load()` method.
Load in memory:
```go
fst, err := vellum.Load(buf.Bytes())
if err != nil {
log.Fatal(err)
}
```
Open from disk:
```go
fst, err := vellum.Open("/tmp/vellum.fst")
if err != nil {
log.Fatal(err)
}
```
Get key/value:
```go
val, exists, err = fst.Get([]byte("dog"))
if err != nil {
log.Fatal(err)
}
if exists {
fmt.Printf("contains dog with val: %d\n", val)
} else {
fmt.Printf("does not contain dog")
}
```
Iterate key/values:
```go
itr, err := fst.Iterator(startKeyInclusive, endKeyExclusive)
for err == nil {
key, val := itr.Current()
fmt.Printf("contains key: %s val: %d", key, val)
err = itr.Next()
}
if err != nil {
log.Fatal(err)
}
```
### How does the FST get built?
A full example of the implementation is beyond the scope of this README, but let's consider a small example where we want to insert 3 key/value pairs.
First we insert "are" with the value 4.
Next, we insert "ate" with the value 2.
Notice how the values associated with the transitions were adjusted so that by summing them while traversing we still get the expected value.
At this point, we see that state 5 looks like state 3, and state 4 looks like state 2. But, we cannot yet combine them because future inserts could change this.
Now, we insert "see" with value 3. Once it has been added, we now know that states 5 and 4 can longer change. Since they are identical to 3 and 2, we replace them.
Again, we see that states 7 and 8 appear to be identical to 2 and 3.
Having inserted our last key, we call `Close()` on the builder.
Now, states 7 and 8 can safely be replaced with 2 and 3.
For additional information, see the references at the bottom of this document.
### What does the serialized format look like?
We've broken out a separate document on the [vellum disk format v1](docs/format.md).
### What if I want to use this on a system that doesn't have mmap?
The mmap library itself is guarded with system/architecture build tags, but we've also added an additional build tag in vellum. If you'd like to Open() a file based representation of an FST, but not use mmap, you can build the library with the `nommap` build tag. NOTE: if you do this, the entire FST will be read into memory.
### Can I use this with Unicode strings?
Yes, however this implementation is only aware of the byte representation you choose. In order to find matches, you must work with some canonical byte representation of the string. In the future, some encoding-aware traversals may be possible on top of the lower-level byte transitions.
### How did this library come to be?
In my work on the [Bleve](https://github.com/blevesearch/bleve) project I became aware of the power of the FST for many search-related tasks. The obvious starting point for such a thing in Go was the [mafsa](https://github.com/smartystreets/mafsa) project. While working with mafsa I encountered some issues. First, it did not stream data to disk while building. Second, it chose to use a rune as the fundamental unit of transition in the FST, but I felt using a byte would be more powerful in the end. My hope is that higher-level encoding-aware traversals will be possible when necessary. Finally, as I reported bugs and submitted PRs I learned that the mafsa project was mainly a research project and no longer being maintained. I wanted to build something that could be used in production. As the project advanced more and more techniques from the [BurntSushi/fst](https://github.com/BurntSushi/fst) were adapted to our implementation.
### Are there tools to work with vellum files?
Under the cmd/vellum subdirectory, there's a command-line tool which
features subcommands that can allow you to create, inspect and query
vellum files.
### How can I generate a state transition diagram from a vellum file?
The vellum command-line tool has a "dot" subcommand that can emit
graphviz dot output data from an input vellum file. The dot file can
in turn be converted into an image using graphviz tools. Example...
$ vellum dot myFile.vellum > output.dot
$ dot -Tpng output.dot -o output.png
## Related Work
Much credit goes to two existing projects:
- [mafsa](https://github.com/smartystreets/mafsa)
- [BurntSushi/fst](https://github.com/BurntSushi/fst)
Most of the original implementation here started with my digging into the internals of mafsa. As the implementation progressed, I continued to borrow ideas/approaches from the BurntSushi/fst library as well.
For a great introduction to this topic, please read the blog post [Index 1,600,000,000 Keys with Automata and Rust](http://blog.burntsushi.net/transducers/)
================================================
FILE: automaton.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum
// Automaton represents the general contract of a byte-based finite automaton
type Automaton interface {
// Start returns the start state
Start() int
// IsMatch returns true if and only if the state is a match
IsMatch(int) bool
// CanMatch returns true if and only if it is possible to reach a match
// in zero or more steps
CanMatch(int) bool
// WillAlwaysMatch returns true if and only if the current state matches
// and will always match no matter what steps are taken
WillAlwaysMatch(int) bool
// Accept returns the next state given the input to the specified state
Accept(int, byte) int
}
// AutomatonContains implements an generic Contains() method which works
// on any implementation of Automaton
func AutomatonContains(a Automaton, k []byte) bool {
i := 0
curr := a.Start()
for a.CanMatch(curr) && i < len(k) {
curr = a.Accept(curr, k[i])
if curr == noneAddr {
break
}
i++
}
if i != len(k) {
return false
}
return a.IsMatch(curr)
}
// AlwaysMatch is an Automaton implementation which always matches
type AlwaysMatch struct{}
// Start returns the AlwaysMatch start state
func (m *AlwaysMatch) Start() int {
return 0
}
// IsMatch always returns true
func (m *AlwaysMatch) IsMatch(int) bool {
return true
}
// CanMatch always returns true
func (m *AlwaysMatch) CanMatch(int) bool {
return true
}
// WillAlwaysMatch always returns true
func (m *AlwaysMatch) WillAlwaysMatch(int) bool {
return true
}
// Accept returns the next AlwaysMatch state
func (m *AlwaysMatch) Accept(int, byte) int {
return 0
}
// creating an alwaysMatchAutomaton to avoid unnecessary repeated allocations.
var alwaysMatchAutomaton = &AlwaysMatch{}
================================================
FILE: builder.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum
import (
"bytes"
"io"
)
var defaultBuilderOpts = &BuilderOpts{
Encoder: 1,
RegistryTableSize: 10000,
RegistryMRUSize: 2,
}
// A Builder is used to build a new FST. When possible data is
// streamed out to the underlying Writer as soon as possible.
type Builder struct {
unfinished *unfinishedNodes
registry *registry
last []byte
len int
lastAddr int
encoder encoder
opts *BuilderOpts
builderNodePool *builderNodePool
}
const noneAddr = 1
const emptyAddr = 0
// NewBuilder returns a new Builder which will stream out the
// underlying representation to the provided Writer as the set is built.
func newBuilder(w io.Writer, opts *BuilderOpts) (*Builder, error) {
if opts == nil {
opts = defaultBuilderOpts
}
builderNodePool := &builderNodePool{}
rv := &Builder{
unfinished: newUnfinishedNodes(builderNodePool),
registry: newRegistry(builderNodePool, opts.RegistryTableSize, opts.RegistryMRUSize),
builderNodePool: builderNodePool,
opts: opts,
lastAddr: noneAddr,
}
var err error
rv.encoder, err = loadEncoder(opts.Encoder, w)
if err != nil {
return nil, err
}
err = rv.encoder.start()
if err != nil {
return nil, err
}
return rv, nil
}
func (b *Builder) Reset(w io.Writer) error {
b.unfinished.Reset()
b.registry.Reset()
b.lastAddr = noneAddr
b.encoder.reset(w)
b.last = nil
b.len = 0
err := b.encoder.start()
if err != nil {
return err
}
return nil
}
// Insert the provided value to the set being built.
// NOTE: values must be inserted in lexicographical order.
func (b *Builder) Insert(key []byte, val uint64) error {
// ensure items are added in lexicographic order
if bytes.Compare(key, b.last) < 0 {
return ErrOutOfOrder
}
if len(key) == 0 {
b.len = 1
b.unfinished.setRootOutput(val)
return nil
}
prefixLen, out := b.unfinished.findCommonPrefixAndSetOutput(key, val)
b.len++
err := b.compileFrom(prefixLen)
if err != nil {
return err
}
b.copyLastKey(key)
b.unfinished.addSuffix(key[prefixLen:], out)
return nil
}
func (b *Builder) copyLastKey(key []byte) {
if b.last == nil {
b.last = make([]byte, 0, 64)
} else {
b.last = b.last[:0]
}
b.last = append(b.last, key...)
}
// Close MUST be called after inserting all values.
func (b *Builder) Close() error {
err := b.compileFrom(0)
if err != nil {
return err
}
root := b.unfinished.popRoot()
rootAddr, err := b.compile(root)
if err != nil {
return err
}
return b.encoder.finish(b.len, rootAddr)
}
func (b *Builder) compileFrom(iState int) error {
addr := noneAddr
for iState+1 < len(b.unfinished.stack) {
var node *builderNode
if addr == noneAddr {
node = b.unfinished.popEmpty()
} else {
node = b.unfinished.popFreeze(addr)
}
var err error
addr, err = b.compile(node)
if err != nil {
return nil
}
}
b.unfinished.topLastFreeze(addr)
return nil
}
func (b *Builder) compile(node *builderNode) (int, error) {
if node.final && len(node.trans) == 0 &&
node.finalOutput == 0 {
return 0, nil
}
found, addr, entry := b.registry.entry(node)
if found {
return addr, nil
}
addr, err := b.encoder.encodeState(node, b.lastAddr)
if err != nil {
return 0, err
}
b.lastAddr = addr
entry.addr = addr
return addr, nil
}
type unfinishedNodes struct {
stack []*builderNodeUnfinished
// cache allocates a reasonable number of builderNodeUnfinished
// objects up front and tries to keep reusing them
// because the main data structure is a stack, we assume the
// same access pattern, and don't track items separately
// this means calls get() and pushXYZ() must be paired,
// as well as calls put() and popXYZ()
cache []builderNodeUnfinished
builderNodePool *builderNodePool
}
func (u *unfinishedNodes) Reset() {
u.stack = u.stack[:0]
for i := 0; i < len(u.cache); i++ {
u.cache[i] = builderNodeUnfinished{}
}
u.pushEmpty(false)
}
func newUnfinishedNodes(p *builderNodePool) *unfinishedNodes {
rv := &unfinishedNodes{
stack: make([]*builderNodeUnfinished, 0, 64),
cache: make([]builderNodeUnfinished, 64),
builderNodePool: p,
}
rv.pushEmpty(false)
return rv
}
// get new builderNodeUnfinished, reusing cache if possible
func (u *unfinishedNodes) get() *builderNodeUnfinished {
if len(u.stack) < len(u.cache) {
return &u.cache[len(u.stack)]
}
// full now allocate a new one
return &builderNodeUnfinished{}
}
// return builderNodeUnfinished, clearing it for reuse
func (u *unfinishedNodes) put() {
if len(u.stack) >= len(u.cache) {
return
// do nothing, not part of cache
}
u.cache[len(u.stack)] = builderNodeUnfinished{}
}
func (u *unfinishedNodes) findCommonPrefixAndSetOutput(key []byte,
out uint64) (int, uint64) {
var i int
for i < len(key) {
if i >= len(u.stack) {
break
}
var addPrefix uint64
if !u.stack[i].hasLastT {
break
}
if u.stack[i].lastIn == key[i] {
commonPre := outputPrefix(u.stack[i].lastOut, out)
addPrefix = outputSub(u.stack[i].lastOut, commonPre)
out = outputSub(out, commonPre)
u.stack[i].lastOut = commonPre
i++
} else {
break
}
if addPrefix != 0 {
u.stack[i].addOutputPrefix(addPrefix)
}
}
return i, out
}
func (u *unfinishedNodes) pushEmpty(final bool) {
next := u.get()
next.node = u.builderNodePool.Get()
next.node.final = final
u.stack = append(u.stack, next)
}
func (u *unfinishedNodes) popRoot() *builderNode {
l := len(u.stack)
var unfinished *builderNodeUnfinished
u.stack, unfinished = u.stack[:l-1], u.stack[l-1]
rv := unfinished.node
u.put()
return rv
}
func (u *unfinishedNodes) popFreeze(addr int) *builderNode {
l := len(u.stack)
var unfinished *builderNodeUnfinished
u.stack, unfinished = u.stack[:l-1], u.stack[l-1]
unfinished.lastCompiled(addr)
rv := unfinished.node
u.put()
return rv
}
func (u *unfinishedNodes) popEmpty() *builderNode {
l := len(u.stack)
var unfinished *builderNodeUnfinished
u.stack, unfinished = u.stack[:l-1], u.stack[l-1]
rv := unfinished.node
u.put()
return rv
}
func (u *unfinishedNodes) setRootOutput(out uint64) {
u.stack[0].node.final = true
u.stack[0].node.finalOutput = out
}
func (u *unfinishedNodes) topLastFreeze(addr int) {
last := len(u.stack) - 1
u.stack[last].lastCompiled(addr)
}
func (u *unfinishedNodes) addSuffix(bs []byte, out uint64) {
if len(bs) == 0 {
return
}
last := len(u.stack) - 1
u.stack[last].hasLastT = true
u.stack[last].lastIn = bs[0]
u.stack[last].lastOut = out
for _, b := range bs[1:] {
next := u.get()
next.node = u.builderNodePool.Get()
next.hasLastT = true
next.lastIn = b
next.lastOut = 0
u.stack = append(u.stack, next)
}
u.pushEmpty(true)
}
type builderNodeUnfinished struct {
node *builderNode
lastOut uint64
lastIn byte
hasLastT bool
}
func (b *builderNodeUnfinished) lastCompiled(addr int) {
if b.hasLastT {
transIn := b.lastIn
transOut := b.lastOut
b.hasLastT = false
b.lastOut = 0
b.node.trans = append(b.node.trans, transition{
in: transIn,
out: transOut,
addr: addr,
})
}
}
func (b *builderNodeUnfinished) addOutputPrefix(prefix uint64) {
if b.node.final {
b.node.finalOutput = outputCat(prefix, b.node.finalOutput)
}
for i := range b.node.trans {
b.node.trans[i].out = outputCat(prefix, b.node.trans[i].out)
}
if b.hasLastT {
b.lastOut = outputCat(prefix, b.lastOut)
}
}
type builderNode struct {
finalOutput uint64
trans []transition
final bool
// intrusive linked list
next *builderNode
}
// reset resets the receiver builderNode to a re-usable state.
func (n *builderNode) reset() {
n.final = false
n.finalOutput = 0
for i := range n.trans {
n.trans[i] = emptyTransition
}
n.trans = n.trans[:0]
n.next = nil
}
func (n *builderNode) equiv(o *builderNode) bool {
if n.final != o.final {
return false
}
if n.finalOutput != o.finalOutput {
return false
}
if len(n.trans) != len(o.trans) {
return false
}
for i, ntrans := range n.trans {
otrans := o.trans[i]
if ntrans.in != otrans.in {
return false
}
if ntrans.addr != otrans.addr {
return false
}
if ntrans.out != otrans.out {
return false
}
}
return true
}
var emptyTransition = transition{}
type transition struct {
out uint64
addr int
in byte
}
func outputPrefix(l, r uint64) uint64 {
if l < r {
return l
}
return r
}
func outputSub(l, r uint64) uint64 {
return l - r
}
func outputCat(l, r uint64) uint64 {
return l + r
}
// builderNodePool pools builderNodes using a singly linked list.
//
// NB: builderNode lifecylce is described by the following interactions -
// +------------------------+ +----------------------+
// | Unfinished Nodes | Transfer once | Registry |
// |(not frozen builderNode)|-----builderNode is ------->| (frozen builderNode) |
// +------------------------+ marked frozen +----------------------+
// ^ |
// | |
// | Put()
// | Get() on +-------------------+ when
// +-new char--------| builderNode Pool |<-----------evicted
// +-------------------+
type builderNodePool struct {
head *builderNode
}
func (p *builderNodePool) Get() *builderNode {
if p.head == nil {
return &builderNode{}
}
head := p.head
p.head = p.head.next
return head
}
func (p *builderNodePool) Put(v *builderNode) {
if v == nil {
return
}
v.reset()
v.next = p.head
p.head = v
}
================================================
FILE: builder_test.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum
import (
"bufio"
"io/ioutil"
"math/rand"
"os"
"sort"
"testing"
)
func init() {
thousandTestWords, _ = loadWords("data/words-1000.txt")
}
// this simple test case only has a shared final state
// it also tests out of order insert
func TestBuilderSimple(t *testing.T) {
b, err := New(ioutil.Discard, nil)
if err != nil {
t.Fatalf("error creating builder: %v", err)
}
// add our first string
err = b.Insert([]byte("jul"), 0)
if err != nil {
t.Errorf("got error inserting string: %v", err)
}
// expect len to be 1
if b.len != 1 {
t.Errorf("expected node count to be 1, got %v", b.len)
}
// try to add a value out of order (not allowed)
err = b.Insert([]byte("abc"), 0)
if err == nil {
t.Errorf("expected err, got nil")
}
// add a second string
err = b.Insert([]byte("mar"), 0)
if err != nil {
t.Errorf("got error inserting string: %v", err)
}
// expect len to grow by 1
if b.len != 2 {
t.Errorf("expected node count to be 2, got %v", b.len)
}
// now close the builder
err = b.Close()
if err != nil {
t.Errorf("got error closing set builder: %v", err)
}
}
func TestBuilderSharedPrefix(t *testing.T) {
b, err := New(ioutil.Discard, nil)
if err != nil {
t.Fatalf("error creating builder: %v", err)
}
// add our first string
err = b.Insert([]byte("car"), 0)
if err != nil {
t.Errorf("got error inserting string: %v", err)
}
// expect len to be 1
if b.len != 1 {
t.Errorf("expected node count to be 1, got %v", b.len)
}
// add a second string
err = b.Insert([]byte("cat"), 0)
if err != nil {
t.Errorf("got error inserting string: %v", err)
}
// expect len to be 2
if b.len != 2 {
t.Errorf("expected node count to be 2, got %v", b.len)
}
// now close the builder
err = b.Close()
if err != nil {
t.Errorf("got error closing set builder: %v", err)
}
}
func randomValues(list []string) []uint64 {
rv := make([]uint64, len(list))
for i := range list {
rv[i] = uint64(rand.Uint64())
}
return rv
}
func insertStrings(b *Builder, list []string, vals []uint64) error {
for i, item := range list {
err := b.Insert([]byte(item), vals[i])
if err != nil {
return err
}
}
return nil
}
var smallSample = map[string]uint64{
"mon": 2,
"tues": 3,
"thurs": 5,
"tye": 99,
}
func insertStringMap(b *Builder, m map[string]uint64) error {
// make list of keys
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
// sort it
sort.Strings(keys)
// insert in sorted order
for _, k := range keys {
err := b.Insert([]byte(k), m[k])
if err != nil {
return err
}
}
return nil
}
func TestBuilderNodeEquiv(t *testing.T) {
tests := []struct {
desc string
a *builderNode
b *builderNode
want bool
}{
{
"both states final",
&builderNode{
final: true,
},
&builderNode{
final: true,
},
true,
},
{
"both states final, different final val",
&builderNode{
final: true,
finalOutput: 7,
},
&builderNode{
final: true,
finalOutput: 9,
},
false,
},
{
"both states final, same transitions, but different trans val",
&builderNode{
final: true,
trans: []transition{
{in: 'a', out: 7},
},
},
&builderNode{
final: true,
trans: []transition{
{in: 'a', out: 9},
},
},
false,
},
}
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
got := test.a.equiv(test.b)
if got != test.want {
t.Errorf("wanted: %t, got: %t", test.want, got)
}
})
}
}
func loadWords(path string) ([]string, error) {
var rv []string
file, err := os.Open(path)
if err != nil {
return nil, err
}
scanner := bufio.NewScanner(file)
for scanner.Scan() {
word := append([]byte(nil), scanner.Bytes()...)
rv = append(rv, string(word))
if err != nil {
return nil, err
}
}
if err = scanner.Err(); err != nil {
return nil, err
}
err = file.Close()
if err != nil {
return nil, err
}
return rv, nil
}
var thousandTestWords []string
func BenchmarkBuilder(b *testing.B) {
dataset := thousandTestWords
randomThousandVals := randomValues(dataset)
b.ResetTimer()
for i := 0; i < b.N; i++ {
builder, err := New(ioutil.Discard, nil)
if err != nil {
b.Fatalf("error creating builder: %v", err)
}
err = insertStrings(builder, dataset, randomThousandVals)
if err != nil {
b.Fatalf("error inserting thousand words: %v", err)
}
err = builder.Close()
if err != nil {
b.Fatalf("error closing builder: %v", err)
}
}
}
================================================
FILE: cmd/vellum/cmd/dot.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cmd
import (
"fmt"
"io"
"os"
"github.com/couchbase/vellum"
"github.com/spf13/cobra"
)
var dotCmd = &cobra.Command{
Use: "dot",
Short: "Dot prints the contents of this vellum FST file in the dot format",
Long: `Dot prints the contents of this vellum FST file in the dot format.`,
PreRunE: func(cmd *cobra.Command, args []string) error {
if len(args) < 1 {
return fmt.Errorf("path is required")
}
return nil
},
RunE: func(cmd *cobra.Command, args []string) error {
fst, err := vellum.Open(args[0])
if err != nil {
return err
}
return dotToWriter(fst, os.Stdout)
},
}
func dotToWriter(fst *vellum.FST, w io.Writer) error {
_, err := fmt.Fprint(w, dotHeader)
if err != nil {
return err
}
err = fst.Debug(func(n int, state interface{}) error {
if d, ok := state.(dotStringer); ok {
_, err = fmt.Fprintf(w, "%s", d.DotString(n))
if err != nil {
return err
}
}
return nil
})
if err != nil {
return err
}
_, err = fmt.Fprint(w, dotFooter)
if err != nil {
return err
}
return nil
}
const dotHeader = `
digraph automaton {
labelloc="l";
labeljust="l";
rankdir="LR";
`
const dotFooter = `}
`
type dotStringer interface {
DotString(int) string
}
func init() {
RootCmd.AddCommand(dotCmd)
}
================================================
FILE: cmd/vellum/cmd/dump.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cmd
import (
"fmt"
"github.com/couchbase/vellum"
"github.com/spf13/cobra"
)
// dumpCmd represents the dump command
var dumpCmd = &cobra.Command{
Use: "dump",
Short: "Dumps the contents of this vellum FST file",
Long: `Dumps the contents of this vellum FST file.`,
PreRunE: func(cmd *cobra.Command, args []string) error {
if len(args) < 1 {
return fmt.Errorf("path is required")
}
return nil
},
RunE: func(cmd *cobra.Command, args []string) error {
fst, err := vellum.Open(args[0])
if err != nil {
return err
}
return fst.Debug(debugPrint)
},
}
func debugPrint(n int, state interface{}) error {
fmt.Printf("%v\n", state)
return nil
}
func init() {
RootCmd.AddCommand(dumpCmd)
}
================================================
FILE: cmd/vellum/cmd/fuzzy.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cmd
import (
"fmt"
"github.com/couchbase/vellum"
"github.com/couchbase/vellum/levenshtein"
"github.com/spf13/cobra"
)
var query string
var distance int
var fuzzyCmd = &cobra.Command{
Use: "fuzzy",
Short: "Fuzzy runs a fuzzy query over the contents of this vellum FST file",
Long: `Fuzzy runs a fuzzy query over the contents of this vellum FST file.`,
PreRunE: func(cmd *cobra.Command, args []string) error {
if len(args) < 1 {
return fmt.Errorf("path is required")
}
if len(args) > 1 {
query = args[1]
}
return nil
},
RunE: func(cmd *cobra.Command, args []string) error {
fst, err := vellum.Open(args[0])
if err != nil {
return err
}
lb, err := levenshtein.NewLevenshteinAutomatonBuilder(uint8(distance), false)
if err != nil {
return err
}
fuzzy, err := lb.BuildDfa(query, uint8(distance))
if err != nil {
return err
}
var startKeyB, endKeyB []byte
if startKey != "" {
startKeyB = []byte(startKey)
}
if endKey != "" {
endKeyB = []byte(endKey)
}
itr, err := fst.Search(fuzzy, startKeyB, endKeyB)
for err == nil {
key, val := itr.Current()
fmt.Printf("%s - %d\n", key, val)
err = itr.Next()
}
return nil
},
}
func init() {
RootCmd.AddCommand(fuzzyCmd)
fuzzyCmd.Flags().StringVar(&startKey, "start", "", "start key inclusive")
fuzzyCmd.Flags().StringVar(&endKey, "end", "", "end key inclusive")
fuzzyCmd.Flags().IntVar(&distance, "distance", 1, "edit distance in Unicode codepoints")
}
================================================
FILE: cmd/vellum/cmd/grep.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cmd
import (
"fmt"
"github.com/couchbase/vellum"
"github.com/couchbase/vellum/regexp"
"github.com/spf13/cobra"
)
var grepCmd = &cobra.Command{
Use: "grep",
Short: "Grep runs regular expression searches over the contents of this " +
"vellum FST file.",
Long: `Grep runs regular expression searches over the contents of this ` +
`vellum FST file.`,
PreRunE: func(cmd *cobra.Command, args []string) error {
if len(args) < 1 {
return fmt.Errorf("path is required")
}
if len(args) > 1 {
query = args[1]
}
return nil
},
RunE: func(cmd *cobra.Command, args []string) error {
fst, err := vellum.Open(args[0])
if err != nil {
return err
}
r, err := regexp.New(query)
if err != nil {
return err
}
var startKeyB, endKeyB []byte
if startKey != "" {
startKeyB = []byte(startKey)
}
if endKey != "" {
endKeyB = []byte(endKey)
}
itr, err := fst.Search(r, startKeyB, endKeyB)
for err == nil {
key, val := itr.Current()
fmt.Printf("%s - %d\n", key, val)
err = itr.Next()
}
return nil
},
}
func init() {
RootCmd.AddCommand(grepCmd)
grepCmd.Flags().StringVar(&startKey, "start", "", "start key inclusive")
grepCmd.Flags().StringVar(&endKey, "end", "", "end key inclusive")
}
================================================
FILE: cmd/vellum/cmd/info.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cmd
import (
"fmt"
"github.com/couchbase/vellum"
"github.com/spf13/cobra"
)
var infoCmd = &cobra.Command{
Use: "info",
Short: "Prints info about this vellum FST file",
Long: `Prints info about this vellum FST file.`,
PreRunE: func(cmd *cobra.Command, args []string) error {
if len(args) < 1 {
return fmt.Errorf("path is required")
}
return nil
},
RunE: func(cmd *cobra.Command, args []string) error {
fst, err := vellum.Open(args[0])
if err != nil {
return err
}
fmt.Printf("version: %d\n", fst.Version())
fmt.Printf("length: %d\n", fst.Len())
return nil
},
}
func init() {
RootCmd.AddCommand(infoCmd)
}
================================================
FILE: cmd/vellum/cmd/map.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cmd
import (
"encoding/csv"
"fmt"
"io"
"log"
"os"
"strconv"
"github.com/couchbase/vellum"
"github.com/spf13/cobra"
)
var mapCmd = &cobra.Command{
Use: "map",
Short: "Map builds a new FST from a CSV file containing key,val pairs",
Long: `Map builds a new FST from a CSV file containing key,val pairs.`,
PreRunE: func(cmd *cobra.Command, args []string) error {
if len(args) < 1 {
return fmt.Errorf("source and target paths are required")
}
if len(args) < 2 {
return fmt.Errorf("target path is required")
}
return nil
},
RunE: func(cmd *cobra.Command, args []string) error {
if !sorted {
return fmt.Errorf("only sorted input supported at this time")
}
file, err := os.Open(args[0])
if err != nil {
log.Fatal(err)
}
defer file.Close()
f, err := os.Create(args[1])
if err != nil {
return err
}
b, err := vellum.New(f, nil)
if err != nil {
return err
}
reader := csv.NewReader(file)
reader.FieldsPerRecord = 2
var record []string
record, err = reader.Read()
for err == nil {
var v uint64
v, err = strconv.ParseUint(record[1], 10, 64)
if err != nil {
return err
}
err = b.Insert([]byte(record[0]), v)
if err != nil {
return err
}
record, err = reader.Read()
}
if err != io.EOF {
return err
}
err = b.Close()
if err != nil {
return err
}
return nil
},
}
func init() {
RootCmd.AddCommand(mapCmd)
mapCmd.Flags().BoolVar(&sorted, "sorted", false, "input already sorted")
}
================================================
FILE: cmd/vellum/cmd/range.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cmd
import (
"fmt"
"github.com/couchbase/vellum"
"github.com/spf13/cobra"
)
var startKey string
var endKey string
var rangeCmd = &cobra.Command{
Use: "range",
Short: "Range iterates over the contents of this vellum FST file",
Long: `Range iterates over the contents of this vellum FST file. You can optionally specify start/end keys after the filename.`,
PreRunE: func(cmd *cobra.Command, args []string) error {
if len(args) < 1 {
return fmt.Errorf("path is required")
}
return nil
},
RunE: func(cmd *cobra.Command, args []string) error {
fst, err := vellum.Open(args[0])
if err != nil {
return err
}
var startKeyB, endKeyB []byte
if startKey != "" {
startKeyB = []byte(startKey)
}
if endKey != "" {
endKeyB = []byte(endKey)
}
itr, err := fst.Iterator(startKeyB, endKeyB)
for err == nil {
key, val := itr.Current()
fmt.Printf("%s - %d\n", key, val)
err = itr.Next()
}
return nil
},
}
func init() {
RootCmd.AddCommand(rangeCmd)
rangeCmd.Flags().StringVar(&startKey, "start", "", "start key inclusive")
rangeCmd.Flags().StringVar(&endKey, "end", "", "end key inclusive")
}
================================================
FILE: cmd/vellum/cmd/root.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cmd
import (
"fmt"
"net/http"
"os"
"github.com/spf13/cobra"
)
var expvarBind string
// RootCmd represents the base command when called without any subcommands
var RootCmd = &cobra.Command{
Use: "vellum",
Short: "A utility to work with vellum FST files",
Long: `A utility to work with vellum FST files.`,
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
if expvarBind != "" {
go http.ListenAndServe(expvarBind, nil)
}
return nil
},
}
// Execute adds all child commands to the root command sets flags appropriately.
// This is called by main.main(). It only needs to happen once to the rootCmd.
func Execute() {
if err := RootCmd.Execute(); err != nil {
fmt.Println(err)
os.Exit(-1)
}
}
func init() {
RootCmd.PersistentFlags().StringVar(&expvarBind, "expvar", "", "bind address for expvar, default none")
}
================================================
FILE: cmd/vellum/cmd/set.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cmd
import (
"bufio"
"fmt"
"log"
"os"
"github.com/couchbase/vellum"
"github.com/spf13/cobra"
)
var sorted bool
var setCmd = &cobra.Command{
Use: "set",
Short: "Set builds a new FST from a file containing new-line separated values",
Long: `Set builds a new FST from a file containing new-line separated values.`,
PreRunE: func(cmd *cobra.Command, args []string) error {
if len(args) < 1 {
return fmt.Errorf("source and target paths are required")
}
if len(args) < 2 {
return fmt.Errorf("target path is required")
}
return nil
},
RunE: func(cmd *cobra.Command, args []string) error {
if !sorted {
return fmt.Errorf("only sorted input supported at this time")
}
file, err := os.Open(args[0])
if err != nil {
log.Fatal(err)
}
defer file.Close()
f, err := os.Create(args[1])
if err != nil {
return err
}
b, err := vellum.New(f, nil)
if err != nil {
return err
}
scanner := bufio.NewScanner(file)
for scanner.Scan() {
word := append([]byte(nil), scanner.Bytes()...)
err = b.Insert(word, 0)
if err != nil {
return err
}
}
if err = scanner.Err(); err != nil {
log.Fatal(err)
}
err = b.Close()
if err != nil {
return err
}
return nil
},
}
func init() {
RootCmd.AddCommand(setCmd)
setCmd.Flags().BoolVar(&sorted, "sorted", false, "input already sorted")
}
================================================
FILE: cmd/vellum/cmd/svg.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cmd
import (
"fmt"
"io"
"io/ioutil"
"os"
"os/exec"
"github.com/couchbase/vellum"
"github.com/spf13/cobra"
)
var svgCmd = &cobra.Command{
Use: "svg",
Short: "SVG prints the contents of this vellum FST file in the SVG format",
Long: `SVG prints the contents of this vellum FST file in the SVG format.`,
PreRunE: func(cmd *cobra.Command, args []string) error {
if len(args) < 1 {
return fmt.Errorf("path is required")
}
return nil
},
RunE: func(cmd *cobra.Command, args []string) error {
fst, err := vellum.Open(args[0])
if err != nil {
return err
}
return svgToWriter(fst, os.Stdout)
},
}
func svgToWriter(fst *vellum.FST, w io.Writer) error {
pr, pw := io.Pipe()
go func() {
defer func() {
_ = pw.Close()
}()
_ = dotToWriter(fst, pw)
}()
cmd := exec.Command("dot", "-Tsvg")
cmd.Stdin = pr
cmd.Stdout = w
cmd.Stderr = ioutil.Discard
err := cmd.Run()
if err != nil {
return err
}
return nil
}
func init() {
RootCmd.AddCommand(svgCmd)
}
================================================
FILE: cmd/vellum/main.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
_ "expvar"
"github.com/couchbase/vellum/cmd/vellum/cmd"
)
func main() {
cmd.Execute()
}
================================================
FILE: common.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum
const maxCommon = 1<<6 - 1
func encodeCommon(in byte) byte {
val := byte((int(commonInputs[in]) + 1) % 256)
if val > maxCommon {
return 0
}
return val
}
func decodeCommon(in byte) byte {
return commonInputsInv[in-1]
}
var commonInputs = []byte{
84, // '\x00'
85, // '\x01'
86, // '\x02'
87, // '\x03'
88, // '\x04'
89, // '\x05'
90, // '\x06'
91, // '\x07'
92, // '\x08'
93, // '\t'
94, // '\n'
95, // '\x0b'
96, // '\x0c'
97, // '\r'
98, // '\x0e'
99, // '\x0f'
100, // '\x10'
101, // '\x11'
102, // '\x12'
103, // '\x13'
104, // '\x14'
105, // '\x15'
106, // '\x16'
107, // '\x17'
108, // '\x18'
109, // '\x19'
110, // '\x1a'
111, // '\x1b'
112, // '\x1c'
113, // '\x1d'
114, // '\x1e'
115, // '\x1f'
116, // ' '
80, // '!'
117, // '"'
118, // '#'
79, // '$'
39, // '%'
30, // '&'
81, // "'"
75, // '('
74, // ')'
82, // '*'
57, // '+'
66, // ','
16, // '-'
12, // '.'
2, // '/'
19, // '0'
20, // '1'
21, // '2'
27, // '3'
32, // '4'
29, // '5'
35, // '6'
36, // '7'
37, // '8'
34, // '9'
24, // ':'
73, // ';'
119, // '<'
23, // '='
120, // '>'
40, // '?'
83, // '@'
44, // 'A'
48, // 'B'
42, // 'C'
43, // 'D'
49, // 'E'
46, // 'F'
62, // 'G'
61, // 'H'
47, // 'I'
69, // 'J'
68, // 'K'
58, // 'L'
56, // 'M'
55, // 'N'
59, // 'O'
51, // 'P'
72, // 'Q'
54, // 'R'
45, // 'S'
52, // 'T'
64, // 'U'
65, // 'V'
63, // 'W'
71, // 'X'
67, // 'Y'
70, // 'Z'
77, // '['
121, // '\\'
78, // ']'
122, // '^'
31, // '_'
123, // '`'
4, // 'a'
25, // 'b'
9, // 'c'
17, // 'd'
1, // 'e'
26, // 'f'
22, // 'g'
13, // 'h'
7, // 'i'
50, // 'j'
38, // 'k'
14, // 'l'
15, // 'm'
10, // 'n'
3, // 'o'
8, // 'p'
60, // 'q'
6, // 'r'
5, // 's'
0, // 't'
18, // 'u'
33, // 'v'
11, // 'w'
41, // 'x'
28, // 'y'
53, // 'z'
124, // '{'
125, // '|'
126, // '}'
76, // '~'
127, // '\x7f'
128, // '\x80'
129, // '\x81'
130, // '\x82'
131, // '\x83'
132, // '\x84'
133, // '\x85'
134, // '\x86'
135, // '\x87'
136, // '\x88'
137, // '\x89'
138, // '\x8a'
139, // '\x8b'
140, // '\x8c'
141, // '\x8d'
142, // '\x8e'
143, // '\x8f'
144, // '\x90'
145, // '\x91'
146, // '\x92'
147, // '\x93'
148, // '\x94'
149, // '\x95'
150, // '\x96'
151, // '\x97'
152, // '\x98'
153, // '\x99'
154, // '\x9a'
155, // '\x9b'
156, // '\x9c'
157, // '\x9d'
158, // '\x9e'
159, // '\x9f'
160, // '\xa0'
161, // '¡'
162, // '¢'
163, // '£'
164, // '¤'
165, // '¥'
166, // '¦'
167, // '§'
168, // '¨'
169, // '©'
170, // 'ª'
171, // '«'
172, // '¬'
173, // '\xad'
174, // '®'
175, // '¯'
176, // '°'
177, // '±'
178, // '²'
179, // '³'
180, // '´'
181, // 'µ'
182, // '¶'
183, // '·'
184, // '¸'
185, // '¹'
186, // 'º'
187, // '»'
188, // '¼'
189, // '½'
190, // '¾'
191, // '¿'
192, // 'À'
193, // 'Á'
194, // 'Â'
195, // 'Ã'
196, // 'Ä'
197, // 'Å'
198, // 'Æ'
199, // 'Ç'
200, // 'È'
201, // 'É'
202, // 'Ê'
203, // 'Ë'
204, // 'Ì'
205, // 'Í'
206, // 'Î'
207, // 'Ï'
208, // 'Ð'
209, // 'Ñ'
210, // 'Ò'
211, // 'Ó'
212, // 'Ô'
213, // 'Õ'
214, // 'Ö'
215, // '×'
216, // 'Ø'
217, // 'Ù'
218, // 'Ú'
219, // 'Û'
220, // 'Ü'
221, // 'Ý'
222, // 'Þ'
223, // 'ß'
224, // 'à'
225, // 'á'
226, // 'â'
227, // 'ã'
228, // 'ä'
229, // 'å'
230, // 'æ'
231, // 'ç'
232, // 'è'
233, // 'é'
234, // 'ê'
235, // 'ë'
236, // 'ì'
237, // 'í'
238, // 'î'
239, // 'ï'
240, // 'ð'
241, // 'ñ'
242, // 'ò'
243, // 'ó'
244, // 'ô'
245, // 'õ'
246, // 'ö'
247, // '÷'
248, // 'ø'
249, // 'ù'
250, // 'ú'
251, // 'û'
252, // 'ü'
253, // 'ý'
254, // 'þ'
255, // 'ÿ'
}
var commonInputsInv = []byte{
't',
'e',
'/',
'o',
'a',
's',
'r',
'i',
'p',
'c',
'n',
'w',
'.',
'h',
'l',
'm',
'-',
'd',
'u',
'0',
'1',
'2',
'g',
'=',
':',
'b',
'f',
'3',
'y',
'5',
'&',
'_',
'4',
'v',
'9',
'6',
'7',
'8',
'k',
'%',
'?',
'x',
'C',
'D',
'A',
'S',
'F',
'I',
'B',
'E',
'j',
'P',
'T',
'z',
'R',
'N',
'M',
'+',
'L',
'O',
'q',
'H',
'G',
'W',
'U',
'V',
',',
'Y',
'K',
'J',
'Z',
'X',
'Q',
';',
')',
'(',
'~',
'[',
']',
'$',
'!',
'\'',
'*',
'@',
'\x00',
'\x01',
'\x02',
'\x03',
'\x04',
'\x05',
'\x06',
'\x07',
'\x08',
'\t',
'\n',
'\x0b',
'\x0c',
'\r',
'\x0e',
'\x0f',
'\x10',
'\x11',
'\x12',
'\x13',
'\x14',
'\x15',
'\x16',
'\x17',
'\x18',
'\x19',
'\x1a',
'\x1b',
'\x1c',
'\x1d',
'\x1e',
'\x1f',
' ',
'"',
'#',
'<',
'>',
'\\',
'^',
'`',
'{',
'|',
'}',
'\x7f',
'\x80',
'\x81',
'\x82',
'\x83',
'\x84',
'\x85',
'\x86',
'\x87',
'\x88',
'\x89',
'\x8a',
'\x8b',
'\x8c',
'\x8d',
'\x8e',
'\x8f',
'\x90',
'\x91',
'\x92',
'\x93',
'\x94',
'\x95',
'\x96',
'\x97',
'\x98',
'\x99',
'\x9a',
'\x9b',
'\x9c',
'\x9d',
'\x9e',
'\x9f',
'\xa0',
'\xa1',
'\xa2',
'\xa3',
'\xa4',
'\xa5',
'\xa6',
'\xa7',
'\xa8',
'\xa9',
'\xaa',
'\xab',
'\xac',
'\xad',
'\xae',
'\xaf',
'\xb0',
'\xb1',
'\xb2',
'\xb3',
'\xb4',
'\xb5',
'\xb6',
'\xb7',
'\xb8',
'\xb9',
'\xba',
'\xbb',
'\xbc',
'\xbd',
'\xbe',
'\xbf',
'\xc0',
'\xc1',
'\xc2',
'\xc3',
'\xc4',
'\xc5',
'\xc6',
'\xc7',
'\xc8',
'\xc9',
'\xca',
'\xcb',
'\xcc',
'\xcd',
'\xce',
'\xcf',
'\xd0',
'\xd1',
'\xd2',
'\xd3',
'\xd4',
'\xd5',
'\xd6',
'\xd7',
'\xd8',
'\xd9',
'\xda',
'\xdb',
'\xdc',
'\xdd',
'\xde',
'\xdf',
'\xe0',
'\xe1',
'\xe2',
'\xe3',
'\xe4',
'\xe5',
'\xe6',
'\xe7',
'\xe8',
'\xe9',
'\xea',
'\xeb',
'\xec',
'\xed',
'\xee',
'\xef',
'\xf0',
'\xf1',
'\xf2',
'\xf3',
'\xf4',
'\xf5',
'\xf6',
'\xf7',
'\xf8',
'\xf9',
'\xfa',
'\xfb',
'\xfc',
'\xfd',
'\xfe',
'\xff',
}
================================================
FILE: common_test.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum
import "testing"
func TestCommonInputs(t *testing.T) {
// first ensure items that can be encoded round trip properly
for i := 0; i < 256; i++ {
roundTrip(t, byte(i))
}
// G maps to 62, +1 is 63, which is highest 6-bit value we can encode
enc := encodeCommon('G')
if enc != 63 {
t.Errorf("expected G to encode to 63, got %d", enc)
}
// W encodes to 63, +1 is 64, which is too big to fit
enc = encodeCommon('W')
if enc != 0 {
t.Errorf("expected W to encode to 0, got %d", enc)
}
}
func roundTrip(t *testing.T, b byte) {
enc := encodeCommon(b)
if enc > 0 {
dec := decodeCommon(enc)
if dec != b {
t.Errorf("error round trip common input: %d", b)
}
}
}
================================================
FILE: data/words-1000.txt
================================================
American
Congress
Democrat
I
Mr
Mrs
PM
Republican
TV
a
ability
able
about
above
accept
according
account
across
act
action
activity
actually
add
address
administration
admit
adult
affect
after
again
against
age
agency
agent
ago
agree
agreement
ahead
air
all
allow
almost
alone
along
already
also
although
always
among
amount
analysis
and
animal
another
answer
any
anyone
anything
appear
apply
approach
area
argue
arm
around
arrive
art
article
artist
as
ask
assume
at
attack
attention
attorney
audience
author
authority
available
avoid
away
baby
back
bad
bag
ball
bank
bar
base
be
beat
beautiful
because
become
bed
before
begin
behavior
behind
believe
benefit
best
better
between
beyond
big
bill
billion
bit
black
blood
blue
board
body
book
born
both
box
boy
break
bring
brother
budget
build
building
business
but
buy
by
call
camera
campaign
can
cancer
candidate
capital
car
card
care
career
carry
case
catch
cause
cell
center
central
century
certain
certainly
chair
challenge
chance
change
character
charge
check
child
choice
choose
church
citizen
city
civil
claim
class
clear
clearly
close
coach
cold
collection
college
color
come
commercial
common
community
company
compare
computer
concern
condition
conference
consider
consumer
contain
continue
control
cost
could
country
couple
course
court
cover
create
crime
cultural
culture
cup
current
customer
cut
dark
data
daughter
day
dead
deal
death
debate
decade
decide
decision
deep
defense
degree
democratic
describe
design
despite
detail
determine
develop
development
die
difference
different
difficult
dinner
direction
director
discover
discuss
discussion
disease
do
doctor
dog
door
down
draw
dream
drive
drop
drug
during
each
early
east
easy
eat
economic
economy
edge
education
effect
effort
eight
either
election
else
employee
end
energy
enjoy
enough
enter
entire
environment
environmental
especially
establish
even
evening
event
ever
every
everybody
everyone
everything
evidence
exactly
example
executive
exist
expect
experience
expert
explain
eye
face
fact
factor
fail
fall
family
far
fast
father
fear
federal
feel
feeling
few
field
fight
figure
fill
film
final
finally
financial
find
fine
finger
finish
fire
firm
first
fish
five
floor
fly
focus
follow
food
foot
for
force
foreign
forget
form
former
forward
four
free
friend
from
front
full
fund
future
game
garden
gas
general
generation
get
girl
give
glass
go
goal
good
government
great
green
ground
group
grow
growth
guess
gun
guy
hair
half
hand
hang
happen
happy
hard
have
he
head
health
hear
heart
heat
heavy
help
her
here
herself
high
him
himself
his
history
hit
hold
home
hope
hospital
hot
hotel
hour
house
how
however
huge
human
hundred
husband
idea
identify
if
image
imagine
impact
important
improve
in
include
including
increase
indeed
indicate
individual
industry
information
inside
instead
institution
interest
interesting
international
interview
into
investment
involve
issue
it
item
its
itself
job
join
just
keep
key
kid
kill
kind
kitchen
know
knowledge
land
language
large
last
late
later
laugh
law
lawyer
lay
lead
leader
learn
least
leave
left
leg
legal
less
let
letter
level
lie
life
light
like
likely
line
list
listen
little
live
local
long
look
lose
loss
lot
love
low
machine
magazine
main
maintain
major
majority
make
man
manage
management
manager
many
market
marriage
material
matter
may
maybe
me
mean
measure
media
medical
meet
meeting
member
memory
mention
message
method
middle
might
military
million
mind
minute
miss
mission
model
modern
moment
money
month
more
morning
most
mother
mouth
move
movement
movie
much
music
must
my
myself
n't
name
nation
national
natural
nature
near
nearly
necessary
need
network
never
new
news
newspaper
next
nice
night
no
none
nor
north
not
note
nothing
notice
now
number
occur
of
off
offer
office
officer
official
often
oh
oil
ok
old
on
once
one
only
onto
open
operation
opportunity
option
or
order
organization
other
others
our
out
outside
over
own
owner
page
pain
painting
paper
parent
part
participant
particular
particularly
partner
party
pass
past
patient
pattern
pay
peace
people
per
perform
performance
perhaps
period
person
personal
phone
physical
pick
picture
piece
place
plan
plant
play
player
point
police
policy
political
politics
poor
popular
population
position
positive
possible
power
practice
prepare
present
president
pressure
pretty
prevent
price
private
probably
problem
process
produce
product
production
professional
professor
program
project
property
protect
prove
provide
public
pull
purpose
push
put
quality
question
quickly
quite
race
radio
raise
range
rate
rather
reach
read
ready
real
reality
realize
really
reason
receive
recent
recently
recognize
record
red
reduce
reflect
region
relate
relationship
religious
remain
remember
remove
report
represent
require
research
resource
respond
response
responsibility
rest
result
return
reveal
rich
right
rise
risk
road
rock
role
room
rule
run
safe
same
save
say
scene
school
science
scientist
score
sea
season
seat
second
section
security
see
seek
seem
sell
send
senior
sense
series
serious
serve
service
set
seven
several
sex
sexual
shake
share
she
shoot
short
shot
should
shoulder
show
side
sign
significant
similar
simple
simply
since
sing
single
sister
sit
site
situation
six
size
skill
skin
small
smile
so
social
society
soldier
some
somebody
someone
something
sometimes
son
song
soon
sort
sound
source
south
southern
space
speak
special
specific
speech
spend
sport
spring
staff
stage
stand
standard
star
start
state
statement
station
stay
step
still
stock
stop
store
story
strategy
street
strong
structure
student
study
stuff
style
subject
success
successful
such
suddenly
suffer
suggest
summer
support
sure
surface
system
table
take
talk
task
tax
teach
teacher
team
technology
television
tell
ten
tend
term
test
than
thank
that
the
their
them
themselves
then
theory
there
these
they
thing
think
third
this
those
though
thought
thousand
threat
three
through
throughout
throw
thus
time
to
today
together
tonight
too
top
total
tough
toward
town
trade
traditional
training
travel
treat
treatment
tree
trial
trip
trouble
true
truth
try
turn
two
type
under
understand
unit
until
up
upon
us
use
usually
value
various
very
victim
view
violence
visit
voice
vote
wait
walk
wall
want
war
watch
water
way
we
weapon
wear
week
weight
well
west
western
what
whatever
when
where
whether
which
while
white
who
whole
whom
whose
why
wide
wife
will
win
wind
window
wish
with
within
without
woman
wonder
word
work
worker
world
worry
would
write
writer
wrong
yard
yeah
year
yes
yet
you
young
your
yourself
================================================
FILE: decoder_v1.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum
import (
"bytes"
"encoding/binary"
"fmt"
"strconv"
)
func init() {
registerDecoder(versionV1, func(data []byte) decoder {
return newDecoderV1(data)
})
}
type decoderV1 struct {
data []byte
}
func newDecoderV1(data []byte) *decoderV1 {
return &decoderV1{
data: data,
}
}
func (d *decoderV1) getRoot() int {
if len(d.data) < footerSizeV1 {
return noneAddr
}
footer := d.data[len(d.data)-footerSizeV1:]
root := binary.LittleEndian.Uint64(footer[8:])
return int(root)
}
func (d *decoderV1) getLen() int {
if len(d.data) < footerSizeV1 {
return 0
}
footer := d.data[len(d.data)-footerSizeV1:]
dlen := binary.LittleEndian.Uint64(footer)
return int(dlen)
}
func (d *decoderV1) stateAt(addr int, prealloc fstState) (fstState, error) {
state, ok := prealloc.(*fstStateV1)
if ok && state != nil {
*state = fstStateV1{} // clear the struct
} else {
state = &fstStateV1{}
}
err := state.at(d.data, addr)
if err != nil {
return nil, err
}
return state, nil
}
type fstStateV1 struct {
data []byte
top int
bottom int
numTrans int
// single trans only
singleTransChar byte
singleTransNext bool
singleTransAddr uint64
singleTransOut uint64
// shared
transSize int
outSize int
// multiple trans only
final bool
transTop int
transBottom int
destTop int
destBottom int
outTop int
outBottom int
outFinal int
}
func (f *fstStateV1) isEncodedSingle() bool {
if f.data[f.top]>>7 > 0 {
return true
}
return false
}
func (f *fstStateV1) at(data []byte, addr int) error {
f.data = data
if addr == emptyAddr {
return f.atZero()
} else if addr == noneAddr {
return f.atNone()
}
if addr > len(data) || addr < 16 {
return fmt.Errorf("invalid address %d/%d", addr, len(data))
}
f.top = addr
f.bottom = addr
if f.isEncodedSingle() {
return f.atSingle(data, addr)
}
return f.atMulti(data, addr)
}
func (f *fstStateV1) atZero() error {
f.top = 0
f.bottom = 1
f.numTrans = 0
f.final = true
f.outFinal = 0
return nil
}
func (f *fstStateV1) atNone() error {
f.top = 0
f.bottom = 1
f.numTrans = 0
f.final = false
f.outFinal = 0
return nil
}
func (f *fstStateV1) atSingle(data []byte, addr int) error {
// handle single transition case
f.numTrans = 1
f.singleTransNext = data[f.top]&transitionNext > 0
f.singleTransChar = data[f.top] & maxCommon
if f.singleTransChar == 0 {
f.bottom-- // extra byte for uncommon
f.singleTransChar = data[f.bottom]
} else {
f.singleTransChar = decodeCommon(f.singleTransChar)
}
if f.singleTransNext {
// now we know the bottom, can compute next addr
f.singleTransAddr = uint64(f.bottom - 1)
f.singleTransOut = 0
} else {
f.bottom-- // extra byte with pack sizes
f.transSize, f.outSize = decodePackSize(data[f.bottom])
f.bottom -= f.transSize // exactly one trans
f.singleTransAddr = readPackedUint(data[f.bottom : f.bottom+f.transSize])
if f.outSize > 0 {
f.bottom -= f.outSize // exactly one out (could be length 0 though)
f.singleTransOut = readPackedUint(data[f.bottom : f.bottom+f.outSize])
} else {
f.singleTransOut = 0
}
// need to wait till we know bottom
if f.singleTransAddr != 0 {
f.singleTransAddr = uint64(f.bottom) - f.singleTransAddr
}
}
return nil
}
func (f *fstStateV1) atMulti(data []byte, addr int) error {
// handle multiple transitions case
f.final = data[f.top]&stateFinal > 0
f.numTrans = int(data[f.top] & maxNumTrans)
if f.numTrans == 0 {
f.bottom-- // extra byte for number of trans
f.numTrans = int(data[f.bottom])
if f.numTrans == 1 {
// can't really be 1 here, this is special case that means 256
f.numTrans = 256
}
}
f.bottom-- // extra byte with pack sizes
f.transSize, f.outSize = decodePackSize(data[f.bottom])
f.transTop = f.bottom
f.bottom -= f.numTrans // one byte for each transition
f.transBottom = f.bottom
f.destTop = f.bottom
f.bottom -= f.numTrans * f.transSize
f.destBottom = f.bottom
if f.outSize > 0 {
f.outTop = f.bottom
f.bottom -= f.numTrans * f.outSize
f.outBottom = f.bottom
if f.final {
f.bottom -= f.outSize
f.outFinal = f.bottom
}
}
return nil
}
func (f *fstStateV1) Address() int {
return f.top
}
func (f *fstStateV1) Final() bool {
return f.final
}
func (f *fstStateV1) FinalOutput() uint64 {
if f.final && f.outSize > 0 {
return readPackedUint(f.data[f.outFinal : f.outFinal+f.outSize])
}
return 0
}
func (f *fstStateV1) NumTransitions() int {
return f.numTrans
}
func (f *fstStateV1) TransitionAt(i int) byte {
if f.isEncodedSingle() {
return f.singleTransChar
}
transitionKeys := f.data[f.transBottom:f.transTop]
return transitionKeys[f.numTrans-i-1]
}
func (f *fstStateV1) TransitionFor(b byte) (int, int, uint64) {
if f.isEncodedSingle() {
if f.singleTransChar == b {
return 0, int(f.singleTransAddr), f.singleTransOut
}
return -1, noneAddr, 0
}
transitionKeys := f.data[f.transBottom:f.transTop]
pos := bytes.IndexByte(transitionKeys, b)
if pos < 0 {
return -1, noneAddr, 0
}
transDests := f.data[f.destBottom:f.destTop]
dest := int(readPackedUint(transDests[pos*f.transSize : pos*f.transSize+f.transSize]))
if dest > 0 {
// convert delta
dest = f.bottom - dest
}
transVals := f.data[f.outBottom:f.outTop]
var out uint64
if f.outSize > 0 {
out = readPackedUint(transVals[pos*f.outSize : pos*f.outSize+f.outSize])
}
return f.numTrans - pos - 1, dest, out
}
func (f *fstStateV1) String() string {
rv := ""
rv += fmt.Sprintf("State: %d (%#x)", f.top, f.top)
if f.final {
rv += " final"
fout := f.FinalOutput()
if fout != 0 {
rv += fmt.Sprintf(" (%d)", fout)
}
}
rv += "\n"
rv += fmt.Sprintf("Data: % x\n", f.data[f.bottom:f.top+1])
for i := 0; i < f.numTrans; i++ {
transChar := f.TransitionAt(i)
_, transDest, transOut := f.TransitionFor(transChar)
rv += fmt.Sprintf(" - %d (%#x) '%s' ---> %d (%#x) with output: %d", transChar, transChar, string(transChar), transDest, transDest, transOut)
rv += "\n"
}
if f.numTrans == 0 {
rv += "\n"
}
return rv
}
func (f *fstStateV1) DotString(num int) string {
rv := ""
label := fmt.Sprintf("%d", num)
final := ""
if f.final {
final = ",peripheries=2"
}
rv += fmt.Sprintf(" %d [label=\"%s\"%s];\n", f.top, label, final)
for i := 0; i < f.numTrans; i++ {
transChar := f.TransitionAt(i)
_, transDest, transOut := f.TransitionFor(transChar)
out := ""
if transOut != 0 {
out = fmt.Sprintf("/%d", transOut)
}
rv += fmt.Sprintf(" %d -> %d [label=\"%s%s\"];\n", f.top, transDest, escapeInput(transChar), out)
}
return rv
}
func escapeInput(b byte) string {
x := strconv.AppendQuoteRune(nil, rune(b))
return string(x[1:(len(x) - 1)])
}
================================================
FILE: decoder_v1_test.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum
import (
"reflect"
"testing"
)
func TestDecoderVersionError(t *testing.T) {
_, err := loadDecoder(629, nil)
if err == nil {
t.Errorf("expected error loading decoder version 629, got nil")
}
}
func TestShortHeader(t *testing.T) {
header := make([]byte, 15)
_, _, err := decodeHeader(header)
if err == nil {
t.Errorf("expected error decoding short header, got nil")
}
}
func TestDecoderRootLen(t *testing.T) {
d := newDecoderV1([]byte{1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0})
if d.getLen() != 1 {
t.Fatalf("expected parsed footer length 1, got %d", d.getLen())
}
if d.getRoot() != 2 {
t.Fatalf("expected parsed footer length 2, got %d", d.getLen())
}
}
func TestDecoderStateAt(t *testing.T) {
tests := []struct {
desc string
data []byte
want *fstStateV1
}{
{
"one trans, trans next, common char",
[]byte{
// header
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
// test node data
oneTransition | transitionNext | encodeCommon('a'),
// footer
1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0,
},
&fstStateV1{
numTrans: 1,
top: 16,
bottom: 16,
singleTransChar: 'a',
singleTransNext: true,
singleTransAddr: 15,
},
},
{
"one trans, trans next, uncommon char",
[]byte{
// header
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
// test node data
0xff,
oneTransition | transitionNext,
// footer
1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0,
},
&fstStateV1{
numTrans: 1,
top: 17,
bottom: 16,
singleTransChar: 0xff,
singleTransNext: true,
singleTransAddr: 15,
},
},
{
"one trans, trans not next, common char",
[]byte{
// header
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
// test node data
4, // delta address packed
1<<4 | 0, // pack sizes
oneTransition | encodeCommon('a'),
// footer
1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0,
},
&fstStateV1{
numTrans: 1,
top: 18,
bottom: 16,
singleTransChar: 'a',
singleTransNext: false,
singleTransAddr: 12,
transSize: 1,
},
},
{
"one trans, trans not next, uncommon char",
[]byte{
// header
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
// test node data
4, // delta address packed
1<<4 | 0, // pack sizes
0xff,
oneTransition,
// footer
1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0,
},
&fstStateV1{
numTrans: 1,
top: 19,
bottom: 16,
singleTransChar: 0xff,
singleTransNext: false,
singleTransAddr: 12,
transSize: 1,
},
},
{
"one trans, trans not next, common char, with value",
[]byte{
// header
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
// test node data
27, // trans value
4, // delta address packed
1<<4 | 1, // pack sizes
oneTransition | encodeCommon('a'),
// footer
1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0,
},
&fstStateV1{
numTrans: 1,
top: 19,
bottom: 16,
singleTransChar: 'a',
singleTransNext: false,
singleTransAddr: 12,
singleTransOut: 27,
transSize: 1,
outSize: 1,
},
},
{
"one trans, trans not next, uncommon char, with value",
[]byte{
// header
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
// test node data
39, // trans val
4, // delta address packed
1<<4 | 1, // pack sizes
0xff,
oneTransition,
// footer
1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0,
},
&fstStateV1{
numTrans: 1,
top: 20,
bottom: 16,
singleTransChar: 0xff,
singleTransNext: false,
singleTransAddr: 12,
singleTransOut: 39,
transSize: 1,
outSize: 1,
},
},
{
"many trans, not final, no values",
[]byte{
// header
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
// test node data
2, // delta addresses packed
3,
4,
'c', // encoded keys reversed
'b',
'a',
1<<4 | 0, // pack sizes
encodeNumTrans(3),
// footer
1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0,
},
&fstStateV1{
numTrans: 3,
top: 23,
bottom: 16,
transSize: 1,
destBottom: 16,
destTop: 19,
transBottom: 19,
transTop: 22,
},
},
{
"many trans, not final, with values",
[]byte{
// header
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
// test node data
7, // values reversed
0,
3,
2, // delta addresses reversed
3,
4,
'c', // encoded keys reversed
'b',
'a',
1<<4 | 1, // pack sizes
encodeNumTrans(3),
// footer
1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0,
},
&fstStateV1{
numTrans: 3,
top: 26,
bottom: 16,
transSize: 1,
outSize: 1,
outBottom: 16,
outTop: 19,
destBottom: 19,
destTop: 22,
transBottom: 22,
transTop: 25,
},
},
{
"many trans, final, with values",
[]byte{
// header
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
// test node data
9, // node final val
7, // values reversed
0,
3,
2, // delta addresses reversed
3,
4,
'c', // encoded keys reversed
'b',
'a',
1<<4 | 1, // pack sizes
stateFinal | encodeNumTrans(3),
// footer
1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0,
},
&fstStateV1{
final: true,
numTrans: 3,
top: 27,
bottom: 16,
transSize: 1,
outSize: 1,
outBottom: 17,
outTop: 20,
destBottom: 20,
destTop: 23,
transBottom: 23,
transTop: 26,
outFinal: 16,
},
},
{
"max trans, ",
[]byte{
// header
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
// test node data
// delta addresses packed
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4, 0x4,
// encoded keys reversed
0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7, 0x8, 0x9, 0xa, 0xb, 0xc, 0xd, 0xe, 0xf,
0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f,
0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f,
0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f,
0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f,
0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f,
0x60, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, 0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f,
0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, 0x78, 0x79, 0x7a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f,
0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f,
0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, 0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f,
0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf,
0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, 0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf,
0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, 0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf,
0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, 0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf,
0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, 0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef,
0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, 0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff,
1<<4 | 0, // pack sizes
1, // actual trans 1 == 256
0, // zero trans (wont fit)
// footer
1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0,
},
&fstStateV1{
numTrans: 256,
top: 530,
bottom: 16,
transSize: 1,
destBottom: 16,
destTop: 272,
transBottom: 272,
transTop: 528,
},
},
}
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
d := newDecoderV1(test.data)
test.want.data = test.data
got, err := d.stateAt(len(test.data)-17, nil)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(got, test.want) {
t.Errorf("wanted: %+v, got: %+v", test.want, got)
}
addr := got.Address()
if addr != test.want.top {
t.Errorf("expected address to match: %d - %d", addr, test.want.top)
}
fin := got.Final()
if fin != test.want.final {
t.Errorf("expected final to match: %t - %t", fin, test.want.final)
}
ntrans := got.NumTransitions()
if ntrans != test.want.numTrans {
t.Errorf("expected num trans to match: %d - %d", ntrans, test.want.numTrans)
}
})
}
}
func TestFSTStateFinalOutput(t *testing.T) {
tests := []struct {
desc string
in *fstStateV1
want uint64
}{
{
"final output for final state",
&fstStateV1{
data: []byte{7},
numTrans: 2,
final: true,
outSize: 1,
outFinal: 0,
},
7,
},
{
"final output for non-final state",
&fstStateV1{
data: []byte{7},
numTrans: 2,
final: false,
outSize: 1,
},
0,
},
}
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
got := test.in.FinalOutput()
if got != test.want {
t.Errorf("wanted: %d, got: %d", test.want, got)
}
})
}
}
func TestDecodeStateZero(t *testing.T) {
var state fstStateV1
err := state.at(nil, 0)
if err != nil {
t.Fatal(err)
}
if state.numTrans != 0 {
t.Errorf("expected 0 states, got %d", state.numTrans)
}
if !state.final {
t.Errorf("expected state final, got %t", state.final)
}
}
func TestDecodeAtInvalid(t *testing.T) {
var state fstStateV1
err := state.at(nil, 15)
if err == nil {
t.Errorf("expected error invalid address, got nil")
}
}
func TestFSTStateTransitionAt(t *testing.T) {
state := fstStateV1{
data: []byte{oneTransition | encodeCommon('a')},
numTrans: 1,
singleTransChar: 'a',
}
got := state.TransitionAt(0)
if got != state.singleTransChar {
t.Errorf("expected %s got %s", string(state.singleTransChar), string(got))
}
state = fstStateV1{
data: []byte{'b', 'a'},
numTrans: 2,
transBottom: 0,
transTop: 2,
}
got = state.TransitionAt(0)
if got != 'a' {
t.Errorf("expected %s got %s", string('a'), string(got))
}
}
================================================
FILE: docs/format.md
================================================
# vellum file format v1
The v1 file format for vellum has been designed by trying to understand the file format used by [BurntSushi/fst](https://github.com/BurntSushi/fst) library. It should be binary compatible, but no attempt has been made to verify this.
## Overview
The file has 3 sections:
- header
- edge/transition data
- footer
### Header
The header is 16 bytes in total.
- 8 bytes version, uint64 little-endian
- 8 bytes type, uint64 little-endian (currently always 0, no meaning assigned)
A side-effect of this header is that when computing transition target addresses at runtime, any address < 16 is invalid.
### State/Transition Data
A state is encoded with the following sections, HOWEVER, many sections are optional and omitted for various combinations of settings. In the order they occur:
- node final output value (packed integer of the computed output size for this node)
- n transition output values (packed integers, of the computed output size for this node, in REVERSE transition order)
- n transition addresses (delta encoded, relative the lowest byte of this node, packed at the computed transition address size for this node, in REVERSE transition order)
- n transition bytes (1 byte for each transition, in REVERSE transition order)
- pack sizes, 1 byte, high 4 bits transition address size, low 4 bits output size
- number of transitions, 1 byte (ONLY if it didn't fit in the top byte), value of 1 in this byte means 256 (1 would have fit into the top byte)
- single transition byte, 1 byte (ONLY if it didn't fit in the top byte)
- top byte, encodes various flags, and uses remaining bits depending on those flags, broken out separate below
#### State Top Byte
- high bit
- 1 means, this edge has just 1 transition
- 0 means, this edge has multiple transitions
##### 1 transition States
- second bit flags jump to previous
- 1 means this transition target is the immediately preceding state
- 0 means there will transition address in the rest of the data
- remaining 6 bits attempt to encode the transition byte
- Obviously this requires 8 bits, but we map the most frequently used bytes into the lowest 6 bits (see common.go). If the byte we need to encode doesn't fit, we encode 0, and read it fully in the following byte. This allows the most common bytes in a single transition edge to fit into just a single byte.
##### Multiple Transition States
- second bit flags final states
- 1 means this is a final state
- 0 means this is not a final state
- remaining 6 bits attempt to encode the number of transitions
- Obviously, this can require 8 bits, be we assume that many states have fewer transition, and will fit. If the number won't fit, we encode 0 here, and read it fully in the following byte. Because we could 256 transitions, that full byte still isn't enough, so we reuse the value 1 to mean 256. The value of 1 would never naturally occur in this position, since 1 transition would have fit into the top byte (NOTE: single transition states that are final are encoded as multi-transition states, but the value of 1 would fit in the top 6 bytes).
### Single Transition Jump To Previous
The flag marking that a single transition state should jump to the previous state works because we encode all of the node data backwards (ie, we start processing state date with the last byte). Since, at runtime, we can always compute the lowest byte of the state we're in, we can trivially compute the start address of the previous node, just by subtracting one. This allows saving another set of bytes in many cases which would have otherwise been needed to encode that address.
### Delta Addresses
All transition target addresses are delta encoded, relative to the lowest byte in the current state.
### Packed Integer Encoding
For both the output values and transition target addresses, we choose a fixed size number of bytes that will work for encoding all the appropriate values in this state. Because this length will be recorded (in the pack sizes section), we don't need to use varint encoding, we can instead simply use the minimum number of bytes required. So, 8-bit values take just 1 byte, etc. This has the advantage that small values take less space, but the sizes are still fixed, so we can easily navigate without excessive computation.
### Footer
The footer is 16 bytes in total.
- 8 bytes number of keys, uint64 little-endian
- 8 bytes root address (absolute, not delta encoded like other addresses in file), uint64 little-endian
## Encoding Streaming
States are written out to the underlying writer as soon as possible. This allows us to get an early start on I/O while still building the FST, reducing the overall time to build, and it also allows us to reduce the memory consumed during the build process.
Because of this, the root node will always be the last node written in the file.
================================================
FILE: encoder_v1.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum
import (
"encoding/binary"
"fmt"
"io"
)
const versionV1 = 1
const oneTransition = 1 << 7
const transitionNext = 1 << 6
const stateFinal = 1 << 6
const footerSizeV1 = 16
func init() {
registerEncoder(versionV1, func(w io.Writer) encoder {
return newEncoderV1(w)
})
}
type encoderV1 struct {
bw *writer
}
func newEncoderV1(w io.Writer) *encoderV1 {
return &encoderV1{
bw: newWriter(w),
}
}
func (e *encoderV1) reset(w io.Writer) {
e.bw.Reset(w)
}
func (e *encoderV1) start() error {
header := make([]byte, headerSize)
binary.LittleEndian.PutUint64(header, versionV1)
binary.LittleEndian.PutUint64(header[8:], uint64(0)) // type
n, err := e.bw.Write(header)
if err != nil {
return err
}
if n != headerSize {
return fmt.Errorf("short write of header %d/%d", n, headerSize)
}
return nil
}
func (e *encoderV1) encodeState(s *builderNode, lastAddr int) (int, error) {
if len(s.trans) == 0 && s.final && s.finalOutput == 0 {
return 0, nil
} else if len(s.trans) != 1 || s.final {
return e.encodeStateMany(s)
} else if !s.final && s.trans[0].out == 0 && s.trans[0].addr == lastAddr {
return e.encodeStateOneFinish(s, transitionNext)
}
return e.encodeStateOne(s)
}
func (e *encoderV1) encodeStateOne(s *builderNode) (int, error) {
start := uint64(e.bw.counter)
outPackSize := 0
if s.trans[0].out != 0 {
outPackSize = packedSize(s.trans[0].out)
err := e.bw.WritePackedUintIn(s.trans[0].out, outPackSize)
if err != nil {
return 0, err
}
}
delta := deltaAddr(start, uint64(s.trans[0].addr))
transPackSize := packedSize(delta)
err := e.bw.WritePackedUintIn(delta, transPackSize)
if err != nil {
return 0, err
}
packSize := encodePackSize(transPackSize, outPackSize)
err = e.bw.WriteByte(packSize)
if err != nil {
return 0, err
}
return e.encodeStateOneFinish(s, 0)
}
func (e *encoderV1) encodeStateOneFinish(s *builderNode, next byte) (int, error) {
enc := encodeCommon(s.trans[0].in)
// not a common input
if enc == 0 {
err := e.bw.WriteByte(s.trans[0].in)
if err != nil {
return 0, err
}
}
err := e.bw.WriteByte(oneTransition | next | enc)
if err != nil {
return 0, err
}
return e.bw.counter - 1, nil
}
func (e *encoderV1) encodeStateMany(s *builderNode) (int, error) {
start := uint64(e.bw.counter)
transPackSize := 0
outPackSize := packedSize(s.finalOutput)
anyOutputs := s.finalOutput != 0
for i := range s.trans {
delta := deltaAddr(start, uint64(s.trans[i].addr))
tsize := packedSize(delta)
if tsize > transPackSize {
transPackSize = tsize
}
osize := packedSize(s.trans[i].out)
if osize > outPackSize {
outPackSize = osize
}
anyOutputs = anyOutputs || s.trans[i].out != 0
}
if !anyOutputs {
outPackSize = 0
}
if anyOutputs {
// output final value
if s.final {
err := e.bw.WritePackedUintIn(s.finalOutput, outPackSize)
if err != nil {
return 0, err
}
}
// output transition values (in reverse)
for j := len(s.trans) - 1; j >= 0; j-- {
err := e.bw.WritePackedUintIn(s.trans[j].out, outPackSize)
if err != nil {
return 0, err
}
}
}
// output transition dests (in reverse)
for j := len(s.trans) - 1; j >= 0; j-- {
delta := deltaAddr(start, uint64(s.trans[j].addr))
err := e.bw.WritePackedUintIn(delta, transPackSize)
if err != nil {
return 0, err
}
}
// output transition keys (in reverse)
for j := len(s.trans) - 1; j >= 0; j-- {
err := e.bw.WriteByte(s.trans[j].in)
if err != nil {
return 0, err
}
}
packSize := encodePackSize(transPackSize, outPackSize)
err := e.bw.WriteByte(packSize)
if err != nil {
return 0, err
}
numTrans := encodeNumTrans(len(s.trans))
// if number of transitions wont fit in edge header byte
// write out separately
if numTrans == 0 {
if len(s.trans) == 256 {
// this wouldn't fit in single byte, but reuse value 1
// which would have always fit in the edge header instead
err = e.bw.WriteByte(1)
if err != nil {
return 0, err
}
} else {
err = e.bw.WriteByte(byte(len(s.trans)))
if err != nil {
return 0, err
}
}
}
// finally write edge header
if s.final {
numTrans |= stateFinal
}
err = e.bw.WriteByte(numTrans)
if err != nil {
return 0, err
}
return e.bw.counter - 1, nil
}
func (e *encoderV1) finish(count, rootAddr int) error {
footer := make([]byte, footerSizeV1)
binary.LittleEndian.PutUint64(footer, uint64(count)) // root addr
binary.LittleEndian.PutUint64(footer[8:], uint64(rootAddr)) // root addr
n, err := e.bw.Write(footer)
if err != nil {
return err
}
if n != footerSizeV1 {
return fmt.Errorf("short write of footer %d/%d", n, footerSizeV1)
}
err = e.bw.Flush()
if err != nil {
return err
}
return nil
}
================================================
FILE: encoder_v1_test.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum
import (
"bytes"
"reflect"
"testing"
)
// FIXME add tests for longjmp (wider delta address)
// FIXME add tests for wider values
// FIXME add tests for mixed value sizes in same edge (fixed size, but padded)
// FIXME add test for final state (must include final val even if 0)
func TestEncoderVersionError(t *testing.T) {
_, err := loadEncoder(629, nil)
if err == nil {
t.Errorf("expected error loading encoder version 629, got nil")
}
}
func TestEncoderStart(t *testing.T) {
var headerV1 = []byte{
1, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0,
}
var buf bytes.Buffer
e := newEncoderV1(&buf)
err := e.start()
if err != nil {
t.Fatal(err)
}
// manually flush
err = e.bw.Flush()
if err != nil {
t.Fatal(err)
}
got := buf.Bytes()
if !reflect.DeepEqual(got, headerV1) {
t.Errorf("expected header: %v, got %v", headerV1, got)
}
}
func TestEncoderStateOneNextWithCommonInput(t *testing.T) {
curr := &builderNode{
trans: []transition{
{
in: 'a',
addr: 27,
},
},
}
var buf bytes.Buffer
e := newEncoderV1(&buf)
// now encode the curr state
_, err := e.encodeState(curr, 27)
if err != nil {
t.Fatal(err)
}
// manually flush
err = e.bw.Flush()
if err != nil {
t.Fatal(err)
}
// now look at the bytes produced
var want = []byte{
oneTransition | transitionNext | encodeCommon('a'),
}
got := buf.Bytes()
if !reflect.DeepEqual(got, want) {
t.Errorf("expected bytes: %v, got %v", want, got)
}
}
func TestEncoderStateOneNextWithUncommonInput(t *testing.T) {
curr := &builderNode{
trans: []transition{
{
in: 0xff,
addr: 27,
},
},
}
var buf bytes.Buffer
e := newEncoderV1(&buf)
// now encode the curr state
_, err := e.encodeState(curr, 27)
if err != nil {
t.Fatal(err)
}
// manually flush
err = e.bw.Flush()
if err != nil {
t.Fatal(err)
}
// now look at the bytes produced
var want = []byte{
0xff,
oneTransition | transitionNext,
}
got := buf.Bytes()
if !reflect.DeepEqual(got, want) {
t.Errorf("expected bytes: %v, got %v", want, got)
}
}
func TestEncoderStateOneNotNextWithCommonInputNoValue(t *testing.T) {
curr := &builderNode{
trans: []transition{
{
in: 'a',
addr: 32,
},
},
}
var buf bytes.Buffer
e := newEncoderV1(&buf)
// pretend we're at a position in the file
e.bw.counter = 64
// now encode the curr state
_, err := e.encodeState(curr, 64)
if err != nil {
t.Fatal(err)
}
// manually flush
err = e.bw.Flush()
if err != nil {
t.Fatal(err)
}
// now look at the bytes produced
var want = []byte{
32, // delta address packed
1<<4 | 0, // pack sizes
oneTransition | encodeCommon('a'),
}
got := buf.Bytes()
if !reflect.DeepEqual(got, want) {
t.Errorf("expected bytes: %v, got %v", want, got)
}
}
func TestEncoderStateOneNotNextWithUncommonInputNoValue(t *testing.T) {
curr := &builderNode{
trans: []transition{
{
in: 0xff,
addr: 32,
},
},
}
var buf bytes.Buffer
e := newEncoderV1(&buf)
// pretend we're at a position in the file
e.bw.counter = 64
// now encode the curr state
_, err := e.encodeState(curr, 64)
if err != nil {
t.Fatal(err)
}
// manually flush
err = e.bw.Flush()
if err != nil {
t.Fatal(err)
}
// now look at the bytes produced
var want = []byte{
32, // delta address packed
1<<4 | 0, // pack sizes
0xff,
oneTransition,
}
got := buf.Bytes()
if !reflect.DeepEqual(got, want) {
t.Errorf("expected bytes: %v, got %v", want, got)
}
}
func TestEncoderStateOneNotNextWithCommonInputWithValue(t *testing.T) {
curr := &builderNode{
trans: []transition{
{
in: 'a',
addr: 32,
out: 27,
},
},
}
var buf bytes.Buffer
e := newEncoderV1(&buf)
// pretend we're at a position in the file
e.bw.counter = 64
// now encode the curr state
_, err := e.encodeState(curr, 64)
if err != nil {
t.Fatal(err)
}
// manually flush
err = e.bw.Flush()
if err != nil {
t.Fatal(err)
}
// now look at the bytes produced
var want = []byte{
27, // trans value
32, // delta address packed
1<<4 | 1, // pack sizes
oneTransition | encodeCommon('a'),
}
got := buf.Bytes()
if !reflect.DeepEqual(got, want) {
t.Errorf("expected bytes: %v, got %v", want, got)
}
}
func TestEncoderStateOneNotNextWithUncommonInputWithValue(t *testing.T) {
curr := &builderNode{
trans: []transition{
{
in: 0xff,
addr: 32,
out: 39,
},
},
}
var buf bytes.Buffer
e := newEncoderV1(&buf)
// pretend we're at a position in the file
e.bw.counter = 64
// now encode the curr state
_, err := e.encodeState(curr, 64)
if err != nil {
t.Fatal(err)
}
// manually flush
err = e.bw.Flush()
if err != nil {
t.Fatal(err)
}
// now look at the bytes produced
var want = []byte{
39, // trans val
32, // delta address packed
1<<4 | 1, // pack sizes
0xff,
oneTransition,
}
got := buf.Bytes()
if !reflect.DeepEqual(got, want) {
t.Errorf("expected bytes: %v, got %v", want, got)
}
}
func TestEncoderStateManyWithNoValues(t *testing.T) {
curr := &builderNode{
trans: []transition{
{
in: 'a',
addr: 32,
},
{
in: 'b',
addr: 45,
},
{
in: 'c',
addr: 52,
},
},
}
var buf bytes.Buffer
e := newEncoderV1(&buf)
// pretend we're at a position in the file
e.bw.counter = 64
// now encode the curr state
_, err := e.encodeState(curr, 64)
if err != nil {
t.Fatal(err)
}
// manually flush
err = e.bw.Flush()
if err != nil {
t.Fatal(err)
}
// now look at the bytes produced
var want = []byte{
12, // delta addresses packed
19,
32,
'c', // encoded keys reversed
'b',
'a',
1<<4 | 0, // pack sizes
encodeNumTrans(3),
}
got := buf.Bytes()
if !reflect.DeepEqual(got, want) {
t.Errorf("expected bytes: %v, got %v", want, got)
}
}
func TestEncoderStateManyWithValues(t *testing.T) {
curr := &builderNode{
trans: []transition{
{
in: 'a',
addr: 32,
out: 3,
},
{
in: 'b',
addr: 45,
out: 0,
},
{
in: 'c',
addr: 52,
out: 7,
},
},
}
var buf bytes.Buffer
e := newEncoderV1(&buf)
// pretend we're at a position in the file
e.bw.counter = 64
// now encode the curr state
_, err := e.encodeState(curr, 64)
if err != nil {
t.Fatal(err)
}
// manually flush
err = e.bw.Flush()
if err != nil {
t.Fatal(err)
}
// now look at the bytes produced
var want = []byte{
7, // values reversed
0,
3,
12, // delta addresses reversed
19,
32,
'c', // encoded keys reversed
'b',
'a',
1<<4 | 1, // pack sizes
encodeNumTrans(3),
}
got := buf.Bytes()
if !reflect.DeepEqual(got, want) {
t.Errorf("expected bytes: %v, got %v", want, got)
}
}
func TestEncoderStateMaxTransitions(t *testing.T) {
testEncoderStateNTransitions(t, 256)
}
func TestEncoderStateMoreTransitionsThanFitInHeader(t *testing.T) {
testEncoderStateNTransitions(t, 1<<6)
}
func testEncoderStateNTransitions(t *testing.T, n int) {
curr := &builderNode{
trans: make([]transition, n),
}
for i := 0; i < n; i++ {
curr.trans[i] = transition{
in: byte(i),
addr: 32,
}
}
var buf bytes.Buffer
e := newEncoderV1(&buf)
// pretend we're at a position in the file
e.bw.counter = 64
// now encode the curr state
_, err := e.encodeState(curr, 64)
if err != nil {
t.Fatal(err)
}
// manually flush
err = e.bw.Flush()
if err != nil {
t.Fatal(err)
}
// now look at the bytes produced
var want []byte
// append 256 delta addresses
for i := 0; i < n; i++ {
want = append(want, 32)
}
// append transition keys (reversed)
for i := n - 1; i >= 0; i-- {
want = append(want, byte(i))
}
// append pack sizes
want = append(want, 1<<4|0)
if n > 1<<6-1 {
// append separate byte of pack sizes
if n == 256 { // 256 is specially encoded as 1
want = append(want, 1)
} else {
want = append(want, byte(n))
}
}
// append header byte, which is all 0 in this case
want = append(want, 0)
got := buf.Bytes()
if !reflect.DeepEqual(got, want) {
t.Errorf("expected bytes: %v, got %v", want, got)
}
}
================================================
FILE: encoding.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum
import (
"encoding/binary"
"fmt"
"io"
)
const headerSize = 16
type encoderConstructor func(w io.Writer) encoder
type decoderConstructor func([]byte) decoder
var encoders = map[int]encoderConstructor{}
var decoders = map[int]decoderConstructor{}
type encoder interface {
start() error
encodeState(s *builderNode, addr int) (int, error)
finish(count, rootAddr int) error
reset(w io.Writer)
}
func loadEncoder(ver int, w io.Writer) (encoder, error) {
if cons, ok := encoders[ver]; ok {
return cons(w), nil
}
return nil, fmt.Errorf("no encoder for version %d registered", ver)
}
func registerEncoder(ver int, cons encoderConstructor) {
encoders[ver] = cons
}
type decoder interface {
getRoot() int
getLen() int
stateAt(addr int, prealloc fstState) (fstState, error)
}
func loadDecoder(ver int, data []byte) (decoder, error) {
if cons, ok := decoders[ver]; ok {
return cons(data), nil
}
return nil, fmt.Errorf("no decoder for version %d registered", ver)
}
func registerDecoder(ver int, cons decoderConstructor) {
decoders[ver] = cons
}
func decodeHeader(header []byte) (ver int, typ int, err error) {
if len(header) < headerSize {
err = fmt.Errorf("invalid header < 16 bytes")
return
}
ver = int(binary.LittleEndian.Uint64(header[0:8]))
typ = int(binary.LittleEndian.Uint64(header[8:16]))
return
}
// fstState represents a state inside the FTS runtime
// It is the main contract between the FST impl and the decoder
// The FST impl should work only with this interface, while only the decoder
// impl knows the physical representation.
type fstState interface {
Address() int
Final() bool
FinalOutput() uint64
NumTransitions() int
TransitionFor(b byte) (int, int, uint64)
TransitionAt(i int) byte
}
================================================
FILE: example_test.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum_test
import (
"bytes"
"fmt"
"log"
"github.com/couchbase/vellum"
)
func Example() {
var buf bytes.Buffer
builder, err := vellum.New(&buf, nil)
if err != nil {
log.Fatal(err)
}
err = builder.Insert([]byte("cat"), 1)
if err != nil {
log.Fatal(err)
}
err = builder.Insert([]byte("dog"), 2)
if err != nil {
log.Fatal(err)
}
err = builder.Insert([]byte("fish"), 3)
if err != nil {
log.Fatal(err)
}
err = builder.Close()
if err != nil {
log.Fatal(err)
}
fst, err := vellum.Load(buf.Bytes())
if err != nil {
log.Fatal(err)
}
val, exists, err := fst.Get([]byte("cat"))
if err != nil {
log.Fatal(err)
}
if exists {
fmt.Println(val)
}
val, exists, err = fst.Get([]byte("dog"))
if err != nil {
log.Fatal(err)
}
if exists {
fmt.Println(val)
}
val, exists, err = fst.Get([]byte("fish"))
if err != nil {
log.Fatal(err)
}
if exists {
fmt.Println(val)
}
// Output: 1
// 2
// 3
}
================================================
FILE: fst.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum
import (
"io"
"github.com/willf/bitset"
)
// FST is an in-memory representation of a finite state transducer,
// capable of returning the uint64 value associated with
// each []byte key stored, as well as enumerating all of the keys
// in order.
type FST struct {
f io.Closer
ver int
len int
typ int
data []byte
decoder decoder
}
func new(data []byte, f io.Closer) (rv *FST, err error) {
rv = &FST{
data: data,
f: f,
}
rv.ver, rv.typ, err = decodeHeader(data)
if err != nil {
return nil, err
}
rv.decoder, err = loadDecoder(rv.ver, rv.data)
if err != nil {
return nil, err
}
rv.len = rv.decoder.getLen()
return rv, nil
}
// Contains returns true if this FST contains the specified key.
func (f *FST) Contains(val []byte) (bool, error) {
_, exists, err := f.Get(val)
return exists, err
}
// Get returns the value associated with the key. NOTE: a value of zero
// does not imply the key does not exist, you must consult the second
// return value as well.
func (f *FST) Get(input []byte) (uint64, bool, error) {
return f.get(input, nil)
}
func (f *FST) get(input []byte, prealloc fstState) (uint64, bool, error) {
var total uint64
curr := f.decoder.getRoot()
state, err := f.decoder.stateAt(curr, prealloc)
if err != nil {
return 0, false, err
}
for _, c := range input {
_, curr, output := state.TransitionFor(c)
if curr == noneAddr {
return 0, false, nil
}
state, err = f.decoder.stateAt(curr, state)
if err != nil {
return 0, false, err
}
total += output
}
if state.Final() {
total += state.FinalOutput()
return total, true, nil
}
return 0, false, nil
}
// Version returns the encoding version used by this FST instance.
func (f *FST) Version() int {
return f.ver
}
// Len returns the number of entries in this FST instance.
func (f *FST) Len() int {
return f.len
}
// Type returns the type of this FST instance.
func (f *FST) Type() int {
return f.typ
}
// Close will unmap any mmap'd data (if managed by vellum) and it will close
// the backing file (if managed by vellum). You MUST call Close() for any
// FST instance that is created.
func (f *FST) Close() error {
if f.f != nil {
err := f.f.Close()
if err != nil {
return err
}
}
f.data = nil
f.decoder = nil
return nil
}
// Start returns the start state of this Automaton
func (f *FST) Start() int {
return f.decoder.getRoot()
}
// IsMatch returns if this state is a matching state in this Automaton
func (f *FST) IsMatch(addr int) bool {
match, _ := f.IsMatchWithVal(addr)
return match
}
// CanMatch returns if this state can ever transition to a matching state
// in this Automaton
func (f *FST) CanMatch(addr int) bool {
if addr == noneAddr {
return false
}
return true
}
// WillAlwaysMatch returns if from this state the Automaton will always
// be in a matching state
func (f *FST) WillAlwaysMatch(int) bool {
return false
}
// Accept returns the next state for this Automaton on input of byte b
func (f *FST) Accept(addr int, b byte) int {
next, _ := f.AcceptWithVal(addr, b)
return next
}
// IsMatchWithVal returns if this state is a matching state in this Automaton
// and also returns the final output value for this state
func (f *FST) IsMatchWithVal(addr int) (bool, uint64) {
s, err := f.decoder.stateAt(addr, nil)
if err != nil {
return false, 0
}
return s.Final(), s.FinalOutput()
}
// AcceptWithVal returns the next state for this Automaton on input of byte b
// and also returns the output value for the transition
func (f *FST) AcceptWithVal(addr int, b byte) (int, uint64) {
s, err := f.decoder.stateAt(addr, nil)
if err != nil {
return noneAddr, 0
}
_, next, output := s.TransitionFor(b)
return next, output
}
// Iterator returns a new Iterator capable of enumerating the key/value pairs
// between the provided startKeyInclusive and endKeyExclusive.
func (f *FST) Iterator(startKeyInclusive, endKeyExclusive []byte) (*FSTIterator, error) {
return newIterator(f, startKeyInclusive, endKeyExclusive, nil)
}
// Search returns a new Iterator capable of enumerating the key/value pairs
// between the provided startKeyInclusive and endKeyExclusive that also
// satisfy the provided automaton.
func (f *FST) Search(aut Automaton, startKeyInclusive, endKeyExclusive []byte) (*FSTIterator, error) {
return newIterator(f, startKeyInclusive, endKeyExclusive, aut)
}
// Debug is only intended for debug purposes, it simply asks the underlying
// decoder visit each state, and pass it to the provided callback.
func (f *FST) Debug(callback func(int, interface{}) error) error {
addr := f.decoder.getRoot()
set := bitset.New(uint(addr))
stack := addrStack{addr}
stateNumber := 0
stack, addr = stack[:len(stack)-1], stack[len(stack)-1]
for addr != noneAddr {
if set.Test(uint(addr)) {
stack, addr = stack.Pop()
continue
}
set.Set(uint(addr))
state, err := f.decoder.stateAt(addr, nil)
if err != nil {
return err
}
err = callback(stateNumber, state)
if err != nil {
return err
}
for i := 0; i < state.NumTransitions(); i++ {
tchar := state.TransitionAt(i)
_, dest, _ := state.TransitionFor(tchar)
stack = append(stack, dest)
}
stateNumber++
stack, addr = stack.Pop()
}
return nil
}
type addrStack []int
func (a addrStack) Pop() (addrStack, int) {
l := len(a)
if l < 1 {
return a, noneAddr
}
return a[:l-1], a[l-1]
}
// Reader() returns a Reader instance that a single thread may use to
// retrieve data from the FST
func (f *FST) Reader() (*Reader, error) {
return &Reader{f: f}, nil
}
func (f *FST) GetMinKey() ([]byte, error) {
var rv []byte
curr := f.decoder.getRoot()
state, err := f.decoder.stateAt(curr, nil)
if err != nil {
return nil, err
}
for !state.Final() {
nextTrans := state.TransitionAt(0)
_, curr, _ = state.TransitionFor(nextTrans)
state, err = f.decoder.stateAt(curr, state)
if err != nil {
return nil, err
}
rv = append(rv, nextTrans)
}
return rv, nil
}
func (f *FST) GetMaxKey() ([]byte, error) {
var rv []byte
curr := f.decoder.getRoot()
state, err := f.decoder.stateAt(curr, nil)
if err != nil {
return nil, err
}
for state.NumTransitions() > 0 {
nextTrans := state.TransitionAt(state.NumTransitions() - 1)
_, curr, _ = state.TransitionFor(nextTrans)
state, err = f.decoder.stateAt(curr, state)
if err != nil {
return nil, err
}
rv = append(rv, nextTrans)
}
return rv, nil
}
// A Reader is meant for a single threaded use
type Reader struct {
f *FST
prealloc fstStateV1
}
func (r *Reader) Get(input []byte) (uint64, bool, error) {
return r.f.get(input, &r.prealloc)
}
================================================
FILE: fst_iterator.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum
import (
"bytes"
)
// Iterator represents a means of visiting key/value pairs in order.
type Iterator interface {
// Current() returns the key/value pair currently pointed to.
// The []byte of the key is ONLY guaranteed to be valid until
// another call to Next/Seek/Close. If you need it beyond that
// point you MUST make a copy.
Current() ([]byte, uint64)
// Next() advances the iterator to the next key/value pair.
// If no more key/value pairs exist, ErrIteratorDone is returned.
Next() error
// Seek() advances the iterator the specified key, or the next key
// if it does not exist.
// If no keys exist after that point, ErrIteratorDone is returned.
Seek(key []byte) error
// Reset resets the Iterator' internal state to allow for iterator
// reuse (e.g. pooling).
Reset(f *FST, startKeyInclusive, endKeyExclusive []byte, aut Automaton) error
// Close() frees any resources held by this iterator.
Close() error
}
// FSTIterator is a structure for iterating key/value pairs in this FST in
// lexicographic order. Iterators should be constructed with the FSTIterator
// method on the parent FST structure.
type FSTIterator struct {
f *FST
aut Automaton
startKeyInclusive []byte
endKeyExclusive []byte
statesStack []fstState
keysStack []byte
keysPosStack []int
valsStack []uint64
autStatesStack []int
nextStart []byte
}
func newIterator(f *FST, startKeyInclusive, endKeyExclusive []byte,
aut Automaton) (*FSTIterator, error) {
rv := &FSTIterator{}
err := rv.Reset(f, startKeyInclusive, endKeyExclusive, aut)
if err != nil {
return nil, err
}
return rv, nil
}
// Reset resets the Iterator' internal state to allow for iterator
// reuse (e.g. pooling).
func (i *FSTIterator) Reset(f *FST,
startKeyInclusive, endKeyExclusive []byte, aut Automaton) error {
if aut == nil {
aut = alwaysMatchAutomaton
}
i.f = f
i.startKeyInclusive = startKeyInclusive
i.endKeyExclusive = endKeyExclusive
i.aut = aut
return i.pointTo(startKeyInclusive)
}
// pointTo attempts to point us to the specified location
func (i *FSTIterator) pointTo(key []byte) error {
// tried to seek before start
if bytes.Compare(key, i.startKeyInclusive) < 0 {
key = i.startKeyInclusive
}
// tried to see past end
if i.endKeyExclusive != nil &&
bytes.Compare(key, i.endKeyExclusive) > 0 {
key = i.endKeyExclusive
}
// reset any state, pointTo always starts over
i.statesStack = i.statesStack[:0]
i.keysStack = i.keysStack[:0]
i.keysPosStack = i.keysPosStack[:0]
i.valsStack = i.valsStack[:0]
i.autStatesStack = i.autStatesStack[:0]
root, err := i.f.decoder.stateAt(i.f.decoder.getRoot(), nil)
if err != nil {
return err
}
autStart := i.aut.Start()
maxQ := -1
// root is always part of the path
i.statesStack = append(i.statesStack, root)
i.autStatesStack = append(i.autStatesStack, autStart)
for j := 0; j < len(key); j++ {
keyJ := key[j]
curr := i.statesStack[len(i.statesStack)-1]
autCurr := i.autStatesStack[len(i.autStatesStack)-1]
pos, nextAddr, nextVal := curr.TransitionFor(keyJ)
if nextAddr == noneAddr {
// needed transition doesn't exist
// find last trans before the one we needed
for q := curr.NumTransitions() - 1; q >= 0; q-- {
if curr.TransitionAt(q) < keyJ {
maxQ = q
break
}
}
break
}
autNext := i.aut.Accept(autCurr, keyJ)
next, err := i.f.decoder.stateAt(nextAddr, nil)
if err != nil {
return err
}
i.statesStack = append(i.statesStack, next)
i.keysStack = append(i.keysStack, keyJ)
i.keysPosStack = append(i.keysPosStack, pos)
i.valsStack = append(i.valsStack, nextVal)
i.autStatesStack = append(i.autStatesStack, autNext)
continue
}
if !i.statesStack[len(i.statesStack)-1].Final() ||
!i.aut.IsMatch(i.autStatesStack[len(i.autStatesStack)-1]) ||
bytes.Compare(i.keysStack, key) < 0 {
return i.next(maxQ)
}
return nil
}
// Current returns the key and value currently pointed to by the iterator.
// If the iterator is not pointing at a valid value (because Iterator/Next/Seek)
// returned an error previously, it may return nil,0.
func (i *FSTIterator) Current() ([]byte, uint64) {
curr := i.statesStack[len(i.statesStack)-1]
if curr.Final() {
var total uint64
for _, v := range i.valsStack {
total += v
}
total += curr.FinalOutput()
return i.keysStack, total
}
return nil, 0
}
// Next advances this iterator to the next key/value pair. If there is none
// or the advancement goes beyond the configured endKeyExclusive, then
// ErrIteratorDone is returned.
func (i *FSTIterator) Next() error {
return i.next(-1)
}
func (i *FSTIterator) next(lastOffset int) error {
// remember where we started with keysStack in this next() call
i.nextStart = append(i.nextStart[:0], i.keysStack...)
nextOffset := lastOffset + 1
allowCompare := false
OUTER:
for true {
curr := i.statesStack[len(i.statesStack)-1]
autCurr := i.autStatesStack[len(i.autStatesStack)-1]
if curr.Final() && i.aut.IsMatch(autCurr) && allowCompare {
// check to see if new keystack might have gone too far
if i.endKeyExclusive != nil &&
bytes.Compare(i.keysStack, i.endKeyExclusive) >= 0 {
return ErrIteratorDone
}
cmp := bytes.Compare(i.keysStack, i.nextStart)
if cmp > 0 {
// in final state greater than start key
return nil
}
}
numTrans := curr.NumTransitions()
INNER:
for nextOffset < numTrans {
t := curr.TransitionAt(nextOffset)
autNext := i.aut.Accept(autCurr, t)
if !i.aut.CanMatch(autNext) {
// TODO: potential optimization to skip nextOffset
// forwards more directly to something that the
// automaton likes rather than a linear scan?
nextOffset += 1
continue INNER
}
pos, nextAddr, v := curr.TransitionFor(t)
// the next slot in the statesStack might have an
// fstState instance that we can reuse
var nextPrealloc fstState
if len(i.statesStack) < cap(i.statesStack) {
nextPrealloc = i.statesStack[0:cap(i.statesStack)][len(i.statesStack)]
}
// push onto stack
next, err := i.f.decoder.stateAt(nextAddr, nextPrealloc)
if err != nil {
return err
}
i.statesStack = append(i.statesStack, next)
i.keysStack = append(i.keysStack, t)
i.keysPosStack = append(i.keysPosStack, pos)
i.valsStack = append(i.valsStack, v)
i.autStatesStack = append(i.autStatesStack, autNext)
nextOffset = 0
allowCompare = true
continue OUTER
}
// no more transitions, so need to backtrack and stack pop
if len(i.statesStack) <= 1 {
// stack len is 1 (root), can't go back further, we're done
break
}
// if the top of the stack represents a linear chain of states
// (i.e., a suffix of nodes linked by single transitions),
// then optimize by popping the suffix in one shot without
// going back all the way to the OUTER loop
var popNum int
for j := len(i.statesStack) - 1; j > 0; j-- {
if j == 1 || i.statesStack[j].NumTransitions() != 1 {
popNum = len(i.statesStack) - 1 - j
break
}
}
if popNum < 1 { // always pop at least 1 entry from the stacks
popNum = 1
}
nextOffset = i.keysPosStack[len(i.keysPosStack)-popNum] + 1
allowCompare = false
i.statesStack = i.statesStack[:len(i.statesStack)-popNum]
i.keysStack = i.keysStack[:len(i.keysStack)-popNum]
i.keysPosStack = i.keysPosStack[:len(i.keysPosStack)-popNum]
i.valsStack = i.valsStack[:len(i.valsStack)-popNum]
i.autStatesStack = i.autStatesStack[:len(i.autStatesStack)-popNum]
}
return ErrIteratorDone
}
// Seek advances this iterator to the specified key/value pair. If this key
// is not in the FST, Current() will return the next largest key. If this
// seek operation would go past the last key, or outside the configured
// startKeyInclusive/endKeyExclusive then ErrIteratorDone is returned.
func (i *FSTIterator) Seek(key []byte) error {
return i.pointTo(key)
}
// Close will free any resources held by this iterator.
func (i *FSTIterator) Close() error {
// at the moment we don't do anything,
// but wanted this for API completeness
return nil
}
================================================
FILE: fst_iterator_test.go
================================================
// Copyright (c) 2017 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package vellum
import (
"bytes"
"reflect"
"testing"
"github.com/couchbase/vellum/levenshtein"
"github.com/couchbase/vellum/regexp"
)
func TestIterator(t *testing.T) {
var buf bytes.Buffer
b, err := New(&buf, nil)
if err != nil {
t.Fatalf("error creating builder: %v", err)
}
err = insertStringMap(b, smallSample)
if err != nil {
t.Fatalf("error building: %v", err)
}
err = b.Close()
if err != nil {
t.Fatalf("error closing: %v", err)
}
fst, err := Load(buf.Bytes())
if err != nil {
t.Fatalf("error loading set: %v", err)
}
got := map[string]uint64{}
itr, err := fst.Iterator(nil, nil)
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(smallSample, got) {
t.Errorf("expected %v, got: %v", smallSample, got)
}
}
func TestIteratorReset(t *testing.T) {
var buf bytes.Buffer
b, err := New(&buf, nil)
if err != nil {
t.Fatalf("error creating builder: %v", err)
}
err = insertStringMap(b, smallSample)
if err != nil {
t.Fatalf("error building: %v", err)
}
err = b.Close()
if err != nil {
t.Fatalf("error closing: %v", err)
}
fst, err := Load(buf.Bytes())
if err != nil {
t.Fatalf("error loading set: %v", err)
}
itr, err := fst.Iterator(nil, nil)
if err != nil {
t.Fatalf("error creating an iterator: %v", err)
}
buf.Reset()
b, err = New(&buf, nil)
if err != nil {
t.Fatalf("error creating builder: %v", err)
}
smallSample2 := map[string]uint64{
"bold": 25,
"last": 1,
"next": 500,
"tank": 0,
}
err = insertStringMap(b, smallSample2)
if err != nil {
t.Fatalf("error building: %v", err)
}
err = b.Close()
if err != nil {
t.Fatalf("error closing: %v", err)
}
fst, err = Load(buf.Bytes())
if err != nil {
t.Fatalf("error loading set: %v", err)
}
got := map[string]uint64{}
err = itr.Reset(fst, nil, nil, nil)
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(smallSample2, got) {
t.Errorf("expected %v, got: %v", smallSample2, got)
}
}
func TestIteratorStartKey(t *testing.T) {
var buf bytes.Buffer
b, err := New(&buf, nil)
if err != nil {
t.Fatalf("error creating builder: %v", err)
}
err = insertStringMap(b, smallSample)
if err != nil {
t.Fatalf("error building: %v", err)
}
err = b.Close()
if err != nil {
t.Fatalf("error closing: %v", err)
}
fst, err := Load(buf.Bytes())
if err != nil {
t.Fatalf("error loading set: %v", err)
}
// with start key < "mon", we should still get it
got := map[string]uint64{}
itr, err := fst.Iterator([]byte("a"), nil)
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(smallSample, got) {
t.Errorf("expected %v, got: %v", smallSample, got)
}
// with start key = "mon", we should still get it
got = map[string]uint64{}
itr, err = fst.Iterator([]byte("mon"), nil)
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(smallSample, got) {
t.Errorf("expected %v, got: %v", smallSample, got)
}
// with start key > "mon", we don't expect to get it
expect := map[string]uint64{
"tues": smallSample["tues"],
"thurs": smallSample["thurs"],
"tye": smallSample["tye"],
}
got = map[string]uint64{}
itr, err = fst.Iterator([]byte("mona"), nil)
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(expect, got) {
t.Errorf("expected %v, got: %v", expect, got)
}
// with start key > "mon", we don't expect to get it
expect = map[string]uint64{
"tues": smallSample["tues"],
"thurs": smallSample["thurs"],
"tye": smallSample["tye"],
}
got = map[string]uint64{}
itr, err = fst.Iterator([]byte("my"), nil)
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(expect, got) {
t.Errorf("expected %v, got: %v", expect, got)
}
}
func TestIteratorEndKey(t *testing.T) {
var buf bytes.Buffer
b, err := New(&buf, nil)
if err != nil {
t.Fatalf("error creating builder: %v", err)
}
err = insertStringMap(b, smallSample)
if err != nil {
t.Fatalf("error building: %v", err)
}
err = b.Close()
if err != nil {
t.Fatalf("error closing: %v", err)
}
fst, err := Load(buf.Bytes())
if err != nil {
t.Fatalf("error loading set: %v", err)
}
// with end key > "tye", we should still get it
got := map[string]uint64{}
itr, err := fst.Iterator(nil, []byte("zeus"))
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(smallSample, got) {
t.Errorf("expected %v, got: %v", smallSample, got)
}
// with end key = "tye", we should NOT get it (end key exclusive)
expect := map[string]uint64{
"mon": smallSample["mon"],
"tues": smallSample["tues"],
"thurs": smallSample["thurs"],
}
got = map[string]uint64{}
itr, err = fst.Iterator(nil, []byte("tye"))
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(expect, got) {
t.Errorf("expected %v, got: %v", expect, got)
}
// with start key < "tye", we don't expect to get it
got = map[string]uint64{}
itr, err = fst.Iterator(nil, []byte("tv"))
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(expect, got) {
t.Errorf("expected %v, got: %v", expect, got)
}
}
func TestIteratorSeek(t *testing.T) {
var buf bytes.Buffer
b, err := New(&buf, nil)
if err != nil {
t.Fatalf("error creating builder: %v", err)
}
err = insertStringMap(b, smallSample)
if err != nil {
t.Fatalf("error building: %v", err)
}
err = b.Close()
if err != nil {
t.Fatalf("error closing: %v", err)
}
fst, err := Load(buf.Bytes())
if err != nil {
t.Fatalf("error loading set: %v", err)
}
// seek past thurs (exactly to tues)
expect := map[string]uint64{
"mon": smallSample["mon"],
"tues": smallSample["tues"],
"tye": smallSample["tye"],
}
got := map[string]uint64{}
itr, err := fst.Iterator(nil, nil)
for err == nil {
key, val := itr.Current()
got[string(key)] = val
if string(key) == "mon" {
err = itr.Seek([]byte("tue"))
} else {
err = itr.Next()
}
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(expect, got) {
t.Errorf("expected %v, got: %v", expect, got)
}
// similar but seek to something after thurs before tues
got = map[string]uint64{}
itr, err = fst.Iterator(nil, nil)
for err == nil {
key, val := itr.Current()
got[string(key)] = val
if string(key) == "mon" {
err = itr.Seek([]byte("thv"))
} else {
err = itr.Next()
}
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(expect, got) {
t.Errorf("expected %v, got: %v", expect, got)
}
// similar but seek to thurs+suffix
got = map[string]uint64{}
itr, err = fst.Iterator(nil, nil)
for err == nil {
key, val := itr.Current()
got[string(key)] = val
if string(key) == "mon" {
err = itr.Seek([]byte("thursday"))
} else {
err = itr.Next()
}
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(expect, got) {
t.Errorf("expected %v, got: %v", expect, got)
}
// seek past last key (still inside iterator boundaries)
expect = map[string]uint64{
"mon": smallSample["mon"],
}
got = map[string]uint64{}
itr, err = fst.Iterator(nil, nil)
for err == nil {
key, val := itr.Current()
got[string(key)] = val
if string(key) == "mon" {
err = itr.Seek([]byte("zzz"))
} else {
err = itr.Next()
}
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(expect, got) {
t.Errorf("expected %v, got: %v", expect, got)
}
}
func TestIteratorSeekOutsideBoundaries(t *testing.T) {
var buf bytes.Buffer
b, err := New(&buf, nil)
if err != nil {
t.Fatalf("error creating builder: %v", err)
}
err = insertStringMap(b, smallSample)
if err != nil {
t.Fatalf("error building: %v", err)
}
err = b.Close()
if err != nil {
t.Fatalf("error closing: %v", err)
}
fst, err := Load(buf.Bytes())
if err != nil {
t.Fatalf("error loading set: %v", err)
}
// first test with boundaries should just see thurs/tues
expect := map[string]uint64{
"thurs": smallSample["thurs"],
"tues": smallSample["tues"],
}
got := map[string]uint64{}
itr, err := fst.Iterator([]byte("th"), []byte("tuesd"))
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(expect, got) {
t.Errorf("expected %v, got: %v", expect, got)
}
// this time try to seek before the start,
// still shouldn't see mon
got = map[string]uint64{}
itr, err = fst.Iterator([]byte("th"), []byte("tuesd"))
if err != nil {
t.Fatalf("error before seeking: %v", err)
}
err = itr.Seek([]byte("cat"))
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(expect, got) {
t.Errorf("expected %v, got: %v", expect, got)
}
// this time try to seek past the end
// should see nothing
itr, err = fst.Iterator([]byte("th"), []byte("tuesd"))
if err != nil {
t.Fatalf("error before seeking: %v", err)
}
err = itr.Seek([]byte("ty"))
if err != ErrIteratorDone {
t.Fatalf("expected ErrIteratorDone, got %v", err)
}
}
var key []byte
var val uint64
func BenchmarkFSTIteratorAllInMem(b *testing.B) {
// first build the FST once
dataset := thousandTestWords
randomThousandVals := randomValues(dataset)
var buf bytes.Buffer
builder, err := New(&buf, nil)
if err != nil {
b.Fatalf("error creating builder: %v", err)
}
err = insertStrings(builder, dataset, randomThousandVals)
if err != nil {
b.Fatalf("error inserting thousand words: %v", err)
}
err = builder.Close()
if err != nil {
b.Fatalf("error closing builder: %v", err)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
fst, err := Load(buf.Bytes())
if err != nil {
b.Fatalf("error loading FST: %v", err)
}
itr, err := fst.Iterator(nil, nil)
for err == nil {
key, val = itr.Current()
err = itr.Next()
}
if err != ErrIteratorDone {
b.Fatalf("iterator error: %v", err)
}
err = fst.Close()
if err != nil {
b.Fatalf("error closing FST: %v", err)
}
}
}
func TestFuzzySearch(t *testing.T) {
var buf bytes.Buffer
b, err := New(&buf, nil)
if err != nil {
t.Fatalf("error creating builder: %v", err)
}
err = insertStringMap(b, smallSample)
if err != nil {
t.Fatalf("error building: %v", err)
}
err = b.Close()
if err != nil {
t.Fatalf("error closing: %v", err)
}
fst, err := Load(buf.Bytes())
if err != nil {
t.Fatalf("error loading set: %v", err)
}
lb, err := levenshtein.NewLevenshteinAutomatonBuilder(uint8(1), false)
if err != nil {
t.Fatalf("error loading set: %v", err)
}
fuzzy, err := lb.BuildDfa("tue", 1)
if err != nil {
t.Fatalf("error building levenshtein automaton: %v", err)
}
want := map[string]uint64{
"tues": 3,
"tye": 99,
}
got := map[string]uint64{}
itr, err := fst.Search(fuzzy, nil, nil)
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(want, got) {
t.Errorf("expected %v, got: %v", want, got)
}
}
func TestRegexpSearch(t *testing.T) {
var buf bytes.Buffer
b, err := New(&buf, nil)
if err != nil {
t.Fatalf("error creating builder: %v", err)
}
err = insertStringMap(b, smallSample)
if err != nil {
t.Fatalf("error building: %v", err)
}
err = b.Close()
if err != nil {
t.Fatalf("error closing: %v", err)
}
fst, err := Load(buf.Bytes())
if err != nil {
t.Fatalf("error loading set: %v", err)
}
r, err := regexp.New(`t.*s`)
if err != nil {
t.Fatalf("error building regexp automaton: %v", err)
}
want := map[string]uint64{
"thurs": 5,
"tues": 3,
}
got := map[string]uint64{}
itr, err := fst.Search(r, nil, nil)
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(want, got) {
t.Errorf("expected %v, got: %v", want, got)
}
got = map[string]uint64{}
itr, err = fst.Search(r, []byte("t"), nil)
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(want, got) {
t.Errorf("with start key t, expected %v, got: %v", want, got)
}
got = map[string]uint64{}
itr, err = fst.Search(r, nil, []byte("u"))
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(want, got) {
t.Errorf("with end key u, expected %v, got: %v", want, got)
}
got = map[string]uint64{}
itr, err = fst.Search(r, []byte("t"), []byte("u"))
for err == nil {
key, val := itr.Current()
got[string(key)] = val
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
if !reflect.DeepEqual(want, got) {
t.Errorf("with start key t, end key u, expected %v, got: %v", want, got)
}
}
func TestIssue32(t *testing.T) {
var buf bytes.Buffer
b, err := New(&buf, nil)
if err != nil {
t.Fatalf("error creating builder: %v", err)
}
err = b.Insert(bytes.Repeat([]byte{'a'}, 1000000), 0)
if err != nil {
t.Fatalf("error inserting large key: %v", err)
}
err = b.Close()
if err != nil {
t.Fatalf("error closing: %v", err)
}
fst, err := Load(buf.Bytes())
if err != nil {
t.Fatalf("error loading set: %v", err)
}
itr, err := fst.Iterator(nil, nil)
for err == nil {
err = itr.Next()
}
if err != ErrIteratorDone {
t.Errorf("iterator error: %v", err)
}
}
================================================
FILE: go.mod
================================================
module github.com/couchbase/vellum
go 1.12
require (
github.com/blevesearch/mmap-go v1.0.2
github.com/spf13/cobra v0.0.5
github.com/willf/bitset v1.1.10
golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a // indirect
)
================================================
FILE: go.sum
================================================
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
github.com/blevesearch/mmap-go v1.0.2 h1:JtMHb+FgQCTTYIhtMvimw15dJwu1Y5lrZDMOFXVWPk0=
github.com/blevesearch/mmap-go v1.0.2/go.mod h1:ol2qBqYaOUsGdm7aRMRrYGgPvnwLe6Y+7LMvAB5IbSA=
github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
github.com/spf13/cobra v0.0.5 h1:f0B+LkLX6DtmRH1isoNA9VTtNUK9K8xYd28JNNfOv/s=
github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
github.com/willf/bitset v1.1.10 h1:NotGKqX0KwQ72NUzqrjZq5ipPNDQex9lo3WpaS8L2sc=
github.com/willf/bitset v1.1.10/go.mod h1:RjeCKbqT1RxIR/KWY6phxZiaY1IyutSBfGjNPySAYV4=
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181221143128-b4a75ba826a6/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a h1:aYOabOQFp6Vj6W1F80affTUvO9UxmJRx8K0gsfABByQ=
golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
================================================
FILE: levenshtein/LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: levenshtein/README.md
================================================
# levenshtein
levenshtein automaton
This package makes it fast and simple to build a finite determinic automaton that computes the levenshtein distance from a given string.
# Sample usage:
```
// build a re-usable builder
lb := NewLevenshteinAutomatonBuilder(2, false)
origTerm := "couchbasefts"
dfa := lb.BuildDfa("couchbases", 2)
ed := dfa.eval([]byte(origTerm))
if ed.distance() != 2 {
log.Errorf("expected distance 2, actual: %d", ed.distance())
}
```
This implementation is inspired by [blog post](https://fulmicoton.com/posts/levenshtein/) and is intended to be
a port of original rust implementation: https://github.com/tantivy-search/levenshtein-automata
Micro Benchmark Results against the current vellum/levenshtein is as below.
```
BenchmarkNewEditDistance1-8 30000 52684 ns/op 89985 B/op 295 allocs/op
BenchmarkOlderEditDistance1-8 10000 132931 ns/op 588892 B/op 363 allocs/op
BenchmarkNewEditDistance2-8 10000 199127 ns/op 377532 B/op 1019 allocs/op
BenchmarkOlderEditDistance2-8 2000 988109 ns/op 4236609 B/op 1898 allocs/op
```
================================================
FILE: levenshtein/alphabet.go
================================================
// Copyright (c) 2018 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package levenshtein
import (
"fmt"
"sort"
"unicode/utf8"
)
type FullCharacteristicVector []uint32
func (fcv FullCharacteristicVector) shiftAndMask(offset, mask uint32) uint32 {
bucketID := offset / 32
align := offset - bucketID*32
if align == 0 {
return fcv[bucketID] & mask
}
left := fcv[bucketID] >> align
right := fcv[bucketID+1] << (32 - align)
return (left | right) & mask
}
type tuple struct {
char rune
fcv FullCharacteristicVector
}
type sortRunes []rune
func (s sortRunes) Less(i, j int) bool {
return s[i] < s[j]
}
func (s sortRunes) Swap(i, j int) {
s[i], s[j] = s[j], s[i]
}
func (s sortRunes) Len() int {
return len(s)
}
func sortRune(r []rune) []rune {
sort.Sort(sortRunes(r))
return r
}
type Alphabet struct {
charset []tuple
index uint32
}
func (a *Alphabet) resetNext() {
a.index = 0
}
func (a *Alphabet) next() (rune, FullCharacteristicVector, error) {
if int(a.index) >= len(a.charset) {
return 0, nil, fmt.Errorf("eof")
}
rv := a.charset[a.index]
a.index++
return rv.char, rv.fcv, nil
}
func dedupe(in string) string {
lookUp := make(map[rune]struct{}, len(in))
var rv string
for len(in) > 0 {
r, size := utf8.DecodeRuneInString(in)
in = in[size:]
if _, ok := lookUp[r]; !ok {
rv += string(r)
lookUp[r] = struct{}{}
}
}
return rv
}
func queryChars(qChars string) Alphabet {
chars := dedupe(qChars)
inChars := sortRune([]rune(chars))
charsets := make([]tuple, 0, len(inChars))
for _, c := range inChars {
tempChars := qChars
var bits []uint32
for len(tempChars) > 0 {
var chunk string
if len(tempChars) > 32 {
chunk = tempChars[0:32]
tempChars = tempChars[32:]
} else {
chunk = tempChars
tempChars = tempChars[:0]
}
chunkBits := uint32(0)
bit := uint32(1)
for _, chr := range chunk {
if chr == c {
chunkBits |= bit
}
bit <<= 1
}
bits = append(bits, chunkBits)
}
bits = append(bits, 0)
charsets = append(charsets, tuple{char: c, fcv: FullCharacteristicVector(bits)})
}
return Alphabet{charset: charsets}
}
================================================
FILE: levenshtein/alphabet_test.go
================================================
// Copyright (c) 2018 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package levenshtein
import "testing"
func TestAlphabet(t *testing.T) {
chars := "happy"
alphabet := queryChars(chars)
c, chi, _ := alphabet.next()
if c != 'a' {
t.Errorf("expecting 'a', got: %v", c)
}
if chi[0] != 2 {
t.Errorf("expecting 2, got: %v", chi[0])
}
c, chi, _ = alphabet.next()
if c != 'h' {
t.Errorf("expecting 'h', got: %v", c)
}
if chi[0] != 1 {
t.Errorf("expecting 1, got: %v", chi[0])
}
c, chi, _ = alphabet.next()
if c != 'p' {
t.Errorf("expecting 'p', got: %v", c)
}
if chi[0] != 12 {
t.Errorf("expecting 12, got: %v", chi[0])
}
c, chi, _ = alphabet.next()
if c != 'y' {
t.Errorf("expecting 'y', got: %v", c)
}
if chi[0] != 16 {
t.Errorf("expecting 16, got: %v", chi[0])
}
}
func TestFullCharacteristic(t *testing.T) {
fcv := FullCharacteristicVector([]uint32{2, 0})
if fcv.shiftAndMask(1, 1) != 1 {
t.Errorf("expected 1, got: %v", fcv.shiftAndMask(1, 1))
}
fcv = FullCharacteristicVector([]uint32{1<<5 + 1<<10, 0})
if fcv.shiftAndMask(3, 63) != 4 {
t.Errorf("expected 4, got: %v", fcv.shiftAndMask(3, 63))
}
}
func TestLongCharacteristic(t *testing.T) {
qChars := "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaabcabewa"
alphabet := queryChars(qChars)
c, chi, _ := alphabet.next()
if c != 'a' {
t.Errorf("expecting 'a', got: %v", c)
}
if chi.shiftAndMask(0, 7) != 7 {
t.Errorf("expecting 7 , got: %v", chi.shiftAndMask(0, 7))
}
if chi.shiftAndMask(28, 7) != 3 {
t.Errorf("expecting 3 , got: %v", chi.shiftAndMask(28, 7))
}
if chi.shiftAndMask(28, 127) != 1+2+16 {
t.Errorf("expecting 19 , got: %v", chi.shiftAndMask(28, 127))
}
if chi.shiftAndMask(28, 4095) != 1+2+16+256 {
t.Errorf("expecting 275 , got: %v", chi.shiftAndMask(28, 4095))
}
c, chi, _ = alphabet.next()
if c != 'b' {
t.Errorf("expecting 'b', got: %v", c)
}
if chi.shiftAndMask(0, 7) != 0 {
t.Errorf("expecting 0 , got: %v", chi.shiftAndMask(0, 7))
}
if chi.shiftAndMask(28, 15) != 4 {
t.Errorf("expecting 4 , got: %v", chi.shiftAndMask(28, 15))
}
if chi.shiftAndMask(28, 63) != 4+32 {
t.Errorf("expecting 36 , got: %v", chi.shiftAndMask(28, 63))
}
}
================================================
FILE: levenshtein/benchmark_test.go
================================================
// Copyright (c) 2018 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package levenshtein
import (
"testing"
)
func BenchmarkNewEvalEditDistance1(b *testing.B) {
lb, _ := NewLevenshteinAutomatonBuilder(1, true)
query := "coucibase"
for i := 0; i < b.N; i++ {
dfa, _ := lb.BuildDfa("couchbase", 1)
ed := dfa.eval([]byte(query))
if ed.distance() != 1 {
b.Errorf("expected distance 1, actual: %d", ed.distance())
}
}
}
func BenchmarkNewEvalEditDistance2(b *testing.B) {
lb, _ := NewLevenshteinAutomatonBuilder(2, false)
query := "couchbasefts"
for i := 0; i < b.N; i++ {
dfa, _ := lb.BuildDfa("couchbases", 2)
ed := dfa.eval([]byte(query))
if ed.distance() != 2 {
b.Errorf("expected distance 2, actual: %d", ed.distance())
}
}
}
func BenchmarkNewEditDistance1(b *testing.B) {
lb, _ := NewLevenshteinAutomatonBuilder(1, true)
query := "coucibase"
for i := 0; i < b.N; i++ {
dfa, _ := lb.BuildDfa("couchbase", 1)
state := dfa.initialState()
for _, b := range []byte(query) {
state = dfa.transition(state, b)
}
if !dfa.IsMatch(state) {
b.Errorf("expected isMatch %t, got %t", true, !dfa.IsMatch(state))
}
}
}
func BenchmarkNewEditDistance2(b *testing.B) {
lb, _ := NewLevenshteinAutomatonBuilder(2, false)
query := "couchbasefts"
for i := 0; i < b.N; i++ {
dfa, _ := lb.BuildDfa("couchbases", 2)
state := dfa.initialState()
for _, b := range []byte(query) {
state = dfa.transition(state, b)
}
if !dfa.IsMatch(state) {
b.Errorf("expected isMatch %t, got %t", true, !dfa.IsMatch(state))
}
}
}
================================================
FILE: levenshtein/dfa.go
================================================
// Copyright (c) 2018 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package levenshtein
import (
"fmt"
"math"
)
const SinkState = uint32(0)
type DFA struct {
transitions [][256]uint32
distances []Distance
initState int
ed uint8
}
/// Returns the initial state
func (d *DFA) initialState() int {
return d.initState
}
/// Returns the Levenshtein distance associated to the
/// current state.
func (d *DFA) distance(stateId int) Distance {
return d.distances[stateId]
}
/// Returns the number of states in the `DFA`.
func (d *DFA) numStates() int {
return len(d.transitions)
}
/// Returns the destination state reached after consuming a given byte.
func (d *DFA) transition(fromState int, b uint8) int {
return int(d.transitions[fromState][b])
}
func (d *DFA) eval(bytes []uint8) Distance {
state := d.initialState()
for _, b := range bytes {
state = d.transition(state, b)
}
return d.distance(state)
}
func (d *DFA) Start() int {
return int(d.initialState())
}
func (d *DFA) IsMatch(state int) bool {
if _, ok := d.distance(state).(Exact); ok {
return true
}
return false
}
func (d *DFA) CanMatch(state int) bool {
return state > 0 && state < d.numStates()
}
func (d *DFA) Accept(state int, b byte) int {
return int(d.transition(state, b))
}
// WillAlwaysMatch returns if the specified state will always end in a
// matching state.
func (d *DFA) WillAlwaysMatch(state int) bool {
return false
}
func fill(dest []uint32, val uint32) {
for i := range dest {
dest[i] = val
}
}
func fillTransitions(dest *[256]uint32, val uint32) {
for i := range dest {
dest[i] = val
}
}
type Utf8DFAStateBuilder struct {
dfaBuilder *Utf8DFABuilder
stateID uint32
defaultSuccessor []uint32
}
func (sb *Utf8DFAStateBuilder) addTransitionID(fromStateID uint32, b uint8,
toStateID uint32) {
sb.dfaBuilder.transitions[fromStateID][b] = toStateID
}
func (sb *Utf8DFAStateBuilder) addTransition(in rune, toStateID uint32) {
fromStateID := sb.stateID
chars := []byte(string(in))
lastByte := chars[len(chars)-1]
for i, ch := range chars[:len(chars)-1] {
remNumBytes := len(chars) - i - 1
defaultSuccessor := sb.defaultSuccessor[remNumBytes]
intermediateStateID := sb.dfaBuilder.transitions[fromStateID][ch]
if intermediateStateID == defaultSuccessor {
intermediateStateID = sb.dfaBuilder.allocate()
fillTransitions(&sb.dfaBuilder.transitions[intermediateStateID],
sb.defaultSuccessor[remNumBytes-1])
}
sb.addTransitionID(fromStateID, ch, intermediateStateID)
fromStateID = intermediateStateID
}
toStateIDDecoded := sb.dfaBuilder.getOrAllocate(original(toStateID))
sb.addTransitionID(fromStateID, lastByte, toStateIDDecoded)
}
type Utf8StateId uint32
func original(stateId uint32) Utf8StateId {
return predecessor(stateId, 0)
}
func predecessor(stateId uint32, numSteps uint8) Utf8StateId {
return Utf8StateId(stateId*4 + uint32(numSteps))
}
// Utf8DFABuilder makes it possible to define a DFA
// that takes unicode character, and build a `DFA`
// that operates on utf-8 encoded
type Utf8DFABuilder struct {
index []uint32
distances []Distance
transitions [][256]uint32
initialState uint32
numStates uint32
maxNumStates uint32
}
func withMaxStates(maxStates uint32) *Utf8DFABuilder {
rv := &Utf8DFABuilder{
index: make([]uint32, maxStates*2+100),
distances: make([]Distance, 0, maxStates),
transitions: make([][256]uint32, 0, maxStates),
maxNumStates: maxStates,
}
for i := range rv.index {
rv.index[i] = math.MaxUint32
}
return rv
}
func (dfab *Utf8DFABuilder) allocate() uint32 {
newState := dfab.numStates
dfab.numStates++
dfab.distances = append(dfab.distances, Atleast{d: 255})
dfab.transitions = append(dfab.transitions, [256]uint32{})
return newState
}
func (dfab *Utf8DFABuilder) getOrAllocate(state Utf8StateId) uint32 {
if int(state) >= cap(dfab.index) {
cloneIndex := make([]uint32, int(state)*2)
copy(cloneIndex, dfab.index)
dfab.index = cloneIndex
}
if dfab.index[state] != math.MaxUint32 {
return dfab.index[state]
}
nstate := dfab.allocate()
dfab.index[state] = nstate
return nstate
}
func (dfab *Utf8DFABuilder) setInitialState(iState uint32) {
decodedID := dfab.getOrAllocate(original(iState))
dfab.initialState = decodedID
}
func (dfab *Utf8DFABuilder) build(ed uint8) *DFA {
return &DFA{
transitions: dfab.transitions,
distances: dfab.distances,
initState: int(dfab.initialState),
ed: ed,
}
}
func (dfab *Utf8DFABuilder) addState(state, default_suc_orig uint32,
distance Distance) (*Utf8DFAStateBuilder, error) {
if state > dfab.maxNumStates {
return nil, fmt.Errorf("State id is larger than maxNumStates")
}
stateID := dfab.getOrAllocate(original(state))
dfab.distances[stateID] = distance
defaultSuccID := dfab.getOrAllocate(original(default_suc_orig))
// creates a chain of states of predecessors of `default_suc_orig`.
// Accepting k-bytes (whatever the bytes are) from `predecessor_states[k-1]`
// leads to the `default_suc_orig` state.
predecessorStates := []uint32{defaultSuccID,
defaultSuccID,
defaultSuccID,
defaultSuccID}
for numBytes := uint8(1); numBytes < 4; numBytes++ {
predecessorState := predecessor(default_suc_orig, numBytes)
predecessorStateID := dfab.getOrAllocate(predecessorState)
predecessorStates[numBytes] = predecessorStateID
succ := predecessorStates[numBytes-1]
fillTransitions(&dfab.transitions[predecessorStateID], succ)
}
// 1-byte encoded chars.
fill(dfab.transitions[stateID][0:192], predecessorStates[0])
// 2-bytes encoded chars.
fill(dfab.transitions[stateID][192:224], predecessorStates[1])
// 3-bytes encoded chars.
fill(dfab.transitions[stateID][224:240], predecessorStates[2])
// 4-bytes encoded chars.
fill(dfab.transitions[stateID][240:256], predecessorStates[3])
return &Utf8DFAStateBuilder{
dfaBuilder: dfab,
stateID: stateID,
defaultSuccessor: predecessorStates}, nil
}
================================================
FILE: levenshtein/dfa_test.go
================================================
// Copyright (c) 2018 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package levenshtein
import (
"testing"
)
func TestLevenshtein2(t *testing.T) {
dfaBuilder := withMaxStates(2)
dfaBuilder.addState(0, 1, Exact{d: 1})
dfaBuilder.addState(1, 0, Exact{d: 0})
dfaBuilder.setInitialState(1)
_ = dfaBuilder.build(1)
}
================================================
FILE: levenshtein/levenshtein.go
================================================
// Copyright (c) 2018 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package levenshtein
import "fmt"
// StateLimit is the maximum number of states allowed
const StateLimit = 10000
// ErrTooManyStates is returned if you attempt to build a Levenshtein
// automaton which requires too many states.
var ErrTooManyStates = fmt.Errorf("dfa contains more than %d states",
StateLimit)
// LevenshteinAutomatonBuilder wraps a precomputed
// datastructure that allows to produce small (but not minimal) DFA.
type LevenshteinAutomatonBuilder struct {
pDfa *ParametricDFA
}
// NewLevenshteinAutomatonBuilder creates a
// reusable, threadsafe Levenshtein automaton builder.
// `maxDistance` - maximum distance considered by the automaton.
// `transposition` - assign a distance of 1 for transposition
//
// Building this automaton builder is computationally intensive.
// While it takes only a few milliseconds for `d=2`, it grows
// exponentially with `d`. It is only reasonable to `d <= 5`.
func NewLevenshteinAutomatonBuilder(maxDistance uint8,
transposition bool) (*LevenshteinAutomatonBuilder, error) {
lnfa := newLevenshtein(maxDistance, transposition)
pdfa, err := fromNfa(lnfa)
if err != nil {
return nil, err
}
return &LevenshteinAutomatonBuilder{pDfa: pdfa}, nil
}
// BuildDfa builds the levenshtein automaton for serving
// queries with a given edit distance.
func (lab *LevenshteinAutomatonBuilder) BuildDfa(query string,
fuzziness uint8) (*DFA, error) {
return lab.pDfa.buildDfa(query, fuzziness, false)
}
// MaxDistance returns the MaxEdit distance supported by the
// LevenshteinAutomatonBuilder builder.
func (lab *LevenshteinAutomatonBuilder) MaxDistance() uint8 {
return lab.pDfa.maxDistance
}
================================================
FILE: levenshtein/levenshtein_nfa.go
================================================
// Copyright (c) 2018 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package levenshtein
import (
"math"
"sort"
)
/// Levenshtein Distance computed by a Levenshtein Automaton.
///
/// Levenshtein automata can only compute the exact Levenshtein distance
/// up to a given `max_distance`.
///
/// Over this distance, the automaton will invariably
/// return `Distance::AtLeast(max_distance + 1)`.
type Distance interface {
distance() uint8
}
type Exact struct {
d uint8
}
func (e Exact) distance() uint8 {
return e.d
}
type Atleast struct {
d uint8
}
func (a Atleast) distance() uint8 {
return a.d
}
func characteristicVector(query []rune, c rune) uint64 {
chi := uint64(0)
for i := 0; i < len(query); i++ {
if query[i] == c {
chi |= 1 << uint64(i)
}
}
return chi
}
type NFAState struct {
Offset uint32
Distance uint8
InTranspose bool
}
type NFAStates []NFAState
func (ns NFAStates) Len() int {
return len(ns)
}
func (ns NFAStates) Less(i, j int) bool {
if ns[i].Offset != ns[j].Offset {
return ns[i].Offset < ns[j].Offset
}
if ns[i].Distance != ns[j].Distance {
return ns[i].Distance < ns[j].Distance
}
return !ns[i].InTranspose && ns[j].InTranspose
}
func (ns NFAStates) Swap(i, j int) {
ns[i], ns[j] = ns[j], ns[i]
}
func (ns *NFAState) imply(other NFAState) bool {
transposeImply := ns.InTranspose
if !other.InTranspose {
transposeImply = !other.InTranspose
}
deltaOffset := ns.Offset - other.Offset
if ns.Offset < other.Offset {
deltaOffset = other.Offset - ns.Offset
}
if transposeImply {
return uint32(other.Distance) >= (uint32(ns.Distance) + deltaOffset)
}
return uint32(other.Distance) > (uint32(ns.Distance) + deltaOffset)
}
type MultiState struct {
states []NFAState
}
func (ms *MultiState) States() []NFAState {
return ms.states
}
func (ms *MultiState) Clear() {
ms.states = ms.states[:0]
}
func newMultiState() *MultiState {
return &MultiState{states: make([]NFAState, 0)}
}
func (ms *MultiState) normalize() uint32 {
minOffset := uint32(math.MaxUint32)
for _, s := range ms.states {
if s.Offset < minOffset {
minOffset = s.Offset
}
}
if minOffset == uint32(math.MaxUint32) {
minOffset = 0
}
for i := 0; i < len(ms.states); i++ {
ms.states[i].Offset -= minOffset
}
sort.Sort(NFAStates(ms.states))
return minOffset
}
func (ms *MultiState) addStates(nState NFAState) {
for _, s := range ms.states {
if s.imply(nState) {
return
}
}
i := 0
for i < len(ms.states) {
if nState.imply(ms.states[i]) {
ms.states = append(ms.states[:i], ms.states[i+1:]...)
} else {
i++
}
}
ms.states = append(ms.states, nState)
}
func extractBit(bitset uint64, pos uint8) bool {
shift := bitset >> pos
bit := shift & 1
return bit == uint64(1)
}
func dist(left, right uint32) uint32 {
if left > right {
return left - right
}
return right - left
}
type LevenshteinNFA struct {
mDistance uint8
damerau bool
}
func newLevenshtein(maxD uint8, transposition bool) *LevenshteinNFA {
return &LevenshteinNFA{mDistance: maxD,
damerau: transposition,
}
}
func (la *LevenshteinNFA) maxDistance() uint8 {
return la.mDistance
}
func (la *LevenshteinNFA) msDiameter() uint8 {
return 2*la.mDistance + 1
}
func (la *LevenshteinNFA) initialStates() *MultiState {
ms := MultiState{}
nfaState := NFAState{}
ms.addStates(nfaState)
return &ms
}
func (la *LevenshteinNFA) multistateDistance(ms *MultiState,
queryLen uint32) Distance {
minDistance := Atleast{d: la.mDistance + 1}
for _, s := range ms.states {
t := s.Distance + uint8(dist(queryLen, s.Offset))
if t <= uint8(la.mDistance) {
if minDistance.distance() > t {
minDistance.d = t
}
}
}
if minDistance.distance() == la.mDistance+1 {
return Atleast{d: la.mDistance + 1}
}
return minDistance
}
func (la *LevenshteinNFA) simpleTransition(state NFAState,
symbol uint64, ms *MultiState) {
if state.Distance < la.mDistance {
// insertion
ms.addStates(NFAState{Offset: state.Offset,
Distance: state.Distance + 1,
InTranspose: false})
// substitution
ms.addStates(NFAState{Offset: state.Offset + 1,
Distance: state.Distance + 1,
InTranspose: false})
n := la.mDistance + 1 - state.Distance
for d := uint8(1); d < n; d++ {
if extractBit(symbol, d) {
// for d > 0, as many deletion and character match
ms.addStates(NFAState{Offset: state.Offset + 1 + uint32(d),
Distance: state.Distance + d,
InTranspose: false})
}
}
if la.damerau && extractBit(symbol, 1) {
ms.addStates(NFAState{
Offset: state.Offset,
Distance: state.Distance + 1,
InTranspose: true})
}
}
if extractBit(symbol, 0) {
ms.addStates(NFAState{Offset: state.Offset + 1,
Distance: state.Distance,
InTranspose: false})
}
if state.InTranspose && extractBit(symbol, 0) {
ms.addStates(NFAState{Offset: state.Offset + 2,
Distance: state.Distance,
InTranspose: false})
}
}
func (la *LevenshteinNFA) transition(cState *MultiState,
dState *MultiState, scv uint64) {
dState.Clear()
mask := (uint64(1) << la.msDiameter()) - uint64(1)
for _, state := range cState.states {
cv := (scv >> state.Offset) & mask
la.simpleTransition(state, cv, dState)
}
sort.Sort(NFAStates(dState.states))
}
func (la *LevenshteinNFA) computeDistance(query, other []rune) Distance {
cState := la.initialStates()
nState := newMultiState()
for _, i := range other {
nState.Clear()
chi := characteristicVector(query, i)
la.transition(cState, nState, chi)
cState, nState = nState, cState
}
return la.multistateDistance(cState, uint32(len(query)))
}
================================================
FILE: levenshtein/levenshtein_test.go
================================================
// Copyright (c) 2018 Couchbase, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package levenshtein
import (
"testing"
)
func TestLevenshtein(t *testing.T) {
hash := make(map[uint8]LevenshteinAutomatonBuilder, 4)
for i := 0; i < 3; i++ {
lb, err := NewLevenshteinAutomatonBuilder(uint8(i), false)
if err != nil {
t.Errorf("NewLevenshteinAutomatonBuilder(%d, false) failed, err: %v", i, err)
}
hash[uint8(i)] = *lb
}
tests := []struct {
desc string
query string
distance uint8
seq []byte
isMatch bool
canMatch bool
}{
{
desc: "cat/0 - c a t",
query: "cat",
distance: 0,
seq: []byte{'c', 'a', 't'},
isMatch: true,
canMatch: true,
},
{
desc: "cat/1 - c a",
query: "cat",
distance: 1,
seq: []byte{'c', 'a'},
isMatch: true,
canMatch: true,
},
{
desc: "cat/1 - c a t s",
query: "cat",
distance: 1,
seq: []byte{'c', 'a', 't', 's'},
isMatch: true,
canMatch: true,
},
{
desc: "cat/0 - c a",
query: "cat",
distance: 0,
seq: []byte{'c', 'a'},
isMatch: false,
canMatch: true,
},
{
desc: "cat/0 - c a t s",
query: "cat",
distance: 0,
seq: []byte{'c', 'a', 't', 's'},
isMatch: false,
canMatch: false,
},
{
desc: "cate/1 - cate",
query: "cate",
distance: 1,
seq: []byte{'c', 'a', 't', 'e'},
isMatch: true,
canMatch: true,
},
{
desc: "cater/1 - cate",
query: "cater",
distance: 1,
seq: []byte{'c', 'a', 't', 'e'},
isMatch: true,
canMatch: true,
},
{
desc: "cater/1 - ctr",
query: "cater",
distance: 1,
seq: []byte{'c', 't', 'r'},
isMatch: false,
canMatch: false,
},
{
desc: "catered/2 - cater",
query: "catered",
distance: 2,
seq: []byte{'c', 'a', 't', 'e', 'r'},
isMatch: true,
canMatch: true,
},
// this section contains cases where the sequence
// of bytes encountered contains utf-8 encoded
// multi-byte characters, which should count as 1
// for the purposes of the levenshtein edit distance
{
desc: "cat/0 - c 0xc3 0xa1 t (cát)",
query: "cat",
distance: 0,
seq: []byte{'c', 0xc3, 0xa1, 't'},
isMatch: false,
canMatch: false,
},
{
desc: "cat/1 - c 0xc3 0xa1 t (cát)",
query: "cat",
distance: 1,
seq: []byte{'c', 0xc3, 0xa1, 't'},
isMatch: true,
canMatch: true,
},
{
desc: "cat/1 - c 0xc3 0xa1 t (cáts)",
query: "cat",
distance: 1,
seq: []byte{'c', 0xc3, 0xa1, 't', 's'},
isMatch: false,
canMatch: false,
},
{
desc: "cat/1 - 0xc3 0xa1 (á)",
query: "cat",
distance: 1,
seq: []byte{0xc3, 0xa1},
isMatch: false,
canMatch: true,
},
{
desc: "cat/1 - c 0xc3 0xa1 t (ácat)",
query: "cat",
distance: 1,
seq: []byte{0xc3, 0xa1, 'c', 'a', 't'},
isMatch: true,
canMatch: true,
},
// this section has utf-8 encoded multi-byte characters
// in the query, which should still just count as 1
// for the purposes of the levenshtein edit distance
{
desc: "cát/0 - c a t (cat)",
query: "cát",
distance: 0,
seq: []byte{'c', 'a', 't'},
isMatch: false,
canMatch: false,
},
{
desc: "cát/1 - c 0xc3 0xa1 (cá)",
query: "cát",
distance: 1,
seq: []byte{'c', 0xc3, 0xa1},
isMatch: true,
canMatch: true,
},
{
desc: "cát/1 - c 0xc3 0xa1 s (cás)",
query: "cát",
distance: 1,
seq: []byte{'c', 0xc3, 0xa1, 's'},
isMatch: true,
canMatch: true,
},
{
desc: "cát/1 - c 0xc3 0xa1 t a (cáta)",
query: "cát",
distance: 1,
seq: []byte{'c', 0xc3, 0xa1, 't', 'a'},
isMatch: true,
canMatch: true,
},
{
desc: "cát/1 - d 0xc3 0xa1 (dát)",
query: "cát",
distance: 1,
seq: []byte{'d', 0xc3, 0xa1, 't'},
isMatch: true,
canMatch: true,
},
{
desc: "cát/1 - c a t (cat)",
query: "cát",
distance: 1,
seq: []byte{'c', 'a', 't'},
isMatch: true,
canMatch: true,
},
{
desc: "cát/1 - c a t (cats)",
query: "cát",
distance: 1,
seq: []byte{'c', 'a', 't', 's'},
isMatch: false,
canMatch: false,
},
{
desc: "cát/1 - 0xc3, 0xa (á)",
query: "cát",
distance: 1,
seq: []byte{0xc3, 0xa1},
isMatch: false,
canMatch: true,
},
{
desc: "cát/1 - a c 0xc3 0xa1 t (acát)",
query: "cát",
distance: 1,
seq: []byte{'a', 'c', 0xc3, 0xa1, 't'},
isMatch: true,
canMatch: true,
},
}
for _, test := range tests {
t.Run(test.desc, func(t *testing.T) {
l, err := hash[uint8(test.distance)].pDfa.buildDfa(test.query, test.distance, false)
if err != nil {
t.Errorf("buildDfa(%s, %d, false) failed, err: %v", test.query,
test.distance, err)
}
s := l.Start()
for _, b := range test.seq {
s = l.Accept(s, b)
if uint32(s) == SinkState {
break
}
}
isMatch := l.IsMatch(s)
if isMatch != test.isMatch {
t.Errorf("expected isMatch %t, got %t", test.isMatch, isMatch)
}
canMatch := l.CanMatch(s)
if canMatch != test.canMatch {
t.Errorf("expectec canMatch %t, got %t", test.canMatch, canMatch)
}
})
}
}
func makeDistance(d uint8, md uint8) Distance {
if d > md {
return Atleast{d: md + 1}
}
return Exact{d: d}
}
func testLevenshteinNfaUtil(left, right string, ed uint8, t *testing.T) {
for _, d := range []uint8{0, 1, 2, 3} {
expectedDistance := makeDistance(ed, uint8(d))
lev := newLevenshtein(d, false)
testSymmetric(lev, left, right, expectedDistance, t)
}
}
func testSymmetric(lev *LevenshteinNFA, left, right string, expected Distance, t *testing.T) {
levd := lev.computeDistance([]rune(left), []rune(right))
if levd.distance() != expected.distance() {
t.Errorf("expected distance: %d, actual: %d", expected.distance(), levd.distance())
}
levd = lev.computeDistance([]rune(right), []rune(left))
if levd.distance() != expected.distance() {
t.Errorf("expected distance: %d, actual: %d", expected.distance(), levd.distance())
}
}
func TestLevenshteinNfa(t *testing.T) {
testLevenshteinNfaUtil("abc", "abc", 0, t)
testLevenshteinNfaUtil("abc", "abcd", 1, t)
testLevenshteinNfaUtil("aab", "ab", 1, t)
}
/*func TestDeadState(t *testing.T) {
nfa := newLevenshtein(2, false)
pdfa := fromNfa(nfa)
dfa := pdfa.buildDfa("abcdefghijklmnop", 0, false)
state := dfa.initialState()
r := []rune("X")
state = dfa.transition(state, uint8(r[0]))
if state != 0 {
t.Errorf("expected state: 0, actual: %d", state)
}
state = dfa.transition(state, uint8(r[0]))
if state != 0 {
t.Errorf("expected state: 0, actual: %d", state)
}
state = dfa.transition(state, uint8(r[0]))
if state != 0 {
t.Errorf("expected state: 0, actual: %d", state)
}
}*/
func TestLevenshteinParametricDfa(t *testing.T) {
lev := newLevenshtein(1, true)
pDfa, err := fromNfa(lev)
if err != nil {
t.Errorf("fromNfa err: %v", err)
}
testStr := "abc"
dfa, err := pDfa.buildDfa(testStr, 1, false)
if err != nil {
t.Errorf("buildDfa(%s, 1, false) failed, err: %v", testStr, err)
}
rd := dfa.eval([]byte("abc"))
if rd.distance() != 0 {
t.Errorf("expected distance 0, actual: %d", rd.distance())
}
rd = dfa.eval([]byte("ab"))
if rd.distance() != 1 {
t.Errorf("expected distance 1, actual: %d", rd.distance())
}
rd = dfa.eval([]byte("ac"))
if rd.distance() != 1 {
t.Errorf("expected distance 1, actual: %d", rd.distance())
}
rd = dfa.eval([]byte("a"))
if rd.distance() != 2 {
t.Errorf("expected distance 2, actual: %d", rd.distance())
}
rd = dfa.eval([]byte("abcd"))
if rd.distance() != 1 {
t.Errorf("expected distance 1, actual: %d", rd.distance())
}
rd = dfa.eval([]byte("abdd"))
if rd.distance() != 2 {
t.Errorf("expected distance 2, actual: %d", rd.distance())
}
testStr = "abcdefghijlmnopqrstuvwxyz" +
"abcdefghijlmnopqrstuvwxyz" +
"abcdefghijlmnopqrstuvwxyz" +
"abcdefghijlmnopqrstuvwxyz"
dfa, err = pDfa.buildDfa(testStr, 1, false)
if err != nil {
t.Errorf("buildDfa(%s, 1, false) failed, err: %v", testStr, err)
}
sample1 := "abcdefghijlmnopqrstuvwxyz" +
"abcdefghijlnopqrstuvwxyz" +
"abcdefghijlmnopqrstuvwxyz" +
"abcdefghijlmnopqrstuvwxyz"
rd = dfa.eval([]byte(sample1))
if rd.distance() != 1 {
t.Errorf("expected distance 1, actual: %d", rd.distance())
}
sample2 := "abcdefghijlmnopqrstuvwxyz" +
"abcdefghijlnopqrstuvwxyz" +
"abcdefghijlmnopqrstuvwxyz" +
"abcdefghijlmnoprqstuvwxyz"
rd = dfa.eval([]byte(sample2))
if rd.distance() != 2 {
t.Errorf("expected distance 2, actual: %d", rd.distance())
}
}
func TestDamerau(t *testing.T) {
nfa := newLevenshtein(2, true)
testSymmetric(nfa, "abc", "abc", Exact{d: 0}, t)
testSymmetric(nfa, "abc", "abcd", Exact{d: 1}, t)
testSymmetric(nfa, "abcdef", "abddef", Exact{d: 1}, t)
testSymmetric(nfa, "abcdef", "abdcef", Exact{d: 1}, t)
}
func TestLevenshteinDfa(t *testing.T) {
nfa := newLevenshtein(2, false)
pDfa, err := fromNfa(nfa)
if err != nil {
t.Errorf("fromNfa failed, err: %v", err)
}
dfa, err := pDfa.buildDfa("abcabcaaabc", 2, false)
if err != nil {
t.Errorf("buildDfa(abcabcaaabc, 1, false) failed, err: %v", err)
}
if dfa.numStates() != 273 {
t.Errorf("expected number of states: 273, actual: %d", dfa.numStates())
}
}
func TestUtf8Simple(t *testing.T) {
nfa := newLevenshtein(1, false)
pDfa, err := fromNfa(nfa)
if err != nil {
t.Errorf("fromNfa failed, err: %v", err)
}
dfa, err := pDfa.buildDfa("あ", 1, false)
if err != nil {
t.Errorf("buildDfa(あ, 1, false) failed, err: %v", err)
}
ed := dfa.eval([]byte("あ"))
if ed.distance() != 0 {
t.Errorf("expected distance 0, actual: %d", ed.distance())
}
}
func TestSimple(t *testing.T) {
query := "abcdef"
nfa := newLevenshtein(2, false)
pDfa, err := fromNfa(nfa)
if err != nil {
t.Errorf("fromNfa failed, err: %v", err)
}
dfa, err := pDfa.buildD
gitextract_4hy_b8n1/ ├── .github/ │ └── workflows/ │ └── tests.yml ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── automaton.go ├── builder.go ├── builder_test.go ├── cmd/ │ └── vellum/ │ ├── cmd/ │ │ ├── dot.go │ │ ├── dump.go │ │ ├── fuzzy.go │ │ ├── grep.go │ │ ├── info.go │ │ ├── map.go │ │ ├── range.go │ │ ├── root.go │ │ ├── set.go │ │ └── svg.go │ └── main.go ├── common.go ├── common_test.go ├── data/ │ └── words-1000.txt ├── decoder_v1.go ├── decoder_v1_test.go ├── docs/ │ └── format.md ├── encoder_v1.go ├── encoder_v1_test.go ├── encoding.go ├── example_test.go ├── fst.go ├── fst_iterator.go ├── fst_iterator_test.go ├── go.mod ├── go.sum ├── levenshtein/ │ ├── LICENSE │ ├── README.md │ ├── alphabet.go │ ├── alphabet_test.go │ ├── benchmark_test.go │ ├── dfa.go │ ├── dfa_test.go │ ├── levenshtein.go │ ├── levenshtein_nfa.go │ ├── levenshtein_test.go │ └── parametric_dfa.go ├── merge_iterator.go ├── merge_iterator_test.go ├── pack.go ├── pack_test.go ├── regexp/ │ ├── compile.go │ ├── compile_test.go │ ├── dfa.go │ ├── inst.go │ ├── regexp.go │ ├── regexp_test.go │ ├── sparse.go │ └── sparse_test.go ├── registry.go ├── registry_test.go ├── transducer.go ├── utf8/ │ ├── utf8.go │ └── utf8_test.go ├── vellum.go ├── vellum_mmap.go ├── vellum_nommap.go ├── vellum_test.go ├── vendor/ │ ├── github.com/ │ │ ├── edsrzf/ │ │ │ └── mmap-go/ │ │ │ ├── LICENSE │ │ │ ├── mmap.go │ │ │ ├── mmap_unix.go │ │ │ ├── mmap_windows.go │ │ │ ├── msync_netbsd.go │ │ │ └── msync_unix.go │ │ ├── inconshreveable/ │ │ │ └── mousetrap/ │ │ │ ├── LICENSE │ │ │ ├── trap_others.go │ │ │ ├── trap_windows.go │ │ │ └── trap_windows_1.4.go │ │ ├── spf13/ │ │ │ ├── cobra/ │ │ │ │ ├── LICENSE.txt │ │ │ │ ├── bash_completions.go │ │ │ │ ├── cobra.go │ │ │ │ ├── command.go │ │ │ │ ├── command_notwin.go │ │ │ │ ├── command_win.go │ │ │ │ └── doc/ │ │ │ │ ├── man_docs.go │ │ │ │ ├── md_docs.go │ │ │ │ ├── util.go │ │ │ │ └── yaml_docs.go │ │ │ └── pflag/ │ │ │ ├── LICENSE │ │ │ ├── bool.go │ │ │ ├── bool_slice.go │ │ │ ├── count.go │ │ │ ├── duration.go │ │ │ ├── flag.go │ │ │ ├── float32.go │ │ │ ├── float64.go │ │ │ ├── golangflag.go │ │ │ ├── int.go │ │ │ ├── int32.go │ │ │ ├── int64.go │ │ │ ├── int8.go │ │ │ ├── int_slice.go │ │ │ ├── ip.go │ │ │ ├── ip_slice.go │ │ │ ├── ipmask.go │ │ │ ├── ipnet.go │ │ │ ├── string.go │ │ │ ├── string_array.go │ │ │ ├── string_slice.go │ │ │ ├── uint.go │ │ │ ├── uint16.go │ │ │ ├── uint32.go │ │ │ ├── uint64.go │ │ │ ├── uint8.go │ │ │ └── uint_slice.go │ │ └── willf/ │ │ └── bitset/ │ │ ├── LICENSE │ │ ├── bitset.go │ │ ├── popcnt.go │ │ ├── popcnt_amd64.go │ │ ├── popcnt_amd64.s │ │ └── popcnt_generic.go │ └── manifest ├── writer.go └── writer_test.go
SYMBOL INDEX (1194 symbols across 103 files)
FILE: automaton.go
type Automaton (line 18) | type Automaton interface
function AutomatonContains (line 40) | func AutomatonContains(a Automaton, k []byte) bool {
type AlwaysMatch (line 57) | type AlwaysMatch struct
method Start (line 60) | func (m *AlwaysMatch) Start() int {
method IsMatch (line 65) | func (m *AlwaysMatch) IsMatch(int) bool {
method CanMatch (line 70) | func (m *AlwaysMatch) CanMatch(int) bool {
method WillAlwaysMatch (line 75) | func (m *AlwaysMatch) WillAlwaysMatch(int) bool {
method Accept (line 80) | func (m *AlwaysMatch) Accept(int, byte) int {
FILE: builder.go
type Builder (line 30) | type Builder struct
method Reset (line 74) | func (b *Builder) Reset(w io.Writer) error {
method Insert (line 91) | func (b *Builder) Insert(key []byte, val uint64) error {
method copyLastKey (line 114) | func (b *Builder) copyLastKey(key []byte) {
method Close (line 124) | func (b *Builder) Close() error {
method compileFrom (line 137) | func (b *Builder) compileFrom(iState int) error {
method compile (line 156) | func (b *Builder) compile(node *builderNode) (int, error) {
constant noneAddr (line 44) | noneAddr = 1
constant emptyAddr (line 45) | emptyAddr = 0
function newBuilder (line 49) | func newBuilder(w io.Writer, opts *BuilderOpts) (*Builder, error) {
type unfinishedNodes (line 175) | type unfinishedNodes struct
method Reset (line 189) | func (u *unfinishedNodes) Reset() {
method get (line 208) | func (u *unfinishedNodes) get() *builderNodeUnfinished {
method put (line 217) | func (u *unfinishedNodes) put() {
method findCommonPrefixAndSetOutput (line 225) | func (u *unfinishedNodes) findCommonPrefixAndSetOutput(key []byte,
method pushEmpty (line 254) | func (u *unfinishedNodes) pushEmpty(final bool) {
method popRoot (line 261) | func (u *unfinishedNodes) popRoot() *builderNode {
method popFreeze (line 270) | func (u *unfinishedNodes) popFreeze(addr int) *builderNode {
method popEmpty (line 280) | func (u *unfinishedNodes) popEmpty() *builderNode {
method setRootOutput (line 289) | func (u *unfinishedNodes) setRootOutput(out uint64) {
method topLastFreeze (line 294) | func (u *unfinishedNodes) topLastFreeze(addr int) {
method addSuffix (line 299) | func (u *unfinishedNodes) addSuffix(bs []byte, out uint64) {
function newUnfinishedNodes (line 197) | func newUnfinishedNodes(p *builderNodePool) *unfinishedNodes {
type builderNodeUnfinished (line 318) | type builderNodeUnfinished struct
method lastCompiled (line 325) | func (b *builderNodeUnfinished) lastCompiled(addr int) {
method addOutputPrefix (line 339) | func (b *builderNodeUnfinished) addOutputPrefix(prefix uint64) {
type builderNode (line 351) | type builderNode struct
method reset (line 361) | func (n *builderNode) reset() {
method equiv (line 371) | func (n *builderNode) equiv(o *builderNode) bool {
type transition (line 398) | type transition struct
function outputPrefix (line 404) | func outputPrefix(l, r uint64) uint64 {
function outputSub (line 411) | func outputSub(l, r uint64) uint64 {
function outputCat (line 415) | func outputCat(l, r uint64) uint64 {
type builderNodePool (line 432) | type builderNodePool struct
method Get (line 436) | func (p *builderNodePool) Get() *builderNode {
method Put (line 445) | func (p *builderNodePool) Put(v *builderNode) {
FILE: builder_test.go
function init (line 26) | func init() {
function TestBuilderSimple (line 32) | func TestBuilderSimple(t *testing.T) {
function TestBuilderSharedPrefix (line 71) | func TestBuilderSharedPrefix(t *testing.T) {
function randomValues (line 104) | func randomValues(list []string) []uint64 {
function insertStrings (line 112) | func insertStrings(b *Builder, list []string, vals []uint64) error {
function insertStringMap (line 129) | func insertStringMap(b *Builder, m map[string]uint64) error {
function TestBuilderNodeEquiv (line 147) | func TestBuilderNodeEquiv(t *testing.T) {
function loadWords (line 204) | func loadWords(path string) ([]string, error) {
function BenchmarkBuilder (line 235) | func BenchmarkBuilder(b *testing.B) {
FILE: cmd/vellum/cmd/dot.go
function dotToWriter (line 46) | func dotToWriter(fst *vellum.FST, w io.Writer) error {
constant dotHeader (line 70) | dotHeader = `
constant dotFooter (line 77) | dotFooter = `}
type dotStringer (line 80) | type dotStringer interface
function init (line 84) | func init() {
FILE: cmd/vellum/cmd/dump.go
function debugPrint (line 44) | func debugPrint(n int, state interface{}) error {
function init (line 49) | func init() {
FILE: cmd/vellum/cmd/fuzzy.go
function init (line 74) | func init() {
FILE: cmd/vellum/cmd/grep.go
function init (line 68) | func init() {
FILE: cmd/vellum/cmd/info.go
function init (line 45) | func init() {
FILE: cmd/vellum/cmd/map.go
function init (line 95) | func init() {
FILE: cmd/vellum/cmd/range.go
function init (line 60) | func init() {
FILE: cmd/vellum/cmd/root.go
function Execute (line 42) | func Execute() {
function init (line 49) | func init() {
FILE: cmd/vellum/cmd/set.go
function init (line 86) | func init() {
FILE: cmd/vellum/cmd/svg.go
function svgToWriter (line 48) | func svgToWriter(fst *vellum.FST, w io.Writer) error {
function init (line 67) | func init() {
FILE: cmd/vellum/main.go
function main (line 23) | func main() {
FILE: common.go
constant maxCommon (line 17) | maxCommon = 1<<6 - 1
function encodeCommon (line 19) | func encodeCommon(in byte) byte {
function decodeCommon (line 27) | func decodeCommon(in byte) byte {
FILE: common_test.go
function TestCommonInputs (line 19) | func TestCommonInputs(t *testing.T) {
function roundTrip (line 39) | func roundTrip(t *testing.T, b byte) {
FILE: decoder_v1.go
function init (line 24) | func init() {
type decoderV1 (line 30) | type decoderV1 struct
method getRoot (line 40) | func (d *decoderV1) getRoot() int {
method getLen (line 49) | func (d *decoderV1) getLen() int {
method stateAt (line 58) | func (d *decoderV1) stateAt(addr int, prealloc fstState) (fstState, er...
function newDecoderV1 (line 34) | func newDecoderV1(data []byte) *decoderV1 {
type fstStateV1 (line 72) | type fstStateV1 struct
method isEncodedSingle (line 99) | func (f *fstStateV1) isEncodedSingle() bool {
method at (line 106) | func (f *fstStateV1) at(data []byte, addr int) error {
method atZero (line 124) | func (f *fstStateV1) atZero() error {
method atNone (line 133) | func (f *fstStateV1) atNone() error {
method atSingle (line 142) | func (f *fstStateV1) atSingle(data []byte, addr int) error {
method atMulti (line 176) | func (f *fstStateV1) atMulti(data []byte, addr int) error {
method Address (line 211) | func (f *fstStateV1) Address() int {
method Final (line 215) | func (f *fstStateV1) Final() bool {
method FinalOutput (line 219) | func (f *fstStateV1) FinalOutput() uint64 {
method NumTransitions (line 226) | func (f *fstStateV1) NumTransitions() int {
method TransitionAt (line 230) | func (f *fstStateV1) TransitionAt(i int) byte {
method TransitionFor (line 238) | func (f *fstStateV1) TransitionFor(b byte) (int, int, uint64) {
method String (line 264) | func (f *fstStateV1) String() string {
method DotString (line 289) | func (f *fstStateV1) DotString(num int) string {
function escapeInput (line 311) | func escapeInput(b byte) string {
FILE: decoder_v1_test.go
function TestDecoderVersionError (line 22) | func TestDecoderVersionError(t *testing.T) {
function TestShortHeader (line 29) | func TestShortHeader(t *testing.T) {
function TestDecoderRootLen (line 37) | func TestDecoderRootLen(t *testing.T) {
function TestDecoderStateAt (line 47) | func TestDecoderStateAt(t *testing.T) {
function TestFSTStateFinalOutput (line 373) | func TestFSTStateFinalOutput(t *testing.T) {
function TestDecodeStateZero (line 412) | func TestDecodeStateZero(t *testing.T) {
function TestDecodeAtInvalid (line 426) | func TestDecodeAtInvalid(t *testing.T) {
function TestFSTStateTransitionAt (line 434) | func TestFSTStateTransitionAt(t *testing.T) {
FILE: encoder_v1.go
constant versionV1 (line 23) | versionV1 = 1
constant oneTransition (line 24) | oneTransition = 1 << 7
constant transitionNext (line 25) | transitionNext = 1 << 6
constant stateFinal (line 26) | stateFinal = 1 << 6
constant footerSizeV1 (line 27) | footerSizeV1 = 16
function init (line 29) | func init() {
type encoderV1 (line 35) | type encoderV1 struct
method reset (line 45) | func (e *encoderV1) reset(w io.Writer) {
method start (line 49) | func (e *encoderV1) start() error {
method encodeState (line 63) | func (e *encoderV1) encodeState(s *builderNode, lastAddr int) (int, er...
method encodeStateOne (line 74) | func (e *encoderV1) encodeStateOne(s *builderNode) (int, error) {
method encodeStateOneFinish (line 100) | func (e *encoderV1) encodeStateOneFinish(s *builderNode, next byte) (i...
method encodeStateMany (line 118) | func (e *encoderV1) encodeStateMany(s *builderNode) (int, error) {
method finish (line 211) | func (e *encoderV1) finish(count, rootAddr int) error {
function newEncoderV1 (line 39) | func newEncoderV1(w io.Writer) *encoderV1 {
FILE: encoder_v1_test.go
function TestEncoderVersionError (line 28) | func TestEncoderVersionError(t *testing.T) {
function TestEncoderStart (line 35) | func TestEncoderStart(t *testing.T) {
function TestEncoderStateOneNextWithCommonInput (line 59) | func TestEncoderStateOneNextWithCommonInput(t *testing.T) {
function TestEncoderStateOneNextWithUncommonInput (line 95) | func TestEncoderStateOneNextWithUncommonInput(t *testing.T) {
function TestEncoderStateOneNotNextWithCommonInputNoValue (line 132) | func TestEncoderStateOneNotNextWithCommonInputNoValue(t *testing.T) {
function TestEncoderStateOneNotNextWithUncommonInputNoValue (line 173) | func TestEncoderStateOneNotNextWithUncommonInputNoValue(t *testing.T) {
function TestEncoderStateOneNotNextWithCommonInputWithValue (line 215) | func TestEncoderStateOneNotNextWithCommonInputWithValue(t *testing.T) {
function TestEncoderStateOneNotNextWithUncommonInputWithValue (line 258) | func TestEncoderStateOneNotNextWithUncommonInputWithValue(t *testing.T) {
function TestEncoderStateManyWithNoValues (line 302) | func TestEncoderStateManyWithNoValues(t *testing.T) {
function TestEncoderStateManyWithValues (line 356) | func TestEncoderStateManyWithValues(t *testing.T) {
function TestEncoderStateMaxTransitions (line 416) | func TestEncoderStateMaxTransitions(t *testing.T) {
function TestEncoderStateMoreTransitionsThanFitInHeader (line 420) | func TestEncoderStateMoreTransitionsThanFitInHeader(t *testing.T) {
function testEncoderStateNTransitions (line 424) | func testEncoderStateNTransitions(t *testing.T, n int) {
FILE: encoding.go
constant headerSize (line 23) | headerSize = 16
type encoderConstructor (line 25) | type encoderConstructor
type decoderConstructor (line 26) | type decoderConstructor
type encoder (line 31) | type encoder interface
function loadEncoder (line 38) | func loadEncoder(ver int, w io.Writer) (encoder, error) {
function registerEncoder (line 45) | func registerEncoder(ver int, cons encoderConstructor) {
type decoder (line 49) | type decoder interface
function loadDecoder (line 55) | func loadDecoder(ver int, data []byte) (decoder, error) {
function registerDecoder (line 62) | func registerDecoder(ver int, cons decoderConstructor) {
function decodeHeader (line 66) | func decodeHeader(header []byte) (ver int, typ int, err error) {
type fstState (line 80) | type fstState interface
FILE: example_test.go
function Example (line 25) | func Example() {
FILE: fst.go
type FST (line 27) | type FST struct
method Contains (line 58) | func (f *FST) Contains(val []byte) (bool, error) {
method Get (line 66) | func (f *FST) Get(input []byte) (uint64, bool, error) {
method get (line 70) | func (f *FST) get(input []byte, prealloc fstState) (uint64, bool, erro...
method Version (line 99) | func (f *FST) Version() int {
method Len (line 104) | func (f *FST) Len() int {
method Type (line 109) | func (f *FST) Type() int {
method Close (line 116) | func (f *FST) Close() error {
method Start (line 129) | func (f *FST) Start() int {
method IsMatch (line 134) | func (f *FST) IsMatch(addr int) bool {
method CanMatch (line 141) | func (f *FST) CanMatch(addr int) bool {
method WillAlwaysMatch (line 150) | func (f *FST) WillAlwaysMatch(int) bool {
method Accept (line 155) | func (f *FST) Accept(addr int, b byte) int {
method IsMatchWithVal (line 162) | func (f *FST) IsMatchWithVal(addr int) (bool, uint64) {
method AcceptWithVal (line 172) | func (f *FST) AcceptWithVal(addr int, b byte) (int, uint64) {
method Iterator (line 183) | func (f *FST) Iterator(startKeyInclusive, endKeyExclusive []byte) (*FS...
method Search (line 190) | func (f *FST) Search(aut Automaton, startKeyInclusive, endKeyExclusive...
method Debug (line 196) | func (f *FST) Debug(callback func(int, interface{}) error) error {
method Reader (line 242) | func (f *FST) Reader() (*Reader, error) {
method GetMinKey (line 246) | func (f *FST) GetMinKey() ([]byte, error) {
method GetMaxKey (line 269) | func (f *FST) GetMaxKey() ([]byte, error) {
function new (line 36) | func new(data []byte, f io.Closer) (rv *FST, err error) {
type addrStack (line 230) | type addrStack
method Pop (line 232) | func (a addrStack) Pop() (addrStack, int) {
type Reader (line 293) | type Reader struct
method Get (line 298) | func (r *Reader) Get(input []byte) (uint64, bool, error) {
FILE: fst_iterator.go
type Iterator (line 22) | type Iterator interface
type FSTIterator (line 50) | type FSTIterator struct
method Reset (line 79) | func (i *FSTIterator) Reset(f *FST,
method pointTo (line 94) | func (i *FSTIterator) pointTo(key []byte) error {
method Current (line 168) | func (i *FSTIterator) Current() ([]byte, uint64) {
method Next (line 184) | func (i *FSTIterator) Next() error {
method next (line 188) | func (i *FSTIterator) next(lastOffset int) error {
method Seek (line 294) | func (i *FSTIterator) Seek(key []byte) error {
method Close (line 299) | func (i *FSTIterator) Close() error {
function newIterator (line 66) | func newIterator(f *FST, startKeyInclusive, endKeyExclusive []byte,
FILE: fst_iterator_test.go
function TestIterator (line 26) | func TestIterator(t *testing.T) {
function TestIteratorReset (line 63) | func TestIteratorReset(t *testing.T) {
function TestIteratorStartKey (line 133) | func TestIteratorStartKey(t *testing.T) {
function TestIteratorEndKey (line 226) | func TestIteratorEndKey(t *testing.T) {
function TestIteratorSeek (line 299) | func TestIteratorSeek(t *testing.T) {
function TestIteratorSeekOutsideBoundaries (line 410) | func TestIteratorSeekOutsideBoundaries(t *testing.T) {
function BenchmarkFSTIteratorAllInMem (line 487) | func BenchmarkFSTIteratorAllInMem(b *testing.B) {
function TestFuzzySearch (line 531) | func TestFuzzySearch(t *testing.T) {
function TestRegexpSearch (line 582) | func TestRegexpSearch(t *testing.T) {
function TestIssue32 (line 671) | func TestIssue32(t *testing.T) {
FILE: levenshtein/alphabet.go
type FullCharacteristicVector (line 23) | type FullCharacteristicVector
method shiftAndMask (line 25) | func (fcv FullCharacteristicVector) shiftAndMask(offset, mask uint32) ...
type tuple (line 36) | type tuple struct
type sortRunes (line 41) | type sortRunes
method Less (line 43) | func (s sortRunes) Less(i, j int) bool {
method Swap (line 47) | func (s sortRunes) Swap(i, j int) {
method Len (line 51) | func (s sortRunes) Len() int {
function sortRune (line 55) | func sortRune(r []rune) []rune {
type Alphabet (line 60) | type Alphabet struct
method resetNext (line 65) | func (a *Alphabet) resetNext() {
method next (line 69) | func (a *Alphabet) next() (rune, FullCharacteristicVector, error) {
function dedupe (line 79) | func dedupe(in string) string {
function queryChars (line 93) | func queryChars(qChars string) Alphabet {
FILE: levenshtein/alphabet_test.go
function TestAlphabet (line 19) | func TestAlphabet(t *testing.T) {
function TestFullCharacteristic (line 60) | func TestFullCharacteristic(t *testing.T) {
function TestLongCharacteristic (line 72) | func TestLongCharacteristic(t *testing.T) {
FILE: levenshtein/benchmark_test.go
function BenchmarkNewEvalEditDistance1 (line 21) | func BenchmarkNewEvalEditDistance1(b *testing.B) {
function BenchmarkNewEvalEditDistance2 (line 35) | func BenchmarkNewEvalEditDistance2(b *testing.B) {
function BenchmarkNewEditDistance1 (line 48) | func BenchmarkNewEditDistance1(b *testing.B) {
function BenchmarkNewEditDistance2 (line 67) | func BenchmarkNewEditDistance2(b *testing.B) {
FILE: levenshtein/dfa.go
constant SinkState (line 22) | SinkState = uint32(0)
type DFA (line 24) | type DFA struct
method initialState (line 32) | func (d *DFA) initialState() int {
method distance (line 38) | func (d *DFA) distance(stateId int) Distance {
method numStates (line 43) | func (d *DFA) numStates() int {
method transition (line 48) | func (d *DFA) transition(fromState int, b uint8) int {
method eval (line 52) | func (d *DFA) eval(bytes []uint8) Distance {
method Start (line 62) | func (d *DFA) Start() int {
method IsMatch (line 66) | func (d *DFA) IsMatch(state int) bool {
method CanMatch (line 73) | func (d *DFA) CanMatch(state int) bool {
method Accept (line 77) | func (d *DFA) Accept(state int, b byte) int {
method WillAlwaysMatch (line 83) | func (d *DFA) WillAlwaysMatch(state int) bool {
function fill (line 87) | func fill(dest []uint32, val uint32) {
function fillTransitions (line 93) | func fillTransitions(dest *[256]uint32, val uint32) {
type Utf8DFAStateBuilder (line 99) | type Utf8DFAStateBuilder struct
method addTransitionID (line 105) | func (sb *Utf8DFAStateBuilder) addTransitionID(fromStateID uint32, b u...
method addTransition (line 110) | func (sb *Utf8DFAStateBuilder) addTransition(in rune, toStateID uint32) {
type Utf8StateId (line 134) | type Utf8StateId
function original (line 136) | func original(stateId uint32) Utf8StateId {
function predecessor (line 140) | func predecessor(stateId uint32, numSteps uint8) Utf8StateId {
type Utf8DFABuilder (line 147) | type Utf8DFABuilder struct
method allocate (line 171) | func (dfab *Utf8DFABuilder) allocate() uint32 {
method getOrAllocate (line 181) | func (dfab *Utf8DFABuilder) getOrAllocate(state Utf8StateId) uint32 {
method setInitialState (line 197) | func (dfab *Utf8DFABuilder) setInitialState(iState uint32) {
method build (line 202) | func (dfab *Utf8DFABuilder) build(ed uint8) *DFA {
method addState (line 211) | func (dfab *Utf8DFABuilder) addState(state, default_suc_orig uint32,
function withMaxStates (line 156) | func withMaxStates(maxStates uint32) *Utf8DFABuilder {
FILE: levenshtein/dfa_test.go
function TestLevenshtein2 (line 21) | func TestLevenshtein2(t *testing.T) {
FILE: levenshtein/levenshtein.go
constant StateLimit (line 20) | StateLimit = 10000
type LevenshteinAutomatonBuilder (line 29) | type LevenshteinAutomatonBuilder struct
method BuildDfa (line 55) | func (lab *LevenshteinAutomatonBuilder) BuildDfa(query string,
method MaxDistance (line 62) | func (lab *LevenshteinAutomatonBuilder) MaxDistance() uint8 {
function NewLevenshteinAutomatonBuilder (line 41) | func NewLevenshteinAutomatonBuilder(maxDistance uint8,
FILE: levenshtein/levenshtein_nfa.go
type Distance (line 29) | type Distance interface
type Exact (line 33) | type Exact struct
method distance (line 37) | func (e Exact) distance() uint8 {
type Atleast (line 41) | type Atleast struct
method distance (line 45) | func (a Atleast) distance() uint8 {
function characteristicVector (line 49) | func characteristicVector(query []rune, c rune) uint64 {
type NFAState (line 59) | type NFAState struct
method imply (line 87) | func (ns *NFAState) imply(other NFAState) bool {
type NFAStates (line 65) | type NFAStates
method Len (line 67) | func (ns NFAStates) Len() int {
method Less (line 71) | func (ns NFAStates) Less(i, j int) bool {
method Swap (line 83) | func (ns NFAStates) Swap(i, j int) {
type MultiState (line 105) | type MultiState struct
method States (line 109) | func (ms *MultiState) States() []NFAState {
method Clear (line 113) | func (ms *MultiState) Clear() {
method normalize (line 121) | func (ms *MultiState) normalize() uint32 {
method addStates (line 142) | func (ms *MultiState) addStates(nState NFAState) {
function newMultiState (line 117) | func newMultiState() *MultiState {
function extractBit (line 162) | func extractBit(bitset uint64, pos uint8) bool {
function dist (line 168) | func dist(left, right uint32) uint32 {
type LevenshteinNFA (line 175) | type LevenshteinNFA struct
method maxDistance (line 186) | func (la *LevenshteinNFA) maxDistance() uint8 {
method msDiameter (line 190) | func (la *LevenshteinNFA) msDiameter() uint8 {
method initialStates (line 194) | func (la *LevenshteinNFA) initialStates() *MultiState {
method multistateDistance (line 201) | func (la *LevenshteinNFA) multistateDistance(ms *MultiState,
method simpleTransition (line 220) | func (la *LevenshteinNFA) simpleTransition(state NFAState,
method transition (line 267) | func (la *LevenshteinNFA) transition(cState *MultiState,
method computeDistance (line 280) | func (la *LevenshteinNFA) computeDistance(query, other []rune) Distance {
function newLevenshtein (line 180) | func newLevenshtein(maxD uint8, transposition bool) *LevenshteinNFA {
FILE: levenshtein/levenshtein_test.go
function TestLevenshtein (line 21) | func TestLevenshtein(t *testing.T) {
function makeDistance (line 263) | func makeDistance(d uint8, md uint8) Distance {
function testLevenshteinNfaUtil (line 270) | func testLevenshteinNfaUtil(left, right string, ed uint8, t *testing.T) {
function testSymmetric (line 278) | func testSymmetric(lev *LevenshteinNFA, left, right string, expected Dis...
function TestLevenshteinNfa (line 289) | func TestLevenshteinNfa(t *testing.T) {
function TestLevenshteinParametricDfa (line 315) | func TestLevenshteinParametricDfa(t *testing.T) {
function TestDamerau (line 387) | func TestDamerau(t *testing.T) {
function TestLevenshteinDfa (line 395) | func TestLevenshteinDfa(t *testing.T) {
function TestUtf8Simple (line 412) | func TestUtf8Simple(t *testing.T) {
function TestSimple (line 430) | func TestSimple(t *testing.T) {
function TestJapanese (line 462) | func TestJapanese(t *testing.T) {
function TestJapaneseEnglish (line 496) | func TestJapaneseEnglish(t *testing.T) {
function TestTooManyStatesError (line 520) | func TestTooManyStatesError(t *testing.T) {
FILE: levenshtein/parametric_dfa.go
type ParametricState (line 24) | type ParametricState struct
method isDeadEnd (line 33) | func (ps *ParametricState) isDeadEnd() bool {
function newParametricState (line 29) | func newParametricState() ParametricState {
type Transition (line 37) | type Transition struct
method apply (line 42) | func (t *Transition) apply(state ParametricState) ParametricState {
type ParametricStateIndex (line 54) | type ParametricStateIndex struct
method numStates (line 79) | func (psi *ParametricStateIndex) numStates() int {
method maxNumStates (line 83) | func (psi *ParametricStateIndex) maxNumStates() int {
method get (line 87) | func (psi *ParametricStateIndex) get(stateID uint32) ParametricState {
method getOrAllocate (line 91) | func (psi *ParametricStateIndex) getOrAllocate(ps ParametricState) uin...
function newParametricStateIndex (line 60) | func newParametricStateIndex(queryLen,
type ParametricDFA (line 104) | type ParametricDFA struct
method initialState (line 112) | func (pdfa *ParametricDFA) initialState() ParametricState {
method isPrefixSink (line 118) | func (pdfa *ParametricDFA) isPrefixSink(state ParametricState, queryLe...
method numStates (line 141) | func (pdfa *ParametricDFA) numStates() int {
method transition (line 152) | func (pdfa *ParametricDFA) transition(state ParametricState,
method getDistance (line 157) | func (pdfa *ParametricDFA) getDistance(state ParametricState,
method computeDistance (line 170) | func (pdfa *ParametricDFA) computeDistance(left, right string) Distance {
method buildDfa (line 186) | func (pdfa *ParametricDFA) buildDfa(query string, distance uint8,
function min (line 145) | func min(x, y uint32) uint32 {
function fromNfa (line 248) | func fromNfa(nfa *LevenshteinNFA) (*ParametricDFA, error) {
type hash (line 313) | type hash struct
method getOrAllocate (line 325) | func (h *hash) getOrAllocate(m MultiState) int {
method getFromID (line 338) | func (h *hash) getFromID(id int) *MultiState {
function newHash (line 318) | func newHash() *hash {
function getHash (line 342) | func getHash(ms *MultiState) [16]byte {
FILE: merge_iterator.go
type MergeFunc (line 26) | type MergeFunc
type MergeIterator (line 32) | type MergeIterator struct
method init (line 63) | func (m *MergeIterator) init() {
method updateMatches (line 70) | func (m *MergeIterator) updateMatches() {
method Current (line 106) | func (m *MergeIterator) Current() ([]byte, uint64) {
method Next (line 112) | func (m *MergeIterator) Next() error {
method Seek (line 131) | func (m *MergeIterator) Seek(key []byte) error {
method Close (line 147) | func (m *MergeIterator) Close() error {
function NewMergeIterator (line 47) | func NewMergeIterator(itrs []Iterator, f MergeFunc) (*MergeIterator, err...
function MergeMin (line 160) | func MergeMin(vals []uint64) uint64 {
function MergeMax (line 171) | func MergeMax(vals []uint64) uint64 {
function MergeSum (line 182) | func MergeSum(vals []uint64) uint64 {
FILE: merge_iterator_test.go
function TestMergeIterator (line 23) | func TestMergeIterator(t *testing.T) {
type testIterator (line 150) | type testIterator struct
method Current (line 170) | func (m *testIterator) Current() ([]byte, uint64) {
method Next (line 177) | func (m *testIterator) Next() error {
method Seek (line 185) | func (m *testIterator) Seek(key []byte) error {
method Reset (line 193) | func (m *testIterator) Reset(f *FST, startKeyInclusive, endKeyExclusiv...
method Close (line 197) | func (m *testIterator) Close() error {
method Exists (line 201) | func (m *testIterator) Exists(key []byte) (bool, error) {
function newTestIterator (line 156) | func newTestIterator(in map[string]uint64) (*testIterator, error) {
function TestMergeFunc (line 205) | func TestMergeFunc(t *testing.T) {
function TestEmptyMergeIterator (line 242) | func TestEmptyMergeIterator(t *testing.T) {
FILE: pack.go
function deltaAddr (line 17) | func deltaAddr(base, trans uint64) uint64 {
constant packOutMask (line 25) | packOutMask = 1<<4 - 1
function encodePackSize (line 27) | func encodePackSize(transSize, outSize int) byte {
function decodePackSize (line 34) | func decodePackSize(pack byte) (transSize int, packSize int) {
constant maxNumTrans (line 40) | maxNumTrans = 1<<6 - 1
function encodeNumTrans (line 42) | func encodeNumTrans(n int) byte {
function readPackedUint (line 49) | func readPackedUint(data []byte) (rv uint64) {
FILE: pack_test.go
function TestEncodeDecodePackSize (line 22) | func TestEncodeDecodePackSize(t *testing.T) {
function TestEncodeNumTrans (line 35) | func TestEncodeNumTrans(t *testing.T) {
FILE: regexp/compile.go
type compiler (line 26) | type compiler struct
method compile (line 45) | func (c *compiler) compile(ast *syntax.Regexp) (prog, error) {
method c (line 56) | func (c *compiler) c(ast *syntax.Regexp) (err error) {
method checkSize (line 229) | func (c *compiler) checkSize() error {
method compileClass (line 236) | func (c *compiler) compileClass(ast *syntax.Regexp) error {
method compileClassRange (line 270) | func (c *compiler) compileClassRange(startR, endR rune) (err error) {
method compileUtf8Ranges (line 297) | func (c *compiler) compileUtf8Ranges(seq utf8.Sequence) {
method emptySplit (line 307) | func (c *compiler) emptySplit() uint {
method emptyJump (line 314) | func (c *compiler) emptyJump() uint {
method setSplit (line 321) | func (c *compiler) setSplit(i, pc1, pc2 uint) {
method setJump (line 327) | func (c *compiler) setJump(i, pc uint) {
method top (line 332) | func (c *compiler) top() uint {
method allocInst (line 336) | func (c *compiler) allocInst() *inst {
function newCompiler (line 37) | func newCompiler(sizeLimit uint) *compiler {
FILE: regexp/compile_test.go
function TestCompiler (line 23) | func TestCompiler(t *testing.T) {
FILE: regexp/dfa.go
constant StateLimit (line 23) | StateLimit = 10000
type dfaBuilder (line 30) | type dfaBuilder struct
method build (line 52) | func (d *dfaBuilder) build() (*dfa, error) {
method runState (line 81) | func (d *dfaBuilder) runState(cur, next *sparseSet, state int, b byte,...
method cachedState (line 106) | func (d *dfaBuilder) cachedState(set *sparseSet,
function newDfaBuilder (line 36) | func newDfaBuilder(insts prog) *dfaBuilder {
function instsKey (line 94) | func instsKey(insts []uint, buf []byte) []byte {
type dfa (line 141) | type dfa struct
method add (line 146) | func (d *dfa) add(set *sparseSet, ip uint) {
method run (line 160) | func (d *dfa) run(from, to *sparseSet, b byte) bool {
type state (line 178) | type state struct
type intStack (line 184) | type intStack
method Push (line 186) | func (s intStack) Push(v int) intStack {
method Pop (line 190) | func (s intStack) Pop() (intStack, int) {
FILE: regexp/inst.go
type instOp (line 20) | type instOp
constant OpMatch (line 24) | OpMatch instOp = iota
constant OpJmp (line 25) | OpJmp
constant OpSplit (line 26) | OpSplit
constant OpRange (line 27) | OpRange
constant instSize (line 31) | instSize = 40
type inst (line 33) | type inst struct
method String (line 42) | func (i *inst) String() string {
type prog (line 54) | type prog
method String (line 56) | func (p prog) String() string {
FILE: regexp/regexp.go
type Regexp (line 42) | type Regexp struct
method Start (line 85) | func (r *Regexp) Start() int {
method IsMatch (line 90) | func (r *Regexp) IsMatch(s int) bool {
method CanMatch (line 99) | func (r *Regexp) CanMatch(s int) bool {
method WillAlwaysMatch (line 108) | func (r *Regexp) WillAlwaysMatch(int) bool {
method Accept (line 114) | func (r *Regexp) Accept(s int, b byte) int {
function New (line 51) | func New(expr string) (*Regexp, error) {
function NewWithLimit (line 59) | func NewWithLimit(expr string, size uint) (*Regexp, error) {
function NewParsedWithLimit (line 67) | func NewParsedWithLimit(expr string, parsed *syntax.Regexp, size uint) (...
FILE: regexp/regexp_test.go
function TestRegexp (line 22) | func TestRegexp(t *testing.T) {
function BenchmarkNewWildcard (line 248) | func BenchmarkNewWildcard(b *testing.B) {
FILE: regexp/sparse.go
type sparseSet (line 17) | type sparseSet struct
method Len (line 31) | func (s *sparseSet) Len() int {
method Add (line 35) | func (s *sparseSet) Add(ip uint) uint {
method Get (line 43) | func (s *sparseSet) Get(i uint) uint {
method Contains (line 47) | func (s *sparseSet) Contains(ip uint) bool {
method Clear (line 52) | func (s *sparseSet) Clear() {
function newSparseSet (line 23) | func newSparseSet(size uint) *sparseSet {
FILE: regexp/sparse_test.go
function TestSparse (line 19) | func TestSparse(t *testing.T) {
FILE: registry.go
type registryCell (line 17) | type registryCell struct
type registry (line 22) | type registry struct
method Reset (line 40) | func (r *registry) Reset() {
method entry (line 48) | func (r *registry) entry(node *builderNode) (bool, int, *registryCell) {
method hash (line 61) | func (r *registry) hash(b *builderNode) int {
function newRegistry (line 29) | func newRegistry(p *builderNodePool, tableSize, mruSize int) *registry {
constant fnvPrime (line 59) | fnvPrime = 1099511628211
type registryCache (line 78) | type registryCache
method entry (line 80) | func (r registryCache) entry(node *builderNode, pool *builderNodePool)...
method promote (line 105) | func (r registryCache) promote(i int) {
method swap (line 112) | func (r registryCache) swap(i, j int) {
FILE: registry_test.go
function TestRegistry (line 21) | func TestRegistry(t *testing.T) {
FILE: transducer.go
type Transducer (line 18) | type Transducer interface
function TransducerGet (line 37) | func TransducerGet(t Transducer, k []byte) (bool, uint64) {
FILE: utf8/utf8.go
type Sequences (line 23) | type Sequences
function NewSequences (line 27) | func NewSequences(start, end rune) (Sequences, error) {
function NewSequencesPrealloc (line 32) | func NewSequencesPrealloc(start, end rune,
type Sequence (line 115) | type Sequence
method Matches (line 147) | func (u Sequence) Matches(bytes []byte) bool {
method String (line 159) | func (u Sequence) String() string {
function SequenceFromEncodedRange (line 118) | func SequenceFromEncodedRange(start, end []byte) (Sequence, error) {
type Range (line 175) | type Range struct
method matches (line 182) | func (u Range) matches(b byte) bool {
method String (line 189) | func (u Range) String() string {
type scalarRange (line 196) | type scalarRange struct
method String (line 203) | func (s *scalarRange) String() string {
method split (line 208) | func (s *scalarRange) split() (scalarRange, scalarRange) {
method valid (line 222) | func (s *scalarRange) valid() bool {
method ascii (line 226) | func (s *scalarRange) ascii() Range {
method encode (line 237) | func (s *scalarRange) encode(start, end []byte) (int, int) {
type RangeStack (line 243) | type RangeStack
method Push (line 245) | func (s RangeStack) Push(v scalarRange) RangeStack {
method Pop (line 249) | func (s RangeStack) Pop() (RangeStack, scalarRange) {
function maxScalarValue (line 257) | func maxScalarValue(nbytes int) rune {
FILE: utf8/utf8_test.go
function TestUtf8Sequences (line 24) | func TestUtf8Sequences(t *testing.T) {
function TestCodepointsNoSurrogates (line 65) | func TestCodepointsNoSurrogates(t *testing.T) {
function neverAcceptsSurrogateCodepoints (line 73) | func neverAcceptsSurrogateCodepoints(start, end rune) error {
FILE: vellum.go
type BuilderOpts (line 56) | type BuilderOpts struct
function New (line 64) | func New(w io.Writer, opts *BuilderOpts) (*Builder, error) {
function Open (line 69) | func Open(path string) (*FST, error) {
function Load (line 74) | func Load(data []byte) (*FST, error) {
function Merge (line 80) | func Merge(w io.Writer, opts *BuilderOpts, itrs []Iterator, f MergeFunc)...
FILE: vellum_mmap.go
type mmapWrapper (line 25) | type mmapWrapper struct
method Close (line 30) | func (m *mmapWrapper) Close() (err error) {
function open (line 45) | func open(path string) (*FST, error) {
FILE: vellum_nommap.go
function open (line 21) | func open(path string) (*FST, error) {
FILE: vellum_test.go
function TestRoundTripSimple (line 25) | func TestRoundTripSimple(t *testing.T) {
function TestRoundTripThousand (line 151) | func TestRoundTripThousand(t *testing.T) {
function TestRoundTripEmpty (line 226) | func TestRoundTripEmpty(t *testing.T) {
function TestRoundTripEmptyString (line 283) | func TestRoundTripEmptyString(t *testing.T) {
function TestRoundTripEmptyStringAndOthers (line 348) | func TestRoundTripEmptyStringAndOthers(t *testing.T) {
function TestMerge (line 418) | func TestMerge(t *testing.T) {
function BenchmarkKey4000K (line 583) | func BenchmarkKey4000K(b *testing.B) {
function BenchmarkKey1000K (line 587) | func BenchmarkKey1000K(b *testing.B) {
function BenchmarkKey100K (line 591) | func BenchmarkKey100K(b *testing.B) {
function BenchmarkKey10K (line 595) | func BenchmarkKey10K(b *testing.B) {
function BenchmarkKey1K (line 599) | func BenchmarkKey1K(b *testing.B) {
function benchmarkBigKey (line 603) | func benchmarkBigKey(b *testing.B, n int) {
function TestMaxWithSubstring (line 626) | func TestMaxWithSubstring(t *testing.T) {
FILE: vendor/github.com/edsrzf/mmap-go/mmap.go
constant RDONLY (line 27) | RDONLY = 0
constant RDWR (line 30) | RDWR = 1 << iota
constant COPY (line 33) | COPY
constant EXEC (line 35) | EXEC
constant ANON (line 40) | ANON = 1 << iota
type MMap (line 44) | type MMap
method header (line 76) | func (m *MMap) header() *reflect.SliceHeader {
method Lock (line 82) | func (m MMap) Lock() error {
method Unlock (line 90) | func (m MMap) Unlock() error {
method Flush (line 96) | func (m MMap) Flush() error {
method Unmap (line 107) | func (m *MMap) Unmap() error {
function Map (line 48) | func Map(f *os.File, prot, flags int) (MMap, error) {
function MapRegion (line 56) | func MapRegion(f *os.File, length int, prot, flags int, offset int64) (M...
FILE: vendor/github.com/edsrzf/mmap-go/mmap_unix.go
function mmap (line 13) | func mmap(len int, inprot, inflags, fd uintptr, off int64) ([]byte, erro...
function flush (line 37) | func flush(addr, len uintptr) error {
function lock (line 45) | func lock(addr, len uintptr) error {
function unlock (line 53) | func unlock(addr, len uintptr) error {
function unmap (line 61) | func unmap(addr, len uintptr) error {
FILE: vendor/github.com/edsrzf/mmap-go/mmap_windows.go
function mmap (line 25) | func mmap(len int, prot, flags, hfile uintptr, off int64) ([]byte, error) {
function flush (line 74) | func flush(addr, len uintptr) error {
function lock (line 92) | func lock(addr, len uintptr) error {
function unlock (line 97) | func unlock(addr, len uintptr) error {
function unmap (line 102) | func unmap(addr, len uintptr) error {
FILE: vendor/github.com/edsrzf/mmap-go/msync_netbsd.go
constant _SYS_MSYNC (line 7) | _SYS_MSYNC = 277
constant _MS_SYNC (line 8) | _MS_SYNC = 0x04
FILE: vendor/github.com/edsrzf/mmap-go/msync_unix.go
constant _SYS_MSYNC (line 13) | _SYS_MSYNC = syscall.SYS_MSYNC
constant _MS_SYNC (line 14) | _MS_SYNC = syscall.MS_SYNC
FILE: vendor/github.com/inconshreveable/mousetrap/trap_others.go
function StartedByExplorer (line 13) | func StartedByExplorer() bool {
FILE: vendor/github.com/inconshreveable/mousetrap/trap_windows.go
constant th32cs_snapprocess (line 15) | th32cs_snapprocess uintptr = 0x2
type processEntry32 (line 26) | type processEntry32 struct
function getProcessEntry (line 39) | func getProcessEntry(pid int) (pe *processEntry32, err error) {
function getppid (line 69) | func getppid() (pid int, err error) {
function StartedByExplorer (line 85) | func StartedByExplorer() bool {
FILE: vendor/github.com/inconshreveable/mousetrap/trap_windows_1.4.go
function getProcessEntry (line 12) | func getProcessEntry(pid int) (*syscall.ProcessEntry32, error) {
function StartedByExplorer (line 40) | func StartedByExplorer() bool {
FILE: vendor/github.com/spf13/cobra/bash_completions.go
constant BashCompFilenameExt (line 15) | BashCompFilenameExt = "cobra_annotation_bash_completion_filename_ext...
constant BashCompCustom (line 16) | BashCompCustom = "cobra_annotation_bash_completion_custom"
constant BashCompOneRequiredFlag (line 17) | BashCompOneRequiredFlag = "cobra_annotation_bash_completion_one_required...
constant BashCompSubdirsInDir (line 18) | BashCompSubdirsInDir = "cobra_annotation_bash_completion_subdirs_in_dir"
function preamble (line 21) | func preamble(out io.Writer, name string) error {
function postscript (line 255) | func postscript(w io.Writer, name string) error {
function writeCommands (line 303) | func writeCommands(cmd *Command, w io.Writer) error {
function writeFlagHandler (line 319) | func writeFlagHandler(name string, annotations map[string][]string, w io...
function writeShortFlag (line 370) | func writeShortFlag(flag *pflag.Flag, w io.Writer) error {
function writeFlag (line 384) | func writeFlag(flag *pflag.Flag, w io.Writer) error {
function writeLocalNonPersistentFlag (line 398) | func writeLocalNonPersistentFlag(flag *pflag.Flag, w io.Writer) error {
function writeFlags (line 410) | func writeFlags(cmd *Command, w io.Writer) error {
function writeRequiredFlag (line 470) | func writeRequiredFlag(cmd *Command, w io.Writer) error {
function writeRequiredNouns (line 506) | func writeRequiredNouns(cmd *Command, w io.Writer) error {
function writeArgAliases (line 519) | func writeArgAliases(cmd *Command, w io.Writer) error {
function gen (line 532) | func gen(cmd *Command, w io.Writer) error {
method GenBashCompletion (line 572) | func (cmd *Command) GenBashCompletion(w io.Writer) error {
function nonCompletableFlag (line 587) | func nonCompletableFlag(flag *pflag.Flag) bool {
method GenBashCompletionFile (line 592) | func (cmd *Command) GenBashCompletionFile(filename string) error {
method MarkFlagRequired (line 603) | func (cmd *Command) MarkFlagRequired(name string) error {
method MarkPersistentFlagRequired (line 608) | func (cmd *Command) MarkPersistentFlagRequired(name string) error {
function MarkFlagRequired (line 613) | func MarkFlagRequired(flags *pflag.FlagSet, name string) error {
method MarkFlagFilename (line 619) | func (cmd *Command) MarkFlagFilename(name string, extensions ...string) ...
method MarkFlagCustom (line 625) | func (cmd *Command) MarkFlagCustom(name string, f string) error {
method MarkPersistentFlagFilename (line 631) | func (cmd *Command) MarkPersistentFlagFilename(name string, extensions ....
function MarkFlagFilename (line 637) | func MarkFlagFilename(flags *pflag.FlagSet, name string, extensions ...s...
function MarkFlagCustom (line 643) | func MarkFlagCustom(flags *pflag.FlagSet, name string, f string) error {
FILE: vendor/github.com/spf13/cobra/cobra.go
function AddTemplateFunc (line 51) | func AddTemplateFunc(name string, tmplFunc interface{}) {
function AddTemplateFuncs (line 57) | func AddTemplateFuncs(tmplFuncs template.FuncMap) {
function OnInitialize (line 64) | func OnInitialize(y ...func()) {
function Gt (line 71) | func Gt(a interface{}, b interface{}) bool {
function Eq (line 99) | func Eq(a interface{}, b interface{}) bool {
function trimRightSpace (line 114) | func trimRightSpace(s string) string {
function appendIfNotPresent (line 119) | func appendIfNotPresent(s, stringToAppend string) string {
function rpad (line 127) | func rpad(s string, padding int) string {
function tmpl (line 133) | func tmpl(w io.Writer, text string, data interface{}) error {
function ld (line 141) | func ld(s, t string, ignoreCase bool) int {
FILE: vendor/github.com/spf13/cobra/command.go
type Command (line 34) | type Command struct
method SetArgs (line 137) | func (c *Command) SetArgs(a []string) {
method SetOutput (line 143) | func (c *Command) SetOutput(output io.Writer) {
method SetUsageFunc (line 148) | func (c *Command) SetUsageFunc(f func(*Command) error) {
method SetUsageTemplate (line 153) | func (c *Command) SetUsageTemplate(s string) {
method SetFlagErrorFunc (line 159) | func (c *Command) SetFlagErrorFunc(f func(*Command, error) error) {
method SetHelpFunc (line 164) | func (c *Command) SetHelpFunc(f func(*Command, []string)) {
method SetHelpCommand (line 169) | func (c *Command) SetHelpCommand(cmd *Command) {
method SetHelpTemplate (line 174) | func (c *Command) SetHelpTemplate(s string) {
method SetGlobalNormalizationFunc (line 180) | func (c *Command) SetGlobalNormalizationFunc(n func(f *flag.FlagSet, n...
method OutOrStdout (line 191) | func (c *Command) OutOrStdout() io.Writer {
method OutOrStderr (line 196) | func (c *Command) OutOrStderr() io.Writer {
method getOut (line 200) | func (c *Command) getOut(def io.Writer) io.Writer {
method UsageFunc (line 212) | func (c *Command) UsageFunc() (f func(*Command) error) {
method Usage (line 233) | func (c *Command) Usage() error {
method HelpFunc (line 239) | func (c *Command) HelpFunc() func(*Command, []string) {
method checkHelpFunc (line 253) | func (c *Command) checkHelpFunc() func(*Command, []string) {
method Help (line 269) | func (c *Command) Help() error {
method UsageString (line 275) | func (c *Command) UsageString() string {
method FlagErrorFunc (line 287) | func (c *Command) FlagErrorFunc() (f func(*Command, error) error) {
method UsagePadding (line 303) | func (c *Command) UsagePadding() int {
method CommandPathPadding (line 313) | func (c *Command) CommandPathPadding() int {
method NamePadding (line 323) | func (c *Command) NamePadding() int {
method UsageTemplate (line 331) | func (c *Command) UsageTemplate() string {
method HelpTemplate (line 367) | func (c *Command) HelpTemplate() string {
method resetChildrensParents (line 381) | func (c *Command) resetChildrensParents() {
method Find (line 461) | func (c *Command) Find(args []string) (*Command, []string, error) {
method SuggestionsFor (line 528) | func (c *Command) SuggestionsFor(typedName string) []string {
method VisitParents (line 549) | func (c *Command) VisitParents(fn func(*Command)) {
method Root (line 565) | func (c *Command) Root() *Command {
method ArgsLenAtDash (line 582) | func (c *Command) ArgsLenAtDash() int {
method execute (line 586) | func (c *Command) execute(a []string) (err error) {
method preRun (line 672) | func (c *Command) preRun() {
method errorMsgFromParse (line 678) | func (c *Command) errorMsgFromParse() string {
method Execute (line 692) | func (c *Command) Execute() error {
method ExecuteC (line 698) | func (c *Command) ExecuteC() (cmd *Command, err error) {
method initHelpFlag (line 759) | func (c *Command) initHelpFlag() {
method initHelpCmd (line 766) | func (c *Command) initHelpCmd() {
method ResetCommands (line 795) | func (c *Command) ResetCommands() {
method Commands (line 808) | func (c *Command) Commands() []*Command {
method AddCommand (line 818) | func (c *Command) AddCommand(cmds ...*Command) {
method RemoveCommand (line 847) | func (c *Command) RemoveCommand(cmds ...*Command) {
method Print (line 881) | func (c *Command) Print(i ...interface{}) {
method Println (line 886) | func (c *Command) Println(i ...interface{}) {
method Printf (line 892) | func (c *Command) Printf(format string, i ...interface{}) {
method CommandPath (line 898) | func (c *Command) CommandPath() string {
method UseLine (line 909) | func (c *Command) UseLine() string {
method DebugFlags (line 919) | func (c *Command) DebugFlags() {
method Name (line 963) | func (c *Command) Name() string {
method HasAlias (line 977) | func (c *Command) HasAlias(s string) bool {
method NameAndAliases (line 987) | func (c *Command) NameAndAliases() string {
method HasExample (line 992) | func (c *Command) HasExample() bool {
method Runnable (line 997) | func (c *Command) Runnable() bool {
method HasSubCommands (line 1002) | func (c *Command) HasSubCommands() bool {
method IsAvailableCommand (line 1008) | func (c *Command) IsAvailableCommand() bool {
method IsAdditionalHelpTopicCommand (line 1029) | func (c *Command) IsAdditionalHelpTopicCommand() bool {
method HasHelpSubCommands (line 1049) | func (c *Command) HasHelpSubCommands() bool {
method HasAvailableSubCommands (line 1063) | func (c *Command) HasAvailableSubCommands() bool {
method HasParent (line 1078) | func (c *Command) HasParent() bool {
method GlobalNormalizationFunc (line 1083) | func (c *Command) GlobalNormalizationFunc() func(f *flag.FlagSet, name...
method Flags (line 1089) | func (c *Command) Flags() *flag.FlagSet {
method LocalNonPersistentFlags (line 1101) | func (c *Command) LocalNonPersistentFlags() *flag.FlagSet {
method LocalFlags (line 1114) | func (c *Command) LocalFlags() *flag.FlagSet {
method InheritedFlags (line 1132) | func (c *Command) InheritedFlags() *flag.FlagSet {
method NonInheritedFlags (line 1161) | func (c *Command) NonInheritedFlags() *flag.FlagSet {
method PersistentFlags (line 1166) | func (c *Command) PersistentFlags() *flag.FlagSet {
method ResetFlags (line 1178) | func (c *Command) ResetFlags() {
method HasFlags (line 1188) | func (c *Command) HasFlags() bool {
method HasPersistentFlags (line 1193) | func (c *Command) HasPersistentFlags() bool {
method HasLocalFlags (line 1198) | func (c *Command) HasLocalFlags() bool {
method HasInheritedFlags (line 1203) | func (c *Command) HasInheritedFlags() bool {
method HasAvailableFlags (line 1209) | func (c *Command) HasAvailableFlags() bool {
method HasAvailablePersistentFlags (line 1214) | func (c *Command) HasAvailablePersistentFlags() bool {
method HasAvailableLocalFlags (line 1220) | func (c *Command) HasAvailableLocalFlags() bool {
method HasAvailableInheritedFlags (line 1226) | func (c *Command) HasAvailableInheritedFlags() bool {
method Flag (line 1231) | func (c *Command) Flag(name string) (flag *flag.Flag) {
method persistentFlag (line 1242) | func (c *Command) persistentFlag(name string) (flag *flag.Flag) {
method ParseFlags (line 1254) | func (c *Command) ParseFlags(args []string) (err error) {
method Parent (line 1264) | func (c *Command) Parent() *Command {
method mergePersistentFlags (line 1268) | func (c *Command) mergePersistentFlags() {
function hasNoOptDefVal (line 387) | func hasNoOptDefVal(name string, f *flag.FlagSet) bool {
function shortHasNoOptDefVal (line 395) | func shortHasNoOptDefVal(name string, fs *flag.FlagSet) bool {
function stripFlags (line 405) | func stripFlags(args []string, c *Command) []string {
function argsMinusFirstX (line 447) | func argsMinusFirstX(args []string, x string) []string {
type commandSorterByName (line 801) | type commandSorterByName
method Len (line 803) | func (c commandSorterByName) Len() int { return len(c) }
method Swap (line 804) | func (c commandSorterByName) Swap(i, j int) { c[i], c[j] = c[j], ...
method Less (line 805) | func (c commandSorterByName) Less(i, j int) bool { return c[i].Name() ...
FILE: vendor/github.com/spf13/cobra/command_win.go
function preExecHook (line 20) | func preExecHook(c *Command) {
FILE: vendor/github.com/spf13/cobra/doc/man_docs.go
function GenManTree (line 36) | func GenManTree(cmd *cobra.Command, header *GenManHeader, dir string) er...
function GenManTreeFromOpts (line 46) | func GenManTreeFromOpts(cmd *cobra.Command, opts GenManTreeOptions) error {
type GenManTreeOptions (line 80) | type GenManTreeOptions struct
type GenManHeader (line 90) | type GenManHeader struct
function GenMan (line 101) | func GenMan(cmd *cobra.Command, header *GenManHeader, w io.Writer) error {
function fillHeader (line 112) | func fillHeader(header *GenManHeader, name string) {
function manPreamble (line 129) | func manPreamble(out io.Writer, header *GenManHeader, cmd *cobra.Command...
function manPrintFlags (line 147) | func manPrintFlags(out io.Writer, flags *pflag.FlagSet) {
function manPrintOptions (line 175) | func manPrintOptions(out io.Writer, command *cobra.Command) {
function genMan (line 190) | func genMan(cmd *cobra.Command, header *GenManHeader) []byte {
FILE: vendor/github.com/spf13/cobra/doc/md_docs.go
function printOptions (line 28) | func printOptions(w io.Writer, cmd *cobra.Command, name string) error {
function GenMarkdown (line 55) | func GenMarkdown(cmd *cobra.Command, w io.Writer) error {
function GenMarkdownCustom (line 59) | func GenMarkdownCustom(cmd *cobra.Command, w io.Writer, linkHandler func...
function GenMarkdownTree (line 144) | func GenMarkdownTree(cmd *cobra.Command, dir string) error {
function GenMarkdownTreeCustom (line 150) | func GenMarkdownTreeCustom(cmd *cobra.Command, dir string, filePrepender...
FILE: vendor/github.com/spf13/cobra/doc/util.go
function hasSeeAlso (line 25) | func hasSeeAlso(cmd *cobra.Command) bool {
function forceMultiLine (line 40) | func forceMultiLine(s string) string {
type byName (line 47) | type byName
method Len (line 49) | func (s byName) Len() int { return len(s) }
method Swap (line 50) | func (s byName) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
method Less (line 51) | func (s byName) Less(i, j int) bool { return s[i].Name() < s[j].Name() }
FILE: vendor/github.com/spf13/cobra/doc/yaml_docs.go
type cmdOption (line 29) | type cmdOption struct
type cmdDoc (line 36) | type cmdDoc struct
function GenYamlTree (line 51) | func GenYamlTree(cmd *cobra.Command, dir string) error {
function GenYamlTreeCustom (line 58) | func GenYamlTreeCustom(cmd *cobra.Command, dir string, filePrepender, li...
function GenYaml (line 86) | func GenYaml(cmd *cobra.Command, w io.Writer) error {
function GenYamlCustom (line 91) | func GenYamlCustom(cmd *cobra.Command, w io.Writer, linkHandler func(str...
function genFlagResult (line 139) | func genFlagResult(flags *pflag.FlagSet) []cmdOption {
FILE: vendor/github.com/spf13/pflag/bool.go
type boolFlag (line 7) | type boolFlag interface
type boolValue (line 13) | type boolValue
method Set (line 20) | func (b *boolValue) Set(s string) error {
method Type (line 26) | func (b *boolValue) Type() string {
method String (line 30) | func (b *boolValue) String() string { return strconv.FormatBool(bool(*...
method IsBoolFlag (line 32) | func (b *boolValue) IsBoolFlag() bool { return true }
function newBoolValue (line 15) | func newBoolValue(val bool, p *bool) *boolValue {
function boolConv (line 34) | func boolConv(sval string) (interface{}, error) {
method GetBool (line 39) | func (f *FlagSet) GetBool(name string) (bool, error) {
method BoolVar (line 49) | func (f *FlagSet) BoolVar(p *bool, name string, value bool, usage string) {
method BoolVarP (line 54) | func (f *FlagSet) BoolVarP(p *bool, name, shorthand string, value bool, ...
function BoolVar (line 61) | func BoolVar(p *bool, name string, value bool, usage string) {
function BoolVarP (line 66) | func BoolVarP(p *bool, name, shorthand string, value bool, usage string) {
method Bool (line 73) | func (f *FlagSet) Bool(name string, value bool, usage string) *bool {
method BoolP (line 78) | func (f *FlagSet) BoolP(name, shorthand string, value bool, usage string...
function Bool (line 86) | func Bool(name string, value bool, usage string) *bool {
function BoolP (line 91) | func BoolP(name, shorthand string, value bool, usage string) *bool {
FILE: vendor/github.com/spf13/pflag/bool_slice.go
type boolSliceValue (line 10) | type boolSliceValue struct
method Set (line 24) | func (s *boolSliceValue) Set(val string) error {
method Type (line 57) | func (s *boolSliceValue) Type() string {
method String (line 62) | func (s *boolSliceValue) String() string {
function newBoolSliceValue (line 15) | func newBoolSliceValue(val []bool, p *[]bool) *boolSliceValue {
function boolSliceConv (line 74) | func boolSliceConv(val string) (interface{}, error) {
method GetBoolSlice (line 93) | func (f *FlagSet) GetBoolSlice(name string) ([]bool, error) {
method BoolSliceVar (line 103) | func (f *FlagSet) BoolSliceVar(p *[]bool, name string, value []bool, usa...
method BoolSliceVarP (line 108) | func (f *FlagSet) BoolSliceVarP(p *[]bool, name, shorthand string, value...
function BoolSliceVar (line 114) | func BoolSliceVar(p *[]bool, name string, value []bool, usage string) {
function BoolSliceVarP (line 119) | func BoolSliceVarP(p *[]bool, name, shorthand string, value []bool, usag...
method BoolSlice (line 125) | func (f *FlagSet) BoolSlice(name string, value []bool, usage string) *[]...
method BoolSliceP (line 132) | func (f *FlagSet) BoolSliceP(name, shorthand string, value []bool, usage...
function BoolSlice (line 140) | func BoolSlice(name string, value []bool, usage string) *[]bool {
function BoolSliceP (line 145) | func BoolSliceP(name, shorthand string, value []bool, usage string) *[]b...
FILE: vendor/github.com/spf13/pflag/count.go
type countValue (line 6) | type countValue
method Set (line 13) | func (i *countValue) Set(s string) error {
method Type (line 24) | func (i *countValue) Type() string {
method String (line 28) | func (i *countValue) String() string { return strconv.Itoa(int(*i)) }
function newCountValue (line 8) | func newCountValue(val int, p *int) *countValue {
function countConv (line 30) | func countConv(sval string) (interface{}, error) {
method GetCount (line 39) | func (f *FlagSet) GetCount(name string) (int, error) {
method CountVar (line 50) | func (f *FlagSet) CountVar(p *int, name string, usage string) {
method CountVarP (line 55) | func (f *FlagSet) CountVarP(p *int, name, shorthand string, usage string) {
function CountVar (line 61) | func CountVar(p *int, name string, usage string) {
function CountVarP (line 66) | func CountVarP(p *int, name, shorthand string, usage string) {
method Count (line 73) | func (f *FlagSet) Count(name string, usage string) *int {
method CountP (line 80) | func (f *FlagSet) CountP(name, shorthand string, usage string) *int {
function Count (line 87) | func Count(name string, usage string) *int {
function CountP (line 92) | func CountP(name, shorthand string, usage string) *int {
FILE: vendor/github.com/spf13/pflag/duration.go
type durationValue (line 8) | type durationValue
method Set (line 15) | func (d *durationValue) Set(s string) error {
method Type (line 21) | func (d *durationValue) Type() string {
method String (line 25) | func (d *durationValue) String() string { return (*time.Duration)(d).S...
function newDurationValue (line 10) | func newDurationValue(val time.Duration, p *time.Duration) *durationValue {
function durationConv (line 27) | func durationConv(sval string) (interface{}, error) {
method GetDuration (line 32) | func (f *FlagSet) GetDuration(name string) (time.Duration, error) {
method DurationVar (line 42) | func (f *FlagSet) DurationVar(p *time.Duration, name string, value time....
method DurationVarP (line 47) | func (f *FlagSet) DurationVarP(p *time.Duration, name, shorthand string,...
function DurationVar (line 53) | func DurationVar(p *time.Duration, name string, value time.Duration, usa...
function DurationVarP (line 58) | func DurationVarP(p *time.Duration, name, shorthand string, value time.D...
method Duration (line 64) | func (f *FlagSet) Duration(name string, value time.Duration, usage strin...
method DurationP (line 71) | func (f *FlagSet) DurationP(name, shorthand string, value time.Duration,...
function Duration (line 79) | func Duration(name string, value time.Duration, usage string) *time.Dura...
function DurationP (line 84) | func DurationP(name, shorthand string, value time.Duration, usage string...
FILE: vendor/github.com/spf13/pflag/flag.go
type ErrorHandling (line 115) | type ErrorHandling
constant ContinueOnError (line 119) | ContinueOnError ErrorHandling = iota
constant ExitOnError (line 121) | ExitOnError
constant PanicOnError (line 123) | PanicOnError
type NormalizedName (line 128) | type NormalizedName
type FlagSet (line 131) | type FlagSet struct
method SetNormalizeFunc (line 195) | func (f *FlagSet) SetNormalizeFunc(n func(f *FlagSet, name string) Nor...
method GetNormalizeFunc (line 207) | func (f *FlagSet) GetNormalizeFunc() func(f *FlagSet, name string) Nor...
method normalizeFlagName (line 214) | func (f *FlagSet) normalizeFlagName(name string) NormalizedName {
method out (line 219) | func (f *FlagSet) out() io.Writer {
method SetOutput (line 228) | func (f *FlagSet) SetOutput(output io.Writer) {
method VisitAll (line 234) | func (f *FlagSet) VisitAll(fn func(*Flag)) {
method HasFlags (line 241) | func (f *FlagSet) HasFlags() bool {
method HasAvailableFlags (line 247) | func (f *FlagSet) HasAvailableFlags() bool {
method Visit (line 264) | func (f *FlagSet) Visit(fn func(*Flag)) {
method Lookup (line 277) | func (f *FlagSet) Lookup(name string) *Flag {
method lookup (line 282) | func (f *FlagSet) lookup(name NormalizedName) *Flag {
method getFlagType (line 287) | func (f *FlagSet) getFlagType(name string, ftype string, convFunc func...
method ArgsLenAtDash (line 310) | func (f *FlagSet) ArgsLenAtDash() int {
method MarkDeprecated (line 317) | func (f *FlagSet) MarkDeprecated(name string, usageMessage string) err...
method MarkShorthandDeprecated (line 332) | func (f *FlagSet) MarkShorthandDeprecated(name string, usageMessage st...
method MarkHidden (line 346) | func (f *FlagSet) MarkHidden(name string) error {
method Set (line 362) | func (f *FlagSet) Set(name, value string) error {
method SetAnnotation (line 386) | func (f *FlagSet) SetAnnotation(name, key string, values []string) err...
method Changed (line 401) | func (f *FlagSet) Changed(name string) bool {
method PrintDefaults (line 417) | func (f *FlagSet) PrintDefaults() {
method FlagUsagesWrapped (line 559) | func (f *FlagSet) FlagUsagesWrapped(cols int) string {
method FlagUsages (line 625) | func (f *FlagSet) FlagUsages() string {
method NFlag (line 654) | func (f *FlagSet) NFlag() int { return len(f.actual) }
method Arg (line 661) | func (f *FlagSet) Arg(i int) string {
method NArg (line 675) | func (f *FlagSet) NArg() int { return len(f.args) }
method Args (line 681) | func (f *FlagSet) Args() []string { return f.args }
method Var (line 692) | func (f *FlagSet) Var(value Value, name string, usage string) {
method VarPF (line 697) | func (f *FlagSet) VarPF(value Value, name, shorthand, usage string) *F...
method VarP (line 711) | func (f *FlagSet) VarP(value Value, name, shorthand, usage string) {
method AddFlag (line 716) | func (f *FlagSet) AddFlag(flag *Flag) {
method AddFlagSet (line 754) | func (f *FlagSet) AddFlagSet(newSet *FlagSet) {
method failf (line 782) | func (f *FlagSet) failf(format string, a ...interface{}) error {
method usage (line 791) | func (f *FlagSet) usage() {
method setFlag (line 801) | func (f *FlagSet) setFlag(flag *Flag, value string, origArg string) er...
method parseLongArg (line 829) | func (f *FlagSet) parseLongArg(s string, args []string, fn parseFunc) ...
method parseSingleShortArg (line 867) | func (f *FlagSet) parseSingleShortArg(shorthands string, args []string...
method parseShortArg (line 906) | func (f *FlagSet) parseShortArg(s string, args []string, fn parseFunc)...
method parseArgs (line 920) | func (f *FlagSet) parseArgs(args []string, fn parseFunc) (err error) {
method Parse (line 955) | func (f *FlagSet) Parse(arguments []string) error {
method ParseAll (line 984) | func (f *FlagSet) ParseAll(arguments []string, fn func(flag *Flag, val...
method Parsed (line 1007) | func (f *FlagSet) Parsed() bool {
method SetInterspersed (line 1052) | func (f *FlagSet) SetInterspersed(interspersed bool) {
method Init (line 1059) | func (f *FlagSet) Init(name string, errorHandling ErrorHandling) {
type Flag (line 152) | type Flag struct
method defaultIsZeroValue (line 424) | func (f *Flag) defaultIsZeroValue() bool {
type Value (line 168) | type Value interface
function sortFlags (line 175) | func sortFlags(flags map[NormalizedName]*Flag) []*Flag {
function VisitAll (line 258) | func VisitAll(fn func(*Flag)) {
function Visit (line 272) | func Visit(fn func(*Flag)) {
function Lookup (line 357) | func Lookup(name string) *Flag {
function Set (line 411) | func Set(name, value string) error {
function UnquoteUsage (line 459) | func UnquoteUsage(flag *Flag) (name string, usage string) {
function wrapN (line 494) | func wrapN(i, slop int, s string) (string, string) {
function wrap (line 510) | func wrap(i, w int, s string) string {
function PrintDefaults (line 630) | func PrintDefaults() {
function defaultUsage (line 635) | func defaultUsage(f *FlagSet) {
function NFlag (line 657) | func NFlag() int { return len(CommandLine.actual) }
function Arg (line 670) | func Arg(i int) string {
function NArg (line 678) | func NArg() int { return len(CommandLine.args) }
function Args (line 684) | func Args() []string { return CommandLine.args }
function Var (line 771) | func Var(value Value, name string, usage string) {
function VarP (line 776) | func VarP(value Value, name, shorthand, usage string) {
function containsShorthand (line 820) | func containsShorthand(arg, shorthand string) bool {
type parseFunc (line 977) | type parseFunc
function Parse (line 1013) | func Parse() {
function ParseAll (line 1021) | func ParseAll(fn func(flag *Flag, value string) error) {
function SetInterspersed (line 1027) | func SetInterspersed(interspersed bool) {
function Parsed (line 1032) | func Parsed() bool {
function NewFlagSet (line 1041) | func NewFlagSet(name string, errorHandling ErrorHandling) *FlagSet {
FILE: vendor/github.com/spf13/pflag/float32.go
type float32Value (line 6) | type float32Value
method Set (line 13) | func (f *float32Value) Set(s string) error {
method Type (line 19) | func (f *float32Value) Type() string {
method String (line 23) | func (f *float32Value) String() string { return strconv.FormatFloat(fl...
function newFloat32Value (line 8) | func newFloat32Value(val float32, p *float32) *float32Value {
function float32Conv (line 25) | func float32Conv(sval string) (interface{}, error) {
method GetFloat32 (line 34) | func (f *FlagSet) GetFloat32(name string) (float32, error) {
method Float32Var (line 44) | func (f *FlagSet) Float32Var(p *float32, name string, value float32, usa...
method Float32VarP (line 49) | func (f *FlagSet) Float32VarP(p *float32, name, shorthand string, value ...
function Float32Var (line 55) | func Float32Var(p *float32, name string, value float32, usage string) {
function Float32VarP (line 60) | func Float32VarP(p *float32, name, shorthand string, value float32, usag...
method Float32 (line 66) | func (f *FlagSet) Float32(name string, value float32, usage string) *flo...
method Float32P (line 73) | func (f *FlagSet) Float32P(name, shorthand string, value float32, usage ...
function Float32 (line 81) | func Float32(name string, value float32, usage string) *float32 {
function Float32P (line 86) | func Float32P(name, shorthand string, value float32, usage string) *floa...
FILE: vendor/github.com/spf13/pflag/float64.go
type float64Value (line 6) | type float64Value
method Set (line 13) | func (f *float64Value) Set(s string) error {
method Type (line 19) | func (f *float64Value) Type() string {
method String (line 23) | func (f *float64Value) String() string { return strconv.FormatFloat(fl...
function newFloat64Value (line 8) | func newFloat64Value(val float64, p *float64) *float64Value {
function float64Conv (line 25) | func float64Conv(sval string) (interface{}, error) {
method GetFloat64 (line 30) | func (f *FlagSet) GetFloat64(name string) (float64, error) {
method Float64Var (line 40) | func (f *FlagSet) Float64Var(p *float64, name string, value float64, usa...
method Float64VarP (line 45) | func (f *FlagSet) Float64VarP(p *float64, name, shorthand string, value ...
function Float64Var (line 51) | func Float64Var(p *float64, name string, value float64, usage string) {
function Float64VarP (line 56) | func Float64VarP(p *float64, name, shorthand string, value float64, usag...
method Float64 (line 62) | func (f *FlagSet) Float64(name string, value float64, usage string) *flo...
method Float64P (line 69) | func (f *FlagSet) Float64P(name, shorthand string, value float64, usage ...
function Float64 (line 77) | func Float64(name string, value float64, usage string) *float64 {
function Float64P (line 82) | func Float64P(name, shorthand string, value float64, usage string) *floa...
FILE: vendor/github.com/spf13/pflag/golangflag.go
type flagValueWrapper (line 17) | type flagValueWrapper struct
method String (line 48) | func (v *flagValueWrapper) String() string {
method Set (line 52) | func (v *flagValueWrapper) Set(s string) error {
method Type (line 56) | func (v *flagValueWrapper) Type() string {
type goBoolFlag (line 24) | type goBoolFlag interface
function wrapFlagValue (line 29) | func wrapFlagValue(v goflag.Value) Value {
function PFlagFromGoFlag (line 64) | func PFlagFromGoFlag(goflag *goflag.Flag) *Flag {
method AddGoFlag (line 85) | func (f *FlagSet) AddGoFlag(goflag *goflag.Flag) {
method AddGoFlagSet (line 94) | func (f *FlagSet) AddGoFlagSet(newSet *goflag.FlagSet) {
FILE: vendor/github.com/spf13/pflag/int.go
type intValue (line 6) | type intValue
method Set (line 13) | func (i *intValue) Set(s string) error {
method Type (line 19) | func (i *intValue) Type() string {
method String (line 23) | func (i *intValue) String() string { return strconv.Itoa(int(*i)) }
function newIntValue (line 8) | func newIntValue(val int, p *int) *intValue {
function intConv (line 25) | func intConv(sval string) (interface{}, error) {
method GetInt (line 30) | func (f *FlagSet) GetInt(name string) (int, error) {
method IntVar (line 40) | func (f *FlagSet) IntVar(p *int, name string, value int, usage string) {
method IntVarP (line 45) | func (f *FlagSet) IntVarP(p *int, name, shorthand string, value int, usa...
function IntVar (line 51) | func IntVar(p *int, name string, value int, usage string) {
function IntVarP (line 56) | func IntVarP(p *int, name, shorthand string, value int, usage string) {
method Int (line 62) | func (f *FlagSet) Int(name string, value int, usage string) *int {
method IntP (line 69) | func (f *FlagSet) IntP(name, shorthand string, value int, usage string) ...
function Int (line 77) | func Int(name string, value int, usage string) *int {
function IntP (line 82) | func IntP(name, shorthand string, value int, usage string) *int {
FILE: vendor/github.com/spf13/pflag/int32.go
type int32Value (line 6) | type int32Value
method Set (line 13) | func (i *int32Value) Set(s string) error {
method Type (line 19) | func (i *int32Value) Type() string {
method String (line 23) | func (i *int32Value) String() string { return strconv.FormatInt(int64(...
function newInt32Value (line 8) | func newInt32Value(val int32, p *int32) *int32Value {
function int32Conv (line 25) | func int32Conv(sval string) (interface{}, error) {
method GetInt32 (line 34) | func (f *FlagSet) GetInt32(name string) (int32, error) {
method Int32Var (line 44) | func (f *FlagSet) Int32Var(p *int32, name string, value int32, usage str...
method Int32VarP (line 49) | func (f *FlagSet) Int32VarP(p *int32, name, shorthand string, value int3...
function Int32Var (line 55) | func Int32Var(p *int32, name string, value int32, usage string) {
function Int32VarP (line 60) | func Int32VarP(p *int32, name, shorthand string, value int32, usage stri...
method Int32 (line 66) | func (f *FlagSet) Int32(name string, value int32, usage string) *int32 {
method Int32P (line 73) | func (f *FlagSet) Int32P(name, shorthand string, value int32, usage stri...
function Int32 (line 81) | func Int32(name string, value int32, usage string) *int32 {
function Int32P (line 86) | func Int32P(name, shorthand string, value int32, usage string) *int32 {
FILE: vendor/github.com/spf13/pflag/int64.go
type int64Value (line 6) | type int64Value
method Set (line 13) | func (i *int64Value) Set(s string) error {
method Type (line 19) | func (i *int64Value) Type() string {
method String (line 23) | func (i *int64Value) String() string { return strconv.FormatInt(int64(...
function newInt64Value (line 8) | func newInt64Value(val int64, p *int64) *int64Value {
function int64Conv (line 25) | func int64Conv(sval string) (interface{}, error) {
method GetInt64 (line 30) | func (f *FlagSet) GetInt64(name string) (int64, error) {
method Int64Var (line 40) | func (f *FlagSet) Int64Var(p *int64, name string, value int64, usage str...
method Int64VarP (line 45) | func (f *FlagSet) Int64VarP(p *int64, name, shorthand string, value int6...
function Int64Var (line 51) | func Int64Var(p *int64, name string, value int64, usage string) {
function Int64VarP (line 56) | func Int64VarP(p *int64, name, shorthand string, value int64, usage stri...
method Int64 (line 62) | func (f *FlagSet) Int64(name string, value int64, usage string) *int64 {
method Int64P (line 69) | func (f *FlagSet) Int64P(name, shorthand string, value int64, usage stri...
function Int64 (line 77) | func Int64(name string, value int64, usage string) *int64 {
function Int64P (line 82) | func Int64P(name, shorthand string, value int64, usage string) *int64 {
FILE: vendor/github.com/spf13/pflag/int8.go
type int8Value (line 6) | type int8Value
method Set (line 13) | func (i *int8Value) Set(s string) error {
method Type (line 19) | func (i *int8Value) Type() string {
method String (line 23) | func (i *int8Value) String() string { return strconv.FormatInt(int64(*...
function newInt8Value (line 8) | func newInt8Value(val int8, p *int8) *int8Value {
function int8Conv (line 25) | func int8Conv(sval string) (interface{}, error) {
method GetInt8 (line 34) | func (f *FlagSet) GetInt8(name string) (int8, error) {
method Int8Var (line 44) | func (f *FlagSet) Int8Var(p *int8, name string, value int8, usage string) {
method Int8VarP (line 49) | func (f *FlagSet) Int8VarP(p *int8, name, shorthand string, value int8, ...
function Int8Var (line 55) | func Int8Var(p *int8, name string, value int8, usage string) {
function Int8VarP (line 60) | func Int8VarP(p *int8, name, shorthand string, value int8, usage string) {
method Int8 (line 66) | func (f *FlagSet) Int8(name string, value int8, usage string) *int8 {
method Int8P (line 73) | func (f *FlagSet) Int8P(name, shorthand string, value int8, usage string...
function Int8 (line 81) | func Int8(name string, value int8, usage string) *int8 {
function Int8P (line 86) | func Int8P(name, shorthand string, value int8, usage string) *int8 {
FILE: vendor/github.com/spf13/pflag/int_slice.go
type intSliceValue (line 10) | type intSliceValue struct
method Set (line 22) | func (s *intSliceValue) Set(val string) error {
method Type (line 42) | func (s *intSliceValue) Type() string {
method String (line 46) | func (s *intSliceValue) String() string {
function newIntSliceValue (line 15) | func newIntSliceValue(val []int, p *[]int) *intSliceValue {
function intSliceConv (line 54) | func intSliceConv(val string) (interface{}, error) {
method GetIntSlice (line 74) | func (f *FlagSet) GetIntSlice(name string) ([]int, error) {
method IntSliceVar (line 84) | func (f *FlagSet) IntSliceVar(p *[]int, name string, value []int, usage ...
method IntSliceVarP (line 89) | func (f *FlagSet) IntSliceVarP(p *[]int, name, shorthand string, value [...
function IntSliceVar (line 95) | func IntSliceVar(p *[]int, name string, value []int, usage string) {
function IntSliceVarP (line 100) | func IntSliceVarP(p *[]int, name, shorthand string, value []int, usage s...
method IntSlice (line 106) | func (f *FlagSet) IntSlice(name string, value []int, usage string) *[]int {
method IntSliceP (line 113) | func (f *FlagSet) IntSliceP(name, shorthand string, value []int, usage s...
function IntSlice (line 121) | func IntSlice(name string, value []int, usage string) *[]int {
function IntSliceP (line 126) | func IntSliceP(name, shorthand string, value []int, usage string) *[]int {
FILE: vendor/github.com/spf13/pflag/ip.go
type ipValue (line 10) | type ipValue
method String (line 17) | func (i *ipValue) String() string { return net.IP(*i).String() }
method Set (line 18) | func (i *ipValue) Set(s string) error {
method Type (line 27) | func (i *ipValue) Type() string {
function newIPValue (line 12) | func newIPValue(val net.IP, p *net.IP) *ipValue {
function ipConv (line 31) | func ipConv(sval string) (interface{}, error) {
method GetIP (line 40) | func (f *FlagSet) GetIP(name string) (net.IP, error) {
method IPVar (line 50) | func (f *FlagSet) IPVar(p *net.IP, name string, value net.IP, usage stri...
method IPVarP (line 55) | func (f *FlagSet) IPVarP(p *net.IP, name, shorthand string, value net.IP...
function IPVar (line 61) | func IPVar(p *net.IP, name string, value net.IP, usage string) {
function IPVarP (line 66) | func IPVarP(p *net.IP, name, shorthand string, value net.IP, usage strin...
method IP (line 72) | func (f *FlagSet) IP(name string, value net.IP, usage string) *net.IP {
method IPP (line 79) | func (f *FlagSet) IPP(name, shorthand string, value net.IP, usage string...
function IP (line 87) | func IP(name string, value net.IP, usage string) *net.IP {
function IPP (line 92) | func IPP(name, shorthand string, value net.IP, usage string) *net.IP {
FILE: vendor/github.com/spf13/pflag/ip_slice.go
type ipSliceValue (line 11) | type ipSliceValue struct
method Set (line 25) | func (s *ipSliceValue) Set(val string) error {
method Type (line 58) | func (s *ipSliceValue) Type() string {
method String (line 63) | func (s *ipSliceValue) String() string {
function newIPSliceValue (line 16) | func newIPSliceValue(val []net.IP, p *[]net.IP) *ipSliceValue {
function ipSliceConv (line 75) | func ipSliceConv(val string) (interface{}, error) {
method GetIPSlice (line 94) | func (f *FlagSet) GetIPSlice(name string) ([]net.IP, error) {
method IPSliceVar (line 104) | func (f *FlagSet) IPSliceVar(p *[]net.IP, name string, value []net.IP, u...
method IPSliceVarP (line 109) | func (f *FlagSet) IPSliceVarP(p *[]net.IP, name, shorthand string, value...
function IPSliceVar (line 115) | func IPSliceVar(p *[]net.IP, name string, value []net.IP, usage string) {
function IPSliceVarP (line 120) | func IPSliceVarP(p *[]net.IP, name, shorthand string, value []net.IP, us...
method IPSlice (line 126) | func (f *FlagSet) IPSlice(name string, value []net.IP, usage string) *[]...
method IPSliceP (line 133) | func (f *FlagSet) IPSliceP(name, shorthand string, value []net.IP, usage...
function IPSlice (line 141) | func IPSlice(name string, value []net.IP, usage string) *[]net.IP {
function IPSliceP (line 146) | func IPSliceP(name, shorthand string, value []net.IP, usage string) *[]n...
FILE: vendor/github.com/spf13/pflag/ipmask.go
type ipMaskValue (line 10) | type ipMaskValue
method String (line 17) | func (i *ipMaskValue) String() string { return net.IPMask(*i).String() }
method Set (line 18) | func (i *ipMaskValue) Set(s string) error {
method Type (line 27) | func (i *ipMaskValue) Type() string {
function newIPMaskValue (line 12) | func newIPMaskValue(val net.IPMask, p *net.IPMask) *ipMaskValue {
function ParseIPv4Mask (line 33) | func ParseIPv4Mask(s string) net.IPMask {
function parseIPv4Mask (line 59) | func parseIPv4Mask(sval string) (interface{}, error) {
method GetIPv4Mask (line 68) | func (f *FlagSet) GetIPv4Mask(name string) (net.IPMask, error) {
method IPMaskVar (line 78) | func (f *FlagSet) IPMaskVar(p *net.IPMask, name string, value net.IPMask...
method IPMaskVarP (line 83) | func (f *FlagSet) IPMaskVarP(p *net.IPMask, name, shorthand string, valu...
function IPMaskVar (line 89) | func IPMaskVar(p *net.IPMask, name string, value net.IPMask, usage strin...
function IPMaskVarP (line 94) | func IPMaskVarP(p *net.IPMask, name, shorthand string, value net.IPMask,...
method IPMask (line 100) | func (f *FlagSet) IPMask(name string, value net.IPMask, usage string) *n...
method IPMaskP (line 107) | func (f *FlagSet) IPMaskP(name, shorthand string, value net.IPMask, usag...
function IPMask (line 115) | func IPMask(name string, value net.IPMask, usage string) *net.IPMask {
function IPMaskP (line 120) | func IPMaskP(name, shorthand string, value net.IPMask, usage string) *ne...
FILE: vendor/github.com/spf13/pflag/ipnet.go
type ipNetValue (line 10) | type ipNetValue
method String (line 12) | func (ipnet ipNetValue) String() string {
method Set (line 17) | func (ipnet *ipNetValue) Set(value string) error {
method Type (line 26) | func (*ipNetValue) Type() string {
function newIPNetValue (line 30) | func newIPNetValue(val net.IPNet, p *net.IPNet) *ipNetValue {
function ipNetConv (line 35) | func ipNetConv(sval string) (interface{}, error) {
method GetIPNet (line 44) | func (f *FlagSet) GetIPNet(name string) (net.IPNet, error) {
method IPNetVar (line 54) | func (f *FlagSet) IPNetVar(p *net.IPNet, name string, value net.IPNet, u...
method IPNetVarP (line 59) | func (f *FlagSet) IPNetVarP(p *net.IPNet, name, shorthand string, value ...
function IPNetVar (line 65) | func IPNetVar(p *net.IPNet, name string, value net.IPNet, usage string) {
function IPNetVarP (line 70) | func IPNetVarP(p *net.IPNet, name, shorthand string, value net.IPNet, us...
method IPNet (line 76) | func (f *FlagSet) IPNet(name string, value net.IPNet, usage string) *net...
method IPNetP (line 83) | func (f *FlagSet) IPNetP(name, shorthand string, value net.IPNet, usage ...
function IPNet (line 91) | func IPNet(name string, value net.IPNet, usage string) *net.IPNet {
function IPNetP (line 96) | func IPNetP(name, shorthand string, value net.IPNet, usage string) *net....
FILE: vendor/github.com/spf13/pflag/string.go
type stringValue (line 4) | type stringValue
method Set (line 11) | func (s *stringValue) Set(val string) error {
method Type (line 15) | func (s *stringValue) Type() string {
method String (line 19) | func (s *stringValue) String() string { return string(*s) }
function newStringValue (line 6) | func newStringValue(val string, p *string) *stringValue {
function stringConv (line 21) | func stringConv(sval string) (interface{}, error) {
method GetString (line 26) | func (f *FlagSet) GetString(name string) (string, error) {
method StringVar (line 36) | func (f *FlagSet) StringVar(p *string, name string, value string, usage ...
method StringVarP (line 41) | func (f *FlagSet) StringVarP(p *string, name, shorthand string, value st...
function StringVar (line 47) | func StringVar(p *string, name string, value string, usage string) {
function StringVarP (line 52) | func StringVarP(p *string, name, shorthand string, value string, usage s...
method String (line 58) | func (f *FlagSet) String(name string, value string, usage string) *string {
method StringP (line 65) | func (f *FlagSet) StringP(name, shorthand string, value string, usage st...
function String (line 73) | func String(name string, value string, usage string) *string {
function StringP (line 78) | func StringP(name, shorthand string, value string, usage string) *string {
FILE: vendor/github.com/spf13/pflag/string_array.go
type stringArrayValue (line 4) | type stringArrayValue struct
method Set (line 16) | func (s *stringArrayValue) Set(val string) error {
method Type (line 26) | func (s *stringArrayValue) Type() string {
method String (line 30) | func (s *stringArrayValue) String() string {
function newStringArrayValue (line 9) | func newStringArrayValue(val []string, p *[]string) *stringArrayValue {
function stringArrayConv (line 35) | func stringArrayConv(sval string) (interface{}, error) {
method GetStringArray (line 45) | func (f *FlagSet) GetStringArray(name string) ([]string, error) {
method StringArrayVar (line 56) | func (f *FlagSet) StringArrayVar(p *[]string, name string, value []strin...
method StringArrayVarP (line 61) | func (f *FlagSet) StringArrayVarP(p *[]string, name, shorthand string, v...
function StringArrayVar (line 68) | func StringArrayVar(p *[]string, name string, value []string, usage stri...
function StringArrayVarP (line 73) | func StringArrayVarP(p *[]string, name, shorthand string, value []string...
method StringArray (line 80) | func (f *FlagSet) StringArray(name string, value []string, usage string)...
method StringArrayP (line 87) | func (f *FlagSet) StringArrayP(name, shorthand string, value []string, u...
function StringArray (line 96) | func StringArray(name string, value []string, usage string) *[]string {
function StringArrayP (line 101) | func StringArrayP(name, shorthand string, value []string, usage string) ...
FILE: vendor/github.com/spf13/pflag/string_slice.go
type stringSliceValue (line 10) | type stringSliceValue struct
method Set (line 42) | func (s *stringSliceValue) Set(val string) error {
method Type (line 56) | func (s *stringSliceValue) Type() string {
method String (line 60) | func (s *stringSliceValue) String() string {
function newStringSliceValue (line 15) | func newStringSliceValue(val []string, p *[]string) *stringSliceValue {
function readAsCSV (line 22) | func readAsCSV(val string) ([]string, error) {
function writeAsCSV (line 31) | func writeAsCSV(vals []string) (string, error) {
function stringSliceConv (line 65) | func stringSliceConv(sval string) (interface{}, error) {
method GetStringSlice (line 75) | func (f *FlagSet) GetStringSlice(name string) ([]string, error) {
method StringSliceVar (line 85) | func (f *FlagSet) StringSliceVar(p *[]string, name string, value []strin...
method StringSliceVarP (line 90) | func (f *FlagSet) StringSliceVarP(p *[]string, name, shorthand string, v...
function StringSliceVar (line 96) | func StringSliceVar(p *[]string, name string, value []string, usage stri...
function StringSliceVarP (line 101) | func StringSliceVarP(p *[]string, name, shorthand string, value []string...
method StringSlice (line 107) | func (f *FlagSet) StringSlice(name string, value []string, usage string)...
method StringSliceP (line 114) | func (f *FlagSet) StringSliceP(name, shorthand string, value []string, u...
function StringSlice (line 122) | func StringSlice(name string, value []string, usage string) *[]string {
function StringSliceP (line 127) | func StringSliceP(name, shorthand string, value []string, usage string) ...
FILE: vendor/github.com/spf13/pflag/uint.go
type uintValue (line 6) | type uintValue
method Set (line 13) | func (i *uintValue) Set(s string) error {
method Type (line 19) | func (i *uintValue) Type() string {
method String (line 23) | func (i *uintValue) String() string { return strconv.FormatUint(uint64...
function newUintValue (line 8) | func newUintValue(val uint, p *uint) *uintValue {
function uintConv (line 25) | func uintConv(sval string) (interface{}, error) {
method GetUint (line 34) | func (f *FlagSet) GetUint(name string) (uint, error) {
method UintVar (line 44) | func (f *FlagSet) UintVar(p *uint, name string, value uint, usage string) {
method UintVarP (line 49) | func (f *FlagSet) UintVarP(p *uint, name, shorthand string, value uint, ...
function UintVar (line 55) | func UintVar(p *uint, name string, value uint, usage string) {
function UintVarP (line 60) | func UintVarP(p *uint, name, shorthand string, value uint, usage string) {
method Uint (line 66) | func (f *FlagSet) Uint(name string, value uint, usage string) *uint {
method UintP (line 73) | func (f *FlagSet) UintP(name, shorthand string, value uint, usage string...
function Uint (line 81) | func Uint(name string, value uint, usage string) *uint {
function UintP (line 86) | func UintP(name, shorthand string, value uint, usage string) *uint {
FILE: vendor/github.com/spf13/pflag/uint16.go
type uint16Value (line 6) | type uint16Value
method Set (line 13) | func (i *uint16Value) Set(s string) error {
method Type (line 19) | func (i *uint16Value) Type() string {
method String (line 23) | func (i *uint16Value) String() string { return strconv.FormatUint(uint...
function newUint16Value (line 8) | func newUint16Value(val uint16, p *uint16) *uint16Value {
function uint16Conv (line 25) | func uint16Conv(sval string) (interface{}, error) {
method GetUint16 (line 34) | func (f *FlagSet) GetUint16(name string) (uint16, error) {
method Uint16Var (line 44) | func (f *FlagSet) Uint16Var(p *uint16, name string, value uint16, usage ...
method Uint16VarP (line 49) | func (f *FlagSet) Uint16VarP(p *uint16, name, shorthand string, value ui...
function Uint16Var (line 55) | func Uint16Var(p *uint16, name string, value uint16, usage string) {
function Uint16VarP (line 60) | func Uint16VarP(p *uint16, name, shorthand string, value uint16, usage s...
method Uint16 (line 66) | func (f *FlagSet) Uint16(name string, value uint16, usage string) *uint16 {
method Uint16P (line 73) | func (f *FlagSet) Uint16P(name, shorthand string, value uint16, usage st...
function Uint16 (line 81) | func Uint16(name string, value uint16, usage string) *uint16 {
function Uint16P (line 86) | func Uint16P(name, shorthand string, value uint16, usage string) *uint16 {
FILE: vendor/github.com/spf13/pflag/uint32.go
type uint32Value (line 6) | type uint32Value
method Set (line 13) | func (i *uint32Value) Set(s string) error {
method Type (line 19) | func (i *uint32Value) Type() string {
method String (line 23) | func (i *uint32Value) String() string { return strconv.FormatUint(uint...
function newUint32Value (line 8) | func newUint32Value(val uint32, p *uint32) *uint32Value {
function uint32Conv (line 25) | func uint32Conv(sval string) (interface{}, error) {
method GetUint32 (line 34) | func (f *FlagSet) GetUint32(name string) (uint32, error) {
method Uint32Var (line 44) | func (f *FlagSet) Uint32Var(p *uint32, name string, value uint32, usage ...
method Uint32VarP (line 49) | func (f *FlagSet) Uint32VarP(p *uint32, name, shorthand string, value ui...
function Uint32Var (line 55) | func Uint32Var(p *uint32, name string, value uint32, usage string) {
function Uint32VarP (line 60) | func Uint32VarP(p *uint32, name, shorthand string, value uint32, usage s...
method Uint32 (line 66) | func (f *FlagSet) Uint32(name string, value uint32, usage string) *uint32 {
method Uint32P (line 73) | func (f *FlagSet) Uint32P(name, shorthand string, value uint32, usage st...
function Uint32 (line 81) | func Uint32(name string, value uint32, usage string) *uint32 {
function Uint32P (line 86) | func Uint32P(name, shorthand string, value uint32, usage string) *uint32 {
FILE: vendor/github.com/spf13/pflag/uint64.go
type uint64Value (line 6) | type uint64Value
method Set (line 13) | func (i *uint64Value) Set(s string) error {
method Type (line 19) | func (i *uint64Value) Type() string {
method String (line 23) | func (i *uint64Value) String() string { return strconv.FormatUint(uint...
function newUint64Value (line 8) | func newUint64Value(val uint64, p *uint64) *uint64Value {
function uint64Conv (line 25) | func uint64Conv(sval string) (interface{}, error) {
method GetUint64 (line 34) | func (f *FlagSet) GetUint64(name string) (uint64, error) {
method Uint64Var (line 44) | func (f *FlagSet) Uint64Var(p *uint64, name string, value uint64, usage ...
method Uint64VarP (line 49) | func (f *FlagSet) Uint64VarP(p *uint64, name, shorthand string, value ui...
function Uint64Var (line 55) | func Uint64Var(p *uint64, name string, value uint64, usage string) {
function Uint64VarP (line 60) | func Uint64VarP(p *uint64, name, shorthand string, value uint64, usage s...
method Uint64 (line 66) | func (f *FlagSet) Uint64(name string, value uint64, usage string) *uint64 {
method Uint64P (line 73) | func (f *FlagSet) Uint64P(name, shorthand string, value uint64, usage st...
function Uint64 (line 81) | func Uint64(name string, value uint64, usage string) *uint64 {
function Uint64P (line 86) | func Uint64P(name, shorthand string, value uint64, usage string) *uint64 {
FILE: vendor/github.com/spf13/pflag/uint8.go
type uint8Value (line 6) | type uint8Value
method Set (line 13) | func (i *uint8Value) Set(s string) error {
method Type (line 19) | func (i *uint8Value) Type() string {
method String (line 23) | func (i *uint8Value) String() string { return strconv.FormatUint(uint6...
function newUint8Value (line 8) | func newUint8Value(val uint8, p *uint8) *uint8Value {
function uint8Conv (line 25) | func uint8Conv(sval string) (interface{}, error) {
method GetUint8 (line 34) | func (f *FlagSet) GetUint8(name string) (uint8, error) {
method Uint8Var (line 44) | func (f *FlagSet) Uint8Var(p *uint8, name string, value uint8, usage str...
method Uint8VarP (line 49) | func (f *FlagSet) Uint8VarP(p *uint8, name, shorthand string, value uint...
function Uint8Var (line 55) | func Uint8Var(p *uint8, name string, value uint8, usage string) {
function Uint8VarP (line 60) | func Uint8VarP(p *uint8, name, shorthand string, value uint8, usage stri...
method Uint8 (line 66) | func (f *FlagSet) Uint8(name string, value uint8, usage string) *uint8 {
method Uint8P (line 73) | func (f *FlagSet) Uint8P(name, shorthand string, value uint8, usage stri...
function Uint8 (line 81) | func Uint8(name string, value uint8, usage string) *uint8 {
function Uint8P (line 86) | func Uint8P(name, shorthand string, value uint8, usage string) *uint8 {
FILE: vendor/github.com/spf13/pflag/uint_slice.go
type uintSliceValue (line 10) | type uintSliceValue struct
method Set (line 22) | func (s *uintSliceValue) Set(val string) error {
method Type (line 41) | func (s *uintSliceValue) Type() string {
method String (line 45) | func (s *uintSliceValue) String() string {
function newUintSliceValue (line 15) | func newUintSliceValue(val []uint, p *[]uint) *uintSliceValue {
function uintSliceConv (line 53) | func uintSliceConv(val string) (interface{}, error) {
method GetUintSlice (line 72) | func (f *FlagSet) GetUintSlice(name string) ([]uint, error) {
method UintSliceVar (line 82) | func (f *FlagSet) UintSliceVar(p *[]uint, name string, value []uint, usa...
method UintSliceVarP (line 87) | func (f *FlagSet) UintSliceVarP(p *[]uint, name, shorthand string, value...
function UintSliceVar (line 93) | func UintSliceVar(p *[]uint, name string, value []uint, usage string) {
function UintSliceVarP (line 98) | func UintSliceVarP(p *[]uint, name, shorthand string, value []uint, usag...
method UintSlice (line 104) | func (f *FlagSet) UintSlice(name string, value []uint, usage string) *[]...
method UintSliceP (line 111) | func (f *FlagSet) UintSliceP(name, shorthand string, value []uint, usage...
function UintSlice (line 119) | func UintSlice(name string, value []uint, usage string) *[]uint {
function UintSliceP (line 124) | func UintSliceP(name, shorthand string, value []uint, usage string) *[]u...
FILE: vendor/github.com/willf/bitset/bitset.go
constant wordSize (line 53) | wordSize = uint(64)
constant log2WordSize (line 56) | log2WordSize = uint(6)
constant allBits (line 59) | allBits uint64 = 0xffffffffffffffff
type BitSet (line 62) | type BitSet struct
method safeSet (line 71) | func (b *BitSet) safeSet() []uint64 {
method Bytes (line 84) | func (b *BitSet) Bytes() []uint64 {
method Len (line 121) | func (b *BitSet) Len() uint {
method extendSetMaybe (line 126) | func (b *BitSet) extendSetMaybe(i uint) {
method Test (line 143) | func (b *BitSet) Test(i uint) bool {
method Set (line 151) | func (b *BitSet) Set(i uint) *BitSet {
method Clear (line 158) | func (b *BitSet) Clear(i uint) *BitSet {
method SetTo (line 167) | func (b *BitSet) SetTo(i uint, value bool) *BitSet {
method Flip (line 175) | func (b *BitSet) Flip(i uint) *BitSet {
method String (line 184) | func (b *BitSet) String() string {
method NextSet (line 212) | func (b *BitSet) NextSet(i uint) (uint, bool) {
method NextClear (line 236) | func (b *BitSet) NextClear(i uint) (uint, bool) {
method ClearAll (line 258) | func (b *BitSet) ClearAll() *BitSet {
method wordCount (line 268) | func (b *BitSet) wordCount() int {
method Clone (line 273) | func (b *BitSet) Clone() *BitSet {
method Copy (line 284) | func (b *BitSet) Copy(c *BitSet) (count uint) {
method Count (line 299) | func (b *BitSet) Count() uint {
method Equal (line 320) | func (b *BitSet) Equal(c *BitSet) bool {
method Difference (line 348) | func (b *BitSet) Difference(compare *BitSet) (result *BitSet) {
method DifferenceCardinality (line 363) | func (b *BitSet) DifferenceCardinality(compare *BitSet) uint {
method InPlaceDifference (line 378) | func (b *BitSet) InPlaceDifference(compare *BitSet) {
method Intersection (line 403) | func (b *BitSet) Intersection(compare *BitSet) (result *BitSet) {
method IntersectionCardinality (line 415) | func (b *BitSet) IntersectionCardinality(compare *BitSet) uint {
method InPlaceIntersection (line 426) | func (b *BitSet) InPlaceIntersection(compare *BitSet) {
method Union (line 447) | func (b *BitSet) Union(compare *BitSet) (result *BitSet) {
method UnionCardinality (line 460) | func (b *BitSet) UnionCardinality(compare *BitSet) uint {
method InPlaceUnion (line 473) | func (b *BitSet) InPlaceUnion(compare *BitSet) {
method SymmetricDifference (line 495) | func (b *BitSet) SymmetricDifference(compare *BitSet) (result *BitSet) {
method SymmetricDifferenceCardinality (line 508) | func (b *BitSet) SymmetricDifferenceCardinality(compare *BitSet) uint {
method InPlaceSymmetricDifference (line 521) | func (b *BitSet) InPlaceSymmetricDifference(compare *BitSet) {
method isLenExactMultiple (line 542) | func (b *BitSet) isLenExactMultiple() bool {
method cleanLastWord (line 547) | func (b *BitSet) cleanLastWord() {
method Complement (line 554) | func (b *BitSet) Complement() (result *BitSet) {
method All (line 566) | func (b *BitSet) All() bool {
method None (line 573) | func (b *BitSet) None() bool {
method Any (line 587) | func (b *BitSet) Any() bool {
method IsSuperSet (line 593) | func (b *BitSet) IsSuperSet(other *BitSet) bool {
method IsStrictSuperSet (line 603) | func (b *BitSet) IsStrictSuperSet(other *BitSet) bool {
method DumpAsBits (line 608) | func (b *BitSet) DumpAsBits() string {
method BinaryStorageSize (line 621) | func (b *BitSet) BinaryStorageSize() int {
method WriteTo (line 626) | func (b *BitSet) WriteTo(stream io.Writer) (int64, error) {
method ReadFrom (line 641) | func (b *BitSet) ReadFrom(stream io.Reader) (int64, error) {
method MarshalBinary (line 666) | func (b *BitSet) MarshalBinary() ([]byte, error) {
method UnmarshalBinary (line 681) | func (b *BitSet) UnmarshalBinary(data []byte) error {
method MarshalJSON (line 691) | func (b *BitSet) MarshalJSON() ([]byte, error) {
method UnmarshalJSON (line 703) | func (b *BitSet) UnmarshalJSON(data []byte) error {
type Error (line 68) | type Error
function From (line 79) | func From(buf []uint64) *BitSet {
function wordsNeeded (line 89) | func wordsNeeded(i uint) int {
function New (line 97) | func New(length uint) (bset *BitSet) {
function Cap (line 116) | func Cap() uint {
function trailingZeroes64 (line 313) | func trailingZeroes64(v uint64) uint {
function panicIfNull (line 340) | func panicIfNull(b *BitSet) {
function sortByLength (line 392) | func sortByLength(a *BitSet, b *BitSet) (ap *BitSet, bp *BitSet) {
FILE: vendor/github.com/willf/bitset/popcnt.go
function popcount (line 6) | func popcount(x uint64) (n uint64) {
function popcntSliceGo (line 15) | func popcntSliceGo(s []uint64) uint64 {
function popcntMaskSliceGo (line 23) | func popcntMaskSliceGo(s, m []uint64) uint64 {
function popcntAndSliceGo (line 31) | func popcntAndSliceGo(s, m []uint64) uint64 {
function popcntOrSliceGo (line 39) | func popcntOrSliceGo(s, m []uint64) uint64 {
function popcntXorSliceGo (line 47) | func popcntXorSliceGo(s, m []uint64) uint64 {
FILE: vendor/github.com/willf/bitset/popcnt_amd64.go
function hasAsm (line 9) | func hasAsm() bool
function popcntSliceAsm (line 16) | func popcntSliceAsm(s []uint64) uint64
function popcntMaskSliceAsm (line 20) | func popcntMaskSliceAsm(s, m []uint64) uint64
function popcntAndSliceAsm (line 24) | func popcntAndSliceAsm(s, m []uint64) uint64
function popcntOrSliceAsm (line 28) | func popcntOrSliceAsm(s, m []uint64) uint64
function popcntXorSliceAsm (line 32) | func popcntXorSliceAsm(s, m []uint64) uint64
function popcntSlice (line 34) | func popcntSlice(s []uint64) uint64 {
function popcntMaskSlice (line 41) | func popcntMaskSlice(s, m []uint64) uint64 {
function popcntAndSlice (line 48) | func popcntAndSlice(s, m []uint64) uint64 {
function popcntOrSlice (line 55) | func popcntOrSlice(s, m []uint64) uint64 {
function popcntXorSlice (line 62) | func popcntXorSlice(s, m []uint64) uint64 {
FILE: vendor/github.com/willf/bitset/popcnt_generic.go
function popcntSlice (line 5) | func popcntSlice(s []uint64) uint64 {
function popcntMaskSlice (line 9) | func popcntMaskSlice(s, m []uint64) uint64 {
function popcntAndSlice (line 13) | func popcntAndSlice(s, m []uint64) uint64 {
function popcntOrSlice (line 17) | func popcntOrSlice(s, m []uint64) uint64 {
function popcntXorSlice (line 21) | func popcntXorSlice(s, m []uint64) uint64 {
FILE: writer.go
type writer (line 24) | type writer struct
method Reset (line 35) | func (w *writer) Reset(newWriter io.Writer) {
method WriteByte (line 40) | func (w *writer) WriteByte(c byte) error {
method Write (line 49) | func (w *writer) Write(p []byte) (int, error) {
method Flush (line 55) | func (w *writer) Flush() error {
method WritePackedUintIn (line 59) | func (w *writer) WritePackedUintIn(v uint64, n int) error {
method WritePackedUint (line 70) | func (w *writer) WritePackedUint(v uint64) error {
function newWriter (line 29) | func newWriter(w io.Writer) *writer {
function packedSize (line 75) | func packedSize(n uint64) int {
FILE: writer_test.go
function TestPackedSize (line 24) | func TestPackedSize(t *testing.T) {
type stubWriter (line 59) | type stubWriter struct
method Write (line 63) | func (s *stubWriter) Write(p []byte) (n int, err error) {
function TestWriteByteErr (line 68) | func TestWriteByteErr(t *testing.T) {
function TestWritePackedUintErr (line 82) | func TestWritePackedUintErr(t *testing.T) {
Condensed preview — 121 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (580K chars).
[
{
"path": ".github/workflows/tests.yml",
"chars": 504,
"preview": "on:\n push:\n branches:\n - master\n pull_request:\nname: Tests\njobs:\n test:\n strategy:\n matrix:\n go-"
},
{
"path": "CONTRIBUTING.md",
"chars": 995,
"preview": "# Contributing to Vellum\n\nWe look forward to your contributions, but ask that you first review these guidelines.\n\n### Si"
},
{
"path": "LICENSE",
"chars": 11357,
"preview": "\n Apache License\n Version 2.0, January 2004\n "
},
{
"path": "README.md",
"chars": 7577,
"preview": "#  vellum\n\n# NOTE: active development of the vellum library has moved to [https://github.com/ble"
},
{
"path": "automaton.go",
"chars": 2291,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "builder.go",
"chars": 10282,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "builder_test.go",
"chars": 5138,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "cmd/vellum/cmd/dot.go",
"chars": 1875,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "cmd/vellum/cmd/dump.go",
"chars": 1322,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "cmd/vellum/cmd/fuzzy.go",
"chars": 2090,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "cmd/vellum/cmd/grep.go",
"chars": 1850,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "cmd/vellum/cmd/info.go",
"chars": 1250,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "cmd/vellum/cmd/map.go",
"chars": 2113,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "cmd/vellum/cmd/range.go",
"chars": 1749,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "cmd/vellum/cmd/root.go",
"chars": 1461,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "cmd/vellum/cmd/set.go",
"chars": 1973,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "cmd/vellum/cmd/svg.go",
"chars": 1603,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "cmd/vellum/main.go",
"chars": 712,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "common.go",
"chars": 6441,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "common_test.go",
"chars": 1294,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "data/words-1000.txt",
"chars": 6522,
"preview": "American\nCongress\nDemocrat\nI\nMr\nMrs\nPM\nRepublican\nTV\na\nability\nable\nabout\nabove\naccept\naccording\naccount\nacross\nact\nacti"
},
{
"path": "decoder_v1.go",
"chars": 7308,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "decoder_v1_test.go",
"chars": 12429,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "docs/format.md",
"chars": 4898,
"preview": "# vellum file format v1\n\nThe v1 file format for vellum has been designed by trying to understand the file format used by"
},
{
"path": "encoder_v1.go",
"chars": 5337,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "encoder_v1_test.go",
"chars": 8739,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "encoding.go",
"chars": 2355,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "example_test.go",
"chars": 1551,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "fst.go",
"chars": 7265,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "fst_iterator.go",
"chars": 8703,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "fst_iterator_test.go",
"chars": 15344,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "go.mod",
"chars": 225,
"preview": "module github.com/couchbase/vellum\n\ngo 1.12\n\nrequire (\n\tgithub.com/blevesearch/mmap-go v1.0.2\n\tgithub.com/spf13/cobra v0"
},
{
"path": "go.sum",
"chars": 3763,
"preview": "github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=\ngithub.com/armon/consul-api v0."
},
{
"path": "levenshtein/LICENSE",
"chars": 11359,
"preview": "\n Apache License\n Version 2.0, January 2004\n "
},
{
"path": "levenshtein/README.md",
"chars": 1135,
"preview": "# levenshtein\nlevenshtein automaton \n\nThis package makes it fast and simple to build a finite determinic automaton that "
},
{
"path": "levenshtein/alphabet.go",
"chars": 2666,
"preview": "// Copyright (c) 2018 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "levenshtein/alphabet_test.go",
"chars": 2722,
"preview": "// Copyright (c) 2018 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "levenshtein/benchmark_test.go",
"chars": 2101,
"preview": "// Copyright (c) 2018 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "levenshtein/dfa.go",
"chars": 6527,
"preview": "// Copyright (c) 2018 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "levenshtein/dfa_test.go",
"chars": 848,
"preview": "// Copyright (c) 2018 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "levenshtein/levenshtein.go",
"chars": 2248,
"preview": "// Copyright (c) 2018 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "levenshtein/levenshtein_nfa.go",
"chars": 6156,
"preview": "// Copyright (c) 2018 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "levenshtein/levenshtein_test.go",
"chars": 13199,
"preview": "// Copyright (c) 2018 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "levenshtein/parametric_dfa.go",
"chars": 8588,
"preview": "// Copyright (c) 2018 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "merge_iterator.go",
"chars": 4875,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "merge_iterator_test.go",
"chars": 5385,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "pack.go",
"chars": 1307,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "pack_test.go",
"chars": 1324,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "regexp/compile.go",
"chars": 7550,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "regexp/compile_test.go",
"chars": 6012,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "regexp/dfa.go",
"chars": 4248,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "regexp/inst.go",
"chars": 1410,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "regexp/regexp.go",
"chars": 3447,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "regexp/regexp_test.go",
"chars": 5078,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "regexp/sparse.go",
"chars": 1200,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "regexp/sparse_test.go",
"chars": 1050,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "registry.go",
"chars": 2660,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "registry_test.go",
"chars": 1305,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "transducer.go",
"chars": 1809,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "utf8/utf8.go",
"chars": 6129,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "utf8/utf8_test.go",
"chars": 2006,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "vellum.go",
"chars": 3425,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "vellum_mmap.go",
"chars": 1286,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "vellum_nommap.go",
"chars": 790,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "vellum_test.go",
"chars": 13005,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "vendor/github.com/edsrzf/mmap-go/LICENSE",
"chars": 1518,
"preview": "Copyright (c) 2011, Evan Shaw <edsrzf@gmail.com>\nAll rights reserved.\n\nRedistribution and use in source and binary forms"
},
{
"path": "vendor/github.com/edsrzf/mmap-go/mmap.go",
"chars": 3565,
"preview": "// Copyright 2011 Evan Shaw. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license that "
},
{
"path": "vendor/github.com/edsrzf/mmap-go/mmap_unix.go",
"chars": 1435,
"preview": "// Copyright 2011 Evan Shaw. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license that "
},
{
"path": "vendor/github.com/edsrzf/mmap-go/mmap_windows.go",
"chars": 3819,
"preview": "// Copyright 2011 Evan Shaw. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license that "
},
{
"path": "vendor/github.com/edsrzf/mmap-go/msync_netbsd.go",
"chars": 214,
"preview": "// Copyright 2011 Evan Shaw. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license that "
},
{
"path": "vendor/github.com/edsrzf/mmap-go/msync_unix.go",
"chars": 320,
"preview": "// Copyright 2011 Evan Shaw. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license that "
},
{
"path": "vendor/github.com/inconshreveable/mousetrap/LICENSE",
"chars": 551,
"preview": "Copyright 2014 Alan Shreve\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file ex"
},
{
"path": "vendor/github.com/inconshreveable/mousetrap/trap_others.go",
"chars": 483,
"preview": "// +build !windows\n\npackage mousetrap\n\n// StartedByExplorer returns true if the program was invoked by the user\n// doubl"
},
{
"path": "vendor/github.com/inconshreveable/mousetrap/trap_windows.go",
"chars": 2393,
"preview": "// +build windows\n// +build !go1.4\n\npackage mousetrap\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"syscall\"\n\t\"unsafe\"\n)\n\nconst (\n\t// defined"
},
{
"path": "vendor/github.com/inconshreveable/mousetrap/trap_windows_1.4.go",
"chars": 1185,
"preview": "// +build windows\n// +build go1.4\n\npackage mousetrap\n\nimport (\n\t\"os\"\n\t\"syscall\"\n\t\"unsafe\"\n)\n\nfunc getProcessEntry(pid in"
},
{
"path": "vendor/github.com/spf13/cobra/LICENSE.txt",
"chars": 10140,
"preview": " Apache License\n Version 2.0, January 2004\n "
},
{
"path": "vendor/github.com/spf13/cobra/bash_completions.go",
"chars": 17591,
"preview": "package cobra\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\t\"os\"\n\t\"sort\"\n\t\"strings\"\n\n\t\"github.com/spf13/pflag\"\n)\n\n// Annotations for Bash comp"
},
{
"path": "vendor/github.com/spf13/cobra/cobra.go",
"chars": 5006,
"preview": "// Copyright © 2013 Steve Francia <spf@spf13.com>.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");"
},
{
"path": "vendor/github.com/spf13/cobra/command.go",
"chars": 36953,
"preview": "// Copyright © 2013 Steve Francia <spf@spf13.com>.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");"
},
{
"path": "vendor/github.com/spf13/cobra/command_notwin.go",
"chars": 68,
"preview": "// +build !windows\n\npackage cobra\n\nvar preExecHookFn func(*Command)\n"
},
{
"path": "vendor/github.com/spf13/cobra/command_win.go",
"chars": 482,
"preview": "// +build windows\n\npackage cobra\n\nimport (\n\t\"os\"\n\t\"time\"\n\n\t\"github.com/inconshreveable/mousetrap\"\n)\n\nvar preExecHookFn ="
},
{
"path": "vendor/github.com/spf13/cobra/doc/man_docs.go",
"chars": 6640,
"preview": "// Copyright 2015 Red Hat Inc. All rights reserved.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\")"
},
{
"path": "vendor/github.com/spf13/cobra/doc/md_docs.go",
"chars": 4576,
"preview": "//Copyright 2015 Red Hat Inc. All rights reserved.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");"
},
{
"path": "vendor/github.com/spf13/cobra/doc/util.go",
"chars": 1559,
"preview": "// Copyright 2015 Red Hat Inc. All rights reserved.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\")"
},
{
"path": "vendor/github.com/spf13/cobra/doc/yaml_docs.go",
"chars": 4789,
"preview": "// Copyright 2016 French Ben. All rights reserved.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");"
},
{
"path": "vendor/github.com/spf13/pflag/LICENSE",
"chars": 1531,
"preview": "Copyright (c) 2012 Alex Ogier. All rights reserved.\nCopyright (c) 2012 The Go Authors. All rights reserved.\n\nRedistribut"
},
{
"path": "vendor/github.com/spf13/pflag/bool.go",
"chars": 3072,
"preview": "package pflag\n\nimport \"strconv\"\n\n// optional interface to indicate boolean flags that can be\n// supplied without \"=value"
},
{
"path": "vendor/github.com/spf13/pflag/bool_slice.go",
"chars": 4587,
"preview": "package pflag\n\nimport (\n\t\"io\"\n\t\"strconv\"\n\t\"strings\"\n)\n\n// -- boolSlice Value\ntype boolSliceValue struct {\n\tvalue *[]bo"
},
{
"path": "vendor/github.com/spf13/pflag/count.go",
"chars": 2792,
"preview": "package pflag\n\nimport \"strconv\"\n\n// -- count Value\ntype countValue int\n\nfunc newCountValue(val int, p *int) *countValue "
},
{
"path": "vendor/github.com/spf13/pflag/duration.go",
"chars": 3317,
"preview": "package pflag\n\nimport (\n\t\"time\"\n)\n\n// -- time.Duration Value\ntype durationValue time.Duration\n\nfunc newDurationValue(val"
},
{
"path": "vendor/github.com/spf13/pflag/flag.go",
"chars": 32474,
"preview": "// Copyright 2009 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "vendor/github.com/spf13/pflag/float32.go",
"chars": 3168,
"preview": "package pflag\n\nimport \"strconv\"\n\n// -- float32 Value\ntype float32Value float32\n\nfunc newFloat32Value(val float32, p *flo"
},
{
"path": "vendor/github.com/spf13/pflag/float64.go",
"chars": 3105,
"preview": "package pflag\n\nimport \"strconv\"\n\n// -- float64 Value\ntype float64Value float64\n\nfunc newFloat64Value(val float64, p *flo"
},
{
"path": "vendor/github.com/spf13/pflag/golangflag.go",
"chars": 2675,
"preview": "// Copyright 2009 The Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license "
},
{
"path": "vendor/github.com/spf13/pflag/int.go",
"chars": 2780,
"preview": "package pflag\n\nimport \"strconv\"\n\n// -- int Value\ntype intValue int\n\nfunc newIntValue(val int, p *int) *intValue {\n\t*p = "
},
{
"path": "vendor/github.com/spf13/pflag/int32.go",
"chars": 3013,
"preview": "package pflag\n\nimport \"strconv\"\n\n// -- int32 Value\ntype int32Value int32\n\nfunc newInt32Value(val int32, p *int32) *int32"
},
{
"path": "vendor/github.com/spf13/pflag/int64.go",
"chars": 2952,
"preview": "package pflag\n\nimport \"strconv\"\n\n// -- int64 Value\ntype int64Value int64\n\nfunc newInt64Value(val int64, p *int64) *int64"
},
{
"path": "vendor/github.com/spf13/pflag/int8.go",
"chars": 2935,
"preview": "package pflag\n\nimport \"strconv\"\n\n// -- int8 Value\ntype int8Value int8\n\nfunc newInt8Value(val int8, p *int8) *int8Value {"
},
{
"path": "vendor/github.com/spf13/pflag/int_slice.go",
"chars": 3785,
"preview": "package pflag\n\nimport (\n\t\"fmt\"\n\t\"strconv\"\n\t\"strings\"\n)\n\n// -- intSlice Value\ntype intSliceValue struct {\n\tvalue *[]int"
},
{
"path": "vendor/github.com/spf13/pflag/ip.go",
"chars": 3043,
"preview": "package pflag\n\nimport (\n\t\"fmt\"\n\t\"net\"\n\t\"strings\"\n)\n\n// -- net.IP value\ntype ipValue net.IP\n\nfunc newIPValue(val net.IP, "
},
{
"path": "vendor/github.com/spf13/pflag/ip_slice.go",
"chars": 4675,
"preview": "package pflag\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\t\"net\"\n\t\"strings\"\n)\n\n// -- ipSlice Value\ntype ipSliceValue struct {\n\tvalue *[]net"
},
{
"path": "vendor/github.com/spf13/pflag/ipmask.go",
"chars": 4052,
"preview": "package pflag\n\nimport (\n\t\"fmt\"\n\t\"net\"\n\t\"strconv\"\n)\n\n// -- net.IPMask value\ntype ipMaskValue net.IPMask\n\nfunc newIPMaskVa"
},
{
"path": "vendor/github.com/spf13/pflag/ipnet.go",
"chars": 3330,
"preview": "package pflag\n\nimport (\n\t\"fmt\"\n\t\"net\"\n\t\"strings\"\n)\n\n// IPNet adapts net.IPNet for use as a flag.\ntype ipNetValue net.IPN"
},
{
"path": "vendor/github.com/spf13/pflag/string.go",
"chars": 2925,
"preview": "package pflag\n\n// -- string Value\ntype stringValue string\n\nfunc newStringValue(val string, p *string) *stringValue {\n\t*p"
},
{
"path": "vendor/github.com/spf13/pflag/string_array.go",
"chars": 3825,
"preview": "package pflag\n\n// -- stringArray Value\ntype stringArrayValue struct {\n\tvalue *[]string\n\tchanged bool\n}\n\nfunc newString"
},
{
"path": "vendor/github.com/spf13/pflag/string_slice.go",
"chars": 4055,
"preview": "package pflag\n\nimport (\n\t\"bytes\"\n\t\"encoding/csv\"\n\t\"strings\"\n)\n\n// -- stringSlice Value\ntype stringSliceValue struct {\n\tv"
},
{
"path": "vendor/github.com/spf13/pflag/uint.go",
"chars": 2935,
"preview": "package pflag\n\nimport \"strconv\"\n\n// -- uint Value\ntype uintValue uint\n\nfunc newUintValue(val uint, p *uint) *uintValue {"
},
{
"path": "vendor/github.com/spf13/pflag/uint16.go",
"chars": 3072,
"preview": "package pflag\n\nimport \"strconv\"\n\n// -- uint16 value\ntype uint16Value uint16\n\nfunc newUint16Value(val uint16, p *uint16) "
},
{
"path": "vendor/github.com/spf13/pflag/uint32.go",
"chars": 3088,
"preview": "package pflag\n\nimport \"strconv\"\n\n// -- uint32 value\ntype uint32Value uint32\n\nfunc newUint32Value(val uint32, p *uint32) "
},
{
"path": "vendor/github.com/spf13/pflag/uint64.go",
"chars": 3085,
"preview": "package pflag\n\nimport \"strconv\"\n\n// -- uint64 Value\ntype uint64Value uint64\n\nfunc newUint64Value(val uint64, p *uint64) "
},
{
"path": "vendor/github.com/spf13/pflag/uint8.go",
"chars": 3007,
"preview": "package pflag\n\nimport \"strconv\"\n\n// -- uint8 Value\ntype uint8Value uint8\n\nfunc newUint8Value(val uint8, p *uint8) *uint8"
},
{
"path": "vendor/github.com/spf13/pflag/uint_slice.go",
"chars": 3893,
"preview": "package pflag\n\nimport (\n\t\"fmt\"\n\t\"strconv\"\n\t\"strings\"\n)\n\n// -- uintSlice Value\ntype uintSliceValue struct {\n\tvalue *[]u"
},
{
"path": "vendor/github.com/willf/bitset/LICENSE",
"chars": 1480,
"preview": "Copyright (c) 2014 Will Fitzgerald. All rights reserved.\n\nRedistribution and use in source and binary forms, with or wit"
},
{
"path": "vendor/github.com/willf/bitset/bitset.go",
"chars": 17184,
"preview": "/*\nPackage bitset implements bitsets, a mapping\nbetween non-negative integers and boolean values. It should be more\neffi"
},
{
"path": "vendor/github.com/willf/bitset/popcnt.go",
"chars": 1020,
"preview": "package bitset\n\n// bit population count, take from\n// https://code.google.com/p/go/issues/detail?id=4988#c11\n// credit: "
},
{
"path": "vendor/github.com/willf/bitset/popcnt_amd64.go",
"chars": 1166,
"preview": "// +build amd64,!appengine\n\npackage bitset\n\n// *** the following functions are defined in popcnt_amd64.s\n\n//go:noescape\n"
},
{
"path": "vendor/github.com/willf/bitset/popcnt_amd64.s",
"chars": 1725,
"preview": "// +build amd64,!appengine\n\nTEXT ·hasAsm(SB),4,$0-1\nMOVQ $1, AX\nCPUID\nSHRQ $23, CX\nANDQ $1, CX\nMOVB CX, ret+0(FP)\nRET\n\n#"
},
{
"path": "vendor/github.com/willf/bitset/popcnt_generic.go",
"chars": 421,
"preview": "// +build !amd64 appengine\n\npackage bitset\n\nfunc popcntSlice(s []uint64) uint64 {\n\treturn popcntSliceGo(s)\n}\n\nfunc popcn"
},
{
"path": "vendor/manifest",
"chars": 1180,
"preview": "{\n\t\"version\": 0,\n\t\"dependencies\": [\n\t\t{\n\t\t\t\"importpath\": \"github.com/edsrzf/mmap-go\",\n\t\t\t\"repository\": \"https://github.c"
},
{
"path": "writer.go",
"chars": 1912,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
},
{
"path": "writer_test.go",
"chars": 2024,
"preview": "// Copyright (c) 2017 Couchbase, Inc.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may "
}
]
About this extraction
This page contains the full source code of the couchbase/vellum GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 121 files (517.3 KB), approximately 156.8k tokens, and a symbol index with 1194 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.